Neural Network

class coremltools.models.neural_network.NeuralNetworkBuilder(input_features=None, output_features=None, mode=None, spec=None, nn_spec=None, disable_rank5_shape_mapping=False, training_features=None, use_float_arraytype=False)

Neural network builder class to construct Core ML models.

The NeuralNetworkBuilder constructs a Core ML neural network specification layer by layer. The layers should be added in such an order that the inputs to each layer (referred to as blobs) of each layer has been previously defined. The builder can also set pre-processing steps to handle specialized input format (e.g. images), and set class labels for neural network classifiers. Refer to the protobuf messages in specification (NeuralNetwork.proto) for more details.

See also

MLModel, datatypes, save_spec

Examples

from coremltools.models.neural_network import datatypes, NeuralNetworkBuilder
from coremltools.models.utils import save_spec

# Create a neural network binary classifier that classifies
# 3-dimensional data points
# Specify input and output dimensions
>>> input_dim = (3,)
>>> output_dim = (2,)

# Specify input and output features
>>> input_features = [('data', datatypes.Array(*input_dim))]
>>> output_features = [('probs', datatypes.Array(*output_dim))]

# Build a simple neural network with 1 inner product layer
>>> builder = NeuralNetworkBuilder(input_features, output_features)
>>> builder.add_inner_product(name='ip_layer', W=weights, b=bias, input_channels=3, output_channels=2,
... has_bias=True, input_name='data', output_name='probs')

# save the spec by the builder
>>> save_spec(builder.spec, 'network.mlmodel')

Utilities to annotate Neural Network Features with flexible shape information. Only available in coremltools 2.0b1 and onwards

class coremltools.models.neural_network.flexible_shape_utils.NeuralNetworkImageSize(height=None, width=None)

An object representing a size for an image feature inside a neural network. Valid sizess for height and width are > 0.

__init__(height=None, width=None)

Initialize self. See help(type(self)) for accurate signature.

class coremltools.models.neural_network.flexible_shape_utils.NeuralNetworkImageSizeRange(height_range=None, width_range=None)

An object representing a range of sizes for an image feature inside a neural network. Valid ranges for height and width are > 0. A "-1" upper bound value for either width or height represents an unbounded size for that dimension.

__init__(height_range=None, width_range=None)

Initialize self. See help(type(self)) for accurate signature.

class coremltools.models.neural_network.flexible_shape_utils.NeuralNetworkMultiArrayShape(channel=None, height=None, width=None)

An object representing a shape for a multiArray feature in a neural network. Valid shapes must have have only the Channel [C] shape or the Channel, Height and Width [C, H, W] shapes populated

__init__(channel=None, height=None, width=None)

Initialize self. See help(type(self)) for accurate signature.

class coremltools.models.neural_network.flexible_shape_utils.NeuralNetworkMultiArrayShapeRange(input_ranges=None)

An object representing a range of shapes for a multiArray feature in a neural network. Valid shape ranges must have have only the Channel [C] range or the Channel, Height and Width [C, H, W] ranges populated. A "-1" value in an upper bound represents an unbounded range.

__init__(input_ranges=None)

Initialize self. See help(type(self)) for accurate signature.

isFlexible()

Returns true if any one of the channel, height, or width ranges of this shape allow more than one input value.

coremltools.models.neural_network.flexible_shape_utils.add_enumerated_image_sizes(spec, feature_name, sizes)

Annotate an input or output image feature in a Neural Network spec to to accommodate a list of enumerated image sizes

Parameters
  • spec -- MLModel The MLModel spec containing the feature

  • feature_name -- str The name of the image feature for which to add size information. If the feature is not found in the input or output descriptions then an exception is thrown

  • sizes -- [] | NeuralNetworkImageSize A single or a list of NeuralNetworkImageSize objects which encode valid size information for a image feature

Examples

>>> import coremltools
>>> from coremltools.models.neural_network import flexible_shape_utils
>>> spec = coremltools.utils.load_spec('mymodel.mlmodel')
>>> image_sizes = [flexible_shape_utils.NeuralNetworkImageSize(128, 128)]
>>> image_sizes.append(flexible_shape_utils.NeuralNetworkImageSize(256, 256))
>>> flexible_shape_utils.add_enumerated_image_sizes(spec, feature_name='my_multiarray_featurename', sizes=image_sizes)
Returns

None. The spec object is updated

coremltools.models.neural_network.flexible_shape_utils.add_enumerated_multiarray_shapes(spec, feature_name, shapes)

Annotate an input or output multiArray feature in a Neural Network spec to to accommodate a list of enumerated array shapes

Parameters
  • spec -- MLModel The MLModel spec containing the feature

  • feature_name -- str The name of the image feature for which to add shape information. If the feature is not found in the input or output descriptions then an exception is thrown

  • shapes -- [] | NeuralNetworkMultiArrayShape A single or a list of NeuralNetworkImageSize objects which encode valid size information for a image feature

Examples

>>> import coremltools
>>> from coremltools.models.neural_network import flexible_shape_utils
>>> spec = coremltools.utils.load_spec('mymodel.mlmodel')
>>> array_shapes = [flexible_shape_utils.NeuralNetworkMultiArrayShape(3)]
>>> second_shape = flexible_shape_utils.NeuralNetworkMultiArrayShape()
>>> second_shape.set_channel_shape(3)
>>> second_shape.set_height_shape(10)
>>> second_shape.set_width_shape(15)
>>> array_shapes.append(second_shape)
>>> flexible_shape_utils.add_enumerated_multiarray_shapes(spec, feature_name='my_multiarray_featurename', shapes=array_shapes)
Returns

None. The spec object is updated

coremltools.models.neural_network.flexible_shape_utils.add_multiarray_ndshape_enumeration(spec, feature_name, enumerated_shapes)

Annotate an input or output MLMultiArray feature in a Neural Network spec to accommodate a range of shapes. Add provided enumerated shapes to the list of shapes already present. This method is different from "add_enumerated_multiarray_shapes", which is applicable for rank 5 mapping, SBCHW, arrays.

Parameters
  • spec -- MLModel The MLModel spec containing the feature

  • feature_name -- str The name of the feature for which to add shape range information. If the feature is not found in the input or output descriptions then an exception is thrown

  • enumerated_shapes -- List[Tuple(int)] list of shapes, where each shape is specified as a tuple of integers.

Examples

>>> import coremltools
>>> from coremltools.models.neural_network import flexible_shape_utils
>>> spec = coremltools.utils.load_spec('mymodel.mlmodel')
>>> # say, the default shape of "my_multiarray_featurename" is (2,3)
>>> flexible_shape_utils.add_multiarray_ndshape_enumeration(spec, feature_name='my_multiarray_featurename', enumerated_shapes=[(2,4), (2,6)])
Returns

None. The spec is updated

coremltools.models.neural_network.flexible_shape_utils.set_multiarray_ndshape_range(spec, feature_name, lower_bounds, upper_bounds)

Annotate an input or output MLMultiArray feature in a Neural Network spec to accommodate a range of shapes. This is different from "update_multiarray_shape_range", which works with rank 5 SBCHW mapping.

Parameters
  • spec -- MLModel The MLModel spec containing the feature

  • feature_name -- str The name of the feature for which to add shape range information. If the feature is not found in the input or output descriptions then an exception is thrown

  • lower_bounds -- List[int] list of integers specifying the lower bounds of each dimension. Length must be same as the rank (length of shape) of the feature_name.

  • upper_bounds -- List[int] list of integers specifying the upper bounds of each dimension. -1 corresponds to unbounded range. Length must be same as the rank (length of shape) of the feature_name.

Examples

>>> import coremltools
>>> from coremltools.models.neural_network import flexible_shape_utils
>>> spec = coremltools.utils.load_spec('mymodel.mlmodel')
>>> # say, the default shape of "my_multiarray_featurename" is (2,3)
>>> flexible_shape_utils.set_multiarray_ndshape_range(spec, feature_name='my_multiarray_featurename', lower_bounds=[1,2], upper_bounds=[10,-1])
Returns

None. The spec is updated

coremltools.models.neural_network.flexible_shape_utils.update_image_size_range(spec, feature_name, size_range)

Annotate an input or output Image feature in a Neural Network spec to to accommodate a range of image sizes

Parameters
  • spec -- MLModel The MLModel spec containing the feature

  • feature_name -- str The name of the Image feature for which to add shape information. If the feature is not found in the input or output descriptions then an exception is thrown

  • size_range -- NeuralNetworkImageSizeRange A NeuralNetworkImageSizeRange object with the populated image size range information.

Examples

>>> import coremltools
>>> from coremltools.models.neural_network import flexible_shape_utils
>>> spec = coremltools.utils.load_spec('mymodel.mlmodel')
>>> img_size_ranges = flexible_shape_utils.NeuralNetworkImageSizeRange()
>>> img_size_ranges.add_height_range(64, 128)
>>> img_size_ranges.add_width_range(128, -1)
>>> flexible_shape_utils.update_image_size_range(spec, feature_name='my_multiarray_featurename', size_range=img_size_ranges)
Returns

None. The spec object is updated

coremltools.models.neural_network.flexible_shape_utils.update_multiarray_shape_range(spec, feature_name, shape_range)

Annotate an input or output MLMultiArray feature in a Neural Network spec to accommodate a range of shapes

Parameters
  • spec -- MLModel The MLModel spec containing the feature

  • feature_name -- str The name of the feature for which to add shape range information. If the feature is not found in the input or output descriptions then an exception is thrown

  • shape_range -- NeuralNetworkMultiArrayShapeRange A NeuralNetworkMultiArrayShapeRange object with the populated shape range information. The shape_range object must either contain only shape information for channel or channel, height and width. If the object is invalid then an exception is thrown

Examples

>>> import coremltools
>>> from coremltools.models.neural_network import flexible_shape_utils
>>> spec = coremltools.utils.load_spec('mymodel.mlmodel')
>>> shape_range = flexible_shape_utils.NeuralNetworkMultiArrayShapeRange()
>>> shape_range.add_channel_range((1, 3))
>>> shape_range.add_width_range((128, 256))
>>> shape_range.add_height_range((128, 256))
>>> flexible_shape_utils.update_multiarray_shape_range(spec, feature_name='my_multiarray_featurename', shape_range=shape_range)
Returns

None. The spec is updated

Utilities to compress Neural Network Models. Only available in coremltools 2.0b1 and onwards

class coremltools.models.neural_network.quantization_utils.AdvancedQuantizedLayerSelector(skip_layer_types=[], minimum_conv_kernel_channels=4, minimum_conv_weight_count=4096)

Quantized layer selector allowing the user to specify some types of layers to skip during quantization process and the minimum size parameters in quantized convolution layers.

Examples

from coremltools.models.neural_network.quantization_utils import AdvancedQuantizedLayerSelector
selector = AdvancedQuantizedLayerSelector(
        skip_layer_types=['batchnorm', 'bias', 'depthwiseConv'],
        minimum_conv_kernel_channels=4,
        minimum_conv_weight_count=4096)
quantized_model = quantize_weights(model, 8, selector=selector)
__init__(skip_layer_types=[], minimum_conv_kernel_channels=4, minimum_conv_weight_count=4096)

Initialize self. See help(type(self)) for accurate signature.

do_quantize(layer, weight_param=None)

weight_param - should be name of the WeightParam field

class coremltools.models.neural_network.quantization_utils.MatrixMultiplyLayerSelector(minimum_weight_count=1, minimum_input_channels=1, minimum_output_channels=1, maximum_input_channels=None, maximum_output_channels=None, include_layers_with_names=None)

Layer selector object that allows users to select matrix multiplication layers with one of the matrices being constant, based on some criterions like total numbers of parameters/weights, number of input or output channels and/or layer names. If any of the criterion is not valid, the corresponding layer is not selected.

__init__(minimum_weight_count=1, minimum_input_channels=1, minimum_output_channels=1, maximum_input_channels=None, maximum_output_channels=None, include_layers_with_names=None)

Initialize self. See help(type(self)) for accurate signature.

do_quantize(layer, weight_param=None)

weight_param - should be name of the WeightParam field

class coremltools.models.neural_network.quantization_utils.ModelMetrics(spec)

A utility class to hold evaluation metrics

__init__(spec)

Initialize self. See help(type(self)) for accurate signature.

class coremltools.models.neural_network.quantization_utils.OutputMetric(name, type)

Utility class to calculate and hold metrics between two model outputs

__init__(name, type)

Initialize self. See help(type(self)) for accurate signature.

class coremltools.models.neural_network.quantization_utils.QuantizedLayerSelector

This is the base class to implement custom selectors to skip certain layers during quantization. To implement a custom selector, create a class that inherits this class and override do_quantize() method.

Examples

class MyLayerSelector(QuantizedLayerSelector):
    def __init__(self):
        super(MyLayerSelector, self).__init__()

    def do_quantize(self, layer, **kwargs):
        ret = super(MyLayerSelector, self).do_quantize(layer)
        if not ret or layer.name == 'dense_2':
            return False
        return True

selector = MyLayerSelector()
quantized_model = quantize_weights(mlmodel, 8, quantization_mode='linear', selector=selector)
__init__()

Initialize self. See help(type(self)) for accurate signature.

coremltools.models.neural_network.quantization_utils.activate_int8_int8_matrix_multiplications(spec, selector=None)

Utility function that takes in either a full precision (float) spec or an nbit quantized spec to selectively enable int8 activation + weight quantization of matrix multiplication operations where the second matrix represents a constant weight.

spec: MLModel.get_spec()

Currently conversion for only neural network models is supported. If a pipeline model is passed in then all embedded neural network models embedded within will be modified.

selector: (optional) MatrixMultiplyLayerSelector

A MatrixMultiplyLayerSelector object that enables int8 activation + weight quantization only on those layers for which the user-specified criterion on the minimum/maximum number of size/channels in constant weight parameters is met. It can also be derived to provide custom selection.

coremltools.models.neural_network.quantization_utils.compare_models(full_precision_model, quantized_model, sample_data)

Utility function to compare the performance of a full precision vs quantized model

full_precision_model: MLModel

The full precision model with float32 weights

quantized_model: MLModel

Quantized version of the model with quantized weights

sample_data: str | [dict]

Data used to characterize performance of the quantized model in comparison to the full precision model. Either a list of sample input dictionaries or an absolute path to a directory containing images. Path to a directory containing images is only valid for models with one image input. For all other models a list of sample inputs must be provided.

Returns

None. Performance metrics are printed out

coremltools.models.neural_network.quantization_utils.quantize_weights(full_precision_model, nbits, quantization_mode='linear', sample_data=None, **kwargs)

Utility function to convert a full precision (float) MLModel to a nbit quantized MLModel (float16).

full_precision_model: MLModel

Model which will be converted to half precision. Currently conversion for only neural network models is supported. If a pipeline model is passed in then all embedded neural network models embedded within will be converted.

nbits: int
Number of bits per quantized weight. Only 16-bit float point and

1-8 bit is supported

quantization_mode: str

One of the following:

"linear":

Linear quantization with scale and bias assuming the range of weight values is [A, B], where A = min(weight), B = max(weight)

"linear_lut":

Simple linear quantization represented as a lookup table

"kmeans_lut":

LUT based quantization, where LUT is generated by K-Means clustering

"custom_lut":

LUT quantization where LUT and quantized weight params are calculated using a custom function. If this mode is selected then a custom function must be passed in kwargs with key lut_function. The function must have input params (nbits, wp) where nbits is the number of quantization bits and wp is the list of weights for a given layer. The function should return two parameters (lut, qw) where lut is an array of length (2^n bits)containing LUT values and qw is the list of quantized weight parameters. See _get_linear_lookup_table_and_weight for a sample implementation.

"linear_symmetric":

Linear quantization with scale and bias assuming the range of weight values is [-A, A], where A = max(abs(weight)).

sample_data: str | [dict]

Data used to characterize performance of the quantized model in comparison to the full precision model. Either a list of sample input dictionaries or an absolute path to a directory containing images. Path to a directory containing images is only valid for models with one image input. For all other models a list of sample inputs must be provided.

kwargs: keyword arguments
lut_function(callable function)

A callable function provided when quantization mode is set to _QUANTIZATION_MODE_CUSTOM_LOOKUP_TABLE. See quantization_mode for more details.

selector: QuantizedLayerSelector

A QuanatizedLayerSelector object that can be derived to provide custom quantization selection.

Returns
model: MLModel

The quantized MLModel instance if running on macOS 10.14 or later, otherwise the quantized model specification is returned

Examples

>>> import coremltools
>>> from coremltools.models.neural_network import quantization_utils
>>> model = coremltools.models.MLModel('my_model.mlmodel')
>>> quantized_model = quantization_utils.quantize_weights(model, 8, "linear")

Neural Network optimizer utilities.

class coremltools.models.neural_network.update_optimizer_utils.AdamParams(lr=0.01, batch=10, beta1=0.9, beta2=0.999, eps=1e-08)

Adam - A Method for Stochastic Optimization.

Attributes
lr: float

The learning rate that controls learning step size. Adjustable in progress, default: 0.01.

batch: int

The mini-batch size, number of examples used to compute single gradient step, default: 10.

beta1: float

Controls the exponential decay rate for the first moment estimates, default: 0.9.

beta2: float

Controls the exponential decay rate for the second moment estimates, default: 0.999.

eps: float

The epsilon, a very small number to prevent any division by zero in the implementation, default: 1e-8.

Methods

set_lr(value, min, max)

Set value for learning rate.

set_batch(value, allow_set)

Set value for batch size.

set_beta1(value, min, max)

Set value for beta1.

set_beta2(value, min, max)

Set value for beta2.

set_eps(value, min, max)

Set value for epsilon.

__init__(lr=0.01, batch=10, beta1=0.9, beta2=0.999, eps=1e-08)

Initialize self. See help(type(self)) for accurate signature.

class coremltools.models.neural_network.update_optimizer_utils.SgdParams(lr=0.01, batch=10, momentum=0)

SGD - Stochastic Gradient Descent optimizer.

Attributes
lr: float

The learning rate that controls learning step size. Adjustable in progress, default: 0.01.

batch: int

The mini-batch size, number of examples used to compute single gradient step, default: 10.

momentum: float

The momentum factor that helps accelerate gradients vectors in the right direction, default 0.

Methods

set_lr(value, min, max)

Set value for learning rate.

set_batch(value, allow_set)

Set value for batch size.

set_momentum(value, min, max)

Set value for momentum.

__init__(lr=0.01, batch=10, momentum=0)

Initialize self. See help(type(self)) for accurate signature.