Share via


Introduction to CoreML in Xamarin.iOS

CoreML brings machine learning to iOS – apps can take advantage of trained machine learning models to perform all sorts of tasks, from problem solving to image recognition.

This introduction covers the following:

Getting Started with CoreML

These steps describe how to add CoreML to an iOS project.

Mars Habitat Price Predictor sample screenshot

1. Add the CoreML model to the project

Add a CoreML model (a file with the .mlmodel extension) to the Resources directory of the project.

In the model file's properties, its Build action is set to CoreMLModel. This means that it will be compiled into an .mlmodelc file when the application is built.

2. Load the model

Load the model using the MLModel.Create static method:

var assetPath = NSBundle.MainBundle.GetUrlForResource("NameOfModel", "mlmodelc");
model = MLModel.Create(assetPath, out NSError error1);

3. Set the parameters

Model parameters are passed in and out using a container class that implements IMLFeatureProvider.

Feature provider classes behave like a dictionary of string and MLFeatureValues, where each feature value could be a simple string or number, an array or data, or a pixel buffer containing an image.

Code for a single-value feature provider is shown below:

public class MyInput : NSObject, IMLFeatureProvider
{
  public double MyParam { get; set; }
  public NSSet<NSString> FeatureNames => new NSSet<NSString>(new NSString("myParam"));
  public MLFeatureValue GetFeatureValue(string featureName)
  {
    if (featureName == "myParam")
      return MLFeatureValue.FromDouble(MyParam);
    return MLFeatureValue.FromDouble(0); // default value
  }

Using classes like this, input parameters can be provided in a way that is understood by CoreML. The names of the features (such as myParam in the code example) must match what the model expects.

4. Run the model

Using the model requires that the feature provider to be instantiated and parameters set, then that the GetPrediction method be called:

var input = new MyInput {MyParam = 13};
var outFeatures = model.GetPrediction(inputFeatures, out NSError error2);

5. Extract the results

The prediction result outFeatures is also an instance of IMLFeatureProvider; output values can be accessed using GetFeatureValue with the name of each output parameter (such as theResult), as in this example:

var result = outFeatures.GetFeatureValue("theResult").DoubleValue; // eg. 6227020800

Using CoreML with the Vision Framework

CoreML can also be used in conjunction with the Vision framework to perform operations on image, such as shape recognition, object identification, and other tasks.

The steps below describe how CoreML and Vision can be used together. The sample combines the rectangles recognition from the Vision framework with the MNINSTClassifier CoreML model to identify a handwritten digit in a photograph.

Image recognition of number 3 Image recognition of number 5

1. Create a Vision CoreML model

The CoreML model MNISTClassifier is loaded and then wrapped in a VNCoreMLModel which makes the model available for Vision tasks. This code also creates two Vision requests: first for finding rectangles in an image, and then for processing a rectangle with the CoreML model:

// Load the ML model
var bundle = NSBundle.MainBundle;
var assetPath = bundle.GetUrlForResource("MNISTClassifier", "mlmodelc");
NSError mlErr, vnErr;
var mlModel = MLModel.Create(assetPath, out mlErr);
var model = VNCoreMLModel.FromMLModel(mlModel, out vnErr);

// Initialize Vision requests
RectangleRequest = new VNDetectRectanglesRequest(HandleRectangles);
ClassificationRequest = new VNCoreMLRequest(model, HandleClassification);

The class still needs to implement the HandleRectangles and HandleClassification methods for the Vision requests, shown in steps 3 and 4 below.

2. Start the Vision processing

The following code starts processing the request. In the CoreMLVision sample, this code runs after the user has selected an image:

// Run the rectangle detector, which upon completion runs the ML classifier.
var handler = new VNImageRequestHandler(ciImage, uiImage.Orientation.ToCGImagePropertyOrientation(), new VNImageOptions());
DispatchQueue.DefaultGlobalQueue.DispatchAsync(()=>{
  handler.Perform(new VNRequest[] {RectangleRequest}, out NSError error);
});

This handler passes the ciImage to the Vision framework VNDetectRectanglesRequest that was created in step 1.

3. Handle the results of Vision processing

Once the rectangle detection is complete, it executes the HandleRectangles method, which crops the image to extract the first rectangle, converts the rectangle image to greyscale, and passes it to the CoreML model for classification.

The request parameter passed to this method contains the details of the Vision request, and using the GetResults<VNRectangleObservation>() method, it returns a list of rectangles found in the image. The first rectangle observations[0] is extracted and passed to the CoreML model:

void HandleRectangles(VNRequest request, NSError error) {
  var observations = request.GetResults<VNRectangleObservation>();
  // ... omitted error handling ...
  var detectedRectangle = observations[0]; // first rectangle
  // ... omitted cropping and greyscale conversion ...
  // Run the Core ML MNIST classifier -- results in handleClassification method
  var handler = new VNImageRequestHandler(correctedImage, new VNImageOptions());
  DispatchQueue.DefaultGlobalQueue.DispatchAsync(() => {
    handler.Perform(new VNRequest[] {ClassificationRequest}, out NSError err);
  });
}

The ClassificationRequest was initialized in step 1 to use the HandleClassification method defined in the next step.

4. Handle the CoreML

The request parameter passed to this method contains the details of the CoreML request, and using the GetResults<VNClassificationObservation>() method, it returns a list of possible results ordered by confidence (highest confidence first):

void HandleClassification(VNRequest request, NSError error){
  var observations = request.GetResults<VNClassificationObservation>();
  // ... omitted error handling ...
  var best = observations[0]; // first/best classification result
  // render in UI
  DispatchQueue.MainQueue.DispatchAsync(()=>{
    ClassificationLabel.Text = $"Classification: {best.Identifier} Confidence: {best.Confidence * 100f:#.00}%";
  });
}