Form Recognizer custom and composed models

Form Recognizer uses advanced machine learning technology to detect and extract information from document images and return the extracted data in a structured JSON output. With Form Recognizer, you can train standalone custom models, create composed models, or get started with our prebuilt models:

  • Custom models. Form Recognizer custom models enable you to analyze and extract data from forms and documents specific to your business. Custom models are trained for your distinct data and use cases.

  • Composed models. A composed model is created by taking a collection of custom models and assigning them to a single model that encompasses your form types. When a document is submitted to a composed model, the service performs a classification step to decide which custom model accurately represents the form presented for analysis.

  • Prebuilt models. Form Recognizer currently supports prebuilt models for business cards, layout, identity documents, invoices, and receipts.

In this article, we'll examine the process for creating Form Recognizer custom and composed models using our Form Recognizer sample labeling tool, REST APIs, or client-library SDKs.

What is a custom model?

A custom model is a machine learning program trained to recognize form fields within your distinct content and extract key-value pairs and table data. You only need five examples of the same form type to get started and your custom model can be trained with or without labeled datasets.

What is a composed model?

With composed models, you can assign multiple custom models to a composed model called with a single model ID. This is useful when you have trained several models and want to group them to analyze similar form types. For example, your composed model may be comprised of custom models trained to analyze your supply, equipment, and furniture purchase orders. Instead of manually trying to select the appropriate model, you can use a composed model to determine the appropriate custom model for each analysis and extraction.

Try it

Get started with our Form Recognizer sample labeling tool:

Create your models

The steps for building, training, and using custom and composed models are as follows:

Assemble your training dataset

Building a custom model begins with establishing your training dataset. You'll need a minimum of five completed forms of the same type for your sample dataset. They can be of different file types (jpg, png, pdf, tiff) and contain both text and handwriting. Your forms must follow the input requirements for Form Recognizer.

Upload your training dataset

You'll need to upload your training data to an Azure blob storage container. If you don't know how to create an Azure storage account with a container, see Azure Storage quickstart for Azure portal. You can use the free pricing tier (F0) to try the service, and upgrade later to a paid tier for production.

Train your custom model

You can train your model with or without labeled data sets. Unlabeled datasets rely solely on the Layout API to detect and identify key information without added human input. Labeled datasets also rely on the Layout API, but supplementary human input is included such as your specific labels and field locations. To use both labeled and unlabeled data, start with at least five completed forms of the same type for the labeled training data and then add unlabeled data to the required data set.

Create a composed model

Note

Model Compose is only available for custom models trained with labels. Attempting to compose unlabeled models will produce an error.

With the Model Compose operation, you can assign up to 100 trained custom models to a single model ID. When you call Analyze with the composed model ID, Form Recognizer will first classify the form you submitted, choose the best matching assigned model, and then return results for that model. This operation is useful when incoming forms may belong to one of several templates.

Using the Form Recognizer sample labeling tool, the REST API, or the Client-library SDKs, follow the steps below to set up a composed model:

  1. Gather your custom model IDs
  2. Compose your custom models

Gather your custom model IDs

Once the training process has successfully completed, your custom model will be assigned a model ID. You can retrieve a model ID as follows:

When you train models using the Form Recognizer sample labeling tool, the model ID is located in the Train Result window:

Screenshot: training results window.

Compose your custom models

After you have gathered your custom models corresponding to a single form type, you can compose them into a single model.

The sample labeling tool enables you to quickly get started training models and composing them to a single model ID.

After you have completed training, compose your models as follows:

  1. On the left rail menu, select the Model Compose icon (merging arrow).

  2. In the main window, select the models you wish to assign to a single model ID. Models with the arrows icon are already composed models.

  3. Choose the Compose button from the upper-left corner.

  4. In the pop-up window, name your newly composed model and select Compose.

When the operation completes, your newly composed model will appear in the list.

Screenshot: model compose window.

Analyze documents with your custom or composed model

The custom form Analyzeoperation requires you to provide the modelID in the call to Form Recognizer . You can provide a single custom model ID or a composed model ID for the modelID parameter.

  1. On the tool's left-pane menu, select the Analyze icon (lightbulb).

  2. Choose a local file or image URL to analyze.

  3. Select the Run Analysis button.

  4. The tool will apply tags in bounding boxes and report the confidence percentage for each tag.

Screenshot: Form Recognizer tool analyze-a-custom-form window.

Test your newly trained models by analyzing forms that were not part of the training dataset. Depending on the reported accuracy, you may want to do further training to improve the model. You can continue further training to improve results.

Manage your custom models

You can manage your custom models throughout their lifecycle by viewing a list of all custom models under your subscription, retrieving information about a specific custom model, and deleting custom models from your account.

Great! You have learned the steps to create custom and composed models and use them in your Form Recognizer projects and applications.

Next steps

Learn more about the Form Recognizer client library by exploring our API reference documentation.