Create a form processing custom model

After you review the requirements, you can get started creating your form processing model.

Sign in to AI Builder

  1. Sign in to Power Apps or Power Automate.

  2. In the left pane, select AI Builder > Build.

  3. Select Form processing.

  4. Type a name for your model.

  5. If you want to create your model by using your own documents, make sure that you have at least five examples that use the same layout. Otherwise, you can use sample data to create the model.

  6. Select Create.

Define information to extract

On the Choose information to extract screen, you define the fields, tables, and checkboxes you want to teach your model to extract. Select the +Add button to start defining these.

  • For each field, provide a name you would like the field to have in the model.

    'Define fields' animation.

  • For each table, provide the name you would like the table to have. Also, define the different columns that the model should extract.

    'Define tables' animation.

  • For each checkbox, provide a name you would like the checkbox to have in the model. Define separate checkboxes for each item that can be checked in a document.

    'Define checkboxes' animation.

Group documents by collections

A collection is a group of documents that share the same layout. Create as many collections as document layouts that you want your model to process. For example, if you're building an AI model to process invoices from two different vendors, each having their own invoice template, create two collections.

'Add collections of documents' animation.

For each collection that you create, you need to upload at least five sample documents per collection. Files with formats JPG, PNG, and PDF files are currently accepted.

'Select documents to upload' animation.

Note

You can create up to 200 collections per model.

Tag documents

By tagging the documents you've uploaded, you're teaching your AI Builder model to extract the fields and tables you've specified.

To start tagging, select one of the collections on the right panel.

Tag fields

To tag a field, draw a rectangle around the field you're interested in and select the field name that it corresponds to.

'Draw a rectangle to select a field' animation.

At any time, you can resize to adjust your selection.

When you hover over words in your documents, light blue boxes may appear. These indicate that you can draw a rectangle around those words to select a field.

Select fields.

Tag tables

  1. Draw a rectangle around the table in the document you're interested in, and then select the table name that it corresponds to. The content of the panel on the right will change.

  2. Draw rows by left-clicking between row separators.

  3. Draw columns by pressing Ctrl + left-click.

  4. Once the rows and columns have been set, assign the headers to extract by selecting the header column and mapping it to the desired one.

  5. A preview of how the table will be extracted appears on the panel on the right.

  6. If the header of the table has been tagged, select Ignore first row so the header of the table isn't extracted as the table content.

The following animation illustrates the process:

'Tag table' animation.

An alternative way to define the rows and columns for a table is by selecting Delimit rows and columns at the top of the screen.

Screenshot of Delimit rows and columns option.

Use the advanced tagging mode

Advanced tagging mode allows you to tag tables at the cell level. Use this mode for complex tables like:

  • Tables that are skewed, where tagging with a grid is not possible.
  • When you need to extract nested items, like an item within a cell.

Given the table from the following example, to extract the unit price, we'll define it as a separate column on the Choose information to extract step. We define Description, Unit price, Quantity, and Amount each as a column of the table and tag them accordingly using advanced tagging mode. See the animation below.

Screenshot of delimit rows and columns screen.

You can start tagging in the default mode to quickly capture all rows and columns. Then switch to advanced mode to adjust each cell and tag nested items.

Nested items in tables

You can tag items that are nested within a row by defining these as columns. Given the table from the example below, to extract the unit price, define it as a separate column on the Choose information to extract step earlier in this topic. Define Description, Unit price, Quantity, and Amount each as a column of the table and then tag them accordingly.

'Tag tables' animation.

Multipage tables

AI Builder supports extracting tables that span across multiple pages as a single table with an experimental feature. For details, go to Process multipage tables in form processor (experimental).

If you don't want to try the experimental feature, you can extract tables from different pages by defining each page as a separate table in the Choose information to extract step. For example, if you have a document with a table that spans over two pages, you'll need to define them as two separate tables.

Tag checkboxes

To tag a checkbox, draw a rectangle around the checkbox you're interested in extracting and select the checkbox name that it corresponds to.

'Tag checkbox' animation.

If the quality of the document is low, AI Builder might not be able to detect the checkbox. If you're unable to tag a checkbox, do the following:

  1. On the panel on the right, select the three dots next to the checkbox you want to extract.

  2. Select Not available in document.

AI Builder supports detection and extraction of selection marks such as checkboxes and radio buttons, with different markers to indicate whether the selection is marked or not.

Field, checkbox, or table not in document

If a field, checkbox, or table isn't present in one of the documents you've uploaded for training, select Not available in document on the panel to the right, next to a field, checkbox, or table.

Select Analyze.

Tag all documents across all collections

All the documents that you've uploaded are presented for you to tag. Some of the fields might be automatically detected in successive documents. In that case, confirm that the selection is correct.

If you've created multiple collections, tag all documents across all the collections.

Next step

Train and publish your form processing model