Create a document processing custom model

After you review the requirements, you can get started creating your document processing model.

Sign in to AI Builder

  1. Sign in to Power Apps or Power Automate.

  2. On the left pane, select AI Builder > Explore.

  3. Select Extract custom information from documents.

  4. Select Get Started.

  5. A step-by-step wizard will walk you through the process by asking you to list all data you want to extract from your document. If you want to create your model by using your own documents, make sure you have at least five examples that use the same layout. Otherwise, you can use sample data to create the model.

  6. Select Train.

  7. Test the model by selecting Quick test.

Select the type of document

On the Choose document type step, select the type of document you want to build an AI model to automate data extraction. There are two options:

  • Structured and semi-structured documents. Structured and semi-structured documents are those where for a given layout, the fields, tables, checkboxes, and other items can be found in similar places. Examples of structured and semi-structured documents are invoices, purchase orders, delivery orders, tax documents, and more.

  • Unstructured and free-form documents (preview). Unstructured documents are those where there's no set structure, usually free documents with a varying number of paragraphs. Examples of unstructured documents are contracts, statement of work, letters, and more. This document type is currently in preview. We welcome your feedback on this new preview functionality. You can post your feedback on the AI Builder forum.

    Choose document type

Define information to extract

On the Choose information to extract screen, define the fields, tables, and checkboxes you want to teach your model to extract. Select the +Add button to start defining these.

  • For each field, provide a name you would like the field to have in the model.

    'Define fields' animation.

  • For each table, provide the name you would like the table to have. Also, define the different columns that the model should extract.

    'Define tables' animation.

    Note

    Table extraction isn't currently available when choosing ‘Unstructured and free-form documents’ as the document type.

  • For each checkbox, provide a name you would like the checkbox to have in the model. Define separate checkboxes for each item that can be checked in a document.

    'Define checkboxes' animation.

Group documents by collections

A collection is a group of documents that share the same layout. Create as many collections as document layouts that you want your model to process. For example, if you're building an AI model to process invoices from two different vendors, each having their own invoice template, create two collections.

'Add collections of documents' animation.

For each collection that you create, you need to upload at least five sample documents per collection. Files with formats JPG, PNG, and PDF files are currently accepted.

'Select documents to upload' animation.

Note

You can create up to 200 collections per model.

Tag documents

By tagging the documents you've uploaded, you're teaching your AI Builder model to extract the fields and tables you've specified.

To start tagging, select one of the collections on the right panel.

Tag fields

To tag a field, draw a rectangle around the field you're interested in and select the field name that it corresponds to.

'Draw a rectangle to select a field' animation.

At any time, you can resize to adjust your selection.

If a field ends on one line and begins on another line, you can tag it using the ‘Continue tagging’ option.

Continue tagging

You can also use the Control key in Windows or Command ⌘ in macOS to tag multiple words at once. You can also use the Shift key to select the first and last word to tag all words between the two.

When you hover over words in your documents, light blue boxes may appear. These indicate that you can draw a rectangle around those words to select a field.

Select fields.

Tag tables

  1. Draw a rectangle around the table in the document you're interested in, and then select the table name that it corresponds to. The content of the panel on the right will change.

  2. Draw rows by left-clicking between row separators.

  3. Draw columns by pressing Ctrl + left-click.

  4. Once the rows and columns have been set, assign the headers to extract by selecting the header column and mapping it to the desired one.

  5. A preview of how the table will be extracted appears on the panel on the right.

  6. If the header of the table has been tagged, select Ignore first row so the header of the table isn't extracted as the table content.

The following animation illustrates the process:

'Tag table' animation.

An alternative way to define the rows and columns for a table is by selecting Delimit rows and columns at the top of the screen.

Screenshot of Delimit rows and columns option.

Use the advanced tagging mode

Advanced tagging mode allows you to tag tables at the cell level. Use this mode for complex tables like:

  • Tables that are skewed, where tagging with a grid isn't possible.
  • When you need to extract nested items, like an item within a cell.

Given the table from the following example, to extract the unit price, we'll define it as a separate column on the Choose information to extract step. We define Description, Unit price, Quantity, and Amount each as a column of the table and tag them accordingly using advanced tagging mode. See the animation below.

Screenshot of delimit rows and columns screen.

You can start tagging in the default mode to quickly capture all rows and columns. Then switch to advanced mode to adjust each cell and tag nested items.

Nested items in tables

You can tag items that are nested within a row by defining these as columns. Given the table from the example below, to extract the unit price, define it as a separate column on the Choose information to extract step earlier in this topic. Define Description, Unit price, Quantity, and Amount each as a column of the table and then tag them accordingly.

'Tag tables' animation.

Multipage tables

AI Builder supports extracting tables that span across multiple pages as a single table with an experimental feature. For details, go to Process multipage tables in form processor (experimental).

If you don't want to try the experimental feature, you can extract tables from different pages by defining each page as a separate table in the Choose information to extract step. For example, if you have a document with a table that spans over two pages, you'll need to define them as two separate tables.

Tag checkboxes

To tag a checkbox, draw a rectangle around the checkbox you're interested in extracting and select the checkbox name that it corresponds to.

'Tag checkbox' animation.

If the quality of the document is low, AI Builder might not be able to detect the checkbox. If you're unable to tag a checkbox, do the following:

  1. On the panel on the right, select the three dots next to the checkbox you want to extract.

  2. Select Not available in document.

AI Builder supports detection and extraction of selection marks such as checkboxes and radio buttons, with different markers to indicate whether the selection is marked or not.

Field, checkbox, or table not in document

If a field, checkbox, or table isn't present in one of the documents you've uploaded for training, select Not available in document on the panel to the right, next to a field, checkbox, or table.

Select Analyze.

Tag all documents across all collections

All the documents that you've uploaded are presented for you to tag. If you've created multiple collections, tag all documents across all the collections.

Next step

Train and publish your document processing model