Considerations for initial synchronization

Note

Effective November 2020:

  • Common Data Service has been renamed to Microsoft Dataverse. For more information, see Power Automate Blog.
  • Some terminology in Microsoft Dataverse has been updated. For example, entity is now table and field is now column. For more information, see Terminology updates.

This topic will be updated soon to reflect the latest terminology.

Before you start dual-write on an entity, you can run an initial synchronization to handle existing data on both sides: Finance and Operations apps and customer engagement apps. You can skip the initial synchronization if you don't have to sync data between the two environments.

The initial synchronization lets you copy existing data from one app to another bidirectionally, and there are several considerations when you run it. For example, you might have to migrate data before your go-live. In this case, data can be loaded into one side through data migration and then synced to the other side through the initial synchronization.

We recommend that you use the following approach for the initial synchronization:

  • Single-threaded entities: First migrate data into the Finance and Operations app, and then trigger the initial synchronization to move the data over to Common Data Service. Based on lab testing that Microsoft has done, this sequence has better performance than synchronization from Common Data Service to Finance and Operations apps.
  • Multi-threaded entities: First migrate data into Common Data Service, and then trigger the initial synchronization to move the data over to the Finance and Operations app.

Constraints

Data migration slow-down with enabled dual-write

If you first activate the map in dual-write and then start to import data, migration performance will be poor. We recommend that you not activate running maps in dual-write until the data migration is completed.

Limit of 500,000 records per run

The maximum number of records that is allowed through initial synchronization is 500,000 per run. The limit of 500,000 records applies to each legal entity, because each legal entity runs separately. For more information, see Integrate data into Common Data Service. In particular, pay attention to the note that states, "To optimize performance and not overload the apps, we currently limit project executions to 500k rows per execution per project."

If there must be more than 500,000 records in a run when you the initial synchronization, we recommend that you migrate data into the Finance and Operations app and Common Data Service separately, and skip the initial synchronization.

Twenty-four-hour limit

If you're running the initial synchronization from Common Data Service to the Finance and Operations app, the import result must be received back from the Finance and Operations app within 24 hours. Otherwise, a time-out occurs. Therefore, if you're syncing lots of data, and the single run takes more than 24 hours, the initial synchronization might fail because of a time-out. For example, an initial synchronization from Common Data Service to a Finance and Operations app for the Customer/Account entity involves 70,000 records. Therefore, the run might take more than 24 hours and time out.

Don't run the initial synchronization from Common Data Service to a Finance and Operations app for single-threaded entities if the data volume is more than 70,000 records. Because these entities don't support multi-threading during import, a time-out might occur if the volume is more 70,000 records. In this situation, you should migrate data into the Finance and Operations app and Common Data Service separately, and skip the initial synchronization.

Currently, there is a limit of 40 legal entities while the environments are being linked. If you try to enable maps where more than 40 legal entities are linked between the environments, you will receive the following error message:

Dual-write failure - Plugin registration failed: [(Unable to get partition map for project DWM-1ae35e60-4bc2-4905-88ea-XXXXX. Error Exceeds the maximum partitions allowed for mapping DWM-1ae35e60-4bc2-4905-88ea- XXXXX)], One or more errors occurred.

Initial synchronization isn't currently supported for entity maps that have 10 or more lookups

This limitation applies only to the initial synchronization from Common Data Service for entity maps that have 10 or more lookups. If you run the initial synchronization against an entity map that has 10 or more lookups, you might receive the following error message:

5 Attempts to get data from https://XXXX.azure-apim.net/apim... Failed

As a workaround you can split the initial sync into these steps:

  1. Remove some of the lookup fields that are not mandatory from the dual-write entity map and bring the number of lookups to 10.
  2. After the lookup fields are removed, save the map and do the initial sync.
  3. After the initial sync for the first step is successful, add the remaining lookup fields and remove the lookup fields that were synced in first step. Once again make sure the number of lookup fields is 10. Save the map and run the initial sync. Repeat these steps to make sure all the lookup fields are synced.
  4. Add all the lookup fields back to the map, save the map and run the map with skip initial sync. This will enable the map for live sync mode.

Five-minute limit for Finance and Operations data export

If you're running the initial synchronization from the Finance and Operations app to Common Data Service and the Finance and Operations data export takes more than five minutes, then the initial sync might time out. The time-out can happen if the data entity has virtual fields with the postLoad method, or the export query isn't optimized (for example, if it has missing indexes).

This type of synchronization is supported in Platform update 37 (PU37) and later. Therefore, you should update your Finance and Operations app to PU37 or later.

Error handling capabilities

Initial synchronization is always a full push

If an individual record fails to be synced, you can't resync only that individual record. The initial synchronization always pushes the whole data set. This behavior is known as a full push. If the initial synchronization only partially succeeds, a second synchronization runs for all the records, not just the records that failed to be synced during the initial synchronization.

Only the top five errors can be viewed

You can view only the top five errors from the initial synchronization error log.

Known issues

For information about known issues, see Troubleshoot issues during initial synchronization.

Guidance matrix

Finance and Operations app instance Common Data Service instance Has data to run the initial synchronization Description Maximum volume in an entity Single-threaded or multi-threaded Approach
New New No A new Finance and Operations app instance and a new customer engagement app instance, where neither app has initial data Not applicable Any
  • Activate dual-write, and skip the initial synchronization.
New New Yes A new Finance and Operations app instance and a new customer engagement app instance, where one of the apps has migrated data < 500,000 Single-threaded
  1. Migrate data to the Finance and Operations app.
  2. Run the initial synchronization.
< 500,000 Multi-threaded
  1. Migrate data to Common Data Service.
  2. Run the initial synchronization.
> 500,000 Any
  1. Migrate data to each app outside the initial synchronization.
  2. Activate dual-write, and skip the initial synchronization.
New Existing Yes A new Finance and Operations app instance and an existing customer engagement app instance < 70,000 Single-threaded
  1. Create a new company in the Finance and Operations app.
  2. Bootstrap Common Data Service for the company code.
  3. Run the initial synchronization.
> 70,000 Single-threaded
  1. Create a new company in the Finance and Operations app.
  2. Bootstrap Common Data Service for the company code.
  3. Migrate data to each app outside the initial synchronization.
  4. Activate dual-write, and skip the initial synchronization.
< 500,000 Multi-threaded
  1. Create a new company in the Finance and Operations app.
  2. Bootstrap Common Data Service for the company code.
  3. Run the initial synchronization.
> 500,000 Any
  1. Create a new company in the Finance and Operations app.
  2. Bootstrap Common Data Service for the company code.
  3. Migrate data to each app outside the initial synchronization.
  4. Activate dual-write, and skip the initial synchronization.
Existing New Yes An existing Finance and Operations app instance and a new customer engagement app instance < 500,000 Any
  • Run the initial synchronization.
> 500,000 Any
  1. Migrate data to each app.
  2. Activate dual-write, and skip the initial synchronization.
Existing Existing Yes An existing Finance and Operations app instance and an existing customer engagement app instance < 70,000 Single-threaded
  1. Bootstrap Common Data Service for the company code.
  2. Run the initial synchronization.
> 70,000 Single-threaded
  1. Bootstrap Common Data Service for the company code.
  2. Migrate data to each app outside the initial synchronization.
  3. Activate dual-write, and skip the initial synchronization.
< 500,000 Multi-threaded
  1. Bootstrap Common Data Service for the company code.
  2. Run the initial synchronization.
> 500,000 Any
  1. Bootstrap Common Data Service for the company code.
  2. Migrate data to each app outside the initial synchronization.
  3. Activate dual-write, and skip the initial synchronization.

Single-threaded entities

  • Sales tax codes (msdyn_taxcodes)
  • Customers V3 (accounts)
  • Vendors V2 (msdyn_vendors)
  • Warehouses (msdyn_warehouses)
  • Product categories (msdyn_productcategories)
  • Employment (cdm_employments)
  • Position worker assignments (cdm_positionworkerassignmentmaps)
  • Warehouse locations (msdyn_inventorylocations)
  • Modes of delivery (msdyn_shipvias)