Synchronize Dynamics 365 Customer Engagement (on-premises) data with external systems

Sometimes you’ll need to synchronize and integrate Dynamics 365 Customer Engagement (on-premises) data with data that is stored in other systems. The common data integration patterns include taking data from an external system and pushing it into Dynamics 365 Customer Engagement (on-premises), taking data from Dynamics 365 Customer Engagement (on-premises) and synchronizing it to some external data store, or updating Dynamics 365 Customer Engagement (on-premises) with external data. You can now use several new capabilities to make it easier to write code to achieve these scenarios.

These new features can be used separately as needed in any situation, but together they address common issues related to synchronizing and integrating data with external data. The following table introduces these new features.

Feature	Description
Removing specialized messages	Dynamics 365 Customer Engagement (on-premises) has a number of specialized messages for specific operations that update records. These messages are deprecated in this release and you should now simply use Update to perform the same operations. The deprecated messages are: - Assign - SetParentSystemUser - SetParentTeam - SetParentBusinessUnit - SetBusinessEquipment - SetBusinessUnit - SetState Simply updating the record is much simpler than using these messages and should streamline your development for data integration and synchronization scenarios. More information: Perform specialized operations using Update
Alternate Keys	In enterprise deployments of Dynamics 365 Customer Engagement (on-premises), it’s common for data from external enterprise systems to be loaded into Dynamics 365 Customer Engagement (on-premises) so that it can be presented to users. These external systems often can’t be extended to store the Dynamics 365 Customer Engagement (on-premises) record identifiers, known as GUIDs, required for system synchronization. A common solution is to add a custom attribute to an entity in Dynamics 365 Customer Engagement (on-premises) that can be used to store the identifier of the related record in the external system. When you build data load processes that update records in Dynamics 365 Customer Engagement (on-premises) and assign references to related records in Dynamics 365 Customer Engagement (on-premises), you first have to make an extra Dynamics 365 Customer Engagement (on-premises) web service call to retrieve the target Dynamics 365 Customer Engagement (on-premises) record based on this external identifier. This lookup can be slow if an appropriate index is not in place for the custom attribute, and in Dynamics 365 Customer Engagement (on-premises) scenarios, each of these lookups requires a costly round-trip across the Internet. These extra round trips can increase by an order of magnitude the time it takes to update each Dynamics 365 Customer Engagement (on-premises) record and can reduce overall throughput drastically. Now, web service operations can target a Dynamics 365 Customer Engagement (on-premises) record using one or more alternate keys instead of a GUID. In addition, entity references to related records can be specified using one or more alternate keys. Because alternate keys are indexed, lookup operations show increased performance as compared to adding a custom attribute as an identifier. If something goes wrong, the system will throw an error and roll back all the changes. More information: Define alternate keys for an entity
Change tracking	When organizations need to maintain Dynamics 365 Customer Engagement (on-premises) data in external storage there is now a way to keep that data synchronized in a performant way by detecting what data has changed since the data was initially extracted or last synchronized. The RetrieveEntityChangesRequest message is used to retrieve the changes for an entity. See Use change tracking to synchronize data with external systems for more information.
Upsert	When loading data into Dynamics 365 Customer Engagement (on-premises) from an external system, you may not know if a record already exists in Dynamics 365 Customer Engagement (on-premises) and should be updated, or whether you must create a new record. Use the new UpsertRequest message to update the record if it exists, or create a new record if it doesn’t exist, in one API call. More information: Update Dynamics 365 Customer Engagement (on-premises) with external data using Upsert

The following table compares the complexity of synchronizing with and without these new features.

Before

Description

For each record: 1. Query Dynamics 365 Customer Engagement (on-premises) to see if the account exists. If it exists, get its account ID (for example, ABC123) 2. Query the contacts to verify that the contact exists. If it exists, get the email ID of the contact (for example, contact@company.com). 3. Query to get or set the region ID (for example, NW). 4. Query to get the user ID to set the owner (for example, user@mycompany.com) 5. Update the account. 6. Set the state of the account by calling the SetState API. 7. Assign the owner by calling the Assign API.

Now, with the new features, it just takes one call to the server to perform the same operations as previously shown.

After	Description
	Just one call to verify that there is an account with the unique ID ABC123, set the primary contact to contact@company.com, set the region to NW, set the owner to user@mycompany.com and the status to active.

Define alternate keys for an entity

All Dynamics 365 Customer Engagement (on-premises) records have unique identifiers defined as GUIDs. These are the primary key for each entity. When you need to integrate with an external data store, you might be able to add a column to the external database tables to contain a reference to the unique identifier in Dynamics 365 Customer Engagement (on-premises). This allows you to have a local reference to link to the Dynamics 365 Customer Engagement (on-premises) record. However, sometimes you can’t modify the external database. With alternate keys you can now define an attribute in a Dynamics 365 Customer Engagement (on-premises) entity to correspond to a unique identifier (or unique combination of columns) used by the external data store. This alternate key can be used to uniquely identify a record in Dynamics 365 Customer Engagement (on-premises) in place of the primary key. You must be able to define which attributes represent a unique identity for your records. Once you identify the attributes that are unique to the entity, you can declare them as alternate keys through the customization user interface (UI) or in the code. This topic provides information about defining alternate keys in the data model. To learn more, see Define alternate keys for an entity in the Dataverse documentation.

Use alternate keys

You can now use alternate keys to create instances of Entity and EntityReference classes. This topic discusses the usage patterns and possible exceptions that might be thrown when using alternate keys. To understand how to define alternate keys for an entity, see Define alternate keys for an entity. To learn more, see Using alternate keys in the Dataverse documentation.

Synchronize data with external systems using change tracking

The change tracking feature in Dynamics 365 Customer Engagement (on-premises) provides a way to keep the data synchronized in a performant way by detecting what data has changed since the data was initially extracted or last synchronized. Previously, without this new feature, it was difficult to build a reliable and efficient mechanism to determine what records had changed in Dynamics 365 Customer Engagement (on-premises). This topic discusses how to retrieve changes for an entity. To learn more, see Use change tracking to synchronize data with external systems in the Dataverse documentation.

Using upsert

You can reduce the complexity involved with data integration scenarios by using the UpsertRequest message. When loading data into Dynamics 365 Customer Engagement (on-premises)from an external system, for example in a bulk data integration scenario, you may not know if a record already exists in Dynamics 365 Customer Engagement (on-premises). In such cases you won’t know if you should call an UpdateRequest or a CreateRequest operation. This results in your querying for the record first to determine if it exists before performing the appropriate operation. You can now reduce this complexity and load data into Customer Engagement more efficiently by using the new UpsertRequest (Update or Insert)message. To learn more, see Use upsert to update records in the Dataverse documentation.

Samples

Sample: Insert or update a record using Upsert
Sample: Synchronize data with external systems using change tracking

Related Sections

Perform specialized operations using Update
Developers guide to customization for Dynamics 365 Customer Engagement (on-premises)