Understanding Data Synchronization with External Systems
The ability to manage distributed identity information from a central point is key component of the Microsoft® Forefront™ Identity Manager (FIM 2010 R2) 2010 architecture. This process is governed by a well-defined and customizable set of synchronization rules.
The objective of this document is to explain how you can use the FIM Synchronization Service to synchronize data with external systems.
The conceptual information outlined in this document is complemented by these documents, which provide step-by-step, hands-on guidance for these features:
For an overview of FIM 2010 documentation and guidance for using it, see the Documentation Roadmap.
If you have questions regarding the content of this document or if you have general feedback, post a message to the Forefront Identity Manager 2010 TechNet Forum.
What is Data Synchronization
In a typical enterprise environment, identity information is distributed among various data sources. With FIM 2010, you can manage your distributed identity information from a central point.
In the FIM architecture, the synchronization service has a central role; it manages your distributed identities according to your business needs. When configuring your FIM environment, you need to translate your business requirements into processing instructions that are known as synchronization rules.
The process that is used to apply these synchronization rules to the objects in your environment is also known as declarative provisioning. Synchronization rules govern the object and attribute flow between the FIM Service data store and your configured external systems. In declarative provisioning, you configure synchronization rules in the FIM Portal and they are applied to your objects inside the FIM Synchronization Service's data store during a synchronization run.
So that the FIM Synchronization Service can apply a synchronization rule to a resource, you need to bring your objects into the scope of your synchronization rules and the synchronization rules from the FIM Service data store into the data store of the FIM Synchronization Service. This enables the FIM Synchronization Service to locate the correct set of synchronization rules that need to be applied during a synchronization run. In a well configured environment, the data synchronization service controls the entire lifecycle of your distributed identity data from a central point according to the configuration of your synchronization rules.
How Data Synchronization Works
The objective of this section is to give you a detailed overview of how data synchronization works in FIM 2010. Later, during a general introduction to the object and attribute flow, you will be introduced to synchronization rules, synchronization policies, and how synchronization rules are applied to your managed objects.
Introduction to object and attribute flow
From a high-level perspective, FIM 2010 consists of two main components for data synchronization:
FIM Synchronization Service
FIM Service and Portal
The following illustration outlines the high-level design of FIM 2010.
The objective of the FIM Synchronization Service, also referred to simply as the synchronization service, is to control the data exchange between FIM 2010 and your configured external systems, for example, a human resources (HR) database or Active Directory® Domain Services (AD DS).
To exchange identity data with external data sources, FIM 2010 uses an interface called a management agent (MA). The sole purpose of the MA is to either request identity data from an external data source or to push out identity data to an external data source. The MA is also used in the data exchange between the FIM Synchronization Service and the FIM Service. The following illustration shows an example for a FIM 2010 system that is connected to two external data sources.
The synchronization service uses two data layers to store object information:
The connector space is a staging area that your FIM 2010 system can use to process identity data at any time without the need to maintain an active connection to the external system based on the last known state of your objects. The primary purpose of the connector space data is to support the synchronization process.
Each identity object in an external system must have a representation in the connector space. The following illustration shows a FIM 2010 system that stages representations of identity objects in the connector space.
The actual data exchange between your FIM 2010 system and an external system source is initiated by a component called a run profile. The data exchange in a run profile step of the MA with an external system is always unidirectional—either an import from an external system or to an external system. Inside the synchronization service, the identity data parts that are staged in the connector space are aggregated into one unified view of a physical identity. The layer used to store this unified view is called a metaverse. The creation of an object in the metaverse is always initiated by an object in the connector space. This process is also known as projection. In addition to projecting a new object in the metaverse, a connector space object can also join to an existing metaverse object. Both processes, projection and join, establish a link relationship between a connector space object and a metaverse object. In the FIM terminology, a connector space object that is linked to a metaverse object is known as a connector. If a connector space object does not have a link relationship, it is known as a disconnector. The following illustration shows an example of this.
As soon as a connection between a connector space object and a metaverse object exists, you can use the attribute values that are staged on a connector space object to populate the values of a metaverse object. This process is also known as an import attribute flow.
Projection, join, and import attribute flow represent the elementary steps to create an aggregated view of your distributed identity data in the metaverse. The data stored on a metaverse object is also known as authoritative identity data. In addition to aggregating existing identity information inside the metaverse, you also might need to distribute authoritative information to other connector spaces. The distribution of authoritative metaverse data to an external system requires a connector space object and a link relationship between the metaverse object and the affected connector space object. If such a connector space object does not exist yet, the synchronization service can create one. In FIM, this process is known as provisioning. The following illustration shows an example of this.
To summarize, the synchronization service performs three major object- and attribute-related tasks:
Import of identity data from an external system
Synchronization of the stored identity data between the connector spaces and the metaverse
Export of identity data from to an external system
The following illustration outlines this process:
Introduction to synchronization rules
In the previous section, you have been introduced to the three main tasks the synchronization service performs. Inside the synchronization service, there are two main object-and-attribute-flow directions:
From one connector space to the metaverse
From the metaverse object to multiple connector spaces
These two object-and-attribute-flow directions are encapsulated into a process called synchronization. In FIM 2010, a synchronization process consists of two distinct phases that are related to the object and attributes flow directions. These phases are also known as inbound synchronization and outbound synchronization.
The following illustration outlines the object-and-attribute-flow directions in these two phases.
The objective of the inbound synchronization phase is to update the metaverse with authoritative identity data. A connector space object can only be linked to one metaverse object and can therefore only update one metaverse object.
A metaverse object can be linked to several connector space objects. As soon as the metaverse has been updated, the synchronization service determines whether there is a need to distribute the new data to the connected connector space objects, which is the objective of the outbound synchronization phase.
As an administrator of your FIM environment, you can configure how the synchronization service processes identity data during the inbound and outbound synchronization phase. The related configuration settings are stored in a synchronization rule object. In FIM 2010, you define three types of synchronization rules:
Inbound Synchronization Rule (ISR) –the CS to MV transition of your identity data
Outbound Synchronization Rule (OSR) –the MV to CS transition of your identity data.
Inbound and Outbound Synchronization Rule – combines the two synchronization rules listed above
In FIM, you define synchronization rules in the FIM Portal. However, since they are used by the synchronization service to make processing decisions, you need to bring synchronization rule objects into the metaverse by using the same concept that is used for your managed identity objects—an import from the FIM Portal into the FIM connector space and a projection into the metaverse. The following illustration shows an example of this.
Applying synchronization rules to identity objects
The synchronization service uses synchronization rules to make processing decisions during a synchronization run. When you configure a synchronization rule, you must define the scope of it. The scope includes three settings:
Metaverse Resource Type
External System Resource Type
In an inbound synchronization phase, the scope information is sufficient for the synchronization service to locate the synchronization rules it needs to apply to the objects. For example, when you start a synchronization run on the MA called Fabrikam ADMA, the synchronization service queries the metaverse for all inbound related synchronization rules that have the Fabrikam ADMA configured as External System. In other words, for the inbound synchronization phase, the scope information is used to link synchronization rules to identity objects.
Inbound synchronization rules are connector space–centric.
However, this concept does not work in the outbound synchronization phase. To recap, the outbound synchronization phase starts with a change that was applied to a metaverse object and the synchronization service needs to determine whether this change requires updates to your managed connector spaces. For instance, in an inbound synchronization rule, you must configure the scope of an outbound related synchronization rule. However, for an inbound related synchronization rule, the scope information is used to define the source objects the rule should be applied to. For the outbound synchronization phase, the scope information is used to determine the target of an operation.
Outbound synchronization rules are metaverse object–centric.
As a result, a mechanism is required that links outbound related synchronization rules to metaverse objects. In the FIM architecture, this mechanism is implemented by an operational attribute called an Expected Rules List (ERL). The information stored in this attribute enables the synchronization service to locate all outbound related synchronization rules that need to be applied to an object.
The following illustration shows an example of this.
Since the scope information is tied to one external system and you can have several external systems that are managed by your FIM environment, the ERL attribute is multivalued. You can use this to link several outbound related synchronization rules to a metaverse object. As mentioned previously, the synchronization service uses the values of the ERL attribute to locate the outbound related synchronization rules that need to be applied to an object. However, the ERL values do not directly point to a synchronization rules object; they point to an intermediate object called Expected Rules Entry (ERE). The following illustration shows this architecture.
The ERE is required to track information that includes, for example, the status of the relationship between an identity object and the synchronization rule. For instance, you can retrieve from an ERE object whether a synchronization rule has been applied to an object. The following screenshot shows an example of the ERE status of an object. The status of Pending indicates that FileMA Outbound Synchronization Rule has not been applied yet by the synchronization engine to the object Jimmy Bischoff.
If an outbound synchronization rule has been applied successfully to an object, the ERE status changes to Applied as outlined in the following illustration.
The ERL attribute is managed by the FIM Service. As a consequence, you need to bring all managed identity objects into the FIM service database. For example, if you have an HR system that is used as a data source for the objects in your Active Directory environment, you need to bring all HR objects into the FIM Service database where a link relationship between your objects and the Active Directory outbound related synchronization rule is established.
All managed objects need to be brought into the FIM Service database to either link them with the required outbound synchronization rules or to remove an existing relationship.
Outbound synchronization rules are applied by the synchronization service. To determine, which synchronization rules the synchronization engine needs to apply to an object, the service reviews the expectedRulesList attribute of an object in the metaverse. The following illustration shows an example for an object in the metaverse that has the expectedRulesList attribute populated:
To summarize, for outbound related operations, it is necessary to tie a managed resource together with the related outbound synchronization rules. In FIM, this link relationship is implemented by using an additional object called Expected Rules Entry (ERE). This means, your managed resource points to an ERE object and the ERE object points to the related outbound synchronization rule.
The following illustration shows the relationship between the three objects.
In FIM, a complete outbound synchronization object set is known as an outbound synchronization triple. The outbound synchronization triple relationship is managed by the FIM Service and applied by the FIM synchronization service.
Introduction to synchronization policies
In the previous sections, you have learned that outbound synchronization–related operations are metaverse object–centric and require the existence of an outbound synchronization triple. The triple relationship is managed by the FIM Service.
Creating an outbound synchronization triple is also known as “to bring an object into the scope of an outbound synchronization rule”. The opposite operation is known as “to remove a resource from the scope of an outbound synchronization rule”. Both operations involve at least three additional FIM objects:
Management policy rule (MPR)
The complete architecture of a set, an MPR, a workflow, and an outbound synchronization rule is known as a synchronization policy.
The following illustration outlines this architecture:
When you design your outbound synchronization logic, you need to define what the condition is that brings an object into the scope of an outbound synchronization rule. In FIM, you express this condition as sets. For example, if your business policy states that all full-time employees must have an account in the corporations Active Directory, it is advisable to create a set called All full-time employees to track the related objects. When a user becomes a member of All full-time employees, you can take advantage of the set transition to bring the related object into the scope of the outbound synchronization rule that manages your Active Directory objects. The controlling entity in this process is a set transition-based MPR. MPRs define the required responses to conditions in your FIM environment. The condition in the current example is the transition of an object in the All full-time employees set. Set transition-based MPRs invoke workflows as a response to a condition.
An activity related to a synchronization rule is one option you can configure on a workflow in FIM. In conjunction with a synchronization rule, a workflow either brings an object into the scope or removes the object from the scope of an outbound synchronization rule.
To summarize, the relationship between a managed object and an outbound synchronization rule is governed by a set transition-based MPR. When the condition that triggers the MPR occurs, the MPR invokes a workflow that either adds or removes an object from the scope of a synchronization rule.
It is important to note that synchronization policies are only required for outbound synchronization rules.
Introduction to the FIM management agent
In FIM, MAs are used to exchange data between the synchronization service and the external systems. This includes the data exchange between the synchronization service and the FIM Service.
However, although the interface to exchange data is the same, the FIM MA has a special role in the FIM architecture. In the previous section, you have learned that you need to bring all managed resources into the FIM service database. This means there is only one specific business rule for the FIM connector space—to function as a mirror for the metaverse. As a consequence, there is no inbound or outbound synchronization rule that you need to configure for the data exchange between the metaverse and the FIM connector space. In FIM, the data exchange between the metaverse and the FIM connector space is governed by a preconfigured set of nondeclarative rules. When you open the management designer, you find some nondeclarative settings that you can configure. This includes the list of object types, the list of attributes, the connector filter, object type mappings, attribute flow, and deprovisioning.
In addition to the configuration of the selected object types and the object type mappings, all scenarios typically require a configuration of additional import and export attribute flow mappings.
The following screenshot shows an example for the related dialog box page.
In Applying synchronization rules to identity objects earlier in this document, you have learned that you need to bring the outbound synchronization triple into the metaverse so that the synchronization can apply outbound synchronization rules to your resources. In addition to bringing the related objects into the metaverse, you also need to ensure that you have an inbound attribute flow mapping configured on your resource for the ExpectedRulesList attribute. As mentioned earlier in this section, the synchronization service uses the values of this attribute to locate the outbound synchronization rules that need to be applied to the resource. If the values for this attribute are not populated in the metaverse, outbound synchronization does not work on your resource. Since there is no declarative inbound or outbound synchronization rule for the data exchange between the metaverse and the FIM connector space, another mechanism is needed that activates provisioning for metaverse objects into the FIM connector space.
In FIM, this is accomplished by the Create Resource In FIM setting of an inbound synchronization rule.
The following illustration shows an example for this setting.
Since the FIM connector space is considered to be a mirror of the metaverse, all objects that are brought into the metaverse by an inbound synchronization rule are automatically also provisioned into the FIM connector space if a representation doesn’t exist there yet.
Configuring Synchronization Rules
In FIM, synchronization rules govern the object and attribute flow between the connector spaces and the metaverse. The synchronization process consists of two distinct phases called inbound and outbound synchronization.
In FIM, you configure related synchronization rules that are scoped based on these two phases. From the previous sections, you know how synchronization rules are applied to your resources. The objective of this section is to take a look at what synchronization rules can do and you how you can configure them.
Configuring inbound synchronization rules
The purpose of a synchronization rule is to define the object and attribute flow of an identity object. In FIM, inbound synchronization rules govern the object and attribute flow during the CS to MV transition. To define the object and attribute flow conditions during the CS to MV transition, an inbound synchronization rule consists of four main building blocks:
General –The definition of the display name, a description, a dependency, and the data flow direction.
Scope - The scope of an inbound synchronization rule defines the relationship between a metaverse resource type and a resource type in your external system. In addition to the relationship, you can also define an external system scoping filter to prevent certain objects from being processed by the synchronization service.
Relationship – The relationship defines the conditions under which connector space resources are linked to existing metaverse resources through relationship criteria. If no matching partner exists in the metaverse, you can configure your inbound synchronization rule to create a resource in FIM, which creates a new metaverse resource and replicates it into the FIM connector space.
Inbound Attribute Flow – Flow rules that define attribute values that should flow from the connector space to the metaverse resource. In addition to direct flow rules, you can also define flow rules that transform, through the use of a set of built-in functions, the actual value that is applied to a metaverse resource.
As outlined earlier in this document, the scope of an inbound synchronization rule is the connector space that the rule is defined for. In your inbound synchronization rule, you can define a dependency to another synchronization rule. When you select a synchronization rule as dependency, the selected synchronization rule must be applied to a resource before that synchronization rule with the dependency configured is applied. In the scope definition, you can define an external system scoping filter to prevent certain objects from being processed towards the metaverse. The scoping filter defines conditions for the attributes you have selected. If a condition is met, the object is not processed further by the synchronization service. For example, if you want to filter all resources that have a sAMAccountName attribute that starts with A, you can define this criteria as an external system scoping filter condition.The following illustration shows an example of this.
The inbound system scoping filter does not support multi-valued attributes.
Configuring outbound synchronization rules
The purpose of a synchronization rule is to define the object and attribute flow of an identity object. In FIM, outbound synchronization rules govern the object and attribute flow during the MV to CS transition. To define the object and attribute flow conditions during the MV to CS transition, an inbound synchronization rule consists of five main building blocks:
General – The definition of the display name, a description, a dependency, and the data flow direction.
Scope - The scope of an inbound synchronization rule defines the relationship between a metaverse resource type and a resource type in your external system.
Relationship – The relationship defines the conditions under which connector space resources are linked to existing metaverse resources through relationship criteria. You can configure an outbound synchronization rule to create a resource in the external system that the synchronization rule is defined for. In addition, you can configure the synchronization rule to disconnect the metaverse object from a linked connector space object when the metaverse object is removed from the scope of the synchronization rule. The relationship criteria in an outbound synchronization rule is used during the inbound synchronization phase on the target system to link related objects.
Workflow Parameters – Workflow parameters are used to pass a value of a property from a workflow activity to a synchronization rule. For example, when you create a new user in Active Directory Domain Services (AD DS), you want to set a new random password. By generating a random password in a workflow activity, you can send an e-mail message to the manager of the new user and the workflow parameter allows you to pass this workflow value to the synchronization rule at run time.
Outbound Attribute Flow – Flow rules define attribute values that should flow from the metaverse to the connector space resource. In addition to direct flow rules, you can also define flow rules that transform, through the use of a set of built-in functions, the actual value that is applied to a metaverse resource.
Attribute Value Transformations
When you configure synchronization rules, you typically also configure attribute flow mappings. The objective of attribute flow mappings is to define how an attribute is supposed to be populated. In the simplest case, the target attribute is populated with the value of the configured source attribute. This type of configuration is also known as direct attribute flow mapping.
However, there are also situations when you need to:
Tie a flow to a condition
Calculate the actual value
Transform a data type
To address these requirements, FIM provides a set of built-in functions.
The following table lists some examples for each transformation case.
|If you need to||Use the following function|
Apply a flow if an attribute has a value
Select the first four characters of an attribute
Convert a binary SID into a string
Some data transformations require more than just one single operation. To address the requirement for more complex conditions, FIM supports the concept of custom expressions, which are combinations of the built-in functions into one expression.For example, if you only want to flow Value A if the attribute myAttribute 1 has a value and flow the value of myAttribute 2 if not, you can implement this requirement by using the following custom expression:
The built-in functions and the combination of custom expressions represent a powerful way to calculate attribute values in attribute flow mappings.
Introduction to Precedence
When you configure multiple synchronization rules, it is possible to create conflicts in overlapping settings. An overlapping setting is a configuration in which the same type of information is contributed by various sources. An example of an overlapping setting is a metaverse attribute with inbound flow mappings from several MAs. The following illustration shows an example for four MAs that have an inbound flow configured for the same metaverse attribute.
From the perspective of the synchronization service, overlapping settings represent conflicts that require a resolution mechanism. The resolution mechanism to handle overlapping settings is known as precedence. In FIM, you can find two scopes for overlapping settings:
In the following sections, you find more details on how both types of overlapping settings are handled in FIM.
Attribute flow precedence
When you configure synchronization rules, it is possible to have several attribute flow mappings from various sources for the same metaverse attribute. The following illustration shows an example for the firstName attribute that has an inbound flow mapping from the Fabrikam File MA and the Fabrikam FIMMA configured.
When either of these MAs tries to flow a value to the firstName attribute, the synchronization service needs to determine what to do with the attribute value. By default, the synchronization service uses the regular flow precedence configuration to make a flow decision. Regular attribute flow precedence is governed by the order of the MAs. You can change the order of your attribute flow precedence configuration as outlined in the following illustration.
To make a regular attribute flow precedence decision, the synchronization service tracks in the metaverse next to the actual attribute value as well as the management agent that has contributed it. The synchronization service uses the following logic to make a flow decision:
As long as the metaverse attribute has not been populated yet, all MAs that have an inbound attribute flow mapping for the attribute configured can flow a value.
As soon as the metaverse attribute has a value, the value can only be updated by a management agent that has the same or a higher precedence as the last contributor.
If a management agent with a lower precedence than the last contributor tries to flow an attribute value, the flow is rejected.
By using regular attribute flow precedence, the synchronization service ensures that the metaverse attribute is always populated by the management agent with the highest precedence for the attribute.In the context of attribute flow precedence, it is important to note you have in FIM an inbound and an outbound aspect. The synchronization service always rejects outbound attribute flows of metaverse attribute values that have a lower precedence than the flow target. For example, if, as shown in the illustration above, Fabrikam File MA is the first contributor for the firstName attribute, and you have also an outbound attribute mapping for the firstName attribute on the Fabrikam FIMMA configured, the attribute value does not flow out to the Fabrikam FIMMA. In real world environments, it is possible that you do not have a precedence order. In other words, all MAs have the same authority to populate an attribute value. To address this case, FIM offers the option to configure equal flow precedence. The following illustration shows the related setting.
When configuring equal precedence for a metaverse attribute, the synchronization service uses a last writer wins logic to make a flow decision.
Synchronization rule precedence
A conflict resolution mechanism might also be required in synchronization rules. For example, if a connector space object is in the scope of several inbound synchronization rules and you have overlapping settings configured, the synchronization service needs a mechanism to resolve the conflict.To address the conflict, FIM also provides a precedence implementation on the synchronization rule level. It is important to note that synchronization rule precedence only affects overlapping settings inside the synchronization rules. In other words, if an object is in the scope of several synchronization rules, it does not mean that only the settings from the synchronization rule with the highest precedence are applied to the object. Synchronization rule precedence applies to inbound and outbound synchronization rules.
Designing Your Data Synchronization Model
Before you start implementing synchronization rules in your environment, you should develop a plan that describes in nontechnical terms the behavior of your environment that you want. For each connected system and also for each object type that you intend to manage, your plan should describe what should happen in each of the following events:
Create – a new object has been imported
Update – an update for an object has been imported
Delete – a deletion for an object has been imported
Especially deletions in this context require special care. This is because you cannot detect deletions from a metaverse perspective. When you process a staged deletion, the only information that is available from the metaverse perspective is the removal of the link relationship between the connector space and the metaverse object. However, a link removal during the inbound synchronization phase can also be a result of the external system scoping filter that has been applied to a connector space object that is linked to a metaverse object. In addition to the three basic operations, your plan should address link management:
Establish link – when a link relationship between a metaverse and a connector space object has been established.
Remove link – when a link relationship between a metaverse and a connector space object has been removed.
The listed operations in combination with an object type describe the conditions that can occur during a synchronization run. In your design, you should add to each condition the intended response of your synchronization logic:
"When <condition happens>, do <this>."
Using this simple structure helps you to design your object and data flow model for your scenario. Because the synchronization process is separated into two distinct phases, you should develop a design for the inbound case and for the outbound case. When you are done designing your data synchronization model, you are ready to translate your design into synchronization rules.
Advanced Data Synchronization Considerations
There are some cases you need to handle outside of your synchronization rule implementation. One example of this is a complex data transformation requirement that cannot be implemented by using the declarative provisioning model. In this case, you can still use the nondeclarative synchronization rule model that was introduced by the predecessor of FIM. These complex cases may have to be handled within a rules extension. Another example is the removal of a link relationship in the outbound synchronization phase. In the FIM terminology, this activity is also known as deprovisioning. While the link removal represents the deprovisioning trigger, you need to configure the required response to this condition. The response defines what the system should do with disconnected connector space object.
You configure the related response to this condition in the Configure Deprovisioning section of the Management Agent Designer. Possible responses to this condition are:
Make them disconnectors
Make them explicit disconnectors
Stage a delete on the object for the next export run
Determine with a rules extension
For example, if your requirement is to delete an object in an external system when the link relationship between a metaverse object and a connector space object has been removed, you need to configure deprovisioning to "Stage a delete on the object for the next export run." This configuration does not delete a connector space object. However, it sends a deletion request for the affected object to the external system during the next synchronization run. To actually also delete the connector space object, you need to import the deletion of the related object from the external system.
Another example is the configuration of a response to the link removal during the inbound synchronization phase. This condition triggers the object deletion rule. By default, a metaverse object is always deleted when the last link relationship to a connector space object has been removed. In addition to this, you can either select MAs that are supposed to be ignored in a last connector evaluation, or you can select a list of MAs that should initiate the deletions of a metaverse object.
The following screenshot shows the related configuration dialog:
To synchronize data with external systems, FIM provides a very flexible and customizable architecture to define your object and attribute flow requirements by using synchronization rules. There are two types of synchronization rules available—declarative and non-declarative synchronization rules. You configure declarative synchronization rules in the FIM Portal. However, they are applied to your objects inside the data store of the synchronization service during a synchronization run. Before you start configuring synchronization rules, you should first design your intended object and attribute flow model, which can significantly help you to improve your actual synchronization rule implementation. All managed objects need to be replicated into the FIM Service data store because bringing objects into or out of the scope of a synchronization is handled by the FIM Service based on your configured synchronization policy. Because the FIM connector space is intended to function as a mirror for the metaverse, declarative synchronization rules do not apply to this connector space. While FIM supports declarative and non-declarative synchronization rules, your preferred synchronization rule implementation should be based on declarative synchronization rules.