Refresh data sources

Completed

As mentioned in the previous unit, whether you import data using Power Query or ingest data from a Common Data Model or Microsoft Dataverse, data sources aren't refreshed automatically. If you want to ensure that new data from your data sources is available in Audience Insights, you will need to refresh manually. Once your data is refreshed, the new information will be reflected in the entities. After you complete the data unification process, the refresh also refreshes any segments, measures, activities, enrichment, search, data preparation, and insights that are configured in the application.

Refreshing data manually isn't an ideal scenario for most organizations. For Audience Insights to be as effective as possible for your organization, the data should be as current as possible. This helps ensure that the organization has the clearest picture of their customers possible. To assist with this, Audience Insights provide multiple options for refreshing data sources.

These options include:

  • Scheduled data refreshes - Refreshes of all your data sources on a predefined schedule.

  • Incremental refresh - Refreshes smaller subsets of data for a data source based on the configured incremental refresh settings.

  • Near real-time data ingestion - Can ingest profile and activity-related information in near real time.

Scheduled refresh

The easiest way to automate refreshing of your data is to set up scheduled refreshes. Scheduled refreshes are configured by selecting System under the admin heading in the application navigation. You can modify the schedule from the Schedule tab. When you enable scheduled refreshes, it will impact all your data sources. Not only will it refresh the data, but it also refreshes all other configured items such as segments, measures, activities, enrichment, search, data preparation, and insights.

You'll need to provide the following options:

  • Repeat (Frequency) - Defines how often to refresh the data. Data can be refreshed daily or weekly. Selecting weekly allows you to define the specific days the schedule should run.

  • Time Zone - Defines the time zone the job is running in.

  • Time - Specifies the time that you want the refresh to occur. If you want refreshes to happen multiple times a day, you can add additional times.

Turn on scheduled refreshes and select a time.

Important

Refresh schedules perform a complete data refresh on ALL configured data sources. It is not possible to configure different schedules for different data sources. Scheduled refreshes typically include large numbers of records and several complex operations.

When a scheduled refresh runs, the following occurs:

  1. Data is pulled from the data source.

  2. Data is unified.

  3. Data is enriched with additional information.

Depending on factors such as the number of data sources and the volume of data scheduled runs can range from minutes to hours.

Incremental refresh

Incremental refreshes aren't available for all ingestion methods. Currently, only data sources imported using Power Query support incremental ingestion. Incremental ingestion support for connections to Common Data Model is planned. Not all data sources imported through Power Query support incremental refresh. You'll need to verify that it's allowed for the data source you're working with. When you select a data source that allows incremental refresh such as an Azure SQL DB, you'll be provided with incremental refresh settings to configure after you've transformed your data. You'll be able to modify the refresh settings for all entities selected when you create the data source.

Incremental refreshing data provides the following advantages:

  • Faster refreshes - Only data that has changed gets refreshed.

  • Increased reliability - Due to smaller refreshes, connections to volatile source systems don't need to be maintained for as long, reducing the risk of connection issues.

  • Reduced resource consumption - Refreshing only a subset of your total data leads to more efficient use of computing resources and decreases the environmental footprint.

For example, a historical data set stored in an Azure SQL database could contain thousands or even millions for records that span multiple years. With incremental refreshes, you can choose to refresh only records that were created or modified in the last five days.

For each entity, you'll need to provide the following details:

  • Primary key - Select a primary key for the entity or table.

  • Last updated field - Specifies the attribute that indicates when the records were last updated. It's used to identify records that fall within the incremental refresh time frame.

  • Check for updates every - Specifies how long the incremental refresh time frame will be.

schedule incremental refreshes by setting the primary key, last updated field, and check for updates.

Real-time data ingestion

Audience Insights near real-time functionality allow you to see, within seconds, the latest interactions that your customers have made with your products or services. The purpose of real-time updates is to keep customer profile and activity information current in near real time so it can be consumed immediately, until a scheduled refresh job updates the data from the data source.

Real-time operations take place after data unification has happened and they only apply to unified customer profiles. For this reason, existing customer profiles can be updated, but new profiles cannot be created. Existing profiles cannot be deleted. So, real-time profile changes won't update measures, segment membership, or enrichments. Once the normal schedule refresh occurs, items like measures, segments, and enrichments will be updated as normal.

Note

Exporting real-time updates to external systems, like Power BI, is not possible at the moment.

Using the real-time API allows you to publish a new activity from your source system (an individual source record) to a unified customer profile. The new activity will be available as a unified activity in that unified customer profile's timeline within seconds. You can see the timeline in the customer card view or any other timeline integration you configured. Like customer profile information that is updated in real-time, new activities won't be reflected in segments or measures until a schedule refresh occurs. Activities added only through the real-time API aren't part of exports and won't show up in Power BI.

There are two ways to connect to the real-time API:

  • Indirectly - Uses the¬†Dynamics 365 Customer Insights connector.

  • Directly - Programmatically through code.

Regardless of the method you select, both methods require a Dynamics 365 Customer Insights environment, unified customer profiles have been created, activities have been configured and run, and that the account used to configure has either contributor or administrator permissions to authenticate your account

The information discussed in the unit was designed to provide a starting point in working with real-time ingestion. For more detailed information Real-time data ingestion, refer to the following links: