Push Data Factory lineage data to Azure Purview (Preview)
APPLIES TO:
Azure Data Factory
Azure Synapse Analytics
In this tutorial, you'll use the Data Factory user interface (UI) to create a pipeline that run activities and report lineage data to Azure Purview account. Then you can view all the lineage information in your Azure Purview account.
Currently, lineage is supported for Copy, Data Flow, and Execute SSIS activities. Learn more details on the supported capabilities from Supported Azure Data Factory activities.
Prerequisites
- Azure subscription. If you don't have an Azure subscription, create a free Azure account before you begin.
- Azure Data Factory. If you don't have an Azure Data Factory, see Create an Azure Data Factory.
- Azure Purview account. The Purview account captures all lineage data generated by data factory. If you don't have an Azure Purview account, see Create an Azure Purview.
Run pipeline and push lineage data to Azure Purview
Step 1: Connect Data Factory to your Purview account
You can establish the connection between Data Factory and Purview account by following the steps in Connect Data Factory to Azure Purview.
Step 2: Run pipeline in Data Factory
You can create pipelines, Copy activities and Dataflow activities in Data Factory. You don't need any additional configuration for lineage data capture. The lineage data will automatically be captured during the activities execution.
Learn more about how to create Copy, Data Flow and Execute SSIS activities from Copy data from Azure Blob storage to a database in Azure SQL Database by using Azure Data Factory, Transform data using mapping data flows and Run SSIS Packages in Azure.
Step 3: Monitor lineage reporting status
After you run the pipeline, in the pipeline monitoring view, you can check the lineage reporting status by clicking the following Lineage status button. The same information is also available in the activity output JSON -> reportLineageToPurvew section.
Step 4: View lineage information in your Purview account
On Purview UI, you can browse assets and choose type "Azure Data Factory". You can also search the Data Catalog using keywords.
On the activity asset, click the Lineage tab, you can see all the lineage information.
Copy activity:
Data Flow activity:
Note
For the lineage of Dataflow activity, we only support source and sink. The lineage for Dataflow transformation is not supported yet.
Execute SSIS Package activity:
Note
For the lineage of Execute SSIS Package activity, we only support source and destination. The lineage for transformation is not supported yet.