Connect to Azure Data Lake Gen2 in Azure Purview

This article outlines the process to register an Azure Data Lake Storage Gen2 data source in Azure Purview including instructions to authenticate and interact with the Azure Data Lake Storage Gen2 source

Supported capabilities

** Lineage is supported if dataset is used as a source/sink in Data Factory Copy activity

Prerequisites

• An Azure account with an active subscription. Create an account for free.

• An active Purview resource.

• You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Purview Studio. See our Azure Purview Permissions page for details.

Register

This section will enable you to register the ADLS Gen2 data source and set up an appropriate authentication mechanism to ensure successful scanning of the data source.

Steps to register

It is important to register the data source in Azure Purview prior to setting up a scan for the data source.

1. Go to the Azure portal, and navigate to the Purview accounts page and select your Purview account

   

2. Open Purview Studio and navigate to the Data Map --> Sources

   

   

3. Create the Collection hierarchy using the Collections menu and assign permissions to individual subcollections, as required

   

4. Navigate to the appropriate collection under the Sources menu and select the Register icon to register a new ADLS Gen2 data source

   

5. Select the Azure Data Lake Storage Gen2 data source and select Continue

   

6. Provide a suitable Name for the data source, select the relevant Azure subscription, existing Data Lake Store account name and the collection and select Apply

   

7. The ADLS Gen2 storage account will be shown under the selected Collection

   

Scan

Prerequisites for scan

In order to have access to scan the data source, an authentication method in the ADLS Gen2 Storage account needs to be configured. The following options are supported:

• System-assigned managed identity (Recommended) - As soon as the Azure Purview Account is created, a system-assigned managed identity (SAMI) is created automatically in Azure AD tenant. Depending on the type of resource, specific RBAC role assignments are required for the Azure Purview system-assigned managed identity (SAMI) to perform the scans.

• User-assigned managed identity (preview) - Similar to a system managed identity, a user-assigned managed identity (UAMI) is a credential resource that can be used to allow Azure Purview to authenticate against Azure Active Directory. For more information, you can see our User-assigned managed identity guide.

• Account Key - Secrets can be created inside an Azure Key Vault to store credentials in order to enable access for Azure Purview to scan data sources securely using the secrets. A secret can be a storage account key, SQL login password, or a password.

  Note

  If you use this option, you need to deploy an Azure key vault resource in your subscription and assign Azure Purview account’s SAMI with required access permission to secrets inside Azure key vault.

• Service Principal - In this method, you can create a new or use an existing service principal in your Azure Active Directory tenant.

Authentication for a scan

Using a system or user assigned managed identity for scanning

It is important to give your Purview account or user-assigned managed identity (UAMI) the permission to scan the ADLS Gen2 data source. You can add your Purview account's system-assigned managed identity (which has the same name as your Purview account) or UAMI at the Subscription, Resource Group, or Resource level, depending on what level scan permissions are needed.

1. From the Azure portal, find either the subscription, resource group, or resource (for example, an Azure Data Lake Storage Gen2 storage account) that you would like to allow the catalog to scan.

   

2. Select Access Control (IAM) in the left navigation and then select + Add --> Add role assignment

   

3. Set the Role to Storage Blob Data Reader and enter your Azure Purview account name or user-assigned managed identity under the Select input box. Then, select Save to give this role assignment to your Purview account.

   

1. Go into your ADLS Gen2 storage account in Azure portal

2. Navigate to Security + networking > Networking

   

3. Choose Selected Networks under Allow access from

   

4. In the Exceptions section, select Allow trusted Microsoft services to access this storage account and hit Save

   

Using Account Key for scanning

When authentication method selected is Account Key, you need to get your access key and store in the key vault:

1. Navigate to your ADLS Gen2 storage account

2. Select Security + networking > Access keys

   

3. Copy your key and save it separately for the next steps

   

4. Navigate to your key vault

   

5. Select Settings > Secrets and select + Generate/Import

6. Enter the Name and Value as the key from your storage account

   

7. Select Create to complete

   

8. If your key vault is not connected to Purview yet, you will need to create a new key vault connection

9. Finally, create a new credential using the key to set up your scan

Using Service Principal for scanning

Creating a new service principal

If you need to Create a new service principal, it is required to register an application in your Azure AD tenant and provide access to Service Principal in your data sources. Your Azure AD Global Administrator or other roles such as Application Administrator can perform this operation.

Getting the Service Principal's Application ID

1. Copy the Application (client) ID present in the Overview of the Service Principal already created

   

Granting the Service Principal access to your ADLS Gen2 account

It is important to give your service principal the permission to scan the ADLS Gen2 data source. You can add access for the service principal at the Subscription, Resource Group, or Resource level, depending on what level scan permissions are needed.

1. From the Azure portal, find either the subscription, resource group, or resource (for example, an Azure Data Lake Storage Gen2 storage account) that you would like to allow the catalog to scan.

   

2. Select Access Control (IAM) in the left navigation and then select + Add --> Add role assignment

   

3. Set the Role to Storage Blob Data Reader and enter your service principal under Select input box. Then, select Save to give this role assignment to your Purview account.

   

Create the scan

1. Open your Purview account and select the Open Purview Studio

2. Navigate to the Data map --> Sources to view the collection hierarchy

3. Select the New Scan icon under the ADLS Gen2 data source registered earlier

   

If using a system or user assigned managed identity

1. Provide a Name for the scan, select the system-assigned or user-assigned managed identity under Credential, choose the appropriate collection for the scan, and select Test connection. On a successful connection, select Continue.

   

If using Account Key

1. Provide a Name for the scan, choose the appropriate collection for the scan, and select Authentication method as Account Key

   

If using Service Principal

1. Provide a Name for the scan, choose the appropriate collection for the scan, and select the + New under Credential

   

2. Select the appropriate Key vault connection and the Secret name that was used while creating the Service Principal. The Service Principal ID is the Application (client) ID copied earlier.

   

3. Select Test connection. On a successful connection, select Continue

Scope and run the scan

1. You can scope your scan to specific folders and subfolders by choosing the appropriate items in the list.

   

2. Then select a scan rule set. You can choose between the system default, existing custom rule sets, or create a new rule set inline.

   

3. If creating a new scan rule set, select the file types to be included in the scan rule.

   

4. You can select the classification rules to be included in the scan rule

   

   

5. Choose your scan trigger. You can set up a schedule or run the scan once.

   

6. Review your scan and select Save and run.

   

View your scans and scan runs

To view existing scans, do the following:

1. Go to the Purview Studio. Select the Data Map tab under the left pane.

2. Select the desired data source. You will see a list of existing scans on that data source under Recent scans, or can view all scans under the Scans tab.

3. Select the scan that has results you want to view.

4. This page will show you all of the previous scan runs along with the status and metrics for each scan run. It will also display whether your scan was scheduled or manual, how many assets had classifications applied, how many total assets were discovered, the start and end time of the scan, and the total scan duration.

Manage your scans - edit, delete, or cancel

To manage or delete a scan, do the following:

1. Go to the Purview Studio. Select the Data Map tab under the left pane.

2. Select the desired data source. You will see a list of existing scans on that data source under Recent scans, or can view all scans under the Scans tab.

3. Select the scan you would like to manage. You can edit the scan by selecting Edit scan.

4. You can cancel an in progress scan by selecting Cancel scan run.

5. You can delete your scan by selecting Delete scan.

Access policy

Supported regions

Azure Purview (management side)

The Purview access policies capability is available in all Azure Purview regions

Azure Storage (enforcement side)

Purview access policies can only be enforced in the following Azure Storage regions

• France Central
• Canada Central

Enable access policy enforcement for the Azure Storage account

The following PowerShell commands need to be executed in the subscription where the Azure Storage account resides. This will cover all Azure Storage accounts in that subscription.

# Install the Az module
Install-Module -Name Az -Scope CurrentUser -Repository PSGallery -Force
# Login into the subscription
Connect-AzAccount -Subscription <SubscriptionID>
# Register the feature
Register-AzProviderFeature -FeatureName AllowPurviewPolicyEnforcement -ProviderNamespace Microsoft.Storage

If the output of the last command shows value of “RegistrationState” as “Registered”, then your subscription is enabled for this functionality.

Follow this configuration guide to enable access policies on an Azure Storage account

Next steps

Now that you have registered your source, follow the below guides to learn more about Purview and your data.

• Data insights in Azure Purview
• Lineage in Azure Purview
• Search Data Catalog