Tutorial: Accept and receive data using Azure Data Share

In this tutorial, you'll learn how to accept a data share invitation using Azure Data Share. You'll learn how to receive data being shared with you, and how to enable a regular refresh interval to ensure that you always have the most recent snapshot of the data being shared with you.

• How to accept an Azure Data Share invitation
• Create an Azure Data Share account
• Specify a destination for your data
• Create a subscription to your data share for scheduled refresh

Prerequisites

Before you can accept a data share invitation, you must create some Azure resources, which are listed below.

Ensure that all prerequisites are complete before accepting a data share invitation.

• Azure Subscription: If you don't have an Azure subscription, create a free account before you begin.
• A Data Share invitation: An invitation from Microsoft Azure with a subject titled "Azure Data Share invitation from yourdataprovider@domain.com".
• Register the Microsoft.DataShare resource provider in the Azure subscription where you'll create a Data Share resource and the Azure subscription where your target Azure data stores are located.

Receive data into a storage account

• An Azure Storage account: If you don't already have one, you can create an Azure Storage account.
• Permission to write to the storage account, which is present in Microsoft.Storage/storageAccounts/write. This permission exists in the Storage Account Contributor role.
• Permission to add role assignment to the storage account, which is present in Microsoft.Authorization/role assignments/write. This permission exists in the Owner role.

Receive data into a SQL-based target

If you choose to receive data into Azure SQL Database, Azure Synapse Analytics, below is the list of prerequisites.

Prerequisites for receiving data into Azure SQL Database or Azure Synapse Analytics (formerly Azure SQL DW)

• An Azure SQL Database or Azure Synapse Analytics (formerly Azure SQL DW).
• Permission to write to databases on the SQL server, which is present in Microsoft.Sql/servers/databases/write. This permission exists in the Contributor role.
• Microsoft Entra Admin of the SQL server
• SQL Server Firewall access. This can be done through the following steps:
  1. In SQL server in Azure portal, navigate to Firewalls and virtual networks
  2. Select Yes for Allow Azure services and resources to access this server.
  3. Select +Add client IP. Client IP address is subject to change. This process might need to be repeated the next time you're sharing SQL data from Azure portal. You can also add an IP range.
  4. Select Save.

Prerequisites for receiving data into Azure Synapse Analytics (workspace) SQL pool

• An Azure Synapse Analytics (workspace) dedicated SQL pool. Receiving data into serverless SQL pool isn't currently supported.

• Permission to write to the SQL pool in Synapse workspace, which is present in Microsoft.Synapse/workspaces/sqlPools/write. This permission exists in the Contributor role.

• Permission for the Data Share resource's managed identity to access the Synapse workspace SQL pool. This can be done through the following steps:

  1. In Azure portal, navigate to Synapse workspace. Select SQL Active Directory admin from left navigation and set yourself as the Microsoft Entra admin.

  2. Open Synapse Studio, select Manage from the left navigation. Select Access control under Security. Assign yourself SQL admin or Workspace admin role.

  3. In Synapse Studio, select Develop from the left navigation. Execute the following script in SQL pool to add the Data Share resource Managed Identity as a 'db_datareader, db_datawriter, db_ddladmin'.

     create user "<share_acc_name>" from external provider; 
     exec sp_addrolemember db_datareader, "<share_acc_name>"; 
     exec sp_addrolemember db_datawriter, "<share_acc_name>"; 
     exec sp_addrolemember db_ddladmin, "<share_acc_name>";
     

     The <share_acc_name> is the name of your Data Share resource. If you haven't created a Data Share resource as yet, you can come back to this prerequisite later.

• Synapse workspace Firewall access. This can be done through the following steps:

  1. In Azure portal, navigate to Synapse workspace. Select Firewalls from left navigation.
  2. Select ON for Allow Azure services and resources to access this workspace.
  3. Select +Add client IP. Client IP address is subject to change. This process might need to be repeated the next time you're sharing SQL data from Azure portal. You can also add an IP range.
  4. Select Save.

Receive data into an Azure Data Explorer cluster:

• An Azure Data Explorer cluster in the same Azure data center as the data provider's Data Explorer cluster: If you don't already have one, you can create an Azure Data Explorer cluster. If you don't know the Azure data center of the data provider's cluster, you can create the cluster later in the process.
• Permission to write to the Azure Data Explorer cluster, which is present in Microsoft.Kusto/clusters/write. This permission exists in the Contributor role.

Sign in to the Azure portal

Sign in to the Azure portal.

Open invitation

Portal

1. You can open invitation from email or directly from Azure portal.

   To open invitation from email, check your inbox for an invitation from your data provider. The invitation is from Microsoft Azure, titled Azure Data Share invitation from yourdataprovider@domain.com. Select on View invitation to see your invitation in Azure.

   To open invitation from Azure portal directly, search for Data Share Invitations in Azure portal. This action takes you to the list of Data Share invitations.

   If you're a guest user of a tenant, you'll be asked to verify your email address for the tenant prior to viewing Data Share invitation for the first time. Once verified, it's valid for 12 months.

   

2. Select the invitation you would like to view.

Azure CLI

Prepare your Azure CLI environment and then view your invitations.

Start by preparing your environment for the Azure CLI:

• Use the Bash environment in Azure Cloud Shell. For more information, see Quickstart for Bash in Azure Cloud Shell.

  

• If you prefer to run CLI reference commands locally, install the Azure CLI. If you're running on Windows or macOS, consider running Azure CLI in a Docker container. For more information, see How to run the Azure CLI in a Docker container.

  • If you're using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Sign in with the Azure CLI.

  • When you're prompted, install the Azure CLI extension on first use. For more information about extensions, see Use extensions with the Azure CLI.

  • Run az version to find the version and dependent libraries that are installed. To upgrade to the latest version, run az upgrade.

Run the az datashare consumer-invitation list-invitation command to see your current invitations:

az datashare consumer consumer-invitation list-invitation

Copy your invitation ID for use in the next section.

PowerShell

Start by preparing your environment for PowerShell. You can either run PowerShell commands locally or using the Bash environment in the Azure Cloud Shell.

• If you choose to use Azure PowerShell locally:
  • Install the latest version of the Az PowerShell module.
  • Connect to your Azure account using the Connect-AzAccount cmdlet.
• If you choose to use Azure Cloud Shell:
  • See Overview of Azure Cloud Shell for more information.

1. Use the Connect-AzAccount command to connect to your Azure account.

   Connect-AzAccount
   

2. Run the Set-AzContext command to set the correct subscription, if you have multiple subscriptions.

   Set-AzContext [SubscriptionID/SubscriptionName]
   

3. Run the Get-AzDataShareReceivedInvitation command to see your current invitations:

   Get-AzDataShareReceivedInvitation
   

Copy your invitation ID for use in the next section.

Accept invitation

Portal

1. Make sure all fields are reviewed, including the Terms of Use. If you agree to the terms of use, you'll be required to check the box to indicate you agree.

   

2. Under Target Data Share Account, select the Subscription and Resource Group that you'll be deploying your Data Share into.

   For the Data Share Account field, select Create new if you don't have an existing Data Share account. Otherwise, select an existing Data Share account that you'd like to accept your data share into.

   For the Received Share Name field, you can leave the default specified by the data provide, or specify a new name for the received share.

   Once you've agreed to the terms of use and specified a Data Share account to manage your received share, Select Accept and configure. A share subscription will be created.

   

   This action takes you to the received share in your Data Share account.

   If you don't want to accept the invitation, Select Reject.

Azure CLI

Use the az datashare share-subscription create command to create the Data Share.

az datashare share-subscription create --resource-group share-rg \
  --name "fabrikamsolutions" --account-name FabrikamDataShareAccount \
  --invitation-id 89abcdef-0123-4567-89ab-cdef01234567 \
  --source-share-location "East US 2"

PowerShell

Use the New-AzDataShareSubscription command to create the Data Share. The InvitationId will be the ID you gathered from the previous step.

New-AzDataShareSubscription -ResourceGroupName <String> -AccountName <String> -Name <String> -InvitationId <String>

Configure received share

Portal

Follow the steps below to configure where you want to receive data.

1. Select Datasets tab. Check the box next to the dataset you'd like to assign a destination to. Select + Map to target to choose a target data store.

   

2. Select a target data store type that you'd like the data to land in. Any data files or tables in the target data store with the same path and name will be overwritten. If you're receiving data into Azure SQL Database or Azure Synapse Analytics (formerly Azure SQL DW), check the checkbox Allow Data Share to run the above 'create user' script on my behalf.

   For in-place sharing, select a data store in the Location specified. The Location is the Azure data center where data provider's source data store is located at. Once dataset is mapped, you can follow the link in the Target Path to access the data.

   

3. For snapshot-based sharing, if the data provider has created a snapshot schedule to provide regular update to the data, you can also enable snapshot schedule by selecting the Snapshot Schedule tab. Check the box next to the snapshot schedule and select + Enable. The first scheduled snapshot will start within one minute of the schedule time and subsequent snapshots will start within seconds of the scheduled time.

   

   The metadata of copied files isn't persisted after each run. This is by design.

Azure CLI

Use these commands to configure where you want to receive data.

1. Run the az datashare consumer-source-data-set list command to get the data set ID:

   az datashare consumer-source-data-set list --resource-group "share-rg" \
    --account-name "FabrikamDataShareAccount" \
      --share-subscription-name "fabrikamsolutions" \
   

2. If you need a storage account, run the az storage account create command to create a storage account for this Data Share:

   az storage account create --resource-group "share-rg" --name "FabrikamDataShareStorageAccount" \
   

3. Use the az storage account show command to get the storage account ID:

   az storage account show --resource-group "share-rg" --name "FabrikamDataShareStorageAccount" \
    --query "id"
   

4. Use the following command to get the account principal ID:

   az datashare account show --resource-group "share-rg" --name "FabrikamDataShareAccount" \
     --query "identity.principalId"
   

5. Use the az role assignment create command to create a role assignment for the account principal using the account principal ID and your storage account ID:

   az role assignment create --role "Contributor" \
     --assignee-object-id 6789abcd-ef01-2345-6789-abcdef012345 
     --assignee-principal-type ServicePrincipal --scope "your\storage\account\id\path"
   

6. Create a variable for the mapping based on the data set ID from step 1:

   $mapping='{\"data_set_id\":\"' + $dataset_id + '\",\"container_name\":\"newcontainer\",
     \"storage_account_name\":\"datashareconsumersa\",\"kind\":\"BlobFolder\",\"prefix\":\"consumer\"}'
   

7. Use the az datashare data-set-mapping create command to create the dataset mapping:

   az datashare data-set-mapping create --account-name "FabrikamDataShareAccount" \
    --data-set-mapping-name "datasetmapping" --resource-group "share-rg" \
    --share-subscription-name "fabrikamsolutions" --blob-folder-data-set-mapping $mapping
   

8. Run the az datashare share-subscription synchronize command to start dataset synchronization.

   az datashare share-subscription synchronize \
     --resource-group "share-rg" --account-name "FabrikamDataShareAccount"  \
     --name "Fabrikam Solutions" --synchronization-mode "Incremental" \
   

   Run the az datashare share-subscription list-synchronization command to see a list of your synchronizations:

   az datashare share-subscription list-synchronization \
     --resource-group "share-rg" --account-name "FabrikamDataShareAccount" \
     --share-subscription-name "Fabrikam Solutions" \
   

   Use the az datashare share-subscription list-source-share-synchronization-setting command to see synchronization settings set on your share.

   az datashare share-subscription list-source-share-synchronization-setting \
     --resource-group "share-rg" --account-name "FabrikamDataShareAccount" \
     --share-subscription-name "Fabrikam Solutions"
   

PowerShell

Use these commands to configure where you want to receive data.

1. Run the Get-AzDataShareSourceDataSet command to get the data set ID:

   Get-AzDataShareSourceDataSet -ResourceGroupName <String> -AccountName <String> -ShareSubscriptionName <String>
   

2. If you don't already have a location where you would like to store the shared data, you can follow these steps to create a storage account. If you already have storage, you can skip to the next steps.

   1. Run the New-AzStorageAccount command to create an Azure Storage account:

      $storageAccount = New-AzStorageAccount -ResourceGroupName <String> -AccountName <String> -Location <String> -SkuName <String>
      
      $ctx = $storageAccount.Context
      

   2. Run the New-AzStorageContainer command to create a container in your new Azure Storage account that will hold your data:

      $containerName = <String>
      
      New-AzStorageContainer -Name $containerName -Context $ctx -Permission blob
      

   3. Run the Set-AzStorageBlobContent command to upload a file. The following example uploads textfile.csv from the D:\testFiles folder on local memory, to the container you created.

      Set-AzStorageBlobContent -File "D:\testFiles\textfile.csv" -Container $containerName -Blob "textfile.csv" -Context $ctx
      

   For more information about working with Azure Storage in PowerShell, follow this Azure Storage PowerShell guide.

3. Use the Get-AzStorageAccount command to get the storage account ID:

   Get-AzStorageAccount -ResourceGroupName <String> -Name <String>
   

4. Use the data set ID from the first step, then run the New-AzDataShareDataSetMapping command to create the dataset mapping:

   New-AzDataShareDataSetMapping -ResourceGroupName <String> -AccountName <String> -ShareSubscriptionName <String> -Name <String> -StorageAccountResourceId <String> -DataSetId <String> -Container <String>
   

5. Run the Start-AzDataShareSubscriptionSynchronization command to start dataset synchronization.

   Start-AzDataShareSubscriptionSynchronization -ResourceGroupName <String> -AccountName <String> -ShareSubscriptionName <String> -SynchronizationMode <String>
   

   Run the Get-AzDataShareSubscriptionSynchronization command to see a list of your synchronizations:

   Get-AzDataShareSubscriptionSynchronization -ResourceGroupName <String> -AccountName <String> -ShareSubscriptionName <String>
   

   Use the Get-AzDataShareSubscriptionSynchronizationDetail command to see synchronization settings set on your share.

   Get-AzDataShareSubscriptionSynchronizationDetail -ResourceGroupName <String> -AccountName <String> -ShareSubscriptionName <String> -SynchronizationId <String>
   

Trigger a snapshot

Portal

These steps only apply to snapshot-based sharing.

1. You can trigger a snapshot by selecting Details tab followed by Trigger snapshot. Here, you can trigger a full or incremental snapshot of your data. If it is your first time receiving data from your data provider, select full copy.

   

2. When the last run status is successful, go to target data store to view the received data. Select Datasets, and select the link in the Target Path.

   

Azure CLI

Run the az datashare trigger create command to trigger a snapshot:

az datashare trigger create --account-name "FabrikamDataShareAccount" --resource-group "share-rg" --share-subscription-name "Fabrikam Solutions" --scheduled-trigger recurrence-interval="Day" synchronization-mode="Incremental" synchronization-time="2018-11-14T04:47:52.9614956Z" --name "Trigger1"

PowerShell

These steps only apply to snapshot-based sharing.

Run the New-AzDataShareTrigger command to trigger a snapshot:

New-AzDataShareTrigger -ResourceGroupName <String> -AccountName <String> -Name <String> -RecurrenceInterval <String> -SynchronizationTime <DateTime>

View history

This step only applies to snapshot-based sharing. To view history of your snapshots, select History tab. Here you'll find history of all snapshots that were generated for the past 30 days.

Clean up resources

When the resource is no longer needed, go to the Data Share Overview page, and select Delete to remove it.

Next steps

In this tutorial, you learned how to accept and receive an Azure Data Share. To learn more about Azure Data Share concepts, continue to Azure Data Share Terminology.

Azure Data Share Concepts