Migrate ADF - Datasets which are linked with Linked Services and Pipelines to Synapse Analytics

Question

Hi All -

We Need to migrate Datasets from ADF which are linked with Linked Services and Pipelines only to Synapse Analytics.

The GITHUB solution (from previous posts https://learn.microsoft.com/en-us/answers/questions/533505/import-bulk-pipelines-from-azure-data-factory-to-a.html)
migrates entire all datasets, pipelines, linked services from ADF to Synapse Analytics.

But we need to migrate Datasets, linked services and pipelines which are linked each other and don't need to migrate which were not linked.

Thanks in advance !!!

Answer 1

The following solution works only on existing published assets. It does not look or touch repository code. @Naga

This solution relies on the fact that attempting to delete a referenced asset throws an error. (see below picture)

Assumptions: If a dataset is not referenced by a pipeline, then you want to delete it. If a linked service is not referenced by a dataset, then you want to delete it. This is not checking for pipelines which are not referenced by triggers.

$ADFresourceGroup = resourceGroupName  
$ADFname = FactoryName  
$datasets = Get-AzDataFactoryV2Dataset -ResourceGroupName $ADFresourceGroup -DataFactoryName $ADFname  
$linkedserveces = Get-AzDataFactoryV2LinkedService -ResourceGroupName $ADFresourceGroup -DataFactoryName $ADFname  
  
Foreach ($set in $datasets) { Remove-AzDataFactoryV2Dataset -ResourceGroupName $ADFresourceGroup -DataFactoryName $ADFname -Name $set.Name -Force }  
Foreach ($linked in $linkedserveces) { Remove-AzDataFactoryV2LinkedService -ResourceGroupName $ADFresourceGroup -DataFactoryName $ADFname -Name $linked.Name -Force }  

Answer 2

Hi Team,

Above scripts worked for me. Could you also please share the PowerShell scripts in the same way from Synapse.

Regards,
NagaSri