question

NavinDondapati avatar image
0 Votes"
NavinDondapati asked PRADEEPCHEEKATLA-MSFT commented

Azure Data Factory V2 Arm template to Synpase Data Integration without Git

Hi Guys,

We have Azure data factory as separate resources within same resource group, now we exported out the arm template from ADF. And we recently spin up Azure synapse component on azure; We do see Data integration component which is similar to ADF, But we could not find a way to deploy our old pipelines to Azure Synapse.

How can i deploy my arm template to synapse and make sure my pipeline works fine?

Regards,
Navin

azure-synapse-analytics
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @NavinDondapati,

Thanks for the question and using MS Q&A platform.

Currently, there is no direct method to migrate ADF v2 templates to Azure Synapse Analytics.

Alternate Solution:

You can use Azure PowerShell to upload the downloaded JSON supported files to Azure Synapse Analytics and use it.

Here in the below example I'm trying to move ADF pipeline from (ADF:chepra) to Azure Synapse Analytics Workspace (Synapse Analytics: cheprasynapse).

After upload the JSON files to Azure Synapse Analytics workspace, make sure to modify the linked services configuration. Else, you get this error message Cloud stored credentials cannot be found with retry, please check whether related resource had been deleted before.

Step1: Once you download the supported JSON files from the ADF.

97411-image.png

Step2: Log in to PowerShell (Make sure to use other subscription where you want to upload the JSON files)

 Run the following command, and enter the same Azure user name and password that you use to sign in to the Azure portal:
    
 Connect-AzAccount
    
 Run the following command to view all the subscriptions for this account:
    
 Get-AzSubscription
    
 If you see multiple subscriptions associated with your account, run the following command to select the subscription that you want to work with. Replace SubscriptionId with the ID/Name of your Azure subscription:
    
 Select-AzSubscription -SubscriptionName "<SubscriptionName>"

98469-image.png

Step3: To upload downloaded JSON support files from ADF v2 to Azure Synapse Analytics workspace .

     To Create a linked service:
        
     Set-AzSynapseLinkedService -WorkspaceName ContosoWorkspace -Name ContosoLinkedService -DefinitionFile "C:\\samples\\LinkedService.json"
        
     To create a dataset: 
        
    Set-AzSynapseDataset -WorkspaceName ContosoWorkspace -Name ContosoDataset -DefinitionFile "C:\\samples\\Dataset.json"
        
     To create a pipeline:
        
     Set-AzSynapsePipeline -WorkspaceName ContosoWorkspace -Name ContosoPipeline -DefinitionFile "C:\pipeline.json"

98586-image.png

Now we had successfully uploaded the downloaded JSON files ADF to Azure Synapse Analytics Workspace.

98567-image.png

For more details, refer Azure PowerShell cmdlets for Azure Synapse Analytics..

Hope this helps. Do let us know if you any further queries.


Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.


image.png (37.7 KiB)
image.png (222.3 KiB)
image.png (105.5 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Is it same approach for non prod Synapse to prod synapse (different subscription) migration with out git configuration or azure devops?

0 Votes 0 ·

Hello @NavinDondapati,

Yes, it was the same approach.

0 Votes 0 ·
SaikatDutta-6535 avatar image
0 Votes"
SaikatDutta-6535 answered PRADEEPCHEEKATLA-MSFT commented

This is good for single pipeline. However we have 1300 pipelines in our environment. How can we do this? Or do we need to write parameterized powershell queries? Surely, there should be support for an easier way out.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @SaikatDutta-6535,

You can use this open source community PowerShell tool to help to migrate from ADF to Synapse: https://github.com/Azure-Samples/Synapse/tree/main/Pipelines/ImportADFtoSynapse

0 Votes 0 ·