How to automate Synapse pipelines and connections ?

Question

Is it possible to automate the generation of Synapse pipelines in code ? instead of using the interface and clicking all the time?

Other providers like Snowflake allow you to just use worksheets (sql scripts) where you write all the code and you do not need to go to the interface and start clicking buttons selecting the source and the sink.
The same question applies for creation of linked services, sql pools etc

Is this also possible in Synapse ?

Thank you!

Answer 1

Hello and welcome to Microsoft Q&A @kosmos

As I understand, you would like to create Azure Synapse integration assets without using the graphical interface.

There are ways to do this. These include REST API, Powershell, SDKs (dotnet, Go), Azure CLI.
There is a caveat you should be made aware of. All of these have the effect of 'instant-publish', bypassing any code repository you may have set up.

REST:
https://learn.microsoft.com/en-us/rest/api/synapse/data-plane/linked-service/create-or-update-linked-service?tabs=HTTP
https://learn.microsoft.com/en-us/rest/api/synapse/data-plane/dataset/create-or-update-dataset?tabs=HTTP
https://learn.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/create-or-update-pipeline?tabs=HTTP

Powershell:
https://learn.microsoft.com/en-us/powershell/module/az.synapse/set-azsynapselinkedservice?view=azps-8.3.0
https://learn.microsoft.com/en-us/powershell/module/az.synapse/set-azsynapsedataset?view=azps-8.3.0
https://learn.microsoft.com/en-us/powershell/module/az.synapse/set-azsynapsepipeline?view=azps-8.3.0