Synapse Pipeline Parameters parametrising in ci/cd not appearing

NAGA SANDEEP KUMAR KAPA 51

Hi,

I have a pipeline with parameters, when I published in synapse i don't see any parameters in templateparameters json file for deploying it to different environments.
Parameter value changes for each environment. How can I achieve it if there is another way.

Thanks in advance.

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2022-08-15T22:52:20.583+00:00

Hello @NAGA SANDEEP KUMAR KAPA ,

Thanks for the question and using MS Q&A platform.

Could you please let me know how you are publishing the synapse pipelines? Are you using the "synapse workspace deployment" or "ARM template deployment" task?
NAGA SANDEEP KUMAR KAPA 51 Reputation points

2022-08-16T03:26:21.9+00:00

Hi Bhargav ,

Synapse workspace deployment task.

Regards,
Sandeep
BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2022-08-16T18:46:17.717+00:00

Hello @NAGA SANDEEP KUMAR KAPA ,

When you select the “Synapse Workspace Deployment” to add the agent job, you won’t see the override parameters automatically populate based on the parameters file. This is expected behavior

We will need to add the override parameters manually.

Please see this documentation about how to use CI/CD integration of synapse workspace to multiple environments.

You will need to create variable groups and create .yaml file that contains parameters/values and pass the file in OverrideArmParameter.

Please follow Step4 from the document and let me know for any questions.

In case, if you use ARM template deployment, Override template parameters will automatically populate based on the parameters file.

If you have any further questions, please let us know.
NAGA SANDEEP KUMAR KAPA 51 Reputation points

2022-08-16T19:05:22.953+00:00

Hi,

Working fine with the Azure devops pipeline parameters using pipeline variables.

But my doubt is I have a etl pipeline which has parameters which are also environment specific. Those parameters are not available in templateparametersforworkspace json file to override. But those parameters are present in templateforworkspace json file.

Do you say that for those kind of parameters also create a variable and assign the environment specific value to them and it will override automatically?

Or is it something wrong with my workspcae_publish? Can we delete and publish again ?

Regards,
Sandeep

1 answer

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2022-08-16T19:51:29.64+00:00
Hello @NAGA SANDEEP KUMAR KAPA ,

TemplateParametersForWorkspace.json file containing a global parameter for your workspace name and a parameter for each workspace
ex: default linked service: the default SQL Server and the default Storage account.

But if you check the TemplateForWorkspace.json in your publish branch, you will find several other properties(the artifact whose properties are not exposed by the default workspace parameters template.)

On your end, you are seeing correct behavior. You no need to delete and publish again.

So looks like you are trying to override the artifact properties(environment specific) that are not parameterized by the default parameters template(TemplateParametersforworkspace.json) right?

In this case, you can use a customized parameters template. Please see the below documents.

https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/cicd-automation-in-synapse-analytics-taking-advantage-of-custom/ba-p/3513546

https://learn.microsoft.com/en-us/azure/synapse-analytics/cicd/continuous-integration-delivery#create-custom-parameters-in-the-workspace-template

I hope this helps. Please let me know if you have any further questions.

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Want a reminder to come back and check responses? Here is how to subscribe to a notification

If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
Please sign in to rate this answer.
BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2022-08-17T21:30:12.363+00:00

Hello @NAGA SANDEEP KUMAR KAPA ,
I am just checking in to see if you get a chance to look into my above answer. Please let me know if you have any further questions.

NAGA SANDEEP KUMAR KAPA 51 Reputation points

2022-08-18T03:34:57.833+00:00

Hi,

I am working on it. I will update you soon.

Regards,
Sandeep

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2022-08-18T13:35:37.583+00:00

Thank you @NAGA SANDEEP KUMAR KAPA

NAGA SANDEEP KUMAR KAPA 51 Reputation points

2022-08-19T15:44:21.287+00:00

Hi,

Thanks for replying. While deploying i have got an issue like it is not overriding the pipeline parameters( it's default value ) though i have given the override for them as well, for the rest linked services it is overriding.

I have modified the below section in custom template json

Microsoft.Synapse/workspaces/pipelines": {
"properties": {
"activities": [{
"typeProperties": {
"waitTimeInSeconds": "-::int",
"headers": "=::object",
"activities": [
{
"typeProperties": {
"url": "-:-webUrl:string"
} }
]
}
}]
}
}

To
Microsoft.Synapse/workspaces/pipelines": {
"properties": {
"parameters" : { "*" : "="}
}
}

I have also observed that
pipeline parameters is of type string but in templateworkspace json it is being shown as type object. is it causing the issue?. Thanks in advance.

Regards,
Sandeep

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2022-08-20T15:18:13.43+00:00

Hi @NAGA SANDEEP KUMAR KAPA ,

After adding the code below to my custom template-parameters-definition.json, I see more properties related to the pipeline parameters.

{
"Microsoft.Synapse/workspaces/pipelines": {
"properties": {
"parameters" : { "*" : "="}
}
}
}

If your pipeline parameters are not overriding, I believe you are not correctly using override parameters on the release synapse deployment task.

And after adding the custom template, are you not seeing the type as a string on the templateforworkspace JSON file?

I see the below from my end.

NAGA SANDEEP KUMAR KAPA 51 Reputation points

2022-08-21T06:50:20.673+00:00

Hi ,

Thanks for the reply. I have done exactly the same way and I want to override the default value as you have shown. But it didn't happen when I deployed to higher environments those fields became blank. Can you please try deploying to other environments by over riding the default value. Can you please check that for me.

Regards,
Sandeep

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2022-08-23T22:18:56.167+00:00

Hi @NAGA SANDEEP KUMAR KAPA ,

Yes. you are correct, I tried to override the default values of the pipeline parameters, and they are not changing.

Can you please provide me the details of how you use the pipeline parameters on the pipeline variables section and the override parameters section?

Meanwhile, I am checking with my internal team on this. I will get back to you once I hear more on this.

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2022-08-29T23:58:57.137+00:00

Hi @NAGA SANDEEP KUMAR KAPA ,
Sorry for the delayed response. You can use the below syntax in your custom template Json file to parameterize the activity level pipeline parameters.

I tested this from my end and able to replace the parameters when deploying using DevOps.

{ "Microsoft.Synapse/workspaces/pipelines": { "properties": { "activities": [{ "typeProperties": { "*": "=" } }] } } }

Please let me know if you have any further questions.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Want a reminder to come back and check responses? Here is how to subscribe to a notification

If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2022-08-31T00:37:37.827+00:00

Hello @NAGA SANDEEP KUMAR KAPA ,
I am just checking to see if you get a chance to look into my above response.
Sign in to comment