Are linked service parameters disappearing now?

Question

question

BrentLeslie-2638 asked Dec 2, '20 FangLiuADMS-3292 published Apr 18, '22

Are linked service parameters disappearing now?

It feels like something has changed in data factory linked service parameters recently. I have setup my data factory as follows:

I have created a linked service pointing to on-prem SQL boxes
I have added parameters to this linked service, using the "+ New" parameter box for Server/DB/User
I have then referenced these linked service parameters in the linked service Server/DB name/SQL Auth boxes:

This all works fine. This forms part of a release pipeline. I deployed this to UAT and noticed UAT no longer has most of these parameters specified, however it does have the Key Vault parameter:

The parameter definitions come through to UAT, but not the link to these parameters in the linked services:

I can also confirm this, when I Export the templates from the user interface in Data Factory, I check the ArmTemplate_0.json file (in the Linked Templates folder of the exported file) and the arm_template.json file from the root, to find where the Linked Service parameters should be specified. All I see is this in both files:

I would also expect to see additional "typeProperties" for the other parameters specified.

This seems to confirm that when specifying Linked Service parameters, not all values are being saved in the arm templates files. Only Azure Key Vault parameters are being saved? I just tried doing the same thing with a different data factory and get the same result. I thought there was something wrong with my CI/CD process, however it looks like the problem exists even without going through Azure DevOps release processes.

azure-data-factory

image.png (15.7 KiB)

image.png (42.9 KiB)

image.png (36.2 KiB)

image.png (56.4 KiB)

image.png (51.3 KiB)

image.png (10.7 KiB)

· 5

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SaurabhSharma-msft · Dec 04, 2020 at 01:25 AM

@BrentLeslie-2638 Thanks for using Microsoft Q&A.

Sorry for your trouble. I am looking this issue and get back to you with findings.

0 ·

SaurabhSharma-msft SaurabhSharma-msft · Dec 04, 2020 at 07:33 PM

@BrentLeslie-2638 I am able to reproduce this in my environment (without CI/CD) and checking internally with the products team on this. I will update you here as soon as I hear back from them.

0 ·

BrentLeslie-2638 SaurabhSharma-msft · Dec 06, 2020 at 09:10 PM

Cool thanks @SaurabhSharma0303 - it's broken a few of our ETL processes at the moment. So currently we have a few failing processes which previously used to work. I think I can work around them by manually publishing the parameters, but I will have to do this every evening due to us continually updating our data factories with nightly builds.

0 ·

BrentLeslie-2638 SaurabhSharma-msft · Dec 09, 2020 at 07:32 PM

@SaurabhSharma-msft - any updates on this?

0 ·

Show more comments

Answer 1 · 2020-12-10T19:05:09.007Z

SaurabhSharma-msft answered Dec 10, '20 FangLiuADMS-3292 published Apr 18, '22

@BrentLeslie-2638 Here is the response from the products team -
Connection string is treated as a sensitive field even though the password is specified with a akv. In this case, the connection string value is stored in an arm template parameter named "onprem_SQL_Server_linked_service_use_parameters_connectionString" and it should look like below

So, you need to either manually edit the arm template to give a default value to the "onprem_SQL_Server_linked_service_use_parameters_connectionString" or overwrite the parameter value in the deployment pipeline like

We tried the second method and the linked service was successfully deployed like shown below-

Please let me know if you have any other questions.

image.png (65.5 KiB)

image.png (43.4 KiB)

image.png (207.2 KiB)

· 4

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

BrentLeslie-2638 · Dec 16, 2020 at 03:30 AM

Thanks SaurabhSharma-msft - I have implemented your suggested solution which looks to be working. I did need to put in a "TrustServerCertificate=True" into the connection string as well (not a problem as the host is in Azure on a VM as well, so nothing across the public network). Obviously this would be a problem if we were true "on prem".

There is something odd going on with DevOps releases and Linked Services which I am struggling to track down (sometimes they are overwritten on release, sometimes they aren't). If I am able to get to the source of it, I will get back to you.

1 ·

SaurabhSharma-msft BrentLeslie-2638 · Dec 16, 2020 at 06:48 PM

@BrentLeslie-2638 Great to hear the solution works for you. Sure, please let me know over here or open a new Q&A thread for any other issues you see.

Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.

0 ·

AayushKuSoni-8493 · Apr 08 at 10:07 AM

Hello Team, I did tried 2nd method. But then database name is not coming as expected. It's getting created in "Additional connection properties". Can you please help me out for this.

0 ·

FangLiuADMS-3292 AayushKuSoni-8493 · Apr 18 at 02:26 PM

Database should be "Initial Catalog". What's your connection string?

0 ·

question