question

pavnis avatar image
0 Votes"
pavnis asked MartinJaffer-MSFT commented

Azure Data Factory ADF - parameterize the From and To File Paths under Data Flow > Source Options > After Completion

Problem: I need help with ADF to parameterize the From and To File Paths under Data Flow > Source Options > After Completion.

Environment: We are using Data Flow Activity to transform and load data from ADLS Gen 2 to a sink. I am moving a file after completion to keep files organized using Source Options in Data Flow. We use Azure DevOps and bitbucket for Source Control and CICD. We also share ADF for multiple projects so we need to have this taken care at CI-CD only.

We are using one ADLS service/Container to store both Dev and Test curated datasets in .parquet format. Hence, I need to parameterize File path for CICD. So in dev we have ContainerName & folderName/dev/subFolderName and in test we have ContainerName & folderName/tst/subFolderName along with Stage and Production get their respective paths. ContainerName are same in both Dev ADLS and Production ADLS.

So far: I added a parameters for both From and To in Data Flow > Source Options but when I export ARM, I don't see those parameters.

Thanks to all the geniuses out here.



azure-data-factoryazure-data-lake-storage
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

If you found your own solution, please share it here with the community.

0 Votes 0 ·

1 Answer

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered SaurabhSharma-msft commented

Hello @pavnis and welcome to Microsoft Q&A.

I haven't parameterized that part of Data Flow before. There are two routes to try.

  1. Have the Data Flow parameters be set by the pipeline (which calls the Data Flow). Then alter the pipeline parameters in CI/CD.

  2. Investigate the parameterization template options.

  3. Use Global Parameters.


· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello Martin,

I will look into these and get back to you with my findings.

Thanks,
D Patel

0 Votes 0 ·

@pavnis Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.

0 Votes 0 ·