How to dynamically fetch specific partion /folder from synapse spark pool table

Question

Hi,

We are using big synapse sparkpool table which internally store data in parquet file.
Those files are stored in data lake based on the partition key columns.

We need to fetch this table in the ADF dataflow to dynamically filter based on other source system.
We could do with single value in source query by using variable (append Variable).
However we need to run new Dataflow instance for every single record.
Please suggest solution to dynamically filter source data for multiple value.

Thanks

Answer 1

Hello @Ganesh Pathak and welcome to Microsoft Q&A.

I am confused why you need a dataflow for every variable. You can pass array type variable to dataflow. See below pictures.

Inside the dataflow, declare parameter of appropriate array type:

In pipeline, make array type variable:

In pipeline's dataflow activity, pass the variable: