question

GaneshPathak-4726 avatar image
0 Votes"
GaneshPathak-4726 asked GaneshPathak-4726 commented

ADF Dataflow lookup dynamically filter records from source stream

Hi ,

We are having 100 million of data from the source table(sparkpool table).
However we need to fetch less number of record based on the other source stream (file).
Now if we consider the data flow to lookup synapse table, we need to fetch complete data which is later joined with other source stream.
Ideally in lookup we should restrict the data from source.
If this is the way lookup designed in data-flow, please suggest any workaround.

Thanks

azure-data-factoryazure-synapse-analytics
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @GaneshPathak-4726 ,

Just checking in to see if the above answer helped. If this answers your query, do click 130616-image.png and upvote 130671-image.png for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hi @GaneshPathak-4726 ,,

Following up to see if the above answer helped. If this answers your query, do click 130616-image.png and upvote 130671-image.png for the same. And, if you have any further query do let us know.

0 Votes 0 ·

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered

Hi @GaneshPathak-4726 ,

Welcome to Microsoft Q&A Platform. Thank you for posting query here.

You can use Query option to query only specific data from your source table. In dataflows, If your source transformation pointing to table then under source options you may be able to see query option.

Lookup in transformation in dataflows, will try to perform kind of left join. It will take all rows from your Primary stream and lookup streams and perform lookup to return results. Please check below link to know more about lookup transformation.
https://docs.microsoft.com/en-us/azure/data-factory/data-flow-lookup

You can also consider creating a new table(using Spark notebook may be) from your source table with subset of data and then consume that in your ETL workflow.

Hope this will help. Please let us know if any further queries. Thank you.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.