question

AbdullaMahammadKhanDawood-5577 avatar image
0 Votes"
AbdullaMahammadKhanDawood-5577 asked ShaikMaheer-MSFT commented

How to read Data Partitioned files from ADLS using ADF mapping data flow source options

Hi All,

Good Day!!

We have a requirement to join multiple files which are available in ADLS folders which we have ingested from SQL Source and derive business logic accordingly. Since we store the Table data in ADLS with table name and date partitioned folder for incremental load from Source System. Now in order to read multiple files and join like SQL Query, initially I have to read date partitioned files in mapping data flow dynamically as per the current date.

ADLS folder structure: Tenant1/Raw/PolicyAdmin/SRC_ConceptOne/Entity/2021/07/09/

113452-image.png



Can any give me idea how to read date partitioned files dynamically in ADF mapping data flow.

Thank you in anticipation!!

Regards,
Mahammad Khan

azure-data-factory
image.png (15.3 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @AbdullaMahammadKhanDawood-5577 ,

Following up to check is below provided answer helped. If yes, Please Accept Answer. Accepting answer will help community too.

Please feel free to ask if any further queries. Thank you.

0 Votes 0 ·

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered ShaikMaheer-MSFT commented

Hi @AbdullaMahammadKhanDawood-5577 ,

Thank you for posting query in Microsoft Q&A Platform. Kindly check below detailed example, In which I am using wildcard to copy all dates files for "07" month folder.

Folder structure I had on my ADLS demofiles/yyyy/MM/dd/emp_yyyyMMdd.csv
113788-filestructure.gif

Source Transformation in dataflow to copy all .csv files from all dates folders(01,02,03)
113789-sourcetransformation.gif

Add sink transformation after above step to load data in to your Target storage.

Please check below documentation to know more about Wild card file paths in Source Transformations.
https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-storage#source-transformation

Hope this will help. Thank you.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.




· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you Maheer for giving the solution, However I am expecting dynamic expression for YYYY/MM/dd folder structure in ADLS and it should read only files from current date folder from the date parition folder in ADLS.


Thank you in anticipation!!

Regard,
Mahammad Khan

0 Votes 0 ·
ShaikMaheer-MSFT avatar image ShaikMaheer-MSFT AbdullaMahammadKhanDawood-5577 ·

Hi @AbdullaMahammadKhanDawood-5577 ,

Thank you for follow up question.

Use utcnow() function to get current date and then use formatDateTime() to get your dynamic date partition folder.

Kindly check below example for same.

Here I created a variable called "DynamicPath" and using dynamic expression @formatDateTime(utcnow(),'yyyy/MM/dd') to get date partition folder structure for current date.
114333-dynamicexpression.gif

Hope this will help. Thank you.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.


0 Votes 0 ·

Thank you Maheer!!

0 Votes 0 ·
Show more comments