question

ImranMondal-3977 avatar image
0 Votes"
ImranMondal-3977 asked KranthiPakala-MSFT commented

Azure Data Factory

EveryDay My Data Bricks job dumping data in YYYY/MM/DD/HH format folder in Blob Storage
like this -

hourlydata/2021/03/10/00/part-00000-tid-8b61-4d85-a6cb-249245ccad76-39980-1-c000.csv
hourlydata/2021/03/10/01/part-00000-tid-9c81-dc555aaa0f8-41519-1-8c3-46ca-97-c000.csv
hourlydata/2021/03/10/02/part-00000-tid-9c81-dc555aaa0f8-41519-1-8c3-46ca-97-c000.csv

77445-image.png

I have a data set that resides under a folder path where the date is dynamic, inside each hour folder along with CSV files, we have other files also. But I want to select only the CSV file.

As soon as my Databricks Jobs dumps data every hour, I would like to read those new data using data factory and load it into a database, please suggest



azure-data-factory
image.png (18.9 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @imranmondal-3977,

Just checking in to see if the below suggestion from @VaibhavChaudhari was helpful. If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread.

And, if you have any further query do let us know.

Thanks

0 Votes 0 ·

Hi @imranmondal-3977,

We still have not heard back from you. Just wanted to check if you still need assistance on this query? Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

And if you have further query, do let us know.

0 Votes 0 ·

1 Answer

VaibhavChaudhari avatar image
0 Votes"
VaibhavChaudhari answered VaibhavChaudhari commented

Try using Event based triggers in ADF. You should configure it such way that as soon as new file is created ending with .csv, pipeline will use this file to copy/process

See point 6 in below doc

https://docs.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger


Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

So, My data bricks are job creating this year, month, Day, hour folder dynamically when it runs,

So these triggers in ADF will work in these scenarios also, like every hour when Data dump happens it will start the data Factory job.

0 Votes 0 ·

Yes, as soon as file is created or modified, based on these events, ADF pipeline will get triggered.. You have to configure the event trigger as per your requirement

See step by step tutorial - https://www.mssqltips.com/sqlservertip/6063/create-event-based-trigger-in-azure-data-factory/

0 Votes 0 ·