question

ranjitkurian-6670 avatar image
0 Votes"
ranjitkurian-6670 asked ·

ADF pipeline to increment the date back to six months

Hi,

i have folders created on each day [20201114], so there are 12 months folders created based on date.
i want my pipeline to copy the files from each folders ; Starting from current date [20201114] till 6 months back folders data [20200514]

i have created two variables start date [@formatDateTime(AddToTime(utcnow(), -6, 'Month'), 'yyyy-MM-dd')] and end date [@formatDateTime((utcnow(), 'yyyy-MM-dd')]

now how to increment dates one by one from start to end date

Now i want the copy activity to increment 1 + day to this variable and copy the data from respective date folder

for eample: varibale output is [2020-05-14] >> adding +1 [2020-05-14 + 1] = [2020-05-15]>> copy activity should search for the folder name [2020-05-15] and process

like wise the process should continue from [2020-05-14] till [2020-11-14]

how to achieve this through ADF Pipeline

azure-data-lake-storage
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

HarithaMaddi-MSFT avatar image
0 Votes"
HarithaMaddi-MSFT answered ·

Hi @ranjitkurian-6670,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

I have used below approach using Until activity to implement the same. For testing, I used one month folders to loop and copy activity can be placed inside until loop and it can use "Date" variable in below example to access corresponding date folder which I skipped adding in below repro.

Due to limitation in using self referencing variable in "Set variable" activity, I have used index value and counter variables to add 1 day in every loop to the date. Also, attaching the JSON of the pipeline.

39830-incrementalfolderaccessadf.gif

Please let us know if requirement does not align with implementation or for further queries and we will be glad to assist.



  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.



· 3 ·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you Haritha, its working as expected.

If the folder of specified date does not exist then the pipeline process fails at copy activity and the loop moves on to next folder date.
do we need to add any error handler so that if folder of specified date does not exist it should not come to copy activity and move to next loop.
is this possible in ADF?

1 Vote 1 ·

Thanks @ranjitkurian-6670 for confirming the approach. Yes, it can be implemented as below using "Get Metadata" and "IF Condition" activities. Attached is the pipeline JSON after modification.

40361-image.png

40237-image.png

40362-pipelinejson.txt

Hope this helps! Please let us know for further queries and we will be glad to assist.

1 Vote 1 ·
image.png (71.2 KiB)
image.png (55.0 KiB)
pipelinejson.txt (11.8 KiB)

Hello,

Thanks for your answer. I have similar requirement.
But how to connect this to copy activity and give start date and end date as parameters while pipeline run time.

0 Votes 0 ·