question

ShreyasiSrivastava-1468 avatar image
0 Votes"
ShreyasiSrivastava-1468 asked ShreyasiSrivastava-1468 published

Using Until for looping copy activity which has Rest Api call set as source

Hi,
I have created a copy activity which has source for REST API and Sink as Azure ADLS. It work for the 1000 records.

I need to apply custom pagination to track if has more than 1000 records any time. I am trying to parameterize the copy activity dataset but not sure how to use until for looping in this case.

How can I achieve this custom pagination?
Please help ASAP.

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered ShaikMaheer-MSFT commented

Hi @ShreyasiSrivastava-1468 ,

Thank you for posting query in Microsoft Q&A Platform.

Sorry your ask is little unclear to me. What is mean when you say API call worked for 1000 records? You mean your API call will give you 1000 records in one call?

You mentioned that you want to apply custom pagination using until loop. Could you please elaborate more with some simple example?

between, could you please check below video, where pagination handled using ForEach loop and copy activity inside of it? may be this will be helpful in your case.
https://www.youtube.com/watch?v=EbURwy2a8dM

Above video contains Foreach loop and copy activity. If in your case you want to use Until activity then yes you can use it. But make sure to end Until loop at some point by using some flag. Please check below video as well to understand how until loop works.
https://www.youtube.com/watch?v=n8e_exWMH5k

Hope this helps. Please let us know it goes. Thank you.

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @ShreyasiSrivastava-1468 , Just checking if you get chance to see above and share updates? If above helps, then please consider hitting Accept Answer. Accepted answers helps community as well. Please let us know if any further queries.

0 Votes 0 ·

Hi,
The problem got resolved.
I have used Until Activity with 4 variables i.e. TOP, SKIP, NewCounter and Increment. The expression that I have set for the Until loop is @bool(greaterOrEquals(int(variables('Skip')),int(variables('Top')))).

Until has 3 activities inside, one is Copy data and others are Set Variables.

1.Copy Data Activity- It has Base URL (API)
It also has a Relative URL set to @concat('apiurl?','top=',dataset().top,'&skip=',dataset().skip)
It has two dataset parameter Top and Skip which has dataset properties set to @variables('Top')
and @variables('Skip')

  1. Set Variable- Given Name is Increment_Counter
    (Temp Variable) Variable- NewCounter Value-@string(add(int(variables('Skip')), int(variables('Increment'))))

  2. Set Variable- Given Name is Set_Skip
    Variable- Skip Value- @variables('NewCounter')


0 Votes 0 ·
ShaikMaheer-MSFT avatar image ShaikMaheer-MSFT ShreyasiSrivastava-1468 ·

Hi @ShreyasiSrivastava-1468 , Awesome. Thanks for sharing your work around. This helps community as well. Please consider marking it as Accepted Answer. Thank you.

0 Votes 0 ·
ShreyasiSrivastava-1468 avatar image
0 Votes"
ShreyasiSrivastava-1468 answered ShreyasiSrivastava-1468 published

Hi @ShaikMaheer-MSFT, Following the same line of work. Our task is now to copy files from multiple relative urls of a single API instead of just one into same folder but different files.
For the solution I am using For each loop and I have set a parameter at pipeline level as below-
199267-image.png


I have kept the until activity inside the for loop and for the settings of for loop I have as below-
199282-image.png


Inside for loop it looks like- 199249-image.png


Inside until activity I have one copy data which is copying from api and two variables for top and skip.
199277-image.png


Inside copy data inside data set I have set three parameters
199207-image.png


And passing these values to them
199175-image.png


Inside connection there is api and relative url is set
199257-image.png


I have set the relative url as 199268-image.png


For the sink data set we have set parameter 199235-image.png


For connection in sink dataset
199229-image.png



With all these settings, still pipeline is failing.


image.png (46.8 KiB)
image.png (36.6 KiB)
image.png (23.6 KiB)
image.png (36.6 KiB)
image.png (45.0 KiB)
image.png (43.7 KiB)
image.png (62.4 KiB)
image.png (14.8 KiB)
image.png (30.7 KiB)
image.png (55.7 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.