Using Until for looping copy activity which has Rest Api call set as source

Shreyasi Srivastava 21 Reputation points
2022-03-25T19:00:37.82+00:00

Hi,
I have created a copy activity which has source for REST API and Sink as Azure ADLS. It work for the 1000 records.

I need to apply custom pagination to track if has more than 1000 records any time. I am trying to parameterize the copy activity dataset but not sure how to use until for looping in this case.

How can I achieve this custom pagination?
Please help ASAP.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,339 questions
0 comments No comments
{count} votes

Accepted answer
  1. ShaikMaheer-MSFT 37,566 Reputation points Microsoft Employee
    2022-03-29T10:13:57.237+00:00

    Hi @Shreyasi Srivastava ,

    Thank you for posting query in Microsoft Q&A Platform.

    Sorry your ask is little unclear to me. What is mean when you say API call worked for 1000 records? You mean your API call will give you 1000 records in one call?

    You mentioned that you want to apply custom pagination using until loop. Could you please elaborate more with some simple example?

    between, could you please check below video, where pagination handled using ForEach loop and copy activity inside of it? may be this will be helpful in your case.
    https://www.youtube.com/watch?v=EbURwy2a8dM

    Above video contains Foreach loop and copy activity. If in your case you want to use Until activity then yes you can use it. But make sure to end Until loop at some point by using some flag. Please check below video as well to understand how until loop works.
    https://www.youtube.com/watch?v=n8e_exWMH5k

    Hope this helps. Please let us know it goes. Thank you.


1 additional answer

Sort by: Most helpful
  1. Shreyasi Srivastava 21 Reputation points
    2022-05-05T14:40:12.62+00:00

    Hi @ShaikMaheer-MSFT , Following the same line of work. Our task is now to copy files from multiple relative urls of a single API instead of just one into same folder but different files.
    For the solution I am using For each loop and I have set a parameter at pipeline level as below-
    199267-image.png

    I have kept the until activity inside the for loop and for the settings of for loop I have as below-
    199282-image.png

    Inside for loop it looks like- 199249-image.png

    Inside until activity I have one copy data which is copying from api and two variables for top and skip.
    199277-image.png

    Inside copy data inside data set I have set three parameters
    199207-image.png

    And passing these values to them
    199175-image.png

    Inside connection there is api and relative url is set
    199257-image.png

    I have set the relative url as 199268-image.png

    For the sink data set we have set parameter 199235-image.png

    For connection in sink dataset
    199229-image.png

    With all these settings, still pipeline is failing.

    0 comments No comments