How to Fetch 1000 records data from Rest API connect dynamically with Parameters

Question

Hi,

I am trying to fetch 1000 using the rest API connector link services in ADF and storing in ADLS but I am not able to set pagination for the URL.

Example of my URL :https://api.instantwebtools.net/v1/passenger?page=1&size=10

Answer 1

Hello @KumariShwetha-1759

I am assuming that you are making use of the Copy Data activity to move data from REST API to ADLS. If this is not case. Please feel free to provide some context on the activity.

The copy data by default supports the pagination for the below scenarios :

Reference : https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support

Now, coming back to your scenario

For fetching 1000 records, I am guessing that you would like recursively hit the page like below

https://api.instantwebtools.net/v1/passenger?page=1&size=10  
https://api.instantwebtools.net/v1/passenger?page=2&size=10  
https://api.instantwebtools.net/v1/passenger?page=3&size=10  
...  
...  
...  
https://api.instantwebtools.net/v1/passenger?page=10&size=10  
   

if the response header/response body have the url for the next page ( for instance, https://api.instantwebtools.net/v1/passenger?page=2&size=10) , you could configure it in the below way.

If the next page link or query parameter doesn't come as response headers/body then you're less likely to be able to use the feature.

The alternative would be to make use of loop logic & Perform the Copy Activity

In the Rest Connector create two parameters :

Now populate the relative URL for the RestConnector with the parameters.

@concat(dataset().page,dataset().size)  

This is to form the Relative URL dynamically. In your case "?page=1&size=10"

Now, In the Copy Activity :

You could set the page number as a Variable.

@concat('?page=',variables('page'))  

This variable value would be incremented in a loop by making use of the Set Variable activity. for each iteration --> ?page=1, ?page=2,?page=3

The URL for the Copy Activity is dynamically set for each iteration.

To Loop / iterate - You could either make use of Until activity

Condition : Until the variable is less than 100 or so.

After the page 1 is executed - the variable is updated 2.

The request is made with the variable value 2 i.e. the page 2 is queried.

https://reqres.in/api/users?page=2&size=10  

OR

FOREACH Loop (The implementation is discussed here : https://learn.microsoft.com/en-us/answers/questions/59467/how-to-pass-pagination-rules-during-a-rest-api-cal.html)

Note : if needed you could then merge the data for all the files (generated for each page)

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators