how to perform pagination in azure data factory using Rest API?

Navdeep Kaur 21

Can you please share in detail with steps.

1 answer

Saurabh Sharma 23,751 Reputation points Microsoft Employee

2021-01-19T20:26:55.897+00:00
@Navdeep Kaur Thanks for using Microsoft Q&A !!

Can you please share what kind of pagination you are trying to implement ?

If your API response contains a property that points to the next page url then it can be used to load the next page using the "AbsoluteUrl" key in the pagination rules page in your ADF. As mentioned in the example Facebook Graph API returns response in the below structure:
{ ...
"paging" : {
"previous": "https://graph.facebook.com/me/albums?limit=25&before=NDMyNzQyODI3OTQw",
"next": "https://graph.facebook.com/me/albums?limit=25&after=MTAxNTExOTQ1MjAwNzI5NDE="
}
}
so your pagination rule will look like below:

If you are using a Microsoft Graph API like https://graph.microsoft.com/v1.0/users then Microsoft Graph returns the next url as "@odata.nextLink" in the body as shown below :

You pagination rule will be something like below:

Please refer to the official documentation to get the supported pagination rules:

Next request’s absolute or relative URL = property value in current response body

Next request’s absolute or relative URL = header value in current response headers

Next request’s query parameter = property value in current response body

Next request’s query parameter = header value in current response headers

Next request’s header = property value in current response body

Next request’s header = header value in current response headers

You can also refer to documentation to get more on using REST in copy data.
Please sign in to rate this answer.

2 people found this answer helpful.
Saurabh Sharma 23,751 Reputation points Microsoft Employee

2021-01-22T19:54:07.5+00:00

@Navdeep Kaur Please let me know if you find above reply useful. If yes, do click on 'Mark as answer' link in above reply. This will help other community members facing similar query to refer to this solution. Thanks.

Saurabh Sharma 23,751 Reputation points Microsoft Employee

2021-01-26T18:10:36.707+00:00

@Navdeep Kaur Following up to know if you are able to look into my previous response.

Saurabh Sharma 23,751 Reputation points Microsoft Employee

2021-01-28T18:36:52.28+00:00

@Navdeep Kaur I have not heard back from you. Did my answer solve your issue? If so, please mark as accepted answer. If not, please let me know how I may better assist.

Dervishi, Erald 66 Reputation points

2021-03-29T18:24:19.33+00:00

Hello @Saurabh Sharma , I have a scenario related to this post I would appreciate if you can give me any hints.
So basically I have a resource REST API that I would like to get all the data from it and insert to a table in SQL Server, but the request it includes pagination at the end, so basically when I do the request I get one part of the data, and at the end I have pagination as shown below, what I want is the whole data including the pagination.

"Paging": {
"Next": "https://ustest60-services.dayforcehcm.com/Api/winn/V1/Reports/Report?cursor=7bCOeZDzU9MuYYaPMapyBnDk%252Ff33ZjHfeCu5bTTl2nGcA233yrTuSFjbKlW%252FjpCm"
}
How can I map in the sink part to retrieve the data from the
"Next": "https://ustest60-services.dayforcehcm.com/Api/winn/V1/Reports/Report?cursor=7bCOeZDzU9MuYYaPMapyBnDk%252Ff33ZjHfeCu5bTTl2nGcA233yrTuSFjbKlW%252FjpCm"

Thank You.

Fabio Carello 31 Reputation points

2021-12-13T11:32:51.057+00:00

Is this feature fully supported on Data flow too? It seems not. This is my post on StackOverflow describing the issue.
https://stackoverflow.com/questions/70302879/data-factory-data-flow-rest-paging

SanjayR 1 Reputation point

2021-12-16T07:25:10.123+00:00

Hello @Fabio Carello ,

Am also having same issue and working with MS support for more than 2 weeks now but no resolution yet.

Just wanted share one thing that at once it did worked for me to fetch more that 1000 records after increasing row sample to more than 1000 in Debug settings and also changing Request interval (ms) from 0 to 10. But since then it had never worked to fetch more than 1000 :(

Thanks,
Sanjay

CC: @Saurabh Sharma SaurabhSharma-msft

Saurabh Sharma 23,751 Reputation points Microsoft Employee

2021-12-21T19:59:01+00:00

Hi @SanjayR ,

sorry for the trouble. Can you please share the support ticket number over an email to azcommunity[at]microsoft[dot]com with this Q&A thread link and subject as "Attn: Saurabh' and I can try checking internally on the same.

Thanks
Saurabh

Ruben Aster 6 Reputation points

2022-07-04T17:23:48.677+00:00

Does the OData connector does pagination automatically? It doesn't contain pagination options.
Sign in to comment