question

DervishiErald avatar image
0 Votes"
DervishiErald asked MartinJaffer-MSFT commented

Pagination in azure data factory using Rest API

Hello, Team
I have a project scenario that requires me to use REST APIs with pagination, basically I have a resource REST API that I would like to get all the data from it and insert to a table in SQL Server, but the request it includes paging at the end, so basically when I do the request I get one part of the data, and at the end I have pagination as shown below, what I want is the whole data including the paging.

"Paging": {
"Next": "https://ustest60-services.dayforcehcm.com/Api/winn/V1/Reports/Report?cursor=7bCOeZDzU9MuYYaPMapyBnDk%252Ff33ZjHfeCu5bTTl2nGcA233yrTuSFjbKlW%252FjpCm";
}

How can I map in the sink part to retrieve the data from the "Next"

I would appreciate any hints or recommendations, also I can provide my pipeline id to check the runs if need it.

Looking forward for a response.
Thank You.

azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@DervishiErald-7950 did my answer help you? If it solved your issue, please mark as accepted answer, otherwise let me know how I may better assist.

0 Votes 0 ·

1 Answer

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered

Hello @DervishiErald-7950 and welcome to Microsoft Q&A.

In the Copy Activity, in the source, try adding an Additional Column to duplicate the paging if it is being removed.

"MyPagingDuplicate" , "$$COLUMN" , "$.Paging.Next"

82991-image.png

Please let me know if this helps. I wasn't able to test against your website, but I was able to duplicate in another one.
82972-image.png



image.png (91.7 KiB)
image.png (32.1 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.