question

ShannonMoran-4889 avatar image
0 Votes"
ShannonMoran-4889 asked MeekhaESajiBLRGSS-4627 commented

Azure data factory REST API with pagination

I am new to Data Factory and trying to use copy data function to pull data from Delighted API, which is paginated. I have successfully configured a different dataflow that pulls data from Delighted which is not paginated.

There are two parameters that you pass via form-data in curl.. per_page and page. i have tested this in Postman and have pasted the Code snippet for reference. I can't figure out how to configure this in ADF and would greatly appreciate any help. The API doesn't appear to pass value to indicate end of pages or number of pages.

121266-image.png


azure-data-factoryazure-synapse-analytics
image.png (22.8 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

II did find an article that mentioned entering pagination rules: AbsoluteURL $.d.__next which pulled back to records and failed with this error message.


Failure happened on 'Source' side. ErrorCode=UserErrorMoreThanOneObjectsReturned,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=More than one objects returned\r\n{<!-- -->


0 Votes 0 ·

Hello @ShannonMoran-4889 and welcome to Microsoft Q&A. You are correct that the pagination features of the Copy Activity REST API connector do not support this particular mechanism, sadly. I do have a work-around utilizing a loop of Copy Activities, each instance passed a different page value.

If you know how many pages there are in total, you can use the range(start_page,num_pages) function to generate an array of page numbers. This then can be passed to a forEach loop to be iterated over. Inside the loop place the copy activity, with the page number bein @item().

If you do not know the total number of pages, let me know and I will share a different solution.

0 Votes 0 ·

1 Answer

ShannonMoran-4889 avatar image
0 Votes"
ShannonMoran-4889 answered MeekhaESajiBLRGSS-4627 commented

Hello @MartinJaffer-MSFT

Thank you so much for the response.

Unfortunately this API doesn't support GET number of pages. I will figure out how to deal with that later once I can find solution to this:

This API requires URL parameters to change page in place of ADF not handling pagination properly.

As an example: https://api.delighted.com/v1/survey_responses.json?per_page=100&page=3

Using the copy data activity I was not able to use the Request Body to add the page parameter value, nor was I able to append to URL in the Linked Service from the pipeline array ex. ["1","2"... ].

Since I could not figure out how to pass the page number from parameter URL to the LinkedService URL, I switched to using the Web Activity, which is fetching the data properly within the parameterized ForLoop... I can't determine how to get the output data from Web Activity to sink to a CSV (Not JSON if possible - or could you help me with article on how to convert JSON file to CSV) file in blob storage with unique name for each page ex... file_[pipeline parameter value]





· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi, @ShannonMoran-4889, can you tell, how did you set up the webactivity to do that.., I tried , but I am getting the 1028 error, message:\r\r \n, FailureType: UserError.

0 Votes 0 ·