ADF Rest API Copy Tool - How to configure pagination using partial URL from response.

Question

Trying to configure ADF pipeline to read from a REST API and write to SQL db using pagination.

I'm able to read data from the API and write to SQL, but only the first 200 records (due to default pagination from vendor)

Here is a sample response showing the json syntax.

I'm attempting to concat the base url (https://api.weaveconnect.com/) with the variable part of the url listed in the response at meta.links.next

Here is where the API documentation refers to pagination:

I would be grateful for any guidance around configuring the pagination. I've been all over learn.microsoft, stackoverflow, youtube, etc. and can't find the correct setup.

Thanks in advance.

Answer 1

Your pagination is this case is a little bit tricky, here is my idea :

1. Create a Lookup activity to retrieve the initial set of data from the REST API. In the settings of the Lookup activity, configure it to retrieve the data from the base URL (https://api.weaveconnect.com/). Don't forget to adjust the pagination mode and the number of records to retrieve per request (if needed)
2. Then add a ForEach activity after the Lookup activity to iterate over the results you obtained from the Lookup activity where you need to specify the items to iterate over, which will be the records retrieved in the previous step.
3. Inside the ForEach activity, use an expression to append the partial URL (meta.links.next) from each record to the base URL. You can concatenate the base URL with the value of meta.links.next and then store this concatenated URL in a variable.
4. Add another Lookup activity inside the ForEach activity to get data from the dynamically generated URL stored in the variable.

Don't forget to implement control flow logic to handle when to stop pagination. You can set conditions based on whether the meta.links.next field is present in the response. If it's not present, it indicates that there are no more pages to retrieve, and the pagination process can be stopped.