question

sesuk-8939 avatar image
0 Votes"
sesuk-8939 asked MartinJaffer-MSFT commented

How to pass pagination token during the API request(HTTP Connector) in copy activity ?

Hi,

I have an API that provides the response with 200 results and for the other records, I need to use the token string which I get from the XML response. the offset token is available in the root of the XML

I need to request for the next page till I get the token string in the response.

I know we have a REST connector that supports the pagination. But I am trying with the HTTP Connector and I would like to know if this can be achieved in my Data factory.

azure-data-factory
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @sesuk-8939 and welcome to Microsoft Q&A.

The HTTP Connector does not have pagination support. However, it can still be used in a pagination scenario by placing the copy activity inside a loop. In this case you will need to parameterize the HTTP dataset, so the pagination details can be passed in at runtime.

The exact details of how to implement and advance the pagination depend upon the pagination method.

For cases where it is merely a numbered offset, and you know how many pages, the details can be pre-computed and iterated over via forEach loop.

For cases where the next page token is unknown , or is delivered in the response of each request, the loop must be done sequentially, perhaps using an until loop.
A web activity may be necessary to fetch the next page token, if it is embedded in the data.

0 Votes 0 ·

Hi, @MartinJaffer-MSFT Thanks for the information. And yes in my case the token will be delivered in response to each request. So I am trying to use until activity and get a token for each request and setting a variable. But I am getting an error when I try to set it as a variable.

The expression 'activity('Lookup1').output.value.Token' cannot be evaluated because property 'Token' cannot be selected. Array elements can only be selected using an integer index. I understand we need to use value[0].

But my token is the last column @Token": "dfasdfsdfty" in the value array and outside the request root array I am not sure how to add this in a set variable?

Could you please provide me some clarity?

0 Votes 0 ·

@sesuk-8939 did my response solve your issue? If so , please mark as accepted answer. If you still need more help, please tell me how I may better assist.

0 Votes 0 ·

1 Answer

MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered

@sesuk-8939 My apologies for the delay. Could you share a brief sample of what the return looks like? (redact if needed)

It sounds like you are using a Lookup on top of an HTTP JSON dataset.
This type returns the whole response.

Let me share some bits that might help.

For an example, I use a Lookup (http JSON) pointing to https://reqres.in/api/users

108286-image.png

If I wanted to get the last object in an array, I used expression:

 @{activity('Lookup1').output.value[0].data[sub(length(activity('Lookup1').output.value[0].data),1)]}

To simplify, I get the length of the array, subtract 1, and use the result in index of array.

The [0] in output.value[0] comes from the lookup activity. This would change if I enabled the "First row only" option in lookup activity. Before you think too hard, remember that Lookup activity can also be used on SQL or other tabular sources which have rows.


image.png (105.7 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.