question

KUMARAbhishek-2110 avatar image
KUMARAbhishek-2110 asked ·

How to Use Dynamic values in pagination Rule (Azure data factory)?

I'm using pagination rule in copy data activity from rest endpoint to blob storage.

I have applied pagination rule with dynamic value like this:

AbsoluteUrl = @replace('$.nextLink','old_base_url','new_base_url')

But it is still accessing the old_base_url which is I'm getting in response, doesn't replace it with new through string function replace.

Error in data factory:

ErrorCode=UserErrorFailToReadFromRestResource,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred while sending the request.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.Http.HttpRequestException,Message=An error occurred while sending the request.,Source=mscorlib,''Type=System.Net.WebException,Message=Unable to connect to the remote server,Source=System,''Type=System.Net.Sockets.SocketException,Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond xx.xx.xx.xx:443,Source=System,'", "failureType": "UserError", "target": "cdr", "details": [] }

@KranthiPakala-MSFT
@SubhashVasarapu-8577

I have also asked this here
https://github.com/MicrosoftDocs/azure-docs/issues/56266

azure-data-factory
4 comments
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Also tried this but not working

9040-capture1.png


1 Vote 1 · ·
capture1.png (3.7 KiB)

@ChiragMishraMSFT-1092

that solution was not working.

0 Votes 0 · ·

Based on the official document, ADF pagination rules only support below patterns.

FxARs.png

I think you could adopt the pattern: Next request’s query parameter = property value in current response body to set the page size, then pass it into next request as parameter.

As mentioned in the rules, the connector will stop iterating when it gets HTTP status code 204 (No Content), or any of the JSONPath expression in "paginationRules" returns null.

Ref - https://stackoverflow.com/a/56436989/10653466

We are working internally to see if there could be a workaround for it.

0 Votes 0 · ·

I have already gone through the official docs from Microsoft for pagination support.

I need to pass the AbsoluteUrl not the query parameter in this case :
because i'm using usage API for Azurestack

https://docs.microsoft.com/en-us/rest/api/consumption/usagedetails/list#uri-parameters

and replacing the base URL is very important for the pagination as we cannot access directly the azurestack URL from datafactory, we have go through the relay proxy.

0 Votes 0 · ·
KranthiPakala-MSFT avatar image
KranthiPakala-MSFT answered ·

Hi @KUMARAbhishek-2110 ,

I have communicated with @ChiragMishraMSFT-1092 about this and we have verified with ADF engineering team and confirmed that the requirement you are trying to achieve is not supported.

We request you to please provide your suggestion/feedback in ADF feedback forum and do share the feedback link here, so that other users with similar idea/suggestion can up-vote and/or comment on your feedback to increase the priority of the feature request.

ADF User voice forum: https://feedback.azure.com/forums/270578-azure-data-factory

All the feedback shared in this forum are actively monitored and reviewed by ADF engineering team.



Thank you

Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.




Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KUMARAbhishek-2110 avatar image
KUMARAbhishek-2110 answered ·
1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks for sharing the feedback link here @KUMARAbhishek-2110 .

0 Votes 0 · ·