Thanks for using MS Q&A platform and posting your query.
The problem you're facing with the end condition in Azure Data Factory's copy activity pagination rule is a common challenge when dealing with dynamic values within a loop. Here are two possible solutions to achieve the desired outcome:
Using a Until Loop
- Set a Variable: Create a variable in ADF to store the link value from the previous iteration (e.g.,
previousLink
). - Until Loop: Implement a
Until
loop within the foreach activity. The loop condition should be@equals(variables('previousLink'), @activity('CopyActivity1').output.firstRow.Links[0].href)
. This checks if thepreviousLink
variable is equal to the first element'shref
property in the current iteration'sLinks
array. - Update Variable: Inside the loop, after the copy activity (
CopyActivity1
in this example) finishes, update thepreviousLink
variable with the current iteration's first element'shref
property using a similar expression:set_variable('previousLink', @activity('CopyActivity1').output.firstRow.Links[0].href
Custom Script
- Add a Web Activity: Add a web activity after the copy activity within the foreach loop.
- Script Logic: Within the web activity, use a scripting language like PowerShell or Python to access the previous iteration's output and compare the
href
values as needed. You can leverage ADF functions likeactivity('CopyActivity1').output
to access the previous copy activity's output. - Exit Code: Based on the comparison result, set the web activity's exit code to indicate the loop continuation (non-zero) or termination (zero).
Hope this helps. Do let us know if you any further queries.