Hi Experts, Something strange observed with the Until activity in our ADF pipeline. We have some series of activities(7activities) inside the 'Until' activity; We call an external api which produces the result in paginated fashion and the Until activity is basically iterating through multiple pages of the api response with the counter approach and Until acitivity is the very last activity of the ADF pipeline.
As seen in the above log picture, the Until activity(Loop all Pages) started at 9:40:35pm and it had to go through approx. 174 iterations to get the complete data from the api. The expression inside Until activity is shown below. @equals(variables('counter'),string(int(variables('maxpage')))) Inside the loop, we keep increasing the counter and that counter is being evaluated against the maxpage variable which, for instance, is 174. The last activity(swap counter) inside the Until loop(from the last iteration) was completed at 11:04:55pm as shown in the below log picture. After this last activity, the Until activity should immediately turn to 'Succeeded' status at somewhere close to 11:04:55pm but it actually was completed at 11:24:57pm as you can see this in the log of the duration of 'Loop all Pages' activity picture; duration says 01:44:20. This means that Until activity has taken 20mins more to turn to 'Succeeded' status eventhough all the inner activities are completed and the condition expression evaluates to true. Why is the Until activity spending extra minutes after all inner activities are completed. I have tested this by running the pipeline at different timings of the day and same behavior was observed. If I run the pipeline for smaller date range wherein the number of iterations is less, say 3 or 4, the extra time it spends is close to 1min or 2mins. So, more the iterations, more the time spends(wasted) though all inner activities are done. Is there something that can be done to avoid this extra time wastage?
