How to send parallel rest-API request through Copy data activity in ADF with Pagination rule AbsoluteURL

Question

The pipeline flow is to send request to rest-api and load the data to a json file in ADLS2. Due to huge data requests, there is a delay in the Copy task completion. To enhance the performance of the ADF pipeline, I wanted to send 4 requests in parallel in the Copy Data activity with Source as rest-API. I have used the pagination rule as AbsoluteUrl. Please find the below screenshot below for the pipeline flow.

Could you please confirm, whether the 4 different requests can be sent in parallel in the Copy task to improve the pipeline performance

Answer 1

Based on what you shared, I think you need to add multiple Copy Data activities in your pipeline.

So, in this case, you need to configure each Copy Data activity to target your REST API endpoint (with each activity targeting different parts of your data if necessary).

Then for each Copy Data activity, in the Source settings, set up the pagination rule where you select AbsoluteUrl and specify the absolute URL for pagination.

Then create a pipeline and add the multiple Copy Data activities created in the 1st step (where to dd multiple Copy Data activities)

You can use the parallel branching feature to execute these activities in parallel. Drag and drop each Copy Data activity in the pipeline editor to create parallel branches.