question

ShaikRiyazSGRECOGDVLSW-9079 avatar image
0 Votes"
ShaikRiyazSGRECOGDVLSW-9079 asked KranthiPakala-MSFT commented

Azure data factory pipelines are sometimes stuck in status queued and takes 1-5 mins to kick off

Hi Team
we have a setup of event trigger(blob storage file upload) which triggers the data factory pipeline. And the adf pipelines runs sequentially. Randomly we have observed that in (approx) 2 out of 10 pipeline runs are stuck in queued and takes about 1-5 mins to kick off the first activity/or move the status to in progress.

Example Run id for reference: a4611573-d389-40da-a7bf-c74692167003
Trigger name: TRG_EQUIPMENT_DETAILS
Pipeline name: PL_EquipmentDetailsToSQL
Region: West Europe
No self hosted IR is used in this pipeline

Please advise what is the issue here.

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @ShaikRiyazSGRECOGDVLSW-9079,

Welcome to Microsoft Q&A forum and thanks for reaching out.

From the description provided, only few runs are queued for before they actually start executing the activities. This usually happens when a number is set for concurrent pipeline runs: from pipeline paneSettingsConcurrency property. Please check if this property is set to 1. If yes, then that confirms why the pipeline runs are getting queued. Also I would recommend to check when the pipelines runs are queued, are they any in-progress pipeline runs?


101561-image.png


If that is not the case, then you might be using an Azure Integration Runtime that is in a not optimal region resulting in slow read/write, suggest configuring to use an IR in another region which is closer to your data store.

Also please refer to this document and see if that helps: Troubleshoot copy activity on Azure IR

Hope this info helps. Do let us know if you have further query.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.




image.png (25.7 KiB)
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @ShaikRiyazSGRECOGDVLSW-9079,

Just checking to see if the above information was helpful? Do let us know if you have further query and if the above information helps, please do consider to accept answer and up-vote as it can be beneficial to other users.

Thank you

0 Votes 0 ·

Hi @ShaikRiyazSGRECOGDVLSW-9079,

We still have not heard back from you. Just wanted to check if you are still facing the issue or need assistance on this? In case If you already found a solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue. Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

0 Votes 0 ·

@KranthiPakala-MSFT yes we have setup the concurrency level to 1 intentionally, as we want them to run pipelines to run in sequence.
The issue is not that it is getting queued actually. Lets say there are 10 pipeline runs in queue. We see 7-8 out of 10 runs the next pipeline kicks off immediately once a previous run is completed. However noticed 2-3 pipelines doesn't kick off immediately and waits some more time (between 1-5 mins). And these delays are at pipeline level not at activity level.
And the integration runtime we are using is Azure IR not self hosted in this scenario. And as this is happening at pipeline level that it is taking time before kicking off the pipeline, i am thinking it might not even be IR issue.

Any advise what this could be.

0 Votes 0 ·
KranthiPakala-MSFT avatar image KranthiPakala-MSFT ShaikRiyazSGRECOGDVLSW-9079 ·

Hey @ShaikRiyazSGRECOGDVLSW-9079 sorry for the delayed response and thanks for sharing additional clarification. Could you please provide few consecutive pipeline runID's where the behavior is Run1 executed without delay, and next run2 delayed and next run 3 executed without delay. This would help us to troubleshoot better. In case if this is a PROD issue and a blocker, if you have a I would recommend filing a support ticket else let us know.

We look forward to your response.

0 Votes 0 ·

Hi @ShaikRiyazSGRECOGDVLSW-9079,

We still have not heard back from you. Could you please share the above requested details for further analysis?
Thanks

0 Votes 0 ·