Data Factory activities stuck in Queued

Chi, Dave 61

Hello!

Since the outage of Microsoft services on 15/03, we started to experience problems with the Data Factory runs in our Azure platform on a daily basis. Every day the ADF activities connected to the self-hosted integration runtime on our on-prem server get stuck in the Queued state. We checked the IR and we notice that the IR itself is in the Running state, but none of the workers are actually running. To resolve this, we tried restarting the IR. This works fine for a while, but after each day the activities start queuing again. We've been restarting the self-hosted IR every day and every time we face the same problem the next day. Before the outage, we never faced this issue. As far as we know we haven't changed any settings.

Roberto Velasco 1 Reputation point

2021-03-19T13:14:19.05+00:00

Hi

Is there anything new about this problem?

I have the same problem in the Production instance, and I have not been able to successfully complete the loads since 15/3
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-03-19T13:51:12.84+00:00

Hi @Chi, Dave , @Roberto Velasco ,

Thanks for reaching out and sorry for your experience with this issue.

I have verified with internal product team and confirmed that there is a potential defect identified in the newer SHIR version (5.2 and above), which could cause the SHIR's capacity degradation under big loads. This might be the reason you are seeing this issue. In order to confirm if this is the exact issue impacting your pipeline runs, I would request you to please share your Queued pipeline runID and activity runID so that I can double check with product team.

Mitigation : While product team is actively working on permeant fix, In order to mitigate the issue, it is recommended to try downgrading your SHIR to version lower than 5.2 and let us know if that helps.

Since apologizes for your experience because of this issue.

Thank you
Chi, Dave 61 Reputation points

2021-03-19T14:12:17.717+00:00

Hi @KranthiPakala-MSFT ,

Thanks for your answer! Can you check pipeline runID 63e671dd-c3f6-45f3-9571-23bc31b33b03 and activity runID 341cc605-0685-47c8-b9f9-17c04ca5bd20?
Danny van Buren 16 Reputation points

2021-05-19T07:13:07.267+00:00

Hi @KranthiPakala-MSFT ,

I ran into this problem around the same time as the ticket was opened.
The proposed solution fixed the issue, but currently we are again having similar issues.
Are newer versions of the integration runtime stable at the moment?
We are still running on the Integration Runtime version 5.2.7740.4.

Is there anything known about current issues?

Example Runid: e4d6945c-7101-4e5d-b9ee-efd8b5aedd92 , c903cbe7-8cd4-482a-9a0f-33392248ca67
Abhijith Kadam 0 Reputation points

2024-04-30T10:07:43.7266667+00:00

Above problem has been re-surfaced. I'm trying to run Databricks notebook from ADF its in Queue state forever.

Accepted answer

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-03-19T14:00:55.43+00:00

Hi @Chi, Dave , @Roberto Velasco ,

Update: A QFE SHIR is available in download center now. Product team have already sent a notification to the potential impacted customers with the mitigation steps and QFE download link.

Here is the link: https://download.microsoft.com/download/E/4/7/E4771905-1079-445B-8BF9-8A1A075D8A10/IntegrationRuntime_5.2.7740.4.msi. Please download and do the manually update, in the meanwhile a new round of auto update is also in progress by the product team.

Let us know how it goes.

Thanks
Please sign in to rate this answer.
Roberto Velasco 1 Reputation point

2021-03-19T15:31:00.077+00:00

Thanks for the feedback.

The SHIR version 5.2.7740.4 is stable?

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-03-19T18:07:00.09+00:00

Thanks for your response @Roberto Velasco . Yes it is a Stable version.

Do let us know if you see any issue after installing the suggestion QFE version.

Thanks

Roberto Velasco 1 Reputation point

2021-03-22T19:11:12.627+00:00

Hi

I downgraded to Integration Runtime version 4.15.7611.1.

After downgraded, the ambient appears stable.

Thankyou for help.

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-03-23T23:55:46.583+00:00

Thanks for confirming @Roberto Velasco ! :)

Chi, Dave 61 Reputation points

2021-04-12T09:42:28.707+00:00

Hi @KranthiPakala-MSFT ,

Sorry for the late response. I downloaded version 5.2.7740.4 and it's a stable version. Thanks for your help!
Sign in to comment

Share via

Data Factory activities stuck in Queued

0 additional answers