question

GCFM avatar image
0 Votes"
GCFM asked KranthiPakala-MSFT commented

Azure Data Factory Pipeline Activities Stuck In Queued status

We have been experiencing a number of failed jobs over the last week because of pipeline activities becoming stuck in a queued state. These seemed to have been working until we upgraded to the latest IR version. An example pipelinerunID is c560992a-d974-4806-9c39-1b6b71e5c7f4. We have looked at the IR server and nothing seems to be wrong but we can't figure out why activities keep becoming stuck and won't start.

azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @GCFM,

Thanks for the ask and also for using the Microsoft Q&A forum.

We are reaching out to the internal team to get more help on this, I will update you once we hear back from them.

0 Votes 0 ·

1 Answer

KranthiPakala-MSFT avatar image
1 Vote"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @GCFM,

Sorry for your experience with this issue and thanks for reporting this to us.

I have verified with internal product team and confirmed that there is a potential defect identified in the newer SHIR version (5.2 and above), which could cause the SHIR's capacity degradation under big loads. This might be the reason you are seeing this issue.

For mitigation, a QFE SHIR is available in download center now. Product team have already sent a notification to the potential impacted customers with the mitigation steps and QFE download link.

Here is the link: https://download.microsoft.com/download/E/4/7/E4771905-1079-445B-8BF9-8A1A075D8A10/IntegrationRuntime_5.2.7740.4.msi. Please download and do the manually update, in the meanwhile a new round of auto update is also in progress by the product team.

Apologizes for the inconvenience due to this issue.

Let us know how it goes.

· 8
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you for the info. I will share this with our internal teams.

0 Votes 0 ·

Thanks for your response @GCFM. Do let us know if you notice any issues after installing the suggested SHIR version.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

0 Votes 0 ·
GCFM avatar image GCFM KranthiPakala-MSFT ·

We were seeing a lot of odd network activity during this period. Is there a way to get more details about the bug so we can understand if the network activity we are seeing was impacted?

0 Votes 0 ·

When I try to install the QFE it complains that "The same or newer version already installed". We're running 5.3.77287.2. All the jobs that should be run on that IR are stuck in a queue.

0 Votes 0 ·

@tp-2671 - Thanks for reaching out. Could you please try uninstalling the the version you are having and try to manually download the QFE version and then install it.

Let us know how it goes.

Thanks

0 Votes 0 ·
Tp-2671 avatar image Tp-2671 KranthiPakala-MSFT ·

Restart of the IR service solved the problem for now. I'll look into uninstall option if needed.

Can you give any estimate as for when a proper fix will be issued, we might be able to wait for that?

0 Votes 0 ·
Show more comments