question

IlliaKlievtsov-2622 avatar image
0 Votes"
IlliaKlievtsov-2622 asked ShaikMaheer-MSFT commented

Azure Integration Runtime Error on Auto Resolve or East Europe region

Hi all.

Last month I started to receive this error on 2 of my Azure Integration Runtime ( Azure 1 and Azure 2).
"There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime 'AzureCluster1-Quick', ActivityId: '5dadd0b2-03ba-47fb-8177-21e2ea576615'"

Here is some of my investigations:
- All 4 of my runtimes had Auto Resolve region.
- There is no concurent executions, got failure on the very first step or Debug via default IR.
- If I switch the same Data Flows to my other Azure IR (3 or 4) works better but sometimes have same error.
- If I change region to East Europe - same error.
- If I change region to other regions - no error.
- My Azure Database placed East Europe.

Currently got error every time. Have the same infrastructure on Dev/Staging/Preprod/Prod environments. There are problems only with Staging one. Previously had some migration between resourse groups for this Data Factory (changing subscriptions). Can be the reason. If so - please provide me instructions how to avoid it on other env's.

Here are some other Activity Id's:
Azure Runtime 1
'5dadd0b2-03ba-47fb-8177-21e2ea576615'
'455c1e7d-89a1-4752-b5db-a9282e3e56b5'
'007e8d92-5b41-455f-8a43-6ad05568e73b'

Azure Runtime 2
'337ce6dd-0b50-4c5a-a3fd-cdb970342db4'
'f7f44127-1b00-4009-84ea-30c65cb178c3'
'ec85b394-1388-4dde-98f9-6134acc6e688'


Will be thanful for any help.

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered ShaikMaheer-MSFT commented

Hi @IlliaKlievtsov-2622 ,

Welcome to Microsoft Q&A Platform. Thank you for posting query on Microsoft Q&A Platform.

You may have reached integration runtimes capacity limit or you might be running large amount of data flow using same integration runtime at same time.

You should consider running your pipeline data flow at different times or split your data flow execution's to across multiple integration runtimes.

Kindly check below thread where detailed resolution is discussed.
https://docs.microsoft.com/en-us/answers/questions/282303/azure-datafactory-error.html

Hope this will help. Please let us know if any further queries. Thank you.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.

· 8
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi,
Thank you for your answer.

I read that resolution, and don't think that's my case.

No concurent data flows. No paralelism. Amount of data can be < 1000 rows. Even debug for 1 data flow will be failed.

The same configuration not causing issues on other environments.

My staging runs really rare. Please explain if "reached integration runtimes capacity limit" means smth else.

Regards,
Illia

0 Votes 0 ·
ShaikMaheer-MSFT avatar image ShaikMaheer-MSFT IlliaKlievtsov-2622 ·

Hi @IlliaKlievtsov-2622 ,

We are reaching internal team on this to check more details. We will revert back soon with updates. Thank you.

0 Votes 0 ·

Hi @ShaikMaheer-MSFT , any updates on this?

0 Votes 0 ·
ShaikMaheer-MSFT avatar image ShaikMaheer-MSFT IlliaKlievtsov-2622 ·

Hi @IlliaKlievtsov-2622 ,

Thank you for your response. I was waiting for your inputs from long time to debug this further. Could you please check above private messages and share requested details. That will help to dig in to this issue more. Thank you.

0 Votes 0 ·

Sent you an email with the details on August, 17

0 Votes 0 ·
Show more comments