Azure Integration Runtime Error on Auto Resolve or East Europe region

Illia Klievtsov 1 Reputation point
2021-08-10T15:13:42.95+00:00

Hi all.

Last month I started to receive this error on 2 of my Azure Integration Runtime ( Azure 1 and Azure 2).
"There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime 'AzureCluster1-Quick', ActivityId: '5dadd0b2-03ba-47fb-8177-21e2ea576615'"

Here is some of my investigations:

  • All 4 of my runtimes had Auto Resolve region.
  • There is no concurent executions, got failure on the very first step or Debug via default IR.
  • If I switch the same Data Flows to my other Azure IR (3 or 4) works better but sometimes have same error.
  • If I change region to East Europe - same error.
  • If I change region to other regions - no error.
  • My Azure Database placed East Europe.

Currently got error every time. Have the same infrastructure on Dev/Staging/Preprod/Prod environments. There are problems only with Staging one. Previously had some migration between resourse groups for this Data Factory (changing subscriptions). Can be the reason. If so - please provide me instructions how to avoid it on other env's.

Here are some other Activity Id's:
Azure Runtime 1
'5dadd0b2-03ba-47fb-8177-21e2ea576615'
'455c1e7d-89a1-4752-b5db-a9282e3e56b5'
'007e8d92-5b41-455f-8a43-6ad05568e73b'

Azure Runtime 2
'337ce6dd-0b50-4c5a-a3fd-cdb970342db4'
'f7f44127-1b00-4009-84ea-30c65cb178c3'
'ec85b394-1388-4dde-98f9-6134acc6e688'

Will be thanful for any help.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,609 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee
    2021-08-11T05:45:43.9+00:00

    Hi @Illia Klievtsov ,

    Welcome to Microsoft Q&A Platform. Thank you for posting query on Microsoft Q&A Platform.

    You may have reached integration runtimes capacity limit or you might be running large amount of data flow using same integration runtime at same time.

    You should consider running your pipeline data flow at different times or split your data flow execution's to across multiple integration runtimes.

    Kindly check below thread where detailed resolution is discussed.
    https://learn.microsoft.com/en-us/answers/questions/282303/azure-datafactory-error.html

    Hope this will help. Please let us know if any further queries. Thank you.

    ------------------------------

    • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification.