Azure Automation - Jobs stuck in running and stopped after 3 hours

Question

question

AjayKumar-8667 asked Oct 31, '20 SwathiDhanwada-MSFT commented Nov 24, '20

Azure Automation - Jobs stuck in running and stopped after 3 hours

Hi Team,

I have scheduled an automation job to suspend/delete users based on their last sign-in. I am fetching last Sign-ins from Azure Log analytics.

I am getting below error intermittently. Some times it completes within 32 mins without any issue and some time it stopped with below message.
Error -
"The job has been stopped because it reached the fair share limit of job execution more than 3 hours. For long-running jobs, it's recommended to use a Hybrid Runbook Worker. Hybrid Runbook Workers don't have a limitation on how long a runbook can execute. Refer https://docs.microsoft.com/en-us/azure/automation/automation-runbook-execution#fair-share for more details."

Below are the few Azure commands i am using -
1. Connect-AzureAD
2. Connect-azurermaccount
3. Get-AzureADUser
4. Create users batch of 2000
5. Invoke-AzureRmOperationalInsightsQuery -WorkspaceId "WorkspaceID" -Query $query -ErrorAction Stop -wait 360

The job usually connect with Azure and AzureRM but it stuck while fetching the last sign-in logs from Azure log analytics.

What could be the issue? Is it required to use Hybrid Runbook Workers here? Any clue?

More Information -
1. Azure Automation Account - Region North EU
2. Azure Log Analytics - Region West EU

Thanks
Ajay

azure-automation

· 3

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AjayKumar-8667 · Nov 05, 2020 at 08:57 AM

Hi @SwathiDhanwada-MSFT, For avoiding gateway timeout i have added "-wait" parameter and its working fine. But the main issue is intermittent failure is of script with below message -

"The job has been stopped because it reached the fair share limit of job execution more than 3 hours. For long-running jobs, it's recommended to use a Hybrid Runbook Worker. Hybrid Runbook Workers don't have a limitation on how long a runbook can execute."

Are you saying this is due to AzureRM modules? If yes then i will migrate from AzureRM to Az.

Thanks
Ajay Kumar

0 ·

SwathiDhanwada-MSFT AjayKumar-8667 · Nov 05, 2020 at 10:30 AM

@AjayKumar-8667 Migrating from AzureRM to Az is recommended practice but it isn't related to the error.

For the mentioned error , Seems the execution time of PowerShell script has exceeded in your case , I would suggest you to check the PowerShell Commands execution time. Here is a third party article which I mentioned in my previous comment to check the execution time of script and based on the execution time, you can optimize your script or choose to execute the script to execute on Hybrid Runbook Worker.

0 ·

AjayKumar-8667 SwathiDhanwada-MSFT · Nov 05, 2020 at 03:58 PM

Hi @SwathiDhanwada-MSFT, I have run this automation runbook manually by clicking on Start and it is taking approx. 35 mins to complete and also same execution time it takes, when completed intermittently.

I can see it is authenticating with Azure successfully but when try to fetch the logs it keep stuck on running state for 3 hours and the stopped.

Thanks

0 ·

Answer 1 · 2020-10-31T22:53:51.56Z

AndreasBaumgarten answered Oct 31, '20 AndreasBaumgarten commented Nov 1, '20

I would recommend to migrate from AzureRM to Az.
AzureRM is outdated and is replaced by the new Az module.
https://docs.microsoft.com/en-us/powershell/azure/migrate-from-azurerm-to-az?view=azps-5.0.0

Maybe this will help.

The automation job is a new one? Or it was running fine earlier days and now it's causing issues?

(If the reply was helpful please don't forget to upvote and/or accept as answer, thank you)

Regards
Andreas Baumgarten

· 4

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AjayKumar-8667 · Nov 01, 2020 at 08:50 AM

Hi Andreas,

It was working fine till 2 weeks back. Now, its causing issues intermittently. Recently i have added "-wait" parameter. Not sure if this is creating problem. The default wait time is 3 mins and i have extended it to 6.

Invoke-AzureRmOperationalInsightsQuery -WorkspaceId "WorkspaceID" -Query $query -ErrorAction Stop -wait 360

Thanks
Ajay

0 ·

AndreasBaumgarten AjayKumar-8667 · Nov 01, 2020 at 09:12 AM

If you change back to 3 mins wait? The issue is gone? It's worth a try.

(If the reply was helpful please don't forget to upvote and/or accept as answer, thank you)

Regards
Andreas Baumgarten

0 ·

AjayKumar-8667 AndreasBaumgarten · Nov 01, 2020 at 07:57 PM

Thanks Andreas!

I have changed it to 6 mins because previously, was facing below gateway timeout and this was the work around. Is there any other workaround or suggestion?

"Operation returned an invalid status code 'GatewayTimeout'"

Invoke-AzureRmOperationalInsightsQuery : Operation returned an invalid status code 'GatewayTimeout' At line:255 char:26 + ... InLogResult=Invoke-AzureRmOperationalInsightsQuery -WorkspaceId "2200 ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : CloseError: (:) [Invoke-AzureRmOperationalInsightsQuery], ErrorResponseException + FullyQualifiedErrorId : Microsoft.Azure.Commands.OperationalInsights.Query.InvokeOperationalInsightsQuery

Thanks
Ajay Kumar

0 ·

Show more comments

Answer 2 · 2020-11-05T08:23:33.717Z

SwathiDhanwada-MSFT answered Nov 5, '20

@AjayKumar-8667 As mentioned by @AndreasBaumgarten , it is recommended to migrate from AzureRM to Az modules.

To your questions related to the error, please note that maximum run time allowed for runbook using Azure Sandbox is 3 hours. Seems the runtime has exceeded in your case. Hence , you were prompted with the mentioned error ""The job has been stopped because it reached the fair share limit of job execution more than 3 hours. For long-running jobs, it's recommended to use a Hybrid Runbook Worker. Hybrid Runbook Workers don't have a limitation on how long a runbook can execute. Refer https://docs.microsoft.com/en-us/azure/automation/automation-runbook-execution#fair-share for more details."

Kindly note if a query takes longer than the specified timeout (or default timeout, if unspecificed), it will fail with an status code of 504 Gateway Timeout which you have noticed when executing Invoke-AzureRmOperationalInsightsQuery command.

To check the execution time of your PowerShell Script, I would suggest you to check this third party article which helps to measure time taken to execute the script. Based on your execution time, you can choose to improvise your script or select hybrid runbook worker for execution.

image.png (4.4 KiB)

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 3 · 2020-11-06T15:15:46.147Z

sadomovalex answered Nov 6, '20 AjayKumar-8667 commented Nov 6, '20

did you try to run the same code locally (not from Azure)? Does it also stuck in this case?

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AjayKumar-8667 · Nov 06, 2020 at 03:43 PM

Hi @sadomovalex, Yes, i have executed it on locally as well. No issues. I am trying to migrate from AzureRM to Az.

0 ·

Answer 4 · 2020-11-07T04:33:54.55Z

SwathiDhanwada-MSFT answered Nov 7, '20 SwathiDhanwada-MSFT commented Nov 12, '20

@AjayKumar-8667 Kindly note authentication using AAD inside Azure Runbooks using Azure Sandbox is not supported. The only options are using an Hybrid worker or using Azure Functions.

Reference Links:

image.png (14.5 KiB)

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SwathiDhanwada-MSFT · Nov 12, 2020 at 03:35 PM

@AjayKumar-8667 Hope the above information is helpful. Kindly let us know if you need further assistance.

0 ·

Answer 5 · 2020-11-12T15:55:05.49Z

AjayKumar-8667 answered Nov 12, '20 SwathiDhanwada-MSFT commented Nov 24, '20

Hi @SwathiDhanwada-MSFT, Currently i am working on AzureRM to Az migration and getting below warning with Connect-AzAccount. Can you please share your thoughts on this? Although i get the results.

Warning -
TenantId 'XXXXX' contains more than one active subscription. First one will be selected for further use. To select another subscription, use Set-AzContext.

I can see in the below link that there is an issue with Az module > version1.9.5. I am using version 2.1.2.

Link -
https://github.com/Azure/azure-powershell/issues/13337

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SwathiDhanwada-MSFT · Nov 24, 2020 at 02:20 PM

@AjayKumar-8667 Do you need further assistance on this issue ?

0 ·

question