jobs are not able to run on Hybrid worker

Question

question

AvaTomlinson-7607 asked Apr 18, '22 AnuragSingh-MSFT edited Apr 25, '22

jobs are not able to run on Hybrid worker

We have are using Hybrid Worker(Windows Server 2016) in an Azure Automation Account to run the PowerShell runbooks. We have noticed that the jobs in the Automation account are getting queued and suspended we have thousands of suspended jobs within last 24hours. The jobs are not able to run on Hybrid worker. We have checked the Microsoft SMA logs and found below error.

Event Viewer : Application and Services log- Operations Manager
1. A module of type "Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource" reported an exception System.Net.WebException: The remote server returned an error: (504) Gateway Timeout.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource.TryGetIMDSData(Object state) which was running as part of rule "CollectHealthAssessmentInformation" running for instance "" with id:"{XXXXXXXXXXXX}" in management group "XXXXXXXXXXXX".

A secure connection could not be negotiated with the service XXXXXXXXXXXX.ods.opinsights.azure.com. The article KB3126513 has additional troubleshooting information for connectivity issues. Possible reasons for this include:
The certificate authority present in server certificate is not in the "Third-Party Root Certification Authorities" store. Please add this authority to that store.
TLS 1.0, 1.1, and 1.2 are all disabled.
A suitable cypher suite could not be negotiated.
Other details:
Failure Code: 12175L

Any help is appreciated in advance. Thanks

azure-automation

mma.png (69.1 KiB)

· 6

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AnuragSingh-MSFT · Apr 19 at 10:20 AM

@AvaTomlinson-7607, @AvaTomlinson-7607, the screenshot above shows that LogAnalytics agent can connect to the workspace. Also, the error mentioned is related to one of the "workflow" to collect health of machine - not related to the HybridWorker's runbook execution. The following should help:

a. The Application logs under Windows Logs. This would contain logs related to installation failures/crashes happening for related processes.

b. The Operations Manager logs under the Applications and Services Logs would contain errors related to connection to Log Analytics workspace.

c. The Operational logs available under Applications and Services Logs --> Microsoft-SMA in Event Viewer on the Hybrid worker machine. This would contain logs related to the functioning of runbook workers as well as execution of runbook jobs.

d. Are the jobs failing for a particular runbook or set of runbooks?

e. Are all the "Hybrid Workers" online and running? Try to run a sample runbook using option -> select the Hybrid Worker group, does it fail as well?

f. The troubleshooting guides:
Extension-based
Agent-based

These questions should help narrow down the issue further. Please let me know if you have any questions.

1 ·

image.png (748 B)

AvaTomlinson-7607 AnuragSingh-MSFT · Apr 20 at 07:18 AM

a. No Error and critical log found in Application logs under Windows Logs.

b. Repetitive Error log in Operations Manager logs-

A module of type "Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource" reported an exception System.Net.WebException: The remote server returned an error: (504) Gateway Timeout.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.EnterpriseManagement.HealthService.HealthAssessmentData.HealthAssessmentDataSource.TryGetIMDSData(Object state) which was running as part of rule "CollectHealthAssessmentInformation" running for instance "" with id:"{8E33DF4F-C46F-F0B2-820E-EDDEC57520BF}" in management group "AOI-f7203428-76d2-4905-bf39-d7c21298d5e8".

c. No Error, critical and warning log found in Microsoft-SMA

d. Jobs are getting Suspended for all the runbooks

e. There is only one Agent based hybrid worker and it is online. The jobs are getting queued for about 15minutes and getting Suspended.
(We cannot run any job manually as it is Production)

f. We need some time to go through the troubleshooting articles shared by you.

Under services, Microsoft Monitoring agent is running. We have observed that “Microsoft Monitoring Agent APM”, “Microsoft Agent Auditing Forwarding” are in disable state where we are not sure whether these are relevant to this issue or not.

0 ·

mma-services.png (48.0 KiB)

AnuragSingh-MSFT AvaTomlinson-7607 · Apr 21 at 10:56 AM

@AvaTomlinson-7607, thank you for the reply. The "Microsoft Monitoring Agent APM" is for application performance monitoring (.NET Framework based). This is not related to SMA/Azure Automation Hybrid runbook worker.

I understand that you are going through the troubleshooting guide as of now. Please ensure that you are following the guide depending on the type of Hybrid Worker (extension based OR agent based).

Please let me know with your findings and any questions that you may have.

0 ·

Show more comments

YiLu-MSFT · Apr 19 at 02:04 AM

Hi @AvaTomlinson-7607
This is a quick note to let you know that we are performing research on this issue.

0 ·

CaseyYangMSFT-4714 · Apr 20 at 07:02 AM

Hi @AvaTomlinson-7607,

I removed the tag "office-sharepoint-server-administration" because this issue has little to do with it, Thank you for your understanding.

0 ·

question