Automation Hybrid Runbook Worker overview

Runbooks in Azure Automation might not have access to resources in other clouds or in your on-premises environment because they run on the Azure cloud platform. You can use the Hybrid Runbook Worker feature of Azure Automation to run runbooks directly on the machine hosting the role and against resources in the environment to manage those local resources. Runbooks are stored and managed in Azure Automation and then delivered to one or more assigned machines.

Azure Automation provides native integration of the Hybrid Runbook Worker role through the Azure virtual machine (VM) extension framework. The Azure VM agent is responsible for management of the extension on Azure VMs on Windows and Linux VMs, and on non-Azure machines through the Arc-enabled servers Connected Machine agent. Now there are two Hybrid Runbook Workers installation platforms supported by Azure Automation.

Platform Description
Agent-based (V1) Installed after the Log Analytics agent reporting to an Azure Monitor Log Analytics workspace is completed.
Extension-based (V2) Installed using the Hybrid Runbook Worker VM extension, without any dependency on the Log Analytics agent reporting to an Azure Monitor Log Analytics workspace. This is the recommended platform.

Hybrid worker group showing platform field

Here's a list of benefits available with the extension-based Hybrid Runbook Worker role:

Benefit Description
Seamless onboarding Removes dependency on a Log Analytics solution for onboarding Hybrid Runbook Workers, which is a multi-step process, is time consuming, and error-prone.
Unified onboarding experience Installation is managed using the same supported methods for Azure and non-Azure machines.
Ease of Manageability Native integration with ARM identity for Hybrid Runbook Worker and provides the flexibility for governance at scale through policies and templates.
Azure AD-based authentication Uses VM system assigned-identities provided by Azure AD. This centralizes control and management of identities and resource credentials.

For Hybrid Runbook Worker operations after installation, the process of executing runbooks on Hybrid Runbook Workers is the same. The purpose of the extension-based approach is to simplify the installation and management of the Hybrid Runbook Worker role and remove the complexity working with the agent-based version. The new extension-based installation doesn't affect the installation or management of an agent-based Hybrid Runbook Worker role. Both Hybrid Runbook Worker types can co-exist on the same machine.

The extension-based Hybrid Runbook Worker only supports the user Hybrid Runbook Worker type, and doesn't include the system Hybrid Runbook Worker required for the Update Management feature.

Note

PowerShell support to install the extension-based Hybrid Runbook Worker is not supported at this time.

Runbook Worker types

There are two types of Runbook Workers - system and user. The following table describes the difference between them.

Type Description
System Supports a set of hidden runbooks used by the Update Management feature that are designed to install user-specified updates on Windows and Linux machines.
This type of Hybrid Runbook Worker isn't a member of a Hybrid Runbook Worker group, and therefore doesn't run runbooks that target a Runbook Worker group.
User Supports user-defined runbooks intended to run directly on the Windows and Linux machine that are members of one or more Runbook Worker groups.

Agent-based (V1) Hybrid Runbook Workers rely on the Log Analytics agent reporting to an Azure Monitor Log Analytics workspace. The workspace isn't only to collect monitoring data from the machine, but also to download the components required to install the agent-based Hybrid Runbook Worker.

When Azure Automation Update Management is enabled, any machine connected to your Log Analytics workspace is automatically configured as a system Hybrid Runbook Worker. To configure it as a user Windows Hybrid Runbook Worker, see Deploy an agent-based Windows Hybrid Runbook Worker in Automation and for Linux, see Deploy an agent-based Linux Hybrid Runbook Worker in Automation.

Runbook Worker limits

The following table shows the maximum number of system and user Hybrid Runbook Workers in an Automation account. If you have more than 4,000 machines to manage, we recommend creating another Automation account.

Worker type Maximum number supported per Automation Account.
System 4000
User 4000

How does it work?

Each user Hybrid Runbook Worker is a member of a Hybrid Runbook Worker group that you specify when you install the worker. A group can include a single worker, but you can include multiple workers in a group for high availability. Each machine can host one Hybrid Runbook Worker reporting to one Automation account; you can't register the hybrid worker across multiple Automation accounts. A hybrid worker can only listen for jobs from a single Automation account.

User Hybrid Runbook Worker technical diagram

For machines hosting the system Hybrid Runbook worker managed by Update Management, they can be added to a Hybrid Runbook Worker group. But you must use the same Automation account for both Update Management and the Hybrid Runbook Worker group membership.

System Hybrid Runbook Worker technical diagram

When you start a runbook on a user Hybrid Runbook Worker, you specify the group that it runs on. Each worker in the group polls Azure Automation to see if any jobs are available. If a job is available, the first worker to get the job takes it. The processing time of the jobs queue depends on the hybrid worker hardware profile and load. You can't specify a particular worker. Hybrid worker works on a polling mechanism (every 30 secs) and follows an order of first-come, first-serve. Depending on when a job was pushed, whichever hybrid worker pings the Automation service picks up the job. A single hybrid worker can generally pick up four jobs per ping (that is, every 30 seconds). If your rate of pushing jobs is higher than four per 30 seconds, then there's a high possibility another hybrid worker in the Hybrid Runbook Worker group picked up the job.

A Hybrid Runbook Worker doesn't have many of the Azure sandbox resource limits on disk space, memory, or network sockets. The limits on a hybrid worker are only related to the worker's own resources, and they aren't constrained by the fair share time limit that Azure sandboxes have.

To control the distribution of runbooks on Hybrid Runbook Workers and when or how the jobs are triggered, you can register the hybrid worker against different Hybrid Runbook Worker groups within your Automation account. Target the jobs against the specific group or groups in order to support your execution arrangement.

Hybrid Runbook Worker installation

The process to install a user Hybrid Runbook Worker depends on the operating system. The table below defines the deployment types.

Operating System Deployment Types
Windows Automated
Manual.
Linux Manual
Either For user Hybrid Runbook Workers, see Deploy an extension-based Windows or Linux user Hybrid Runbook Worker in Automation. This is the recommended method.

Network planning

Check Azure Automation Network Configuration for detailed information on the ports, URLs, and other networking details required for the Hybrid Runbook Worker.

Proxy server use

If you use a proxy server for communication between Azure Automation and machines running the Log Analytics agent, ensure that the appropriate resources are accessible. The timeout for requests from the Hybrid Runbook Worker and Automation services is 30 seconds. After three attempts, a request fails.

Firewall use

If you use a firewall to restrict access to the Internet, you must configure the firewall to permit access. If using the Log Analytics gateway as a proxy, ensure that it's configured for Hybrid Runbook Workers. See Configure the Log Analytics gateway for Automation Hybrid Runbook Workers.

Service tags

Azure Automation supports Azure virtual network service tags, starting with the service tag GuestAndHybridManagement. You can use service tags to define network access controls on network security groups or Azure Firewall. Service tags can be used in place of specific IP addresses when you create security rules. By specifying the service tag name GuestAndHybridManagement in the appropriate source or destination field of a rule, you can allow or deny the traffic for the Automation service. This service tag doesn't support allowing more granular control by restricting IP ranges to a specific region.

The service tag for the Azure Automation service only provides IPs used for the following scenarios:

  • Trigger webhooks from within your virtual network
  • Allow Hybrid Runbook Workers or State Configuration agents on your VNet to communicate with the Automation service

Note

The service tag GuestAndHybridManagement currently doesn't support runbook job execution in an Azure sandbox, only directly on a Hybrid Runbook Worker.

Support for Impact Level 5 (IL5)

Azure Automation Hybrid Runbook Worker can be used in Azure Government to support Impact Level 5 workloads in either of the following two configurations:

  • Isolated virtual machine. When deployed, they consume the entire physical host for that machine providing the necessary level of isolation required to support IL5 workloads.

  • Azure Dedicated Hosts, which provides physical servers that are able to host one or more virtual machines, dedicated to one Azure subscription.

Note

Compute isolation through the Hybrid Runbook Worker role is available for Azure Commercial and US Government clouds.

Update Management addresses for Hybrid Runbook Worker

In addition to the standard addresses and ports required for the Hybrid Runbook Worker, Update Management has other network configuration requirements described under the network planning section.

Azure Automation State Configuration on a Hybrid Runbook Worker

You can run Azure Automation State Configuration on a Hybrid Runbook Worker. To manage the configuration of servers that support the Hybrid Runbook Worker, you must add the servers as DSC nodes. See Enable machines for management by Azure Automation State Configuration.

Runbooks on a Hybrid Runbook Worker

You might have runbooks that manage resources on the local machine or run against resources in the local environment where a user Hybrid Runbook Worker is deployed. In this case, you can choose to run your runbooks on the hybrid worker instead of in an Automation account. Runbooks run on a Hybrid Runbook Worker are identical in structure to those that you run in the Automation account. See Run runbooks on a Hybrid Runbook Worker.

Hybrid Runbook Worker jobs

Hybrid Runbook Worker jobs run under the local System account on Windows or the nxautomation account on Linux. Azure Automation handles jobs on Hybrid Runbook Workers differently from jobs run in Azure sandboxes. See Runbook execution environment.

If the Hybrid Runbook Worker host machine reboots, any running runbook job restarts from the beginning, or from the last checkpoint for PowerShell Workflow runbooks. After a runbook job is restarted more than three times, it's suspended.

Runbook permissions for a Hybrid Runbook Worker

Since they access non-Azure resources, runbooks running on a user Hybrid Runbook Worker can't use the authentication mechanism typically used by runbooks authenticating to Azure resources. A runbook either provides its own authentication to local resources, or configures authentication using managed identities for Azure resources. You can also specify a Run As account to provide a user context for all runbooks.

View system Hybrid Runbook Workers

After the Update Management feature is enabled on Windows or Linux machines, you can inventory the list of system Hybrid Runbook Workers group in the Azure portal. You can view up to 2,000 workers in the portal by selecting the tab System hybrid workers group from the option Hybrid workers group from the left-hand pane for the selected Automation account.

Automation account system hybrid worker groups page

If you have more than 2,000 hybrid workers, to get a list of all of them, you can run the following PowerShell script:

"Get-AzSubscription -SubscriptionName "<subscriptionName>" | Set-AzContext
$workersList = (Get-AzAutomationHybridWorkerGroup -ResourceGroupName "<resourceGroupName>" -AutomationAccountName "<automationAccountName>").Runbookworker
$workersList | export-csv -Path "<Path>\output.csv" -NoClobber -NoTypeInformation"

Next steps