Automatic update of the Mobility service in Azure-to-Azure replication
Azure Site Recovery uses a monthly release cadence to fix any issues and enhance existing features or add new ones. To remain current with the service, you must plan for patch deployment each month. To avoid the overhead associated with each upgrade, you can allow Site Recovery to manage component updates.
As mentioned in Azure-to-Azure disaster recovery architecture, the Mobility service is installed on all Azure virtual machines (VMs) that have replication enabled from one Azure region to another. When you use automatic updates, each new release updates the Mobility service extension.
Note
We recommend that you use the Azure Az PowerShell module to interact with Azure. See Install Azure PowerShell to get started. To learn how to migrate to the Az PowerShell module, see Migrate Azure PowerShell from AzureRM to Az.
How automatic updates work
When you use Site Recovery to manage updates, it deploys a global runbook (used by Azure services) via an automation account, created in the same subscription as the vault. Each vault uses one automation account. For each VM in a vault, the runbook checks for active auto-updates. If a newer version of the Mobility service extension is available, the update is installed.
The default runbook schedule occurs daily at 12:00 AM in the time zone of the replicated VM's geography. You can also change the runbook schedule via the automation account.
Note
Starting with Update Rollup 35, you can choose an existing automation account to use for updates. Prior to Update Rollup 35, Site Recovery created the automation account by default. You can only select this option when you enable replication for a VM. It isn't available for a VM that already has replication enabled. The setting you select applies to all Azure VMs protected in the same vault.
Turning on automatic updates doesn't require a restart of your Azure VMs or affect ongoing replication.
Job billing in the automation account is based on the number of job runtime minutes used in a month. Job execution takes a few seconds to about a minute each day and is covered as free units. By default, 500 minutes are included as free units for an automation account, as shown in the following table:
| Free units included (each month) | Price |
|---|---|
| Job runtime 500 minutes | ₹0.14/minute |
Enable automatic updates
There are several ways that Site Recovery can manage the extension updates:
- Manage as part of the enable replication step
- Toggle the extension update settings inside the vault
- Manage updates manually
Manage as part of the enable replication step
When you enable replication for a VM either starting from the VM view or from the recovery services vault, you can either allow Site Recovery to manage updates for the Site Recovery extension or manage it manually.
Toggle the extension update settings inside the vault
From the Recovery Services vault, go to Manage > Site Recovery Infrastructure.
Under For Azure Virtual Machines > Extension Update Settings > Allow Site Recovery to manage, select On.
To manage the extension manually, select Off.
Important
When you choose Allow Site Recovery to manage, the setting is applied to all VMs in the vault.
Select Save.
Note
Either option notifies you of the automation account used for managing updates. If you're using this feature in a vault for the first time, a new automation account is created by default. Alternately, you can customize the setting, and choose an existing automation account. Once defined, all subsequent actions to enable replication in the same vault will use that selected automation account. Currently, the drop-down menu will only list automation accounts that are in the same Resource Group as the vault.
For a custom automation account, use the following script:
Important
Run the following script in the context of an automation account. This script leverages System Assigned Managed Identities as its authentication type.
param(
[Parameter(Mandatory=$true)]
[String] $VaultResourceId,
[Parameter(Mandatory=$true)]
[ValidateSet("Enabled",'Disabled')]
[Alias("Enabled or Disabled")]
[String] $AutoUpdateAction,
[Parameter(Mandatory=$false)]
[String] $AutomationAccountArmId
)
$SiteRecoveryRunbookName = "Modify-AutoUpdateForVaultForPatner"
$TaskId = [guid]::NewGuid().ToString()
$SubscriptionId = "00000000-0000-0000-0000-000000000000"
$AsrApiVersion = "2021-12-01"
$ArmEndPoint = "https://management.azure.com"
$AadAuthority = "https://login.windows.net/"
$AadAudience = "https://management.core.windows.net/"
$AzureEnvironment = "AzureCloud"
$Timeout = "160"
$AuthenticationType = "SystemAssignedIdentity"
function Throw-TerminatingErrorMessage
{
Param
(
[Parameter(Mandatory=$true)]
[String]
$Message
)
throw ("Message: {0}, TaskId: {1}.") -f $Message, $TaskId
}
function Write-Tracing
{
Param
(
[Parameter(Mandatory=$true)]
[ValidateSet("Informational", "Warning", "ErrorLevel", "Succeeded", IgnoreCase = $true)]
[String]
$Level,
[Parameter(Mandatory=$true)]
[String]
$Message,
[Switch]
$DisplayMessageToUser
)
Write-Output $Message
}
function Write-InformationTracing
{
Param
(
[Parameter(Mandatory=$true)]
[String]
$Message
)
Write-Tracing -Message $Message -Level Informational -DisplayMessageToUser
}
function ValidateInput()
{
try
{
if(!$VaultResourceId.StartsWith("/subscriptions", [System.StringComparison]::OrdinalIgnoreCase))
{
$ErrorMessage = "The vault resource id should start with /subscriptions."
throw $ErrorMessage
}
$Tokens = $VaultResourceId.SubString(1).Split("/")
if(!($Tokens.Count % 2 -eq 0))
{
$ErrorMessage = ("Odd Number of tokens: {0}." -f $Tokens.Count)
throw $ErrorMessage
}
if(!($Tokens.Count/2 -eq 4))
{
$ErrorMessage = ("Invalid number of resource in vault ARM id expected:4, actual:{0}." -f ($Tokens.Count/2))
throw $ErrorMessage
}
if($AutoUpdateAction -ieq "Enabled" -and [string]::IsNullOrEmpty($AutomationAccountArmId))
{
$ErrorMessage = ("The automation account ARM id should not be null or empty when AutoUpdateAction is enabled.")
throw $ErrorMessage
}
}
catch
{
$ErrorMessage = ("ValidateInput failed with [Exception: {0}]." -f $_.Exception)
Write-Tracing -Level ErrorLevel -Message $ErrorMessage -DisplayMessageToUser
Throw-TerminatingErrorMessage -Message $ErrorMessage
}
}
function Initialize-SubscriptionId()
{
try
{
$Tokens = $VaultResourceId.SubString(1).Split("/")
$Count = 0
$ArmResources = @{}
while($Count -lt $Tokens.Count)
{
$ArmResources[$Tokens[$Count]] = $Tokens[$Count+1]
$Count = $Count + 2
}
return $ArmResources["subscriptions"]
}
catch
{
Write-Tracing -Level ErrorLevel -Message ("Initialize-SubscriptionId: failed with [Exception: {0}]." -f $_.Exception) -DisplayMessageToUser
throw
}
}
function Invoke-InternalRestMethod($Uri, $Headers, [ref]$Result)
{
$RetryCount = 0
$MaxRetry = 3
do
{
try
{
$ResultObject = Invoke-RestMethod -Uri $Uri -Headers $Headers
($Result.Value) += ($ResultObject)
break
}
catch
{
Write-InformationTracing ("Retry Count: {0}, Exception: {1}." -f $RetryCount, $_.Exception)
$RetryCount++
if(!($RetryCount -le $MaxRetry))
{
throw
}
Start-Sleep -Milliseconds 2000
}
}while($true)
}
function Invoke-InternalWebRequest($Uri, $Headers, $Method, $Body, $ContentType, [ref]$Result)
{
$RetryCount = 0
$MaxRetry = 3
do
{
try
{
$ResultObject = Invoke-WebRequest -Uri $UpdateUrl -Headers $Header -Method 'PATCH' `
-Body $InputJson -ContentType "application/json" -UseBasicParsing
($Result.Value) += ($ResultObject)
break
}
catch
{
Write-InformationTracing ("Retry Count: {0}, Exception: {1}." -f $RetryCount, $_.Exception)
$RetryCount++
if(!($RetryCount -le $MaxRetry))
{
throw
}
Start-Sleep -Milliseconds 2000
}
}while($true)
}
function Get-Header([ref]$Header, $AadAudience){
try
{
$Header.Value['Content-Type'] = 'application\json'
Write-InformationTracing ("The Authentication Type is system Assigned Identity based.")
$endpoint = $env:IDENTITY_ENDPOINT
$endpoint
$Headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$Headers.Add("X-IDENTITY-HEADER", $env:IDENTITY_HEADER)
$Headers.Add("Metadata", "True")
$authenticationResult = Invoke-RestMethod -Method Get -Headers $Headers -Uri ($endpoint +'?resource=' +$AadAudience)
$accessToken = $authenticationResult.access_token
$Header.Value['Authorization'] = "Bearer " + $accessToken
$Header.Value["x-ms-client-request-id"] = $TaskId + "/" + (New-Guid).ToString() + "-" + (Get-Date).ToString("u")
}
catch
{
$ErrorMessage = ("Get-BearerToken: failed with [Exception: {0}]." -f $_.Exception)
Write-Tracing -Level ErrorLevel -Message $ErrorMessage -DisplayMessageToUser
Throw-TerminatingErrorMessage -Message $ErrorMessage
}
}
function Get-ProtectionContainerToBeModified([ref] $ContainerMappingList)
{
try
{
Write-InformationTracing ("Get protection container mappings : {0}." -f $VaultResourceId)
$ContainerMappingListUrl = $ArmEndPoint + $VaultResourceId + "/replicationProtectionContainerMappings" + "?api-version=" + $AsrApiVersion
Write-InformationTracing ("Getting the bearer token and the header.")
Get-Header ([ref]$Header) $AadAudience
$Result = @()
Invoke-InternalRestMethod -Uri $ContainerMappingListUrl -Headers $header -Result ([ref]$Result)
$ContainerMappings = $Result[0]
Write-InformationTracing ("Total retrieved container mappings: {0}." -f $ContainerMappings.Value.Count)
foreach($Mapping in $ContainerMappings.Value)
{
if(($Mapping.properties.providerSpecificDetails -eq $null) -or ($Mapping.properties.providerSpecificDetails.instanceType -ine "A2A"))
{
Write-InformationTracing ("Mapping properties: {0}." -f ($Mapping.properties))
Write-InformationTracing ("Ignoring container mapping: {0} as the provider does not match." -f ($Mapping.Id))
continue;
}
if($Mapping.Properties.State -ine "Paired")
{
Write-InformationTracing ("Ignoring container mapping: {0} as the state is not paired." -f ($Mapping.Id))
continue;
}
Write-InformationTracing ("Provider specific details {0}." -f ($Mapping.properties.providerSpecificDetails))
$MappingAutoUpdateStatus = $Mapping.properties.providerSpecificDetails.agentAutoUpdateStatus
$MappingAutomationAccountArmId = $Mapping.properties.providerSpecificDetails.automationAccountArmId
$MappingHealthErrorCount = $Mapping.properties.HealthErrorDetails.Count
if($AutoUpdateAction -ieq "Enabled" -and
($MappingAutoUpdateStatus -ieq "Enabled") -and
($MappingAutomationAccountArmId -ieq $AutomationAccountArmId) -and
($MappingHealthErrorCount -eq 0))
{
Write-InformationTracing ("Provider specific details {0}." -f ($Mapping.properties))
Write-InformationTracing ("Ignoring container mapping: {0} as the auto update is already enabled and is healthy." -f ($Mapping.Id))
continue;
}
($ContainerMappingList.Value).Add($Mapping.id)
}
}
catch
{
$ErrorMessage = ("Get-ProtectionContainerToBeModified: failed with [Exception: {0}]." -f $_.Exception)
Write-Tracing -Level ErrorLevel -Message $ErrorMessage -DisplayMessageToUser
Throw-TerminatingErrorMessage -Message $ErrorMessage
}
}
$OperationStartTime = Get-Date
$ContainerMappingList = New-Object System.Collections.Generic.List[System.String]
$JobsInProgressList = @()
$JobsCompletedSuccessList = @()
$JobsCompletedFailedList = @()
$JobsFailedToStart = 0
$JobsTimedOut = 0
$Header = @{}
$AzureRMProfile = Get-Module -ListAvailable -Name AzureRM.Profile | Select Name, Version, Path
$AzureRmProfileModulePath = Split-Path -Parent $AzureRMProfile.Path
Add-Type -Path (Join-Path $AzureRmProfileModulePath "Microsoft.IdentityModel.Clients.ActiveDirectory.dll")
$Inputs = ("Tracing inputs VaultResourceId: {0}, Timeout: {1}, AutoUpdateAction: {2}, AutomationAccountArmId: {3}." -f $VaultResourceId, $Timeout, $AutoUpdateAction, $AutomationAccountArmId)
Write-Tracing -Message $Inputs -Level Informational -DisplayMessageToUser
$CloudConfig = ("Tracing cloud configuration ArmEndPoint: {0}, AadAuthority: {1}, AadAudience: {2}." -f $ArmEndPoint, $AadAuthority, $AadAudience)
Write-Tracing -Message $CloudConfig -Level Informational -DisplayMessageToUser
ValidateInput
$SubscriptionId = Initialize-SubscriptionId
Get-ProtectionContainerToBeModified ([ref]$ContainerMappingList)
$Input = @{
"properties"= @{
"providerSpecificInput"= @{
"instanceType" = "A2A"
"agentAutoUpdateStatus" = $AutoUpdateAction
"automationAccountArmId" = $AutomationAccountArmId
"automationAccountAuthenticationType" = $AuthenticationType
}
}
}
$InputJson = $Input | ConvertTo-Json
if ($ContainerMappingList.Count -eq 0)
{
Write-Tracing -Level Succeeded -Message ("Exiting as there are no container mappings to be modified.") -DisplayMessageToUser
exit
}
Write-InformationTracing ("Container mappings to be updated has been retrieved with count: {0}." -f $ContainerMappingList.Count)
try
{
Write-InformationTracing ("Start the modify container mapping jobs.")
ForEach($Mapping in $ContainerMappingList)
{
try {
$UpdateUrl = $ArmEndPoint + $Mapping + "?api-version=" + $AsrApiVersion
Get-Header ([ref]$Header) $AadAudience
$Result = @()
Invoke-InternalWebRequest -Uri $UpdateUrl -Headers $Header -Method 'PATCH' `
-Body $InputJson -ContentType "application/json" -Result ([ref]$Result)
$Result = $Result[0]
$JobAsyncUrl = $Result.Headers['Azure-AsyncOperation']
Write-InformationTracing ("The modify container mapping job invoked with async url: {0}." -f $JobAsyncUrl)
$JobsInProgressList += $JobAsyncUrl;
# Rate controlling the set calls to maximum 60 calls per minute.
# ASR throttling for set calls is 200 in 1 minute.
Start-Sleep -Milliseconds 1000
}
catch{
Write-InformationTracing ("The modify container mappings job creation failed for: {0}." -f $Ru)
Write-InformationTracing $_
$JobsFailedToStart++
}
}
Write-InformationTracing ("Total modify container mappings has been initiated: {0}." -f $JobsInProgressList.Count)
}
catch
{
$ErrorMessage = ("Modify container mapping jobs failed with [Exception: {0}]." -f $_.Exception)
Write-Tracing -Level ErrorLevel -Message $ErrorMessage -DisplayMessageToUser
Throw-TerminatingErrorMessage -Message $ErrorMessage
}
try
{
while($JobsInProgressList.Count -ne 0)
{
Sleep -Seconds 30
$JobsInProgressListInternal = @()
ForEach($JobAsyncUrl in $JobsInProgressList)
{
try
{
Get-Header ([ref]$Header) $AadAudience
$Result = Invoke-RestMethod -Uri $JobAsyncUrl -Headers $header
$JobState = $Result.Status
if($JobState -ieq "InProgress")
{
$JobsInProgressListInternal += $JobAsyncUrl
}
elseif($JobState -ieq "Succeeded" -or `
$JobState -ieq "PartiallySucceeded" -or `
$JobState -ieq "CompletedWithInformation")
{
Write-InformationTracing ("Jobs succeeded with state: {0}." -f $JobState)
$JobsCompletedSuccessList += $JobAsyncUrl
}
else
{
Write-InformationTracing ("Jobs failed with state: {0}." -f $JobState)
$JobsCompletedFailedList += $JobAsyncUrl
}
}
catch
{
Write-InformationTracing ("The get job failed with: {0}. Ignoring the exception and retrying the next job." -f $_.Exception)
# The job on which the tracking failed, will be considered in progress and tried again later.
$JobsInProgressListInternal += $JobAsyncUrl
}
# Rate controlling the get calls to maximum 120 calls each minute.
# ASR throttling for get calls is 10000 in 60 minutes.
Start-Sleep -Milliseconds 500
}
Write-InformationTracing ("Jobs remaining {0}." -f $JobsInProgressListInternal.Count)
$CurrentTime = Get-Date
if($CurrentTime -gt $OperationStartTime.AddMinutes($Timeout))
{
Write-InformationTracing ("Tracing modify cloud pairing jobs has timed out.")
$JobsTimedOut = $JobsInProgressListInternal.Count
$JobsInProgressListInternal = @()
}
$JobsInProgressList = $JobsInProgressListInternal
}
}
catch
{
$ErrorMessage = ("Tracking modify cloud pairing jobs failed with [Exception: {0}]." -f $_.Exception)
Write-Tracing -Level ErrorLevel -Message $ErrorMessage -DisplayMessageToUser
Throw-TerminatingErrorMessage -Message $ErrorMessage
}
Write-InformationTracing ("Tracking modify cloud pairing jobs completed.")
Write-InformationTracing ("Modify cloud pairing jobs success: {0}." -f $JobsCompletedSuccessList.Count)
Write-InformationTracing ("Modify cloud pairing jobs failed: {0}." -f $JobsCompletedFailedList.Count)
Write-InformationTracing ("Modify cloud pairing jobs failed to start: {0}." -f $JobsFailedToStart)
Write-InformationTracing ("Modify cloud pairing jobs timedout: {0}." -f $JobsTimedOut)
if($JobsTimedOut -gt 0)
{
$ErrorMessage = "One or more modify cloud pairing jobs has timedout."
Write-Tracing -Level ErrorLevel -Message ($ErrorMessage)
Throw-TerminatingErrorMessage -Message $ErrorMessage
}
elseif($JobsCompletedSuccessList.Count -ne $ContainerMappingList.Count)
{
$ErrorMessage = "One or more modify cloud pairing jobs failed."
Write-Tracing -Level ErrorLevel -Message ($ErrorMessage)
Throw-TerminatingErrorMessage -Message $ErrorMessage
}
Write-Tracing -Level Succeeded -Message ("Modify cloud pairing completed.") -DisplayMessageToUser
Manage updates manually
If there are new updates for the Mobility service installed on your VMs, you'll see the following notification: New Site Recovery replication agent update is available. Click to install.
Select the notification to open the VM selection page.
Choose the VMs you want to upgrade, and then select OK. The Update Mobility service will start for each selected VM.
Common issues and troubleshooting
If there's an issue with the automatic updates, you'll see an error notification under Configuration issues in the vault dashboard.
If you can't enable automatic updates, see the following common errors and recommended actions:
Error: You do not have permissions to create an Azure Run As account (service principal) and grant the Contributor role to the service principal.
Recommended action: Make sure that the signed-in account is assigned as Contributor and try again. For more information about assigning permissions, see the required permissions section of How to: Use the portal to create a Microsoft Entra application and service principal that can access resources.
To fix most issues after you enable automatic updates, select Repair. If the repair button isn't available, see the error message displayed in the extension update settings pane.
Error: The Run As account does not have the permission to access the recovery services resource.
Recommended action: Delete and then re-create the Run As account. Or, make sure that the Automation Run As account's Microsoft Entra application can access the recovery services resource.
Error: Run As account is not found. Either one of these was deleted or not created - Microsoft Entra Application, Service Principal, Role, Automation Certificate asset, Automation Connection asset - or the Thumbprint is not identical between Certificate and Connection.
Recommended action: Delete and then re-create the Run As account.
Error: The Azure Run as Certificate used by the automation account is about to expire.
The self-signed certificate that is created for the Run As account expires one year from the date of creation. You can renew it at any time before it expires. If you have signed up for email notifications, you will also receive emails when an action is required from your side. This error will be shown two months prior to the expiry date, and will change to a critical error if the certificate has expired. Once the certificate has expired, auto update will not be functional until you renew the same.
Recommended action: To resolve this issue, select Repair and then Renew Certificate.
Note
After you renew the certificate, refresh the page to display the current status.
Next steps
Learn more on how to migrate the authentication type of the Automation accounts to Managed Identities.
Feedback
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback.
Submit and view feedback for