This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
Hello,
I am trying to perform a rolling upgrade on a virtual machine scale set but it is failing with the following error,
The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'. (Code: ResourceDeploymentFailure, Target: /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Compute/virtualMachineScaleSets/<vmss-name>)
The scale set deployment status is conflict with error
An internal execution error occurred. Please retry later. (Code: InternalExecutionError)
The logs doesn't give much information about the cause of the failure. I would appreciate any help here.
This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 83%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
We have the same issue across multiple VMSS in different Azure subscriptions/tenants. The issue started happening in the last two weeks. A support ticket has been opened for the past few days but we haven't been getting much help.
Before this issue, our deployment pipeline has been working since May 2023.
Got a response from support. It seems there was a regression introduced for ephemeral OS disks. MSFT is working on a hotfix.
@Andrew Collard Are you also using ephemeral OS disks?
We are! Is this be tracked anywhere so we can get status updates?
@Andrew Collard I don't think so but, I will post any new updates here.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 40%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJD%3C/text%3E%3C/svg%3E)
@Rushikesh Gaidhani Did Microsoft mention a timeline for the hotfix?
@ Jamie Davis They do not have an ETA, but their latest response is that they are working on a hotfix that will fix the issue across all the regions.
@Jamie Davis They do not have an ETA, but their latest response is that they are working on a hotfix that will fix the issue across all the regions.
Update:
MSFT has identified the issue and have a fix ready but, they will push the fix in the first week of January 2024 (because of the holiday season).
The issue impacts platform triggered AutoOSUpgrade operations only to the VMSS using Ephemeral OS disk.
FYI Andrew Collard Jamie Davis
Hello Rushikesh Gaidhani
Thanks for sharing the resolution details for the benefit of community.
@vipullag-MSFT Do you have a timeline of when this week we should expect this hot fix to go live?
We are now more than half way past the end of the week, and we still cannot update our production builds due to this issue.
@Jamie Davis as a workaround, you can use managed disk instead of ephemeral OS disks but, the catch here is that you'll have to delete your existing deployment/VMSS.
I am yet to get confirmation about the hot fix. I'll keep this thread updated.
Hello Rushikesh Gaidhani
Thanks for sharing this.
@Rushikesh Gaidhani Sorry for asking once again, have you heard anything from Microsoft on an update here? The first week of January is well past now.
Jamie Davis No update. The issue still persist with the ephemeral disk.
Update: The MSFT team is going to roll out the fix periodically across the regions, so some regions may receive the fix sooner. They anticipate completing this activity by the end of January.
FYI Jamie Davis
@Rushikesh Gaidhani Any chance that you have heard anything from Microsoft on the rollout status? We are nearing the end of the month and there's no change on our end. Any luck on your end yet?
Hi Jamie Davis As of right present, the hotfix is scheduled to be finished by the end of this month and tested on January 31 or February 1. I did a quick test deployment on WestUS but had no luck.
Good news @Rushikesh Gaidhani, the hot fix appears to be live in EastUS, hopefully you receive the fix in WestUS soon as well!
Indeed, the hotfix appears to be functioning. Over the coming days, I'll be keeping an eye on the rolling upgrade using the ephemeral OS disk.
The InternalExecutionError is often caused by a temporary problem within Azure itself. Microsoft suggests retrying the deployment as a primary step
https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/common-deployment-errors
Thanks for the reply @Sedat SALMAN
But even after retrying the deployment I still get the same error message also, I did not find the error code "InternalExecutionError" in the documentation.
Hello Rushikesh Gaidhani
Firstly, apologies for the delay in responding here and any inconvenience this issue may have caused.
This issue needs deeper investigation. Support team will be able to check and help on this. I would recommend you to open a azure support case.
If you don't have a support plan enabled on your subscription, I request you to send an email to AzCommunity@Microsoft.com with Subject as "Attn:Vikas" referencing this thread along with your subscription ID. I can then enable your subscription with a one-time free technical support.
Once the issue is resolved, request you to post the resolution here for the benefit of community.
Try deploying a smaller subset of changes first. This can help pinpoint the troublemaker if it's specific to a configuration update.
Not observing any more failures. The fix is rolled out across all the regions.