FABRIC_E_IMAGEBUILDER_UNEXPECTED_ERROR while Registering Application Type
We are receiving the following error with increasing frequency while deploying using Azure Devops deployment pipeline, to our 3 node development cluster:
[error]Exception calling "Invoke" with "1" argument(s): "Failed to register application type. Status: Failed. Details: FABRIC_E_IMAGEBUILDER_UNEXPECTED_ERROR."
We also recently experienced a replica going down on ImageStoreService. This was noticed after investigating one of the above deploy failures. I restarted the node in that case and was able to successfully deploy. In another instance, the above failure occurred without a correlated problem in ImageStoreService. In that case, redeploy succeeded.
Also possibly related, I see the following ApplicationProcessExited events for each service associated with recent successful deploys, seemingly failing to deactivate gracefully during the upgrade:
ActivityId: bb4e514d-2a9f-4b24-b126-fb96495527d0:0
CodePackageName: Code
EntryPointType: Exe
ExeName: [service name].exe
ExitCode: 7148
ExitReason: The process/container terminated with exit code:7148. Aborting since deactivation failed. Deactivating as part of upgrade since code package changed. For information about common termination errors, please visit https://aka.ms/service-fabric-termination-errors
HostId: ec178b3a-24a2-472a-b47b-306e8fb964a4
IsExclusive: false
ProcessId: 8428
ServiceName:
ServicePackageActivationId:
ServicePackageName: [service name]
StartTime: 2021-06-13T17:36:06Z
UnexpectedTermination: false
The increasing frequency of this error causing failed upgrade deployments concerns me, and I want to see how I can get more intel into what the root cause might be so I can resolve it. Thank you in advance!