question

Simon-7285 avatar image
0 Votes"
Simon-7285 asked prmanhas-MSFT commented

The api IReplicator.BuildReplica(<ID>) on node <NODE NAME> is stuck

We are experiencing many occurrences of replicas drifting in and out of warning state and the description reads 'The api IReplicator.BuildReplica(<ID>) on node <NODE NAME> is stuck' (see image 1 below). This often leads to nodes getting placed into a Disabling/Disabled state (as seen in image 2 below) and system services such as the FailoverManagerService and the ImageStoreService entering warning state also.

Can anybody assist as to what are the best ways of inspecting what is going on here and understanding why it is dropping into these warning states?

125649-image.png
125702-image.png


Thanks!

azure-service-fabric
image.png (225.4 KiB)
image.png (255.3 KiB)
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@Simon-7285 Apologies for the delay in response and all the inconvenience caused because of the issue.

I have reached out to our internal team on this and will keep you posted once I have an update.

Thanks

1 Vote 1 ·

@Simon-7285 Below is the response I got from the internal team:

Few possibilites we can rule out are:
• Nodes are being disabled due to some Tenant Job or Platform Job or someone trying to change some something in VMSS
• While the node is being disabled, SF will move replicas out of the node and hitting the warning when trying to build a new replica


What is the size of data in the replicas? Do the warnings eventually go away?\

Thanks


0 Votes 0 ·

@Simon-7285 Any update on the issue?

Following up to check if you have any inputs on the ask mentioned in my previous ask that will be helpful in resolving the issue.

Thanks

0 Votes 0 ·

0 Answers