Service Fabric ReliableConcurrentQueue warning
Recently we've been seeing a strange error associated with a replica of one of our services. We want to track down what's causing it, but we don't understand the error.
'System.RA' reported Warning for property 'ReplicaOpenStatus'.
Replica had multiple failures during open on <one of our servers>. API call: IStatefulServiceReplica.Open(); Error = System.IO.InvalidDataException (-2146233087)
versionRead '0' != FileVersion '1'
at Microsoft.ServiceFabric.Data.Collections.ReliableConcurrentQueue.CheckpointManager1.<TryReadCheckpointFile>d__26.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ServiceFabric.Data.Collections.ReliableConcurrentQueue.ReliableConcurrentQueue
1.<Microsoft-ServiceFabric-Replicator-IStateProvider2-RecoverCheckpointAsync>d__86.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.DynamicStateManager.<RecoverStateProvidersAsync>d__187.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.DynamicStateManager.<OpenAsync>d__118.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.TransactionalReplicator.<OpenAsync>d__82.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Replicator.StatefulServiceReplica.<System-Fabric-IStatefulServiceReplica-OpenAsync>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Data.ReliableStateManagerImpl.<Microsoft-ServiceFabric-Data-IStateProviderReplica-OpenAsync>d__46.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Services.Runtime.StatefulServiceReplicaAdapter.<System-Fabric-IStatefulServiceReplica-OpenAsync>d__17.MoveNext()
For more information see: <a href='https://aka.ms/sfhealth' target='_blank'>https://aka.ms/sfhealth</a>
Any help would be appreciated.