Service Fabric Actors unable to create replicas

Question

question

NishantSharma-7365 asked · May 13 at 03:34 AM

Service Fabric Actors unable to create replicas

I am trying to deploy 2 stateless services, 1 stateful service and two actor services on a 3 node service fabric cluster. Other services are getting deployed without errors but actor service seems to have gotten into trouble.

.... "Description": "Partition is below target replica or instance count.".....

It seems that primaries are getting created fine, but it is having trouble creating/configuring replicas. The exception I see in SF explorer is

'System.RA' reported Warning for property 'ReplicaOpenStatus'.
Replica had multiple failures during open on jsiugm57a_2. API call: IStatefulServiceReplica.Open(); Error = System.InvalidCastException (-2147467262)
Unable to cast COM object of type 'System.ComObject' to interface type 'IFabricKeyValueStoreReplica8'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{FB87EC27-FFF9-4C03-820E-41A7EF38E732}' failed due to the following error: No such interface supported (0x80004002 (E_NOINTERFACE)).
at System.StubHelpers.InterfaceMarshaler.ConvertToManaged(IntPtr pUnk, IntPtr itfMT, IntPtr classMT, Int32 flags)
at System.Fabric.Interop.NativeRuntime.FabricCreateKeyValueStoreReplica5(Guid& riid, IntPtr storeName, Guid partitionId, Int64 replicaId, IntPtr serviceName, IntPtr fabricReplicatorSettings, IntPtr kvsSettings, FABRIC_LOCAL_STORE_KIND localStoreKind, IntPtr localStorageSettings, IFabricStoreEventHandler storeEventHandler, IFabricSecondaryEventHandler secondaryEventHandler)
at System.Fabric.KeyValueStoreReplica.CreateNativeKeyValueStoreReplicaHelper(String storeName, Guid partitionId, Int64 replicaId, Uri serviceName, ReplicatorSettings replicatorSettings, LocalStoreSettings localStoreSettings, KeyValueStoreReplicaSettings kvsSettings, IFabricStoreEventHandler storeEventHandler, IFabricSecondaryEventHandler secondaryEventHandler)
at System.Fabric.KeyValueStoreReplica.<>cDisplayClass106_0.<CreateNativeKeyValueStoreReplica>b_0()
at System.Fabric.Interop.Utility.WrapNativeSyncInvoke[TResult](Func`1 func, String functionTag, String functionArgs)
at System.Fabric.KeyValueStoreReplica.CreateNativeKeyValueStoreReplica(StatefulServiceInitializationParameters initParams, IFabricStoreEventHandler storeEventHandler, IFabricSecondaryEventHandler secondaryEventHandler)
at System.Fabric.KeyValueStoreReplica.Store_Initialize(StatefulServiceInitializationParameters initializationParameters)
at System.Fabric.KeyValueStoreReplica.Initialize(StatefulServiceInitializationParameters initializationParameters)
at Microsoft.ServiceFabric.Actors.Runtime.KvsActorStateProvider.OnCreateAndInitializeReplica(StatefulServiceInitializationParameters initParams, Action`1 copyHandler, Action`1 replicationHandler, Func`2 onDataLossHandler, Func`2 restoreCompletedHandler)
at Microsoft.ServiceFabric.Actors.Runtime.KvsActorStateProviderBase.Microsoft.ServiceFabric.Data.IStateProviderReplica.Initialize(StatefulServiceInitializationParameters initializationParameters)
at System.Fabric.ServiceFactoryBroker.CreateHelper[TFactory,TReturnValue](IntPtr nativeServiceType, IntPtr nativeServiceName, UInt32 initializationDataLength, IntPtr nativeInitializationData, Guid partitionId, Func`3 creationFunc, Action`2 initializationFunc, ServiceInitializationParameters initializationParameters)
For more information see: <a href='https://aka.ms/sfhealth' target='_blank'>https://aka.ms/sfhealth</a>

Following is the configuration I have for partitions and replicas.
<Parameter Name="UserActorsService_PartitionCount" Value="10" />
<Parameter Name="UserActorsService_MinReplicaSetSize" Value="2" />
<Parameter Name="UserActorsService_TargetReplicaSetSize" Value="3" />

 &lt;Parameter Name=&#34;QueueActorsService_PartitionCount&#34; Value=&#34;10&#34; /&gt;
 &lt;Parameter Name=&#34;QueueActorsService_MinReplicaSetSize&#34; Value=&#34;2&#34; /&gt;
 &lt;Parameter Name=&#34;QueueActorsService_TargetReplicaSetSize&#34; Value=&#34;3&#34; /&gt;

Please suggest.
Thanks

azure-service-fabric

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2020-05-15T10:00:45.02Z

AntnioAlbuquerque-5887 answered · May 15 at 10:00 AM

Hello, I'm facing a similar problem, did you do any code changes? Was this working before?
What nuget version of service fabric are you using?

Thanks

Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 2 · 2020-05-15T15:57:48.677Z

RasmusSvendsen-4894 answered · May 15 at 03:57 PM

We also hit the same issue today, when deployed our applications that contains actor services. Latest SF.runtime 7.1 and nuget version: 4.1.409.

Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 3 · 2020-05-15T16:03:19.237Z

NishantSharma-7365 answered · May 15 at 04:03 PM

It seems that you should never use the latest nuget version of service fabric.

https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-versions

As per link above, the latest nuget version never supports the latest cluster SDK.

So, downgrade your SDK to to 4.0.*

1 Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RasmusSvendsen-4894 · May 15 at 04:43 PM

The release note from 7.1 indicate that you should use the *.409 build. https://github.com/microsoft/service-fabric/blob/master/release_notes/Service-Fabric-71-releasenotes.md
On one of our SF clusters works on the other it does not work.

0 · ·

Answer 4 · 2020-05-15T17:53:30.29Z

RasmusSvendsen-4894 answered · May 15 at 05:53 PM

Workaround: Downgrade all SF related nuget packages to version: 4.0.470, still on a 7.1 cluster - worked for me.

Share

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

question