Service Fabric Actors unable to create replicas

Nishant Sharma 1 Reputation point
2020-05-13T03:34:06.4+00:00

I am trying to deploy 2 stateless services, 1 stateful service and two actor services on a 3 node service fabric cluster. Other services are getting deployed without errors but actor service seems to have gotten into trouble.

.... "Description": "Partition is below target replica or instance count.".....

It seems that primaries are getting created fine, but it is having trouble creating/configuring replicas. The exception I see in SF explorer is

'System.RA' reported Warning for property 'ReplicaOpenStatus'.
Replica had multiple failures during open on _jsiugm57a_2. API call: IStatefulServiceReplica.Open(); Error = System.InvalidCastException (-2147467262)
Unable to cast COM object of type 'System.__ComObject' to interface type 'IFabricKeyValueStoreReplica8'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{FB87EC27-FFF9-4C03-820E-41A7EF38E732}' failed due to the following error: No such interface supported (0x80004002 (E_NOINTERFACE)).
at System.StubHelpers.InterfaceMarshaler.ConvertToManaged(IntPtr pUnk, IntPtr itfMT, IntPtr classMT, Int32 flags)
at System.Fabric.Interop.NativeRuntime.FabricCreateKeyValueStoreReplica5(Guid& riid, IntPtr storeName, Guid partitionId, Int64 replicaId, IntPtr serviceName, IntPtr fabricReplicatorSettings, IntPtr kvsSettings, FABRIC_LOCAL_STORE_KIND localStoreKind, IntPtr localStorageSettings, IFabricStoreEventHandler storeEventHandler, IFabricSecondaryEventHandler secondaryEventHandler)
at System.Fabric.KeyValueStoreReplica.CreateNativeKeyValueStoreReplicaHelper(String storeName, Guid partitionId, Int64 replicaId, Uri serviceName, ReplicatorSettings replicatorSettings, LocalStoreSettings localStoreSettings, KeyValueStoreReplicaSettings kvsSettings, IFabricStoreEventHandler storeEventHandler, IFabricSecondaryEventHandler secondaryEventHandler)
at System.Fabric.KeyValueStoreReplica.<>c__DisplayClass106_0.<CreateNativeKeyValueStoreReplica>b__0()
at System.Fabric.Interop.Utility.WrapNativeSyncInvoke[TResult](Func1 func, String functionTag, String functionArgs) at System.Fabric.KeyValueStoreReplica.CreateNativeKeyValueStoreReplica(StatefulServiceInitializationParameters initParams, IFabricStoreEventHandler storeEventHandler, IFabricSecondaryEventHandler secondaryEventHandler) at System.Fabric.KeyValueStoreReplica.Store_Initialize(StatefulServiceInitializationParameters initializationParameters) at System.Fabric.KeyValueStoreReplica.Initialize(StatefulServiceInitializationParameters initializationParameters) at Microsoft.ServiceFabric.Actors.Runtime.KvsActorStateProvider.OnCreateAndInitializeReplica(StatefulServiceInitializationParameters initParams, Action1 copyHandler, Action1 replicationHandler, Func2 onDataLossHandler, Func2 restoreCompletedHandler) at Microsoft.ServiceFabric.Actors.Runtime.KvsActorStateProviderBase.Microsoft.ServiceFabric.Data.IStateProviderReplica.Initialize(StatefulServiceInitializationParameters initializationParameters) at System.Fabric.ServiceFactoryBroker.CreateHelper[TFactory,TReturnValue](IntPtr nativeServiceType, IntPtr nativeServiceName, UInt32 initializationDataLength, IntPtr nativeInitializationData, Guid partitionId, Func3 creationFunc, Action`2 initializationFunc, ServiceInitializationParameters initializationParameters)
For more information see: <a href='https://aka.ms/sfhealth' target='_blank'>https://aka.ms/sfhealth</a>

Following is the configuration I have for partitions and replicas.
<Parameter Name="UserActorsService_PartitionCount" Value="10" />
<Parameter Name="UserActorsService_MinReplicaSetSize" Value="2" />
<Parameter Name="UserActorsService_TargetReplicaSetSize" Value="3" />

&lt;Parameter Name=&#34;QueueActorsService_PartitionCount&#34; Value=&#34;10&#34; /&gt;
&lt;Parameter Name=&#34;QueueActorsService_MinReplicaSetSize&#34; Value=&#34;2&#34; /&gt;
&lt;Parameter Name=&#34;QueueActorsService_TargetReplicaSetSize&#34; Value=&#34;3&#34; /&gt;

Please suggest.
Thanks

Azure Service Fabric
Azure Service Fabric
An Azure service that is used to develop microservices and orchestrate containers on Windows and Linux.
252 questions
0 comments

4 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. António Albuquerque 1 Reputation point
    2020-05-15T10:00:45.02+00:00

    Hello, I'm facing a similar problem, did you do any code changes? Was this working before?
    What nuget version of service fabric are you using?

    Thanks

    0 comments
  2. Rasmus Svendsen 1 Reputation point
    2020-05-15T15:57:48.677+00:00

    We also hit the same issue today, when deployed our applications that contains actor services. Latest SF.runtime 7.1 and nuget version: 4.1.409.

    0 comments
  3. Nishant Sharma 1 Reputation point
    2020-05-15T16:03:19.237+00:00

    It seems that you should never use the latest nuget version of service fabric.

    https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-versions

    As per link above, the latest nuget version never supports the latest cluster SDK.

    So, downgrade your SDK to to 4.0.*

  4. Rasmus Svendsen 1 Reputation point
    2020-05-15T17:53:30.29+00:00

    Workaround: Downgrade all SF related nuget packages to version: 4.0.470, still on a 7.1 cluster - worked for me.

    0 comments