question

NishantSharma-7365 avatar image
NishantSharma-7365 asked ·

Service Fabric Actors unable to create replicas

I am trying to deploy 2 stateless services, 1 stateful service and two actor services on a 3 node service fabric cluster. Other services are getting deployed without errors but actor service seems to have gotten into trouble.

.... "Description": "Partition is below target replica or instance count.".....

It seems that primaries are getting created fine, but it is having trouble creating/configuring replicas. The exception I see in SF explorer is

'System.RA' reported Warning for property 'ReplicaOpenStatus'.
Replica had multiple failures during open on jsiugm57a_2. API call: IStatefulServiceReplica.Open(); Error = System.InvalidCastException (-2147467262)
Unable to cast COM object of type 'System.
ComObject' to interface type 'IFabricKeyValueStoreReplica8'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{FB87EC27-FFF9-4C03-820E-41A7EF38E732}' failed due to the following error: No such interface supported (0x80004002 (E_NOINTERFACE)).
at System.StubHelpers.InterfaceMarshaler.ConvertToManaged(IntPtr pUnk, IntPtr itfMT, IntPtr classMT, Int32 flags)
at System.Fabric.Interop.NativeRuntime.FabricCreateKeyValueStoreReplica5(Guid& riid, IntPtr storeName, Guid partitionId, Int64 replicaId, IntPtr serviceName, IntPtr fabricReplicatorSettings, IntPtr kvsSettings, FABRIC_LOCAL_STORE_KIND localStoreKind, IntPtr localStorageSettings, IFabricStoreEventHandler storeEventHandler, IFabricSecondaryEventHandler secondaryEventHandler)
at System.Fabric.KeyValueStoreReplica.CreateNativeKeyValueStoreReplicaHelper(String storeName, Guid partitionId, Int64 replicaId, Uri serviceName, ReplicatorSettings replicatorSettings, LocalStoreSettings localStoreSettings, KeyValueStoreReplicaSettings kvsSettings, IFabricStoreEventHandler storeEventHandler, IFabricSecondaryEventHandler secondaryEventHandler)
at System.Fabric.KeyValueStoreReplica.<>c
DisplayClass106_0.<CreateNativeKeyValueStoreReplica>b_0()
at System.Fabric.Interop.Utility.WrapNativeSyncInvoke[TResult](Func`1 func, String functionTag, String functionArgs)
at System.Fabric.KeyValueStoreReplica.CreateNativeKeyValueStoreReplica(StatefulServiceInitializationParameters initParams, IFabricStoreEventHandler storeEventHandler, IFabricSecondaryEventHandler secondaryEventHandler)
at System.Fabric.KeyValueStoreReplica.Store_Initialize(StatefulServiceInitializationParameters initializationParameters)
at System.Fabric.KeyValueStoreReplica.Initialize(StatefulServiceInitializationParameters initializationParameters)
at Microsoft.ServiceFabric.Actors.Runtime.KvsActorStateProvider.OnCreateAndInitializeReplica(StatefulServiceInitializationParameters initParams, Action`1 copyHandler, Action`1 replicationHandler, Func`2 onDataLossHandler, Func`2 restoreCompletedHandler)
at Microsoft.ServiceFabric.Actors.Runtime.KvsActorStateProviderBase.Microsoft.ServiceFabric.Data.IStateProviderReplica.Initialize(StatefulServiceInitializationParameters initializationParameters)
at System.Fabric.ServiceFactoryBroker.CreateHelper[TFactory,TReturnValue](IntPtr nativeServiceType, IntPtr nativeServiceName, UInt32 initializationDataLength, IntPtr nativeInitializationData, Guid partitionId, Func`3 creationFunc, Action`2 initializationFunc, ServiceInitializationParameters initializationParameters)
For more information see: <a href='https://aka.ms/sfhealth' target='_blank'>https://aka.ms/sfhealth</a>

Following is the configuration I have for partitions and replicas.
<Parameter Name="UserActorsService_PartitionCount" Value="10" />
<Parameter Name="UserActorsService_MinReplicaSetSize" Value="2" />
<Parameter Name="UserActorsService_TargetReplicaSetSize" Value="3" />

 <Parameter Name="QueueActorsService_PartitionCount" Value="10" />
 <Parameter Name="QueueActorsService_MinReplicaSetSize" Value="2" />
 <Parameter Name="QueueActorsService_TargetReplicaSetSize" Value="3" />

Please suggest.
Thanks

azure-service-fabric
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AntnioAlbuquerque-5887 avatar image
AntnioAlbuquerque-5887 answered ·

Hello, I'm facing a similar problem, did you do any code changes? Was this working before?
What nuget version of service fabric are you using?

Thanks

Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RasmusSvendsen-4894 avatar image
RasmusSvendsen-4894 answered ·

We also hit the same issue today, when deployed our applications that contains actor services. Latest SF.runtime 7.1 and nuget version: 4.1.409.

Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

NishantSharma-7365 avatar image
NishantSharma-7365 answered ·

It seems that you should never use the latest nuget version of service fabric.

https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-versions

As per link above, the latest nuget version never supports the latest cluster SDK.

So, downgrade your SDK to to 4.0.*

1 comment Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

The release note from 7.1 indicate that you should use the *.409 build. https://github.com/microsoft/service-fabric/blob/master/release_notes/Service-Fabric-71-releasenotes.md
On one of our SF clusters works on the other it does not work.

0 Votes 0 · ·
RasmusSvendsen-4894 avatar image
RasmusSvendsen-4894 answered ·

Workaround: Downgrade all SF related nuget packages to version: 4.0.470, still on a 7.1 cluster - worked for me.

Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.