question

mashhour avatar image
0 Votes"
mashhour asked mashhour commented

Exchange 2019 CU10 database switch / failover is not working between two nodes in single dag

Hello all;
I have Exchange 2019 organization which consists of two nodes exchange 2019 in a DAG.
the two exchange servers are in one AD site, AD itself has one single AD site.
after upgrading the two Exchange servers to CU10. I face an issue in the failover / switch over the DBs between the two nodes, actually the DBs AP is distributed between the two Ex2019 servers. I uninstalled all 3rd party apps including the file level antivirus. and restarted both servers multiple times. I have one network adapter on each server, no replication dedicated adapter, the servers are on the same subnet, and are hosted on VMWare, I checked both scenarios where the servers are on the same host or on different VMWare host. all got the same failure results. now all DBs are active in the one of the servers, running the script ".\RedistributeActiveDatabases.ps1 -DagName DAG -BalanceDbsByActivationPreference". the script completes successfully, the DBs are balanced for a second then the DBs failover to the second server, multiple errors are there including the following:
MSExchangeIS error 1001 in application log in the server that is not accepting mounting DBs
"Microsoft Exchange Server Information Store has encountered an internal logic error. Internal error text is (ProcessId perf counter (0) does not match actual process id (20172).) with a call stack of ( at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.Common.Globals.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.PerformanceCounterFactory.CreateDatabaseInstance(StoreDatabase database)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.Globals.DatabaseMounting(Context context, StoreDatabase database, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StartupShutdown.Globals.DatabaseMountingHandler(Context context, StoreDatabase database, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.FinishMount(Context context, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.ActivatePassive(Context context)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountDatabase(Context context, MountFlags flags, Boolean& errorOnTheThreadExecutingTheMount)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.Storage.MountDatabase(Context context, StoreDatabase database, MountFlags flags)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcMountDatabase.EcExecuteRpc(MapiContext context)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics, ClientType clientType)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.<>c_DisplayClass37_0.<EcExecute>b_0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch[T](Action tryDelegate, GenericFilterDelegate filterDelegate, GenericCatchDelegate catchDelegate, T state)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute()
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcServer.EcMountDatabase50(ClientSecurityContext callerSecurityContext, Guid mdbGuid, UInt32 flags, Byte[] auxiliaryIn, Byte[]& auxiliaryOut)
at EcMountDatabaseRpc.EcDispatchCall(EcMountDatabaseRpc* , IAdminRpcServer server, ClientSecurityContext callerSecurityContext, Byte[] auxiliaryIn, Byte[]& auxiliaryOut)
at Microsoft.Exchange.Rpc.AdminRpc.AdminRpcServer_Wrapper.InternalExecute(AdminRpcServer_Wrapper* )
at Microsoft.Exchange.Rpc.ManagedExceptionCrashWrapper.Execute(ManagedExceptionCrashWrapper* )
at EcMountDatabase50_Managed(Void* hBinding, _GUID* pguidStorageGroup, GUID* pguidMdb, UInt32 ulFlags, UInt32 cbAuxIn, Byte* rgbAuxIn, UInt32* pcbAuxOut, Byte** prgbAuxOut)
)."
this error is followed by MSExchangeIS error 1002 in the app log
"Unhandled exception (Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: ProcessId perf counter (0) does not match actual process id (20172).
at Microsoft.Exchange.Diagnostics.ExAssert.AssertInternal(String formatString, Object[] parameters)
at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.Common.Globals.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.PerformanceCounterFactory.CreateDatabaseInstance(StoreDatabase database)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.Globals.DatabaseMounting(Context context, StoreDatabase database, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StartupShutdown.Globals.DatabaseMountingHandler(Context context, StoreDatabase database, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.FinishMount(Context context, Boolean readOnly)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.ActivatePassive(Context context)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.StoreDatabase.MountDatabase(Context context, MountFlags flags, Boolean& errorOnTheThreadExecutingTheMount)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.Storage.MountDatabase(Context context, StoreDatabase database, MountFlags flags)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpcMountDatabase.EcExecuteRpc(MapiContext context)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics, ClientType clientType)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.EcExecute_Unwrapped(AdminExecutionDiagnostics executionDiagnostics)
at Microsoft.Exchange.Server.Storage.AdminInterface.AdminRpc.<>c
DisplayClass37_0.<EcExecute>b_0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch[T](Action tryDelegate, GenericFilterDelegate filterDelegate, GenericCatchDelegate catchDelegate, T state))."

then followed by MSExchange Common error 4999
"Watson report about to be sent for process id: 20172, with parameters: E12, c-RTL-AMD64, 15.02.0922.013, M.E.Store.Worker, M.E.S.Storage.StoreCommonServices, M.E.S.S.S.PerformanceCounterFactory.CreateDatabaseInstance, M.E.Diagnostics.ExAssertException, 6c3-dumptidset, 15.02.0922.013.
ErrorReportingEnabled: True

the issue is not yet been solved.

office-exchange-server-ha
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

the DB balancing issue happened directly after upgrading the Exchange 2019 from CU9 to CU10 with the security update KB5004780

0 Votes 0 ·
ZhengqiLou-MSFT avatar image
0 Votes"
ZhengqiLou-MSFT answered mashhour commented

Hi @mashhour ,

Have you tried removing and recreating the DAG?

the two exchange servers are in one AD site, AD itself has one single AD site.

Do you mean the DC and the Exchange servers are not in one AD site?

Out situation is that now the databases are all mounted on one server, assuming it is Server1. When trying to manually mount the database copy on Server2, it failed, is that right?
I think you could remove the DAG and uninstall the Failover Cluster Service in Server Manager, and then retry creating the DAG or first re-install the Failover Cluster Service and create the DAG.

As for the Error event 4999, I did a lot research but still have no methods, I will do some tests and if I have any updates, I will post here.

Best regards,
Lou


If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you, removing and recreating the database availability group solved the issue

1 Vote 1 ·

Thank you Lou;
The AD and Exchange are on same AD site. actually the AD forest one domain with 2 DCs in one AD site, where the Exchange servers are installed.
the DBs are currently mounted on server2, when running move-activemailboxdatabase for any DB, it moves successfully, directly after when I run get-mailboxdatabasecopystatus for the respective DB, it gives me it is initializing on server1 and mounting on server2, after few seconds in server1 show the DB is healthy and server2 show the DB is mounted, and in between I find the logs above recorded on server1. I tried also the switches SkipActiveCopyChecks and SkipMoveSuppressionChecks. the logs are recorded in event viewer.
I did not retry to remove the DAG and the cluster, I will try that if there is no other method to solve this issue.

Regards;
Mashhour

0 Votes 0 ·

Hi @mashhour ,

Thank you for sharing the details about the server.

You could try restarting the MS Exchange IIS service and MS Exchange Repl service, as the error message seems to be related with them.
And what about the mounted database number of the servers, based on my knowledge it should be less than 5 in some situation.

Best regards,
Lou

0 Votes 0 ·
SanketSalvi-9791 avatar image
0 Votes"
SanketSalvi-9791 answered

hi @mashhour

If the issue is still not fixed. please check the link below.

https://ingogegenwarth.wordpress.com/2017/02/07/exchange-2016-and-filemappingsize/

Regards,
Sanket

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.