Just installed new Exchange 2019 CU10, with Security Update on both of our Exchange Servers, on Windows Server 2019 June Edition.
Created two new Databases, with one DAG.
A copy of both Databases is on each Server.
We have about 340 Active Users Mailboxes.
Primary Database is 120 Gigs.
Secondary (Archive) Database is about 40 Gigs (so far).
As soon as the environment was up and running, users began complaining that their Outlook Desktop Clients on Windows 10 keeps getting disconnected.
Investigation discovered that a few user mailboxes are instantly getting quarantined EVERY SINGLE TIME they open the Outlook Desktop Client, AND causing the database to dismount and failover.
System and Application Logs on Exchange Servers only give vague descriptions:
**Event ID 126
At '7/30/2021 6:27:54 PM' the Exchange store database <Database Name> copy on this server encountered an error that caused the database to be dismounted. For more detail about the failure, consult the Event log on the server for other "ExchangeStoreDb" or "msexchangerepl" events. A successful failover restored service.
Event Id 1013
The mailbox with mailbox guid <Mailbox GUID> caused a crash or resource outage on database <Database Name> (<Database GUID>).
Version: 15.02.0922.013
Description: ExAssertException: MED.ExAssert.AssertInternal,MESSC.ErrorHelper.AssertRetail,MESSSCS.ExecutionDiagnostics.OnExceptionCatch,MESSC.SecurityHelper.CreateSecurityIdentifier,MESSLDM.TopMessage.GetCreatorSecurityIdentifier,MEPMAPI.MapiMessage.CalculateSecurityRelatedProperty,MEPM
Event ID 4999
Watson report about to be sent for process id: 15092, with parameters: E12, c-RTL-AMD64, 15.02.0922.013, M.E.Store.Worker, M.E.S.Storage.StoreCommonServices, M.E.S.S.S.ExecutionDiagnostics.OnExceptionCatch, M.E.Diagnostics.ExAssertException, 96d4-dumptidset, 15.02.0922.013.
ErrorReportingEnabled: True
Event ID 1002
Unhandled exception (Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: Too many exceptions
at Microsoft.Exchange.Diagnostics.ExAssert.AssertInternal(String formatString, Object[] parameters)
at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.ExecutionDiagnostics.OnExceptionCatch(Exception exception, Object diagnosticData)
at Microsoft.Exchange.Server.Storage.Common.SecurityHelper.CreateSecurityIdentifier(IExecutionContext context, Byte[] buffer)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.TopMessage.GetCreatorSecurityIdentifier(Context context)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.CalculateSecurityRelatedProperty(MapiContext context, StorePropTag propertyTag)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.GetPropertyValueImp(MapiContext context, StorePropTag propTag, Boolean truncateLargeProperties)
at Microsoft.Exchange.Protocols.MAPI.MapiPropBagBase.InternalGetOneProp(MapiContext context, StorePropTag propTag, Boolean truncateLargeProperties)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.InternalGetAllProperties(MapiContext context, GetPropListFlags flags, Boolean loadValues, Boolean truncateLargeProperties, Predicate`1 propertyFilter)
at Microsoft.Exchange.Protocols.MAPI.MapiPropBagBase.GetAllProperties(MapiContext context, GetPropListFlags flags, Boolean loadValue, Boolean truncateLargeProperties)
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferPropertyBag.LoadAllPropertiesImp()
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferMessage.LoadAllPropertiesImp()
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferPropertyBag.<GetPropertyList>d_26.MoveNext()
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferPropertyBag.<GetAnnotatedProperties>d28.MoveNext()
at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferPropList.<Microsoft-Exchange-RpcClientAccess-FastTransfer-IFastTransferProcessor<Microsoft-Exchange-RpcClientAccess-FastTransfer-Parser-FastTransferDownloadContext>-Process>d16.MoveNext()
at Microsoft.Exchange.RpcClientAccess.FastTransfer.FastTransferStateMachine.Step()
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferContext`1.Process(TDataInterface dataInterface)
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferDownloadContext.Process(FastTransferWriter dataInterface)
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferDownloadContext.GetNextBuffer(ArraySegment`1 buffer)
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferDownloadContext.GetNextBuffer(MapiContext operationContext, ArraySegment`1 buffer)
at Microsoft.Exchange.Server.Storage.MapiDisp.RopHandler.FastTransferSourceGetBufferExtended(MapiContext context, MapiBase downloadContext, UInt16 bufferSize, FastTransferSourceGetBufferExtendedResultFactory resultFactory)
at Microsoft.Exchange.Server.Storage.MapiDisp.RopHandlerBase.FastTransferSourceGetBufferExtended(IServerObject serverObject, UInt16 bufferSize, FastTransferSourceGetBufferExtendedResultFactory resultFactory)
at Microsoft.Exchange.RpcClientAccess.Parser.RopFastTransferSourceGetBufferExtended.InternalExecute(IServerObject serverObject, IRopHandler ropHandler, ArraySegment`1 outputBuffer)
at Microsoft.Exchange.RpcClientAccess.Parser.InputRop.Execute(IConnectionInformation connection, IRopDriver ropDriver, ServerObjectHandleTable handleTable, ArraySegment`1 outputBuffer)
at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.ExecuteRops(List`1 inputArraySegmentList, ServerObjectHandleTable serverObjectHandleTable, ArraySegment`1 outputBuffer, Int32 outputIndex, Int32 maxOutputSize, Boolean isOutputBufferMaxSize, Int32& outputSize, AuxiliaryData auxiliaryData, Boolean isFake, Byte[]& fakeOut)
at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.ExecuteOrBackoff(IList`1 inputBufferArray, ArraySegment`1 outputBuffer, Int32& outputSize, AuxiliaryData auxiliaryData, Boolean isFake, Byte[]& fakeOut)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.<>cDisplayClass29_1.<DoRpc>b0(MapiContext operationContext, MapiSession& session, Boolean& deregisterSession, AuxiliaryData auxiliaryData)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.Execute(IExecutionDiagnostics executionDiagnostics, MapiContext outerContext, String functionName, Boolean isRpc, IntPtr& contextHandle, Boolean tryLockSession, String userDn, IList`1 dataIn, Int32 sizeInMegabytes, ArraySegment`1 auxIn, ArraySegment`1 auxOut, Int32& sizeAuxOut, ExecuteDelegate executeDelegate)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.DoRpc(IExecutionDiagnostics executionDiagnostics, IntPtr& contextHandle, IList`1 ropInArraySegments, ArraySegment`1 ropOut, Int32& sizeRopOut, Boolean internalAccessPrivileges, ArraySegment`1 auxIn, ArraySegment`1 auxOut, Int32& sizeAuxOut, Boolean fakeRequest, Byte[]& fakeOut)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcDoRpc(MapiExecutionDiagnostics executionDiagnostics, IntPtr& sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request, ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion completion)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcPoolSessionDoRpc_Unwrapped(MapiExecutionDiagnostics executionDiagnostics, IntPtr contextHandle, UInt32 sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request, ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion completion)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.<>cDisplayClass48_0.<EcPoolSessionDoRpc>b_0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch[T](Action tryDelegate, GenericFilterDelegate filterDelegate, GenericCatchDelegate catchDelegate, T state)).
Event ID 1001
Microsoft Exchange Server Information Store has encountered an internal logic error. Internal error text is (Too many exceptions) with a call stack of ( at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.ExecutionDiagnostics.OnExceptionCatch(Exception exception, Object diagnosticData)
at Microsoft.Exchange.Server.Storage.Common.SecurityHelper.CreateSecurityIdentifier(IExecutionContext context, Byte[] buffer)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.TopMessage.GetCreatorSecurityIdentifier(Context context)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.CalculateSecurityRelatedProperty(MapiContext context, StorePropTag propertyTag)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.GetPropertyValueImp(MapiContext context, StorePropTag propTag, Boolean truncateLargeProperties)
at Microsoft.Exchange.Protocols.MAPI.MapiPropBagBase.InternalGetOneProp(MapiContext context, StorePropTag propTag, Boolean truncateLargeProperties)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.InternalGetAllProperties(MapiContext context, GetPropListFlags flags, Boolean loadValues, Boolean truncateLargeProperties, Predicate`1 propertyFilter)
at Microsoft.Exchange.Protocols.MAPI.MapiPropBagBase.GetAllProperties(MapiContext context, GetPropListFlags flags, Boolean loadValue, Boolean truncateLargeProperties)
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferPropertyBag.LoadAllPropertiesImp()
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferMessage.LoadAllPropertiesImp()
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferPropertyBag.<GetPropertyList>d_26.MoveNext()
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferPropertyBag.<GetAnnotatedProperties>d28.MoveNext()
at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferPropList.<Microsoft-Exchange-RpcClientAccess-FastTransfer-IFastTransferProcessor<Microsoft-Exchange-RpcClientAccess-FastTransfer-Parser-FastTransferDownloadContext>-Process>d16.MoveNext()
at Microsoft.Exchange.RpcClientAccess.FastTransfer.FastTransferStateMachine.Step()
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferContext`1.Process(TDataInterface dataInterface)
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferDownloadContext.Process(FastTransferWriter dataInterface)
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferDownloadContext.GetNextBuffer(ArraySegment`1 buffer)
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferDownloadContext.GetNextBuffer(MapiContext operationContext, ArraySegment`1 buffer)
at Microsoft.Exchange.Server.Storage.MapiDisp.RopHandler.FastTransferSourceGetBufferExtended(MapiContext context, MapiBase downloadContext, UInt16 bufferSize, FastTransferSourceGetBufferExtendedResultFactory resultFactory)
at Microsoft.Exchange.Server.Storage.MapiDisp.RopHandlerBase.FastTransferSourceGetBufferExtended(IServerObject serverObject, UInt16 bufferSize, FastTransferSourceGetBufferExtendedResultFactory resultFactory)
at Microsoft.Exchange.RpcClientAccess.Parser.RopFastTransferSourceGetBufferExtended.InternalExecute(IServerObject serverObject, IRopHandler ropHandler, ArraySegment`1 outputBuffer)
at Microsoft.Exchange.RpcClientAccess.Parser.InputRop.Execute(IConnectionInformation connection, IRopDriver ropDriver, ServerObjectHandleTable handleTable, ArraySegment`1 outputBuffer)
at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.ExecuteRops(List`1 inputArraySegmentList, ServerObjectHandleTable serverObjectHandleTable, ArraySegment`1 outputBuffer, Int32 outputIndex, Int32 maxOutputSize, Boolean isOutputBufferMaxSize, Int32& outputSize, AuxiliaryData auxiliaryData, Boolean isFake, Byte[]& fakeOut)
at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.ExecuteOrBackoff(IList`1 inputBufferArray, ArraySegment`1 outputBuffer, Int32& outputSize, AuxiliaryData auxiliaryData, Boolean isFake, Byte[]& fakeOut)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.<>cDisplayClass29_1.<DoRpc>b0(MapiContext operationContext, MapiSession& session, Boolean& deregisterSession, AuxiliaryData auxiliaryData)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.Execute(IExecutionDiagnostics executionDiagnostics, MapiContext outerContext, String functionName, Boolean isRpc, IntPtr& contextHandle, Boolean tryLockSession, String userDn, IList`1 dataIn, Int32 sizeInMegabytes, ArraySegment`1 auxIn, ArraySegment`1 auxOut, Int32& sizeAuxOut, ExecuteDelegate executeDelegate)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.DoRpc(IExecutionDiagnostics executionDiagnostics, IntPtr& contextHandle, IList`1 ropInArraySegments, ArraySegment`1 ropOut, Int32& sizeRopOut, Boolean internalAccessPrivileges, ArraySegment`1 auxIn, ArraySegment`1 auxOut, Int32& sizeAuxOut, Boolean fakeRequest, Byte[]& fakeOut)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcDoRpc(MapiExecutionDiagnostics executionDiagnostics, IntPtr& sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request, ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion completion)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcPoolSessionDoRpc_Unwrapped(MapiExecutionDiagnostics executionDiagnostics, IntPtr contextHandle, UInt32 sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request, ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion completion)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.<>cDisplayClass48_0.<EcPoolSessionDoRpc>b_0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch[T](Action tryDelegate, GenericFilterDelegate filterDelegate, GenericCatchDelegate catchDelegate, T state)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcPoolSessionDoRpc(IntPtr contextHandle, UInt32 sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request, ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion completion)
at EcPoolSessionDoRpcRpc.EcDispatchCall(EcPoolSessionDoRpcRpc* , SafeRpcAsyncStateHandle pAsyncState, IPoolRpcServer server)
at PoolRpcServer_Wrapper.InternalExecute(PoolRpcServer_Wrapper* , SafeRpcAsyncStateHandle pAsyncState)
at Microsoft.Exchange.Rpc.ManagedExceptionAsyncCrashWrapper.Execute<class Microsoft::Exchange::Rpc::PoolRpc::SafeEcPoolSessionDoRpcRpcAsyncStateHandle>(ManagedExceptionAsyncCrashWrapper* , _RPC_ASYNC_STATE* pAsyncState)
at EcPoolSessionDoRpc_Managed(_RPC_ASYNC_STATE* pAsyncState, Void* cpxh, UInt32 ulSessionHandle, UInt32* pulFlags, UInt32 cbIn, Byte* rgbIn, UInt32* pcbOut, Byte* ppbOut, UInt32 cbAuxIn, Byte rgbAuxIn, UInt32* pcbAuxOut, Byte* ppbAuxOut)
).
*
That's it... No other Event ID's or errors in logs.
This occurs EVERY SINGLE TIME one of these users open Outlook Desktop Client.
Web Mail works fine.
E-Mail on mobile devices work fine.
I tried deleting and re-creating Outlook profiles.
I tried a different computer.
I deleted their mailboxes and re-created it.
I created a third database and moved their mailbox to it.
(This at least prevented all the other users from getting disconnected when the database failvoer occurs)
Nothing is working to fixe these few users that have this problem.
Before upgrading to this version of Exchange, none of these problems existed.
I have been on the phone with Microsoft PAID Support for DAYS!
They guy providing the support has to keep researching and collaborating with his team.
He sounds very young.
So far, they are not much help.
Users are getting angry.
Anyone out there have any ideas???