question

GeorgeMorris-1805 avatar image
0 Votes"
GeorgeMorris-1805 asked GeorgeMorris-1805 answered

Exchange Server 2019 CU10 - Outlook Desktop Client causes mailbox to get quarantined and database failover

Just installed new Exchange 2019 CU10, with Security Update on both of our Exchange Servers, on Windows Server 2019 June Edition.
Created two new Databases, with one DAG.
A copy of both Databases is on each Server.
We have about 340 Active Users Mailboxes.
Primary Database is 120 Gigs.
Secondary (Archive) Database is about 40 Gigs (so far).

As soon as the environment was up and running, users began complaining that their Outlook Desktop Clients on Windows 10 keeps getting disconnected.
Investigation discovered that a few user mailboxes are instantly getting quarantined EVERY SINGLE TIME they open the Outlook Desktop Client, AND causing the database to dismount and failover.
System and Application Logs on Exchange Servers only give vague descriptions:


**Event ID 126
At '7/30/2021 6:27:54 PM' the Exchange store database <Database Name> copy on this server encountered an error that caused the database to be dismounted. For more detail about the failure, consult the Event log on the server for other "ExchangeStoreDb" or "msexchangerepl" events. A successful failover restored service.


Event Id 1013
The mailbox with mailbox guid <Mailbox GUID> caused a crash or resource outage on database <Database Name> (<Database GUID>).
Version: 15.02.0922.013
Description: ExAssertException: MED.ExAssert.AssertInternal,MESSC.ErrorHelper.AssertRetail,MESSSCS.ExecutionDiagnostics.OnExceptionCatch,MESSC.SecurityHelper.CreateSecurityIdentifier,MESSLDM.TopMessage.GetCreatorSecurityIdentifier,MEPMAPI.MapiMessage.CalculateSecurityRelatedProperty,MEPM


Event ID 4999
Watson report about to be sent for process id: 15092, with parameters: E12, c-RTL-AMD64, 15.02.0922.013, M.E.Store.Worker, M.E.S.Storage.StoreCommonServices, M.E.S.S.S.ExecutionDiagnostics.OnExceptionCatch, M.E.Diagnostics.ExAssertException, 96d4-dumptidset, 15.02.0922.013.
ErrorReportingEnabled: True


Event ID 1002
Unhandled exception (Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: Too many exceptions
at Microsoft.Exchange.Diagnostics.ExAssert.AssertInternal(String formatString, Object[] parameters)
at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.ExecutionDiagnostics.OnExceptionCatch(Exception exception, Object diagnosticData)
at Microsoft.Exchange.Server.Storage.Common.SecurityHelper.CreateSecurityIdentifier(IExecutionContext context, Byte[] buffer)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.TopMessage.GetCreatorSecurityIdentifier(Context context)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.CalculateSecurityRelatedProperty(MapiContext context, StorePropTag propertyTag)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.GetPropertyValueImp(MapiContext context, StorePropTag propTag, Boolean truncateLargeProperties)
at Microsoft.Exchange.Protocols.MAPI.MapiPropBagBase.InternalGetOneProp(MapiContext context, StorePropTag propTag, Boolean truncateLargeProperties)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.InternalGetAllProperties(MapiContext context, GetPropListFlags flags, Boolean loadValues, Boolean truncateLargeProperties, Predicate`1 propertyFilter)
at Microsoft.Exchange.Protocols.MAPI.MapiPropBagBase.GetAllProperties(MapiContext context, GetPropListFlags flags, Boolean loadValue, Boolean truncateLargeProperties)
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferPropertyBag.LoadAllPropertiesImp()
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferMessage.LoadAllPropertiesImp()
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferPropertyBag.<GetPropertyList>d_26.MoveNext()
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferPropertyBag.<GetAnnotatedProperties>d
28.MoveNext()
at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferPropList.<Microsoft-Exchange-RpcClientAccess-FastTransfer-IFastTransferProcessor<Microsoft-Exchange-RpcClientAccess-FastTransfer-Parser-FastTransferDownloadContext>-Process>d
16.MoveNext()
at Microsoft.Exchange.RpcClientAccess.FastTransfer.FastTransferStateMachine.Step()
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferContext`1.Process(TDataInterface dataInterface)
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferDownloadContext.Process(FastTransferWriter dataInterface)
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferDownloadContext.GetNextBuffer(ArraySegment`1 buffer)
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferDownloadContext.GetNextBuffer(MapiContext operationContext, ArraySegment`1 buffer)
at Microsoft.Exchange.Server.Storage.MapiDisp.RopHandler.FastTransferSourceGetBufferExtended(MapiContext context, MapiBase downloadContext, UInt16 bufferSize, FastTransferSourceGetBufferExtendedResultFactory resultFactory)
at Microsoft.Exchange.Server.Storage.MapiDisp.RopHandlerBase.FastTransferSourceGetBufferExtended(IServerObject serverObject, UInt16 bufferSize, FastTransferSourceGetBufferExtendedResultFactory resultFactory)
at Microsoft.Exchange.RpcClientAccess.Parser.RopFastTransferSourceGetBufferExtended.InternalExecute(IServerObject serverObject, IRopHandler ropHandler, ArraySegment`1 outputBuffer)
at Microsoft.Exchange.RpcClientAccess.Parser.InputRop.Execute(IConnectionInformation connection, IRopDriver ropDriver, ServerObjectHandleTable handleTable, ArraySegment`1 outputBuffer)
at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.ExecuteRops(List`1 inputArraySegmentList, ServerObjectHandleTable serverObjectHandleTable, ArraySegment`1 outputBuffer, Int32 outputIndex, Int32 maxOutputSize, Boolean isOutputBufferMaxSize, Int32& outputSize, AuxiliaryData auxiliaryData, Boolean isFake, Byte[]& fakeOut)
at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.ExecuteOrBackoff(IList`1 inputBufferArray, ArraySegment`1 outputBuffer, Int32& outputSize, AuxiliaryData auxiliaryData, Boolean isFake, Byte[]& fakeOut)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.<>c
DisplayClass29_1.<DoRpc>b0(MapiContext operationContext, MapiSession& session, Boolean& deregisterSession, AuxiliaryData auxiliaryData)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.Execute(IExecutionDiagnostics executionDiagnostics, MapiContext outerContext, String functionName, Boolean isRpc, IntPtr& contextHandle, Boolean tryLockSession, String userDn, IList`1 dataIn, Int32 sizeInMegabytes, ArraySegment`1 auxIn, ArraySegment`1 auxOut, Int32& sizeAuxOut, ExecuteDelegate executeDelegate)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.DoRpc(IExecutionDiagnostics executionDiagnostics, IntPtr& contextHandle, IList`1 ropInArraySegments, ArraySegment`1 ropOut, Int32& sizeRopOut, Boolean internalAccessPrivileges, ArraySegment`1 auxIn, ArraySegment`1 auxOut, Int32& sizeAuxOut, Boolean fakeRequest, Byte[]& fakeOut)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcDoRpc(MapiExecutionDiagnostics executionDiagnostics, IntPtr& sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request, ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion completion)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcPoolSessionDoRpc_Unwrapped(MapiExecutionDiagnostics executionDiagnostics, IntPtr contextHandle, UInt32 sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request, ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion completion)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.<>c
DisplayClass48_0.<EcPoolSessionDoRpc>b_0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch[T](Action tryDelegate, GenericFilterDelegate filterDelegate, GenericCatchDelegate catchDelegate, T state)).


Event ID 1001
Microsoft Exchange Server Information Store has encountered an internal logic error. Internal error text is (Too many exceptions) with a call stack of ( at Microsoft.Exchange.Server.Storage.Common.ErrorHelper.AssertRetail(Boolean assertCondition, String message)
at Microsoft.Exchange.Server.Storage.StoreCommonServices.ExecutionDiagnostics.OnExceptionCatch(Exception exception, Object diagnosticData)
at Microsoft.Exchange.Server.Storage.Common.SecurityHelper.CreateSecurityIdentifier(IExecutionContext context, Byte[] buffer)
at Microsoft.Exchange.Server.Storage.LogicalDataModel.TopMessage.GetCreatorSecurityIdentifier(Context context)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.CalculateSecurityRelatedProperty(MapiContext context, StorePropTag propertyTag)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.GetPropertyValueImp(MapiContext context, StorePropTag propTag, Boolean truncateLargeProperties)
at Microsoft.Exchange.Protocols.MAPI.MapiPropBagBase.InternalGetOneProp(MapiContext context, StorePropTag propTag, Boolean truncateLargeProperties)
at Microsoft.Exchange.Protocols.MAPI.MapiMessage.InternalGetAllProperties(MapiContext context, GetPropListFlags flags, Boolean loadValues, Boolean truncateLargeProperties, Predicate`1 propertyFilter)
at Microsoft.Exchange.Protocols.MAPI.MapiPropBagBase.GetAllProperties(MapiContext context, GetPropListFlags flags, Boolean loadValue, Boolean truncateLargeProperties)
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferPropertyBag.LoadAllPropertiesImp()
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferMessage.LoadAllPropertiesImp()
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferPropertyBag.<GetPropertyList>d_26.MoveNext()
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferPropertyBag.<GetAnnotatedProperties>d
28.MoveNext()
at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferPropList.<Microsoft-Exchange-RpcClientAccess-FastTransfer-IFastTransferProcessor<Microsoft-Exchange-RpcClientAccess-FastTransfer-Parser-FastTransferDownloadContext>-Process>d
16.MoveNext()
at Microsoft.Exchange.RpcClientAccess.FastTransfer.FastTransferStateMachine.Step()
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferContext`1.Process(TDataInterface dataInterface)
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferDownloadContext.Process(FastTransferWriter dataInterface)
at Microsoft.Exchange.RpcClientAccess.FastTransfer.Parser.FastTransferDownloadContext.GetNextBuffer(ArraySegment`1 buffer)
at Microsoft.Exchange.Protocols.FastTransfer.FastTransferDownloadContext.GetNextBuffer(MapiContext operationContext, ArraySegment`1 buffer)
at Microsoft.Exchange.Server.Storage.MapiDisp.RopHandler.FastTransferSourceGetBufferExtended(MapiContext context, MapiBase downloadContext, UInt16 bufferSize, FastTransferSourceGetBufferExtendedResultFactory resultFactory)
at Microsoft.Exchange.Server.Storage.MapiDisp.RopHandlerBase.FastTransferSourceGetBufferExtended(IServerObject serverObject, UInt16 bufferSize, FastTransferSourceGetBufferExtendedResultFactory resultFactory)
at Microsoft.Exchange.RpcClientAccess.Parser.RopFastTransferSourceGetBufferExtended.InternalExecute(IServerObject serverObject, IRopHandler ropHandler, ArraySegment`1 outputBuffer)
at Microsoft.Exchange.RpcClientAccess.Parser.InputRop.Execute(IConnectionInformation connection, IRopDriver ropDriver, ServerObjectHandleTable handleTable, ArraySegment`1 outputBuffer)
at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.ExecuteRops(List`1 inputArraySegmentList, ServerObjectHandleTable serverObjectHandleTable, ArraySegment`1 outputBuffer, Int32 outputIndex, Int32 maxOutputSize, Boolean isOutputBufferMaxSize, Int32& outputSize, AuxiliaryData auxiliaryData, Boolean isFake, Byte[]& fakeOut)
at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.ExecuteOrBackoff(IList`1 inputBufferArray, ArraySegment`1 outputBuffer, Int32& outputSize, AuxiliaryData auxiliaryData, Boolean isFake, Byte[]& fakeOut)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.<>c
DisplayClass29_1.<DoRpc>b0(MapiContext operationContext, MapiSession& session, Boolean& deregisterSession, AuxiliaryData auxiliaryData)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.Execute(IExecutionDiagnostics executionDiagnostics, MapiContext outerContext, String functionName, Boolean isRpc, IntPtr& contextHandle, Boolean tryLockSession, String userDn, IList`1 dataIn, Int32 sizeInMegabytes, ArraySegment`1 auxIn, ArraySegment`1 auxOut, Int32& sizeAuxOut, ExecuteDelegate executeDelegate)
at Microsoft.Exchange.Server.Storage.MapiDisp.MapiRpc.DoRpc(IExecutionDiagnostics executionDiagnostics, IntPtr& contextHandle, IList`1 ropInArraySegments, ArraySegment`1 ropOut, Int32& sizeRopOut, Boolean internalAccessPrivileges, ArraySegment`1 auxIn, ArraySegment`1 auxOut, Int32& sizeAuxOut, Boolean fakeRequest, Byte[]& fakeOut)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcDoRpc(MapiExecutionDiagnostics executionDiagnostics, IntPtr& sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request, ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion completion)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcPoolSessionDoRpc_Unwrapped(MapiExecutionDiagnostics executionDiagnostics, IntPtr contextHandle, UInt32 sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request, ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion completion)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.<>c
DisplayClass48_0.<EcPoolSessionDoRpc>b_0()
at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch[T](Action tryDelegate, GenericFilterDelegate filterDelegate, GenericCatchDelegate catchDelegate, T state)
at Microsoft.Exchange.Server.Storage.MapiDisp.PoolRpcServer.EcPoolSessionDoRpc(IntPtr contextHandle, UInt32 sessionHandle, UInt32 flags, UInt32 maximumResponseSize, ArraySegment`1 request, ArraySegment`1 auxiliaryIn, IPoolSessionDoRpcCompletion completion)
at EcPoolSessionDoRpcRpc.EcDispatchCall(EcPoolSessionDoRpcRpc* , SafeRpcAsyncStateHandle pAsyncState, IPoolRpcServer server)
at PoolRpcServer_Wrapper.InternalExecute(PoolRpcServer_Wrapper* , SafeRpcAsyncStateHandle pAsyncState)
at Microsoft.Exchange.Rpc.ManagedExceptionAsyncCrashWrapper.Execute<class Microsoft::Exchange::Rpc::PoolRpc::SafeEcPoolSessionDoRpcRpcAsyncStateHandle>(ManagedExceptionAsyncCrashWrapper* , _RPC_ASYNC_STATE* pAsyncState)
at EcPoolSessionDoRpc_Managed(_RPC_ASYNC_STATE* pAsyncState, Void* cpxh, UInt32 ulSessionHandle, UInt32* pulFlags, UInt32 cbIn, Byte* rgbIn, UInt32* pcbOut, Byte* ppbOut, UInt32 cbAuxIn, Byte rgbAuxIn, UInt32* pcbAuxOut, Byte* ppbAuxOut)
).
*


That's it... No other Event ID's or errors in logs.
This occurs EVERY SINGLE TIME one of these users open Outlook Desktop Client.
Web Mail works fine.
E-Mail on mobile devices work fine.
I tried deleting and re-creating Outlook profiles.
I tried a different computer.
I deleted their mailboxes and re-created it.
I created a third database and moved their mailbox to it.
(This at least prevented all the other users from getting disconnected when the database failvoer occurs)

Nothing is working to fixe these few users that have this problem.
Before upgrading to this version of Exchange, none of these problems existed.
I have been on the phone with Microsoft PAID Support for DAYS!
They guy providing the support has to keep researching and collaborating with his team.
He sounds very young.
So far, they are not much help.
Users are getting angry.
Anyone out there have any ideas???

office-exchange-server-connectivity
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I saw the following discussion and inspected the MaxSendSize and MaxReceiveSize settings they mentioned.
But my settings were already correct.
https://social.msdn.microsoft.com/Forums/en-US/5d226be9-a586-4581-8ec1-674559790a3a/mailboxes-are-quarantined-under-new-exchange-2019-server?forum=Exch2019

0 Votes 0 ·

Re: https://social.msdn.microsoft.com/Forums/en-US/5d226be9-a586-4581-8ec1-674559790a3a/mailboxes-are-quarantined-under-new-exchange-2019-server?forum=Exch2019

I also tried all the same trouble-shooting steps as the OP of this thread.
I am beyond frustrated and exhausted.
Mailbox Repairs, Mailbox Moves...
Mailbox Export - then disable/delete mailbox -then create new database - then recreate mailbox on new database - then PST Repair - then Mailbox Import from PST import, etc.
Then I tried deleting the mailbox and recreating a fresh database and mailbox, and NOT importing anything.
Outlook still caused database failover and quarantined mailbox. ??? >:< What??

Been sifting through Exchange logs for Hours and Hours!

0 Votes 0 ·

1 Answer

GeorgeMorris-1805 avatar image
1 Vote"
GeorgeMorris-1805 answered

I FIGURED IT OUT!!
The Contacts were somehow causing the entire problem!
Exported Contacts from account.
Deleted contacts from Account.
Key was removing account from mobile devices, THEN deleting contacts and re-importing.
I noticed that every time the users got quarantined, their Outlook Clients all said "updating contacts" on the status bar.
This prompted me to try removing contacts.
I noticed every time I deleted all of them, they started coming back almost immediately.
I had to remove the account from the users' iphones.
Then I was able to delete and re-import the contacts.
Then added account back to iphone.
All affected users used iphones.
WOW! I cannot believe the fix was this simple.
I have been thinking it was a problem with the Exchange Server the entire time!
When I imported the contacts, the "conflicts" folder had several sync conflicts in it that were not there before.
Told it to delete duplicates.
Everything is as it should be now.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.