MSExchangeAppPool crashing Event 4999 AddFromSchemaPropertyList KeyNotFoundException

Question

Error:

Watson report about to be sent for process id: 4832, with parameters: E12IIS, c-RTL-AMD64, 15.01.1847.003, w3wp#MSExchangeServicesAppPool, M.Exchange.Services, M.E.S.C.D.ToServiceObjectPropertyListBase.AddFromSchemaPropertyList, S.C.Generic.KeyNotFoundException, ad37-dumptidset, 15.01.1847.003.
ErrorReportingEnabled: True

No effects on Exchange that I can see, no other errors, no one reporting issues, etc.

Not sure what related to - Is it DAG property I missed configuring, or AD, or .Net (4.7.2 is installed on the E2K16 servers).

Can pinpoint first time that error started: We have 2 sites across WAN (Site A, Site B), 2 E2K16 CU14 servers in each site (A1, A2, B1, B2). Error started during first copy (seeding) of A1 db across WAN to B1, about an hour into seeding, on B1. This was first time ever that we copied a Site A db to Site B. I subsequently copied all A1 and A2 databases over next 3 days. Error started on B2 next day, A1 6 days later, and A2 2 days after A1. Now, several times a day each server.

Troubleshooting steps taken:

Test-ReplicationHealth has NO errors. Get-MailboxDatabaseCopyStatus has NO errors. Get-ServerHealth for EWS, EWS.Protocol, EWS.Proxy all Healthy.

Restarting app pool on all servers did not clear error. Rebooting all servers (installed July W2K16 updates) did not clear error.

Running CU16 Schema, AD, Domain prep few days ago (directly on DCs) did not clear error. (CU16 cause we try stay N-1)

Searching internet yielded nothing. EWS Managed API NOT installed, btw, found posts on 4999 related to that.

Any ideas?

Thanks,
Joan

Answer 1

Hi YukiSun,

I was able to find more information in the EWS logs, thanks to an old forum post I found. The error is related to users using Search/Find People in Outlook. I spot checked one log each on all 4 servers, all 4999 related EWS log entries the same. Outlook is O365 semi-annual channel. Different users. Pretty sure desktop app. (Not all users have phones, ActiveSync is only turned on for those who do and most don't use the Outlook app on their phone. OWA is turned off for everyone, we don't use it.) I can only post partial log entry, but the rest repeats same issue.

2020-08-10T14:23:17.468Z,<GUID>,15,1,1979,3,Unknown,,Negotiate,true,<user>,<domain>,Microsoft Office/16.0 (Windows NT 10.0; Microsoft Outlook 16.0.12527; Pro),Target=None;Req=V2016_06_30/V2016_06_30;,<IP>,<ServerB2>,<ServerA1.FQDN>,FindPeople,500,1161,,ErrorInternalServerError,<user>,,,<GUID>,PrimaryServer,LocalTask,0,0,0,0,0,,,,,,,,,,,,,,,,,,,,...

...ServiceDiagnostics_SendWatsonReportOnUnhandledException=System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary. at System.ThrowHelper.ThrowKeyNotFoundException() at System.Collections.Generic.Dictionary2.get_Item(TKey key) at Microsoft.Exchange.Services.Core.DataConverter.ToServiceObjectPropertyListBase.AddFromSchemaPropertyList(Shape shapeToAddFrom ShapeEnum shapeEnum) at Microsoft.Exchange.Services.Core.DataConverter.ToServiceObjectPropertyListBase.SelectProperties(IMinimalCallContext callContext) at Microsoft.Exchange.Services.Core.DataConverter.ToServiceObjectPropertyListBase..ctor(IMinimalCallContext callContext Shape shape ResponseShape responseShape) at Microsoft.Exchange.Services.Core.Types.Persona.GetPropertyListForPersonaResponseShape(PersonaResponseShape personaResponseShape) at Microsoft.Exchange.Services.Core.SearchPeopleInDirectory.ExecuteInternal() at Microsoft.Exchange.Services.Core.SearchPeopleInDirectory.Execute() at Microsoft.Exchange.Services.Core.SearchPeopleInMailboxAndDirectory.Execute() at Microsoft.Exchange.Services.Core.FindPeople.Execute() at Microsoft.Exchange.Services.Core.ExceptionHandler1.Execute(IRequestDetailsLogger logger CreateServiceResult createServiceResult Int32 index ExecutionOption executionOption) at Microsoft.Exchange.Services.Core.BaseStepServiceCommand3.InternalExecuteStep(Boolean& isBatchStopResponse) at Microsoft.Exchange.Services.Core.ServiceCommandBase1.<ExecuteStep>b__82_0() at Microsoft.Exchange.Services.Core.ServiceCommandBase1.<>c__DisplayClass88_0.<ExecuteHelper>b__0() at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate Func2 filterDelegate Action1 catchDelegate);BaseServiceTask_SendWatsonReportOnGrayException=Microsoft.Exchange.Common.GrayException ---> System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary. at System.ThrowHelper.ThrowKeyNotFoundException() at System.Collections.Generic.Dictionary2.get_Item(TKey key) at Microsoft.Exchange.Services.Core.DataConverter.ToServiceObjectPropertyListBase.AddFromSchemaPropertyList(Shape shapeToAddFrom ShapeEnum shapeEnum) at Microsoft.Exchange.Services.Core.DataConverter.ToServiceObjectPropertyListBase.SelectProperties(IMinimalCallContext callContext) at Microsoft.Exchange.Services.Core.DataConverter.ToServiceObjectPropertyListBase..ctor(IMinimalCallContext callContext Shape shape ResponseShape responseShape) at Microsoft.Exchange.Services.Core.Types.Persona.GetPropertyListForPersonaResponseShape(PersonaResponseShape personaResponseShape) at Microsoft.Exchange.Services.Core.SearchPeopleInDirectory.ExecuteInternal() at Microsoft.Exchange.Services.Core.SearchPeopleInDirectory.Execute() at Microsoft.Exchange.Services.Core.SearchPeopleInMailboxAndDirectory.Execute() at Microsoft.Exchange.Services.Core.FindPeople.Execute() at Microsoft.Exchange.Services.Core.ExceptionHandler1.Execute(IRequestDetailsLogger logger CreateServiceResult createServiceResult Int32 index ExecutionOption executionOption) at Microsoft.Exchange.Services.Core.BaseStepServiceCommand3.InternalExecuteStep(Boolean& isBatchStopResponse) at Microsoft.Exchange.Services.Core.ServiceCommandBase1.<ExecuteStep>b__82_0() at Microsoft.Exchange.Services.Core.ServiceCommandBase1.<>c__DisplayClass88_0.<ExecuteHelper>b__0() at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate Func2 filterDelegate Action1 catchDelegate) at Microsoft.Exchange.Services.Core.ServiceDiagnostics.

Here is the old forum post I found, BUT his cause is NOT my cause. We don't have MACs, and none of the users in the logs I checked have delegate/Send as/Full permissions on any other mailbox but their own. The post just led me to the EWS logs.

https://social.microsoft.com/Forums/windows/en-us/d7d2e87f-052a-4c69-9106-136cdcac22fa/4999-event-id-and-sometimes-eid-1000-when-ews-client-attempts-appointment-response-exchage-2010?forum=smallbusinessserver

Joan

Answer 2

Have you noticed if there were any other events which could be related to this error?

Running CU16 Schema, AD, Domain prep few days ago (directly on DCs) did not clear error. (CU16 cause we try stay N-1)

Do you mean you are now running CU14 and are planning to upgrade to CU16? If this is the case, I'd like to recommend moving on to installing CU16 and see if there is any improvment. You would need to install .NET Framework 4.8 as required by CU16, see Supported .NET Framework versions for Exchange 2016.

Answer 3

YukiSun,

NO OTHER EVENTS, as I had stated in my post ("no errors" includes no events.)

I installed .Net 4.8, AND the .Net 4.8 update that Windows Update found after reboot, AND CU16 on Server B2, and the error DID NOT STOP, it continues. I am going to install .Net 4.8 and CU16 on the other 3 servers over next two days, but I do NOT expect the error to stop on these either. They ALL have the SAME configuration.

At least, TG, the error did not prevent the .Net 4.8 and CU16 install on B2, I did have that thouhght.

Joan

Answer 4

YukiSun,

All servers have .Net 4.8 and CU16, and error continues.

ANY IDEAS WHY?

Joan

Answer 5

Hi Joan,

Regarding the error, I searched a lot and did find some posts about Event 4999, but none of them are related to "AddFromSchemaPropertyList" or "KeyNotFoundException". As it now occurs several times a day, may I know if there are any operations or events that may probably trigger the error?