EX2019-CU10 OWA/ECP not working after July Security Update

Question

Hello,

After installing the July Security update access to ECP and OWA is broken.
Mail Flow works, but accessing OWA or ECP returns the following error:

ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1

It also logs error 1003 to the Event Logs.

As many others have suggested, we have tried replacing the OATH Certificate according to this: https://learn.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-oauth-expired several times, we also waited >60 minutes after doing it - but the error persists. Even after full server reboot.

Please advice on what to do next.

Full Stack Trace Here:

Server Error in '/owa' Application.  
ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1  
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.  
  
Exception Details: Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1  
  
Source Error:  
  
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.  
  
Stack Trace:  
  
  
[ExAssertException: ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1]  
   Microsoft.Exchange.Diagnostics.ExAssert.AssertInternal(String formatString, Object[] parameters) +241  
   Microsoft.Exchange.Diagnostics.ExAssert.RetailAssert(Boolean condition, String formatString, T1 parameter1, T2 parameter2) +2694334  
   Microsoft.Exchange.Clients.Common.HmacProvider.GetCertificates() +363  
   Microsoft.Exchange.Clients.Common.HmacProvider.GetHmacProvider() +140  
   Microsoft.Exchange.Clients.Common.HmacProvider.ComputeHmac(Byte[][] messageArrays) +14  
   Microsoft.Exchange.HttpProxy.FbaModule.SetCadataCookies(HttpApplication httpApplication) +1032  
   Microsoft.Exchange.HttpProxy.FbaFormPostProxyRequestHandler.HandleFbaFormPost(BackEndServer backEndServer) +3581  
   Microsoft.Exchange.HttpProxy.FbaFormPostProxyRequestHandler.ShouldContinueProxy() +20  
   Microsoft.Exchange.HttpProxy.ProxyRequestHandler.BeginProxyRequestOrRecalculate() +257  
   Microsoft.Exchange.HttpProxy.ProxyRequestHandler.InternalOnCalculateTargetBackEndCompleted(TargetCalculationCallbackBeacon beacon) +1528  
   Microsoft.Exchange.HttpProxy.<>c__DisplayClass280_0.<OnCalculateTargetBackEndCompleted>b__0() +303  
   Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate) +35  
   Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CallThreadEntranceMethod(Action method) +59  
  
[AggregateException: One or more errors occurred.]  
   Microsoft.Exchange.HttpProxy.ProxyRequestHandler.EndProcessRequest(IAsyncResult result) +414  
   System.Web.CallHandlerExecutionStep.InvokeEndHandler(IAsyncResult ar) +231  
   System.Web.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar) +172  
  
  
Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.8.4330.0   

Answer 1

Followed this instructions has solved the problem:
https://learn.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-oauth-expired

But is does take time (more then one hour) before it works.

Answer 2

It worked for me after 5 hours of waiting, EXCH2016CU21

Answer 3

Just out of curiosity: I had this problem with a 2019 Exchange Server in Hybrid mode and another one but a fresh install (AD & exchange - lab demo). Both servers took hours to get it resolved but the last one as it is hosted in a Demo Environment I shutdown the whole network and on the next day the Exchange server was back on.
It seems like the waiting hours are not related with leaving the server on but something regarding the certificate publishing/date/time. Does anybody know what happens under the hood and could give an better explanation, rather than "In some environments, it may take an hour for the OAuth certificate to be published." from https://learn.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-oauth-expired ?

Answer 4

Had the exact same issue with Exchange 2013 CU23 just last night. My Exchange Auth Certs had years left on them so proceeded with the patch installs on all servers and completed the AD Schema Update. Rebooted all servers and tested OWA and ECP access and could no longer access either of them. Log in pages would load fine but could not get passed sign in. Decided to renew the Exchange Auth certificates and recycled the App Pools as per the article but still couldn't get into OWA or ECP. Checked each Exchange sever with Get-AuthConfig to verify the new certificate had propagated to all Exchange servers and, based on its thumbprint, it had. ECP and OWA still weren't working. Waited an hour or so and tested both again and could then log in fine. No idea what happens in that 60mins which allows it to then start working, even though the new certificate had propagated to all Exchange servers within a few minutes. However, pleased to say, simply waiting did the trick for me so hope this helps someone else. Very annoying how this is documented as a "known issue" but ONLY when the existing Exchange Auth Certificate/s is expired. These patches are clearly breaking the existing certificate...

Answer 5

I suspect this command:

Set-AuthConfig -NewCertificateThumbprint <ThumbprintFromStep1> -NewCertificateEffectiveDate (Get-Date)

Does not take the timezone into account, i'm in NZ with a +12 timezone and that's about how long it took after I ran the command for it to start working.

Some people are saying it worked immediately, some 1 hour and someone posted it took 4 hours for them. This may correlate to their timezone?

Maybe try:

$Time = Get-Date

Set-AuthConfig -NewCertificateThumbprint <ThumbprintFromStep1> -NewCertificateEffectiveDate $Time.ToUniversalTime()