question

EmilGustafsson-1189 avatar image
3 Votes"
EmilGustafsson-1189 asked AntonYundin-6306 commented

EX2019-CU10 OWA/ECP not working after July Security Update

Hello,


After installing the July Security update access to ECP and OWA is broken.
Mail Flow works, but accessing OWA or ECP returns the following error:

ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1

It also logs error 1003 to the Event Logs.

As many others have suggested, we have tried replacing the OATH Certificate according to this: https://docs.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-oauth-expired several times, we also waited >60 minutes after doing it - but the error persists. Even after full server reboot.


Please advice on what to do next.




Full Stack Trace Here:

 Server Error in '/owa' Application.
 ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1
 Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
    
 Exception Details: Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1
    
 Source Error:
    
 An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
    
 Stack Trace:
    
    
 [ExAssertException: ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1]
    Microsoft.Exchange.Diagnostics.ExAssert.AssertInternal(String formatString, Object[] parameters) +241
    Microsoft.Exchange.Diagnostics.ExAssert.RetailAssert(Boolean condition, String formatString, T1 parameter1, T2 parameter2) +2694334
    Microsoft.Exchange.Clients.Common.HmacProvider.GetCertificates() +363
    Microsoft.Exchange.Clients.Common.HmacProvider.GetHmacProvider() +140
    Microsoft.Exchange.Clients.Common.HmacProvider.ComputeHmac(Byte[][] messageArrays) +14
    Microsoft.Exchange.HttpProxy.FbaModule.SetCadataCookies(HttpApplication httpApplication) +1032
    Microsoft.Exchange.HttpProxy.FbaFormPostProxyRequestHandler.HandleFbaFormPost(BackEndServer backEndServer) +3581
    Microsoft.Exchange.HttpProxy.FbaFormPostProxyRequestHandler.ShouldContinueProxy() +20
    Microsoft.Exchange.HttpProxy.ProxyRequestHandler.BeginProxyRequestOrRecalculate() +257
    Microsoft.Exchange.HttpProxy.ProxyRequestHandler.InternalOnCalculateTargetBackEndCompleted(TargetCalculationCallbackBeacon beacon) +1528
    Microsoft.Exchange.HttpProxy.<>c__DisplayClass280_0.<OnCalculateTargetBackEndCompleted>b__0() +303
    Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate) +35
    Microsoft.Exchange.HttpProxy.ProxyRequestHandler.CallThreadEntranceMethod(Action method) +59
    
 [AggregateException: One or more errors occurred.]
    Microsoft.Exchange.HttpProxy.ProxyRequestHandler.EndProcessRequest(IAsyncResult result) +414
    System.Web.CallHandlerExecutionStep.InvokeEndHandler(IAsyncResult ar) +231
    System.Web.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar) +172
    
    
 Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.8.4330.0 


office-exchange-server-administrationoffice-exchange-online-itprooffice-exchange-server-connectivity
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Same problem here since Exchange security update last night. Recreating and assigning a new certificate will not resolve the problem.
Outlook works fine!, but OWA and ECP only work to signin page, after typing credentials it gives the message.

Hope someone can help to fix this!

1 Vote 1 ·

Hi,
Are you accessing ECP/OWA via a load balanced URL? If so, try directly accessed from the server URL.

1 Vote 1 ·

I experience the same error. Currently 2 servers, on one server the workaround to re-create the OAUTH certificate worked well. On the other one, it worked at first glance, but after the next reboot the error appeared again. After removing the Security Update KB5007480 everything's ok again.

As it is designated to be a "hybrid" server by next week, I of course do not want to operate this server without that update.

Does anybody experience the same issue where the re-creation of the OAUTH cert is not a permanent solution?
Has anybody solved that issue?

Cheers, Thomas

0 Votes 0 ·

If you are running a local exchange server in Hybrid mode just run the latest version of the Hybrid Configuration Wizard and follow the steps to reconfigure the connection. Gave it 30 minutes after going through the configuration and now everything is working again.

0 Votes 0 ·

If you are running a local exchange server in Hybrid mode just run the latest version of the Hybrid Configuration Wizard and follow the steps to reconfigure the connection. Gave it 30 minutes after going through the configuration and now everything is working again.

0 Votes 0 ·
WillemHendrikBerkhof-9467 avatar image
4 Votes"
WillemHendrikBerkhof-9467 answered AntonYundin-6306 commented

Followed this instructions has solved the problem:
https://docs.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-oauth-expired

But is does take time (more then one hour) before it works.

· 10
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Can confirm this.

I renew the cert for like the fifth time and then went to bed.

Woke up and saw this comment and checked, and mine works now too.

1 Vote 1 ·
RomeoS-7348 avatar image RomeoS-7348 EmilGustafsson-1189 ·

Same here, never had so much anxiety after waking up but at least I'm awake now :-))

0 Votes 0 ·

Yeah really got the system going and jump out of bed didn't it :D

0 Votes 0 ·

Yeah, you are right, my cert gonna expire on year 26 but at event viewer it don´´t stop to alert me since I did yesterday the "security Exchange 2016 update" and this link to renew and apply the new cert only to the auth, and wait a couple of hours, did it only in one server and it fix every OWAs and ECPs

thanks a lot, so fast to report this Microsoft fail, in my case I updated test enviroment, on saturday I´m gonna do on producction, we´ll see but I hope I only will have to do this on first room servers.

thanks a lot Willem

Best regards and keep save

0 Votes 0 ·

Thank you so much for this link. We have a simple configuration, so going through all the steps all the way down through restarting IIS fixed our issue immediately. Now I can go back to being on vacation.

0 Votes 0 ·

You can try to copy "Microsoft Exchange Server Auth Certificate" to the "Trusted Root Certification Authorities" on all servers in order to restore the ecp and owa functionality through hardware balancer.
For me these worked: if you go to specific server with http://server.domain/owa worked OK but for the https://vip.domain/owa the page prompted continously for credentials.

0 Votes 0 ·
Show more comments
EmilGustafsson-1189 avatar image
0 Votes"
EmilGustafsson-1189 answered EmilGustafsson-1189 commented

Our Exchange does handle multiple SMTP-domains. The certificate was issued for the one marked default by Get-AcceptedDomain

Would I have to do this for all domains in the server?

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I have an environment with multiple domains as well, and those domains are used as the primary email address depending on which department you are in. Did running this using the default domain from the Get-AcceptedDomain resolve the issue for you?

Thanks

0 Votes 0 ·

Yeah it did, but it took a few hours.

First time I waited around 65 minutes, then I started troubleshooting - to no success. Redoing it multiple times etc.

So I did it for the default domain one last time, and went to bed since Mail Flow was working, OWA / ECP not working wasn't a deal breaker.

When I woke up, it worked.

Other have reported it taking up to 4 hours, so do the cert renewal and just say it'll take a few hours for it to fix.

I'm sure it's gonna work for you.

0 Votes 0 ·
WillemHendrikBerkhof-9467 avatar image
0 Votes"
WillemHendrikBerkhof-9467 answered mavsafe-9397 commented

Same problem here since Exchange security update last night. Recreating and assigning a new certificate will not resolve the problem.
Outlook works fine!, but OWA and ECP only work to signin page, after typing credentials it gives the message.

Hope someone can help to fix this!

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

We manually uninstalled the Exchange 2019 CU9 security update (KB5004780) and OWA started working again.

See description-of-the-security-update-for-microsoft-exchange-server-2019-july-13-2021-kb5004780-fc5b3fa1-1f7a-47b0-8014-699257256bb5+


0 Votes 0 ·

Do you mind sharing the steps/process to uninstall the security update?

0 Votes 0 ·

Same issue with Exchange 2013 CU23 after uninstalling July SU KB5004778 the issue was resolved. Then after checking the Exchange Team Blog you can see they added "Installation Tips"

https://techcommunity.microsoft.com/t5/exchange-team-blog/released-july-2021-exchange-server-security-updates/ba-p/2523421

0 Votes 0 ·
AshleyMartin-7395 avatar image
0 Votes"
AshleyMartin-7395 answered

We had this exact error from OWA/ECP and replacing the OAuth "Microsoft Exchange Server Auth Certificate" does work but the time for the certificate to "publish" seems to be inordinate. It was at least four hours for us, which happened to be over night so i'm not sure exactly how long.

To replace the OAuth cert, we followed these steps https://docs.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-oauth-expired with the exception of the domain being a wildcard, so -DomainName "*.contoso.com"

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

CdricRichard-7272 avatar image
0 Votes"
CdricRichard-7272 answered CdricRichard-7272 published

same error with CU9 and now with CU10.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RomeoS-7348 avatar image
1 Vote"
RomeoS-7348 answered RomeoS-7348 commented

Same error with Exchange 2013 CU23. Replaced the OAuth certificate about an hour ago but no luck yet. Our original OAuth certificate did not expire until 2024.

I followed these steps to replace the OAuth cert:
https://docs.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-oauth-expired

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Same here, I ran the July update from windows update, and had ECP and OWA not work with error above after restart. My OAuth doesn't expire until 2024. I ran the steps to replace it, and restarted the services Microsoft Exchange Service Host Service.
ran IISReset command to restart IIS. Then it started to work right away it seems, fingers crossed.

0 Votes 0 ·

In my case it just took approx. 2 - 3 hours until it started working again. Seems to be all OK now, knock on wood...

2 Votes 2 ·
stefanseidltelecrewde-0838 avatar image
0 Votes"
stefanseidltelecrewde-0838 answered KK-1143 commented

hello, we have the same problem after install the Exchange Server 2013 CU23 (KB5004778) AND Exchange Server 2019 CU10 (KB5004780] july update
3x ex134 and 1x ex19 OWA is corupt !

this option: https://docs.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-oauth-expired dont work for me, becuase my Exchange Server Open Authentication (OAuth) certificate is NOT! expired

so anyone habe a other idea ?


sorry @microsoft WHATS out Problem?

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@stefanseidltelecrewde-0838 My OAUTH cert was not expired either. Something in this security update seems to break the OAUTH cert, even if it is not expired. Going through this process fixed our problem. https://docs.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-oauth-expired

It is not immediate. Microsoft says this can take an hour to process the change. It took about 20 minutes for ours to work, but I have seen people say that it took them a couple of hours.

Good luck.

1 Vote 1 ·

and this morning:
drum roll
works it again
after 1, 2, 3 hours no function was possible

0 Votes 0 ·

I tried this solution as well and waited over 19 hours because i'm EST and i wasn't sure if I could check at 5 ours or wait 19 hours.

When I did this, man I was really tired having wrestled with this issue for three days, but I think I recall seeing a message that said something about time to publish and the number 48. I hope that doesn't mean I have to wait two days from about 4PM EST yesterday. nobody here has said it takes anything like that long.

i have another thread "On Premisis Exchange 2019 EAC text only, no pictures after CU9". In my case OWA opens, but EAC doesn't. i get the EAC logon page, and thin it presents as text only, and none of the links work. There is a screenshot attached over there.

Has anyone here had that exact issue?

0 Votes 0 ·
MikeGrant-5625 avatar image
3 Votes"
MikeGrant-5625 answered AlexanderKondrachuk-9716 published

I suspect this command:

Set-AuthConfig -NewCertificateThumbprint <ThumbprintFromStep1> -NewCertificateEffectiveDate (Get-Date)

Does not take the timezone into account, i'm in NZ with a +12 timezone and that's about how long it took after I ran the command for it to start working.

Some people are saying it worked immediately, some 1 hour and someone posted it took 4 hours for them. This may correlate to their timezone?

Maybe try:

$Time = Get-Date

Set-AuthConfig -NewCertificateThumbprint <ThumbprintFromStep1> -NewCertificateEffectiveDate $Time.ToUniversalTime()

· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Yes, I too think this is Time Zone related.

I tried using:

$Time = Get-Date

Set-AuthConfig -NewCertificateThumbprint <ThumbprintFromStep1> -NewCertificateEffectiveDate $Time.ToUniversalTime()

However I got:

You cannot call a method on a null-valued expression.
At line:1 char:1
+ Set-AuthConfig -NewCertificateThumbprint D2FB78A9D8BC8CFF37DAF5D8BD34AADB46EBABA ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull


I had this issue on 3 Servers all in UK. All 3 had to wait 1 hour after commands issued for it to take effect. I'm wondering iif this is because we are currently UTC +1 as its BST (British Summer Time). Would be interesting to know if this would have worked instantly if we was on UTC 0 when BST ends.

If anyone knows how to get this work work instantly in future would be appreicated.

0 Votes 0 ·

Change timezone on Exchange server to UTC.
open new Exchange powershell Window
Check time with get-date (Existing powershell windows will have old timezone)
Run commands to generate new certificate.
And restart IIS and Service
Set timezone back to what it should be.

Should now work.

Mike.

3 Votes 3 ·

IIt worked for me with the following steps (I'm in +1 timezone):


  • KeySize 2048 -PrivateKeyExportable $true -SubjectName "cn=Microsoft Exchange Server Auth Certificate" -FriendlyName "Microsoft Exchange Server Auth Certificate" -DomainName "domain.com"

  • set timezone to UTC

  • $Time = Get-Date

  • Set-AuthConfig -NewCertificateThumbprint <ThumbprintFromStep1> -NewCertificateEffectiveDate $Time.ToUniversalTime()

  • Set-AuthConfig -PublishCertificate

  • Set-AuthConfig -ClearPreviousCertificate

  • service restart (Microsoft Exchange Service Host )

  • Restart-WebAppPool MSExchangeOWAAppPool

  • Restart-WebAppPool MSExchangeECPAppPool

  • set timezone back

  • restart



1 Vote 1 ·
Show more comments

Have all 3 of my servers online now, will try it on any future ones though

0 Votes 0 ·
AshExchangeAdmin-1808 avatar image
0 Votes"
AshExchangeAdmin-1808 answered

Had the exact same issue with Exchange 2013 CU23 just last night. My Exchange Auth Certs had years left on them so proceeded with the patch installs on all servers and completed the AD Schema Update. Rebooted all servers and tested OWA and ECP access and could no longer access either of them. Log in pages would load fine but could not get passed sign in. Decided to renew the Exchange Auth certificates and recycled the App Pools as per the article but still couldn't get into OWA or ECP. Checked each Exchange sever with Get-AuthConfig to verify the new certificate had propagated to all Exchange servers and, based on its thumbprint, it had. ECP and OWA still weren't working. Waited an hour or so and tested both again and could then log in fine. No idea what happens in that 60mins which allows it to then start working, even though the new certificate had propagated to all Exchange servers within a few minutes. However, pleased to say, simply waiting did the trick for me so hope this helps someone else. Very annoying how this is documented as a "known issue" but ONLY when the existing Exchange Auth Certificate/s is expired. These patches are clearly breaking the existing certificate...

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MarcelLima-8684 avatar image
0 Votes"
MarcelLima-8684 answered

Just out of curiosity: I had this problem with a 2019 Exchange Server in Hybrid mode and another one but a fresh install (AD & exchange - lab demo). Both servers took hours to get it resolved but the last one as it is hosted in a Demo Environment I shutdown the whole network and on the next day the Exchange server was back on.
It seems like the waiting hours are not related with leaving the server on but something regarding the certificate publishing/date/time. Does anybody know what happens under the hood and could give an better explanation, rather than "In some environments, it may take an hour for the OAuth certificate to be published." from https://docs.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-oauth-expired ?

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.