question

JohnRuddy-9997 avatar image
0 Votes"
JohnRuddy-9997 asked YukiSun-MSFT commented

Exchange 2010 / 2016 mixed - OWA/ECP invalid canary

Hi

Already posted a couple of questions regarding our Exchange 2010 -> 2016 migration and here's another.

Exchange 2010 SP3 with Rollup 32. Exchange 2016 CU 20

Mail flow is now via the 2016 server.

Staff and IT use OWA to set out of office messages for themselves and other staff.

They are now getting the "Invalid Canary" message when trying to do this. I've seen references to this from years ago which say it's fixed in 2010 SP3, but we already have this installed.

I have also found this link where someone says they have just started getting the issue after installing the recent 2021 Exchange security updates. Our 2016 is a fresh install so would have had these updates installed already.

https://practical365.com/microsoft-issues-critical-security-updates-for-exchange-server/

The second comment on the above link is the same setup as us and no-one replied to the person's query.

I have checked the case of both the owa and ecp URLs and they are all now lowercase. I have also rebooted both servers with no success.

If I change the link in the browser for owa/ecp to the local 2010 servername we don't then get the error. But we can't get staff to do this as they know the mail server by external url.

This could end up being a major issue so I'm hoping there is a fix for this.

Thanks







office-exchange-server-administrationoffice-exchange-server-mailflowoffice-exchange-server-connectivityoffice-exchange-hybrid-itprooffice-exchange-server-itpro
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

YukiSun-MSFT avatar image
0 Votes"
YukiSun-MSFT answered

Hi @JohnRuddy-9997,

Before going futher, in order to view the build number and check if all the security updates for your Exchange server have been applied, you can follow this link, download and run the latest version of HealthChecker script.

Besides, according to your description, this is only affecting user mailboxes still resides on Exchange 2010, right? If this is the case, I'd suggest moving one of the affected mailbox to Exchange 2016 and see the result. If it works after moving to Exchange 2016, considering that Exchange 2010 has already reached the end of support, I'd recommend speeding up the migration process and proceeding to move all the remaining mailboxes to Exchange 2016 to avoid the "Invalid Canary" issue.

In case currently you are not able to move all mailboxes to Exchange 2016, then please collect the following informaiton for further troubleshooting:

  1. If possible, could you remove all sensitive informaiton involved and share a screenshot or the detailed error message of "Invalid Canary" so that we can see if more clues can be found?

  2. When the error occurs, check the Event Viewer on the Exchange 2010 server and see if there are any relevant events recorded out there.

  3. Try testing with different browers and also check it by opening the browser in private mode to see if there's any difference.

Any findings, feel free to post back.


If an Answer is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

JohnRuddy-9997 avatar image
0 Votes"
JohnRuddy-9997 answered YukiSun-MSFT commented

105782-invalid-canary.jpg




Hi
I have uploaded a screenshot of the error. It happens when you click Save when making any changes to the Out of Office message.

The 2010 server seems to give 3 event messages - 39, 38 & 4

Event 39
Current User: '"domain.name/User1" on behalf of "domain.name/user2"'
Unique Key: 'S-1-5-21-545268291-1463314663-1478062314-4000'
Cookie Name: 'msExchEcpCanary'
Exchange Control Panel detected an invalid canary in cookie from request for URL 'https://exch2010-server/ecp/Organize/AutomaticReplies.svc/SetObject?msExchEcpCanary=foItQpLsFkCaujUC3CM6rp_2mR-KH9kIk_tefe4gFw7EgI9GCSR83bsKcGkTu1kzaSxqZrWMsUQ.'.
Canary in cookie: 'foItQpLsFkCaujUC3CM6rp_2mR-KH9kIk_tefe4gFw7EgI9GCSR83bsKcGkTu1kzaSxqZrWMsUQ.'.
Reset canary cookie for user

Event 38
Current User: '" domain.name/User1" on behalf of " domain.name/user2"'
Exchange Control Panel detected an invalid canary from request for URL 'https://exch2010-server/ecp/Organize/AutomaticReplies.svc/SetObject'.
Canary in cookie: '6dK4SoN_UUuszeM-oISiMIi0nw9uMdkITfUU6lnNx23yN8HqvIb7KNuCB_g4paYgJOQXVmzN8uk.' mismatch with canary in header/form: ', in URL '.

Event 4
Current user: '" domain.name/User1" on behalf of "domain.name/user2"'
Request for URL 'https:// exch2010-server/ecp/Organize/AutomaticReplies.svc/SetObject' failed with the following error:
System.ServiceModel.FaultException: Invalid Canary
at Microsoft.Exchange.Management.ControlPanel.RbacModule.Application_PostAuthenticateRequest(Object sender, EventArgs e)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)


The process seems to work, for me, using Internet Explorer but not Chrome or Edge. While using I.E. might be a workaround, we don't really want to encourage people to be using I.E. so we'd rather get a proper fix for the issue.

Agree that migrating all the users is likely to get round the issue but there are hundreds of mailboxes and I'm not happy just to rush through them all without mitigating any problems as we go.

Thanks


invalid-canary.jpg (49.5 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @JohnRuddy-9997,

Have you tried downloading the latest version of Chrome or Edge to check how it goes?
If changing the link in the browser to the local 2016 servername, will the error persist?

Supposing the issue can be reproduced when using the local 2016 servername in the link, seems to me that it could be related to the Canary Data on Exchange 2016. Please follow the steps below to clear the CanaryData on Exchange 2016 servers after backing up and see if it helps:

  1. Open ADSIedit, connect to Configuration, expand and highlight the object “Client Access”, right click it, choose “properties”.

  2. Click on each key(MSExchCanaryData0, MSExchCanaryData1, MSExchCanaryData2), click "Edit", back up the values, then click "Clear":
    106115-1.jpeg
    106123-2.jpeg

  3. Recycle the Applicationpools for OWA and ECP.

0 Votes 0 ·
1.jpeg (140.3 KiB)
2.jpeg (54.6 KiB)

Hi @JohnRuddy-9997,

I am writing to see if there is any progress on the issue. Have you had a chance to try clearing the CanaryData to see the result?


If an Answer is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

0 Votes 0 ·
JeffLanni-7267 avatar image
0 Votes"
JeffLanni-7267 answered YukiSun-MSFT commented

@JohnRuddy-9997, Were you able to find a permanent fix to this issue or have you decided to proceed to migrate your users without a fix in the current environment?

@YukiSun-MSFT, I tried the process you laid out in your message above and it did not help fix the issue. We also see the workaround succeed using IE, but would like to avoid that as a solution as well.

Could it potentially be something in the Chrome/Edge Browsers that is causing the issue?

Thanks for any suggestion or help you can provide.

Jeff

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi

I haven't found a solution to this yet. But I haven't tried the adsiedit thing yet - I'm always reluctant to do this type of thing unless I know it's definitely required. But I will try if I get the chance. But from what you are saying, JeffLanni, it didn't fix it for you anyway.

I've advised people to use I.E. for now but I think it's only a matter of time before someone higher up the food chain wants a permanent solution.

As you say, there's obviously something with Chrome/Edge, even if it's just a setting somewhere. I've just tried Firefox and it seemed to work but I don't know if the issue is when you've used a browser to access OWA previously which causes the problem. I haven't used Firefox for our webmail before. Also we don't use Firefox in our work environment so this wouldn't be a solution anyway.

Thanks

0 Votes 0 ·

Hi @JeffLanni-7267 and @JohnRuddy-9997,

I personally agree with you guys on that it might be something with the Chrome/Edge Browsers. Also during my research, I've see similar situation occurred in Chrome which could be worked around by disabling the flag "Cookies without SameSite must be secure". But this flag is not available when I checked the current version of Chrome on my end(Version 91), so I didn't mention this in my earlier post.

Then I searched further on that flag and found the following article which could be relevant:
Effect on customer websites and Microsoft services and products in Chrome version 80 or later
From the Recommendations section in the article, as for Exchange products, updates are only available to Exchange 2016 and Exchange 2019, and I am thinking maybe this could have something to do with the issue which affects to mailboxes on Exchange 2010.


If an Answer is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

0 Votes 0 ·
JohnRuddy-9997 avatar image
1 Vote"
JohnRuddy-9997 answered JohnRuddy-9997 rolled back

Hi

I did some testing with Chrome in a VM. I downloaded a couple of older versions of Chrome (7.6 & 7.9) plus v8.1

When I was running 7.6 & 7.9 I could do what I needed to in OWA. As soon as I updated to 8.1 the invalid canary message appeared. I would think that now that Edge is based on the Chrome engine, this is why it will have the same problem.

I then found this page:

https://piunikaweb.com/2021/06/14/google-chrome-flags-for-samesite-cookies-taken-away-after-update-v91/

Part way down it states:

"Those who wish to disable the said SameSite flags can do so by adding –disable-features=SameSiteByDefaultCookies or –disable-features=CookieswithoutSameSitemustbesecure in the Target field of the Google Chrome or Microsoft Edge properties and restart the web browser."

I tried this and it does seem to work with both Chrome and Edge. This is workable for myself and other admins but not suitable for end users. They'd be as well sticking to I.E. so long as it keeps working.

Unfortunately it also states further down that page:

"However, this workaround will only work until the Google Chrome 94 update as the said command line flags will be removed after that."

I don't know if it means the flags setting won't take effect after this or it just means the setting would be removed from the command line. I think it's probably the former so this fix would no longer work.

We're not the only ones to have this issue so I wish Microsoft would come up with a proper solution.

I know I could probably log a call with MS but the cost is probably prohibitive - don't know how much support calls cost these days.

Thanks



5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

JohnRuddy-9997 avatar image
1 Vote"
JohnRuddy-9997 answered JohnRuddy-9997 edited

Hi
OK - I've done a bit more digging and found various references to the sitecookie behaviour setting in Chrome/Edge.

https://admx.help/?Category=Chrome&Policy=Google.Policies.Chrome::LegacySameSiteCookieBehaviorEnabled

Also this one - but I haven't yet worked out the format for this one to enter specific domains, which I think would be better for what we need.

https://admx.help/?Category=Chrome&Policy=Google.Policies.Chrome::LegacySameSiteCookieBehaviorEnabledForDomainList

The first link above I followed and edited the registry on my own pc and created the registry entry in the screenshot.

114311-image.png


This actually seems to work so I'm hoping I might be able to create a GPO with this or preferably the entry which allows using a specific domain.

I'll keep testing but hope this helps someone if it works.

Thanks



image.png (13.7 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

JohnRuddy-9997 avatar image
1 Vote"
JohnRuddy-9997 answered YukiSun-MSFT commented

Hi

Further update.

Looks like both the settings work for Chrome and Edge and it should hopefully, in theory, be a matter of implementing either of these in a GPO rather than manually using regedit.

The format for the domain specific setting is shown in this link:

https://docs.microsoft.com/en-us/deployedge/microsoft-edge-policies#legacysamesitecookiebehaviorenabledfordomainlist

SOFTWARE\Policies\Microsoft\Edge\LegacySameSiteCookieBehaviorEnabledForDomainList\1 = "www.example.com"
SOFTWARE\Policies\Microsoft\Edge\LegacySameSiteCookieBehaviorEnabledForDomainList\2 = "[*.]example.edu"

I used the [*.]domain-name.com option and it seemed to work.

On Google's site it mentions that support for this setting is likely to end at some point but I'm hoping it will work long enough for me to migrate our mailboxes.

Again, hope this helps someone who might have the same issues.

Thanks


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @JohnRuddy-9997,

Great to see that it seemed to work now and really appreciate it for your valuable sharing! Thanks!

0 Votes 0 ·