Unable to expand legacy onPrem Public Folders with O365 account
I have been seeing a few scenarios lately where users who log into Exchange using an account other than the OnPrem User Principal Name (UPN) are running into various difficulties. Today we will focus on one of those scenarios. We will discuss how we figured it out and what needs to be done to get the desired functionality.
Scenario:
Tenant has an onPrem environment which consists of Exchange 2007 servers and two Exchange 2013 hybrid servers. The hybrid servers are set up to allow users to migrate to O365 and users in O365 to have the ability to access resources from the onPrem environment. The public folders are in the 2007 environment and were set up properly according to the following article:
https://technet.microsoft.com/en-us/library/dn249373(v=exchg.150).aspx
2007 and 2013 users have no issues accessing any of the legacy public folders, however, when an O365 user attempts to access the public folders, they are unable to expand them. They get the following error:
"Cannot expand the folder. Microsoft Exchange is not available. Either there are network problems or the Exchange server is down for maintenance."
How to Troubleshoot this issue:
The first thing you want to understand is what is happening with the requests when they leave Outlook and go into the Exchange environment. To do this, you will need the Fiddler tool. Fiddler can be downloaded from the following location:
Once Fiddler is installed, start it up and then reproduce the problem by starting Outlook with the O365 account and trying to expand public folders. We are looking for any 401 statuses against the Hybrid namespace when using the /rpc/rpcproxy.dll?servername.contoso.com:6001 URL. It should look similar to the following (minus the redactions, of course):
So what do we know so far:
- When we attempt to expand the legacy public folders, we are getting a 401 Unauthorized error. This means the account we are logging in with does not have permissions to access the public folder, or the account it is using is not recognized by the OnPrem active directory environment.
- We know the Unauthorized request is spawned by the Hybrid server as it tries to connect to the legacy folders (which is a netlogon connection).
- We know the users who are on OnPrem are not having this issue regardless of whether they are on 2007 or 2013. If 2013 users didn't work, we would suspect there was something wrong with the proxy request back to the 2007 public folders - possibly a MaxConcurrentAPI issue but in this scenario, that is not the case.
What don't we know yet?
Why exactly are we getting Unauthorized when our environment appears to be set up properly?
To answer this, we need more logging. In this case, we need the Netlogon logs on the Hybrid servers. By default, the Netlogon logs are disabled so we first need to enable them. To do that, simply fire up command prompt (as administrator) and run the following commands. Netlogon will be restarted in this process:
Nltest /DBFlag:2080FFFF<
net stop netlogon && net start netlogon
That full process is described in the following article:
Enabling debug logging for the Netlogon service
Now, with Netlogon logging enabled, we will reproduce the issue again and see what we see.
Within the Netlogon logs, we see the following error:
06/17 13:36:07 [CRITICAL] [5468] NlPrintRpcDebug: Couldn't get EEInfo for I_NetLogonSamLogonEx: 1761 (may be legitimate for 0xc0000064)
The 0xc0000064 error means that the account does not exist. Now why would AD think the account does not exist?
We looked at the account they were logging into Outlook with. It was user@contoso.com. We then looked at the OnPrem mail user for that account and it had a primary SMTP address of user@contoso.com, but its UPN was @contoso.org. When we tried to change the UPN onPrem to match the primary SMTP address, the UPN drop-down box did not have @contoso.com listed. This told me that no alternative UPN suffix was set up for that domain onPrem. They had it as an accepted domain so it could receive email, but not as an UPN so it couldn't be used for permissions onPrem, hence the 401 unauthorized.
To verify, we then went into Active Directory Domains and Trust and checked to see if it had any alternative UPN suffixes set. To do this, once in Active Directory Domains and Trusts, you right-click on Active Directory Domains and Trusts (you do not need to expand it), and then click on Properties. It should look as follows (no redaction needed in this screenshot as it is from my test lab):
The Fix:
As you can see, there are no Alternative UPN suffixes. We need to add the contoso.com domain to the Alternative UPN suffixes so the O365 user's onPrem mail user account can have it selected as their UPN.
Once the UPN was added to the Alternative UPN Suffixes, we waited for AD replication to complete. Then we went back into the user account onPrem and changed its UPN to match that of the Primary SMTP address which is what is used by the O365 user to log in. We waited for AD to replicate again.
After that, the O365 user was able to expand and use the legacy public folders.
Note:
Other issues can occur when the alternative UPN suffixes are not set with the remaining SMTP domains used by your organization. Lately, in addition to legacy public folder access from O365, we have seen the issue manifest when an application does authenticated relaying using the SMTP address of the service account. If that SMTP address does not have UPN suffix associated with it, it will reject the connection as unauthorized.
The following KB article discusses the two methods of changing the UPN on a user account. In our scenario, we went with option 2.
Hopefully, this article will be of use to organizations transitioning off of the legacy environments.
Thanks to Miguel Ortiz (miguelo) for helping me to isolate this issue via the Netlogon logging and to Matthew Huynh (mahuynh), Steven Cohen (stcohe) and Chris McGurk (cmcgurk) for helping me clean this article up.