question

Scryden-7312 avatar image
0 Votes"
Scryden-7312 asked LucasLiu-MSFT commented

Setup of Exchange 2013 CU23 is frozen on Mailbox Role: Transport Service

In lights of the Hafnium hack, I am upgrading our Exchange 2013 DAG cluster to 2013 CU23 from CU19 and then apply the security patch. However I was once again remembered today why I dislike upgrading Exchange so much...

The first server in the DAG upgraded flawlessly. Put it in maintenance mode, installed .NET 4.8 and C++ redist 2013, prepared AD schema etc. and then started the CU23 setup. Within an hour it was done installing and then I applied the Hafnium patch. Upon completion, I took it out of maintenance mode and it worked like a charm.

The second Exchange server in my DAG however is a completely different story. I followed the same procedure: Moved all mailbox databases to the first Exchange server, put the second exchange in maintenance mode, installed .NET 4.8 and c++ redist 2013, did a reboot of the server and then started the CU23 setup. The setup itself is not throwing any error messages, but it has been sitting in the same spot for the past 6 hours: Step 9 of 17: Mailbox Role: Transport Service: 54%. It does not go beyond the 54% mark.

Upon checking my task manager I did notice that .NET optimization service is consuming a lot of CPU. Upon checking the eventvwr, I noticed many warning messages Event ID: 1310

Event code: 3008
Event message: A configuration error has occurred.
Event time: 3/11/2021 5:15:03 PM
Event time (UTC): 3/12/2021 12:15:03 AM
Event ID: a4ad6357af7d4a979fd0f984e2812d96
Event sequence: 1
Event occurrence: 1
Event detail code: 0


Application information:
Application domain: /LM/W3SVC/1/ROOT/Autodiscover-21-132599817036183493
Trust level: Full
Application Virtual Path: /Autodiscover
Application Path: C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\Autodiscover\
Machine name: x


Process information:
Process ID: 4276
Process name: w3wp.exe
Account name: NT AUTHORITY\SYSTEM


Exception information:
Exception type: ConfigurationErrorsException
Exception message: Could not load file or assembly 'Microsoft.Exchange.Data.Directory, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified. (C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\Autodiscover\web.config line 39)
at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
at System.Web.Configuration.CompilationSection.LoadAssembly(AssemblyInfo ai)
at System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig)
at System.Web.Compilation.BuildManager.GetPreStartInitMethodsFromReferencedAssemblies()
at System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded)
at System.Web.Compilation.BuildManager.ExecutePreAppStart()
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)

Could not load file or assembly 'Microsoft.Exchange.Data.Directory, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection)
at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
at System.Reflection.Assembly.Load(String assemblyString)
at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)

Request information:
Request URL: https://x:443/autodiscover/autodiscover.xml
Request path: /autodiscover/autodiscover.xml
User host address: x
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\SYSTEM


Thread information:
Thread ID: 13
Thread account name: NT AUTHORITY\SYSTEM
Is impersonating: False
Stack trace: at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
at System.Web.Configuration.CompilationSection.LoadAssembly(AssemblyInfo ai)
at System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig)
at System.Web.Compilation.BuildManager.GetPreStartInitMethodsFromReferencedAssemblies()
at System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded)
at System.Web.Compilation.BuildManager.ExecutePreAppStart()
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)


this is only one example. I have a whole bunch of these messages in my eventviewer complaining about could not load file or assembly. All of them of which have issues in the httpproxy subfolders. I already tried many different things including copying over the SharedWebConfig from my working Exchange server into this HttpProxy folder but to no avail. The setup is just stuck and won't report anything other than the warnings in eventviewer. There are also no Exchange setup logs which makes it even more difficult to say anything.

Anyone else encountered this problem? Client is running fine on one Exchange server now but we need to the second server up and running again.

office-exchange-server-administrationoffice-exchange-server-itprooffice-exchange-server-deployment
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

LucasLiu-MSFT avatar image
0 Votes"
LucasLiu-MSFT answered

Hi @Scryden-7312 ,
According to research on the error information, I think copying the SharedWebCongif from another Exchange server is a feasible solution. But please noted the following points:
1.The SharedWebConfig file must be copied from a server running the same Cumulative Update. I noted that you have an Exchange that has been successfully upgraded, so please copying the file from the Exchange that is still CU19 version.

2.If the server you pull this file from has a different install path for Exchange you will need to do a find/replace of all paths inside the SharedWebConfig file and update with the destination server’s install path.

3.After copied, please run the iisreset in the Run start as Administrator to restart the IIS.

In addition, according to the following error. Under the the path(C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\Autodiscover). You could try to rename the web.config file to web.config.old, then rename the web.config.bak file to replace it. Restart the IIS and try to upgrade Exchange server again.

Exception message: Could not load file or assembly 'Microsoft.Exchange.Data.Directory, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified. (C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\Autodiscover\web.config line 39)



If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.




5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Scryden-7312 avatar image
0 Votes"
Scryden-7312 answered LucasLiu-MSFT commented

Thanks for the response. After some thorough checks we actually found out that this server did get compromised by the Hafnium hack. I immediately put the server into maintenance mode in the DAG cluster and disconnected it from the network. Our first Exchange server that is already fully upgraded is confirmed clean.

I have no intentions of ever bringing this server back online, so I already deployed a brand new VM that is fully patched and updated but I am looking for a proper procedure to recover a DAG member node and I can't seem to find a unanimous answer to this. I found that by pulling it offline like that, the server is basically considered "crashed" and I will have to manually evict it from the DAG and reset the AD computer account. Then if I name the new server the exact same, I can install Exchange 2013 CU23 with the /m:recoverserver flag and all settings will be restored from AD.

Is this the correct procedure or did I forget anything? The newly installed VM holds no mailbox database copies yet. Once I add it back to the DAG, will server 1 automatically copy over all Exchange settings and databases to the second Exchange server? I understand that I will have to copy over a .pfx export of our SSL certificate manually, but other than that, is there anything else I need to do manually?

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @Scryden-7312 ,
According to your description, has the machine where the Exchange server is located is no longer working? If so, you could following the specific steps in the below link to recover the database availability group member server.
Please refer to: Recover a database availability group member server

It should be noted that this process has very high requirements for the matching of the target server and the source server. You can also refer to: Recover Exchange servers



If the response is helpful, please click "Accept Answer" and upvote it.



0 Votes 0 ·

Hi @LucasLiu-MSFT ,

Yes indeed. The server failed. I have a VM that is the same operating system and with the same hostname as the old one ready. I will also give it the same IP addresses as the old one. I read the article you linked and I think I got all bases covered. However since I've never done this before I was more looking for someone who actually has done this before confirm this is the right procedure.

I opened a case with Microsoft and paid for 24x7 support but now 1 week later they still didn't call me back because their Exchange team is swamped with Hafnium support calls. However I can't wait forever.

0 Votes 0 ·

Hi @Scryden-7312 ,
Your approach is correct. I have encountered some users successfully recovering the server using the above method, but the actual restoration process is very complicated, and unexpected errors may occur. It is recommended that you do it with the assistance of a professional engineer.



If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.


0 Votes 0 ·
Scryden-7312 avatar image
1 Vote"
Scryden-7312 answered LucasLiu-MSFT commented

For anyone who is wondering or is in the same situation, I went through the Microsoft article today with a Microsoft tech: https://docs.microsoft.com/en-us/exchange/recover-a-database-availability-group-member-server-exchange-2013-help

It was literally as easy and straight forward as that.

  1. Deploy a new VM with the same OS, same computer name, same IP addresses and same hard drives

  2. Remove database copies from old server

  3. Remove old server from DAG

  4. Evict old server from Failover cluster

  5. Reset computer account in AD

  6. Join new server to AD domain with the exact same computer name as the old server you're replacing

  7. Install Exchange with the /m:RecoverServer switch

  8. When done, reconfigure your External URLs for virtual directories and reimport your SSL certificate

  9. Add new server to DAG cluster

  10. Add mailbox database copies to new server and let it seed

It was as simple as that! Big relief.



· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @Scryden-7312 ,
Just like the link I provided above, I’m glad it can help you. Thank you very much for the steps you summarized. It is very detailed and I believe it can help more users who encounter the same problem.
If you can, you can mark it as an answer, which makes the answer easier to search for by other users.



If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.





0 Votes 0 ·