In lights of the Hafnium hack, I am upgrading our Exchange 2013 DAG cluster to 2013 CU23 from CU19 and then apply the security patch. However I was once again remembered today why I dislike upgrading Exchange so much...
The first server in the DAG upgraded flawlessly. Put it in maintenance mode, installed .NET 4.8 and C++ redist 2013, prepared AD schema etc. and then started the CU23 setup. Within an hour it was done installing and then I applied the Hafnium patch. Upon completion, I took it out of maintenance mode and it worked like a charm.
The second Exchange server in my DAG however is a completely different story. I followed the same procedure: Moved all mailbox databases to the first Exchange server, put the second exchange in maintenance mode, installed .NET 4.8 and c++ redist 2013, did a reboot of the server and then started the CU23 setup. The setup itself is not throwing any error messages, but it has been sitting in the same spot for the past 6 hours: Step 9 of 17: Mailbox Role: Transport Service: 54%. It does not go beyond the 54% mark.
Upon checking my task manager I did notice that .NET optimization service is consuming a lot of CPU. Upon checking the eventvwr, I noticed many warning messages Event ID: 1310
Event code: 3008
Event message: A configuration error has occurred.
Event time: 3/11/2021 5:15:03 PM
Event time (UTC): 3/12/2021 12:15:03 AM
Event ID: a4ad6357af7d4a979fd0f984e2812d96
Event sequence: 1
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/1/ROOT/Autodiscover-21-132599817036183493
Trust level: Full
Application Virtual Path: /Autodiscover
Application Path: C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\Autodiscover\
Machine name: x
Process information:
Process ID: 4276
Process name: w3wp.exe
Account name: NT AUTHORITY\SYSTEM
Exception information:
Exception type: ConfigurationErrorsException
Exception message: Could not load file or assembly 'Microsoft.Exchange.Data.Directory, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified. (C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\Autodiscover\web.config line 39)
at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
at System.Web.Configuration.CompilationSection.LoadAssembly(AssemblyInfo ai)
at System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig)
at System.Web.Compilation.BuildManager.GetPreStartInitMethodsFromReferencedAssemblies()
at System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded)
at System.Web.Compilation.BuildManager.ExecutePreAppStart()
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)
Could not load file or assembly 'Microsoft.Exchange.Data.Directory, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection)
at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
at System.Reflection.Assembly.Load(String assemblyString)
at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
Request information:
Request URL: https://x:443/autodiscover/autodiscover.xml
Request path: /autodiscover/autodiscover.xml
User host address: x
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\SYSTEM
Thread information:
Thread ID: 13
Thread account name: NT AUTHORITY\SYSTEM
Is impersonating: False
Stack trace: at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
at System.Web.Configuration.CompilationSection.LoadAssembly(AssemblyInfo ai)
at System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig)
at System.Web.Compilation.BuildManager.GetPreStartInitMethodsFromReferencedAssemblies()
at System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded)
at System.Web.Compilation.BuildManager.ExecutePreAppStart()
at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)
this is only one example. I have a whole bunch of these messages in my eventviewer complaining about could not load file or assembly. All of them of which have issues in the httpproxy subfolders. I already tried many different things including copying over the SharedWebConfig from my working Exchange server into this HttpProxy folder but to no avail. The setup is just stuck and won't report anything other than the warnings in eventviewer. There are also no Exchange setup logs which makes it even more difficult to say anything.
Anyone else encountered this problem? Client is running fine on one Exchange server now but we need to the second server up and running again.