Exchange 2016 CU 17 .NET 4.8 Errors after restore

Charlie Sciuto 1 Reputation point
2021-03-09T02:58:41.523+00:00

A client of ours was hit by HAFNIUM on their Exchange 2016 box. We were able to restore from a snap from 2/28 and the system seemed to be functional. Some details: Server 2012 R2 OS with latest critical OS updates applied, Exchange 2016 CU17 currently on the machine. All services start without issue. So, went to apply CU19 prior to the HAFNIUM patch. Ran the usual without issue:

Setup.exe /IAcceptExchangeServerLicenseTerms /PrepareSchema

Setup.exe /IAcceptExchangeServerLicenseTerms /PrepareAD

Setup.exe /IAcceptExchangeServerLicenseTerms /PrepareAllDomains

When it came to run Setup.exe /IAcceptExchangeServerLicenseTerms /Mode:Upgrade, it failed almost immediately:

[03/08/2021 15:24:44.0052] [0] The locally installed version is 15.1.2044.4.
[03/08/2021 15:24:44.0053] [0] Exchange Installation Directory : 'D:\Program Files\Microsoft\Exchange Server\V15'.
[03/08/2021 15:24:44.0121] [0] Setup is determining what organization-level operations to perform.
[03/08/2021 15:24:44.0122] [0] Because the value was specified, setup is setting the argument OrganizationName to the value First Organization.
[03/08/2021 15:24:44.0158] [0] RootDataHandler has 1 DataHandlers
[03/08/2021 15:24:44.0158] [0] Languages
[03/08/2021 15:24:44.0159] [0] Management tools
[03/08/2021 15:24:44.0159] [0] Mailbox role: Transport service
[03/08/2021 15:24:44.0160] [0] Mailbox role: Client Access service
[03/08/2021 15:24:44.0161] [0] Mailbox role: Unified Messaging service
[03/08/2021 15:24:44.0162] [0] Mailbox role: Mailbox service
[03/08/2021 15:24:44.0163] [0] Mailbox role: Front End Transport service
[03/08/2021 15:24:44.0163] [0] Mailbox role: Client Access Front End service
[03/08/2021 15:24:44.0176] [0] Validating options for the 7 requested roles
[03/08/2021 15:24:44.0178] [0] UpgradeModeDataHandler has 17 handlers and 17 work units
[03/08/2021 15:24:44.0233] [0] Performing Microsoft Exchange Server Prerequisite Check
[03/08/2021 15:24:44.0371] [0] [ERROR] Exception has been thrown by the target of an invocation.
[03/08/2021 15:24:44.0385] [0] [ERROR] Could not load file or assembly 'System.Security, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The module was expected to contain an assembly manifest.
[03/08/2021 15:24:44.0386] [0] CurrentResult SetupLauncherHelper.loadassembly:444: 1
[03/08/2021 15:24:44.0389] [0] The Exchange Server setup operation didn't complete. More details can be found in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
[03/08/2021 15:24:44.0389] [0] CurrentResult main.run:235: 1 [03/08/2021 15:24:44.0390] [0] CurrentResult setupbase.maincore:396: 1
[03/08/2021 15:24:44.0391] [0] End of Setup [03/08/2021 15:24:44.0391] [0] **********************************************

Looked to be a .NET issue. Ran the netfxrepairtool.exe and it stated it repaired the issues with .NET 4.8, but when I try to either install or remove that update (KB4486105), the install will state .NET 4.8 is already installed or the removal will state the following:

![75520-image.png][1]

CBS.log is riddled with errors :

2021-03-08 20:41:36, Error CSI 00000012 (F) STATUS_OBJECT_NAME_NOT_FOUND #2937274# from Windows::Rtl::SystemImplementation::DirectFileSystemProvider::SysCreateFile(flags = (AllowSharingViolation), handle = {provider=NULL, handle=0, name= ("null")}, da = (SYNCHRONIZE|FILE_READ_ATTRIBUTES), oa = @0x92146cd3d8->OBJECT_ATTRIBUTES {s:48; rd:NULL; on:[116]"\??\C:\Windows\WinSxS\msil_system.security_b03f5f7f11d50a3a_4.0.9664.17143_none_6c8461fc647f6f2b\System.Security.dll"; a:(OBJ_CASE_INSENSITIVE)}, iosb = @0x92146cd3b8, as = (null), fa = 0, sa = (FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE), cd = FILE_OPEN, co = (FILE_SYNCHRONOUS_IO_NONALERT|0x00004000), eab = NULL, eal = 0, disp = Invalid)
[gle=0xd0000034]
2021-03-08 20:41:36, Error CSI 00000013@2021/3/9:02:41:36.805 (F) base\wcp\sil\merged\ntu\ntsystem.cpp(2155): Error STATUS_OBJECT_NAME_NOT_FOUND originated in function Windows::Rtl::SystemImplementation::DirectFileSystemProvider::SysCreateFile expression: (null)
[gle=0x80004005]
2021-03-08 20:41:36, Info CBS Added C:\Windows\Logs\CBS\CBS.log to WER report.
2021-03-08 20:41:36, Info CBS Added C:\Windows\Logs\CBS\CbsPersist_20210309024007.log to WER report.
2021-03-08 20:41:36, Info CBS Added C:\Windows\Logs\CBS\CbsPersist_20210308143307.log to WER report.
2021-03-08 20:41:36, Info CBS Added C:\Windows\Logs\CBS\CbsPersist_20210217161217.cab to WER report.
2021-03-08 20:41:36, Info CBS Added C:\Windows\Logs\CBS\CbsPersist_20210214102212.cab to WER report.
2021-03-08 20:41:36, Info CBS Added C:\Windows\Logs\CBS\CbsPersist_20210214082735.cab to WER report.
2021-03-08 20:41:36, Info CBS Not able to add pending.xml to Windows Error Report. [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]
2021-03-08 20:41:36, Info CBS Not able to add pending.xml.bad to Windows Error Report. [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]
2021-03-08 20:41:36, Info CBS Not able to add SCM.EVM to Windows Error Report. [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]
2021-03-08 20:42:16, Error CSI 00000014 (F) STATUS_OBJECT_NAME_NOT_FOUND #2937273# from Windows::Rtl::SystemImplementation::CDirectory::OpenExistingFile(...)[gle=0xd0000034]
2021-03-08 20:42:16, Error CSI 00000015 (F) STATUS_OBJECT_NAME_NOT_FOUND #2937272# from Windows::Rtl::SystemImplementation::CDirectory_IRtlDirectoryTearoff::OpenExistingFile(flags = 0, da = (FILE_GENERIC_READ), oa = @0x92146cdeb8->SIL_OBJECT_ATTRIBUTES {s:40; on:"System.Security.dll"; a:(OBJ_CASE_INSENSITIVE)}, sa = (FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE), oo = (FILE_SYNCHRONOUS_IO_NONALERT|FILE_NON_DIRECTORY_FILE|FILE_OPEN_FOR_BACKUP_INTENT), file = NULL, disp = (null))
[gle=0xd0000034]
2021-03-08 20:42:16, Error CSI 00000016 (F) STATUS_OBJECT_NAME_NOT_FOUND #2937035# from CGacPrimitiveInstaller::ProcessComponentChange(...)[gle=0xd0000034]
2021-03-08 20:42:16, Error CSI 00000017 (F) STATUS_OBJECT_NAME_NOT_FOUND #2936903# from PrimitiveInstaller::CCoordinator::ProcessChange(Change 4.0.9696.17061 -> 4.0.9664.17143 for System.Security, pA = PROCESSOR_ARCHITECTURE_MSIL (8), Culture neutral, VersionScope neutral, PublicKeyToken = {l:8 b:b03f5f7f11d50a3a}, Type neutral, TypeName neutral, PublicKey neutral)[gle=0xd0000034]
2021-03-08 20:42:16, Error CSI 00000018 (F) STATUS_OBJECT_NAME_NOT_FOUND #1382625# from CCSDirectTransaction::PerformChangeAnalysis(...)[gle=0xd0000034]
2021-03-08 20:42:16, Error CSI 00000019 (F) STATUS_OBJECT_NAME_NOT_FOUND #1382624# from CCSDirectTransaction::PrepareForCommit(...)[gle=0xd0000034]
2021-03-08 20:42:16, Error CSI 0000001a (F) STATUS_OBJECT_NAME_NOT_FOUND #1382623# from CCSDirectTransaction::ExamineTransaction(...)[gle=0xd0000034]
2021-03-08 20:42:16, Error CSI 0000001b (F) STATUS_OBJECT_NAME_NOT_FOUND #1382622# from CCSDirectTransaction_IRtlTransaction::ExamineTransaction(...)[gle=0xd0000034]
2021-03-08 20:42:17, Info CSI 0000001c Delete of a missing registry value detected in the registry installer!!
Registry value name: [ml:142{71},l:96{48}]"System.AppContext,4.0.0.0,,b03f5f7f11d50a3a,msil".

2021-03-08 20:42:17, Info CSI 0000001d Delete of a missing registry value detected in the registry installer!!
Registry value name: [ml:192{96},l:146{73}]"System.Diagnostics.TextWriterTraceListener,4.0.0.0,,b03f5f7f11d50a3a,msil".

2021-03-08 20:42:17, Info CSI 0000001e Delete of a missing registry value detected in the registry installer!!
Registry value name: [ml:154{77},l:108{54}]"System.Threading.Thread,4.0.0.0,,b03f5f7f11d50a3a,msil".

2021-03-08 20:42:17, Info CSI 0000001f Delete of a missing registry value detected in the registry installer!!
Registry value name: [ml:186{93},l:140{70}]"System.Security.Cryptography.Algorithms,4.0.0.0,,b03f5f7f11d50a3a,msil".

2021-03-08 20:42:17, Info CSI 00000020 Delete of a missing registry value detected in the registry installer!!
Registry value name: [ml:206{103},l:160{80}]"System.Runtime.InteropServices.RuntimeInformation,4.0.0.0,,b03f5f7f11d50a3a,msil".

2021-03-08 20:42:17, Info CSI 00000021 Delete of a missing registry value detected in the registry installer!!
Registry value name: [ml:148{74},l:102{51}]"System.IO.FileSystem,4.0.0.0,,b03f5f7f11d50a3a,msil".

2021-03-08 20:42:17, Info CSI 00000022 Delete of a missing registry value detected in the registry installer!!
Registry value name: [ml:172{86},l:126{63}]"System.ComponentModel.Primitives,4.0.0.0,,b03f5f7f11d50a3a,msil".

2021-03-08 20:42:17, Info CSI 00000023 Delete of a missing registry value detected in the registry installer!!
Registry value name: [ml:176{88},l:130{65}]"System.Diagnostics.FileVersionInfo,4.0.0.0,,b03f5f7f11d50a3a,msil".

2021-03-08 20:42:17, Info CSI 00000024 Delete of a missing registry value detected in the registry installer!!
Registry value name: [ml:168{84},l:122{61}]"System.Diagnostics.TraceSource,4.0.0.0,,b03f5f7f11d50a3a,msil".

2021-03-08 20:42:17, Info CSI 00000025 Delete of a missing registry value detected in the registry installer!!
Registry value name: [ml:168{84},l:122{61}]"System.Globalization.Calendars,4.0.0.0,,b03f5f7f11d50a3a,msil".

2021-03-08 20:42:17, Info CSI 00000026 Delete of a missing registry value detected in the registry installer!!
Registry value name: [ml:158{79},l:112{56}]"System.IO.IsolatedStorage,4.0.0.0,,b03f5f7f11d50a3a,msil".

2021-03-08 20:42:17, Info CSI 00000027 Delete of a missing registry value detected in the registry installer!!
Registry value name: [ml:170{85},l:124{62}]"System.Globalization.Extensions,4.0.0.0,,b03f5f7f11d50a3a,msil".

2021-03-08 20:42:17, Info CSI 00000028 Delete of a missing registry value detected in the registry installer!!
Registry value name: [ml:138{69},l:92{46}]"System.Security,4.0.0.0,,b03f5f7f11d50a3a,msil".

2021-03-08 20:42:17, Error CSI 00000029 (F) STATUS_OBJECT_NAME_NOT_FOUND #3065335# from Windows::Rtl::SystemImplementation::DirectFileSystemProvider::SysCreateFile(flags = (AllowSharingViolation), handle = {provider=NULL, handle=0, name= ("null")}, da = (SYNCHRONIZE|FILE_READ_ATTRIBUTES), oa = @0x92146ccf48->OBJECT_ATTRIBUTES {s:48; rd:NULL; on:[116]"\??\C:\Windows\WinSxS\msil_system.security_b03f5f7f11d50a3a_4.0.9664.17143_none_6c8461fc647f6f2b\System.Security.dll"; a:(OBJ_CASE_INSENSITIVE)}, iosb = @0x92146ccf28, as = (null), fa = 0, sa = (FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE), cd = FILE_OPEN, co = (FILE_SYNCHRONOUS_IO_NONALERT|0x00004000), eab = NULL, eal = 0, disp = Invalid)
[gle=0xd0000034]
2021-03-08 20:42:17, Error CSI 0000002a@2021/3/9:02:42:17.633 (F) base\wcp\sil\merged\ntu\ntsystem.cpp(2155): Error STATUS_OBJECT_NAME_NOT_FOUND originated in function Windows::Rtl::SystemImplementation::DirectFileSystemProvider::SysCreateFile expression: (null)
[gle=0x80004005]
2021-03-08 20:42:17, Info CBS Added C:\Windows\Logs\CBS\CBS.log to WER report.
2021-03-08 20:42:17, Info CBS Added C:\Windows\Logs\CBS\CbsPersist_20210309024007.log to WER report.
2021-03-08 20:42:17, Info CBS Added C:\Windows\Logs\CBS\CbsPersist_20210308143307.log to WER report.
2021-03-08 20:42:17, Info CBS Added C:\Windows\Logs\CBS\CbsPersist_20210217161217.cab to WER report.
2021-03-08 20:42:17, Info CBS Added C:\Windows\Logs\CBS\CbsPersist_20210214102212.cab to WER report.
2021-03-08 20:42:17, Info CBS Added C:\Windows\Logs\CBS\CbsPersist_20210214082735.cab to WER report.
2021-03-08 20:42:17, Info CBS Not able to add pending.xml to Windows Error Report. [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]
2021-03-08 20:42:17, Info CBS Not able to add pending.xml.bad to Windows Error Report. [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]
2021-03-08 20:42:17, Info CBS Not able to add SCM.EVM to Windows Error Report. [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]
2021-03-08 20:42:23, Error CSI 0000002b (F) STATUS_OBJECT_NAME_NOT_FOUND #3065334# from Windows::Rtl::SystemImplementation::CDirectory::OpenExistingFile(...)[gle=0xd0000034]
2021-03-08 20:42:23, Error CSI 0000002c (F) STATUS_OBJECT_NAME_NOT_FOUND #3065333# from Windows::Rtl::SystemImplementation::CDirectory_IRtlDirectoryTearoff::OpenExistingFile(flags = 0, da = (FILE_GENERIC_READ), oa = @0x92146cda28->SIL_OBJECT_ATTRIBUTES {s:40; on:"System.Security.dll"; a:(OBJ_CASE_INSENSITIVE)}, sa = (FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE), oo = (FILE_SYNCHRONOUS_IO_NONALERT|FILE_NON_DIRECTORY_FILE|FILE_OPEN_FOR_BACKUP_INTENT), file = NULL, disp = (null))
[gle=0xd0000034]
2021-03-08 20:42:23, Error CSI 0000002d (F) STATUS_OBJECT_NAME_NOT_FOUND #3064965# from CGacPrimitiveInstaller::ProcessComponentChange(...)[gle=0xd0000034]
2021-03-08 20:42:23, Error CSI 0000002e (F) STATUS_OBJECT_NAME_NOT_FOUND #3064833# from PrimitiveInstaller::CCoordinator::ProcessChange(Change 4.0.9696.17061 -> 4.0.9664.17143 for System.Security, pA = PROCESSOR_ARCHITECTURE_MSIL (8), Culture neutral, VersionScope neutral, PublicKeyToken = {l:8 b:b03f5f7f11d50a3a}, Type neutral, TypeName neutral, PublicKey neutral)[gle=0xd0000034]
2021-03-08 20:42:23, Error CSI 0000002f (F) STATUS_OBJECT_NAME_NOT_FOUND #2937279# from CCSDirectTransaction::PerformChangeAnalysis(...)[gle=0xd0000034]
2021-03-08 20:42:23, Error CSI 00000030 (F) STATUS_OBJECT_NAME_NOT_FOUND #2937278# from CCSDirectTransaction::PrepareForCommit(...)[gle=0xd0000034]
2021-03-08 20:42:23, Error CSI 00000031 (F) STATUS_OBJECT_NAME_NOT_FOUND #2937277# from CCSDirectTransaction::GenerateComponentChangeList(...)[gle=0xd0000034]
2021-03-08 20:42:23, Error CSI 00000032 (F) STATUS_OBJECT_NAME_NOT_FOUND #2937276# from Windows::COM::CPendingTransaction::ExtractInformationFromRtlTransaction(...)[gle=0xd0000034]
2021-03-08 20:42:23, Error CSI 00000033 (F) HRESULT_FROM_WIN32(ERROR_FILE_NOT_FOUND) #1098636# from Windows::COM::CPendingTransaction::IStorePendingTransaction_Analyze(...)[gle=0x80070002]
2021-03-08 20:42:24, Error CSI 00000034 (F) HRESULT_FROM_WIN32(ERROR_FILE_NOT_FOUND) #1053458# from Windows::ServicingAPI::CCSITransaction::ICSITransaction_Commit(Flags = 102 (0x00000066), pSink = NULL, disp = 0, coldpatching = FALSE)[gle=0x80070002]
2021-03-08 20:42:24, Error CSI 00000035 (F) HRESULT_FROM_WIN32(ERROR_FILE_NOT_FOUND) #1053457# 80512710 us from Windows::ServicingAPI::CCSITransaction_ICSITransaction::Commit(flags = 0x00000066, pSink = NULL, disp = 0)
[gle=0x80070002]

So, we decided to go the route of the workarounds until we could get this resolved with Microsoft support via this article: https://msrc-blog.microsoft.com/2021/03/05/microsoft-exchange-server-vulnerabilities-mitigations-march-2021/

BUT, when we try to get into EMS to run PS commands, we get this same .NET error:

VERBOSE: Connecting to EXCHANGE1.domain.org. New-PSSession : [exchange-1.domain.org] Processing data from remote server exchange-1.domain.org failed with the following error message: [ClientAccessServer=EXCHANGE-1,BackEndServer=exchange-1.domain.org,Reques tId=f249ccbf-b98e-4fb0-98a3-4f7dcf1d3df1,TimeStamp=3/9/2021 1:56:36 AM] [AuthZRequestId=7c1a891a-71b5-42c5-acbd-9619f0190b59][FailureCategory=AuthZ-BadImageFormatException] Could not load file or assembly 'System.Security, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The module was expected to contain an assembly manifest. For more information, see the about_Remote_Troubleshooting Help topic. At line:1 char:1 + New-PSSession -ConnectionURI "$connectionUri" -ConfigurationName Microsoft.Excha ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin gTransportException + FullyQualifiedErrorId : IncorrectProtocolVersion,PSSessionOpenFailed

Microsoft Support is running behind, so we have a client without Exchange now for 4 days and we are trying different snaps to see if we have any better luck. Any suggestions? [1]: /api/attachments/75520-image.png?platform=QnA

.NET Runtime
.NET Runtime
.NET: Microsoft Technologies based on the .NET software framework.Runtime: An environment required to run apps that aren't compiled to machine language.
1,119 questions
Exchange Server Management
Exchange Server Management
Exchange Server: A family of Microsoft client/server messaging and collaboration software.Management: The act or process of organizing, handling, directing or controlling something.
7,345 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Kael Yao-MSFT 37,496 Reputation points Microsoft Vendor
    2021-03-10T03:14:42.19+00:00

    Hi, @Charlie Sciuto

    Would you please confirm if .NET Framework 4.8 is installed and EMS is working fine in the backup?
    To check what version of .NET Framework is installed, you may open registry and locate HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full.
    Check the value of Release and compare it to values in this document: Detect .NET Framework 4.5 and later versions

    If .NET Framework 4.8 is installed and the upgrade to CU19 still fails unless you run the netfxrepairtool.exe tool,
    I would suggest recovering the Exchange server on a new hardware and restoring the database from backup.

    You may need to:

    1. turn off the current Exchange server
    2. install the same OS (Windows Server 2012 R2) and the prerequisites of Exchange server on the new hardware
    3. name the new hardware as the same name of your Exchange server and join it to the domain
    4. run the E:\Setup.exe /IAcceptExchangeServerLicenseTerms /Mode:RecoverServer command
    5. restore the database from backup

    For the detailed steps,please refer to these documents:
    Recover Exchange servers
    Using Windows Server Backup to back up and restore Exchange data

    If the response is helpful, please click "Accept Answer" and upvote it.
    Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.