Boot Image Randomly Fails to load after "Boot image has changed since added"

Dan Zabinski 191 Reputation points
2020-09-30T15:10:31.697+00:00

We are using one primary site currently running version 1910 (upgrading this Friday to 2002). We have 12 DPs and we've been dealing with an intermittent issue where one DP will "randomly" stop responding to PXE requests. All the other DPs (which are all configured as PXE/Boot servers for other physical locations in the Site) continue to work uninterrupted.

In each case, looking at the SMSPXE.log will show the following error messages repeatedly:

Unable to open the image file (0x8007000B) SMSPXE 9/30/2020 12:02:31 AM 3432 (0x0D68)
Unable to open image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim.
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 12:02:31 AM 3432 (0x0D68)
Failed query WIM at D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim for information..
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 12:02:31 AM 3432 (0x0D68)

If I trace the logs back to when the issue begins, it seems to be triggered when I see the following message:

PXE Provider finished loading. SMSPXE 9/29/2020 11:37:26 PM 2652 (0x0A5C)
Boot image HCS00077 has changed since added SMSPXE 9/30/2020 12:02:31 AM 3432 (0x0D68)

Based on my troubleshooting, this issue seems to randomly happen after the PXE service restarts. It has happened on multiple different servers, but usually only one randomly each time.

I've been running windows updates on the servers this week in preparation for the site upgrade Friday and today it happened after the server rebooted. Previously, we were doing work related to HTTPS/PKI on the site and I determined that some PKI changes cause the PXE service on the DP to restart which triggered the issue.

Our "Fix" in each case has been to redistribute the Boot Image to the DP that is affected. This resolves the issue.

I was hoping for some help in identifying why the issue may be happening. We are not making any changes to that Boot image, we've been using it consistently without issues up until this started to happen a couple months ago. I don't have any idea of anything that changed that caused this to begin happening.

Here's a larger sample of the SMSPXE.log:
============> Received from client: SMSPXE 9/30/2020 10:49:07 AM 2684 (0x0A7C)
Operation: BootRequest (1) Addr type: 1 Addr Len: 6 Hop Count: 0 ID: 4BD4EC40
Sec Since Boot: 0 Client IP: 010.005.037.219 Your IP: 000.000.000.000 Server IP: 000.000.000.000 Relay Agent IP: 000.000.000.000
Addr: e4:54:e8:91:f2:14:
Magic Cookie: 63538263
Options:
Type=53 Msg Type: 3=Request
Type=55 Param Request List: 0102030405060c0d0f111216171c28292a2b3233363a3b3c4243618081828384858687
Type=57 Max Msg Size: 05c0
Type=60 ClassId: PXEClient:Arch:00007:UNDI:003016
Type=93 Client Arch: EFI BC
Type=94 UNDI: 010310
Type=97 UUID: 0044454c4c580010438034b7c04f483133 SMSPXE 9/30/2020 10:49:07 AM 2684 (0x0A7C)
Not in SSL. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Client lookup reply: <ClientIDReply><Identification Unknown="0" DuplicateSMBIOS="0" DuplicateMACAddress="0" ItemKey="16821590" ServerName=""><Machine><ClientID/><NetbiosName/></Machine></Identification></ClientIDReply>
SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: Device is in the database. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Request using architecture 9. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Client boot action reply: <ClientIDReply><Identification Unknown="0" DuplicateSMBIOS="0" DuplicateMACAddress="0" ItemKey="16821590" ServerName=""><Machine><ClientID>GUID:5F098579-4E92-4696-87ED-2B79639BA68C</ClientID><NetbiosName/></Machine></Identification><PXEBootAction LastPXEAdvertisementID="" LastPXEAdvertisementTime="" OfferID="HCS200AD" OfferIDTime="7/28/2020 11:32:00 AM" PkgID="HCS000C2" PackageVersion="" PackagePath="http://server/SMS_DP_SMSPKG$/HCS00077" BootImageID="HCS00077" Mandatory="0"/></ClientIDReply>
SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: found optional advertisement HCS200AD SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Warning: Matching Processor Architecture Boot Image (7) not found SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: Not serviced. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
============> Received from client: SMSPXE 9/30/2020 10:49:08 AM 2684 (0x0A7C)
Operation: BootRequest (1) Addr type: 1 Addr Len: 6 Hop Count: 0 ID: 4BD4EC40
Sec Since Boot: 1 Client IP: 010.005.037.219 Your IP: 000.000.000.000 Server IP: 000.000.000.000 Relay Agent IP: 000.000.000.000
Addr: e4:54:e8:91:f2:14:
Magic Cookie: 63538263
Options:
Type=53 Msg Type: 3=Request
Type=55 Param Request List: 0102030405060c0d0f111216171c28292a2b3233363a3b3c4243618081828384858687
Type=57 Max Msg Size: 05c0
Type=60 ClassId: PXEClient:Arch:00007:UNDI:003016
Type=93 Client Arch: EFI BC
Type=94 UNDI: 010310
Type=97 UUID: 0044454c4c580010438034b7c04f483133 SMSPXE 9/30/2020 10:49:08 AM 2684 (0x0A7C)
Not in SSL. SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
Request using architecture 9. SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
Client boot action reply: <ClientIDReply><Identification Unknown="0" DuplicateSMBIOS="0" DuplicateMACAddress="0" ItemKey="16821590" ServerName=""><Machine><ClientID>GUID:5F098579-4E92-4696-87ED-2B79639BA68C</ClientID><NetbiosName/></Machine></Identification><PXEBootAction LastPXEAdvertisementID="" LastPXEAdvertisementTime="" OfferID="HCS200AD" OfferIDTime="7/28/2020 11:32:00 AM" PkgID="HCS000C2" PackageVersion="" PackagePath="http://server/SMS_DP_SMSPKG$/HCS00077" BootImageID="HCS00077" Mandatory="0"/></ClientIDReply>
SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: found optional advertisement HCS200AD SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
Warning: Matching Processor Architecture Boot Image (7) not found SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: Not serviced. SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
============> Received from client: SMSPXE 9/30/2020 10:49:10 AM 2684 (0x0A7C)
Operation: BootRequest (1) Addr type: 1 Addr Len: 6 Hop Count: 0 ID: 4BD4EC40
Sec Since Boot: 2 Client IP: 010.005.037.219 Your IP: 000.000.000.000 Server IP: 000.000.000.000 Relay Agent IP: 000.000.000.000
Addr: e4:54:e8:91:f2:14:
Magic Cookie: 63538263
Options:
Type=53 Msg Type: 3=Request
Type=55 Param Request List: 0102030405060c0d0f111216171c28292a2b3233363a3b3c4243618081828384858687
Type=57 Max Msg Size: 05c0
Type=60 ClassId: PXEClient:Arch:00007:UNDI:003016
Type=93 Client Arch: EFI BC
Type=94 UNDI: 010310
Type=97 UUID: 0044454c4c580010438034b7c04f483133 SMSPXE 9/30/2020 10:49:10 AM 2684 (0x0A7C)
Not in SSL. SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
Request using architecture 9. SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
Client boot action reply: <ClientIDReply><Identification Unknown="0" DuplicateSMBIOS="0" DuplicateMACAddress="0" ItemKey="16821590" ServerName=""><Machine><ClientID>GUID:5F098579-4E92-4696-87ED-2B79639BA68C</ClientID><NetbiosName/></Machine></Identification><PXEBootAction LastPXEAdvertisementID="" LastPXEAdvertisementTime="" OfferID="HCS200AD" OfferIDTime="7/28/2020 11:32:00 AM" PkgID="HCS000C2" PackageVersion="" PackagePath="http://server/SMS_DP_SMSPKG$/HCS00077" BootImageID="HCS00077" Mandatory="0"/></ClientIDReply>
SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: found optional advertisement HCS200AD SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
Warning: Matching Processor Architecture Boot Image (7) not found SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: Not serviced. SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
============> Received from client: SMSPXE 9/30/2020 10:49:13 AM 2684 (0x0A7C)
Operation: BootRequest (1) Addr type: 1 Addr Len: 6 Hop Count: 0 ID: 4BD4EC40
Sec Since Boot: 3 Client IP: 010.005.037.219 Your IP: 000.000.000.000 Server IP: 000.000.000.000 Relay Agent IP: 000.000.000.000
Addr: e4:54:e8:91:f2:14:
Magic Cookie: 63538263
Options:
Type=53 Msg Type: 3=Request
Type=55 Param Request List: 0102030405060c0d0f111216171c28292a2b3233363a3b3c4243618081828384858687
Type=57 Max Msg Size: 05c0
Type=60 ClassId: PXEClient:Arch:00007:UNDI:003016
Type=93 Client Arch: EFI BC
Type=94 UNDI: 010310
Type=97 UUID: 0044454c4c580010438034b7c04f483133 SMSPXE 9/30/2020 10:49:13 AM 2684 (0x0A7C)
Not in SSL. SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Request using architecture 9. SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Client boot action reply: <ClientIDReply><Identification Unknown="0" DuplicateSMBIOS="0" DuplicateMACAddress="0" ItemKey="16821590" ServerName=""><Machine><ClientID>GUID:5F098579-4E92-4696-87ED-2B79639BA68C</ClientID><NetbiosName/></Machine></Identification><PXEBootAction LastPXEAdvertisementID="" LastPXEAdvertisementTime="" OfferID="HCS200AD" OfferIDTime="7/28/2020 11:32:00 AM" PkgID="HCS000C2" PackageVersion="" PackagePath="http://server/SMS_DP_SMSPKG$/HCS00077" BootImageID="HCS00077" Mandatory="0"/></ClientIDReply>
SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: found optional advertisement HCS200AD SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Warning: Matching Processor Architecture Boot Image (7) not found SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: Not serviced. SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Unable to open registry key 'Software\Wow6432Node\Microsoft\Windows Kits\Installed Roots', cannot find location of ADK SMSPXE 9/30/2020 10:50:36 AM 7832 (0x1E98)
Loaded Windows Imaging API DLL (version '10.0.14393.3503') from location 'C:\Windows\system32\wimgapi.dll' SMSPXE 9/30/2020 10:50:36 AM 7832 (0x1E98)
Opening image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:50:36 AM 7832 (0x1E98)
Unable to open the image file (0x8007000B) SMSPXE 9/30/2020 10:50:36 AM 7832 (0x1E98)
Unable to open image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim.
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:50:36 AM 7832 (0x1E98)
Failed query WIM at D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim for information..
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:50:36 AM 7832 (0x1E98)
Unable to open registry key 'Software\Wow6432Node\Microsoft\Windows Kits\Installed Roots', cannot find location of ADK SMSPXE 9/30/2020 10:52:36 AM 7832 (0x1E98)
Loaded Windows Imaging API DLL (version '10.0.14393.3503') from location 'C:\Windows\system32\wimgapi.dll' SMSPXE 9/30/2020 10:52:36 AM 7832 (0x1E98)
Opening image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:52:36 AM 7832 (0x1E98)
Unable to open the image file (0x8007000B) SMSPXE 9/30/2020 10:52:36 AM 7832 (0x1E98)
Unable to open image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim.
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:52:36 AM 7832 (0x1E98)
Failed query WIM at D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim for information..
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:52:36 AM 7832 (0x1E98)
Unable to open registry key 'Software\Wow6432Node\Microsoft\Windows Kits\Installed Roots', cannot find location of ADK SMSPXE 9/30/2020 10:54:36 AM 7832 (0x1E98)
Loaded Windows Imaging API DLL (version '10.0.14393.3503') from location 'C:\Windows\system32\wimgapi.dll' SMSPXE 9/30/2020 10:54:36 AM 7832 (0x1E98)
Opening image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:54:36 AM 7832 (0x1E98)
Unable to open the image file (0x8007000B) SMSPXE 9/30/2020 10:54:36 AM 7832 (0x1E98)
Unable to open image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim.
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:54:36 AM 7832 (0x1E98)
Failed query WIM at D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim for information..
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:54:36 AM 7832 (0x1E98)
Unable to open registry key 'Software\Wow6432Node\Microsoft\Windows Kits\Installed Roots', cannot find location of ADK SMSPXE 9/30/2020 10:56:36 AM 7832 (0x1E98)
Loaded Windows Imaging API DLL (version '10.0.14393.3503') from location 'C:\Windows\system32\wimgapi.dll' SMSPXE 9/30/2020 10:56:36 AM 7832 (0x1E98)
Opening image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:56:36 AM 7832 (0x1E98)
Unable to open the image file (0x8007000B) SMSPXE 9/30/2020 10:56:36 AM 7832 (0x1E98)
Unable to open image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim.
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:56:36 AM 7832 (0x1E98)
Failed query WIM at D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim for information..
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:56:36 AM 7832 (0x1E98)
Cannot read the registry value of MACIgnoreListFile (00000000) SMSPXE 9/30/2020 10:57:14 AM 7832 (0x1E98)
MAC Ignore List Filename in registry is empty SMSPXE 9/30/2020 10:57:14 AM 7832 (0x1E98)
Unable to open registry key 'Software\Wow6432Node\Microsoft\Windows Kits\Installed Roots', cannot find location of ADK SMSPXE 9/30/2020 10:58:37 AM 7832 (0x1E98)
Loaded Windows Imaging API DLL (version '10.0.14393.3503') from location 'C:\Windows\system32\wimgapi.dll' SMSPXE 9/30/2020 10:58:37 AM 7832 (0x1E98)
Opening image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:58:37 AM 7832 (0x1E98)
Unable to open the image file (0x8007000B) SMSPXE 9/30/2020 10:58:37 AM 7832 (0x1E98)
Unable to open image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim.
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:58:37 AM 7832 (0x1E98)
Failed query WIM at D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim for information..
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:58:37 AM 7832 (0x1E98)
Found new image HCS00077 SMSPXE 9/30/2020 10:58:40 AM 2732 (0x0AAC)
Unable to open registry key 'Software\Wow6432Node\Microsoft\Windows Kits\Installed Roots', cannot find location of ADK SMSPXE 9/30/2020 10:58:40 AM 2732 (0x0AAC)
Loaded Windows Imaging API DLL (version '10.0.14393.3503') from location 'C:\Windows\system32\wimgapi.dll' SMSPXE 9/30/2020 10:58:40 AM 2732 (0x0AAC)
Opening image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:58:40 AM 2732 (0x0AAC)
Found Image file: D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim
PackageID: HCS00077
ProductName: Microsoft® Windows® Operating System
Architecture: 9
Description: Microsoft Windows PE (x64)
Version:
Creator:
SystemDir: WINDOWS
SMSPXE 9/30/2020 10:58:40 AM 2732 (0x0AAC)
Closing image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:58:40 AM 2732 (0x0AAC)

Microsoft Configuration Manager Deployment
Microsoft Configuration Manager Deployment
Microsoft Configuration Manager: An integrated solution for for managing large groups of personal computers and servers.Deployment: The process of delivering, assembling, and maintaining a particular version of a software system at a site.
906 questions
Microsoft Configuration Manager
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Simon Ren-MSFT 30,501 Reputation points Microsoft Vendor
    2020-10-01T08:19:55.283+00:00

    Hi,

    Thanks for posting in TechNet.

    The error code 0x8007000B means "An attempt was made to load a program with an incorrect format".

    1.This issue may be due to some kind of mismatch between x86 and x64 boot images vs. OS image or possibly with WinPE. What version of Windows ADK and boot images you are using? If possible, please upgarde Windows ADK and Windows PE Add-on to the latest version and "Reload this boot image with the current Windows PE version from the Windows ADK" to have a try.
    For more detailed steps, please refer to: Step-By-Step Guide To Update Windows ADK On SCCM Server

    2.When we have an environment with HTTPS only, based on my experience, every DP should have two certificate: one for the DP itself and one for the client. The client must have a valid Boot images for deploying operating systems certificate for the client to communicate with the site and for the PXE OSD deployment to continue. And the PXE-enabled DP also need a certificate for Site systems that have a distribution point installed to communicate with HTTPS-enabled management point. Please help check if the certificates are ok.
    For more information, please refer to: PKI certificates for clients

    3.If possible, please help create an identical task sequence and deploy it to your collections to have a try.

    Hope my answer could help you. Thanks for your time.
     
    Best regards,
    Simon
    If the response is helpful, please click "Accept Answer" and upvote it.