We are using one primary site currently running version 1910 (upgrading this Friday to 2002). We have 12 DPs and we've been dealing with an intermittent issue where one DP will "randomly" stop responding to PXE requests. All the other DPs (which are all configured as PXE/Boot servers for other physical locations in the Site) continue to work uninterrupted.
In each case, looking at the SMSPXE.log will show the following error messages repeatedly:
Unable to open the image file (0x8007000B) SMSPXE 9/30/2020 12:02:31 AM 3432 (0x0D68)
Unable to open image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim.
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 12:02:31 AM 3432 (0x0D68)
Failed query WIM at D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim for information..
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 12:02:31 AM 3432 (0x0D68)
If I trace the logs back to when the issue begins, it seems to be triggered when I see the following message:
PXE Provider finished loading. SMSPXE 9/29/2020 11:37:26 PM 2652 (0x0A5C)
Boot image HCS00077 has changed since added SMSPXE 9/30/2020 12:02:31 AM 3432 (0x0D68)
Based on my troubleshooting, this issue seems to randomly happen after the PXE service restarts. It has happened on multiple different servers, but usually only one randomly each time.
I've been running windows updates on the servers this week in preparation for the site upgrade Friday and today it happened after the server rebooted. Previously, we were doing work related to HTTPS/PKI on the site and I determined that some PKI changes cause the PXE service on the DP to restart which triggered the issue.
Our "Fix" in each case has been to redistribute the Boot Image to the DP that is affected. This resolves the issue.
I was hoping for some help in identifying why the issue may be happening. We are not making any changes to that Boot image, we've been using it consistently without issues up until this started to happen a couple months ago. I don't have any idea of anything that changed that caused this to begin happening.
Here's a larger sample of the SMSPXE.log:
============> Received from client: SMSPXE 9/30/2020 10:49:07 AM 2684 (0x0A7C)
Operation: BootRequest (1) Addr type: 1 Addr Len: 6 Hop Count: 0 ID: 4BD4EC40
Sec Since Boot: 0 Client IP: 010.005.037.219 Your IP: 000.000.000.000 Server IP: 000.000.000.000 Relay Agent IP: 000.000.000.000
Addr: e4:54:e8:91:f2:14:
Magic Cookie: 63538263
Options:
Type=53 Msg Type: 3=Request
Type=55 Param Request List: 0102030405060c0d0f111216171c28292a2b3233363a3b3c4243618081828384858687
Type=57 Max Msg Size: 05c0
Type=60 ClassId: PXEClient:Arch:00007:UNDI:003016
Type=93 Client Arch: EFI BC
Type=94 UNDI: 010310
Type=97 UUID: 0044454c4c580010438034b7c04f483133 SMSPXE 9/30/2020 10:49:07 AM 2684 (0x0A7C)
Not in SSL. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Client lookup reply: <ClientIDReply><Identification Unknown="0" DuplicateSMBIOS="0" DuplicateMACAddress="0" ItemKey="16821590" ServerName=""><Machine><ClientID/><NetbiosName/></Machine></Identification></ClientIDReply>
SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: Device is in the database. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Request using architecture 9. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Client boot action reply: <ClientIDReply><Identification Unknown="0" DuplicateSMBIOS="0" DuplicateMACAddress="0" ItemKey="16821590" ServerName=""><Machine><ClientID>GUID:5F098579-4E92-4696-87ED-2B79639BA68C</ClientID><NetbiosName/></Machine></Identification><PXEBootAction LastPXEAdvertisementID="" LastPXEAdvertisementTime="" OfferID="HCS200AD" OfferIDTime="7/28/2020 11:32:00 AM" PkgID="HCS000C2" PackageVersion="" PackagePath="http://server/SMS_DP_SMSPKG$/HCS00077" BootImageID="HCS00077" Mandatory="0"/></ClientIDReply>
SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: found optional advertisement HCS200AD SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Warning: Matching Processor Architecture Boot Image (7) not found SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: Not serviced. SMSPXE 9/30/2020 10:49:07 AM 3476 (0x0D94)
============> Received from client: SMSPXE 9/30/2020 10:49:08 AM 2684 (0x0A7C)
Operation: BootRequest (1) Addr type: 1 Addr Len: 6 Hop Count: 0 ID: 4BD4EC40
Sec Since Boot: 1 Client IP: 010.005.037.219 Your IP: 000.000.000.000 Server IP: 000.000.000.000 Relay Agent IP: 000.000.000.000
Addr: e4:54:e8:91:f2:14:
Magic Cookie: 63538263
Options:
Type=53 Msg Type: 3=Request
Type=55 Param Request List: 0102030405060c0d0f111216171c28292a2b3233363a3b3c4243618081828384858687
Type=57 Max Msg Size: 05c0
Type=60 ClassId: PXEClient:Arch:00007:UNDI:003016
Type=93 Client Arch: EFI BC
Type=94 UNDI: 010310
Type=97 UUID: 0044454c4c580010438034b7c04f483133 SMSPXE 9/30/2020 10:49:08 AM 2684 (0x0A7C)
Not in SSL. SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
Request using architecture 9. SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
Client boot action reply: <ClientIDReply><Identification Unknown="0" DuplicateSMBIOS="0" DuplicateMACAddress="0" ItemKey="16821590" ServerName=""><Machine><ClientID>GUID:5F098579-4E92-4696-87ED-2B79639BA68C</ClientID><NetbiosName/></Machine></Identification><PXEBootAction LastPXEAdvertisementID="" LastPXEAdvertisementTime="" OfferID="HCS200AD" OfferIDTime="7/28/2020 11:32:00 AM" PkgID="HCS000C2" PackageVersion="" PackagePath="http://server/SMS_DP_SMSPKG$/HCS00077" BootImageID="HCS00077" Mandatory="0"/></ClientIDReply>
SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: found optional advertisement HCS200AD SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
Warning: Matching Processor Architecture Boot Image (7) not found SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: Not serviced. SMSPXE 9/30/2020 10:49:08 AM 3476 (0x0D94)
============> Received from client: SMSPXE 9/30/2020 10:49:10 AM 2684 (0x0A7C)
Operation: BootRequest (1) Addr type: 1 Addr Len: 6 Hop Count: 0 ID: 4BD4EC40
Sec Since Boot: 2 Client IP: 010.005.037.219 Your IP: 000.000.000.000 Server IP: 000.000.000.000 Relay Agent IP: 000.000.000.000
Addr: e4:54:e8:91:f2:14:
Magic Cookie: 63538263
Options:
Type=53 Msg Type: 3=Request
Type=55 Param Request List: 0102030405060c0d0f111216171c28292a2b3233363a3b3c4243618081828384858687
Type=57 Max Msg Size: 05c0
Type=60 ClassId: PXEClient:Arch:00007:UNDI:003016
Type=93 Client Arch: EFI BC
Type=94 UNDI: 010310
Type=97 UUID: 0044454c4c580010438034b7c04f483133 SMSPXE 9/30/2020 10:49:10 AM 2684 (0x0A7C)
Not in SSL. SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
Request using architecture 9. SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
Client boot action reply: <ClientIDReply><Identification Unknown="0" DuplicateSMBIOS="0" DuplicateMACAddress="0" ItemKey="16821590" ServerName=""><Machine><ClientID>GUID:5F098579-4E92-4696-87ED-2B79639BA68C</ClientID><NetbiosName/></Machine></Identification><PXEBootAction LastPXEAdvertisementID="" LastPXEAdvertisementTime="" OfferID="HCS200AD" OfferIDTime="7/28/2020 11:32:00 AM" PkgID="HCS000C2" PackageVersion="" PackagePath="http://server/SMS_DP_SMSPKG$/HCS00077" BootImageID="HCS00077" Mandatory="0"/></ClientIDReply>
SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: found optional advertisement HCS200AD SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
Warning: Matching Processor Architecture Boot Image (7) not found SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: Not serviced. SMSPXE 9/30/2020 10:49:10 AM 3476 (0x0D94)
============> Received from client: SMSPXE 9/30/2020 10:49:13 AM 2684 (0x0A7C)
Operation: BootRequest (1) Addr type: 1 Addr Len: 6 Hop Count: 0 ID: 4BD4EC40
Sec Since Boot: 3 Client IP: 010.005.037.219 Your IP: 000.000.000.000 Server IP: 000.000.000.000 Relay Agent IP: 000.000.000.000
Addr: e4:54:e8:91:f2:14:
Magic Cookie: 63538263
Options:
Type=53 Msg Type: 3=Request
Type=55 Param Request List: 0102030405060c0d0f111216171c28292a2b3233363a3b3c4243618081828384858687
Type=57 Max Msg Size: 05c0
Type=60 ClassId: PXEClient:Arch:00007:UNDI:003016
Type=93 Client Arch: EFI BC
Type=94 UNDI: 010310
Type=97 UUID: 0044454c4c580010438034b7c04f483133 SMSPXE 9/30/2020 10:49:13 AM 2684 (0x0A7C)
Not in SSL. SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Request using architecture 9. SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Client boot action reply: <ClientIDReply><Identification Unknown="0" DuplicateSMBIOS="0" DuplicateMACAddress="0" ItemKey="16821590" ServerName=""><Machine><ClientID>GUID:5F098579-4E92-4696-87ED-2B79639BA68C</ClientID><NetbiosName/></Machine></Identification><PXEBootAction LastPXEAdvertisementID="" LastPXEAdvertisementTime="" OfferID="HCS200AD" OfferIDTime="7/28/2020 11:32:00 AM" PkgID="HCS000C2" PackageVersion="" PackagePath="http://server/SMS_DP_SMSPKG$/HCS00077" BootImageID="HCS00077" Mandatory="0"/></ClientIDReply>
SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: found optional advertisement HCS200AD SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Warning: Matching Processor Architecture Boot Image (7) not found SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Not in SSL. SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
E4:54:E8:91:F2:14, 4C4C4544-0058-4310-8034-B7C04F483133: Not serviced. SMSPXE 9/30/2020 10:49:13 AM 3476 (0x0D94)
Unable to open registry key 'Software\Wow6432Node\Microsoft\Windows Kits\Installed Roots', cannot find location of ADK SMSPXE 9/30/2020 10:50:36 AM 7832 (0x1E98)
Loaded Windows Imaging API DLL (version '10.0.14393.3503') from location 'C:\Windows\system32\wimgapi.dll' SMSPXE 9/30/2020 10:50:36 AM 7832 (0x1E98)
Opening image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:50:36 AM 7832 (0x1E98)
Unable to open the image file (0x8007000B) SMSPXE 9/30/2020 10:50:36 AM 7832 (0x1E98)
Unable to open image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim.
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:50:36 AM 7832 (0x1E98)
Failed query WIM at D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim for information..
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:50:36 AM 7832 (0x1E98)
Unable to open registry key 'Software\Wow6432Node\Microsoft\Windows Kits\Installed Roots', cannot find location of ADK SMSPXE 9/30/2020 10:52:36 AM 7832 (0x1E98)
Loaded Windows Imaging API DLL (version '10.0.14393.3503') from location 'C:\Windows\system32\wimgapi.dll' SMSPXE 9/30/2020 10:52:36 AM 7832 (0x1E98)
Opening image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:52:36 AM 7832 (0x1E98)
Unable to open the image file (0x8007000B) SMSPXE 9/30/2020 10:52:36 AM 7832 (0x1E98)
Unable to open image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim.
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:52:36 AM 7832 (0x1E98)
Failed query WIM at D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim for information..
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:52:36 AM 7832 (0x1E98)
Unable to open registry key 'Software\Wow6432Node\Microsoft\Windows Kits\Installed Roots', cannot find location of ADK SMSPXE 9/30/2020 10:54:36 AM 7832 (0x1E98)
Loaded Windows Imaging API DLL (version '10.0.14393.3503') from location 'C:\Windows\system32\wimgapi.dll' SMSPXE 9/30/2020 10:54:36 AM 7832 (0x1E98)
Opening image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:54:36 AM 7832 (0x1E98)
Unable to open the image file (0x8007000B) SMSPXE 9/30/2020 10:54:36 AM 7832 (0x1E98)
Unable to open image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim.
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:54:36 AM 7832 (0x1E98)
Failed query WIM at D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim for information..
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:54:36 AM 7832 (0x1E98)
Unable to open registry key 'Software\Wow6432Node\Microsoft\Windows Kits\Installed Roots', cannot find location of ADK SMSPXE 9/30/2020 10:56:36 AM 7832 (0x1E98)
Loaded Windows Imaging API DLL (version '10.0.14393.3503') from location 'C:\Windows\system32\wimgapi.dll' SMSPXE 9/30/2020 10:56:36 AM 7832 (0x1E98)
Opening image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:56:36 AM 7832 (0x1E98)
Unable to open the image file (0x8007000B) SMSPXE 9/30/2020 10:56:36 AM 7832 (0x1E98)
Unable to open image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim.
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:56:36 AM 7832 (0x1E98)
Failed query WIM at D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim for information..
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:56:36 AM 7832 (0x1E98)
Cannot read the registry value of MACIgnoreListFile (00000000) SMSPXE 9/30/2020 10:57:14 AM 7832 (0x1E98)
MAC Ignore List Filename in registry is empty SMSPXE 9/30/2020 10:57:14 AM 7832 (0x1E98)
Unable to open registry key 'Software\Wow6432Node\Microsoft\Windows Kits\Installed Roots', cannot find location of ADK SMSPXE 9/30/2020 10:58:37 AM 7832 (0x1E98)
Loaded Windows Imaging API DLL (version '10.0.14393.3503') from location 'C:\Windows\system32\wimgapi.dll' SMSPXE 9/30/2020 10:58:37 AM 7832 (0x1E98)
Opening image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:58:37 AM 7832 (0x1E98)
Unable to open the image file (0x8007000B) SMSPXE 9/30/2020 10:58:37 AM 7832 (0x1E98)
Unable to open image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim.
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:58:37 AM 7832 (0x1E98)
Failed query WIM at D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim for information..
An attempt was made to load a program with an incorrect format. (Error: 8007000B; Source: Windows) SMSPXE 9/30/2020 10:58:37 AM 7832 (0x1E98)
Found new image HCS00077 SMSPXE 9/30/2020 10:58:40 AM 2732 (0x0AAC)
Unable to open registry key 'Software\Wow6432Node\Microsoft\Windows Kits\Installed Roots', cannot find location of ADK SMSPXE 9/30/2020 10:58:40 AM 2732 (0x0AAC)
Loaded Windows Imaging API DLL (version '10.0.14393.3503') from location 'C:\Windows\system32\wimgapi.dll' SMSPXE 9/30/2020 10:58:40 AM 2732 (0x0AAC)
Opening image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:58:40 AM 2732 (0x0AAC)
Found Image file: D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim
PackageID: HCS00077
ProductName: Microsoft® Windows® Operating System
Architecture: 9
Description: Microsoft Windows PE (x64)
Version:
Creator:
SystemDir: WINDOWS
SMSPXE 9/30/2020 10:58:40 AM 2732 (0x0AAC)
Closing image file D:\RemoteInstall\SMSImages\HCS00077\boot.HCS00077.wim SMSPXE 9/30/2020 10:58:40 AM 2732 (0x0AAC)