Server 2012 R2 BSOD after installing November Updates

Imad Ubadat 1 Reputation point
2020-12-03T00:53:06.503+00:00

Hey All,

I deployed November updates to 800+ servers as part of our monthly patching. the servers are scheduled to got the updates @ 8 pm and if they are pending reboots they get rebooted automatically @ 4 am the following morning.

post reboots 10 servers have failed to load OS (All Server 2012 R2) with 2 different symptoms:

  1. Servers completely fail to load OS and go straight to recovery mode
  2. Servers try to load, display a message collecting information then it reboot to recovery mode.

reboot in safe mode wont work.

to fix this:

  1. Attach Windows Server 2012 installation ISO to CD-ROM
  2. Boot from CD-ROM
  3. Go to Repair section and then select Command prompt
  4. From the VM menu
    Go to Guest
    Install/Upgrade VMWare Tools
    Click Okay
  5. Go back to Command prompt
  6. Type this: drvload "D:\Program Files\VMware\VMware Tools\Drivers\pvscsi\Win8\amd64\pvscsi.inf" (hit enter)
  7. Diskpart (Enter)
    List Vol
    Exit 
                  (Our aim is to find the Windows drive. Try each drive one by one to find which drive has the Windows folder)
  8. del X:\windows\winsxs\pending.xml (X: is representing the Windows Drive. replace it with the one you found)
  9. sfc /scannow /offbootdir=X:\ /offwindir=X:\windows (X: is representing the Windows Drive. replace it with the one you found)

all servers are VM's running on VMWare.

we have 2 Data Centers, all broken Servers are in the same Data Center. other Server 2012 R2 in the same Data Center got updated and rebooted without any issues.

Cannot find anything in any log or Event Viewer. I found one .dmp file on one of the servers:

Microsoft (R) Windows Debugger Version 10.0.20153.1000 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Users\xxxxx\Desktop\113020-41218-01.dmp]
Mini Kernel Dump File: Only registers and stack trace are available

************* Path validation summary **************
Response Time (ms) Location
Deferred srv*
Symbol search path is: srv*
Executable search path is:
Windows 8.1 Kernel Version 9600 MP (2 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Edition build lab: 9600.19846.amd64fre.winblue_ltsb_escrow.200923-1735
Machine Name:
Kernel base = 0xfffff801f0e7f000 PsLoadedModuleList = 0xfffff801f11445d0
Debug session time: Mon Nov 30 05:17:11.633 2020 (UTC + 11:00)
System Uptime: 0 days 0:00:42.326
Loading Kernel Symbols
...............................................................
.......................................
Loading User Symbols
Loading unloaded module list
....
For analysis of this file, run !analyze -v
nt!KeBugCheckEx:
fffff801f0fbf4c0 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:ffffd000453e4800=000000000000004c
1: kd> !analyze -v

**

Bugcheck Analysis

**

WINLOGON_FATAL_ERROR (c000021a)
The Winlogon process terminated unexpectedly.
Arguments:
Arg1: ffffc00152ea4800, String that identifies the problem.
Arg2: ffffffffc0000428, Error Code.
Arg3: 0000000000000000
Arg4: 0000005f19d10768

Debugging Details:

------------------

ETW minidump data unavailable
fffff801f1111e58: Unable to get Flags value from nt!KdVersionBlock
GetUlongPtrFromAddress: unable to read from fffff801f11ce308

KEY_VALUES_STRING: 1 

Key  : Analysis.CPU.mSec  
Value: 2436  
 
Key  : Analysis.DebugAnalysisProvider.CPP  
Value: Create: 8007007e on AU69054  
 
Key  : Analysis.DebugData  
Value: CreateObject  
 
Key  : Analysis.DebugModel  
Value: CreateObject  
 
Key  : Analysis.Elapsed.mSec  
Value: 2470  
 
Key  : Analysis.Memory.CommitPeak.Mb  
Value: 63  
 
Key  : Analysis.System  
Value: CreateObject  
 
Key  : WER.OS.Branch  
Value: winblue_ltsb_escrow  
 
Key  : WER.OS.Timestamp  
Value: 2020-09-23T17:35:00Z  
 
Key  : WER.OS.Version  
Value: 8.1.9600.19846

ADDITIONAL_XML: 1

OS_BUILD_LAYERS: 1

VIRTUAL_MACHINE: VMware

ERROR_CODE: (NTSTATUS) 0xc000021a - {Fatal System Error} The %hs system process terminated unexpectedly with a status of 0x

EXCEPTION_CODE_STR: c000021a

EXCEPTION_PARAMETER1: ffffc00152ea4800

EXCEPTION_PARAMETER2: ffffffffc0000428

EXCEPTION_PARAMETER3: 0000000000000000

EXCEPTION_PARAMETER4: 5f19d10768

BUGCHECK_CODE: c000021a

BUGCHECK_P1: ffffc00152ea4800

BUGCHECK_P2: ffffffffc0000428

BUGCHECK_P3: 0

BUGCHECK_P4: 5f19d10768

PROCESS_NAME: smss.exe

ADDITIONAL_DEBUG_TEXT: initial session process or

IMAGE_NAME: ntkrnlmp.exe

MODULE_NAME: nt

CUSTOMER_CRASH_COUNT: 1

STACK_TEXT:
ffffd000453e47f8 fffff801f11f78a9 : 000000000000004c 00000000c000021a ffffd000463fd538 ffffe00123fb7860 : nt!KeBugCheckEx
ffffd000453e4800 fffff801f11ef194 : 0000000000000000 ffffd000453e4919 0000000000000000 0000000000000002 : nt!PopGracefulShutdown+0x2c9
ffffd000453e4840 fffff801f0fcf3e3 : ffffe00123c7d880 0000000000000000 00000000c0000004 ffffd000453e4a00 : nt! ?? ::OKHAJAOM::string'+0x1d64 ffffd000453e4980 fffff801f0fc34a0 : fffff801f13f678d 0000000000000001 ffffd000453e4b98 00000000c0000004 : nt!KiSystemServiceCopyEnd+0x13 ffffd000453e4b18 fffff801f13f678d : 0000000000000001 ffffd000453e4b98 00000000c0000004 fffff801f1160180 : nt!KiServiceLinkage ffffd000453e4b20 fffff801f1339d83 : 0000000000000000 0000000000000000 fffff801f1160180 ffffe00123c7d9c0 : nt! ?? ::NNGAKEGL::string'+0x6341d
ffffd000453e4be0 fffff801f0f28722 : fffff801f0f28668 0000000000000000 0000000000000002 0000000000000000 : nt!PopPolicyWorkerAction+0x63
ffffd000453e4c50 fffff801f0eec5bf : fffff80100000002 ffffe00123c7d880 fffff801f112b040 fffff801f1124e60 : nt!PopPolicyWorkerThread+0xba
ffffd000453e4c90 fffff801f0f4293e : 351bdc594dba662b fffff801f1160180 0000000000000080 ffffe001238b1040 : nt!ExpWorkerThread+0x69f
ffffd000453e4d40 fffff801f0fc6f66 : fffff801f1160180 ffffe00123c7d880 ffffe00123898880 000000022d5f0f64 : nt!PspSystemThreadStartup+0x18a
ffffd000453e4da0 0000000000000000 : ffffd000453e5000 ffffd000453df000 0000000000000000 0000000000000000 : nt!KiStartSystemThread+0x16

SYMBOL_NAME: nt! ?? ::OKHAJAOM::`string'+1d64

IMAGE_VERSION: 6.3.9600.19846

STACK_COMMAND: .thread ; .cxr ; kb

BUCKET_ID_FUNC_OFFSET: 1d64

FAILURE_BUCKET_ID: 0xc000021a_SmpDestroyControlBlock_smss.exe_Terminated_c0000428_nt!??::OKHAJAOM::string

OS_VERSION: 8.1.9600.19846

BUILDLAB_STR: winblue_ltsb_escrow

OSPLATFORM_TYPE: x64

OSNAME: Windows 8.1

FAILURE_ID_HASH: {341f3cd7-cdf2-aefe-6083-b8fc309194a4}

Followup: MachineOwner

Has anyone else had issues with last month updates?

Much appriciated.

Windows Server 2012
Windows Server 2012
A Microsoft server operating system that supports enterprise-level management, data storage, applications, and communications.
1,534 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Jenny Feng 14,081 Reputation points
    2020-12-03T07:13:08.92+00:00

    @Imad Ubadat
    Hi,
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired\
     
    This key should generate and delete as updates are processed. If you see it, check for values. If it has them, back up the key, delete the key. Reset Update Services. Check for updates. That may resolve your issue.

    Hope above information can help you.

    ============================================

    If the Answer is helpful, please click "Accept Answer" and upvote it.
    Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.