Windows Server 2022 - HTTP.sys BSOD

ChrisB3127 6 Reputation points
2022-04-10T04:14:44.04+00:00

I upgraded our servers to Windows Server 2022 Standard 2 weeks ago, from Windows Server 2016. System appeared to be rock-solid since the upgrade.

On Saturday, around noon, the server handling IIS had a BSOD, which caused a spontaneous reboot. What's odd is after the BSOD, the reboot back into Windows had a "Configuring Features xx%" screen on, as if it was doing updates. Initially, I assumed maybe some random Windows Update decided to do a reboot mid-day, but this was not the case.

The Event Viewer simply shows the BSOD, and I was able to grab the dump file. When I run the debug file through WinDbg I get this: (edited for conciseness, if more info is needed, let me know)

This is the first time this happened, but it appears to be something involving HTTP.sys which I assume would be IIS-related. We do have HTTP/3 enabled, in case this matters. (I read there could be some performance issues with HTTP/3 and memory) I will keep an eye on things, since this is the first occurrence of this problem.

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the BugCheck
Arg2: fffff8064ebba2af, Address of the instruction which caused the BugCheck
Arg3: ffffc98b80836100, Address of the context record for the exception that caused the BugCheck
Arg4: 0000000000000000, zero.

======

FILE_IN_CAB: DUMP5ff2.tmp

BUGCHECK_CODE: 3b

BUGCHECK_P1: c0000005

BUGCHECK_P2: fffff8064ebba2af

BUGCHECK_P3: ffffc98b80836100

BUGCHECK_P4: 0

====

SYMBOL_NAME: HTTP!UxpTpRestartBufferSend+f

MODULE_NAME: HTTP

IMAGE_NAME: HTTP.sys

STACK_COMMAND: .cxr 0xffffc98b80836100 ; kb

BUCKET_ID_FUNC_OFFSET: f

FAILURE_BUCKET_ID: AV_HTTP!UxpTpRestartBufferSend

OS_VERSION: 10.0.20348.1

BUILDLAB_STR: fe_release

Internet Information Services
Windows Server
Windows Server
A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.
12,152 questions

6 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Docs 15,141 Reputation points
    2022-04-10T06:09:25.727+00:00

    Please run the DM log collector and post a share link into this thread using one drive, drop box, or google drive.

    If the server is able to run the V2 log collector it will collect more files useful for troubleshooting.

    https://www.tenforums.com/bsod-crashes-debugging/2198-bsod-posting-instructions.html

    https://www.elevenforum.com/t/bsod-posting-instructions.103/

    If needed, indicate if there is the option for testing with server downtime.

    .
    .
    .
    .
    .

    Please remember to vote and to mark the replies as answers if they help.

    On the bottom of each post there is:

    Propose as answer = answered the question

    On the left side of each post there is /\ with a number: click = a helpful post
    .
    .
    .
    .
    .

  2. ChrisB3127 6 Reputation points
    2022-05-03T01:20:01.38+00:00

    Here is a more complete paste of the dump file analyze:

    SYSTEM_SERVICE_EXCEPTION (3b)
    An exception happened while executing a system service routine.
    Arguments:
    Arg1: 00000000c0000005, Exception code that caused the BugCheck
    Arg2: fffff8068761ab5f, Address of the instruction which caused the BugCheck
    Arg3: ffff81014abbe100, Address of the context record for the exception that caused the BugCheck
    Arg4: 0000000000000000, zero.

    KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
Value: 1906

Key  : Analysis.DebugAnalysisManager
Value: Create

Key  : Analysis.Elapsed.mSec
Value: 1932

Key  : Analysis.Init.CPU.mSec
Value: 358

Key  : Analysis.Init.Elapsed.mSec
Value: 3199

Key  : Analysis.Memory.CommitPeak.Mb
Value: 109

Key  : WER.OS.Branch
Value: fe_release

Key  : WER.OS.Timestamp
Value: 2021-05-07T15:00:00Z

Key  : WER.OS.Version
Value: 10.0.20348.1

    FILE_IN_CAB: MEMORY.DMP

    BUGCHECK_CODE: 3b

    BUGCHECK_P1: c0000005

    BUGCHECK_P2: fffff8068761ab5f

    BUGCHECK_P3: ffff81014abbe100

    BUGCHECK_P4: 0

    CONTEXT: ffff81014abbe100 -- (.cxr 0xffff81014abbe100)
    rax=0000000000000000 rbx=00000000c0000184 rcx=ffff8705c9264810
    rdx=00000000c0000184 rsi=ffff8705cb6ba000 rdi=ffff8705c3517270
    rip=fffff8068761ab5f rsp=ffff81014abbeb20 rbp=ffff81014abbec29
    r8=0000000000000000 r9=ffff8705c9264810 r10=ffff8705bdb1d0c0
    r11=ffff8705c9264810 r12=fffff8068761ab50 r13=ffff8705c9264810
    r14=0000000000000004 r15=ffff8705c1919908
    iopl=0 nv up ei ng nz na pe nc
    cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00050282
    HTTP!UxpTpRestartBufferSend+0xf:
    fffff8068761ab5f 488b5008 mov rdx,qword ptr [rax+8] ds:002b:0000000000000008=????????????????
    Resetting default scope

    BLACKBOXBSD: 1 (!blackboxbsd)

    BLACKBOXNTFS: 1 (!blackboxntfs)

    BLACKBOXPNP: 1 (!blackboxpnp)

    BLACKBOXWINLOGON: 1

    PROCESS_NAME: w3wp.exe

    STACK_TEXT:
    ffff81014abbeb20 fffff806875f2a72 : ffff8705c1919908 0000000000000004 ffff8705c9264810 fffff8068761ab50 : HTTP!UxpTpRestartBufferSend+0xf
    ffff81014abbeb60 fffff806876b2256 : ffff8705c1db8060 00000000c0000184 ffff81014abbec29 ffff8705c3517270 : HTTP!UlInvokeCompletionRoutine+0x1a
    ffff81014abbeb90 fffff806875ca878 : ffff8705c83dea01 ffff8705c3517200 ffff8705000101ac ffff870500000000 : HTTP!UxpTpFlushMdlRuns+0x31b76
    ffff81014abbec90 fffff806875ca12d : ffff8705c3517270 ffff8705c1919908 0000000000000000 0000000000000000 : HTTP!UxpTpProcessMdlRuns+0x3c8
    ffff81014abbed60 fffff8068768b3b0 : ffff8705cb6ba000 000000000000004d ffff8705c83dd310 0400000000002401 : HTTP!UxpTpEnqueueTransmitPacket+0x13d
    ffff81014abbedb0 fffff806875c9f49 : ffff8705c1af00d0 ffff8705c1919590 0000000000000000 0000000000000000 : HTTP!UxTpTransmitPacket+0x200
    ffff81014abbee80 fffff8068768b760 : ffff8705c83dd000 0000000000000007 000000000000000c ffff8705c1919590 : HTTP!UlSendData+0x129
    ffff81014abbef40 fffff806875c9a4e : ffff8705c83dd000 ffff8705c1af0001 ffff8705c83dd200 ffff8705c83dd270 : HTTP!UlpSendHttpResponseWorker+0x1e0
    ffff81014abbf020 fffff8068768324f : 0000000000000002 0000000000000002 0000000000000000 ffff8f0b2f8672a0 : HTTP!UlSendHttpResponse+0x30e
    ffff81014abbf120 fffff806875c2fe4 : 0000000000000000 ffff81014abbfaa0 ffff8f0b2f8672a0 0000000000000168 : HTTP!UlSendHttpResponseIoctl+0x1adf
    ffff81014abbf770 fffff80671154ed5 : ffff8f0b2f8672a0 0000000000000002 0000000000000000 0000000000000000 : HTTP!UxDeviceControl+0x84
    ffff81014abbf7b0 fffff80671583a69 : ffff8f0b2f8672a0 0000000000000000 ffff8f0b2f8672a0 0000000000000000 : nt!IofCallDriver+0x55
    ffff81014abbf7f0 fffff80671593cb1 : 0000000000000000 ffff81014abbfb20 000000000012403f ffff81014abbfb20 : nt!IopSynchronousServiceTail+0x189
    ffff81014abbf890 fffff80671593d56 : 0000000000000001 0000000000000000 0000000000000000 000001e89fd3dcf8 : nt!IopXxxControlFile+0xc61
    ffff81014abbf9c0 fffff80671228e35 : 0000000000000000 0000000000001900 0000000000000000 ffff8705c94ae760 : nt!NtDeviceIoControlFile+0x56
    ffff81014abbfa30 00007ffb4944efd4 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!KiSystemServiceCopyEnd+0x25
    00000084e21ff048 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : 0x00007ffb`4944efd4

    SYMBOL_NAME: HTTP!UxpTpRestartBufferSend+f

    MODULE_NAME: HTTP

    IMAGE_NAME: HTTP.sys

    STACK_COMMAND: .cxr 0xffff81014abbe100 ; kb

    BUCKET_ID_FUNC_OFFSET: f

    FAILURE_BUCKET_ID: AV_HTTP!UxpTpRestartBufferSend

    OS_VERSION: 10.0.20348.1

    BUILDLAB_STR: fe_release

    OSPLATFORM_TYPE: x64

    OSNAME: Windows 10

    FAILURE_ID_HASH: {edb15224-7b43-c41e-e441-3f2974ff82b7}

    Followup: MachineOwner

    0: kd> .cxr 0xffff81014abbe100
    rax=0000000000000000 rbx=00000000c0000184 rcx=ffff8705c9264810
    rdx=00000000c0000184 rsi=ffff8705cb6ba000 rdi=ffff8705c3517270
    rip=fffff8068761ab5f rsp=ffff81014abbeb20 rbp=ffff81014abbec29
    r8=0000000000000000 r9=ffff8705c9264810 r10=ffff8705bdb1d0c0
    r11=ffff8705c9264810 r12=fffff8068761ab50 r13=ffff8705c9264810
    r14=0000000000000004 r15=ffff8705c1919908
    iopl=0 nv up ei ng nz na pe nc
    cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00050282
    HTTP!UxpTpRestartBufferSend+0xf:
    fffff8068761ab5f 488b5008 mov rdx,qword ptr [rax+8] ds:002b:0000000000000008=????????????????

    0 comments
  3. Docs 15,141 Reputation points
    2022-05-03T07:52:38.907+00:00

    Please post a share link for the DM log collector.

    If additional files are needed I'll comment into this thread.

    For dump files the actual files are needed so that commands can run on the debugging software.

    No commands can run on the text file.

    Indicate if you will or will not be able to arrange server downtime if additional testing is needed.

    0 comments
  4. Einar Arne 1 Reputation point
    2022-08-05T07:12:04.94+00:00

    We had the same blue screen on a Windows 2022 server running in IBM cloud. All windows updates applied.

    Mini Kernel Dump File: Only registers and stack trace are available

    Symbol search path is: srv*
    Executable search path is:
    Windows 10 Kernel Version 20348 MP (8 procs) Free x64
    Product: Server, suite: TerminalServer SingleUserTS
    Edition build lab: 20348.1.amd64fre.fe_release.210507-1500
    Machine Name:
    Kernel base = 0xfffff80057800000 PsLoadedModuleList = 0xfffff80058433950
    Debug session time: Thu Aug 4 14:50:38.262 2022 (UTC + 2:00)
    System Uptime: 1 days 4:56:12.058
    Loading Kernel Symbols
    ...............................................................
    ................................................................
    .....................................
    Loading User Symbols
    Loading unloaded module list
    ........
    For analysis of this file, run !analyze -v
    nt!KeBugCheckEx:
    fffff80057c17df0 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:ffffb806623e5710=000000000000003b
    5: kd> !analyze -v

    **

    Bugcheck Analysis

    **

    SYSTEM_SERVICE_EXCEPTION (3b)
    An exception happened while executing a system service routine.
    Arguments:
    Arg1: 00000000c0000005, Exception code that caused the BugCheck
    Arg2: fffff8005f4aaf6f, Address of the instruction which caused the BugCheck
    Arg3: ffffb806623e6040, Address of the context record for the exception that caused the BugCheck
    Arg4: 0000000000000000, zero.

    Debugging Details:

    ------------------

    KEY_VALUES_STRING: 1 

    Key  : Analysis.CPU.mSec  
Value: 2483  

Key  : Analysis.DebugAnalysisManager  
Value: Create  

Key  : Analysis.Elapsed.mSec  
Value: 29526  

Key  : Analysis.Init.CPU.mSec  
Value: 343  

Key  : Analysis.Init.Elapsed.mSec  
Value: 50648  

Key  : Analysis.Memory.CommitPeak.Mb  
Value: 95  

Key  : Bugcheck.Code.DumpHeader  
Value: 0x3b  

Key  : Bugcheck.Code.Register  
Value: 0x3b  

Key  : WER.OS.Branch  
Value: fe_release  

Key  : WER.OS.Timestamp  
Value: 2021-05-07T15:00:00Z  

Key  : WER.OS.Version  
Value: 10.0.20348.1

    FILE_IN_CAB: 080422-5656-01.dmp

    BUGCHECK_CODE: 3b

    BUGCHECK_P1: c0000005

    BUGCHECK_P2: fffff8005f4aaf6f

    BUGCHECK_P3: ffffb806623e6040

    BUGCHECK_P4: 0

    CONTEXT: ffffb806623e6040 -- (.cxr 0xffffb806623e6040)
    rax=0000000000000000 rbx=00000000c0000184 rcx=ffff800be9fd0910
    rdx=00000000c0000184 rsi=ffff800bc9b06000 rdi=ffff800bd1ae9aa0
    rip=fffff8005f4aaf6f rsp=ffffb806623e6a60 rbp=ffffb806623e6b69
    r8=0000000000000000 r9=ffff800be9fd0910 r10=ffff800bca59a380
    r11=ffff800be9fd0910 r12=fffff8005f4aaf60 r13=ffff800be9fd0910
    r14=0000000000000004 r15=ffff800bc96bfe88
    iopl=0 nv up ei ng nz na po nc
    cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00050286
    HTTP!UxpTpRestartBufferSend+0xf:
    fffff8005f4aaf6f 488b5008 mov rdx,qword ptr [rax+8] ds:002b:0000000000000008=????????????????
    Resetting default scope

    BLACKBOXBSD: 1 (!blackboxbsd)

    BLACKBOXNTFS: 1 (!blackboxntfs)

    BLACKBOXPNP: 1 (!blackboxpnp)

    BLACKBOXWINLOGON: 1

    CUSTOMER_CRASH_COUNT: 1

    PROCESS_NAME: w3wp.exe

    STACK_TEXT:
    ffffb806623e6a60 fffff8005f4831f2 : ffff800bc96bfe88 0000000000000004 ffff800be9fd0910 fffff8005f4aaf60 : HTTP!UxpTpRestartBufferSend+0xf
    ffffb806623e6aa0 fffff8005f542796 : ffff800bd14898a0 00000000c0000184 ffffb806623e6b69 ffff800bd1ae9aa0 : HTTP!UlInvokeCompletionRoutine+0x1a
    ffffb806623e6ad0 fffff8005f45acf8 : ffff800bc9cd4b01 ffff800bd1ae9a00 ffff800b0001a5ce ffff800b00000000 : HTTP!UxpTpFlushMdlRuns+0x31cd6
    ffffb806623e6bd0 fffff8005f45a5ad : ffff800bd1ae9aa0 ffff800bc96bfe88 0000000000000000 0000000000000000 : HTTP!UxpTpProcessMdlRuns+0x3c8
    ffffb806623e6ca0 fffff8005f51b790 : ffff800bc9b06000 000000000000001b ffff800bc9cd4310 0400000000002401 : HTTP!UxpTpEnqueueTransmitPacket+0x13d
    ffffb806623e6cf0 fffff8005f45a3c9 : ffff800bf6776050 ffff800bc96bfb10 0000000000000000 0000000000000000 : HTTP!UxTpTransmitPacket+0x200
    ffffb806623e6dc0 fffff8005f51bb40 : ffff800bc9cd4000 0000000000000006 000000000000000c ffff800bc96bfb10 : HTTP!UlSendData+0x129
    ffffb806623e6e80 fffff8005f459ece : ffff800bc9cd4000 ffff800bf6776000 ffff800bc9cd4200 ffff800bc9cd4270 : HTTP!UlpSendHttpResponseWorker+0x1e0
    ffffb806623e6f60 fffff8005f51362f : 0000000000000002 0000000000000002 0000000000000000 ffff800bc91d0030 : HTTP!UlSendHttpResponse+0x30e
    ffffb806623e7060 fffff8005f452fe4 : ffff800bc4503d20 fffff80057a672c9 ffff800bc91d0030 0000000000000168 : HTTP!UlSendHttpResponseIoctl+0x1adf
    ffffb806623e76b0 fffff80057b74aa5 : ffff800bc91d0030 0000000000000002 0000000000000000 0000000000000000 : HTTP!UxDeviceControl+0x84
    ffffb806623e76f0 fffff80057fb7b19 : ffff800bc91d0030 0000000000000000 ffff800bc91d0030 0000000000000000 : nt!IofCallDriver+0x55
    ffffb806623e7730 fffff80057e9aed1 : 0000000000000000 ffffb806623e7a60 000000000012403f ffffb806623e7a60 : nt!IopSynchronousServiceTail+0x189
    ffffb806623e77d0 fffff80057e9b236 : 0000000100000000 0000000000000000 0000000000000000 000001ee18e1f938 : nt!IopXxxControlFile+0xc61
    ffffb806623e7900 fffff80057c2a335 : ffff800bca9b1080 000000c30c43f9c8 ffffb806623e7988 000001ee15ae34e0 : nt!NtDeviceIoControlFile+0x56
    ffffb806623e7970 00007ff99ea3f854 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!KiSystemServiceCopyEnd+0x25
    000000c30c43e638 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : 0x00007ff9`9ea3f854

    SYMBOL_NAME: HTTP!UxpTpRestartBufferSend+f

    MODULE_NAME: HTTP

    IMAGE_NAME: HTTP.sys

    IMAGE_VERSION: 10.0.20348.1511

    STACK_COMMAND: .cxr 0xffffb806623e6040 ; kb

    BUCKET_ID_FUNC_OFFSET: f

    FAILURE_BUCKET_ID: AV_HTTP!UxpTpRestartBufferSend

    OS_VERSION: 10.0.20348.1

    BUILDLAB_STR: fe_release

    OSPLATFORM_TYPE: x64

    OSNAME: Windows 10

    FAILURE_ID_HASH: {edb15224-7b43-c41e-e441-3f2974ff82b7}

    Followup: MachineOwner
    ---------

  5. H L 1 Reputation point
    2022-09-14T19:29:20.067+00:00

    any solution? I have the same crash on windows 2022.

    in the mini dump file:

    HTTP!UxpTpRestartBufferSend+0xf caused a kernel BugCheck 3B

    BugCheck Code 0x3B

    I think it has to do with HTTP3, which is new and not well tested.