Windows Server 2022 - HTTP.sys BSOD

Question

question

ChrisB3127-4407 asked Apr 10, '22 ChrisB3127-4407 commented Aug 5, '22

Windows Server 2022 - HTTP.sys BSOD

I upgraded our servers to Windows Server 2022 Standard 2 weeks ago, from Windows Server 2016. System appeared to be rock-solid since the upgrade.

On Saturday, around noon, the server handling IIS had a BSOD, which caused a spontaneous reboot. What's odd is after the BSOD, the reboot back into Windows had a "Configuring Features xx%" screen on, as if it was doing updates. Initially, I assumed maybe some random Windows Update decided to do a reboot mid-day, but this was not the case.

The Event Viewer simply shows the BSOD, and I was able to grab the dump file. When I run the debug file through WinDbg I get this: (edited for conciseness, if more info is needed, let me know)

This is the first time this happened, but it appears to be something involving HTTP.sys which I assume would be IIS-related. We do have HTTP/3 enabled, in case this matters. (I read there could be some performance issues with HTTP/3 and memory) I will keep an eye on things, since this is the first occurrence of this problem.

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the BugCheck
Arg2: fffff8064ebba2af, Address of the instruction which caused the BugCheck
Arg3: ffffc98b80836100, Address of the context record for the exception that caused the BugCheck
Arg4: 0000000000000000, zero.

======

FILE_IN_CAB: DUMP5ff2.tmp

BUGCHECK_CODE: 3b

BUGCHECK_P1: c0000005

BUGCHECK_P2: fffff8064ebba2af

BUGCHECK_P3: ffffc98b80836100

BUGCHECK_P4: 0

====

SYMBOL_NAME: HTTP!UxpTpRestartBufferSend+f

MODULE_NAME: HTTP

IMAGE_NAME: HTTP.sys

STACK_COMMAND: .cxr 0xffffc98b80836100 ; kb

BUCKET_ID_FUNC_OFFSET: f

FAILURE_BUCKET_ID: AV_HTTP!UxpTpRestartBufferSend

OS_VERSION: 10.0.20348.1

BUILDLAB_STR: fe_release

windows-server windows-server-iis

· 2

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

BruceZhang-MSFT · Apr 11 at 01:51 AM

Hi @ChrisB3127-4407 ,

About BSOD, you can fill out this survey to help Microsoft troubleshoot this problem better.

According to this docs, 0x0000003B indicates that an exception happened while executing a routine that transitions from non-privileged code to privileged code. 0xC0000005 indicates a memory access violation occurred. Arg4(0000000000000000, zero) of the bug check is the address that the driver attempted to access.

To debug this problem, use the .cxr (display context record) command with Parameter 3, and then use kb (display stack backtrace). You can also set a breakpoint in the code that precedes this stop code and attempt to single-step forward into the faulting code. Use the u, ub, uu (unassemble) commands to see the assembly program code.

0 ·

ChrisB3127-4407 BruceZhang-MSFT · May 03 at 01:12 AM

@BruceZhang-MSFT Thanks for the information! The server went down again today, with the exact same error I have in my original post. I ran the cxr command and it returns this:

rax=0000000000000000 rbx=00000000c0000184 rcx=ffff8705c9264810
rdx=00000000c0000184 rsi=ffff8705cb6ba000 rdi=ffff8705c3517270
rip=fffff8068761ab5f rsp=ffff81014abbeb20 rbp=ffff81014abbec29
r8=0000000000000000 r9=ffff8705c9264810 r10=ffff8705bdb1d0c0
r11=ffff8705c9264810 r12=fffff8068761ab50 r13=ffff8705c9264810
r14=0000000000000004 r15=ffff8705c1919908
iopl=0 nv up ei ng nz na pe nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00050282
HTTP!UxpTpRestartBufferSend+0xf:
fffff806`8761ab5f 488b5008 mov rdx,qword ptr [rax+8] ds:002b:00000000`00000008=????????????????

I will be honest, I'm not a pro in regards to debugging, only that whatever is happening is being caused by HTTP.sys. I do not have a support plan, and would hate to pay $500 for Microsoft to diagnose. Is there anything else I can check? Are you aware of any internal reports with a similar issue? This is an IIS production machine, so I'm wondering if it's somehow related to that that randomly crashes the server.

0 ·

Answer 1 · 2022-04-10T06:09:25.727Z

Docs-4663 answered Apr 10, '22 Docs-4663 converted comment to answer May 3, '22

Please run the DM log collector and post a share link into this thread using one drive, drop box, or google drive.

If the server is able to run the V2 log collector it will collect more files useful for troubleshooting.

https://www.tenforums.com/bsod-crashes-debugging/2198-bsod-posting-instructions.html

https://www.elevenforum.com/t/bsod-posting-instructions.103/

If needed, indicate if there is the option for testing with server downtime.

.
.
.
.
.

Please remember to vote and to mark the replies as answers if they help.

On the bottom of each post there is:

Propose as answer = answered the question

On the left side of each post there is /\ with a number: click = a helpful post
.
.
.
.
.

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ChrisB3127-4407 · May 03 at 01:10 AM

@Docs-4663 Unfortunately, Server 2022 won't run the v2 collector. The other collector doesn't really provide much information. Anything else I can try?

0 ·

Answer 2 · 2022-05-03T01:20:01.38Z

ChrisB3127-4407 answered May 3, '22 ChrisB3127-4407 published May 3, '22

Here is a more complete paste of the dump file analyze:

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the BugCheck
Arg2: fffff8068761ab5f, Address of the instruction which caused the BugCheck
Arg3: ffff81014abbe100, Address of the context record for the exception that caused the BugCheck
Arg4: 0000000000000000, zero.

KEY_VALUES_STRING: 1

 Key  : Analysis.CPU.mSec
 Value: 1906

 Key  : Analysis.DebugAnalysisManager
 Value: Create

 Key  : Analysis.Elapsed.mSec
 Value: 1932

 Key  : Analysis.Init.CPU.mSec
 Value: 358

 Key  : Analysis.Init.Elapsed.mSec
 Value: 3199

 Key  : Analysis.Memory.CommitPeak.Mb
 Value: 109

 Key  : WER.OS.Branch
 Value: fe_release

 Key  : WER.OS.Timestamp
 Value: 2021-05-07T15:00:00Z

 Key  : WER.OS.Version
 Value: 10.0.20348.1

FILE_IN_CAB: MEMORY.DMP

BUGCHECK_CODE: 3b

BUGCHECK_P1: c0000005

BUGCHECK_P2: fffff8068761ab5f

BUGCHECK_P3: ffff81014abbe100

BUGCHECK_P4: 0

CONTEXT: ffff81014abbe100 -- (.cxr 0xffff81014abbe100)
rax=0000000000000000 rbx=00000000c0000184 rcx=ffff8705c9264810
rdx=00000000c0000184 rsi=ffff8705cb6ba000 rdi=ffff8705c3517270
rip=fffff8068761ab5f rsp=ffff81014abbeb20 rbp=ffff81014abbec29
r8=0000000000000000 r9=ffff8705c9264810 r10=ffff8705bdb1d0c0
r11=ffff8705c9264810 r12=fffff8068761ab50 r13=ffff8705c9264810
r14=0000000000000004 r15=ffff8705c1919908
iopl=0 nv up ei ng nz na pe nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00050282
HTTP!UxpTpRestartBufferSend+0xf:
fffff806`8761ab5f 488b5008 mov rdx,qword ptr [rax+8] ds:002b:00000000`00000008=????????????????
Resetting default scope

BLACKBOXBSD: 1 (!blackboxbsd)

BLACKBOXNTFS: 1 (!blackboxntfs)

BLACKBOXPNP: 1 (!blackboxpnp)

BLACKBOXWINLOGON: 1

PROCESS_NAME: w3wp.exe

STACK_TEXT:
ffff8101`4abbeb20 fffff806`875f2a72 : ffff8705`c1919908 00000000`00000004 ffff8705`c9264810 fffff806`8761ab50 : HTTP!UxpTpRestartBufferSend+0xf
ffff8101`4abbeb60 fffff806`876b2256 : ffff8705`c1db8060 00000000`c0000184 ffff8101`4abbec29 ffff8705`c3517270 : HTTP!UlInvokeCompletionRoutine+0x1a
ffff8101`4abbeb90 fffff806`875ca878 : ffff8705`c83dea01 ffff8705`c3517200 ffff8705`000101ac ffff8705`00000000 : HTTP!UxpTpFlushMdlRuns+0x31b76
ffff8101`4abbec90 fffff806`875ca12d : ffff8705`c3517270 ffff8705`c1919908 00000000`00000000 00000000`00000000 : HTTP!UxpTpProcessMdlRuns+0x3c8
ffff8101`4abbed60 fffff806`8768b3b0 : ffff8705`cb6ba000 00000000`0000004d ffff8705`c83dd310 04000000`00002401 : HTTP!UxpTpEnqueueTransmitPacket+0x13d
ffff8101`4abbedb0 fffff806`875c9f49 : ffff8705`c1af00d0 ffff8705`c1919590 00000000`00000000 00000000`00000000 : HTTP!UxTpTransmitPacket+0x200
ffff8101`4abbee80 fffff806`8768b760 : ffff8705`c83dd000 00000000`00000007 00000000`0000000c ffff8705`c1919590 : HTTP!UlSendData+0x129
ffff8101`4abbef40 fffff806`875c9a4e : ffff8705`c83dd000 ffff8705`c1af0001 ffff8705`c83dd200 ffff8705`c83dd270 : HTTP!UlpSendHttpResponseWorker+0x1e0
ffff8101`4abbf020 fffff806`8768324f : 00000000`00000002 00000000`00000002 00000000`00000000 ffff8f0b`2f8672a0 : HTTP!UlSendHttpResponse+0x30e
ffff8101`4abbf120 fffff806`875c2fe4 : 00000000`00000000 ffff8101`4abbfaa0 ffff8f0b`2f8672a0 00000000`00000168 : HTTP!UlSendHttpResponseIoctl+0x1adf
ffff8101`4abbf770 fffff806`71154ed5 : ffff8f0b`2f8672a0 00000000`00000002 00000000`00000000 00000000`00000000 : HTTP!UxDeviceControl+0x84
ffff8101`4abbf7b0 fffff806`71583a69 : ffff8f0b`2f8672a0 00000000`00000000 ffff8f0b`2f8672a0 00000000`00000000 : nt!IofCallDriver+0x55
ffff8101`4abbf7f0 fffff806`71593cb1 : 00000000`00000000 ffff8101`4abbfb20 00000000`0012403f ffff8101`4abbfb20 : nt!IopSynchronousServiceTail+0x189
ffff8101`4abbf890 fffff806`71593d56 : 00000000`00000001 00000000`00000000 00000000`00000000 000001e8`9fd3dcf8 : nt!IopXxxControlFile+0xc61
ffff8101`4abbf9c0 fffff806`71228e35 : 00000000`00000000 00000000`00001900 00000000`00000000 ffff8705`c94ae760 : nt!NtDeviceIoControlFile+0x56
ffff8101`4abbfa30 00007ffb`4944efd4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25
00000084`e21ff048 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffb`4944efd4

SYMBOL_NAME: HTTP!UxpTpRestartBufferSend+f

MODULE_NAME: HTTP

IMAGE_NAME: HTTP.sys

STACK_COMMAND: .cxr 0xffff81014abbe100 ; kb

BUCKET_ID_FUNC_OFFSET: f

FAILURE_BUCKET_ID: AV_HTTP!UxpTpRestartBufferSend

OS_VERSION: 10.0.20348.1

BUILDLAB_STR: fe_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

FAILURE_ID_HASH: {edb15224-7b43-c41e-e441-3f2974ff82b7}

Followup: MachineOwner

0: kd> .cxr 0xffff81014abbe100
rax=0000000000000000 rbx=00000000c0000184 rcx=ffff8705c9264810
rdx=00000000c0000184 rsi=ffff8705cb6ba000 rdi=ffff8705c3517270
rip=fffff8068761ab5f rsp=ffff81014abbeb20 rbp=ffff81014abbec29
r8=0000000000000000 r9=ffff8705c9264810 r10=ffff8705bdb1d0c0
r11=ffff8705c9264810 r12=fffff8068761ab50 r13=ffff8705c9264810
r14=0000000000000004 r15=ffff8705c1919908
iopl=0 nv up ei ng nz na pe nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00050282
HTTP!UxpTpRestartBufferSend+0xf:
fffff806`8761ab5f 488b5008 mov rdx,qword ptr [rax+8] ds:002b:00000000`00000008=????????????????

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 3 · 2022-05-03T07:52:38.907Z

Docs-4663 answered May 3, '22

Please post a share link for the DM log collector.

If additional files are needed I'll comment into this thread.

For dump files the actual files are needed so that commands can run on the debugging software.

No commands can run on the text file.

Indicate if you will or will not be able to arrange server downtime if additional testing is needed.

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 4 · 2022-08-05T07:12:04.94Z

EinarArneSkive-0145 answered Aug 5, '22 ChrisB3127-4407 commented Aug 5, '22

We had the same blue screen on a Windows 2022 server running in IBM cloud. All windows updates applied.

Mini Kernel Dump File: Only registers and stack trace are available

Symbol search path is: srv*
Executable search path is:
Windows 10 Kernel Version 20348 MP (8 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Edition build lab: 20348.1.amd64fre.fe_release.210507-1500
Machine Name:
Kernel base = 0xfffff800`57800000 PsLoadedModuleList = 0xfffff800`58433950
Debug session time: Thu Aug 4 14:50:38.262 2022 (UTC + 2:00)
System Uptime: 1 days 4:56:12.058
Loading Kernel Symbols
...............................................................
................................................................
.....................................
Loading User Symbols
Loading unloaded module list
........
For analysis of this file, run !analyze -v
nt!KeBugCheckEx:
fffff800`57c17df0 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:ffffb806`623e5710=000000000000003b
5: kd> !analyze -v

                     Bugcheck Analysis                                    *

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the BugCheck
Arg2: fffff8005f4aaf6f, Address of the instruction which caused the BugCheck
Arg3: ffffb806623e6040, Address of the context record for the exception that caused the BugCheck
Arg4: 0000000000000000, zero.

Debugging Details:

KEY_VALUES_STRING: 1

 Key  : Analysis.CPU.mSec
 Value: 2483

 Key  : Analysis.DebugAnalysisManager
 Value: Create

 Key  : Analysis.Elapsed.mSec
 Value: 29526

 Key  : Analysis.Init.CPU.mSec
 Value: 343

 Key  : Analysis.Init.Elapsed.mSec
 Value: 50648

 Key  : Analysis.Memory.CommitPeak.Mb
 Value: 95

 Key  : Bugcheck.Code.DumpHeader
 Value: 0x3b

 Key  : Bugcheck.Code.Register
 Value: 0x3b

 Key  : WER.OS.Branch
 Value: fe_release

 Key  : WER.OS.Timestamp
 Value: 2021-05-07T15:00:00Z

 Key  : WER.OS.Version
 Value: 10.0.20348.1

FILE_IN_CAB: 080422-5656-01.dmp

BUGCHECK_CODE: 3b

BUGCHECK_P1: c0000005

BUGCHECK_P2: fffff8005f4aaf6f

BUGCHECK_P3: ffffb806623e6040

BUGCHECK_P4: 0

CONTEXT: ffffb806623e6040 -- (.cxr 0xffffb806623e6040)
rax=0000000000000000 rbx=00000000c0000184 rcx=ffff800be9fd0910
rdx=00000000c0000184 rsi=ffff800bc9b06000 rdi=ffff800bd1ae9aa0
rip=fffff8005f4aaf6f rsp=ffffb806623e6a60 rbp=ffffb806623e6b69
r8=0000000000000000 r9=ffff800be9fd0910 r10=ffff800bca59a380
r11=ffff800be9fd0910 r12=fffff8005f4aaf60 r13=ffff800be9fd0910
r14=0000000000000004 r15=ffff800bc96bfe88
iopl=0 nv up ei ng nz na po nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00050286
HTTP!UxpTpRestartBufferSend+0xf:
fffff800`5f4aaf6f 488b5008 mov rdx,qword ptr [rax+8] ds:002b:00000000`00000008=????????????????
Resetting default scope

BLACKBOXBSD: 1 (!blackboxbsd)

BLACKBOXNTFS: 1 (!blackboxntfs)

BLACKBOXPNP: 1 (!blackboxpnp)

BLACKBOXWINLOGON: 1

CUSTOMER_CRASH_COUNT: 1

PROCESS_NAME: w3wp.exe

STACK_TEXT:
ffffb806`623e6a60 fffff800`5f4831f2 : ffff800b`c96bfe88 00000000`00000004 ffff800b`e9fd0910 fffff800`5f4aaf60 : HTTP!UxpTpRestartBufferSend+0xf
ffffb806`623e6aa0 fffff800`5f542796 : ffff800b`d14898a0 00000000`c0000184 ffffb806`623e6b69 ffff800b`d1ae9aa0 : HTTP!UlInvokeCompletionRoutine+0x1a
ffffb806`623e6ad0 fffff800`5f45acf8 : ffff800b`c9cd4b01 ffff800b`d1ae9a00 ffff800b`0001a5ce ffff800b`00000000 : HTTP!UxpTpFlushMdlRuns+0x31cd6
ffffb806`623e6bd0 fffff800`5f45a5ad : ffff800b`d1ae9aa0 ffff800b`c96bfe88 00000000`00000000 00000000`00000000 : HTTP!UxpTpProcessMdlRuns+0x3c8
ffffb806`623e6ca0 fffff800`5f51b790 : ffff800b`c9b06000 00000000`0000001b ffff800b`c9cd4310 04000000`00002401 : HTTP!UxpTpEnqueueTransmitPacket+0x13d
ffffb806`623e6cf0 fffff800`5f45a3c9 : ffff800b`f6776050 ffff800b`c96bfb10 00000000`00000000 00000000`00000000 : HTTP!UxTpTransmitPacket+0x200
ffffb806`623e6dc0 fffff800`5f51bb40 : ffff800b`c9cd4000 00000000`00000006 00000000`0000000c ffff800b`c96bfb10 : HTTP!UlSendData+0x129
ffffb806`623e6e80 fffff800`5f459ece : ffff800b`c9cd4000 ffff800b`f6776000 ffff800b`c9cd4200 ffff800b`c9cd4270 : HTTP!UlpSendHttpResponseWorker+0x1e0
ffffb806`623e6f60 fffff800`5f51362f : 00000000`00000002 00000000`00000002 00000000`00000000 ffff800b`c91d0030 : HTTP!UlSendHttpResponse+0x30e
ffffb806`623e7060 fffff800`5f452fe4 : ffff800b`c4503d20 fffff800`57a672c9 ffff800b`c91d0030 00000000`00000168 : HTTP!UlSendHttpResponseIoctl+0x1adf
ffffb806`623e76b0 fffff800`57b74aa5 : ffff800b`c91d0030 00000000`00000002 00000000`00000000 00000000`00000000 : HTTP!UxDeviceControl+0x84
ffffb806`623e76f0 fffff800`57fb7b19 : ffff800b`c91d0030 00000000`00000000 ffff800b`c91d0030 00000000`00000000 : nt!IofCallDriver+0x55
ffffb806`623e7730 fffff800`57e9aed1 : 00000000`00000000 ffffb806`623e7a60 00000000`0012403f ffffb806`623e7a60 : nt!IopSynchronousServiceTail+0x189
ffffb806`623e77d0 fffff800`57e9b236 : 00000001`00000000 00000000`00000000 00000000`00000000 000001ee`18e1f938 : nt!IopXxxControlFile+0xc61
ffffb806`623e7900 fffff800`57c2a335 : ffff800b`ca9b1080 000000c3`0c43f9c8 ffffb806`623e7988 000001ee`15ae34e0 : nt!NtDeviceIoControlFile+0x56
ffffb806`623e7970 00007ff9`9ea3f854 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25
000000c3`0c43e638 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ff9`9ea3f854

SYMBOL_NAME: HTTP!UxpTpRestartBufferSend+f

MODULE_NAME: HTTP

IMAGE_NAME: HTTP.sys

IMAGE_VERSION: 10.0.20348.1511

STACK_COMMAND: .cxr 0xffffb806623e6040 ; kb

BUCKET_ID_FUNC_OFFSET: f

FAILURE_BUCKET_ID: AV_HTTP!UxpTpRestartBufferSend

OS_VERSION: 10.0.20348.1

BUILDLAB_STR: fe_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

FAILURE_ID_HASH: {edb15224-7b43-c41e-e441-3f2974ff82b7}

Followup: MachineOwner

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ChrisB3127-4407 · Aug 05 at 09:40 AM

Did you run the latest updates? I've been installing "Patch Tuesday" and the "preview" updates. The preview update at the end of May seems to have resolved this issue... for now..

This is the update that said it improves the reliability of systems in operation for 24 hours a day. So far, I haven't had a BSOD in about 60 days, and I hope it stays that way.

If the issue continues, I guess I'd have no choice but to pay $500 for a support incident with MS. Hoping it's been resolved in one of these updates.

What's interesting, is when it was randomly crashing, it was only the box that runs IIS. My other Server 2022 machine did not have any issues at all like this.

0 ·

question