question

jubileum-4233 avatar image
2 Votes"
jubileum-4233 asked PolkCarl-0048 commented

BSOD dedup.sys after inplace upgrade server2016-2019

After an inplace upgrade van server 2016 to server 2019 and fully patching the 2019 server it keeps coming with a BSOD on dedup.sys. Further analysis shows that it seems to happen when MP4, AVI or PST files are opened which are over 1GB. Possibly excluding these extentions help (currently testing this temporary fix)

The same happens on the VEAAM backup server trying to restore one of these files to a randomly choosen location (even local not dedupped disc) because it uses VSS on the offending fileserver.

windows-server-2019
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PolkCarl-0048 avatar image
1 Vote"
PolkCarl-0048 answered Francisco-7527 commented

Microsoft confirmed this is a known bug and will release this July
This is a known bug which will be fixed in this month's Preview Update (Jul 13) When published it will be under KB5004244 for Server 2019.

· 11
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Is this Preview Update confirmed? We're having similar problems, but looks like we're already past KB5004244, as last one on July 6 is KB5004947. Hopefully this fix is in the July 13 Preview Update as indicated.

1 Vote 1 ·

Hi SteveRai,

Did you happen to have done an in place upgrade from 2016 to 2019 as well?

0 Votes 0 ·

Actually, no. In our case, we were hosting the volumes on 2016, the server crashed and we decided to spin up a 2019 server and re-attached the volumes... a couple of days passed and we noticed some issues with copying some of the larger files out.

0 Votes 0 ·

Mine was in place upgrade. funny thing is i am not seeing it as much on virtual machines but on my servers with SSDs.

0 Votes 0 ·

I had this problem this week and had all the latest updates for server 2019, but it still got a BSOD. What I ended up doing to fix it was to build a 2016 VM, then mounted the VHDX from the 2019 server, and ran the following Powershell command on it

 Start-DedupJob -Type Unoptimization -Volume E:

After unoptimization was complete I shut down the 2016 VM and started back up the 2019, then no more BSOD

0 Votes 0 ·

jJust make sure one does a

get-dedupstatus | fl

and look for UnoptimizedSize as your free disk space needs to be able to accommodate once one runs the job Marc mentioned.

0 Votes 0 ·

This only works when the files are not optimzed under 2019, otherwise a version mismatch occurs (Also when you try to restore with Veeam to an 2016 server)

0 Votes 0 ·

-kb5004244 is out

Looking at the file contents it contains a dedup.sys file

File Version, Date,Time,File Size
10.0.17763.2061,July 5,2021,11:25,599,432

II currently have version 10.0.17763.1971 so hopefully they fixed it. good luck everyone.

1 Vote 1 ·

Just tested and confirmed this fixed the issue for me. I since unoptimized my deduped volumes, disabled dedupe and then re-enabled it and re-ran optimization/garbage collection/scrubbing and it's looking good.

0 Votes 0 ·

Looking good for us as well. There were some installer files and ISOs we had that would consistently crash the entire OS when clicked, and they are all OK now.

1 Vote 1 ·

hey there! wow, this is great, thanks so much. I noticed you posted this back in July 1 but the preview update was only released on the 13th. where can I get the KB info early? is there a site with information on unreleased updates that I can check, like for this case? thanks again!

0 Votes 0 ·
Miles-MSFT avatar image
1 Vote"
Miles-MSFT answered jubileum-4233 edited

Hi

In order to solve BSOD problem , we could doenload windbg first.
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools

After the installation , please reappear the issue then we could analyse the dump.

Best Regards

--please don't forget to Accept as answer if the reply is helpful--

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Unfortunately I am to busy to recover >5TB damaged data to follow these steps, thanks to Callum-3836 who has the same issue and had time to provide input. I am now awaiting the KB5004244 months preview with the solution for this confirmed bug.

0 Votes 0 ·
Callum-3836 avatar image
0 Votes"
Callum-3836 answered Callum-3836 published

Hi @Miles-MSFT , I've also started to experience this issue. Whenever I attempt to access, copy or move the data on the drive (mainly VHDs for Hyper-V) or also when I attempt to start-dedupjob -type Unoptimize the system will crash with 0x07E dedup.sys

Spun up a clean VM of 2016 and 2019, 2016 reports unknown revision when trying to access the data, 2019 crashes.




 Microsoft (R) Windows Debugger Version 10.0.20348.1 AMD64
 Copyright (c) Microsoft Corporation. All rights reserved.
    
    
 Loading Dump File [C:\Windows\MEMORY.DMP]
 Kernel Bitmap Dump File: Kernel address space is available, User address space may not be available.
    
 Symbol search path is: srv*
 Executable search path is: 
 Windows 10 Kernel Version 17763 MP (6 procs) Free x64
 Product: Server, suite: TerminalServer DataCenter SingleUserTS
 Edition build lab: 17763.1.amd64fre.rs5_release.180914-1434
 Machine Name:
 Kernel base = 0xfffff807`7f2b9000 PsLoadedModuleList = 0xfffff807`7f6d4610
 Debug session time: Sun Jun 13 16:23:15.043 2021 (UTC + 1:00)
 System Uptime: 0 days 2:46:45.819
 Loading Kernel Symbols
 ...............................................................
 ................................................................
 .........................................
 Loading User Symbols
    
 Loading unloaded module list
 ..........
 For analysis of this file, run !analyze -v
 5: kd> !analyze -v
 *******************************************************************************
 *                                                                             *
 *                        Bugcheck Analysis                                    *
 *                                                                             *
 *******************************************************************************
    
 SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
 This is a very common bugcheck.  Usually the exception address pinpoints
 the driver/function that caused the problem.  Always note this address
 as well as the link date of the driver/image that contains this address.
 Arguments:
 Arg1: ffffffffc0000094, The exception code that was not handled
 Arg2: fffff804096f2f29, The address that the exception occurred at
 Arg3: ffffd68d498df2a8, Exception Record Address
 Arg4: ffffd68d498deaf0, Context Record Address
    
 Debugging Details:
 ------------------
    
    
 KEY_VALUES_STRING: 1
    
     Key  : Analysis.CPU.mSec
     Value: 1436
    
     Key  : Analysis.DebugAnalysisProvider.CPP
     Value: Create: 8007007e on WIN-OFG109Q4O2D
    
     Key  : Analysis.DebugData
     Value: CreateObject
    
     Key  : Analysis.DebugModel
     Value: CreateObject
    
     Key  : Analysis.Elapsed.mSec
     Value: 18563
    
     Key  : Analysis.Init.CPU.mSec
     Value: 1452
    
     Key  : Analysis.Init.Elapsed.mSec
     Value: 48816
    
     Key  : Analysis.Memory.CommitPeak.Mb
     Value: 89
    
     Key  : Analysis.System
     Value: CreateObject
    
     Key  : WER.OS.Branch
     Value: rs5_release
    
     Key  : WER.OS.Timestamp
     Value: 2018-09-14T14:34:00Z
    
     Key  : WER.OS.Version
     Value: 10.0.17763.1
    
    
 ADDITIONAL_XML: 1
    
 OS_BUILD_LAYERS: 1
    
 VIRTUAL_MACHINE:  VMware
    
 BUGCHECK_CODE:  7e
    
 BUGCHECK_P1: ffffffffc0000094
    
 BUGCHECK_P2: fffff804096f2f29
    
 BUGCHECK_P3: ffffd68d498df2a8
    
 BUGCHECK_P4: ffffd68d498deaf0
    
 EXCEPTION_RECORD:  ffffd68d498df2a8 -- (.exr 0xffffd68d498df2a8)
 ExceptionAddress: fffff804096f2f29 (dedup!DdpStreamFirstStreamMapEntry+0x000000000000011d)
    ExceptionCode: c0000094 (Integer divide-by-zero)
   ExceptionFlags: 00000000
 NumberParameters: 0
    
 CONTEXT:  ffffd68d498deaf0 -- (.cxr 0xffffd68d498deaf0)
 rax=0000000c38400000 rbx=0000000000000000 rcx=0000000000000000
 rdx=0000000000000000 rsi=0000000c78400000 rdi=000000003fff8b47
 rip=fffff804096f2f29 rsp=ffffd68d498df4e0 rbp=0000000080000000
  r8=0000000000003114  r9=0000000000003115 r10=ffff8b8238eb54e0
 r11=ffffd68d498df4b0 r12=0000000040000000 r13=0000000000003114
 r14=0000000040000000 r15=0000000000000000
 iopl=0         nv up ei pl zr ac po cy
 cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00010257
 dedup!DdpStreamFirstStreamMapEntry+0x11d:
 fffff804`096f2f29 48f7f9          idiv    rax,rcx
 Resetting default scope
    
 BLACKBOXBSD: 1 (!blackboxbsd)
    
    
 BLACKBOXPNP: 1 (!blackboxpnp)
    
    
 PROCESS_NAME:  System
    
 ERROR_CODE: (NTSTATUS) 0xc0000094 - {EXCEPTION}  Integer division by zero.
    
 EXCEPTION_CODE_STR:  c0000094
    
 EXCEPTION_STR:  0xc0000094
    
 STACK_TEXT:  
 ffffd68d`498df4e0 fffff804`096f34e5     : ffffc505`66a45be0 ffff8b82`00003114 ffffc505`00003115 00000000`00000000 : dedup!DdpStreamFirstStreamMapEntry+0x11d
 ffffd68d`498df590 fffff804`097410ca     : 00000000`00000000 ffffd68d`498df910 ffffd68d`498df8d0 00000000`00000000 : dedup!DdpTLCRead+0x219
 ffffd68d`498df850 fffff804`0970a291     : 00000000`00400000 00000000`00000000 ffffc505`66540e10 ffff8b82`00000000 : dedup!DdpHsmReadStream+0x1ca
 ffffd68d`498df910 fffff804`0970a8df     : 00000000`0000000c ffffd68d`498dfaf0 ffff8b82`3f5b9001 fffff807`7f3b5401 : dedup!HsmiReadCommon+0xfcd
 ffffd68d`498dfa80 fffff804`096f8f72     : 00000000`00000000 ffff8b82`38d57960 ffffffff`00000010 00000002`00000005 : dedup!HsmiReadDispatch+0xf
 ffffd68d`498dfab0 fffff804`07f18380     : 00000000`00000000 ffff8b82`452c9b90 00000000`00000000 00000000`00000000 : dedup!HsmiDispatchUserRequest+0x52
 ffffd68d`498dfae0 fffff807`7f32dcba     : ffff8b82`38d57960 ffff8b82`3f5b9040 fffff804`07f182e0 ffff8b82`452c9b98 : FLTMGR!FltpProcessDeferredIoWorkItem+0xa0
 ffffd68d`498dfb70 fffff807`7f2f6bf5     : ffff8b82`3f5b9040 ffff8b82`38c75080 ffff8b82`3f5b9040 00000000`00000000 : nt!ExpWorkerThread+0x16a
 ffffd68d`498dfc10 fffff807`7f4744bc     : ffffa201`62925180 ffff8b82`3f5b9040 fffff807`7f2f6ba0 00000000`00000000 : nt!PspSystemThreadStartup+0x55
 ffffd68d`498dfc60 00000000`00000000     : ffffd68d`498e0000 ffffd68d`498da000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x1c
    
    
 SYMBOL_NAME:  dedup!DdpStreamFirstStreamMapEntry+11d
    
 MODULE_NAME: dedup
    
 IMAGE_NAME:  dedup.sys
    
 STACK_COMMAND:  .cxr 0xffffd68d498deaf0 ; kb
    
 BUCKET_ID_FUNC_OFFSET:  11d
    
 FAILURE_BUCKET_ID:  AV_dedup!DdpStreamFirstStreamMapEntry
    
 OS_VERSION:  10.0.17763.1
    
 BUILDLAB_STR:  rs5_release
    
 OSPLATFORM_TYPE:  x64
    
 OSNAME:  Windows 10
    
 FAILURE_ID_HASH:  {c3859490-b50b-4dc5-9bb7-a257e189e85d}
    
 Followup:     MachineOwner
 ---------



5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Docs-4663 avatar image
1 Vote"
Docs-4663 answered Callum-3836 edited

Please run the V2 log collector so files can be troubleshooted:

https://www.windowsq.com/resources/v2-log-collector.8/
https://www.tenforums.com/bsod-crashes-debugging/2198-bsod-posting-instructions.html


Open file explorer> this PC > C: > in the right upper corner search for: C:\Windows\memory.dmp

if the file size is < 1.5 GB then > save to the downloads folder > zip > post a separate share link into the thread using one drive, drop box, or google drive


.
.
.
.
.

Please remember to vote and to mark the replies as answers if they help.

On the bottom of each post there is:

Propose as answer = answered the question

On the left side of each post: Vote = a helpful post
.
.
.
.
.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi, thanks for the response, please find the zipped log file here


0 Votes 0 ·
Docs-4663 avatar image
1 Vote"
Docs-4663 answered JosephM-9745 edited

@Callum-3836 ,

This thread was opened by jubileum-4233.
Results can be confusing when there are results from multiple computers.
Please open a separate thread so that it will have data from only one computer.
Place a link into this thread.

There were 3 bugchecks: 7E, 1E, and 3B.

All had exception code: Integer division by zero

If there is no option for downtime then post share links for memory dumps.

If there are opportunities for downtime then testing can be performed using Windows Driver Verifier (WDV).

If there is downtime instructions for WDV can be posted.


.
.
.
.
.

Please remember to vote and to mark the replies as answers if they help.

On the bottom of each post there is:

Propose as answer = answered the question

On the left side of each post: Vote = a helpful post
.
.
.
.
.




· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you, I've posed here

The drive in question was mainly part of a lab environment so downtime is possible with no lasting repercussions


0 Votes 0 ·

Hello there! We just started running into this issue on our production file server. The problem seems to be sporadic; I noticed for example when clicking on certain ,exe installer files it would crash the OS. Not even copying or opening them, just highlighting them!

We also performed an in place upgrade from Server 2016 but that was months ago and we saw no issues after the upgrade. I specifically spent some time opening up various files. This issue seems to have only started for us within the last week or two. Is it possible a previous update broke something?

I understand there will be a patch for this issue but I am concerned whether or not any of our data has been damaged.

1 Vote 1 ·
TDeman avatar image
1 Vote"
TDeman answered PolkCarl-0048 commented

Hello,

this error has occurred in my lab on one of my HyperV server where I had activated the dedup (a long time ago).

The system was stopping regularly just before I could do anything.

  • I disconnected the disks/volumes where Dedup was active.

  • I installed the dedup feature on another Windows 2019 system patched with the last security patch of 13/7/2021.

  • Then I renamed the old "dedup.sys" (after changing the security), and put the new version coming from the patched machine.
    (Version 10.0.11763.2061).

  • After a correct restart, I was able to install the full patch on the HyperV server.

  • Then I was able to reconnect all the disks and all necessary VMs are now started.

It seems to work.
(And now I know that I cannot disable the deduplication: no enough free place )

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

thank you Tdeman,

this solution saved me,

0 Votes 0 ·

no need to"hack" your OS. Microsoft provided a fix for this in KB5004244. Not documented but a newer dedup.sys is in there.

0 Votes 0 ·