question

PaulRF-6259 avatar image
1 Vote"
PaulRF-6259 asked Sean-Liming commented

Win 10 2019 with Universal Write filter Blue Screen

I have a bunch of Dell Wyse thin clients with Win 10 2019 LTSC installed with the Universal Write Filter (UWF) installed, that seem to randomly have a Blue Screen. Unfortunately I have not managed to catch a local device to witness what the crash dump code is. I have added %SystemRoot%\MEMORY.dmp to the UWF exclusion so that it is maintained and have the system failure settings set to record an "Automatic Memory Dump". however when the machine BSOD there are event log errors "Dump File Creation failed due to an error during dump creation (Evt ID 161)" and "The system could not successfully load the crash dump driver (Evt ID 45)" and "Crash dump initialization failed (Evt ID 46)"

I have seen elsewhere on this forum a recommendation to ensure that no pagefile is created on the UWF protected drive and this is teh configuration I have set.

Any help or suggestions on how to fix this problem so I can work out what is causing teh BSOD would be gratefully received.

windows-10-generalwindows-iot-10core
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi

When it comes to blue screen , this link maybe help you .
Troubleshoot blue screen errors
https://support.microsoft.com/en-us/sbs/windows/troubleshoot-blue-screen-errors-5c62726c-6489-52da-a372-3f73142c14ad

In short , we could try to:
Uninstall third-party software
Rollback, disable, or uninstall drivers
Disable third-party drivers
Uninstall third-party drivers
Remove external hardware

Hope the above information could help you.
Best Regards

0 Votes 0 ·

Hi Miles - Thanks for your reply, The device has no additional hardware and only the necessary system drivers installed, from Dell's web site. The only application installed on the device is Citrix workspace app, so unfortunately there isn't much scope for troubleshooting here which is why I really need the crash dump. Also I have no idea what the bug check code is as it is not being recorded in the event log.

0 Votes 0 ·

Hi

Sorry for the late response.
We notice that you have tried some methods to fix the issue.
Does the problem still exists?
If you resolve it using your own solution, you could share your experience and solution here.
It will be very beneficial for other community members who have similar questions.

Best Regards

0 Votes 0 ·
Show more comments
yagmoth555 avatar image
0 Votes"
yagmoth555 answered PaulRF-6259 commented

Hi

If you disable UWF does the Blue Screen happen ?

If you activate UWF cache on disk, and not memory does the same happen ?

I ask as if you have big update and low space, your device might have limited RAM to work on.

It happened to me on HP Thin Client, and running MS Team was using a lot of heap, and when it when caching to memory it caused a segfault. Caching to HDD it don't happen.

On Win10 LTSC 2019 you are lucky as new overlay option exist, like; wfmgr overlay set-passthrough on. That can help to prevent the cache filling problem.

Thanks

Philippe

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Philippe, Thanks for your answer. I am in the unenviable position of not actually having witnessed the crash myself and I cant find a way to reproduce it to test if disabling UWF works.

I have the following config set
Overlay Set-Type RAM
Overlay Set-Size 2048
Overlay Set-WarningThreshold 1638
Overlay Set CriticalThreshold 1843

Plus various folder and registry exclusions as recommended by MS and to allow event logs, defender updates etc. The only application installed is Citrix Workspace app, and noone has reported that they have been prompted to reboot before a crash.

Do you think these values are sufficient as I'd like to avoid using disk overlay on the ssd if possible.

thanks

0 Votes 0 ·
Sean-Liming avatar image
0 Votes"
Sean-Liming answered

The only way to capture memory dumps is to put the dumps on a non-UWF protected partition. This means on start up you will have to change the location to the partition. If the BSOD is happening during startup then finding the issue becomes a real challenge to be there when the BSOD occurs. You might want to disable auto reboot on BSOD.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Docs-4663 avatar image
0 Votes"
Docs-4663 answered Docs-4663 edited

Hi PaulRF-6259 ,

1) Open administrative command prompt (ACP) and type or copy and paste:
2) sfc /scannow
3) dism /online /cleanup-image /scanhealth
4) dism /online /cleanup-image /restorehealth
5) sfc /scannow
6) chkdsk /scan
7) wmic recoveros set autoreboot = false
8) wmic recoveros set DebugInfoType = 7
9) wmic recoveros get autoreboot
10) wmic recoveros get DebugInfoType
11) wmic Computersystem where name="%computername%" set AutomaticManagedPagefile=True
12) wmic Computersystem where name="%computername%" get AutomaticManagedPagefile
13) bcdedit /enum {badmemory}

14) When these have completed > right click on the top bar or title bar of the administrative command prompt box > left click on edit then select all > right click on the top bar again > left click on edit then copy > paste into the thread


15) After multiple BSODs with the above settings run the V2 log collector and post a share link into this thread:

https://www.windowsq.com/resources/v2-log-collector.8/
https://www.tenforums.com/bsod-crashes-debugging/2198-bsod-posting-instructions.html
https://www.elevenforum.com/t/bsod-posting-instructions.103/



.
.
.
.
.

Please remember to vote and to mark the replies as answers if they help.

On the bottom of each post there is:

Propose as answer = answered the question

On the left side of each post: Vote = a helpful post
.
.
.
.
.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Huskin1-3610 avatar image
0 Votes"
Huskin1-3610 answered Sean-Liming commented

@PaulRF-6259 I don't have a habit of replying to random posts I find during my own Google searches but after reading your post I felt I had to assist as it instantly gave me war flashbacks. We too are using Dell Wyse W10IoT devices (mobility 5470 in my case) running a mix of 2016 LTSB and 2019 LTSC and we have had (and still have) our share of headaches regarding the write filter. We also had a situation where we had BSOD's all the time on the 2016 LTSB builds, we contacted Dell but they could not help us so we turned to Microsoft.

The people who have already replied to this post have good intentions but might lack some experience in terms of UWF and its behavior. My first guess an UWF-exclusions causing damage to one of the critical system files or registry hives. However, we too are experiencing the odd BSOD with the Dell 2019 LTSC builds, the number of occurrences is so low that we have not yet bothered to troubleshoot it further but these BSODs don't seem to occur with the same UWF-exclusions on the Dell 2016 LTSB builds.

Would you mind posting your UWF-exclusions?

Enabling the write filter disables, among many things, default Windows functionality that could interfere with its internals. Some examples of this are page files, system restore points, defrag, indexing, fastboot, .... The reason why the memory dump on the device doesn't work is simple, the paging file is responsible for this, which is not active when UWF is enabled.

Side-note, if you don't get any specific blue screen codes when the crash occurs, you can inject the following reg values (with UWF off) to get some more info.

REG ADD "HKLM\System\CurrentControlSet\Control\CrashControl" /v DisplayParameters /t REG_DWORD /d 1 /f
REG ADD "HKLM\System\CurrentControlSet\Control\CrashControl" /v DisplayDisabled /t REG_DWORD /d 0 /f

So how does one get information about blue screens when it occurs? Download and install the Windows Debugger Tools (included in Windows AIK) on a second machine, preferably a Windows device without UWF. I got all the info I needed to get it working right here. The annoying thing here of course is that you need the second device online and you need to trigger the BSOD. To my knowledge there is no other way to get additional crash dump information.

In my case a forceful shutdown off (pressing the power button until the device powers off) was causing the BSODs and it was linked to a bad UWF-exclusion I had configured due to my inexperience with UWF and lack of understanding of its internals.

I hope this helps you or anyone reading this.

· 13
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Huskin, Thanks for the reply

I am pretty certain that the crash is being caused by the screen going to sleep. If I set the power options to turn the screen off at 1 hour the crashes happen at 62 mins apart if I set it to 15 mi s they are 17 mins apart.

We have an additional graphics (AMD) card supplied in the Wyse 5070s and I have a feeling it is definitely somthing to do with these as the same devices that dont have this card in dont suffer with the issue. I have obviously tried later and older drivers for both this and the on-board Intel card without much success. I have also disabled the intel card driver.

The UWF config is

UWFMgr Volume Protect C:
UWFMgr Overlay Set-Type RAM
UWFMgr Overlay Set-Size 2048
UWFMgr Overlay Set-CriticalThreshold 1843
UWFMgr Overlay Set-WarningThreshold 1638

SCCM

UWFMgr File Add-Exclusion C:\Windows\CCM\Logs
UWFMgr File Add-Exclusion C:\ProgramData\Microsoft\Crypto
UWFMgr Registry Add-Exclusion "HKLM\SOFTWARE\Microsoft\SystemCertificates\SMS\Certificates"

Daylight Saving

UWFMgr Registry Add-Exclusion "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time Zones"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation"

0 Votes 0 ·

Defender


UWFMgr File Add-Exclusion "C:\Program Files\Windows Defender"
UWFMgr File Add-Exclusion "C:\ProgramData\Microsoft\Windows Defender"
UWFMgr File Add-Exclusion "C:\Windows\Temp\MpCmdRun.log"
UWFMgr File Add-Exclusion "C:\Windows\WindowsUpdate.log"
UWFMgr Registry Add-Exclusion "HKLM\SOFTWARE\Microsoft\Windows Defender"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Services\WdBoot"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Services\WdFilter"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Services\WdNisSvc"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Services\WdNisDrv"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Services\WinDefend"

Screen position

UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Control\GraphicsDrivers"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Control\UnitedVideo"

SCEP Exclusions


UWFMgr File Add-Exclusion "C:\Program Files\Microsoft Security Client"
UWFMgr File Add-Exclusion "C:\ProgramData\Microsoft\Microsoft Antimalware"
UWFMgr Registry Add-Exclusion "HKLM\SOFTWARE\Microsoft\Microsoft Antimalware"

Windows Event Logs

UWFMgr File Add-Exclusion "C:\Windows\System32\Winevt\Logs"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\WPA"


0 Votes 0 ·

Are those your only exclusions? Nothing set for wifi or displays?

What you just told me is quite positive, it means that you can trigger the BSOD so you can get the crash dump you're looking for using the Windows Debugger. It's a bit of a hassle but it gets the job done. Good luck.

0 Votes 0 ·

No Been struggling to get the forum to allow me to post more. If you refresh you'll see the rest


0 Votes 0 ·

2nd part of the exclusion is...


Defender


UWFMgr File Add-Exclusion "C:\Program Files\Windows Defender"
UWFMgr File Add-Exclusion "C:\ProgramData\Microsoft\Windows Defender"
UWFMgr File Add-Exclusion "C:\Windows\Temp\MpCmdRun.log"
UWFMgr File Add-Exclusion "C:\Windows\WindowsUpdate.log"
UWFMgr Registry Add-Exclusion "HKLM\SOFTWARE\Microsoft\Windows Defender"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Services\WdBoot"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Services\WdFilter"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Services\WdNisSvc"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Services\WdNisDrv"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Services\WinDefend"

Screen position

UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Control\GraphicsDrivers"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\CurrentControlSet\Control\UnitedVideo"

SCEP Exclusions


UWFMgr File Add-Exclusion "C:\Program Files\Microsoft Security Client"
UWFMgr File Add-Exclusion "C:\ProgramData\Microsoft\Microsoft Antimalware"
UWFMgr Registry Add-Exclusion "HKLM\SOFTWARE\Microsoft\Microsoft Antimalware"

Windows Event Logs

UWFMgr File Add-Exclusion "C:\Windows\System32\Winevt\Logs"
UWFMgr Registry Add-Exclusion "HKLM\SYSTEM\WPA"

0 Votes 0 ·

We don't seem to share many exclusions. Remember less is more when it comes to UWF-exclusions as they can cause more problems than they solve. Regarding the SCEP exclusions, they seem to be somewhat different from what Microsoft says in their common write filter exclusions. You also have an abundance of Windows Defender exclusions.

You also seem to be missing all of the Dell defaults stored in (HKLM\SYSTEM\CurrentControlSet\Services\uwfvol\parameters\static\copy0\Volumes\0\FileExceptionsUserDefined and HKLM\SYSTEM\CurrentControlSet\Services\uwfvol\parameters\static\copy0\RegistryExceptionsUWFSpecific) but I guess those exclusions were implied as they come with the system.

If I were you I would just do the process of elimination, Just remove all your custom exclusions one by one on a test device. If at some point it doesn't happen anymore you have your answer. If by the end you're only left with the system default exclusions you can be quite sure it has nothing to do with the exclusions and you can try the Windows Debugger.



0 Votes 0 ·

Thanks for replying,
I have tried disabling the write filter on a couple of devices and the issue is still occurring, so I don't think it is anything do do with the UWF exclusions.

0 Votes 0 ·
Show more comments
Docs-4663 avatar image
0 Votes"
Docs-4663 answered

Please see the post dated Jul 29 2021.
Please run the V2 log collector and post share links into this thread using one drive, drop box, or google drive.

.
.
.
.
.

Please remember to vote and to mark the replies as answers if they help.

On the bottom of each post there is:

Propose as answer = answered the question

On the left side of each post: Vote = a helpful post
.
.
.
.
.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.