How Storage Spaces handle bad blocks on physical disks?

Question

I'm looking for technical information on how Storage Spaces are handling physical bad blocks/sectors.

Specifically, Storage Spaces drives (i.e. those you see with Get-VirtualDisk) may be formatted as ReFS or NTFS. The latter can track bad sectors as clusters with the $BadClus file, but given that those 250 MB slabs can be moved on the drive, this file and the whole NTFS operation of checking for bad clusters on a drive pool seems pointless.

The fact that ReFS and NTFS volumes can live in the same drive pool makes it even more complicated, as ReFS interacts with the drive pool much more than NTFS does, it seems.

I cannot find anywhere in Microsoft's docs how drive pools handle bad sectors, except for this one, which describes more of a carpet bombing approach and any write error just fails the entire drive.

https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/jj899886%28v%3Dws.11%29#write-errors

I can see via Get-StorageJob how ReFS runs background storage jobs, checking for bad sectors, but it seems that an NTFS volume on a drive pool has no way to track bad sectors, unless slabs with bad sectors are pinned somehow to retain bad sector locations for the sake of $BadClus.

Please, note that I'm not looking for the common wisdom how drives with bad blocks should be immediately tossed. Just want to know how Storage Spaces handle bad blocks.

If somebody could point me to any information on this or provide some insigghts, it would be greatly appreciated. Thanks!

Answer 1

Hi,

After researching, I didn't find information about how Storage space handle bad block on physical disks, since the questions may be complex and need in-deep research, I'd recommend you open a case with MS to do further research.

Below is the link to open a case with MS:

https://support.microsoft.com/en-us/gp/customer-service-phone-numbers

Thanks for your time!
Best Regards,
Anne

-----------------------------

If the Answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Answer 2

@Xiaowei He

Thank you for the reply, Anne.

It appears from bits and pieces that I gathered in my further research is that Storage Spaces relies on the hard drive to remap bad blocks until it cannot, which it takes as if the drive is gone, hence the entire drive fails.

This is a very old thread that describes in good detail how hard drives reallocate bad sectors and they need a write operation against the bad sector to reallocate (remap) it to a spare sector.

https://superuser.com/questions/384095/how-to-force-a-remap-of-sectors-reported-in-s-m-a-r-t-c5-current-pending-sector/688764

From the link in my original post, Storage Spaces will attempt to write a copy of data to the same location after it failed to read it, which appears to be intended as an attempt to trigger a sector reallocation within the hard drive, even though it is not described in this detail. If that's how it is, maybe Microsoft can expand the note on that page to include the intended behavior against a SMART drive.

From the practical standpoint, one can use smartctl on Linux to see whether there are any pending sectors with read errors on the hard drive and whether the reallocated sector count is getting close to the fail point (i.e. how close its normalized value is to the failure threshold).

For one of my experimental hard drive with bad sectors that Storage Spaces had trouble reading from, I could see via smartctl that there were pending sectors, but it took a long time for Storage Spaces to go through even some of those bad sectors. After a while, I removed the drive from the Storage Spaces pool and ran dd against it on Linux, which reallocated all bad sectors.

Unfortunately, I didn't capture the before picture of how many bad sectors this drive had when it was in the storage pool, so I cannot say how many of those were fixed by the storage pool before I ran dd. It also seems that the storage pool needs a read operation from the file system to initiate all these steps and ReFS will do this periodically when it's hunting for bit rot, but for NTFS, it appears that one needs to run chkdsk /B once in a while to trigger those reads and then again to empty $BadClus after bad sectors have been reallocated, but I did not verify this (e.g. how chkdsk /B will behave on a thinly-provisioned Storage Spaces virtual disk).

Anyway, creating a support case probably won't do much good because the first level Support will not dig too deep into this. Hopefully someone from the Storage Spaces team or someone who experimented more with Storage Spaces than I did stumbles upon this thread and clarifies some of these things.

Answer 3

Maybee I have so partial Information. When Storage Spaces came out I did a lot of testing how Storage Spaces behaves when parts of Data are going bad.

I tested: pull a disc, take Power Cord Off, all when doing write operations at the same time, manipulating Sectors of discs when vm was running/not running and so on. Every scenario that usually can happen. Nothing of all these things could really harm the Volume. Sure it went degraded but rebuilding was easy and always correct. No corrupt files. I tested with Mirrored Volumes and ReFs.

I tested the bad sector handling with following scenario:
Virtual Discs, mounted in VmWare with a ParaVirtual Controller to make it possible to add a disc to different VM's, powered on the same time

First I added the disc to a Windows Server 2012 and produced some data, on a volume with Storage Spaces
Next I added the disc also to a second VM and modified the sectors of the discs directly with a Hex-Editor (This does normally not work with a modern OS as direct hardware acces is locked, so I used XP if I can good remember). On next Storage-Checkup the bad sector was corrected with the data of the mirror partner. I have no idea how Storage Spaces really knows which one is correct, but it took always the right one. What was interesting, manipulated NTFS Volumes were also corrected the right way. It seems that a lot of these things are not bound with the Filesystem itself but with Storage Spaces.

What was also interessting is the fact that even NTFS was not currupt when I pulled the power cord and brought everything down when doing Write-Operations. On the other hand same test with a NTFS volume direct on the disc made problems. For me it seems that a NTFS-Volume on Storage Spaces does benefit also of some enhancements of ReFs. But these Full-Power-Down tests I did not that much with NTFS, first goal was to stress ReFs volumes max. So It could be also some kind of luck. Id did the same tests later with Deduplication On. No change at all. In my opinion Storage Spaces is bloody robust since the first hours.
Edit: Thats what makes 3-way mirror so cool, Storage Spaces will be able to handle such silent failures correct too. So full disc failure and also some bad sectores which were not found at the right moment on the mirror partner.

What I do not know is how a physical bad sector works. If it works the same way than a logically modified sector, than storage spaces will handle it correct.

These tests were the reason that I changed a lot from hardware raid-controllers to Software Raid with based on Storage Spaces. Directly from the beginning when it came out. First two years I used it just for VDI and some high performance things. To have a real life test. No problems at all. ESXi>windows as storage app>NFS>ESXi on the same machine (with NTFS, as ReFs has to long FileID's to offer it with NFS, you would need a NFS-Server with its own Filetable). It was really enormous fast and blowed my SAN far away because of low latency and SSD's. Now with NVMe it is even Faster. =)

The only thing that was really bad and does really suck was the "full disc-bug" of ReFs. So when it runs out of space. These was really a pain. But I think these problems are almost gone today. Before it could always happen because it is not the complete free space that counts. But these is another thing than what you asked.
EDIT: Well there is another thing too. Rebuilding with a disc that comes back and still has Data is not 100% sure it takes the Data which was always online as "reliable". I personally did never had this probelm, but other people had.

Tests I've done with high end SSD's and also with usual discs. Well for usual discs, write operations are a real pain because the lack of a good and especially working write-buffer. So I would always go with high quality SSDs with a really good average and a low max. latency on Storage Spaces.

Answer 4

@WineIsFine

Thanks for the feedback. Lots of interesting observations.

I tested: pull a disc, take Power Cord Off, all when doing write operations at the same time, manipulating Sectors of discs when vm was running/not running and so on. Every scenario that usually can happen. Nothing of all these things could really harm the Volume.

I wouldn't go as far, at least for NTFS volumes. One thing Storage Spaces lacks big time is the eject functionality (it errors out for Storage Spaces). At first, years ago when it just came out, I thought Windows was smart enough to handle writes without having to eject, but that wasn't the case and I lost some data. This is not an issue for desktops and servers, but connecting a drive pool to a laptop certainly causes mobility problems.

I solved it by setting -IsManualAttach on the virtual drive for connecting/disconnecting storage pools manually, with a mandatory volume flush via Write-VolumeCache before disconnecting. I tested this by writing gigs worth of data to a local drive and a pool drive, flushing and disconnecting and binary-comparing both files after and it always worked, so this is what I have been using for a few years.

These tests were the reason that I changed a lot from hardware raid-controllers to Software Raid with based on Storage Spaces.

Indeed. I dumped hardware RAID after one of my units failed and the manufacturer said they no longer had these older models and the new ones have different chipset and are not compatible with my model. Anyone who buys hardware RAID nowadays is asking for trouble, unless they buy 3x parts or order from enterprise-level manufacturers that will honor warranty for years.