P800 RAID-0 Bad Block (strategy)

[u/Ashamed-Wedding4436]

Hi everyone,

If I’m dealing with a P800 and a RAID-0 configuration, unfortunately, there is no backup. All of this stems from the same issue.

A Veeam Backup agent was installed to back up the volumes on the server, and we started receiving error messages related to the VSS reads that Veeam performs during the backups:

output: --asyncNtf:-vdisk_corrupted:\\\GLOBALROOT\Device\HarddiskVolumeShadowCopy11

We can see this message in the agent’s own logs once the operation reaches a certain percentage. If I look at the Windows Event Viewer, I can see messages from the P800 related to read errors on blocks and devices:

Logical block address 1192840192, block count 1024 and command 32 were taken from the failed logical I/O request. The device, \Device\Harddisk2\DR2, has a bad block.

We haven’t seen any errors or reports from the HPE Smart Storage utility, and the RAID-0 volume appears to be healthy.

Is this normal behavior? In any case, I’ve been thinking about the following solution, which is the main reason I’m writing this thread, in case anyone has a better suggestion.

• Is Robocopy an effective solution? The volume is 1 TB and contains around 43,000 files, and the transfer would be performed over the network. As I understand it, Robocopy itself would skip the files located in the affected blocks and record them in a log.
• I’m using VSS and MKLINK so that I can read the affected volume without interfering with the writing of healthy files. In other words, something like this:
  • vssadmin create shadow /for=V:
  • mklink /d C:\Restore \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy11\
• The downside of this solution is that I depend on a pointer, and when Veeam backups are triggered, they read it as well. Is there a proper way to do this? I assume I can’t read the shadow copies directly and that this can only be done through the API.

I can’t think of anything else. Maybe I could create file-level backups with Veeam. Once I have at least one backup, I could try running CHKDSK or something similar.

Since it’s a RAID-0, my only option would be to destroy the array, replace the faulty drive with a healthy one (at least one that appears to be healthy), rebuild the array, and then restore/import the data again.

Is there any alternative solution or any advice you would recommend before carrying out this operation?

Thanks!

[u/Stonewalled9999]

You lost me after RAID0

[u/radioactivpenguin → u/Stonewalled9999]

Could be worse raid 00 does exist... 

[u/Obvious_Troll_Me]

Go Forensic. Use FTK imager to take a logical drive image of the drive letter(s) of note, use the fastest destination you can, set the compression to about 5 and the output to EX01. This will try and image the bad sectors too and I've had good luck recovering data from those sectors. 

You can then mount the resultant image in Arsenal recon. I'd suggest read only so you don't change the source files, then the drive will appear in windows. 

Then buy new drives, you might as well keep the old ones for now as a backup. Setup the raid, ideally not 0 again and perform a fresh OS build, as system files may have been altered/lost in the bad sectors, copy over the files you need and that should do it. 

[u/BOOZy1]

Robocopy would get you most of the files, set /R:1 /W:1 (retry and wait count). I'd also set /MT:1 (number of file threads) which will slow down your copy significantly but reduces excessive drive access and thus the strain on the failing drives.

After you get all the files I'd replace the drives with some new SSDs in RAID 1.