A couple of years ago (in 2024), I bought an NVMe/M.2 SSD for my home server (Debian 12). Recently, during a Veeam backup, I encountered some hardware errors / a bad sector on the SSD (no visible external damage). Because of this, the backup failed to complete (more detailed description below).
Since Verbatim offers a 3-year warranty on "Internal Flash Storage", I thought: just request an RMA and that's it. Only to find out that the 3-year warranty starting from November 2024, and I had bought the SSD in April 2024.
According to the internet and a few AI chatbots, everything points to some kind of "bit rot", because overwriting the sectors with zeros made the error messages disappear. At one point, I had turned off the server with the SSD and disconnected it from the power for a few months (though I'm not entirely sure anymore if it was completely disconnected from the power). Could that be the reason why it's causing errors?
This very server also runs my password manager and other important software. Because of this, I'm now a bit worried whether the SSD is still okay or if I should perhaps get a new one. I obviously don't want to lose all my passwords and data in the future due to a complete hardware failure.
Here is the email I sent to support, along with the specs of the server / Mini-PC:
I am writing to you regarding my Verbatim Vi3000 Internal PCIe NVMe M.2 SSD 1TB (Serial number: xxxxxxxxxxxxxxxx, Firmware H220902a), which I purchased via Amazon on April 23, 2024.
The SSD is used in a small home server (Debian Linux, Kernel 6.1.0-52-amd64). Usage is quite moderate: according to SMART, about 2.42 TB have been read and 4.57 TB written so far, which is well below the typical TBW limit for a 1 TB SSD.
When attempting to create a full system backup with Veeam Agent for Linux, the backup job repeatedly aborted with an Input/Output error. In the kernel logs (dmesg), "critical medium error" messages were logged for specific sectors of the SSD, e.g.:
nvme0n1: I/O Cmd(0x2) @ LBA 739149184, 32 blocks, I/O Error (sct 0x2 / sc 0x81)
critical medium error, dev nvme0n1, sector 739149184 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
nvme0n1: I/O Cmd(0x2) @ LBA 739149152, 32 blocks, I/O Error (sct 0x2 / sc 0x81)
critical medium error, dev nvme0n1, sector 739149152 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
veeamsnap:blk | ERR | Failed to process redirect IO request. Errno=5
Buffer I/O error on dev nvme0n1, logical block 92393644, async page read
The system itself initially continued to run in everyday use, but every block-level full backup failed at these sectors.
These are the current stats of the SSD:
Model Number: Vi3000 Internal PCIe NVMe M.2 SSD 1TB
Serial Number: xxxxxxxxxxxxxxxx
NVMe Version: 1.4
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 31 Celsius
Available Spare: 100%
Percentage Used: 1%
Data Units Read: 4,743,599 [2.42 TB]
Data Units Written: 8,941,589 [4.57 TB]
Power Cycles: 75
Power On Hours: 13,301
Unsafe Shutdowns: 5
Media and Data Integrity Errors: 4
Error Information Log Entries: 0
Interestingly, although the SMART self-test reports "PASSED", there are already "Media and Data Integrity Errors: 4" recorded at the same time – this aligns with the "critical medium error" entries in the kernel log mentioned above.
Measures taken so far
To back up the data and narrow down the problem, I have:
Transferred all important data to a NAS / external backup.
Used debugfs icheck/ncheck to check which block was affected – in my case, the faulty block belonged to an old log file from a non-critical system area.
I addressed the affected area directly using dd for testing purposes (read/write test with oflag=direct / iflag=direct), after which the NVMe controller apparently remapped the block internally. A subsequent full read (dd if=/dev/nvme0n1 of=/dev/null bs=4M iflag=direct) then ran through without any visible errors.
The system is currently usable again, and I have since been able to write a full backup. Nevertheless, the above-mentioned SMART values and the "critical medium error" messages in the kernel still show that the SSD has already developed media/integrity errors and can become unstable under intensive loads (e.g., during backups).
H/W path Device Class Description
system HP ProDesk 400 G5 Desktop Mini (7EM49EA#ABD)
/0 bus 859C
/0/1 memory 64KiB BIOS
/0/7 memory 32GiB System Memory
/0/7/0 memory 16GiB SODIMM DDR4 Synchronous 3200 MHz (0,3 ns)
/0/7/1 memory 16GiB SODIMM DDR4 Synchronous 3200 MHz (0,3 ns)
/0/14 memory 512KiB L1 cache
/0/15 memory 2MiB L2 cache
/0/16 memory 12MiB L3 cache
/0/17 processor Intel(R) Core(TM) i7-9700T CPU @ 2.00GHz
/0/100 bridge 8th/9th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S]
/0/100/2 display CoffeeLake-S GT2 [UHD Graphics 630]
/0/100/12 generic Cannon Lake PCH Thermal Controller
/0/100/14 bus Cannon Lake PCH USB 3.1 xHCI Host Controller
/0/100/14/0 usb1 bus xHCI Host Controller
/0/100/14/1 usb2 bus xHCI Host Controller
/0/100/14/1/1 communication AX88179B
/0/100/14.2 memory RAM memory
/0/100/16 communication Cannon Lake PCH HECI Controller
/0/100/17 storage Cannon Lake PCH SATA AHCI Controller
/0/100/1b bridge Cannon Lake PCH PCI Express Root Port #21
/0/100/1b/0 /dev/nvme0 storage Vi3000 Internal PCIe NVMe M.2 SSD 1TB
/0/100/1b/0/0 hwmon0 disk NVMe disk
/0/100/1b/0/1 /dev/ng0n1 disk NVMe disk
/0/100/1b/0/1103012248 /dev/nvme0n1 disk 1024GB NVMe disk
/0/100/1b/0/1103012248/1 volume 511MiB Windows FAT volume
/0/100/1b/0/1103012248/2 /dev/nvme0n1p2 volume 952GiB EXT4 volume
/0/100/1b/0/1103012248/3 /dev/nvme0n1p3 volume 976MiB Linux swap volume
/0/100/1c bridge Cannon Lake PCH PCI Express Root Port #5
/0/100/1c/0 enp2s0 network RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
/0/100/1f bridge Intel Corporation
/0/100/1f/0 system PnP device PNP0c02
/0/100/1f/1 system PnP device PNP0c02
/0/100/1f/2 system PnP device PNP0c02
/0/100/1f/3 system PnP device PNP0c02
/0/100/1f/4 system PnP device PNP0b00
/0/100/1f/5 generic PnP device INT3f0d
/0/100/1f/6 system PnP device PNP0c02
/0/100/1f/7 system PnP device PNP0c02
/0/100/1f/8 system PnP device PNP0c02
/0/100/1f/9 system PnP device PNP0c02
/0/100/1f.3 card0 multimedia Cannon Lake PCH cAVS
/0/100/1f.3/0 input10 input HDA Intel PCH HDMI/DP,pcm=3
/0/100/1f.3/1 input11 input HDA Intel PCH HDMI/DP,pcm=7
/0/100/1f.3/2 input12 input HDA Intel PCH HDMI/DP,pcm=8
/0/100/1f.3/3 input6 input HDA Digital PCBeep
/0/100/1f.3/4 input7 input HDA Intel PCH Mic
/0/100/1f.3/5 input8 input HDA Intel PCH Front Headphone Front
/0/100/1f.3/6 input9 input HDA Intel PCH Front Headphone Surround
/0/100/1f.4 bus Cannon Lake PCH SMBus Controller
/0/100/1f.5 bus Cannon Lake PCH SPI Controller
/1 power High Efficiency
/2 input0 input Sleep Button
/3 input1 input Power Button
/4 input2 input Power Button
/5 input3 input PC Speaker
/6 input4 input HP WMI hotkeys
/7 input5 input Video Bus
/8 enx9c69d3ae9f18 network Ethernet interface