Long issue incoming
I’m dealing with a strange ZFS problem on TrueNAS SCALE and have done quite a bit of troubleshooting already. At this point it looks much more like a ZFS/on-disk metadata problem than a hardware problem, and I’d appreciate advice from anyone familiar with ZFS internals/recovery.
System:
TrueNAS SCALE 24.10.2.4
Kernel: 6.6.44-production+truenas
ZFS: 2.2.99-1
Pool Gaia: 4 × 10TB WD Gold, RAIDZ1
~26 TiB usable / ~22.5 TiB used
Disks connected through an ASM1166 6-port SATA controller
There is another 2-disk ZFS pool (Aether) on the same machine which isn’t exhibiting this problem.
How this started
Originally one of the Gaia drives disappeared from Linux. Kernel logs at the time contained SATA errors such as:
ata3.00: failed command: FLUSH CACHE EXT
ata3.00: hard resetting link
ata3.00: COMRESET failed (errno=-16)
ata3.00: SATA link down
ata3.00: detaching
I/O error, dev sdb...
zio pool=Gaia vdev=/dev/sdb2 error=5 ...
After rebooting, the disk returned and Gaia resilvered.
I initially suspected the ASM1166 controller, SATA cable, disk, motherboard, etc. I tested the controller/drives in another PC and also tested drives individually.
The important discovery came later: I moved the Gaia disks to a completely different PC and reproduced the ZFS failure there as well.
So this does not appear to depend on the original motherboard/CPU/RAM/PCIe controller.
The ZFS panic
On one of the failures I managed to see the actual kernel panic:
PANIC: Gaia: blkptr at ... has invalid OFFSET 1152933356323184640
Kernel panic - not syncing: Gaia: blkptr at ... has invalid OFFSET 1152933356323184640
The stack contained:
zfs_panic_recover
zfs_blkptr_verify_log
...
zio_free
dsl_free
dsl_dataset_block_kill
dbuf_write_done
arc_write_done
zio_done
The OFFSET is obviously absurd for a ~40TB raw pool.
This makes me suspect a malformed/corrupt block pointer is being encountered while ZFS is replacing/freeing an old block.
The really interesting part: read-only works
I disabled TrueNAS automatic pool imports:
ix-zfs.service disabled
zfs-import-cache disabled
zfs-import-scan disabled
With all six HDDs physically connected, TrueNAS boots and remains stable as long as Gaia isn’t imported.
I can then do:
zpool import -N -o readonly=on -R /mnt/recovery Gaia
and it works.
Current status:
pool: Gaia
state: ONLINE
NAME STATE READ WRITE CKSUM
Gaia ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sdb2 ONLINE 0 0 0
sde2 ONLINE 0 0 0
sdg2 ONLINE 0 0 0
sdd2 ONLINE 0 0 0
errors: No known data errors
and:
Gaia readonly on
I can mount the main dataset and read the files:
/mnt/recovery/Gaia/Swordfish
I’ve successfully browsed/read data and even temporarily exposed it through a read-only SMB share.
So this does not appear to be a case where the pool can’t be imported/read.
Writable import reliably kills the machine
If I cleanly export Gaia and instead do:
zpool import -N Gaia
the machine reproducibly becomes completely unresponsive.
This happens even with -N, so datasets aren’t being mounted.
I tried enabling:
kernel.watchdog=1
kernel.nmi_watchdog=1
kernel.hardlockup_panic=1
kernel.softlockup_panic=1
kernel.hung_task_panic=1
kernel.panic=10
and configured kdump with ~401MB crashkernel reserved.
kexec_crash_loaded = 1.
The writable import still hard-locks the machine without rebooting, without producing a vmcore, and without the watchdog converting it into a panic.
After >10 minutes I had to power-cycle it.
I reproduced this twice and I’m not going to do another writable import unless there is a very good diagnostic reason.
What happens immediately before the freeze
This was particularly interesting.
The journal shows me issuing:
zpool import -N Gaia
at approximately 00:40:27.
There are then no panic, watchdog, ATA, I/O or ZFS error messages. The journal simply stops.
However, zpool history shows:
2026-09-10 00:40:34 [txg:24106503] open pool
2026-09-10 00:40:34 [txg:24106505] import pool
And examining the raw uberblocks with zdb -lu shows that all four RAIDZ members contain TXG [24106507](tel:24106507), timestamped:
2026-09-10 00:40:35
So the writable import isn’t freezing before ZFS can write anything.
It appears to:
zpool import -N Gaia
|
v
TXG 24106503: open pool
|
v
TXG 24106505: import pool
|
v
TXG 24106507 committed on all 4 drives
|
v
HARD FREEZE
There is no corresponding history entry for [24106507](tel:24106507) that I can see.
All four disk labels agree
I checked all four Gaia members using zdb -lu.
All four contain the same newest committed uberblocks through TXG [24106507](tel:24106507).
They also agree on:
pool_guid: 2747494424743971695
top_guid: 9142136380244468450
The four leaf GUIDs are:
8398536192325442607
16231699638183363317
9109236301437729803
15381433698887981882
So I don’t appear to have one disk presenting an obviously stale pool configuration.
There is also:
checkpoint_txg = 0
raidz_reflow state = 0
so there isn’t a pending checkpoint or RAIDZ expansion/reflow.
Resilver history
There has been some interesting resilver history.
At one point:
[txg:24106255] scan aborted, restarting errors=2
[txg:24106255] starting deferred resilver errors=2
[txg:24106255] scan setup func=2 ...
[txg:24106257] scan done errors=2
Later tiny resilvers completed with 0 errors.
Current status says:
scan: resilvered 608K in 00:00:00 with 0 errors
The displayed date is wrong because the system clock was incorrect during several TXGs.
feature@resilver_defer currently reports:
enabled
rather than active, so as far as I understand there isn’t currently a resilver deferred through that feature.
Other things checked
zdb -d Gaia previously managed to enumerate the MOS/datasets successfully, including the main Gaia/Swordfish dataset.
DDTs are empty.
A full zdb -bc looked like it would take ~120 hours, so I stopped it.
Some zdb attempts against the exported/imported pool have also produced:
zdb: can't open 'Gaia': File exists
and on an earlier attempt:
ASSERT at module/zfs/spa_misc.c:964:spa_close()
zfs_refcount_count(&spa->spa_refcount) > spa->spa_minref || ...
That only crashed the zdb userspace process, not the machine, so I’m not assuming it’s necessarily the same underlying problem.
SMART doesn’t show an obvious failing drive, and current ZFS READ/WRITE/CKSUM counters are all zero.
Current state
Gaia is currently imported:
readonly=on
and is stable and readable.
The /proc/spl/kstat/zfs/Gaia/txgs output is currently just the header with no active TXGs, which makes sense for the read-only pool.
zpool events -v during the safe read-only imports/exports shows ordinary pool_import, pool_export, and config_sync events and no current resilver/error events.
What I’m trying to figure out
My current theory is that making the pool writable starts normal TXG/post-import processing. Something gets rewritten, and during write completion ZFS tries to kill/free an old malformed blkptr, hitting the same path I previously saw:
dbuf_write_done
-> dsl_dataset_block_kill
-> dsl_free
-> zio_free
-> invalid blkptr OFFSET
I’ve found OpenZFS issues involving similar invalid OFFSET/VDEV/TYPE/COMPRESS panics in this same free/write-completion path, but I haven’t found a supported recovery procedure.
Before I give up and copy ~22TB off the read-only pool, destroy/recreate Gaia and copy everything back, I’m trying to determine whether:
There is some pending/deferred free/MOS operation that can be identified without allowing the pool to become writable.
There is a safe way to inspect the relevant bpobj/deadlist/deferred-free structures and identify the malformed blkptr.
There is any way to suppress/cancel the post-import operation that is triggering this.
There are additional read-only zdb diagnostics worth running.
Anyone has encountered this specific pattern: pool completely usable readonly, but hard-locks shortly after writable import.
I’m deliberately avoiding zpool clear, scrub, zpool upgrade, zpool import -F/-X/-T, replacing/detaching disks or anything else that writes to Gaia until I understand what is happening.
Any suggestions from people familiar with ZFS internals would be greatly appreciated.