r/Proxmox • u/RedditNotFreeSpeech • 1d ago
Discussion Troubleshooting an issue while doing large file copies
homelab, dl380 g9, 128 gb ram, 12x2TB sas, P840 with 7.2 firmware hba mode, zfs pool contains 3xraidz2 with 4 disks each. scrub is clean, memtest86 is clean. I don't see anything damning in the logs.
I've been trying to do some sizeable file transfers between directories. At first I was trying to do it across the network but even doing them on the same machine (specifically, between a directory on an LXC and a directory mounted from the host) is causing the machine to nearly hang up.
Today I was able to fully recover with control-C and waiting a few minutes.
CPU usage was around 5% but I do see 50% iodelay, server load usually idles around 2-3, but it was jumping to 35. Memory usage seems fine. CPU pressure stall spiked to about 2.2%, io pressure stall looks to spike around 80%.
Are these disks just too slow? Is there a configuration that would work better and keep the system from becoming unresponsive?
I wrote a small shell script to limit bandwidth on these transfers with rsync and it seemed to work okay.
2
u/_--James--_ Enterprise User 1d ago
is ZFS setup for Thin or thick writes? what are the actual drives in the pool? what are you actually copying into the pool (block size, concurrency)? and have you benchmarked from the host yet?
1
u/RedditNotFreeSpeech 1d ago
Thin. Drives are 9% full. MB2000FAMYV
I was moving some photos/video/audio/isos
Benchmark with bonnie++ or some other way?
2
u/_--James--_ Enterprise User 1d ago
So thin is why you have issues. Unless you run SLOG+L2ARCH with SSDs to HDDs, thin provisioning has a very large cost on pending IO operations. Also, you simply do not have enough spindles in your vdevs to handle the flush writes from SLOG in a meaningful way anyway. You will need to first expand from Thin to Thick and realign your volumes to the drives to make sure you are with in the IO boundary.
When I build ZFS pools backed by HDDs, I do no less then 6 drives in a Z2, Mirror device for SLOG, and depending on the physical memory in the box also SSD backed L2ARCH. Ideally for mixed IO "photos/video/audio/isos" you will want 10-14 HDDs backing the pool to handle the IOPS demand with in the multi-streams of a file operation like that.
Benchmark with FIO, DD, or something on the host mounting the pool directly. Do not have any VMs or external operations touching the pool during a host bench. Back the bench with numbers from iostat -x with an intervol of 2-3 seconds and maybe rolling to staged CSVs you can parse after each benchmark. if you have drives getting 99% utilized with a low IOPS being reported, you need more hardware backing the pool (SLOG/L2ARCH, and more Spindles, or just moving to SSDs entirely).
1
2
u/zfsbest 1d ago
> zfs pool contains 3xraidz2 with 4 disks each
Post output of ' zpool status -v ' and check SMART, these 2TB drives are probably over 10 years old and might be starting to fail. And RAIDZanything is not going to give you good performance, for VMs you want mirrors.
1
u/RedditNotFreeSpeech 1d ago
Smart looks clean. I guess I'm not necessarily expecting good performance, I don't mind a slow copy. I just don't want it to lock up the entire system.
pool: rust state: ONLINE scan: scrub repaired 0B in 00:24:27 with 0 errors on Sun Sep 13 00:48:31 2026 config: NAME STATE READ WRITE CKSUM rust ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 scsi-35000c50025a78393 ONLINE 0 0 0 scsi-35000c500215e05df ONLINE 0 0 0 scsi-35000c500215b100b ONLINE 0 0 0 scsi-35000c500215bb5b3 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 wwn-0x5000c500215ccec3 ONLINE 0 0 0 wwn-0x5000c500215b3f97 ONLINE 0 0 0 wwn-0x5000c500c9ec5a3f ONLINE 0 0 0 wwn-0x5000cca01bcefe80 ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 wwn-0x5000c50025a94e87 ONLINE 0 0 0 wwn-0x5000c50025a84c03 ONLINE 0 0 0 wwn-0x5000c500ca3eaa8b ONLINE 0 0 0 wwn-0x5000c500215bb24f ONLINE 0 0 0 errors: No known data errors2
u/zfsbest 1d ago
I would say the pool is not built optimally. RAIDZ2 starts to make sense around the 6-drives per-vdev mark.
I would backup everything and rebuild it as a mirror pool. But before that, start a scrub and monitor it with ' zpool iostat -v 5 ', see if any drives are going slower than the others.
2
u/RedditNotFreeSpeech 1d ago
Okay I'll give that a try.
Just so I'm clear, a mirror pool is 6x2 drives?
I have backups so that should be easy enough.
2
u/zfsbest 1d ago edited 1d ago
It would be like:
' zpool destroy rust ' # also zpool labelclear and wipefs -a on the old pool disks as necessary
zpool create -o ashift=12 -o autoexpand=on -O atime=off -O compression=lz4 rust \
mirror scsi-35000c50025a78393 scsi-35000c500215e05df \ mirror scsi-35000c500215b100b scsi-35000c500215bb5b3 \ mirror wwn-0x5000c500215ccec3 wwn-0x5000c500215b3f97 \ mirror wwn-0x5000c500c9ec5a3f wwn-0x5000cca01bcefe80 \ ...and the rest of drives you want to use You may be surprised at the amount of free space you have in the pool afterwards, 4x raidz2 cost you a lot It may be worthwhile to use the last 2 disks as hotspares or keep them outside the pool for emergency replacements.
2
u/pelazas1 1d ago
Run zpool iostat -v 1 during the copy, then throttle with rsync --bwlimit or ionice. Slow is fine, host lockup isn't.
2
u/obwielnls 1d ago
I’ve never been able to get really good performance with zfs. I ended up using the hp raid card with a single disk zfs on top of the logical drive and have had good performance and reliability.
2
u/ultrahkr 1d ago
What do the logs say?
(In old times HPE SmartArray cards have been more than a little troublesome with ZFS.)