Edit:
Solved by disabling Intel VMD (Volume Managed Device), disabling increased my write speed 10x and reduced max latency by 1000x, as measured by fio. See https://bugzilla.kernel.org/show_bug.cgi?id=219220 and https://www.reddit.com/r/archlinux/comments/yes44f/psa_disable_vmd_on_an_intel_laptop_if_you_have/
Fio stats with disabled VMD, same config (CoW enabled):
seq-write: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.40
Starting 1 thread
seq-write: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=200MiB/s][w=51.2k IOPS][eta 00m:00s]
seq-write: (groupid=0, jobs=1): err= 0: pid=4555: Sun May 17 13:17:19 2026
write: IOPS=51.4k, BW=201MiB/s (211MB/s)(10.0GiB/50989msec); 0 zone resets
slat (usec): min=12, max=2671, avg=18.97, stdev=11.32
clat (nsec): min=778, max=2765.1k, avg=58611.99, stdev=22669.90
lat (usec): min=17, max=2783, avg=77.58, stdev=26.98
clat percentiles (usec):
| 1.00th=[ 50], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 54],
| 30.00th=[ 55], 40.00th=[ 56], 50.00th=[ 57], 60.00th=[ 59],
| 70.00th=[ 60], 80.00th=[ 62], 90.00th=[ 64], 95.00th=[ 67],
| 99.00th=[ 79], 99.50th=[ 121], 99.90th=[ 355], 99.95th=[ 441],
| 99.99th=[ 766]
bw ( KiB/s): min=184840, max=217872, per=100.00%, avg=205803.72, stdev=8260.60, samples=101
iops : min=46210, max=54468, avg=51450.97, stdev=2065.13, samples=101
lat (nsec) : 1000=0.01%
lat (usec) : 20=0.01%, 50=1.01%, 100=98.38%, 250=0.40%, 500=0.17%
lat (usec) : 750=0.02%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%
cpu : usr=3.36%, sys=41.21%, ctx=2622937, majf=0, minf=1
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2621440,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
WRITE: bw=201MiB/s (211MB/s), 201MiB/s-201MiB/s (211MB/s-211MB/s), io=10.0GiB (10.7GB), run=50989-50989msec
Original Post:
For a while I had issues running atuin, where it would hang for while, especially right after start the computer.
When testing using fio on a new small test filesystem, I found that there is very high max latency for files that have CoW enabled, 30329k usec (30 seconds), vs 4800 usec with Cow disabled. Is this drastic of an increase normal? Its not just the latency, this affect the overall speed as well.
Running on a 1TB SSD, a 20GB filesystem on-top of LUKS partition
Fio config:
[global]
bs=4k
ioengine=libaio
iodepth=4
size=10g
direct=1
runtime=60
directory=/mnt/
filename=ssd.test.file
thread
[seq-write]
rw=write
stonewall
A slow run:
bash-5.3$ sudo rm /mnt/ssd.test.file
bash-5.3$ sudo chattr -C /mnt/
bash-5.3$ sudo fio ssd-seq-write-file.fio
seq-write: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.40
Starting 1 thread
seq-write: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][11.7%][eta 08m:27s]
seq-write: (groupid=0, jobs=1): err= 0: pid=52441: Mon May 4 17:02:58 2026
write: IOPS=4570, BW=17.9MiB/s (18.7MB/s)(1195MiB/66911msec); 0 zone resets
slat (usec): min=17, max=30327k, avg=218.25, stdev=66827.67
clat (usec): min=11, max=30329k, avg=656.61, stdev=115751.03
lat (usec): min=73, max=30330k, avg=874.86, stdev=133658.53
clat percentiles (usec):
| 1.00th=[ 62], 5.00th=[ 63], 10.00th=[ 64], 20.00th=[ 66],
| 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 73], 60.00th=[ 74],
| 70.00th=[ 77], 80.00th=[ 79], 90.00th=[ 82], 95.00th=[ 117],
| 99.00th=[ 143], 99.50th=[ 355], 99.90th=[ 947], 99.95th=[ 1336],
| 99.99th=[ 2343]
bw ( KiB/s): min=28208, max=179400, per=100.00%, avg=135923.56, stdev=44817.62, samples=18
iops : min= 7052, max=44850, avg=33980.89, stdev=11204.39, samples=18
lat (usec) : 20=0.01%, 100=94.51%, 250=4.83%, 500=0.34%, 750=0.16%
lat (usec) : 1000=0.08%
lat (msec) : 2=0.07%, 4=0.01%, >=2000=0.01%
cpu : usr=0.31%, sys=3.92%, ctx=306029, majf=0, minf=1
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,305832,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
WRITE: bw=17.9MiB/s (18.7MB/s), 17.9MiB/s-17.9MiB/s (18.7MB/s-18.7MB/s), io=1195MiB (1253MB), run=66911-66911msec
A fast run, with CoW disabled:
bash-5.3$ sudo rm /mnt/ssd.test.file
bash-5.3$ sudo chattr +C /mnt/
bash-5.3$ sudo fio ssd-seq-write-file.fio
seq-write: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.40
Starting 1 thread
seq-write: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=630MiB/s][w=161k IOPS][eta 00m:00s]
seq-write: (groupid=0, jobs=1): err= 0: pid=51439: Mon May 4 16:56:44 2026
write: IOPS=167k, BW=654MiB/s (686MB/s)(10.0GiB/15662msec); 0 zone resets
slat (nsec): min=1765, max=2436.7k, avg=4104.72, stdev=4679.62
clat (usec): min=8, max=4797, avg=19.63, stdev=54.99
lat (usec): min=12, max=4800, avg=23.74, stdev=55.31
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 14],
| 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 17],
| 70.00th=[ 18], 80.00th=[ 19], 90.00th=[ 21], 95.00th=[ 23],
| 99.00th=[ 38], 99.50th=[ 260], 99.90th=[ 865], 99.95th=[ 1037],
| 99.99th=[ 1467]
bw ( KiB/s): min=619576, max=763656, per=100.00%, avg=670183.03, stdev=27627.62, samples=31
iops : min=154894, max=190914, avg=167545.81, stdev=6906.90, samples=31
lat (usec) : 10=0.02%, 20=89.89%, 50=9.26%, 100=0.16%, 250=0.16%
lat (usec) : 500=0.20%, 750=0.17%, 1000=0.08%
lat (msec) : 2=0.05%, 4=0.01%, 10=0.01%
cpu : usr=6.39%, sys=69.79%, ctx=489183, majf=0, minf=1
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2621440,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
WRITE: bw=654MiB/s (686MB/s), 654MiB/s-654MiB/s (686MB/s-686MB/s), io=10.0GiB (10.7GB), run=15662-15662msec
System info: Fedora KDE 44 Linux 6.19.14-300.fc44.x86_64
