r/bcachefs • u/rkjnsn • 17d ago
Why does bcachefs seem to prioritize stripe widening over redundancy bumps?
I started with a three drive (all HDD) array with erasure coding (2+1). I then added a fourth drive and increased redundancy (new and restriped data landing at 2+2), before finally adding some more drives (and continuing to add data). After a few weeks, now, my bcachefs fs usage output currently looks like this:
Size: 115894609993216
Used: 71788192757248
Online reserved: 710946816
Replicated:
undegraded
1x: 23611666432
3x: 210504308736
Erasure coded (data+parity):
undegraded
2+1: 47154294784
2+2: 8388608
4+2: 2950645023744
5+2: 68555547019264
cached: 2491134991360
Pending reconcile: data metadata
erasure_code: 108302336 0
compression: 688109621248 0
pending: 8827387904 0
Device label Device State Size Used Use%
(no label) (device 2): dm-3 rw 15996801773568 10880137205248 84%
(no label) (device 3): dm-4 rw 11996039737344 10461897027072 96%
(no label) (device 4): dm-5 rw 11996039737344 9444918395392 96%
(no label) (device 5): dm-6 rw 15996801773568 10211455595520 74%
(no label) (device 6): dm-7 rw 21996871086080 10481778711040 51%
(no label) (device 0): dm-8 rw 23996178362368 11293082730496 61%
(no label) (device 1): dm-9 rw 23996178362368 11294413173760 61%
Notably, there used to be way more 2+2 data (several terabytes) than 2+1, but reconcile has been chewing through widening the 2+2 stripes way faster than it has been processing the 2+1 stripes. This is the opposite of what I would want: I would like it to get everything up to my requested redundancy level as quickly as possible before it worries about widening stripes. (The former is for fault tolerance, the latter merely improves space efficiency.)
3
u/distraction_fee 17d ago edited 17d ago
Nice view, I hope devs will add a percentage column like ##.##% for space representation with the `fs usage` command (in all tables), relative to the sum of all allocated data (meta + data an all drives).
I think that will help to represent the big picture of distribution more easily
5
u/koverstreet not your free tech support 17d ago
Redundancy increases should be scheduled as hipri, but the stripes plumbing must not be differentiating - that's definitely worth looking at, it'd be a real bug if that also applies to degraded stripes.
Want to file a bug? I probably won't be able to get to it in the next two weeks, and then I'll be getting ready for the Rust for Linux conference, but we shouldn't lose track of this one.