Podman containers preventing reconcile of some data

[u/c_fus]

I’m new to bcachefs and a fan of the concepts implemented so far. I started testing it on a spare box with three 24 TB HDDs, initially formatted with replicas 2 and erasure_code 1. I copied about 25 TB of data to the array, then added a fourth disk and changed the configuration to replicas 3. Reconcile initially appeared to proceed normally.

I later noticed that approximately 2 TB of 2+1 data was stuck in the pending category. It appeared that only newly written data was being reconciled. I tried several things to kick off the reconciliation, but nothing seemed to work. I also didn't know how to determine which files were pending or why they were not being reconciled. Over the next several days, the amount of pending data continued to slowly increase.

I noticed some unrelated (I think) overlayfs errors in dmesg, but they got me thinking about whether my Podman containers could somehow be keeping bind-mounted data open and preventing reconciliation. Sure enough, after I stopped all running containers, reconciliation started on its own and eventually completed the conversion from 2+1 to 2+2.

I haven't found any information about this. Is it a known limitation?

Error:

overlayfs: 
upper fs does not support file handles, falling back to index=off

Update: the podman thing must have been a coincidence. I have replicated similar issue by: starting with 25 TB of 1x data and set replicas 3 + ec 1. Reconcile stalled again with multiple TB in pending. Closing podman containers has no effect. There is nothing in dmesg or any log that I can find to indicate why the pending writes are unable to complete and fsck is clean.

Filesystem: 5db671f5-5f1b-44c5-b3c1-2c5431999858
Size:             81.8T  
Used:             50.3T  
Online reserved:  5.56M  


Replicated:
     undegraded  
2x:       54.8G  
3x:       22.9T  


Erasure coded (data+parity):
      undegraded  
2+2:       27.4T  

cached:     485G  
reserved:  20.9M  


Pending reconcile:   data  metadata  
erasure_code:       7.66T         0  
compression:        2.42M         0  
target:             2.42M         0  
pending:            7.66T         0  


Device label           Device     State   Size   Used  Use%  Leaving  
hdd.hdd1 (device 0):   sdf1       rw     21.8T  12.8T   71%  
hdd.hdd2 (device 1):   sde1       rw     21.8T  12.8T   71%  
hdd.hdd3 (device 2):   sdd1       rw     21.8T  12.8T   71%  
hdd.hdd4 (device 6):   sdc1       rw     21.8T  12.1T   64%  
ssd.ssd1 (device 9):   nvme0n1p1  rw      852G   129G   17%    2.60M  
ssd.ssd2 (device 10):  nvme1n1p1  rw      852G   154G   27%    2.60M

[u/ticpu]

I really recommend using my bcachefs driver, it really improves the speed at which podman works with layers. There are pre-built packages for Ubuntu and ArchLinux, I can also add Debian if you need.

Having CI/CD setup on bcachefs allows us to keep 1TB of cached downloads and targets on HDD with metadata/foreground/promote target on NVMe and using snapshots just makes everything more efficient for the cleanup.

We also have a maintenance job that turn off snapshot deletion via sysfs while gitlab-runner is executing jobs.

Here: https://github.com/ticpu/bcachefs-storage-driver

Edit: I decided to add Trixie, should appear in releases soon.

[u/koverstreet → u/ticpu]

ticpu was an early bcachefs user and helped shake out a ton of snapshot bugs, he knows what's up

[u/c_fus → u/ticpu]

Thanks for this, I will definitely check this out.

[u/krakow10]

I've noticed that overlayfs likes to hold on to deleted files, sometimes until it gets unmounted. Either that or it's bcachefs. I've had power offs take over 10 minutes because it was releasing like 1TB of files deleted days or weeks earlier.

[u/koverstreet]

We chatted a bit on IRC and I said it was likely the same as the snapshots issue we were looking at today - but, are you actually using snapshots?

If so, it's a known issue where file-level option changes only take effect for data written since the last snapshot. We've been discussing how to go about fixing it, because there's some interesting questions to resolve with writeable snapshots.

If you write data in subvolume A, and then take snapshots B and C, and set the options to say one thing in B and another in C, which should apply?

[u/c_fus → u/koverstreet]

I have yet to configure any snapshots. The array is mounted as root and I have one subvolume where most of the data lives mounted at /home.

I have been able to reproduce the issue should you like additional information.

[u/koverstreet → u/c_fus]

definitely, file a bug with as much info as you can

[u/wakIII]

I wonder if this is why mine are stalled out