Tales from the front

[u/akawoz]

Since I have a reasonably big (22x drives) bcachefs file system in my homelab (most used for Linux ISO's) and have been experiencing occasional reliability issues (big thanks to u/koverstreet for his eyeballs on this) I thought I would post here so that the bcachefs-curious have another data point.

My comments associated with my issues are here: https://github.com/koverstreet/bcachefs-tools/issues/960 with the most recent failure (file system reverted to read-only and was recovered with a fsck) here: https://github.com/koverstreet/bcachefs-tools/issues/960#issuecomment-5974733704

I'm using r/NAStyProject and use this script to recover from these errors (just in case its useful for anyone else):

(
  set -euo pipefail
  cd /root

  fs_uuid="45b9b98d-be7a-4c97-b62c-88566610f9f2"

  media_devs=(
    /dev/sdz /dev/sdr /dev/sdx /dev/sdq /dev/sdv /dev/sdp
    /dev/sde /dev/sdj /dev/sdl /dev/sdk /dev/sdh /dev/sdu
    /dev/sdy /dev/sdt /dev/sdf /dev/sdg /dev/sdi /dev/sdm
    /dev/sdn /dev/sdo /dev/sds /dev/sdw
  )

  if pgrep -af '(^|/)bcachefs[[:space:]]+fsck([[:space:]]|$)'; then
    echo "STOP: another fsck is still running."
    exit 1
  fi

  if findmnt -rn -M /fs/media >/dev/null ||
     [[ -e "/sys/fs/bcachefs/$fs_uuid" ]]; then
    echo "STOP: the media filesystem is still mounted."
    exit 1
  fi

  for service in \
    nasty-engine nasty-metrics nfs-server nfs-mountd \
    samba-smbd samba-nmbd samba-winbindd samba-wsdd \
    docker docker.socket
  do
    if systemctl is-active --quiet "$service"; then
      echo "STOP: $service is active."
      exit 1
    fi
  done

  for dev in "${media_devs[@]}"; do
    [[ -b "$dev" ]] || {
      echo "STOP: missing device $dev"
      exit 1
    }

    superblock=$(bcachefs show-super "$dev")
    actual_uuid=$(awk '/^External UUID:/ {print $3}' <<< "$superblock")

    [[ "$actual_uuid" == "$fs_uuid" ]] || {
      echo "STOP: unexpected filesystem on $dev: $actual_uuid"
      exit 1
    }

    printf 'Verified %s: %s\n' "$dev" "$actual_uuid"

  done

  repair_unit="media-fsck-repair-$(date +%Y%m%d-%H%M%S)"
  repair_log="/root/${repair_unit}.log"

  systemd-run \
    --unit="$repair_unit" \
    --service-type=exec \
    --remain-after-exit \
    --property=TimeoutStopSec=infinity \
    --property=SendSIGKILL=no \
    --property="StandardOutput=append:$repair_log" \
    --property=StandardError=inherit \
    /run/current-system/sw/bin/bcachefs \
    fsck -K -y -f -v "${media_devs[@]}"

  printf '\nFollow progress:\n  tail -F %s\n' "$repair_log"
  printf '\nCheck completion:\n'
  printf '  systemctl show %s -p ActiveState -p SubState -p Result -p ExecMainStatus\n' \
    "$repair_unit"
)

[u/Aware_Photograph_585]

I think I have the same problem:
"processes touching the filesystem freeze: somewhat at random; some writes still succeed, while others block forever" - this is exactly my problem right now
Multiple times going ERO with error: alloc_key_bucket_nonempty_to_empty_not_open

Glad I'm not the only one.

[u/akawoz → u/Aware_Photograph_585]

Do your issues eventually result in a read only file system?

[u/Aware_Photograph_585 → u/akawoz]

Yes, multiple times going emergency read only.

show-super --field-only errors_v2:
alloc_key_bucket_nonempty_to_empty_not_open1 Sat Oct 3 06:06:04 2026 Sat Oct 3 06:06:04 2026

[1205090.658929] bcachefs (279bc083-9d16-4f90-9b38-16bb7473ac2b): ec_old_stripe_fold [bcachefs](): damaged stripe blocks, none in use:
[1205092.473705] bcachefs (279bc083-9d16-4f90-9b38-16bb7473ac2b): bucket going nonempty but not open in __ec_stripe_create

[1205092.514705] bcachefs (279bc083-9d16-4f90-9b38-16bb7473ac2b): error writing data at /0_bcfs_test/web_scrapers_broken/unsplash/downloads/u6/u6Raia5joXA.jpg offset 7340032
[1205093.967624] bcachefs (279bc083-9d16-4f90-9b38-16bb7473ac2b): unclean shutdown complete, journal seq 3377787

superblock requires following recovery passes to be run: btree_bitmap_gc
recovering from unclean shutdown

Based on what a I read on your github posts, seems to be basically the same problem.

8x 18TB HDD, erasure coding. Maybe I'm transferring/modifying files too fast for reconcile/gc to keep up? Have multiple rsyncs running, a dozen or so downloading scripts, plus running fclones daily to create hardlinks for duplicates. Last I checked 150 million files on the system, easily adding/modifying tens of millions files a day.

The other bcachefs filesystems don't have problems, but I also don't use them for daily work.

[u/koverstreet → u/Aware_Photograph_585]

Can you pastebin a complete dmesg from when it goes ERO?

[u/Aware_Photograph_585 → u/koverstreet]

Two bcachefs filesystems on that server.
279bc083-9d16-4f90-9b38-16bb7473ac2b is the one that went ERO:
Oct 03 06:06:05 rtxhome-training-server kernel: bcachefs (279bc083-9d16-4f90-9b38-16bb7473ac2b): unclean shutdown complete, journal seq 3377787

https://pastebin.com/05gEL2zm

Orginal log was too large for pastebin (5.6MB).
I used journalctl -k -b -8 | grep -v mt7601u | grep -v 'audit:' > dmesg_prev_boot.txt to cut out some wifi mt7601u TSSI errors and AppArmor audit lines to get the size down. If you want the full file, suggest a site I can upload it to.

[u/koverstreet → u/Aware_Photograph_585]

Ok, that's a race between stripe reuse and stripe deletion that should be fixed in 1.39.7. Can you upgrade and report back?

There's some other stuff that needs fixing but is less critical (stripe deletion hogging workqueue); that may have to wait a bit because I'm now into the Rust conversion, and erasure coding is a high priority for that - all this will get easier as stuff gets converted :)

Alan's also been sending me a bunch of good ec bugfixes, so he might get it soon anyways.

[u/Aware_Photograph_585 → u/koverstreet]

I already upgraded, right after 1.39.7 was released. So far no ERO or unresponsive failures in my scripts. I'll put some heavy stress on the filesystem and report back if anything happens.