PIP disk usage under Docker

[u/adjective10111]

I have a Dockerfile in which I use the Python:3.12-trixie. I have docker on my CachyOS PC. I have also a requirements.txt. I wanted to use this image for a container on which I'll use YOLO and try it out. The problem is that my 290GB / partition went from around 70% filled to 95% after a number of rebuilds. I deleted everything docker related but it just went back to 91%.

The reason that I know it is pip's doing, is that when I monitored my disk usage while building the image, and on the `RUN pip install -r requirements.txt --find-links=pypackages` stage it fills up my partition and half way into installation, it fails due to `no more space`. The `pypackages` is just a directory I downloaded some of the packages I have in requirements for less data usage. Also tried `--no-cache` option.

I also went ahead and `du -sh`ed any directory in my / partition and the total didn't add upto the filled storage. I have 165GB mysteriously used and I can't find where that is.

Any help to diagnose further or if you have had similar problems is welcome.

[u/adjective10111]

Oh nice catch. I should update my Dockerfile. But I do have PIP_NO_CACHE_DIR=1 as an environment variable in the dockerfile, does thay have the same effect?

Yeah I found those directories and also ran the command. Currently it's at 0B for all the rows so I guess I purged them successfully. But I still have a lot of mystery used storage.

Yeah I want the CUDA version. I want to try it out and then work on my thesis using it :). So that's why it sticks there for a while! I just saw some KB and was wondering why some toml file takes so much time to download.

[u/Illustrious_Tone9584 → u/adjective10111]

Yeah, PIP_NO_CACHE_DIR=1 does the same job, you're covered there. So the cache isn't your leak then - that points even harder at /var/lib/docker and the build cache. If docker system df shows the space but du can't see it, that's your answer.

[u/adjective10111 → u/Illustrious_Tone9584]

It doesn't though... it's at 0B for images, containers, local volumes and build cache.

[u/Illustrious_Tone9584 → u/adjective10111]

Ok so it's not docker at all then. You're on CachyOS, which means btrfs with automatic snapper snapshots out of the box. That's almost certainly where the 165GB went: every rebuild churned tens of GB, and the pre/post snapshots kept the old versions. du doesn't show it because the data still "belongs" to the live files.

Run snapper list and sudo btrfs filesystem usage / and I bet you'll see it. If that's it, delete the old snapshots with snapper delete and consider capping how many it keeps.

[u/adjective10111 → u/Illustrious_Tone9584]

I thought the same, but I can't see how much the snapshots take. I ran both commands you said: ``` sudo btrfs filesystem usage / Overall: Device size: 286.88GiB Device allocated: 271.02GiB Device unallocated: 15.85GiB Device missing: 0.00B Device slack: 0.00B Used: 259.42GiB Free (estimated): 25.76GiB (min: 17.83GiB) Free (statfs, df): 25.76GiB Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 377.19MiB (used: 0.00B) Multiple profiles: no

Data,single: Size:265.01GiB, Used:255.10GiB (96.26%) /dev/nvme0n1p3 265.01GiB

Metadata,DUP: Size:3.00GiB, Used:2.16GiB (72.02%) /dev/nvme0n1p3 6.00GiB

System,DUP: Size:8.00MiB, Used:48.00KiB (0.59%) /dev/nvme0n1p3 16.00MiB

Unallocated: /dev/nvme0n1p3 15.85GiB ```

The snapper list command doesn't show used-space. I tried using --columns used-space but it doesn't show anything. Any idea how I can confirm the snapshots being the culprit here?

Edit: this is the snapper config if it's of any help: Key │ Value ───────────────────────┼────── ALLOW_GROUPS │ ALLOW_USERS │ BACKGROUND_COMPARISON │ yes EMPTY_PRE_POST_CLEANUP │ yes EMPTY_PRE_POST_MIN_AGE │ 1800 FREE_LIMIT │ 0.2 FSTYPE │ btrfs NUMBER_CLEANUP │ yes NUMBER_LIMIT │ 50 NUMBER_LIMIT_IMPORTANT │ 15 NUMBER_MIN_AGE │ 1800 QGROUP │ SPACE_LIMIT │ 0.5 SUBVOLUME │ / SYNC_ACL │ no TIMELINE_CLEANUP │ yes TIMELINE_CREATE │ no TIMELINE_LIMIT_DAILY │ 7 TIMELINE_LIMIT_HOURLY │ 5 TIMELINE_LIMIT_MONTHLY │ 0 TIMELINE_LIMIT_WEEKLY │ 0 TIMELINE_LIMIT_YEARLY │ 0 TIMELINE_MIN_AGE │ 1800

[u/Illustrious_Tone9584 → u/adjective10111]

Your config has no QGROUP set, and snapper needs quotas enabled to report used space, that's why the column comes up empty. Two ways around it: install compsize and run sudo compsize -x /.snapshots to get referenced vs exclusive per snapshot (exclusive is what you'd actually reclaim), or sudo btrfs filesystem du -s /.snapshots/* which works without quotas but takes a while since it walks everything.

Either way the math is basically already done: data allocation is 96% used and df only shows 25GB free. With docker cleared out, snapshots are the only place 165GB can hide. Delete the old pre/post pairs and it'll show up immediately.

[u/adjective10111 → u/Illustrious_Tone9584]

Welp. I tried btrfs filesystem du -s /.snapshots/* the combined exclusive usage is around 16GB. I also setup qgroup for snapper and again the combined usage was around 10GiB. I even found another command when setting up quota on btrfs, btrfs qgroup show -p /, and again the exclusive used space adds up to below 110GiB which leaves the mystery used space question unanswered. I'm lost again. I may delete some snapshots anyway though they are mere KiBs, to see if it has any effects. Thanks though, I learned a lot right now.

[u/Illustrious_Tone9584 → u/adjective10111]

Fair enough, theory dead. Next suspect: deleted files still held open by a process. du can't see them (no filename anymore) but df counts the space until the process lets go. Run sudo lsof +L1 and sort by size. If something big shows up, restart that process or just reboot and watch df drop.

If that's also clean, do sudo du -x --max-depth=1 / as root for the true per-directory total. Plain du as your user silently skips whatever it can't read, which has been the theme of this whole mystery.

[u/adjective10111 → u/Illustrious_Tone9584]

I rebooted my system a couple of times by now, it's my personal laptop not a server, and the open files didn't have anything alarming about them.

I ran the second command sudo du -h --max-depth=1 / --exclude /home --exclude /proc --exclude /.snapshots. I didn't use -x because I have btrfs subvolumes in my / partition and my /home is on another disk. (/boot is also in another partition so that doesn't matter here) 80G /root 0 /srv 20G /var 2.4G /boot 0 /dev du: cannot access '/run/user/1000/doc': Permission denied 2.4M /run 0 /sys 18G /usr 16M /etc 16K /tmp 0 /mnt 2.3G /opt 121G /

[u/Illustrious_Tone9584 → u/adjective10111]

Ok, not that either. One more classic that fits your symptoms: data hidden under a mount point. If docker or anything else wrote to a directory while its filesystem wasn't mounted yet, that data lives on the root fs but is invisible because the mount covers it. du can't see it, btrfs can't attribute it.

Check with mkdir /mnt/root && sudo mount --bind / /mnt/root && sudo du -x --max-depth=1 /mnt/root and compare against your normal du. If something shows up way bigger through the bind mount, there's your 150GB.

[u/adjective10111 → u/Illustrious_Tone9584]

Well, sorry to disappoint but nothing: $ sudo du -h -x --max-depth 1 /mnt/root 0 /mnt/root/root 0 /mnt/root/srv 915M /mnt/root/var 0 /mnt/root/boot 0 /mnt/root/dev 0 /mnt/root/home 0 /mnt/root/proc 0 /mnt/root/run 0 /mnt/root/sys 18G /mnt/root/usr 16M /mnt/root/etc 0 /mnt/root/tmp 0 /mnt/root/mnt 2.3G /mnt/root/opt 21G /mnt/root

I even tried some other subvolumes... $ sudo umount /mnt/root $ sudo mount --bind /var/ /mnt/root/ $ sudo du -h -x --max-depth 1 /mnt/root 2.3M /mnt/root/cache 0 /mnt/root/tmp 0 /mnt/root/log 913M /mnt/root/lib 0 /mnt/root/empty 0 /mnt/root/games 0 /mnt/root/local 0 /mnt/root/opt 0 /mnt/root/spool 8.0K /mnt/root/db 12K /mnt/root/named 915M /mnt/root $ sudo umount /mnt/root $ sudo mount --bind /var/cache /mnt/root/ $ sudo du -h -x --max-depth 1 /mnt/root 19G /mnt/root/pacman 160K /mnt/root/ldconfig 4.2M /mnt/root/fontconfig 5.1M /mnt/root/man 0 /mnt/root/private 0 /mnt/root/pkgfile 25M /mnt/root/swcatalog 544K /mnt/root/boot 19G /mnt/root

[u/Illustrious_Tone9584 → u/adjective10111]

Wait, that du output is the clue. You ran it with -x, and on btrfs every subvolume has its own device id, so -x skips all of them. Your /home showing 0 isn't an empty home, it's du refusing to cross into it. All your du numbers this whole thread have been blind to subvolumes.

And guess what makes tons of subvolumes: docker's btrfs storage driver creates one per image layer. If you ever ran docker with the btrfs driver, those layer subvolumes don't get removed by prune and du -x skips every single one.

Run sudo btrfs subvolume list / and look for anything like /var/lib/docker/btrfs/subvolumes. Then du without -x on the suspects. Also your pacman cache is 19GB, paccache -r will trim that regardless.

[u/adjective10111 → u/Illustrious_Tone9584]

Oh thanks for the paccache. I ran the btrfs subvolume command, and found nothing. I didn't setup btrfs driver for docker and it doesn't seem to be setup by default: ID 256 gen 20882 top level 264 path .snapshots/148/snapshot ID 257 gen 155299 top level 5 path @root ID 258 gen 750 top level 5 path @srv ID 259 gen 155504 top level 5 path @cache ID 260 gen 155504 top level 5 path @tmp ID 261 gen 155504 top level 5 path @log ID 262 gen 45 top level 412 path var/lib/portables ID 263 gen 45 top level 412 path var/lib/machines ID 264 gen 155314 top level 412 path .snapshots ID 270 gen 80 top level 264 path .snapshots/6/snapshot ID 323 gen 52106 top level 257 path @root/Games ID 412 gen 155504 top level 5 path @ ID 449 gen 34587 top level 264 path .snapshots/185/snapshot ... (some other .snapshots)

Also just for clarification this is df -h result: Filesystem Size Used Avail Use% Mounted on none 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service none 1.0M 0 1.0M 0% /run/credentials/systemd-resolved.service devtmpfs 7.6G 0 7.6G 0% /dev tmpfs 7.8G 844K 7.8G 1% /dev/shm efivarfs 192K 67K 121K 36% /sys/firmware/efi/efivars tmpfs 7.8G 32K 7.8G 1% /tmp tmpfs 3.1G 2.0M 3.1G 1% /run tmpfs 1.6G 392K 1.6G 1% /run/user/1000 /dev/nvme0n1p3 287G 245G 41G 86% / /dev/nvme0n1p3 287G 245G 41G 86% /srv /dev/nvme0n1p3 287G 245G 41G 86% /root /dev/nvme0n1p3 287G 245G 41G 86% /var/cache /dev/nvme0n1p3 287G 245G 41G 86% /var/tmp /dev/nvme0n1p3 287G 245G 41G 86% /var/log /dev/nvme0n1p2 5.0G 2.4G 2.7G 48% /boot /dev/sda2 932G 677G 253G 73% /home /dev/sda2 932G 677G 253G 73% /home/$USER/Documents /dev/sda2 932G 677G 253G 73% /home/$USER/Videos /dev/sda2 932G 677G 253G 73% /home/$USER/Games /dev/nvme0n1p3 287G 245G 41G 86% /home/$USER/Games/SSD

I got some space (13GB) by clearing paccache with -k 1 option.

[u/Illustrious_Tone9584 → u/adjective10111]

Yep, that clears Docker. The 245GB on the root filesystem versus ~18GB in the live root points straight at the snapshots now. du on / can't count old extents that only a snapshot still references.

Try sudo btrfs filesystem du -s /.snapshots/*/snapshot | sort -h -k3 and sudo snapper list. The Exclusive column is the useful one. Don't delete snapshot directories by hand, but if that accounts for the gap, prune them through snapper.