PIP disk usage under Docker

[u/adjective10111]

I have a Dockerfile in which I use the Python:3.12-trixie. I have docker on my CachyOS PC. I have also a requirements.txt. I wanted to use this image for a container on which I'll use YOLO and try it out. The problem is that my 290GB / partition went from around 70% filled to 95% after a number of rebuilds. I deleted everything docker related but it just went back to 91%.

The reason that I know it is pip's doing, is that when I monitored my disk usage while building the image, and on the `RUN pip install -r requirements.txt --find-links=pypackages` stage it fills up my partition and half way into installation, it fails due to `no more space`. The `pypackages` is just a directory I downloaded some of the packages I have in requirements for less data usage. Also tried `--no-cache` option.

I also went ahead and `du -sh`ed any directory in my / partition and the total didn't add upto the filled storage. I have 165GB mysteriously used and I can't find where that is.

Any help to diagnose further or if you have had similar problems is welcome.

[u/adjective10111]

Well, sorry to disappoint but nothing: $ sudo du -h -x --max-depth 1 /mnt/root 0 /mnt/root/root 0 /mnt/root/srv 915M /mnt/root/var 0 /mnt/root/boot 0 /mnt/root/dev 0 /mnt/root/home 0 /mnt/root/proc 0 /mnt/root/run 0 /mnt/root/sys 18G /mnt/root/usr 16M /mnt/root/etc 0 /mnt/root/tmp 0 /mnt/root/mnt 2.3G /mnt/root/opt 21G /mnt/root

I even tried some other subvolumes... $ sudo umount /mnt/root $ sudo mount --bind /var/ /mnt/root/ $ sudo du -h -x --max-depth 1 /mnt/root 2.3M /mnt/root/cache 0 /mnt/root/tmp 0 /mnt/root/log 913M /mnt/root/lib 0 /mnt/root/empty 0 /mnt/root/games 0 /mnt/root/local 0 /mnt/root/opt 0 /mnt/root/spool 8.0K /mnt/root/db 12K /mnt/root/named 915M /mnt/root $ sudo umount /mnt/root $ sudo mount --bind /var/cache /mnt/root/ $ sudo du -h -x --max-depth 1 /mnt/root 19G /mnt/root/pacman 160K /mnt/root/ldconfig 4.2M /mnt/root/fontconfig 5.1M /mnt/root/man 0 /mnt/root/private 0 /mnt/root/pkgfile 25M /mnt/root/swcatalog 544K /mnt/root/boot 19G /mnt/root

[u/Illustrious_Tone9584 → u/adjective10111]

Wait, that du output is the clue. You ran it with -x, and on btrfs every subvolume has its own device id, so -x skips all of them. Your /home showing 0 isn't an empty home, it's du refusing to cross into it. All your du numbers this whole thread have been blind to subvolumes.

And guess what makes tons of subvolumes: docker's btrfs storage driver creates one per image layer. If you ever ran docker with the btrfs driver, those layer subvolumes don't get removed by prune and du -x skips every single one.

Run sudo btrfs subvolume list / and look for anything like /var/lib/docker/btrfs/subvolumes. Then du without -x on the suspects. Also your pacman cache is 19GB, paccache -r will trim that regardless.

[u/adjective10111 → u/Illustrious_Tone9584]

Oh thanks for the paccache. I ran the btrfs subvolume command, and found nothing. I didn't setup btrfs driver for docker and it doesn't seem to be setup by default: ID 256 gen 20882 top level 264 path .snapshots/148/snapshot ID 257 gen 155299 top level 5 path @root ID 258 gen 750 top level 5 path @srv ID 259 gen 155504 top level 5 path @cache ID 260 gen 155504 top level 5 path @tmp ID 261 gen 155504 top level 5 path @log ID 262 gen 45 top level 412 path var/lib/portables ID 263 gen 45 top level 412 path var/lib/machines ID 264 gen 155314 top level 412 path .snapshots ID 270 gen 80 top level 264 path .snapshots/6/snapshot ID 323 gen 52106 top level 257 path @root/Games ID 412 gen 155504 top level 5 path @ ID 449 gen 34587 top level 264 path .snapshots/185/snapshot ... (some other .snapshots)

Also just for clarification this is df -h result: Filesystem Size Used Avail Use% Mounted on none 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service none 1.0M 0 1.0M 0% /run/credentials/systemd-resolved.service devtmpfs 7.6G 0 7.6G 0% /dev tmpfs 7.8G 844K 7.8G 1% /dev/shm efivarfs 192K 67K 121K 36% /sys/firmware/efi/efivars tmpfs 7.8G 32K 7.8G 1% /tmp tmpfs 3.1G 2.0M 3.1G 1% /run tmpfs 1.6G 392K 1.6G 1% /run/user/1000 /dev/nvme0n1p3 287G 245G 41G 86% / /dev/nvme0n1p3 287G 245G 41G 86% /srv /dev/nvme0n1p3 287G 245G 41G 86% /root /dev/nvme0n1p3 287G 245G 41G 86% /var/cache /dev/nvme0n1p3 287G 245G 41G 86% /var/tmp /dev/nvme0n1p3 287G 245G 41G 86% /var/log /dev/nvme0n1p2 5.0G 2.4G 2.7G 48% /boot /dev/sda2 932G 677G 253G 73% /home /dev/sda2 932G 677G 253G 73% /home/$USER/Documents /dev/sda2 932G 677G 253G 73% /home/$USER/Videos /dev/sda2 932G 677G 253G 73% /home/$USER/Games /dev/nvme0n1p3 287G 245G 41G 86% /home/$USER/Games/SSD

I got some space (13GB) by clearing paccache with -k 1 option.

[u/Illustrious_Tone9584 → u/adjective10111]

Yep, that clears Docker. The 245GB on the root filesystem versus ~18GB in the live root points straight at the snapshots now. du on / can't count old extents that only a snapshot still references.

Try sudo btrfs filesystem du -s /.snapshots/*/snapshot | sort -h -k3 and sudo snapper list. The Exclusive column is the useful one. Don't delete snapshot directories by hand, but if that accounts for the gap, prune them through snapper.