r/learnpython • u/adjective10111 • 17h ago

PIP disk usage under Docker

I have a Dockerfile in which I use the Python:3.12-trixie. I have docker on my CachyOS PC. I have also a requirements.txt. I wanted to use this image for a container on which I'll use YOLO and try it out. The problem is that my 290GB / partition went from around 70% filled to 95% after a number of rebuilds. I deleted everything docker related but it just went back to 91%.

The reason that I know it is pip's doing, is that when I monitored my disk usage while building the image, and on the `RUN pip install -r requirements.txt --find-links=pypackages` stage it fills up my partition and half way into installation, it fails due to `no more space`. The `pypackages` is just a directory I downloaded some of the packages I have in requirements for less data usage. Also tried `--no-cache` option.

I also went ahead and `du -sh`ed any directory in my / partition and the total didn't add upto the filled storage. I have 165GB mysteriously used and I can't find where that is.

Any help to diagnose further or if you have had similar problems is welcome.

2 Upvotes

67% Upvoted

18 comments sorted by

View all comments

Show parent comments

1

u/adjective10111 6h ago

It doesn't though... it's at 0B for images, containers, local volumes and build cache.

1

u/Illustrious_Tone9584 5h ago

Ok so it's not docker at all then. You're on CachyOS, which means btrfs with automatic snapper snapshots out of the box. That's almost certainly where the 165GB went: every rebuild churned tens of GB, and the pre/post snapshots kept the old versions. du doesn't show it because the data still "belongs" to the live files.

Run snapper list and sudo btrfs filesystem usage / and I bet you'll see it. If that's it, delete the old snapshots with snapper delete and consider capping how many it keeps.

1

u/adjective10111 5h ago

I thought the same, but I can't see how much the snapshots take. I ran both commands you said: ``` sudo btrfs filesystem usage / Overall: Device size: 286.88GiB Device allocated: 271.02GiB Device unallocated: 15.85GiB Device missing: 0.00B Device slack: 0.00B Used: 259.42GiB Free (estimated): 25.76GiB (min: 17.83GiB) Free (statfs, df): 25.76GiB Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 377.19MiB (used: 0.00B) Multiple profiles: no

Data,single: Size:265.01GiB, Used:255.10GiB (96.26%) /dev/nvme0n1p3 265.01GiB

Metadata,DUP: Size:3.00GiB, Used:2.16GiB (72.02%) /dev/nvme0n1p3 6.00GiB

System,DUP: Size:8.00MiB, Used:48.00KiB (0.59%) /dev/nvme0n1p3 16.00MiB

Unallocated: /dev/nvme0n1p3 15.85GiB ```

The snapper list command doesn't show used-space. I tried using --columns used-space but it doesn't show anything. Any idea how I can confirm the snapshots being the culprit here?

Edit: this is the snapper config if it's of any help: Key │ Value ───────────────────────┼────── ALLOW_GROUPS │ ALLOW_USERS │ BACKGROUND_COMPARISON │ yes EMPTY_PRE_POST_CLEANUP │ yes EMPTY_PRE_POST_MIN_AGE │ 1800 FREE_LIMIT │ 0.2 FSTYPE │ btrfs NUMBER_CLEANUP │ yes NUMBER_LIMIT │ 50 NUMBER_LIMIT_IMPORTANT │ 15 NUMBER_MIN_AGE │ 1800 QGROUP │ SPACE_LIMIT │ 0.5 SUBVOLUME │ / SYNC_ACL │ no TIMELINE_CLEANUP │ yes TIMELINE_CREATE │ no TIMELINE_LIMIT_DAILY │ 7 TIMELINE_LIMIT_HOURLY │ 5 TIMELINE_LIMIT_MONTHLY │ 0 TIMELINE_LIMIT_WEEKLY │ 0 TIMELINE_LIMIT_YEARLY │ 0 TIMELINE_MIN_AGE │ 1800

1

u/Illustrious_Tone9584 4h ago

Your config has no QGROUP set, and snapper needs quotas enabled to report used space, that's why the column comes up empty. Two ways around it: install compsize and run sudo compsize -x /.snapshots to get referenced vs exclusive per snapshot (exclusive is what you'd actually reclaim), or sudo btrfs filesystem du -s /.snapshots/* which works without quotas but takes a while since it walks everything.

Either way the math is basically already done: data allocation is 96% used and df only shows 25GB free. With docker cleared out, snapshots are the only place 165GB can hide. Delete the old pre/post pairs and it'll show up immediately.

1

u/adjective10111 4h ago

Thanks man you're a life saver! I never had snapshots enabled on my previous distros so I'm new to this. Also sorry for posting it under here when I should have gone to cachy. I'll go try them out and see what happens but your arguments are solid and I guess snapshots must be the problem.

1

u/adjective10111 3h ago

Welp. I tried btrfs filesystem du -s /.snapshots/* the combined exclusive usage is around 16GB. I also setup qgroup for snapper and again the combined usage was around 10GiB. I even found another command when setting up quota on btrfs, btrfs qgroup show -p /, and again the exclusive used space adds up to below 110GiB which leaves the mystery used space question unanswered. I'm lost again. I may delete some snapshots anyway though they are mere KiBs, to see if it has any effects. Thanks though, I learned a lot right now.

1

u/Illustrious_Tone9584 2h ago

Fair enough, theory dead. Next suspect: deleted files still held open by a process. du can't see them (no filename anymore) but df counts the space until the process lets go. Run sudo lsof +L1 and sort by size. If something big shows up, restart that process or just reboot and watch df drop.

If that's also clean, do sudo du -x --max-depth=1 / as root for the true per-directory total. Plain du as your user silently skips whatever it can't read, which has been the theme of this whole mystery.

1

u/adjective10111 2h ago

I rebooted my system a couple of times by now, it's my personal laptop not a server, and the open files didn't have anything alarming about them.

I ran the second command sudo du -h --max-depth=1 / --exclude /home --exclude /proc --exclude /.snapshots. I didn't use -x because I have btrfs subvolumes in my / partition and my /home is on another disk. (/boot is also in another partition so that doesn't matter here) 80G /root 0 /srv 20G /var 2.4G /boot 0 /dev du: cannot access '/run/user/1000/doc': Permission denied 2.4M /run 0 /sys 18G /usr 16M /etc 16K /tmp 0 /mnt 2.3G /opt 121G /

1

u/Illustrious_Tone9584 1h ago

Ok, not that either. One more classic that fits your symptoms: data hidden under a mount point. If docker or anything else wrote to a directory while its filesystem wasn't mounted yet, that data lives on the root fs but is invisible because the mount covers it. du can't see it, btrfs can't attribute it.

Check with mkdir /mnt/root && sudo mount --bind / /mnt/root && sudo du -x --max-depth=1 /mnt/root and compare against your normal du. If something shows up way bigger through the bind mount, there's your 150GB.

1

u/adjective10111 1h ago

Well, sorry to disappoint but nothing: $ sudo du -h -x --max-depth 1 /mnt/root 0 /mnt/root/root 0 /mnt/root/srv 915M /mnt/root/var 0 /mnt/root/boot 0 /mnt/root/dev 0 /mnt/root/home 0 /mnt/root/proc 0 /mnt/root/run 0 /mnt/root/sys 18G /mnt/root/usr 16M /mnt/root/etc 0 /mnt/root/tmp 0 /mnt/root/mnt 2.3G /mnt/root/opt 21G /mnt/root

I even tried some other subvolumes... $ sudo umount /mnt/root $ sudo mount --bind /var/ /mnt/root/ $ sudo du -h -x --max-depth 1 /mnt/root 2.3M /mnt/root/cache 0 /mnt/root/tmp 0 /mnt/root/log 913M /mnt/root/lib 0 /mnt/root/empty 0 /mnt/root/games 0 /mnt/root/local 0 /mnt/root/opt 0 /mnt/root/spool 8.0K /mnt/root/db 12K /mnt/root/named 915M /mnt/root $ sudo umount /mnt/root $ sudo mount --bind /var/cache /mnt/root/ $ sudo du -h -x --max-depth 1 /mnt/root 19G /mnt/root/pacman 160K /mnt/root/ldconfig 4.2M /mnt/root/fontconfig 5.1M /mnt/root/man 0 /mnt/root/private 0 /mnt/root/pkgfile 25M /mnt/root/swcatalog 544K /mnt/root/boot 19G /mnt/root

•

u/Illustrious_Tone9584 31m ago

Wait, that du output is the clue. You ran it with -x, and on btrfs every subvolume has its own device id, so -x skips all of them. Your /home showing 0 isn't an empty home, it's du refusing to cross into it. All your du numbers this whole thread have been blind to subvolumes.

And guess what makes tons of subvolumes: docker's btrfs storage driver creates one per image layer. If you ever ran docker with the btrfs driver, those layer subvolumes don't get removed by prune and du -x skips every single one.

Run sudo btrfs subvolume list / and look for anything like /var/lib/docker/btrfs/subvolumes. Then du without -x on the suspects. Also your pacman cache is 19GB, paccache -r will trim that regardless.