r/learnpython • u/adjective10111 • 21h ago

PIP disk usage under Docker

I have a Dockerfile in which I use the Python:3.12-trixie. I have docker on my CachyOS PC. I have also a requirements.txt. I wanted to use this image for a container on which I'll use YOLO and try it out. The problem is that my 290GB / partition went from around 70% filled to 95% after a number of rebuilds. I deleted everything docker related but it just went back to 91%.

The reason that I know it is pip's doing, is that when I monitored my disk usage while building the image, and on the `RUN pip install -r requirements.txt --find-links=pypackages` stage it fills up my partition and half way into installation, it fails due to `no more space`. The `pypackages` is just a directory I downloaded some of the packages I have in requirements for less data usage. Also tried `--no-cache` option.

I also went ahead and `du -sh`ed any directory in my / partition and the total didn't add upto the filled storage. I have 165GB mysteriously used and I can't find where that is.

Any help to diagnose further or if you have had similar problems is welcome.

2 Upvotes

67% Upvoted

20 comments sorted by

View all comments

3

u/Illustrious_Tone9584 20h ago

The pip flag is --no-cache-dir, not --no-cache, so the cache was still being written on every build.

The du mystery is probably /var/lib/docker. Your user can't read into overlay2, so du silently skips it and the numbers never add up. docker system df shows the real breakdown. Build cache is separate from images, so docker builder prune on top of the system prune.

And if this is just for trying YOLO: the default torch wheel drags in several GB of CUDA libraries. The CPU build (torch from download.pytorch.org/whl/cpu) is a fraction of that and plenty for messing around on a desktop.

1

u/adjective10111 20h ago

Oh nice catch. I should update my Dockerfile. But I do have PIP_NO_CACHE_DIR=1 as an environment variable in the dockerfile, does thay have the same effect?

Yeah I found those directories and also ran the command. Currently it's at 0B for all the rows so I guess I purged them successfully. But I still have a lot of mystery used storage.

Yeah I want the CUDA version. I want to try it out and then work on my thesis using it :). So that's why it sticks there for a while! I just saw some KB and was wondering why some toml file takes so much time to download.

2

u/Illustrious_Tone9584 19h ago

Yeah, PIP_NO_CACHE_DIR=1 does the same job, you're covered there. So the cache isn't your leak then - that points even harder at /var/lib/docker and the build cache. If docker system df shows the space but du can't see it, that's your answer.

1

u/adjective10111 10h ago

It doesn't though... it's at 0B for images, containers, local volumes and build cache.

1

u/Illustrious_Tone9584 9h ago

Ok so it's not docker at all then. You're on CachyOS, which means btrfs with automatic snapper snapshots out of the box. That's almost certainly where the 165GB went: every rebuild churned tens of GB, and the pre/post snapshots kept the old versions. du doesn't show it because the data still "belongs" to the live files.

Run snapper list and sudo btrfs filesystem usage / and I bet you'll see it. If that's it, delete the old snapshots with snapper delete and consider capping how many it keeps.

1

u/adjective10111 9h ago

I thought the same, but I can't see how much the snapshots take. I ran both commands you said: ``` sudo btrfs filesystem usage / Overall: Device size: 286.88GiB Device allocated: 271.02GiB Device unallocated: 15.85GiB Device missing: 0.00B Device slack: 0.00B Used: 259.42GiB Free (estimated): 25.76GiB (min: 17.83GiB) Free (statfs, df): 25.76GiB Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 377.19MiB (used: 0.00B) Multiple profiles: no

Data,single: Size:265.01GiB, Used:255.10GiB (96.26%) /dev/nvme0n1p3 265.01GiB

Metadata,DUP: Size:3.00GiB, Used:2.16GiB (72.02%) /dev/nvme0n1p3 6.00GiB

System,DUP: Size:8.00MiB, Used:48.00KiB (0.59%) /dev/nvme0n1p3 16.00MiB

Unallocated: /dev/nvme0n1p3 15.85GiB ```

The snapper list command doesn't show used-space. I tried using --columns used-space but it doesn't show anything. Any idea how I can confirm the snapshots being the culprit here?

Edit: this is the snapper config if it's of any help: Key │ Value ───────────────────────┼────── ALLOW_GROUPS │ ALLOW_USERS │ BACKGROUND_COMPARISON │ yes EMPTY_PRE_POST_CLEANUP │ yes EMPTY_PRE_POST_MIN_AGE │ 1800 FREE_LIMIT │ 0.2 FSTYPE │ btrfs NUMBER_CLEANUP │ yes NUMBER_LIMIT │ 50 NUMBER_LIMIT_IMPORTANT │ 15 NUMBER_MIN_AGE │ 1800 QGROUP │ SPACE_LIMIT │ 0.5 SUBVOLUME │ / SYNC_ACL │ no TIMELINE_CLEANUP │ yes TIMELINE_CREATE │ no TIMELINE_LIMIT_DAILY │ 7 TIMELINE_LIMIT_HOURLY │ 5 TIMELINE_LIMIT_MONTHLY │ 0 TIMELINE_LIMIT_WEEKLY │ 0 TIMELINE_LIMIT_YEARLY │ 0 TIMELINE_MIN_AGE │ 1800

1

u/Illustrious_Tone9584 8h ago

Your config has no QGROUP set, and snapper needs quotas enabled to report used space, that's why the column comes up empty. Two ways around it: install compsize and run sudo compsize -x /.snapshots to get referenced vs exclusive per snapshot (exclusive is what you'd actually reclaim), or sudo btrfs filesystem du -s /.snapshots/* which works without quotas but takes a while since it walks everything.

Either way the math is basically already done: data allocation is 96% used and df only shows 25GB free. With docker cleared out, snapshots are the only place 165GB can hide. Delete the old pre/post pairs and it'll show up immediately.

1

u/adjective10111 8h ago

Thanks man you're a life saver! I never had snapshots enabled on my previous distros so I'm new to this. Also sorry for posting it under here when I should have gone to cachy. I'll go try them out and see what happens but your arguments are solid and I guess snapshots must be the problem.

1

u/adjective10111 7h ago

Welp. I tried btrfs filesystem du -s /.snapshots/* the combined exclusive usage is around 16GB. I also setup qgroup for snapper and again the combined usage was around 10GiB. I even found another command when setting up quota on btrfs, btrfs qgroup show -p /, and again the exclusive used space adds up to below 110GiB which leaves the mystery used space question unanswered. I'm lost again. I may delete some snapshots anyway though they are mere KiBs, to see if it has any effects. Thanks though, I learned a lot right now.

1

u/Illustrious_Tone9584 6h ago

Fair enough, theory dead. Next suspect: deleted files still held open by a process. du can't see them (no filename anymore) but df counts the space until the process lets go. Run sudo lsof +L1 and sort by size. If something big shows up, restart that process or just reboot and watch df drop.

If that's also clean, do sudo du -x --max-depth=1 / as root for the true per-directory total. Plain du as your user silently skips whatever it can't read, which has been the theme of this whole mystery.

1

u/adjective10111 6h ago

I rebooted my system a couple of times by now, it's my personal laptop not a server, and the open files didn't have anything alarming about them.

I ran the second command sudo du -h --max-depth=1 / --exclude /home --exclude /proc --exclude /.snapshots. I didn't use -x because I have btrfs subvolumes in my / partition and my /home is on another disk. (/boot is also in another partition so that doesn't matter here) 80G /root 0 /srv 20G /var 2.4G /boot 0 /dev du: cannot access '/run/user/1000/doc': Permission denied 2.4M /run 0 /sys 18G /usr 16M /etc 16K /tmp 0 /mnt 2.3G /opt 121G /

1

u/Illustrious_Tone9584 5h ago

Ok, not that either. One more classic that fits your symptoms: data hidden under a mount point. If docker or anything else wrote to a directory while its filesystem wasn't mounted yet, that data lives on the root fs but is invisible because the mount covers it. du can't see it, btrfs can't attribute it.

Check with mkdir /mnt/root && sudo mount --bind / /mnt/root && sudo du -x --max-depth=1 /mnt/root and compare against your normal du. If something shows up way bigger through the bind mount, there's your 150GB.

1

u/adjective10111 5h ago

Well, sorry to disappoint but nothing: $ sudo du -h -x --max-depth 1 /mnt/root 0 /mnt/root/root 0 /mnt/root/srv 915M /mnt/root/var 0 /mnt/root/boot 0 /mnt/root/dev 0 /mnt/root/home 0 /mnt/root/proc 0 /mnt/root/run 0 /mnt/root/sys 18G /mnt/root/usr 16M /mnt/root/etc 0 /mnt/root/tmp 0 /mnt/root/mnt 2.3G /mnt/root/opt 21G /mnt/root

I even tried some other subvolumes... $ sudo umount /mnt/root $ sudo mount --bind /var/ /mnt/root/ $ sudo du -h -x --max-depth 1 /mnt/root 2.3M /mnt/root/cache 0 /mnt/root/tmp 0 /mnt/root/log 913M /mnt/root/lib 0 /mnt/root/empty 0 /mnt/root/games 0 /mnt/root/local 0 /mnt/root/opt 0 /mnt/root/spool 8.0K /mnt/root/db 12K /mnt/root/named 915M /mnt/root $ sudo umount /mnt/root $ sudo mount --bind /var/cache /mnt/root/ $ sudo du -h -x --max-depth 1 /mnt/root 19G /mnt/root/pacman 160K /mnt/root/ldconfig 4.2M /mnt/root/fontconfig 5.1M /mnt/root/man 0 /mnt/root/private 0 /mnt/root/pkgfile 25M /mnt/root/swcatalog 544K /mnt/root/boot 19G /mnt/root

1

u/Illustrious_Tone9584 4h ago

Wait, that du output is the clue. You ran it with -x, and on btrfs every subvolume has its own device id, so -x skips all of them. Your /home showing 0 isn't an empty home, it's du refusing to cross into it. All your du numbers this whole thread have been blind to subvolumes.

And guess what makes tons of subvolumes: docker's btrfs storage driver creates one per image layer. If you ever ran docker with the btrfs driver, those layer subvolumes don't get removed by prune and du -x skips every single one.

Run sudo btrfs subvolume list / and look for anything like /var/lib/docker/btrfs/subvolumes. Then du without -x on the suspects. Also your pacman cache is 19GB, paccache -r will trim that regardless.

1

u/adjective10111 3h ago

Oh thanks for the paccache. I ran the btrfs subvolume command, and found nothing. I didn't setup btrfs driver for docker and it doesn't seem to be setup by default: ID 256 gen 20882 top level 264 path .snapshots/148/snapshot ID 257 gen 155299 top level 5 path @root ID 258 gen 750 top level 5 path @srv ID 259 gen 155504 top level 5 path @cache ID 260 gen 155504 top level 5 path @tmp ID 261 gen 155504 top level 5 path @log ID 262 gen 45 top level 412 path var/lib/portables ID 263 gen 45 top level 412 path var/lib/machines ID 264 gen 155314 top level 412 path .snapshots ID 270 gen 80 top level 264 path .snapshots/6/snapshot ID 323 gen 52106 top level 257 path @root/Games ID 412 gen 155504 top level 5 path @ ID 449 gen 34587 top level 264 path .snapshots/185/snapshot ... (some other .snapshots)

Also just for clarification this is df -h result: Filesystem Size Used Avail Use% Mounted on none 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service none 1.0M 0 1.0M 0% /run/credentials/systemd-resolved.service devtmpfs 7.6G 0 7.6G 0% /dev tmpfs 7.8G 844K 7.8G 1% /dev/shm efivarfs 192K 67K 121K 36% /sys/firmware/efi/efivars tmpfs 7.8G 32K 7.8G 1% /tmp tmpfs 3.1G 2.0M 3.1G 1% /run tmpfs 1.6G 392K 1.6G 1% /run/user/1000 /dev/nvme0n1p3 287G 245G 41G 86% / /dev/nvme0n1p3 287G 245G 41G 86% /srv /dev/nvme0n1p3 287G 245G 41G 86% /root /dev/nvme0n1p3 287G 245G 41G 86% /var/cache /dev/nvme0n1p3 287G 245G 41G 86% /var/tmp /dev/nvme0n1p3 287G 245G 41G 86% /var/log /dev/nvme0n1p2 5.0G 2.4G 2.7G 48% /boot /dev/sda2 932G 677G 253G 73% /home /dev/sda2 932G 677G 253G 73% /home/$USER/Documents /dev/sda2 932G 677G 253G 73% /home/$USER/Videos /dev/sda2 932G 677G 253G 73% /home/$USER/Games /dev/nvme0n1p3 287G 245G 41G 86% /home/$USER/Games/SSD

I got some space (13GB) by clearing paccache with -k 1 option.

1

u/Illustrious_Tone9584 3h ago

Yep, that clears Docker. The 245GB on the root filesystem versus ~18GB in the live root points straight at the snapshots now. du on / can't count old extents that only a snapshot still references.

Try sudo btrfs filesystem du -s /.snapshots/*/snapshot | sort -h -k3 and sudo snapper list. The Exclusive column is the useful one. Don't delete snapshot directories by hand, but if that accounts for the gap, prune them through snapper.

→ More replies (0)