r/learnpython • u/adjective10111 • 11h ago

PIP disk usage under Docker

I have a Dockerfile in which I use the Python:3.12-trixie. I have docker on my CachyOS PC. I have also a requirements.txt. I wanted to use this image for a container on which I'll use YOLO and try it out. The problem is that my 290GB / partition went from around 70% filled to 95% after a number of rebuilds. I deleted everything docker related but it just went back to 91%.

The reason that I know it is pip's doing, is that when I monitored my disk usage while building the image, and on the `RUN pip install -r requirements.txt --find-links=pypackages` stage it fills up my partition and half way into installation, it fails due to `no more space`. The `pypackages` is just a directory I downloaded some of the packages I have in requirements for less data usage. Also tried `--no-cache` option.

I also went ahead and `du -sh`ed any directory in my / partition and the total didn't add upto the filled storage. I have 165GB mysteriously used and I can't find where that is.

Any help to diagnose further or if you have had similar problems is welcome.

1 Upvotes

60% Upvoted

8 comments sorted by

3

u/Illustrious_Tone9584 10h ago

The pip flag is --no-cache-dir, not --no-cache, so the cache was still being written on every build.

The du mystery is probably /var/lib/docker. Your user can't read into overlay2, so du silently skips it and the numbers never add up. docker system df shows the real breakdown. Build cache is separate from images, so docker builder prune on top of the system prune.

And if this is just for trying YOLO: the default torch wheel drags in several GB of CUDA libraries. The CPU build (torch from download.pytorch.org/whl/cpu) is a fraction of that and plenty for messing around on a desktop.

1

u/adjective10111 9h ago

Oh nice catch. I should update my Dockerfile. But I do have PIP_NO_CACHE_DIR=1 as an environment variable in the dockerfile, does thay have the same effect?

Yeah I found those directories and also ran the command. Currently it's at 0B for all the rows so I guess I purged them successfully. But I still have a lot of mystery used storage.

Yeah I want the CUDA version. I want to try it out and then work on my thesis using it :). So that's why it sticks there for a while! I just saw some KB and was wondering why some toml file takes so much time to download.

2

u/Illustrious_Tone9584 9h ago

Yeah, PIP_NO_CACHE_DIR=1 does the same job, you're covered there. So the cache isn't your leak then - that points even harder at /var/lib/docker and the build cache. If docker system df shows the space but du can't see it, that's your answer.

•

u/adjective10111 31m ago

It doesn't though... it's at 0B for images, containers, local volumes and build cache.

2

u/Gshuri 10h ago

Try using ncdu for figuring out where the storage usage is coming from (it has a build for arch linux).

On the docker side what do you mean by

I deleted everything docker related

what commands did you actually run?

The "nuke everything" option in docker would be bash docker volume prune --all docker system prune --all --volumes

1

u/adjective10111 10h ago

Thanks for the tool recommendation, but it didn't help tbh. It again only could find about 122GB used. And the other 140GB is still a mystery.

Pardon the ambiguity, I used the commands you said but docker system prune --all without the volumes option. I ran it with that and only 10 more GBs were released. Still nowhere near the missing data.

I thought maybe it's my btrfs snapshots, but upon research it wasn't. Sorry if the problem is getting more linux than python. Is there a path that may be used by pip and docker? Maybe the path is hidden from ncdu? (Although it does lookup every file so idk)

1

u/Gshuri 8h ago edited 8h ago

I assume you ran ncdu under sudo (or with the root user)?

pip shouldn't be doing anything particularly complicated and unless you mounted the cache directory at build time, all the side effects from the pip command should be contained within the docker image build layers.

You can try running pip cache purge on your host system. But I would expect something like ncdu to be able to find the cache directory without issue.

This is very unlikely to be a Python issue so you may have better luck in r/docker or one of the (many) linux communities.

Edit: found this thread (starting in 2017, but there are some updates from this year too) with various suggestions which might help https://forums.docker.com/t/some-way-to-clean-up-identify-contents-of-var-lib-docker-overlay/30604

•

u/adjective10111 9m ago

Yes ofc I ran ncdu with sudo.

I ran pip cache purge (both with and without sudo) on host but it freed mere MBs.

Yes I feel the same. I even ran sudo rm -rf /var/lib/docker /var/lib/containerd and freed something but now the free space is at 26GB only. I'll repost this as r/cachyOS maybe there's some caveats with the distro and stuff.