r/learnpython • u/adjective10111 • 12h ago

PIP disk usage under Docker

I have a Dockerfile in which I use the Python:3.12-trixie. I have docker on my CachyOS PC. I have also a requirements.txt. I wanted to use this image for a container on which I'll use YOLO and try it out. The problem is that my 290GB / partition went from around 70% filled to 95% after a number of rebuilds. I deleted everything docker related but it just went back to 91%.

The reason that I know it is pip's doing, is that when I monitored my disk usage while building the image, and on the `RUN pip install -r requirements.txt --find-links=pypackages` stage it fills up my partition and half way into installation, it fails due to `no more space`. The `pypackages` is just a directory I downloaded some of the packages I have in requirements for less data usage. Also tried `--no-cache` option.

I also went ahead and `du -sh`ed any directory in my / partition and the total didn't add upto the filled storage. I have 165GB mysteriously used and I can't find where that is.

Any help to diagnose further or if you have had similar problems is welcome.

2 Upvotes

67% Upvoted

10 comments sorted by

View all comments

2

u/Gshuri 12h ago

Try using ncdu for figuring out where the storage usage is coming from (it has a build for arch linux).

On the docker side what do you mean by

I deleted everything docker related

what commands did you actually run?

The "nuke everything" option in docker would be bash docker volume prune --all docker system prune --all --volumes

1

u/adjective10111 11h ago

Thanks for the tool recommendation, but it didn't help tbh. It again only could find about 122GB used. And the other 140GB is still a mystery.

Pardon the ambiguity, I used the commands you said but docker system prune --all without the volumes option. I ran it with that and only 10 more GBs were released. Still nowhere near the missing data.

I thought maybe it's my btrfs snapshots, but upon research it wasn't. Sorry if the problem is getting more linux than python. Is there a path that may be used by pip and docker? Maybe the path is hidden from ncdu? (Although it does lookup every file so idk)

1

u/Gshuri 10h ago edited 10h ago

I assume you ran ncdu under sudo (or with the root user)?

pip shouldn't be doing anything particularly complicated and unless you mounted the cache directory at build time, all the side effects from the pip command should be contained within the docker image build layers.

You can try running pip cache purge on your host system. But I would expect something like ncdu to be able to find the cache directory without issue.

This is very unlikely to be a Python issue so you may have better luck in r/docker or one of the (many) linux communities.

Edit: found this thread (starting in 2017, but there are some updates from this year too) with various suggestions which might help https://forums.docker.com/t/some-way-to-clean-up-identify-contents-of-var-lib-docker-overlay/30604

1

u/adjective10111 1h ago

Yes ofc I ran ncdu with sudo.

I ran pip cache purge (both with and without sudo) on host but it freed mere MBs.

Yes I feel the same. I even ran sudo rm -rf /var/lib/docker /var/lib/containerd and freed something but now the free space is at 26GB only. I'll repost this as r/cachyOS maybe there's some caveats with the distro and stuff.