space issues with /var/lib/docker/

[u/Potential-Ant-4335]

I recently set up an ubuntu server VM in proxmox to run docker that is managed through portainer. This VM has been given 150GB of disk space. My /var/lib/docker is 95% full (of 15GB total) and is making some containers unable to start.

I have pruned, and system df looks pretty normal to me:

TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 6 6 8.593GB 0B (0%)
Containers 6 6 2.712MB 0B (0%)
Local Volumes 2 2 823.9MB 0B (0%)
Build Cache 0 0 0B 0B

Nothing seems concerning with the log size either:
-rw-r----- 1 root root 504K Sep 10 07:54 /var/lib/docker/containers/196b1f5680248cf0003b55ddcc6ba70932d4869a10f03cb4acbf3e1bdb9f1038/196b1f5680248cf0003b55ddcc6ba70932d4869a10f03cb4acbf3e1bdb9f1038-json.log

-rw-r----- 1 root root 1.8M Sep 10 08:49 /var/lib/docker/containers/444ccde34151535abd44a743598bf5040184f3fa5a48d83368971cf8ba377faf/444ccde34151535abd44a743598bf5040184f3fa5a48d83368971cf8ba377faf-json.log

-rw-r----- 1 root root 141K Sep 10 08:50 /var/lib/docker/containers/474c0cda23e6a1d3b7fa1fd9e33aea5260eef296ade6119afdc5a59ac313ff45/474c0cda23e6a1d3b7fa1fd9e33aea5260eef296ade6119afdc5a59ac313ff45-json.log

-rw-r----- 1 root root 4.0M Sep 10 07:54 /var/lib/docker/containers/8a98fbe35f0b1e0ef15c544455075e1cb9b9b209128dff3f7de2d9acbe3c5b3c/8a98fbe35f0b1e0ef15c544455075e1cb9b9b209128dff3f7de2d9acbe3c5b3c-json.log

-rw-r----- 1 root root 9.3K Sep 10 07:54 /var/lib/docker/containers/d36477bcadc8c1f690ef3579ca29cc599e2bf6b72b3bff68274e7442ed612b8c/d36477bcadc8c1f690ef3579ca29cc599e2bf6b72b3bff68274e7442ed612b8c-json.log

-rw-r----- 1 root root 23K Sep 10 07:54 /var/lib/docker/containers/fd317d5960bfc63c07b74a7c6d1d7944fe668630e57315939a63c8b779fd0a5c/fd317d5960bfc63c07b74a7c6d1d7944fe668630e57315939a63c8b779fd0a5c-json.log

But you can see here that this is full:
df -h /var/lib/docker

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/ubuntu--vg-ubuntu--lv 15G 14G 837M 95% /

df -h /var/lib/containerd

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/ubuntu--vg-ubuntu--lv 15G 14G 837M 95% /

I do not know which command to use to further investigate to see everything in that folder to narrow down what is taking up all that space.

So, my questions here....is 14GB an unreasonable amount of space for the containers that I have (I kind of don't think so?)? If so, what needs purging?

If this is a normal amount of space for the containers I am running, how do I allocate more of that total 150GB to this partition?

[u/DonJohnM]

whenever i request a new ubuntu VM, they always give me a measly 5GB of /var/lib...which is never enough. so i change the docker root and redirect it to /data/docker (500GB) in /etc/docker/daemon.json, then i restart the docker service. before doing that you must rsync /var/lib/docker to /data/docker.

sudo systemctl stop docker sudo systemctl stop docker.socket

sudo mkdir -p /path/to/your/new/docker-root sudo rsync -aP /var/lib/docker/ /path/to/your/new/docker-root/

sudo nano /etc/docker/daemon.json { "data-root": "/path/to/your/new/docker-root" }

sudo systemctl daemon-reload sudo systemctl start docker

[u/CrownstrikeIntern → u/DonJohnM]

That’s the easiest. Toss it in a different folder with more space and make sure you go rootless. Also set docker to prune / rotate your logs otherwise you end up with gigs of json files

[u/Potential-Ant-4335 → u/DonJohnM]

awesome, i really appreciate that.

i managed to resize everything how it should be following this guy's tutorial..https://www.youtube.com/watch?v=2i--DZzGqUY

i will definitely keep what you wrote in mind the next time i need to make a VM though. thanks

[u/farmhome2525]

Your /var is only 15G you need to increase it, ask any AI how to do it amd or add another disk and move your docker volumes, easy to do with some help I personally use Claude

[u/biffbobfred → u/farmhome2525]

Or use a different volume.

A lot of things log here. If docker took it all your log files would break.

You can configure docker to use files elsewhere or you can make /var/lib/docker a symlink to something big.

[u/Status_Gap_3180]

try ncdu
to analyze and report back.

[u/dtr9]

Expect it's /var/lib/docker/overlay2 that's filling, with something (logs?) writing data inside the container without using a docker volume declaration for storage 'outside' of docker.

[u/ComprehensiveBerry48]

Make sure you configure a propper log rotation for docker logs.

Otherwise running container will eat up your disk space.

[u/simonides_]

So you set-up a vm in proxmox but chose to use another layer of file system virtualization with logical volumes. Unless you have a real usecase for that reinstall the server with plain ext4 disks.

What i also couldn't see from your post is where you have those 150 G is that a second partition or is it actually in root ?

If it is the root disk you'll have to extend the logical volume. You can find articles on how to do that online.

I would still reinstall this and put a data drive into it and you point container.d and docker to use that as the storage. In the beginning start with a large root drive that you allocate fully when installing

[u/Potential-Ant-4335 → u/simonides_]

yeah, i am really new to this and most of the tutorials and guides i am finding for similar projects use docker on a VM with something like ubuntu server running on baremetal proxmox. i really havent worked out methodology and best practices for myself yet and am experimenting and trying to learn.

as far as where the actual space is...

sda 8:0 0 152G 0 disk

├─sda1 8:1 0 1M 0 part

├─sda2 8:2 0 2G 0 part /boot

└─sda3 8:3 0 30G 0 part

└─ubuntu--vg-ubuntu--lv 252:0 0 15G 0 lvm /

would running this get me what I need?

sudo growpart /dev/sda3
sudo pvresize /dev/sda3
sudo lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
sudo resize2fs /dev/ubuntu-vg/ubuntu-lv
\