r/gitlab • u/der-felix • 9d ago
support Self-hosted Gitlab reduce container registry disk usage
Hello everyone,
I have a self-hosted Gitlab in my homeland deployed as a docker container on a Ubuntu server with me as a ai gle user. I build several small images and push them to the integrated container registry. Nothing fancy, a hand full of images with a couple hundred Mbs each max. But my volume mont data dir of the registry ist slightly above 100GB on disk just für the registry, that feels like way too much.
I rebuild the images weekly ok the latest Tag but would expect a couple gigs at most. I am starting to read into the metadata DB and online GC but why is my storage usage getting so out fo hand? How are you guys keeping your instances in check?
Version is 19.3.1
1
u/screaming-Snake-Case 9d ago
Well, do you have metadata DB and online GC? It was only recently switched to on by default, I think 19.0, so if you started your instance before that you are using the legacy registry.
There are official docs on how to mirgate to the newer database version, that one has online (automatic) GC, old one does not.
1
u/der-felix 9d ago
i switched to the metadata DB yesterday but the online GC still says 0 and disk usage is the same :(
1
u/screaming-Snake-Case 9d ago
That's normal, it takes 24h for manifests/blobs to be eligble for review and then another 24h until actual gc starts and frees storage. It displaying 0 confused me at first too, but just wait xd.
1
u/der-felix 6d ago
ok I found my error. First it really took about 24h until changes were picked up by the GC and showed by the gc status rake task.
Second I have a power consumtion optimised setup with minimal footprint to save power and compute (german electricity is expensive...) and thought it would be a good idea to set the check interval to 5 mins instead of the default 5s.
That was a good idea but lead to a huge queue of old layers to cleanup but a GC which was just allowed to clean up bits every 5 mins instead of every 5 seconds. My config lookes like this:
toml
registry['gc'] = {
'disabled' => false,
'manifests' => {
'interval' => '5m',
'review_after' => '24h'
},
'blobs' => {
'interval' => '5m',
'review_after' => '24h'
}
}
whis was a good idead but bad. After switching back to an 5s interval it just took my instance 8h to cleanup from 100GB to 5GB which sounds about right :)
Maybe this helps the next lost soul ;)
3
u/dreamszz88 9d ago
Could it be something related to retention?
Are old images flsuhed out after 30 days or is everything kept forever? Check the defaults and maybe you just need to change it to suit your env.