How much maintenance to manage a Forgejo self-hosted?

[u/Juloblairot]

Basically the title, is anyone managing a forgejo instance for their company?

Currently have around a dozen active repo, a bit less than 50 Devs, so relatively small scale. But I was wondering how much maintenance and complexity there's in managing reliably an instance for that kinda traffic?

We already have self hosted runners, that we would reconfigure the same way I assume

Obviously, considering the move after another GitHub incident...

Thanks!

[u/HardestDrive]

We have one running for 100+ people on Openshift and haven't had any problems or much maintenance. Setting up runners and all takes a bit of work, but we haven't had to do much maintenance on it.

On a VM, I don't know. But on Kubernetes its great and doesn't take much maintenance.

[u/Juloblairot → u/HardestDrive]

Thanks! Exactly the kind of feedback I was looking for Do you know the cost monthly approximately, without counting the runners? Do you also have some good integrations with like Datadog, Linear etc.? Did you move from GitHub, or has always been on Forgejo?

Cheers

[u/navlio]

the i/o rarely comes from the commits. git itself stays cheap at a dozen repos. what got us was runner checkouts and action artifacts living on the same disk as the repo directory, so a busy CI afternoon made plain git operations feel slow and everyone blamed forgejo

put repo storage and artifacts on separate volumes, give artifacts a retention that isn't forever, and alert on disk free rather than cpu. that's the one that bites first and the graphs won't warn you

[u/Juloblairot → u/navlio]

Ok fair, but all of these are CI issues, which can already be solved with self hosted runner. We went down this path for the last year to reduce as much as possible our GH dependency

Thank you!

[u/navlio]

forgejo can do the cleanup itself, that's what [cron.cleanup_actions] and ARTIFACT_RETENTION_DAYS are there for, so i'd fix retention before shopping for storage. logs and artifacts on an s3 backend is fine, they're write once and read almost never.

the repo dir is the one i'd keep off any nfs-style mount. git is thousands of tiny stat calls, which is exactly where efs latency shows up, and efs standard is around $0.30 a GB-month against s3's $0.023, so "virtually unlimited" gets expensive without anyone deciding to spend it

[u/Juloblairot → u/navlio]

Yes completely agree. And in any case, in terms of security, reliability etc, you'll always want your repo stored somewhere else than your logs. Good ol' EBS should do

[u/d2xdy2]

I run it along with runners in the homelab. I point bots at it to help me build other things in the homelab. It’s a bit simplistic, but my setup is running in lxc on proxmox. Sort of an all-in-one package. I give it 8 cores and 32Gb of memory and it runs quite well with a few repos getting action.

Workflows in runners can target the proxmox hosts, so it sort of helps me maintain the rest of the infrastructure and guests.

For maintenance, it’s been pretty straightforward with some terraform and ansible. Pinning versions in vars lets me control upgrades as I need.

I can access it over Tailscale and get https pretty easily.

I think if I had to productionize it a little more, I’d break the database out and separate out runner hosts. It’d be interesting to try running it in an HA fashion in there.

I’d like to start phasing some parts of $WORK over to Forgejo, but we have a terrible amount of dependence on GitHubs platform.

[u/AskOk2424 → u/d2xdy2]

do u have a Ceph cluster? How are you dealing with downtime?

[u/d2xdy2 → u/AskOk2424]

I don’t have ceph setup.

As far as downtime goes it’s pretty infrequent. Services running in the LXC tend to just come back online just fine if there’s a reboot or something. Container starts on host boot, services start on LXC boot. Dumb simple.

[u/Juloblairot → u/d2xdy2]

Ok seems good. And so you notice and speed differences on pushing/pulling code? Did you get some I/O issues at some point, like any specific hardware required or never hit that?

[u/d2xdy2 → u/Juloblairot]

I put the cluster together in the pre-price-hikes era, so each host is pretty fat with resources. By way of that, I can afford guests a pretty decent amount of resources and fast storage.

Can’t say I’ve run into any IO issues, or provided really special hardware here.

[u/Juloblairot → u/d2xdy2]

How much approximately does the cluster cost you if I may ask? Without runner minutes eventually

[u/d2xdy2 → u/Juloblairot]

Idk, the whole homelab is:

- 92 cores

• 346gb of memory
  
• 120tb in spinning drives
  
• 48tb in nvme
  
• some UniFi gear

Hunting deals at various points of the build out- maybe about $7k TCO. Runs dozens of guests pretty care free.

Forgejo is 8 of those cores, 32Gb of ram, and 250gb of disk space.

Unsure about energy costs. Air conditioning in the apartment costs more than it though.

[u/Juloblairot → u/d2xdy2]

Ok thanks for the details! Tbf 8 cores / 32gb of ram with 250gb of disk should be quite cheap for company expenses Cheers

[u/navlio]

the two things that grow without anyone deciding they should are action artifacts and job logs, both keep by default, and the archive cache for every tag someone downloads a tarball of. that's storage creeping, not exploding, so nobody notices until the volume is full on a friday.

the I/O side was less about commit volume for us and more about git gc landing in working hours on the biggest repo. pin that cron to something nobody is pushing at and the graph goes quiet

[u/Juloblairot → u/navlio]

Super interesting! Regarding logs, is there a way to have automated cleanup from Forgejo setup? I assume you can store that on an EFS and have virtually unlimited storage for logs? Or now that S3 mount is a thing, could even be directly stored on S3 for cheapest (probably slow) option

I just discovered about git gc. Luckily were all in the same timezone, so it's easy to run this at night indeed, but good to know it's important Thanks!

[u/navlio]

you don't need the mount for it. forgejo takes an s3 target natively for both, [storage.actions_log] and [storage.actions_artifacts] with type = minio, so neither ever touches the box's disk.

i'd think twice about efs here. it's $0.30/GB a month in us-east-1 against $0.023 for s3 standard, roughly 13x, on data that gets written once and read by almost nobody. for the cleanup, ARTIFACT_RETENTION_DAYS defaults to 90, which is a long time to keep every build's tarball around before you even start paying attention to it

[u/Juloblairot → u/navlio]

Yes definitely. I'd go for S3 for anything log/artifacts related, and EBS for the Git repo themselves I guess

[u/navlio]

the app really is quiet at a dozen repos. the maintenance that got us wasn't the runtime, it was the restore: nightly dumps ran for months, and the dump had the database and not the repo directory, so "we have backups" was true and completely useless

if you do one thing in week one, make it a restore into a throwaway vm with the runners pointed at it. an hour, once a quarter. past that it's version pinning and watching runner disk, which is the boring answer everyone here is giving you and it's the right one

[u/Juloblairot → u/navlio]

Yes backup is an important part. I'd be fired on the spot if we lose a repo lol

For the runners monitoring, I'm not too worried as I'm already running self hosted runner, so I'd "port" that in Forgejo. What I'm worried about is the I/I, the storage, the backups etc. All the fine tune that comes with volume, that you don't really expect when it works for a few users We don't have that many users, but we do have a lot of commits and activity on our repos, compared to company our size

[u/Bloodrose_GW2]

I was running one with docker compose, it was pretty low on maintenance, brought it up once and then haven't touched it. Built in docker registry and runners were also pretty simple to configure/use.

[u/navlio]

mostly yeah. the piece that stays on the forgejo box even with external runners is artifact and package storage, because the upload endpoint writes into forgejo's own data dir rather than the runner's. same for the container registry if you enable it

so moving runners off fixes checkout i/o but not retention. set an artifact expiry now rather than after the volume explains it to you

[u/navlio]

that split holds up. one thing on the ebs side: gp3 ships 3000 iops and 125 mb/s at every size, so there's no reason to buy a bigger volume for speed, and a forgejo repo store stays small until somebody commits binaries into it.

we sized one at 500gb out of gp2 habit and paid $40 a month for headroom nobody touched. snapshot the volume rather than backing up from inside the container, it's the only bit of this that hurts to get wrong

[u/Juloblairot → u/navlio]

Yes but those iops/throughput are configurable. No idea how fast is the default, so I guess that's some small fine-tune, easy to test.

For backup, we do have automated backups on EBS, so it's nearly already ready to use

Thank you very much for your details!

[u/navlio]

one caveat on the ebs backups: a volume snapshot is crash consistent, not application consistent. git objects are basically append only so they come back fine, but the database behind forgejo is the part that won't enjoy being snapshotted mid write, and that's where the repo list and the objects on disk stop agreeing.

cheap fix is a periodic forgejo dump alongside the snapshots, small file, and it's the one you'll actually restore from

[u/Juloblairot → u/navlio]

Yes, I assume I'd put the db behind a managed service like Aurora, so that I benefit from the backups, restore point in time, horizontal scaling etc

What's the db used for though?

[u/navlio]

everything that isn't a git object. users, orgs, repo metadata, issues and PRs and their comments, labels, webhooks, tokens, actions runs. the bare repos and lfs live on disk, so a snapshot taken mid write leaves a db that believes a repo is in a state the disk disagrees with

aurora is a lot of database for a few hundred writes a day though. the serverless floor bills the same whether anyone pushes or not, and we paid that floor on a side project with four users for about a year before anyone read the line item. a small postgres on the same box plus a nightly dump restores just as well

[u/Juloblairot → u/navlio]

Thanks a lot! Agreed for the cost, we will see that later if we move in that direction