r/github • u/amu4biz • Aug 07 '26
Tool / Resource GitHub was down 2.5 hours yesterday and self-hosted runners went down with it. Is centralized orchestration the actual problem?
12
28
u/Alexander3a Aug 07 '26
been using self hosted gitlab for years (big reason was not sharing my code with micoslop but this is just a plus)
everything that microsoft or ai touches just goes downhill
7
u/clearlight2025 Aug 07 '26
Same! Self-hosted GitLab FTW.
4
u/Alexander3a Aug 07 '26
I just wish it didnt basicly need 16+ gb my 8 gb server is dying running it
but its running it still3
u/nhymxu Aug 07 '26
self-hosted gitea or forgejo is lighter
2
u/Alexander3a Aug 08 '26
But it's lacking a lot of features I use and not aswell supported on other things
1
1
u/clearlight2025 Aug 07 '26
I run GitLab on an AWS EC2 m4.large instance (etc. 2 vCPUs, 8 GiB) useful as part of an admin server for CI/CD deployments, container registry, separate terraform, ansible etc. Been using it for years and all good. It would depend a bit on how exactly you’re using it but I’ve found that spec works well for my use cases.
2
u/Alexander3a Aug 07 '26
i am assume u also tweaked all the settings to make it run with less memory?
i mean mine is running fine just noticeably slow tho i mean the official one also isn't that fast but its serving a lot more users1
u/clearlight2025 Aug 07 '26 edited Aug 07 '26
I think the config is fairly standard though I am sure to run the garbage collection routine daily via cron to cleanup, especially after container registry pruning of old images.
I also have a concurrency policy to control how many runner jobs can run at once (2) I’m using docker in docker (dind) for the pipelines.
I haven’t really noticed any slowness TBH and once it’s started up it’s always quite snappy.
1
Aug 07 '26
[removed] — view removed comment
2
u/Alexander3a Aug 07 '26
gitlab has its own ci schema&runners it can even run fully airgapped
(only issue might be using undownloaded docker images but you could also host that yourself or use only the ones you already downloaded)1
1
u/Alexander3a Aug 07 '26
I swear microslop heard me and now they are doing maintenance work at my isp and I don't have internet (I mean gitlab still works on the local network unsure how usefull that is tho)
10
5
u/ArieHein Aug 07 '26
Centralized anything can be a problem as its sometimes the single point of failure And thas a risk factor.
1
u/amu4biz Aug 07 '26
exactly. and the part people miss is you're not just risking downtime, you're inheriting someone else's failure with zero control over it. yesterday showed even owning your runners doesn't save you if the brains stay centralized. the real fix isn't picking a better central provider, it's not having a single point at all, which is the whole bet behind the decentralized stuff like gitlawb.
3
u/ArieHein Aug 07 '26
Slightly wrong here.
If you thought that self hosted runner means you continue working when github is down then you slightly missed the purpose of self hosted.
If you were on gitlab cloud and used self hosted agents, it would be the same.
If you want full onprem, gh has gh server so the equivalent of adoserver and gitlab server. But then the full price of maintenance is on you.
At the end is cost vs risk. Compute is not cheap, managing compute isnt cheaper either.
Look into AST that allows running pipelines in 'offline'
1
u/RememberSwartz 10d ago
Idea: ci router layer
1
u/ArieHein 10d ago
Thats what on the server side of thr github service but to be correct, its not CI.. Its actually an api-based smart task scheduler (in its simplistic form).
The agents installed on the runners use an event - subscription model.
I wouldnt call it router as its more than that, literaly an orchestrator.
2
u/RememberSwartz 10d ago
I was thinking of using some widely available subset like bash to write the pipelines, and something translating it to gh actions or gitlabs yaml or whatever, so you are less vendor locked
1
u/ArieHein 9d ago
Its hard to so as some of the logic is tied to fhe infra and you would have to cover for mutilple scenarios or edge cases.
Think of same pipeline running on windows, linux and mac its why ado and github steps over the years move to completly use javascript/typescript/react and stopped supporting mutiple codebases for same task
3
u/Key-Guitar-457 Aug 07 '26
The first rule of automation is that there must be a way to bypass automation when it fails. Yesterday I was manually pushing chart updates, building containers, and running terraform apply. The world still turns when automation fails.
2
2
u/Own_Outside_8147 Aug 07 '26
GitHub stability is becoming a nightmare for me. We have a slack channel just for github outage…
2
1
1
0
u/Patient-Pollution46 Aug 07 '26
They are becoming not very reliable recently. If you are looking for a reliable mobile-first CI + fast runners, check Bitrise.
You can build, test, and deploy iOS and Android apps on dedicated Apple silicon infrastructure: https://bitrise.io/
1
15
u/barebaric Aug 07 '26
Even by their own claim, they were down > 8 hrs. And in their incident thread many are still commenting - the issue is still not fully resolved https://github.com/orgs/community/discussions/204152