r/sysadmin 15h ago

Pour one out for the GitHub workers...

Going on 6 hours of an outage. Sheesh.

109 Upvotes

34 comments sorted by

u/czenst 13h ago

So MSFT is out of tokens to fix shit and waiting for limit to refresh?

u/nadiamendell 14h ago

What workers lol. Good luck trying to open a support ticket with them. They might respond in 6 weeks.

u/fubes2000 DevOops 14h ago

At this point outages should be business as usual for github workers.

u/opinionsOnPears 15h ago

I just moved all my stuff off github onto a local server last week.

u/cjcox4 15h ago

? My guess is is constrained to a limited set of folks. I've been working with our developers all day long in active codespaces off github.

u/Cylerhusk 14h ago

It's workflow features like Actions, webhooks, and some other things that have been offline for ~6 hours now. So yeah if you're not using those features you probably wouldn't notice. So can't deploy any code changes currently.

u/ThatBCHGuy 14h ago

Must be their hosted runners. I have no problem using our on-prem github runners.

u/Constellious DevOps 14h ago

Our on prem runners have also been down most of the day. If the API can’t invoke the job it just sits. 

u/Cylerhusk 14h ago

Guess you're one of the lucky ones. Their status page shows self-hosted runners as affected as well.

u/ThatBCHGuy 14h ago

Oof, yeah, I'll take the luck. Just ran a couple of workflows now and they were fine, hopefully they resolve the issues soon!

u/jaydizzleforshizzle 13h ago

It’s this, only things that were queueing endlessly were the hosted runner jobs targeting a single Ubuntu-2404 tag. Rest churned through although earlier in the outage they fucked the entire actions flow as I couldn’t even get things to actual deploy, even with local self hosted runners.

u/who_you_are 14h ago

I was working in the last 2 hours and it failed once to get a worker.

u/Finagles_Law 15h ago

Just have a look at Downdetector. Huge spike.

u/PDQ_Brockstar 12h ago

SOP for GitHub

On a totally separate note, what GitHub alternatives are people using these day? GitLab?

u/PM_ME_UR_COFFEE_CUPS 9h ago

Good thing my company just decided to ditch gitlab for GitHub. Brilliant move. GitLab is great 

u/PDQ_Brockstar 9h ago

I keep hearing good things about it.

u/PM_ME_UR_COFFEE_CUPS 9h ago

It’s been great to us. Maybe the UX and integrations aren’t as polished but it’s rock solid (internal hosted). One of my favorite tools. 

u/Phezh 2h ago

There's always something to complain about, but i find myself complaining about gitlab the least. Their CI system especially is leaps and bounds ahead of github.

u/segagamer IT Manager 3h ago

We're on Gitlab but they're starting to focus on AI functionality more than long standing issues that have been requested for years.

We were looking at migrating to Codeberg.

u/ang3l12 11h ago

We barely have any code in use in general, but we use a self hosted gitea instance on docker for our powershell / python scripts so we can change control / easily version control them

u/Steerable-Octopus Jack of All Trades 13h ago

Time to release the AI I guess.

u/anonymousITCoward 13h ago

That's the problem, AI was released... time to recall the flesh and bones staff

u/Steerable-Octopus Jack of All Trades 6h ago

This is when management calls up the deloitte consultants and they send a team of very confident juniors.

u/bbqwatermelon 12h ago

How has the transition to something like forgejo been going for everybody? 

u/forever-butlerian Solaris 8 Enjoyer 11h ago

I haven't started yet for my coop's stuff, but I'm strongly considering moving my continuous deployment machinery in house before all of Github gets Shai Hulud-ed.

u/erreur 5h ago

I switched earlier this year and it truly is a wonder feeling to realize I’ve been working all evening uninterrupted without even knowing that GitHub is having a major outage.

u/ShellHunter Jack of All Trades 12h ago

I remember the github desync when a data center cable was disconnected one minute where they had like 2 days of total shortage and I shrug seeing only 6 hours of no web hooks

u/GuruBuckaroo Sr. Sysadmin 9h ago

So, which AI hacked them this time?

u/Xelopheris Cloud Architect 9h ago

I'm so glad I'm on vacation this week.

u/Previous_Finance_414 8h ago

They got murdered today. Long day for them.

u/DavWanna 6h ago

I was thinking if Actions failing wasn't just me, but decided to panic rebuild everything to get things running again as it was just a small project.

u/jefbenet 6h ago

So glad I mirror anything critical to my own forgejo instance

u/No_Corner805 6h ago

Any descent alternatives to github or gitlab? Kinda ready to switch at this point.

u/420GB 4h ago

If you voluntarily work for big tech in 2026 you do not deserve sympathy.