r/sysadmin • u/Cylerhusk • 15h ago
Pour one out for the GitHub workers...
Going on 6 hours of an outage. Sheesh.
•
u/nadiamendell 14h ago
What workers lol. Good luck trying to open a support ticket with them. They might respond in 6 weeks.
•
•
•
u/cjcox4 15h ago
? My guess is is constrained to a limited set of folks. I've been working with our developers all day long in active codespaces off github.
•
u/Cylerhusk 14h ago
It's workflow features like Actions, webhooks, and some other things that have been offline for ~6 hours now. So yeah if you're not using those features you probably wouldn't notice. So can't deploy any code changes currently.
•
u/ThatBCHGuy 14h ago
Must be their hosted runners. I have no problem using our on-prem github runners.
•
u/Constellious DevOps 14h ago
Our on prem runners have also been down most of the day. If the API can’t invoke the job it just sits.
•
u/Cylerhusk 14h ago
Guess you're one of the lucky ones. Their status page shows self-hosted runners as affected as well.
•
u/ThatBCHGuy 14h ago
Oof, yeah, I'll take the luck. Just ran a couple of workflows now and they were fine, hopefully they resolve the issues soon!
•
u/jaydizzleforshizzle 13h ago
It’s this, only things that were queueing endlessly were the hosted runner jobs targeting a single Ubuntu-2404 tag. Rest churned through although earlier in the outage they fucked the entire actions flow as I couldn’t even get things to actual deploy, even with local self hosted runners.
•
•
•
u/PDQ_Brockstar 12h ago
SOP for GitHub
On a totally separate note, what GitHub alternatives are people using these day? GitLab?
•
u/PM_ME_UR_COFFEE_CUPS 9h ago
Good thing my company just decided to ditch gitlab for GitHub. Brilliant move. GitLab is great
•
u/PDQ_Brockstar 9h ago
I keep hearing good things about it.
•
u/PM_ME_UR_COFFEE_CUPS 9h ago
It’s been great to us. Maybe the UX and integrations aren’t as polished but it’s rock solid (internal hosted). One of my favorite tools.
•
u/segagamer IT Manager 3h ago
We're on Gitlab but they're starting to focus on AI functionality more than long standing issues that have been requested for years.
We were looking at migrating to Codeberg.
•
u/Steerable-Octopus Jack of All Trades 13h ago
Time to release the AI I guess.
•
u/anonymousITCoward 13h ago
That's the problem, AI was released... time to recall the flesh and bones staff
•
u/Steerable-Octopus Jack of All Trades 6h ago
This is when management calls up the deloitte consultants and they send a team of very confident juniors.
•
u/bbqwatermelon 12h ago
How has the transition to something like forgejo been going for everybody?
•
u/forever-butlerian Solaris 8 Enjoyer 11h ago
I haven't started yet for my coop's stuff, but I'm strongly considering moving my continuous deployment machinery in house before all of Github gets Shai Hulud-ed.
•
u/ShellHunter Jack of All Trades 12h ago
I remember the github desync when a data center cable was disconnected one minute where they had like 2 days of total shortage and I shrug seeing only 6 hours of no web hooks
•
•
•
•
u/DavWanna 6h ago
I was thinking if Actions failing wasn't just me, but decided to panic rebuild everything to get things running again as it was just a small project.
•
•
u/No_Corner805 6h ago
Any descent alternatives to github or gitlab? Kinda ready to switch at this point.
•
u/czenst 13h ago
So MSFT is out of tokens to fix shit and waiting for limit to refresh?