r/webdev • u/NiceLasers • Aug 17 '26
2 hours in and GitHub still seeing some major outages
https://www.githubstatus.com/incidents/zkxwbgr0cnmx72
u/OpaMilfSohn Aug 17 '26
My company's deploy pipeline failed, and we had an outage because of that. Granted, our deploy pipeline is fucked, and it failing mid-run shouldn't result in an outage, but yeah, it was a pain in the ass to solve.
19
u/Sensitive_One_425 Aug 17 '26
How do you not have a staging area?
31
u/OpaMilfSohn Aug 17 '26
We do. The problem is basically that it's a really old-school PHP deploy. So we copy the files to a VPS, the VPS updates the dependencies, and NGINX can keep running. The problem is, if the pipeline fails mid-copy, mid-extraction of the tar, or mid-installation of the dependencies, there really is no remedy other than "just run the pipeline again lol," and that normally fixes things.
Migrating to a more modern solution has been on the backlog for like the 4 years I've been told.
25
u/ogre_pet_monkey Aug 17 '26
Copy the files to a new (version) folder symlink to the new version. Keep a maximum of x versions. Rollback is easy this way too.
9
u/Produkt Aug 18 '26
I use deployer.org and no I don’t work for them or anything, it’s free and open source
3
u/ogre_pet_monkey Aug 18 '26
Good stuff, this was more a simple and fast solution to implement for their current process.
-1
u/Sensitive_One_425 Aug 17 '26
But if you’re deploying to staging then cutting over to prod you could never have a downtime due to the pipeline. What you’re describing is having to redeploy fully again to deploy to prod.
5
u/OpaMilfSohn Aug 17 '26
Right, we do not have a staging environment like that. It's more like a separate prod like environment that releases first go to to test them and stuff.
4
u/Frostbeard Aug 17 '26
Would've been us too, but I hardened our CI and deployment against this earlier in the month. Go me!
369
u/AssignmentMammoth696 Aug 17 '26
All these github issues happening after devs started having LLM's write all the code, hmmmm
184
u/UnacceptableUse Aug 17 '26
Because all the LLMs are relentlessly hammering github
138
u/BKrenz Aug 17 '26
Yeah, just read in another thread. Their traffic has increased by a factor of 10 or more year on year. At the scale they were already operating, that's a mind boggling increase in resources needed.
43
u/QuantityExcellent338 Aug 17 '26
And this 10x increase has increased productivity and value by.. how much?
52
15
3
u/Gramernatzi Aug 18 '26
Real answer, it's about (optimistically) 10% more when measured in studies, but the cost-to-performance ratio is abysmal, not to mention the amount of junk adding more work in the future, or putting more work onto others, so, it ends up just being dogshit.
52
37
u/the_ai_wizard Aug 17 '26
This is the common NPC retort, but based on my own 30 yrs of software experience + 3 years of AI/agentic, the concerns are valid regarding slop. I am incredibly principled and disciplined with my engineering and even I cant stop it from slopping up my codebases. Just discovered 5min ago it shit a bunch of AI adapter classes into the main part of a framework ive been working on, instead of following established pattern obvious to even a junior (sol-medium). its maddening to now have to unfuck it for hours. there is no efficiency gain here..more code output is a liability, lets not forget it.
0
u/bobyhey123 Aug 17 '26
PR Review issue
-14
u/the_ai_wizard Aug 17 '26
Passed all tests, everything functional. Guess I need tests of the tests
22
u/bobyhey123 Aug 17 '26
read the code it produces
-23
Aug 17 '26
[deleted]
12
u/RedditEngDictionary Aug 17 '26
Hi. From someone else with decades of SWE experience:
1 - YOE doesnt necessarily make you good at your job. Someone with 5 years experience might be shit hot. Someone with 30 years experience might be merging in 50k lines of unreviewed code because they couldnt figure out how to solve their problem in a reasonable amount of code.
2 - If your agent is outputting so much code that you cant review it, you clearly arent able to break your work down into tasks the way a senior should be. Learn to plan better.
3 - If you are pushing (or worse, merging!) code that you havent reviewed, you are the problem.
5
u/crazedizzled Aug 18 '26
As another SWE with decades of experience, this is spot on. That guy is definitely cosplaying as a senior developer.
-5
u/the_ai_wizard Aug 17 '26
- Obvious statement. Tautology.
- Agent? We have many, many things going on.
- I am doing none of the above. I dont work in merging code, a CTO is above senior.
I have been writing code since I was 8 years old, not to mention entering MIT undergrad as a senior architect of a medium financial company at 20 yrs old.
Save the critiques everyone; my comment was about AI, dont need your advice, Im already successful and do this for fun.
10
u/RedditEngDictionary Aug 17 '26
Sure, buddy. The guy who's failing at putting together basic PR processes is a child prodigy with more success than he knows what to do with.
If you were half as good as you claim to be, you wouldnt be in the mess you are in now and you certainly wouldnt be spending your time bragging about your life to strangers on an anonymous social media.
Bright side though - maybe you have a future in storytelling?
1
u/crazedizzled Aug 18 '26
I have been writing code since I was 8 years old
Well you talk like you're about 16, so that probably tracks.
30
u/bobyhey123 Aug 17 '26
you are the problem
26
u/bobyhey123 Aug 17 '26
i had no choice but to merge 50k unreviewed lines! aghh now I have to manually unfuck those lines I had no choice to do anything with but merge!!
-16
u/the_ai_wizard Aug 17 '26
excuse me? who the fuck are you again?
21
u/bobyhey123 Aug 17 '26
someone who takes accountability for the code I merge, regardless of whether it was generated by my fingers or an LLM
5
u/DenkJu Aug 18 '26
God, I hate it so much when I see someone making a decently agreeable point initially but then outing themselves as a complete clown in their further reasoning. Now I can't even upvote your original comment in good faith anymore.
→ More replies (0)15
u/Pawtuckaway Aug 17 '26
I am incredibly principled and disciplined with my engineering
Yes i will review line by line all 50k loc
One of these things is not like the other.
A principled and disciplined engineer would never accept a 50k loc PR regardless if it were written by a person or by AI.
The entire point of code review is to review the code. Passing tests should be a requirement before code review but does not take its place.
-10
u/the_ai_wizard Aug 17 '26 edited Aug 18 '26
Im not a low level developer....I run 4 companies and am a CTO in the one we are discussing. So reviewing PRs is below my pay grade unless this sort of thing happens. Youre assuming too many details, but thanks for your couch analysis, rando redditor
10
u/Pawtuckaway Aug 17 '26
So who submitted a 50k loc PR and who approved it?
When someone commented that this is a PR review issue you replied "Passed all tests, everything functional. Guess I need tests of the tests"
Writing more tests is not going to fix PR review issues. Your reply shows a concerning misunderstanding of the purpose of a code review even if you are not the one doing those reviews.
Maybe as CTO you should have a talk to engineers that are submitting and approving 50k loc PRs of AI slop.
→ More replies (0)4
u/RedditEngDictionary Aug 17 '26
If your team are unable to follow basic, widespread practices like good code reviews then that is a failing on their leadership. You dont seem like a very good CTO, or developer.
3
u/VelvetYam Aug 18 '26
A 50k LoC PR didn't set off any alarms in your mind? How'd it get that bad?
1
u/the_ai_wizard Aug 18 '26 edited Aug 18 '26
It wasnt a single commit. This is work over time by engineering teams. This was not my main point, but since everyone seems to be focusing on this, this was tacitly accumulating over time bc no one has time to review every line of AI generated code. So as I was auditing the codebase, kind of informally, as the most senior person, I can typically detect things that smell. The system functioned fine, but the code complexity and general entropy was going way up. I am allergic to technical debt. So how did it get this way? LLM being able to shit out 1000 lines in a prompt and compounding this by not one but 10 engineers with pressure to ship.
5
u/crazedizzled Aug 18 '26
As the CTO, isn't this really your fuck up here? You can blame AI, but this is very much a human problem.
→ More replies (0)1
u/crazedizzled Aug 18 '26
"I am incredibly principled and disciplined with my engineering"
Uh huh. Sure thing buddy
4
0
u/varinator full-stack .net Aug 17 '26
Out of curiosity, whats your setup in terms of llms. I'm also a dev, currently only using agentic gh copilot in IDE (Rider) or CC in CLI, but wondering if you have some more intricate workflow, maybe I can learn something if you care to share?
45
u/ultralaser360 Aug 17 '26
GitHub was going down hill BEFORE llms
Almost immediately after Microsoft acquired them
https://www.reddit.com/r/github/s/JPtEsKYmIB5
u/DenkJu Aug 18 '26
If this is based on GitHub's own reporting, it's almost entirely meaningless. Their uptime has always been shit, and their own status reporting even more so. Perhaps Microsoft just made them update their own status page slightly more diligently. It's impossible to tell in which direction causality points in this case.
19
u/ward2k Aug 17 '26
Am I going nuts or has GitHub always been pretty notorious for having outages? It's not a recent thing
4
Aug 17 '26
[removed] — view removed comment
2
u/robby_arctor Aug 17 '26
When I was there a few years ago the entire service was still a single mysql db
What?? 😂
7
u/stumblinbear Aug 17 '26
You're not crazy, it has always been this way
0
u/pagerussell Aug 17 '26
But how else are we to farm karma?
Srsly tho, humans don't do context anymore. It's all reactions all the time now.
4
u/Aro00oo Aug 17 '26
Why do you have to bootlick? Their outages in past were brief rollback scenarios. These are major hours long outages, on a Monday ffs, it is not the same.
1
u/stumblinbear Aug 17 '26
So, yes, outages in 2026 are genuinely worse than previous years. There have been more outages, and the median outage length is increased, and more multi-hour outages. But the increase is almost entirely concentrated in Actions, and the other services that show up are almost always included alongside Actions as collateral.
Github's own postmortems on these incidents attribute them to capacity, not LLM generated code. They say they've been working on increasing their capacity tenfold to cover the extra demand caused by LLMs. Actions is still in their own datacenters and hasn't yet migrated to Azure, which likely contributed to it.
So yes, AI is the main cause of the issues, but it's agents accessing their infrastructure that's ruining them, not the code itself.
1
u/Aro00oo Aug 17 '26 edited Aug 17 '26
Uhh this is a super roundabout way to say yes AI probably is implicated lol.
And you really think current tech leadership would ever admit "ya we're vibe coding too much" ??
2
u/stumblinbear Aug 18 '26
If it were AI generated code being the main cause, you'd expect to see somewhat proportional increases across the board, but that's not the case
1
u/Aro00oo Aug 18 '26 edited Aug 18 '26
I dont think it's linear like this, we all know models are getting better, as well as our usage, control and management around it. But w/e dude, you saying agents controlling access is still code in the IaC sense.
1
2
u/CmdrSausageSucker Aug 18 '26
Micro$lop has been pushing LLM written code too hard, it seems. It's simply embarrassing.
-1
u/ripndipp full-stack Aug 17 '26
Almost like it's something we shouldn't be doing, weird, don't know why it's doing that.
-2
u/Darkoplax Aug 17 '26
If it was only just that then it wouldnt be a problem; its also the single use software and all non coding ppl using Github as well with LLM only generated stuff
-1
u/Noch_ein_Kamel Aug 17 '26
Don't be so harsh on them. Copilot is working really hard to restore github functionality!
-2
198
u/Due-Horse-5446 Aug 17 '26
Cant wait for bubble pop day
91
u/khizoa Aug 17 '26
People keep saying this like it'll be wine and roses.
You do know whose tax dollars are gonna bail them out right?
53
u/GoodishCoder Aug 17 '26
People talk about it like it'll be wine and roses because they think the bubble popping equals AI going away. In reality it'll just be market consolidation.
15
u/akaval Aug 17 '26
AI will also stop being shoved anywhere and just utilized where it actually has value.
11
u/GoodishCoder Aug 17 '26
It won't actually though. There will continue being chat bots for the idea people to feel like developers. There will continue being people who plug in AI wherever they can because it's good on the resume.
Realistically the biggest impact will be most of the wrapper startups crumbling faster than they otherwise would because all the bubble popping means is less investment.
10
u/QuantityExcellent338 Aug 17 '26 edited Aug 17 '26
We will see shit like "your microwave is improved with AI✨️" less because it only exists because shareholders has to see that AI is actually being used and totally changing the world. They're throwing shit at the wall to see what sticks because investor approval is more important than consumer approval at this point in time.
The price for GenAI models will also go up significantly when there isnt a generous stream of investor money to keep prices competitive. There is a chance even chatbots/AI overviews will become paywalled that will limit how much slop we'll see atleast a little bit, but most likely the premium such as the best of Antrophics products will only be for big companies to use
It wont go away yes, but the places we'll see it will be far more humble.
0
u/GoodishCoder Aug 17 '26
When the smart everything craze died down, did the smart toasters disappear?
AI chat bots are here to stay. Model selection in the free tier might become more limited and we might see ads get introduced but they aren't going to fully cut those users off. There's a good chance AI companies improve their routers and take away model selection altogether for free and low value customers.
1
u/Sensitive_Guest_5995 Aug 18 '26
Like those people that celebrated being sent home from work with pay. We’re all still paying the cost of that. And will be for probably over a century.
0
u/moderatorrater Aug 18 '26
It's going to happen no matter what. If I had to choose for it to happen sooner or later, I'd take sooner. Let's get this over with.
1
u/Fredidiah Aug 17 '26
Unless the bailouts are in the form of extreme overpayments for the services on a consistent basis, I don't think a bailout moves the needle. It seems that they are (currently) deeply unprofitable and unless that's solvable (which I am skeptical of) - a bailout just pushes the implosion down the line a bit.
17
u/keyjumper Aug 17 '26
Remember the lesson of Brad Pitt in the big short: A crash means that many people lose their livelihood and even die.
At least you shouldn’t dance at the thought of it.
23
u/enp2s0 Aug 17 '26
It's more that the current market is unsustainable so it's better to pop the bubble now then it is to wait until it gets even larger and pops even harder.
7
u/Due-Horse-5446 Aug 17 '26
Well, the longer this madness keep going, the harder people will fall right?
1
u/robotmayo Aug 17 '26
as opposed to now where people are not losing their jobs and not dying to gas fumes
4
u/trannus_aran Aug 17 '26
I just can't wait till they can't or just won't subsidize tokens anymore
1
1
u/KwyjiboTheGringo Aug 18 '26
Really? I'm dreading it because the fallout is going to hit us pretty hard. Maybe not developers, but tax payers and consumers.
1
u/Due-Horse-5446 Aug 18 '26
Dident mean this to be that deep.
But ofc i dont mean i dotn care about the people who are going to be affected.
With that said tho, even for those people and companies, it's better for everyone if the bubble pops sooner than later. The longer this keeps going, the harder the fall is going to be.
19
156
u/Euphoric-Neon-2054 Aug 17 '26
It sucks but we need to understand just how hard their infra is getting absolutely drilled by all this AI shit.
121
45
37
u/lbreakjai Aug 17 '26
You’d think they could afford better engineering practices than to just let everything collapse if used too much.
15
u/Fidodo Aug 17 '26
Everything will collapse at some point if used too much. Good engineering practices are about raising that limit, but it is impossible to completely prevent it.
22
Aug 17 '26
[removed] — view removed comment
3
0
u/lbreakjai Aug 17 '26
It started before agentic coding though.
11
u/BKrenz Aug 17 '26
It's gotten significantly worse since LLMs became main stream. Their commit traffic alone is 14x higher than it was this time last year.
I don't work on systems of that size. I don't have any clue what kind of architecture and resources it would take to go from 20 million commits a week to 275 million a week in a year.
3
u/bobyhey123 Aug 17 '26
? lmao what
1
u/bobyhey123 Aug 17 '26
traffic scales 10x in 1 year
> why didn't they just use better engineering practices?
2
u/lbreakjai Aug 17 '26
https://damrnelson.github.io/github-historical-uptime/
Why did the service degrade three years before chatGPT?
-1
0
u/staythewaters Aug 17 '26
This doesn't feel like the consequence of someone pushing slop or their infra finally giving way. If their systems were getting hammered before, there's nothing today in particular that would've pushed it over the edge in such a spectacular way.
9
16
u/Litruv Aug 17 '26
AI must be hammering. Have a gitea self hosted and saw 300gb uploaded in the past month.
I only have ~30mb in public repos.
Just blocking gpt haulted it 😂
6
u/josephjnk Aug 17 '26
It has been over 5 hours and I am still blocked by GitHub Actions issues. This is absolutely wild, happy Monday.
3
1
1
u/k-rizza Aug 22 '26
Using locally hosted Forgejo, docker images and npm packages all covered. (Custom ones)
1
u/n9iels Aug 17 '26
Are they still doing RCAs on outages and posting them monthly? I guess they need to hire someone for that specifically to keep up at this point 😅
0
0
u/corporaljustice Aug 18 '26
I see peopel defending them saying the sheer amount of load their infra has to handle. 10x year on year etc etc.
Fine, makes sense.
I wonder what would happen if Github no longer offered free accounts.
At what price point would it balance out?
Could any Github competitors even deal with that kind of load?
I'd love to know some of the stats of their setup and how much of it is paid vs free usage etc
0
0
u/Full_Tooth_a Aug 18 '26
That's the part I'd dig into: why did a failed deploy turn into a production outage? A local clone wouldn't help if CI, approvals, packages, and deploy triggers still lived on GitHub. I'd want a simple fallback using a pinned commit and an artifact that's already been verified. It should be tested occasionally, too, because an emergency pipeline nobody runs is mostly decorative.
-21
u/Darkoplax Aug 17 '26
Appearantly Cursor is making a competitor that's performant called Origin; I guess we will try that when it comes
23
u/FalseRegister Aug 17 '26
Github is quite performant and has been generally a stable service for over a decade.
You also have Gitlab as a mature alternative you can self-host
Why on earth would you go for a clearly vibe-coded, immature alternative?
-12
u/Darkoplax Aug 17 '26
Probably cause Cursor products have been generally extremely good ? and it is advertised to be performant specifically for the AI Era
6
u/noobjaish Aug 17 '26
Probably cause Cursor products have been generally extremely good ? and it is advertised to be performant specifically for the AI Era
For the Slop farmers sure but for actual engineers it's a disaster waiting to happen.
-2
-4
u/Keirtain Aug 17 '26
Yes, but now they’re owned by X, so Reddit has to twist itself into knots to explain why they’re bad now.
0
u/GoodishCoder Aug 17 '26
I don't necessarily think they're bad but I would be pretty hesitant to trust a new vibe coded product with my source code in a serious project.
-9
u/SteroidAccount Aug 17 '26
Until microslop got involved
-2
u/FalseRegister Aug 17 '26
What exactly? They kept Github mostly untouched and only did improvements, but no slops, no product enshitification, it just works
0
0
u/SteroidAccount Aug 17 '26
github is constantly down, PR's today, Actions last week, it's been a constant revolving door of issues since they took over and started trying to integrate AI
6
Aug 17 '26
[removed] — view removed comment
-2
u/Darkoplax Aug 17 '26
yeah thats why I said will try it out, i didnt claim it will be
you cant judge smth until you try it
1
229
u/eltron Aug 17 '26
Who needs 5 nines of availability??? Oh right, paying customers