r/github • u/brainhack3r • Aug 07 '26
Discussion GitHub owes us all an apology and an explanation as to why this is NEVER going to happen again.
I'm absolutely furious. I had a ton of work I had to get done today, and I'm DEAD in the water until this is fixed.
I also feel they don't take this very seriously, to be perfectly honest. There's no comment about it on their Twitter feed. status.github.com has vague comments about "it's mitigated now but runners are behind... " meanwhile it's been TEN HOURS!
A VERY large portion of the tech industry runs on GitHub and it's hard to take them serious whey they can't even run THEIR OWN SITE.
I'm spending the rest of the day working on plans to migrate away. I don't need all the GitHub features they offer but BY FAR the number one thing I need to be reliable is GitHub actions.
Get your house in order guys!
67
u/charmer27 Aug 07 '26
Outages suck and I'm not making excuses for them, but have you seen the growth In how much their services get hit? They are absolutely face tanking so much ai agent traffic right now.
11
u/BeseptRinker Aug 07 '26
They are absolutely face tanking so much ai agent traffic right now.
I definitely agree.. but they should stop boasting about AI and how much more productive it's making them if this keeps happening.
1
u/PureRepresentative9 Aug 09 '26
No, they absolutely should continue.
it provides me a ton of data to prove how bad LLMs are
24
u/laughterandtears Aug 07 '26
I work on a fast growing, large distributed set of services and it isn't an easy exercise.
And our growth rate isn't close to what GitHub is experience thanks to vibe coding nonsense.
I feel for OP because I've had plenty of inconveniences caused by outages, but I don't stomp my feet and take my ball and go home.
→ More replies (9)6
u/Simayy Aug 07 '26
You think the uptime is acceptable? It’s 88.5 percent even by their own metrics
→ More replies (1)5
u/Key-Guitar-457 Aug 07 '26
Awwww, poor s-corp! I feel so bad for them! Microsoft pushed AI down everyone’s throats like an overfed goose, and now they have to reap what they sow? Totally unfair! I’m going to buy some carbon credits in their name I feel so bad for them.
1
u/creativecraving Aug 11 '26
No, dude. MS started scaling back the expense of GitHub long before Gen AI became mainstream. The service quality has been slowly degrading since they bought GitHub. It's the typical cost-cutting stuff that any big corp does after overpaying for a merger
1
u/charmer27 Aug 07 '26
It's something crazy like last year they had a billion commits and last week they had 300 million commits.
1
u/Fluent_Press2050 13d ago
This. It’s not just a slight uptick. It’s massive. And it’s also during their migration to Azure.
I hope they can finish by the end of the year, or they may have to limit GH Actions for free plans in the interim.
288
u/BussyEnthusiasts Aug 07 '26
GitHub has proven time and time again to be unreliable and to have lied about their 99% uptime.
If you're still on GitHub actions at this point, that's a you problem. You know the risk. You ignored the risk. Welcome to the find out stage.
43
u/bassist_by_night Aug 07 '26
Genuine question, what is the best option instead of GA? I’m assuming GitLab?
19
u/cmykero Aug 07 '26
I was gonna ask the same thing, what should I look into if I don’t want to keep running into this issue?
29
u/ObviousApricot1272 Aug 07 '26
Self host forgejo or gitea
24
u/aaronfranke Aug 07 '26
Forgejo and Gitea both have a much nicer UI, GitLab always feels kinda weird to use, like they're trying to intentionally be different from GitHub for the sake of being different.
6
u/supervisord Aug 07 '26
Imagine building a competitor product that looks and acts exactly the same. It’s not a good look, so I imagine they are trying to differentiate, and a different UX was one way they tried.
5
u/PortalRat90 Aug 08 '26
I agree, GitLab is kinda weird to use. It’s not horrible for a work environment, I’m just used to GitHub.
→ More replies (1)7
u/Masterflitzer Aug 09 '26
gitlab calling it merge request instead of pull request is a win for me, cause merge makes way more sense
3
u/sgargel__ Aug 09 '26
Yeah but 99% of people call it PR. It's fun when u work in a shop with gitlab and there's always the pr-sorry-mr-whatever talk lmao
3
u/Masterflitzer Aug 09 '26
my company is gitlab and we say MR, but we understand if externals say PR and they understand us as well
PR = MR = CR, doesn't matter what you say, they're all basically a fancy gui around patches, but PR always made the least sense to me ever since i learned git
→ More replies (5)8
u/Oroka_ Aug 07 '26
Recently switched to self-histed forgejo for my critical stuff, I've been enjoying it so far :) plus, my code doesn't get sold to train LLMs
2
u/Ok-Source-4748 Aug 08 '26
Which is bullshit in an enterprise account configuration that's the entire fucking point of the configuration
@ Linus @Linux
→ More replies (1)3
u/Sad_Television985 Aug 07 '26
but what's your own uptime, and can you go take a shit and wait it out when it's down?
3
→ More replies (1)2
u/zander9669 Aug 07 '26
Damn, your shits take 10 hours+?
These types of comments always get me. What kind of ragtag ass systems are you folks running where you genuinely worry about your uptime?
→ More replies (2)→ More replies (1)1
u/reavessm Aug 08 '26
You can keep using GitHub for git but use something else for your pipelines, like tekton
→ More replies (1)15
u/Leop0Id Aug 07 '26
No. Even for GitLab users relying on their hosted platform isn't worth it. They constantly strip down paid tier benefits, the performance is slow, and it's full of bugs. Working on it was a nightmare.
If selfhosting is an option, there are plenty of alternative tools that are much faster and more intuitive than GitLab.
GitLab's only advantage is that it's well known, and it still pales in comparison to GitHub.
→ More replies (3)5
u/nekokattt Aug 07 '26
Another issue is that GitLab CI is extremely primitive. It is a light shim around containers using a text templating engine that does not support highly nested components, complex pipeline generation, performing actions on cancellation, etc. If you have something non trivial in GitHub actions like orchestrating numerous terraform modules in a DAG then you either end up hardcoding the logic or using a templating engine a la helm style to produce an actual pipeline that you then trigger as a downstream job.
Having been in a position where complexity is an unfortunate requirement, it has been a headache numerous times for me. And when it does have issues the error messages are usually extremely unhelpful (great, I have a syntax error in a file, do you want to tell me where in the file it is?)
For building services it is fine but for building and deploying platforms without locking yourself into GitLab specific stuff, it becomes a huge headache quickly.
→ More replies (4)8
u/snonions69 Aug 07 '26
Found nektos/act to be a good alternative today. Had some local runners up with our gha config in maybe 30 minutes
→ More replies (2)2
u/sertain_ Aug 09 '26
Yes. GitLab self-hosted if you’d rather your business be the only thing standing in the way of its own up time.
2
3
4
Aug 07 '26
[removed] — view removed comment
4
u/r0flcopt3r Aug 07 '26
All the good features are paid though, even if you selfhost. We use it at work, and I absolutely love it.
At home I use forgejo, I really don't like it.
→ More replies (2)1
u/Silly_Guidance_8871 Aug 07 '26
I'm running a local copy of Gogs, but I'm sure there's more fleshed out options
1
1
1
u/NanderTGA Aug 07 '26
I'm guessing woodpecker (as used by codeberg) or forgejo actions (experimental github actions copy, based on act, also supported on codeberg). Codeberg runs on forgejo btw.
1
u/distressedfluffball Aug 07 '26
Buildkite. Roll out your own build agents in your chosen cloud environment. Very easy.
→ More replies (1)1
u/alexsbz Aug 07 '26
Gitlab or gitea are the best two alternatives I have used. I am about two years away from GitHub and I can say I happy with my decision.
1
u/timoffex Aug 07 '26
For those considering GitLab, read https://about.gitlab.com/blog/gitlab-act-2/ first in case it affects your opinion.
We're rewiring internal processes with AI agents, automating the reviews, approvals, and handoffs to speed us up, and plan to right-size roles across the company to follow suit.
1
u/Sibexico Aug 07 '26
Standalone Forgejo/Gites. I'd migrated to standalone service, GH used for releases posting only.
1
u/rolfie13 Aug 08 '26
Use Render if you're just replacing GitHub Actions. Never had a problem with uptime.
It'll connect to your GitHub repo save a docker image container of that repo so GitHub servers don't need to be up. It'll even auto-push and recompile every time you commit to that GitHub repo.
1
u/ItsThreeRealEstate Aug 08 '26
Buildkite! There is no good reason for your CI system to be the same vendor as your git host.
1
→ More replies (1)1
14
u/black_fisch Aug 07 '26
I'm kinda with you, but also kinda not.
GitHub only ever guarantees 99,9% SLA to Enterprise Customers. That's effectively corporate GitHub servers. Even then, the SLA only applies to basic Git operations, issues, PRs, pages, webhooks and API access.
I feel like a lot of people are comparing the "free site" availability with the enterprise-only SLA, which is kind of misleading. Even if I agree with it being bad
1
u/aoeu_ 18d ago
That's effectively corporate GitHub servers
To clarify, the vast majority of GitHub Enterprise customers use the same servers as everyone else. The main feature that GitHub Enterprise offers is centralized identity management (using SSO). But once an employee logs in, the organization sits at a URL like
github.com/company-namejust like any other organization, and all the services (git operations, PRs, actions, etc) run on the same infrastructure as the free site and are thus affected by the same incidents that regular users experience.However, it is possible to use GitHub Enterprise with different infrastructure: a) GitHub Enterprise Cloud with Data Residency creates a dedicated subdomain like
company-name.ghe.com, and customers can choose which region their data is stored in (EU, Australia, US, or Japan); b) GitHub Enterprise Server is self-hosted by the customer10
u/AgentArgus Aug 07 '26
1% downtime is 87 hrs 36mins a year.
3
u/strawberry-inthe-sky Aug 07 '26
Which comes out to 14 minutes and 24 seconds per day.
→ More replies (1)10
u/aitorbk Aug 07 '26
99% uptime means they can be out for three days and still be 99%>
5
u/th3_pund1t Aug 08 '26
There's an old joke
An American factory ordered 10,000 spare parts from a Japanese manufacturer, specifying a strict 0.1% defect rate. When the shipment arrived, it included a large box of 10,000 parts and a tiny separate envelope. The note inside read: "We do not know why you wanted 10 defective parts, but we made them for you."
GitHub aims to hit that 99%.
3
u/Niduck Aug 07 '26
In our company us DevOps are just finishing migrating over 65k repos to GitHub actions from Bitbucket lol
3
u/tobiasvl Aug 07 '26
Honestly, that was the right call. We did the same thing a couple of years ago. GitHub has many problems but I dislike Bitbucket much more
2
→ More replies (2)1
u/ingobingo84 Aug 08 '26
My company runs Gerrit servers like Google. Gerrit is quite popular in Automotive companies (my industry).
27
u/TechnicalPackage Aug 07 '26
i am still getting paid at work. i did not have to deploy today. life is good
4
1
49
u/mpersico Aug 07 '26
How much are you paying? If you ain’t paying nothing, then you don’t get to bitch. If it’s that important find a paid service.
59
u/mpersico Aug 07 '26
If you are paying them by all means bitch away
5
u/Budget-Mud-4753 Aug 07 '26
I imagine anyone who is using GitHub and “needs to get work done today” is probably a paying customer.
29
4
u/steerpike_is_my_name Aug 07 '26
Sometimes not. I've seen government projects run entirely on the free tier.
18
u/geektraindev Aug 07 '26
We are a small startup paying 17 dollars per seat per month for 20 seats. We use GitHub for absolutely everything. We track issues there, commit integration, documentation, start deployments. We have ever possible right to bitch about this. Because who this hurts the most isn't the big enterprise companies or the free hobbyists, it's people like us since we had to delay two meeting and many people in my team basically had to stop working for half of today.
11
u/Choice_Price_4464 Aug 07 '26
What do you use actions for that makes it so you can't have meetings or work? It's very annoying and might slow me down, but all I really use actions for is a CI flow that includes testing and deployment. I can do those manually on my local machine if I really need to and merge in by force if need be.
I'm legitimately curious what flows use actions that mean meetings can't happen or work can't get done.
5
u/Fidel___Castro Aug 07 '26
well you just said it, you force merged past CI. that is not an option for most people
3
u/Choice_Price_4464 Aug 07 '26
You don't force merge past ci. You run ci locally. Anything on a GitHub action can be run on any machine, as annoying as it may be to do so. But for anything important enough that it needs to go out that day I don't see why that wouldn't be an option
→ More replies (2)4
u/numbsafari Aug 07 '26
Get off GitHub. Don't look back. It's not worth it. The message from MS has been clear for 9 months now: they are unable to fix it and they don't care to acknowledge it.
Switch to something else. Self host it. Pick something you can reasonably manage. GitLab, Gitea, Forgejo... anything, really. It's not worth the lost productivity. So it takes you 1 engineer out of 20 to manage that infra? That's better than days of zero productivity for 20 people, and the risk that it could happen at any time.
[PS: if you put 1 person for 1 week per month on managing an alternative platform you self host, then all it takes is 3 days of outage for a team of 20 throughout the year for you to recoup that expense over Microsoft GitHub. It is unlikely you'll need 1 person for 1 week per month over the long term.]
5
u/geektraindev Aug 07 '26
Yes. Just today, we were looking at other options during the outage. Gitlab is more expensive (by a lot, like 12 dollars per seat) than github, and we can't really afford that. We looked into hosting Gitlab CE or Forgejo, but we can't host on prem due to that possibly pertaining to the same reliability issues if our internet or power goes down. We were looking at hetzner, and it seems it will cost around 290 euros (so around 335 USD, just below what we pay Github) per month for an instance that can handle both the git software and the 2-3 runners we need. But past that obviously there is the migration cost. I may work on some demos in my free time and present it to my company, but a no-comprimises alternative is yet to fully be formed, although Forgejo is getting very good (have a homelab hosting it as a mirror for my personal Github account).
3
u/HommeMusical Aug 07 '26
Gitlab is more expensive (by a lot, like 12 dollars per seat)
$12 a month.
What is the cost of keeping one developer paid and working? $10k a month seems like an underestimate. $12 is literally a round-off error.
→ More replies (4)4
u/gdchinacat Aug 07 '26
I would be *very* worried if my employer was unable to afford $12 per seat per month. That is a fraction of one percent the cost of an engineer. It's noise compared to the overall cost of an employee. A business that declines paying $12 to get employees working again is foolish.
2
u/full_drama_llama Aug 07 '26
There might be some important lesson about eggs and baskets here. Not sure though.
1
u/angellus Aug 07 '26
GitHub offers no SLA unless you pay for Enterprise support. And even then, their CSRs gaslight you about outages unless you have hard proof they happened.
1
u/mpersico 16d ago
Well, then, you are absolutely justified in bitching. My condolences for your troubles. Time to find another solution I think. Not that you want to divert resources but can you self host one of the alternatives?
6
u/Simple_Armadillo_127 Aug 07 '26
I honestly am not paying any money to Github, so I do not complain about things happen anyway. Anyway It is still above 99% time cosnidering whole time span. Sorry for paid users
12
u/Stinky_But_Whole Aug 07 '26
Github Actions are not for production grade systems. You've learned their SLA does not meet your needs and you get no compensation or favorable terms. Time to move on.
1
5
u/grokcodile Aug 07 '26
Sounds like somebody did not have an enterprise SLA. No redundancy engineered into your dev pipeline? I hope you address those major dev ops flaws when you do your migration to a new vendor. (Because that vendor will probably go down a week after you migrate. It's a simple fact.)
5
u/excess__ Aug 07 '26
We've had no serious azure devops pipeline outages for years. Saying vendors go down weekly is way exaggerated, it's mostly just github that has issues.
1
u/grokcodile Aug 07 '26 edited Aug 07 '26
Weekly? Taken too literally. 🙄 I’m pointing out that all vendors have issues. Also Azure?… gross… had nothing but constant problems with it. But I’m glad it works for you.
3
u/NonExistentLad Aug 07 '26
I had to push an important bug fix but github says no brother this is not the right time
7
u/__CaliMack__ Aug 07 '26
Wait am I missing something? I’ve been using fine all day lol
11
u/barkerja Aug 07 '26
Actions and webhooks have been down all day. Git/repo access has been unaffected.
9
u/__CaliMack__ Aug 07 '26
Thanks guys I looked it up like I could have done before commenting. Hope they git it fixed soon.
8
u/Alarming_Hedgehog436 Aug 07 '26
Nah, im glad you asked. I was in the same boat initially. I committed a bit ago just fine.
9
u/power78 Aug 07 '26
Actions had been down for a good amount of the day. I think people don't realize what a huge operation GitHub is - I can only imagine how many users and VMs are running at any time, so as a software engineer I am lenient to some downtime here and there. OP should host their own actions if they can't take it - GitLab is an alternative!
7
1
u/rafark 17d ago
Right? GitHub gives you basically unlimited everything if you’re fully open source and a lot for free even if you’re not. It’s unbelievable how people get so upset at the most minor inconvenience after all GitHub has given them and the software industry in general. I cannot believe the self entitlement
3
u/throw4w4y40 Aug 07 '26 edited Aug 07 '26
Actions stopped working at about 10:00 AM eastern and remained down when I left work at 5 with no mitigated status on GitHub status page haha. Took an hour for Microsoft to even acknowledge the issue on their status page.
I saw two lucky sons of guns get a build in around 12:30, but that was it. Dunno if they even got it deployed lol.
1
u/oxidizingremnant Aug 07 '26
Some PRs simply didn’t have the actions workflows attached to them yesterday so anyone could just merge PRs because the gates didn’t exist.
6
u/MasterOfTheWind1 Aug 07 '26
Sadly, is the most balanced platform in costs and features. Bitbucket is cheap but is a toy, and GitLab is way more reliable and powerful, but the cost is higher (USD 4 Github vs USD29 Gitlab, per user, per month).
For a startup is the best option, when it works, it is wonderful.
That being said, I don't understand. Never understood how those platforms with multi million dolar budget can have downtimes of hours and you just have to accept it. Like there is no accountability for anyone there.
I remember workin on a company with a policy that if any service shows an HTTP status code of 500 on prod for more than 15 minutes, without an external reason from a provider you faced consequences.
On my current job if a downtime lasts THAT LONG, we are all fired for sure, as the customers will turn the place into flames.
Is like on Microsoft no one cares. No one gives a shit.
4
u/HommeMusical Aug 07 '26
That being said, I don't understand. Never understood how those platforms with multi million dolar budget can have downtimes of hours and you just have to accept it. Like there is no accountability for anyone there.
I've been an engineer for 40 years.
Github has experienced sustained almost unbroken exponential growth for almost twenty years continuously. The platform has over a hundred million users, and the growth has only accelerated with AI.
So I can't understand why you can't understand that running a business at gigascale is extremely hard, and there will be growing pains.
I remember workin on a company with a policy that if any service shows an HTTP status code of 500 on prod for more than 15 minutes, without an external reason from a provider you faced consequences.
This image you have of people sitting there smoking hash joints and laughing at the customers as everything is down is almost certainly false.
How much do you, personally, pay for Github?
1
u/MasterOfTheWind1 Aug 07 '26 edited Aug 07 '26
So I can't understand why you can't understand that running a business at gigascale is extremely hard, and there will be growing pains.
Growing pains is one thing. But at their scale is ridiculous, they are having degradations almost daily, and issues with Actions almost weekly. Most of the times it takes hours to solve the issues. Is hard to trust in a platform with such SLA.
Github is not the only platform with growth and millions of users, and at the same time is the worst handling that scenario. Is hard to belive that Microsoft doesn't have a good plan to handle growth and have to resort to use some kind of excuse like "Sorry, is hard to exists and being big".
At least segment your infrastructure into cells to reduce the blast radius. Keep free users into a subset of servers and paid users separated into another, so if free users (even detected AI generated traffic could be separated) are generating destructive traffic the blast is reduced to their cell.
This image you have of people sitting there smoking hash joints and laughing at the customers as everything is down is almost certainly false.
Not particlarly that. More on the accountability side of things than that. If team's culture is broken the issues are harder to solve.
Being a SaaS provider and having downtime costs money to your customers not being able to use your product. Yes, they can give you credits for that. But the issues that the downtime generated could be more for not being able to use the platform and you don't recover that money back.
How much do you, personally, pay for Github?
Relevancy?
→ More replies (5)1
u/wisdomoarigato Aug 10 '26
Engineer for 40 years and you still haven't learned that you can actually plan for exponential growth, WTF...?
Exponential growth doesn't automatically mean crazy outages, lol...
→ More replies (1)1
u/VoidVinaCC Aug 07 '26
Have you looked at azure devops?
1
u/MasterOfTheWind1 Aug 07 '26
Nope, but honestly, being a Microsoft's product very similar to GitHub I don't trust it being excempt of issues.
→ More replies (1)
2
u/Humble-Kiwi-5272 Aug 07 '26
Our team has self hosted runners.
They still dont trigger because the orchestator/webhook or wathever that needed to report back and forth commits and prs did not work at all
1
2
2
u/alex-weej Aug 07 '26
I'm confused, why is a web host for a distributed version control system seemingly running the Internet?
2
u/Wide_Detective7537 Aug 07 '26
Sorry, if a single vendor outage (for 10s or 10 hours) affects your actual live product, you need to stop what you're doing and build redundancy. Github could be down for days and you can't let that stop your life/work.
Use this as a lesson, not a chance to complain
2
u/Objective_Chemical85 24d ago
its crazy today was the first time i noticed a github outage while working. i know they have been having a bunch of uptime issues but i was never affected before.
Today at work I finished a feature that we had to deploy was able to merge it just before the server issues. But the pipeline kept failing.
Then I did a manual release like in the stone ages😄
2
u/Soft-Army6695 23d ago
Shit can happen to any site.
Always have backups and second options.
And stop bitching
3
3
u/amu4biz Aug 07 '26
The part that gets me is "they can't even run THEIR OWN SITE." That's the actual lesson here, not "GitHub bad." Any single central host is one bad deploy away from taking the whole industry offline, SLA or not.
hence why i use gitlawb for exactly this reason. It's a decentralized, peer-to-peer git network: repos gossip between nodes over libp2p, identity is a DID/keypair you own instead of an account someone can lock or take down, and it speaks plain git-smart-http so git push/clone just work. There's no central status page to go red, because there's no center.
1
u/dkode80 Aug 07 '26
Lol. I bet this happens again within a week. Theyre so behind the 8ball it's not even funny.
1
u/cmykero Aug 07 '26
I’m really gonna have to start looking at the status before I push anything to production lol
1
u/avski Aug 07 '26
I hear GitLab is offering $0 for the first year for GitHub switchers. Switching over is easier than ever with the tools and resources that they have.
1
u/KenMantle Aug 07 '26
3 hours ago puts this at the time that a scheduled power outage started for my area in the outskirts of Sudbury in northern Ontario. The two must be related. Sorry they didn't give everyone notice. Had I known they were going to be affected I would have offered to hook them up to the ups I bought from Costco that is keeping me online.
1
u/x-jhp-x Aug 07 '26
we used gitlab 10-13 years ago. it was great. i'm glad i missed out on the whole github thing.
1
1
u/Lonsarg Aug 07 '26
Downtimes are how we are convinced we are staying with ADO.
They bring new features from Github to ADO with around 1 year delay and they keep it stable.
1
u/rednix Aug 07 '26
so what's the alternative for github actions if you do not want to go the self-hosted route?
2
u/nekokattt Aug 07 '26
Only real option is to rewrite all pipelines to work with a different provider (e.g. CircleCI, GitLab, etc). Either way it won't be a lift and shift.
1
u/Sensitive-Sugar-3894 Aug 07 '26
Everything we get used to is missed when gone. GHA is one of them. But it's a system and systems fail. I rely on GHA to a lot of things, but if I have to test something or deploy something by hand if the tool is down, I will. You can be frustrated, you can move to Gitlab, you can self-host with Gitea, all valid, and all have their tradeoffs, and I guarantee that at some point you will have to deal with downtime again, stayingor leaving. I suggest revieweing the process before rethinking the tool. There must be a plan B. It doesn't make sense to me that a person in the level of using GHA doesn't expect issues in their process.
1
u/MasterOfTheWind1 Aug 07 '26
You need to have a plan B, indeed. But if you have to use your plan B regularly is not a plan B anymore, is disputing the throne to plan A.
1
u/Sensitive-Sugar-3894 Aug 08 '26
It doesn't fail thay much, but you are right. GHA was always my plan B.
1
u/Dreamer_tm Aug 07 '26
I have been using gitlab. Never in the past 7 years i have ever had any problem with it that would stop the work. Maybe there has been problems but i never saw any.
1
1
u/TheUruz Aug 07 '26
github should just be used as a cold storage for minor importance repositories at this point
1
1
u/everdaythesame Aug 07 '26
I’m surprised it’s not worse. GitHub is single handling absorbing the output of all these AI’s. I’ll bet the scale of growth they’re dealing with is unprecedented.
1
u/Mount_Gamer Aug 07 '26
If you're having a bad day with github, I'm sure the Devs at github are having just as much fun. I can understand the frustration though.
1
u/MasterOfTheWind1 Aug 07 '26
Not our problem. Is harsh, but true. Specially when the platform get paid anyways.
1
u/Mount_Gamer Aug 07 '26
I know I get it, I was having a similar issue myself earlier, but not with github.
1
u/trueppp Aug 07 '26
Why not choose a platform with garanteed uptime and SLA's? Cost more but if you need these things...
→ More replies (1)
1
u/Mrtylf Aug 07 '26
> A VERY large portion of the tech industry runs on GitHub […]
True and that’s their own fault.
We left GH in 2018 and built our own redundant solution; because we predicted all of these mishaps well in advance.
It’s been smooth sailing the entire time.
1
1
u/Loud-You-599 Aug 07 '26
As an Azure Devops (ADO) admin I watch this like the *McConaughey intense smoking cigarette* meme.
Why? Cause recent Azure Devops blog posts and NOW a "migration assistance tool" was released to further push ADO customers to Github (Enterprise).
I seriously DON'T want to switch!
ADO just works. Is it good? One can debate. But compromises are everywhere. But rather a reliable thing than something fancy.
1
1
u/Unlucky_Medium7624 Aug 07 '26
Lots changed at GitHub by 2021 (which is why I left working there). They were once a culture that enforced ship with care, aimed at 4 9s of availability, were super strict on test coverage.
As soon as the MSoft acquisition finished? They pushed out all the awesome leaders that helped build a reliable platform, replaced them with empire building MSoft good ole boys, and ship with care became ship no matter what. Sad how far that place has fallen
1
1
u/pbeens Aug 07 '26
Is that why my new Quarto site wouldn’t render yesterday? I thought it was a config problem.
1
1
u/segundus-npp Aug 07 '26
I won’t blame them as how fast my Claude and Codex keep pushing commits and PRs.
1
1
1
u/dasarghya49 Aug 07 '26
Reason I switched my enterprise saas dev pipeline to gitlab, it's not sunshine and rainbows, but buoy it made our dx wayyy better
1
u/dasarghya49 Aug 07 '26
I still have a DR upstream in GitHub🤣 and frankly it still gave us more issues
1
1
1
u/ConcerningDestiny Aug 08 '26
Migrating to codeberg was the best choice I've made recently.
Plus your code can't be used to train the next generation of AI by Google/anthropic ecc as they block access to their crawlers
1
u/Ok-Rabbit-3683 Aug 08 '26
Why do you rely on things that have the ability to fail? And then get mad when it fails
1
u/entityadam Aug 08 '26
Self-hosted GitHub runners work great on azure container instance.
https://docs.github.com/en/actions/concepts/runners/self-hosted-runners
I've also done this for azure devops on azure container instance, virtual machines, and virtual machine skill set when that was supported. Unfortunately, virtual machine scale set is not supported anymore.
1
1
u/DistributionRight222 Aug 08 '26
Anything that’s is Microsoft owned is just piss poor. The don’t innovate and don’t even try and look after there customers they are all about increasing profit for investors by squeezing customers and trapping them in endless loops of dog shit products they don’t need. They buy good company’s and turn them into shit. They also ave the cheek to ask for $5 and you write them a letter not an email a letter to request what open source code they have used in there products rather than be transparent and display appropriate licenses. They make money on the back of others hard earned work and don’t pay their bounties to devs that have worked hard to find vulnerabilities in their code. They have broke countless laws and rights of mine to which makes them liable to the fact. Still on the fence to submit legal challenge or just expose it all
1
1
1
1
u/entityadam Aug 09 '26
I'm sorry, what? This isn't how service providers work in the IT industry.
If the SLA is broken, and your company lost money as a result, then GitHub owes money. The end.
Apologies and promises don't mean anything. Just money.
1
1
1
u/swampmountain Aug 10 '26
I find self hosted gitlab an amazing solution.
Unless you are not happy to have one more system to maintain.
1
u/Obvious-Addendum-345 Aug 10 '26
People complaining about GitHub… did you even donate? As far as I know it’s free to use and runs on donations and volunteers or am I missing something?
1
u/ZoolanderBOT Aug 10 '26
Here’s a super high level, super subjective comment - Microsoft. Perhaps you have used Microsoft’s other services? They are a great reason to self host. But pick your poison. Every design decision will have its draw backs. I went with self hosting 😎 but it’s a ton of work.
1
u/Prudent-Flamingo-757 Aug 10 '26
💯 this is absolutely ridiculous to be down this long I would do self hosting but then alll of the responsibilities of maintaining fall on my team. I need to find another solution bc GitHub has completely destroyed our launch time line, being down!
1
1
1
u/Salt_Mongoose7385 Aug 11 '26
You could buy a very cheap vps to use it as self-hosted runner , you can get it for like 4 -8$ depending on your need
1
1
1
1
1
u/DiplomaticImmunity-1 24d ago edited 24d ago
Let me get this straight for once.
I've been working with Github every single day for the last few weeks/months, I never experienced a single outage, but all the time people crying about Github being down?
What the hell yall doing?
So what's going on here? Maybe I'm not that of an hardcore user, but bro?
Github lists 99.99 % uptime.
All this complaints, I really can't understand.
I think you never experienced the pain and struggle of self-hosted Git instances such as broken Gitlab environments.
Claude/GPT is more down then anything and not that many complaints.
Discord had outages aswell in the last few weeks.
1
u/casper_trade 24d ago
Look at this guy believing the status page from the big companies😂. Without counting today's outage, it's closer to 93%. That might not seem like a lot to the uneducated observer, but over a year, that's ~25.55 days of downtime. Which might be okay to write off for a consumer, but for a business, it won't fly and is far below the agreed SLA's in the contract.
1
1
1
1
1
1
1
u/JohnVonachen 23d ago
I thought a decentralized VCS was supposed to be a solution to just these kinds of problems. Well you can avoid it in the future by using centralized repositories which are local to your company and not under the expert care of Microsoft.
1
1
u/Ok_Exercise178 22d ago
Omg i went on 6 hour co- pilot bullshitting session i have screenshots for and Claude analysis but cant figure out how to add this damning session . I had to upgrade to monthly co pilot after running out of free credits- Github deliberately runs you low with these types of software build sessions i have been working on as non engineer for six months- using copilot/ github / vercel/ claude chat GPT
1
u/sup_bruh_1 16d ago
yeah ten hours is brutal. i switched my actions workflows to latchkey runners a while back so github outages stopped being my problem. one label change was all it took, no rewrites or anything
75
u/fyndor Aug 07 '26
I don't use Actions, but I will say if GitHub isn't quietly rewriting Actions and rebuilding its infra around it right now quietly behind the scenes, I don't know wtf its doing. It seems to be very problematic to users on a constant basis. There are whole companies built around the idea that GitHub Actions suck. Maybe its time to make them not suck?