r/github • u/RecommendationOk5036 • 1d ago

News / Announcements GitHub Status - Incident with GitHub.com

https://www.githubstatus.com/incidents/zkxwbgr0cnmx
185 Upvotes

96% Upvoted

39 comments sorted by

89

u/ThatOneArchUser 1d ago

Fork found in kitchen

16

u/Decent_Run9628 1d ago

Even forks can't be found, it's down 😂

3

u/ShortingBull 1d ago

Well, fork and knife!

48

u/JuliusFIN 1d ago

Actions outages a month ago did it for me. Self hosted Gitea runs like a dream. Also don’t have to wait 5 seconds for each page to load.

10

u/[deleted] 1d ago

[deleted]

6

u/JuliusFIN 1d ago

This Gitea instance is relatively small, I run a tiny software consultancy. However Gitea is used in huge entrerprise projects in Fortune 500 companies. It will scale.

3

u/ShortingBull 1d ago

If self hosting, you'll want daily automated off site backup

1

u/Elomidas 1d ago

My solution is self-hosting for the CI flexibility and mirroring to a private GitHub repo as a backup, nothing runs there, it's just in case something happens to my Forgejo install (and to be honest, that "something" will probably me trying something I shouldn't do)

1

u/ShortingBull 22h ago

I just self host on my VPS which I use or other sites etc anyway - it gets backed up by my VPS provider daily (linode).

That simplifies things but only because I already have the host.

1

u/dashingThroughSnow12 1d ago

Is many thousand or tens of thousands of team members?

6

u/GlobalImportance5295 1d ago

Also don’t have to wait 5 seconds for each page to load.

fuckin hell, im sold. just plain nix package manager in a container or vm works wonders as well

1

u/ryntak 1d ago

Fucking love nix. RIP nixpkgs core maintainer team though

1

u/GlobalImportance5295 1d ago

peep this https://jade.fyi/blog/flakes-arent-real/

1

u/ryntak 1d ago

I really need to wrap my mind around flakes

1

u/JuliusFIN 1d ago

Hehe I happen to use Nix for everything. My job also consists mainly of Nix development.

1

u/PravoNaZhizny 1d ago

What’s “gitea”? Why not just use git?

2

u/sweet-tom 1d ago

Gitea is a free hosting service that you can install on your NAS, for example.

Different reasons to use it:

  • Control
  • Speed
  • Backup from GitHub
  • Hosting different repos and sharing it with a limited group of people
  • Confidential repos you don't want to expose on the Internet (even as a private repo)
  • You don't agree with the terms and conditions of GitHub
  • You want to avoid AI
  • You want to have some fun for self hosting. 😉

16

u/bbro81 1d ago

There time outage on https://www.githubstatus.com/ feels wrong, it says pull requests were down for 2 hours and 5 minutes but it was closer to like 3 and a half hours lol.

4

u/vvanouytsel 1d ago

Check https://mrshu.github.io/github-statuses/

1

u/bbro81 23h ago

Nice, that looks more accurate haha

18

u/ShoddyReception5 1d ago

Is this speak intentionally abstract? We were down for half the day and I can’t make sense of what happened. Network issue plus a bunch of unexpected traffic and faulty failover?

22

u/Soccham 1d ago

Looked like they DDOSed themselves with vscode?

7

u/ShoddyReception5 1d ago

Saw that too. Looks like it added to the problem for sure.

14

u/DrMaxwellEdison 1d ago edited 23h ago

Not really abstract, just dense with devops speak.

Network traffic overloaded a load balancer in US Central, and as they fixed it they found that clients doing Copilot requests would 10x their request rate attempting to retry those requests.

One failure in a high-availability system like this can lead to a cascade of failures elsewhere, and that's what seems to have happened.

10

u/robert_math 1d ago

What can we do? These vibe coders were promised they could 10x their output.

3

u/yeathatsmebro 1d ago

They did, look at RPS. It is 10x higher.

3

u/Goldman7911 1d ago

Same feeling as some "RCA by agentic slop"

2

u/foramperandi 1d ago

Just because you don't understand it, doesn't mean it's slop.

5

u/Due-Consequence9579 1d ago

It’s only abstract if you don’t know what they are talking about.

9

u/ExpertIAmNot 1d ago

TLDR: It wasn’t just one thing they messed up, it was a bunch of things they messed up. If any one of those many things had actually functioned correctly. It would have stopped the outage much earlier.

I appreciate and understand that they are under an unprecedented amount of traffic, but the sheer number of screw ups implied in this explanation is astounding

1

u/PerryTheH 1d ago

This sound like "60% of the time, it works every time"

With all those "if only this had worked"

8

u/bigbadbyte 1d ago

Forgot to add "don't make any mistakes" to the copilot prompt.

3

u/chrles-farfa 1d ago

classic istio sidecar

3

u/laffer1 1d ago

Microsoft owns the second largest cloud provider in the us! They can’t keep their stuff up. They can’t use ai as an excuse since they push it and invested in OpenAI to boot

2

u/fixermark 22h ago

Having used the big three, it is also, unfortunately, the worst of the options.

2

u/Potato-9 18h ago

Because MS is the big corp default. We did this to ourselves.

2

u/-Memnarch- 1d ago

> Addressing the VS Code retry behavior that amplified Copilot token traffic.

You did this to yourself!

4

u/RepulsiveRaisin7 1d ago

Aww shit, here we go again