> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity.

This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling failures)", but rather "this complex system collapses (rather than degrade gracefully) when demand exceeds capacity".

When components reach capacity limits, the excess traffic of the lowest priority should be rejected. Rejected traffic should not be retried — in fact, not only should clients not retry these errors, these errors should cause client-side throttling. Traffic isolation should be applied — if the cause of the overload is a single client/customer system, no other system should be affected.

Nearly a decade ago I wrote about some of the techniques we applied at Google to implement these protections: https://sre.google/sre-book/handling-overload/ Most other large internet services have since copied them, afaik.

Why does Github not segregate the free offerings from the enterprise or even better, all paid offerings?

It is unacceptable that enterprise plans get impacted by traffic on free and public repos. Our repos are neither on the free plan nor are they open. We have not had more AI stuff happening in the last weeks. Our traffic is stable. I would wager that most enterprises did not spike the traffic all of the sudden. Even if they were, we are paying for our quotas. Still our Github actions were breaking and our PRs not viewable at some times.

I am hoping this instability is going to cause a Cambrian explosion of forges and if that is happening, Github will be the first victim of the AI revolution.

I am working on a truly decentralized / local first code review right now, and a big part of my motivation for this is how bad Github has become. I dont know if I have enough time to build CI as well, but I am hoping others do. Otherwise I will just fall back onto Jenkins.

  "Since April, monthly commits have grown from 1.4 billion to 2.9 billion. "

Wow, that is some incredible growth in a really short time.

I applaud GitHub. However, I think no matter how valiant they are they will not climb out from under this. The scale problem will keep getting worse, and it's getting worse in a way I don't think is translating to more money for them. Sooner or later, they're going to have to charge for things currently free.

I've been saying this for a while: https://news.ycombinator.com/item?id=47534499

> Since April, monthly commits have grown from 1.4 billion to 2.9 billion

Bonkers.

You can tell the entire industry is in a "productivity panic" and here's more proof. There's a velocity zealot crying tears of joy somewhere.

Everyone suggesting that they simply charge users for commits to drive off AI-heavy users forgets that Github is owned by Microsoft, who has a big incentive to keep having developers use AI.

I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.

> Errors in those services triggered a client-side retry loop that increased traffic during recovery

Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours.

> Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.

The detailed root analysis tries to pass this off as a "bug". You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour is functioning exactly as designed. In this case aggressively to try and hide problems if token service responses become flakey.

> Errors in those services triggered a client-side retry loop that increased traffic during recovery.

The worst outages I've been part of always have some version of this :(

"The retry storm in Northern VA was fixed by 1) temporarily reducing gateway retry logic with a PR " - silly question but github uses github for their own PRs and deploy right? Do they have a special dedicated system just for them so they can fix github with a code change even if the rest of us can't?

Distributing across different services wouldn't be a bad idea....

I still can't help but feel a little grateful for what they do across the free side of things. I know it isn't altruism, and I know nobody needs to defend a billion dollar corporation but...

Name another service that does what they do for FREE (and no ads) at this scale. It isn't easy. Wikipedia has probably more usage, but is a simpler endeavor. (except the moderation part, that's just amazing) Open Street map? Smaller and simpler. Internet archive? Again, smaller and simpler. Linux distro mirrors? Again, smaller and simpler than whatever github is doing for free.

> Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits.

I operated services at similar scale, and generally we use to put a bit of slack so that you would get an alarm when capacity goes up to 80%+ (or whatever number makes sense)

This allows to check, in the morning, after coffee, why the load balancer fleet didn't scale up automatically.

I am sure there is a good answer to why this is impractical, but it would be nice to know

Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic.

I feel like I'm mildly stupid in trying to out retries as heresy but I'm not sure.

"... these incidents make clear that we must accelerate this work."

It feels like GitHub maybe needs to slow down? 'We must change things faster' is a wild way to start off an eight hour hard-down postmortem.

>We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.

Crazy.

I wonder how much of the traffic increase is enterprise vs. hobbyists? A 7 hour outage for enterprise customers is really, really bad and it's sad if caused by a mass of non-paying vibe coders. It's becoming absolutely obvious that the unlimited free tier needs to go.

I appreciate the unprecedented load GitHub is currently experiencing, but it's not just the (admittedly extreme) load of commits/pushes that is to blame.

Their Copilot cloud agent offering is suffering with a case of some of the worst corporate ADHD I've seen. We built a cloud agentic development pipeline on it, and it seems like almost every other week they silently change something with zero public announcement that creates real disruption for our team.

Note: that's not bugs in the Copilot platform like the article discusses. That's real, breaking changes to the platform that clearly aren't being tested/reviewed before being pushed to prod, with zero public announcement or documentation.

Support is useless – we're paying customers in the 4-5 figures and our tickets go unanswered.

I love(d) GitHub, but I do think they've lost enough public trust at this point that their time is ticking. With talk of new VCSes designed specifically for agents, I do believe it is just a matter of time. Which pains me somewhat to say.

Linux didn't (yet) kill Microsoft. Microsoft absorbed that shot. Then the Git arrow went straight to cold black heart of Microsoft. The next few months will determine if they survive it. If they do, what will we see from the third draw out of Linus' quiver?

I recently received a PR fully automated by Claude for an 8 years old repo. The bug is legit and the scope it affects is larger than what that PR addressed, but I no longer care too much about that legacy code anyway, so I also let Claude run free for the first time in my life, from handling that PR to fixing all related bugs.

I walked away for half an hour and back, found Claude opened and merged 9 more PRs and added a comprehensive CI for testing for all platforms. It will likely take me months to reach this level of output, but only half an hour for a capable agent.

No wonder why GitHub is down all the time.

I wonder what the ratio of repositories to physical machines is these days. I'd also be curious to see this as change over time.

I have a hard time with the premise that a mere doubling of git ops would be especially crippling for any particular repository. GitHub runs like ass because it's oversubscribed by a huge factor. Not because git is inherently constraining at scale.

Has to be one of the most vague outage summaries of the year

This article seems to say that there is "no excuse" for these issues but look at all these things we changed and are changing. It doesn't really feel transparent and it feels like they aren't really taking true ownership on what has happened.

Centralized source code hosting is going to end up looking like the three credit bureaus in terms of security. It's only a matter of time before the first big hack, when everyone shrugs and says, "Oh well, everyone's source code leaked lol too big to fail."

Is it me or is all of this essentially "we don't want to show the user anything at all when something breaks?"

And what makes this funny (to me) is that this is a website for developers. I would think that of all the audiences you would target, developers would mind seeing the platform display error messages when things break the least.

A lot of these projects and commits would benefit a ton from proper decentralisation. What functionality of GitHub are you *actually* using?

The trend doesn't seem sustainable.

After touting 1 billion commits over 2025 at universe last year… they are now handling 3 billion per month jeez. I’ll give them a little more grace after all…

Haven't they been migrating to Azure for a few years? How is it still only 58% done... Microslop needs to lay off the focus on AI features and get it done

I don't know where else to ask this but it's killing me. Does anyone know what the hell that GitHub physical CD thing was about? Did anyone in the world get theirs?

Interview question. How would you handle the growing traffic needs and traffic spikes. I'm curious whether any existing architectural diagram of theirs would highlight a potential failure post-mordem.

Cant they just put ai agents at optimizing their slow internal paths.

The 'growth in completed actions runs' graph is interesting. I assume the periodic drops are weekends, so intuitively the floor of those drops corresponds more with hobby/personal projects than people at work. It looks like there's a sharp uptick specifically in that floor since July ish.

Sorry to suggest this but if they charged everyone say $1/mo. it would absolutely help the massive surge from AI coding they seem to have had.

I don't like paying for free stuff but gh certainly worth it.

Would it be okay to suspect the recent upsurge in AI agent usage as a possible main cause of this issue?

Not again but..

the increase in traffic over the last few years is way bigger than I thought

The distinction between running out of capacity and collapsing when you run out of capacity is exactly right. Every distributed system hits limits, the question is what happens next.

Maybe Github (and especially things like Actions) just need to become more expensive?

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity.

I'm missing in these descriptions the most obvious approach: Resilience. Shedding load so that you can keep services up even though capacity is too low. If you flip over as soon as load exceeds what you can handle then this problem will never go away, unless you always have insane overprovisioning of resources which is uneconomical. There will always be spikes. You need to plan to handle them, no matter how high.

> we have focused on three priorities: adding capacity, improving efficiency, and removing architectural bottlenecks.

Sorry, but again, that is not good enough. They should ask themselves why they are expecting that trying the same medicine as last time will prevent next time. It won't.

With that mindset I'm not surprised this happened and it will surely happen again.

Edit: In more concrete terms. If you 2x your capacity and in a week you face a burst 2x of what happened last time, you are back in the same seat. If you improve efficiency by 2x, same thing. And after a bottleneck is before a bottleneck. There will always be a bottleneck. The key is to be able to handle a bottleneck. Removing one just pushes the issue to the next one.

Your architecture must be such that your whole system should be able to run on a raspberry pi. Most client requests would be dropped, but those that make it through will be served. If your architecture serves 0% because it crashes when load is 10% over capacity, then capacity increases or efficiency increases or bottleneck removal are not going to prevent the next outage.

5X as much code in <2 years. What is all that software?

I fear to ask, how archive.org keeps up to catch all those events for archiving...

AWS CloudWatch has an option to show the trend and what it will be like after x-period.

Doesn't Azure have such options so that engineers can predict to scale better? Seems like engineers are not ready for this per postmortem

“Required several coordinated actions” is the key moment for reflection.

Is git still the best VCS today? I ask because it seems so much effort in the industry has been invested in making git scale, like Cursor's Origin, or the stories in the past with Facebook's monorepo, but they all seem like bandaids to its intended design.

Github you can only post you are doing stuff about outages if its actually effective.

The vibes are off.

Centralized decentralized code repos. It feels like an oxymoron.

> We have made progress, but these incidents make clear that we must accelerate this work

Pretty sure this line appears in every one of these.

the retry loop that made the outage worse is just the internet being extra helpful. every client decided the best way to help was to ask again, louder.

What would happen if github was down for like a week?

I have setup a gitea instance on my gitea server which I think is good for me and GitHub both. For one, I dont have to worry about GitHub service outages, and GitHub gets to be free from my toy (and mostly AI slop) projects that no one else will ever read/use/participate in :-)

Great read - I'm glad they realize there's work ahead but what I'm missing is: * Paid customers: we know you pay us often a ton of money, and we burn your month on actions during these outages - we'll refund you for the days we spent your money and gave you no value. * Paid customer: We know you put your trust in us, so we'll ensure we have a separate pool of capacity to ensure we can keep that trust. * Paid customer: we'll proactively refund you when we miss our SLA.

What I read from this is: * Scaling is hard, we don't have enough capacity * We give away a shitton of compute for free * I have to talk about Azure not being a steaming pile of poop, otherwise my bonus will get tweaked downward in the next comp cycle.

Notice there's nothing about paid customers, I'll add in what they are missing:

Paid customers: Go F*ck yourself, you don't pays us enough to be an interesting line item compared to windows server.

Why not identify the lunatic top 1% of free user you know are just abusing the hell out of the system and put severe rate limits across the board for those organizations/accounts?

Why let your entire platform suffer?

With all the software being written on github you'd think we were going though a software rennasance. Where are the results? Is it really just all slop?

   Central US data center failed to scale with it

I'm in Europe and I experienced token failures as well.

That is amazing

"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products."

Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak.

Github is ripe for disruption and I hope it is disrupted soon.

> Copilot services took longer. Errors in those services triggered a client-side retry loop that increased traffic during recovery.

Let's pretend that the scale traffic is with the number of commit/pr and not self-inflicted with all the copilot eye candy features that were vibe-coded-added to GitHub.

In addition they say that they will continue their migration to azure and that azure is supporting their actions run. But GitHub actions is one of the things that was the most constantly broken without multiple outages recently. So I have the feeling that it proves the point that part of the stability issues is also due to their forced usage of azure.

With all the outages at GitHub there has to be someone willing to unseat them as the social git repo… how bad does it have to get before folks go elsewhere? Bitbucket and gitlab exist but are pawns compared to a king no?

As I mentioned before (https://news.ycombinator.com/item?id=49333107), they can mitigate these issues with limits, even for failure cascades. There should've been an all-hands-on-deck feature freeze 6 months ago to implement the limits needed. That clearly didn't happen.

I think it's because their leadership actually doesn't care that it goes down. A weekly outage is now an accepted cost of continuing to allow unlimited free access with infrastructure that cannot possibly handle the load. As a result, everyone is looking at their GitHub Enterprise bills and cost of stopped work, calculating how much they'd save by self-hosting.

This is a really good post.

I said in another thread that they can't blame increased demand for these outages, but the demand growth is genuinely insane for a company already operating at huge scale.

I guess we'll have to wait and see if they deliver now, but it seems like they're taking it seriously at least.

1.5 billion commits of worthless slop

github controls the productivity of a large number of very large tech companies. When there is an outage like this they are basically shutting down a significant number of factories for hours at a time. This would be as if during the hayday of detroit they just turned off the power grid at a time randomly at least once a week for hours. It's unacceptable. The amount of productivity lost is staggering.

We should be building tools that help us all move off of github as soon as possible. The amount of action code that will need to be rewritten is daunting.

nice

someone needs to build a open source aws

Related recently:

GitHub has alternatives, but no replacement

https://news.ycombinator.com/item?id=49135365

Why developers are ditching GitHub for Codeberg and self-hosting alternatives

https://news.ycombinator.com/item?id=48842611

and new entry:

Cursor Origin Code Hosting

https://news.ycombinator.com/item?id=49334209

We need to have a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.

GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when they're just starting out only to sell crap when they're addicted.

This seems really bad, to be honest. I think we may be fucked. There's just no way all these lines of code are doing anything useful. We're now just burning stuff in desperation and confusion.

Use less AI slop coding

Exponential growth. No company could handle that without some issues. Good luck to them. And for those who cannot tolerate this, there are many self hosted options.

And another outage. [0] Looking forward to the subsequent post-mortem on that one.

You might want to not go all in on GitHub anymore since it is very unstable to use. A self-hosted instance would have a far better uptime than GitHub over the years.

6 years ahead [1] on not going all in an centralizing everything on GitHub.

[0] https://www.githubstatus.com/incidents/bhbcjn4n3jzp

[1] https://news.ycombinator.com/item?id=22867803

Almost 8 hours of downtime across all core workflows, and the word "sorry" or "apologize" appears nowhere in this post.

"If you were trying to ship software that day, we let you down" is classic corporate non-apology speak.

I’m done.

[dead]

[dead]

[dead]

[dead]

Just add a queue. Now the outage is eventually consistent. /s

[dead]

[flagged]

[flagged]

well written

Ctrl+F "Sorry"

No results.

Cool

Github down, no hard drives available, no memory available, thanks AI!

Seems like we are headed for Tech Gridlock.

i wanna vibecode a replacement for git and call it jit

They should rewrite their Ruby code to a performant language.

No sorry we messed up your work?

> What we have done and what comes next

"You've seen what we've done. The August 21st outage comes next. See you then!"

I think it should be noted that the CTO of GitHub doesn't use his own product. No commits since January 2024: https://github.com/v-fedorov-gh

No side projects? Nothing? Just seems odd.

Reading this port-mortem / plan shocks me, this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started. It seems the solution has been capacity, capacity rather than architectural or data changes.

> Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations

How do you not have read-replicas / read caches at this scale yet? Which is what I am reading from this statement. You can of course get really far with sharding and whatnot. But at some point it might become worth it to engineer your data into a model that scales better.

Here's Vladimir Fedorov's GitHub contribution graph, as linked to as the author of this post:

https://imgur.com/a/zIbT0Gi

It shows zero contributions in the past year, on this account. This is a huge, huge red flag.