Redesigning source control at Anthropic
403 points - today at 3:49 PM
SourceComments
> ..., the latent demand for merging code is practically infinite, and every bottleneck we remove immediately leads to a further increase in the amount of code we produce.
If the goal is to write software which solves real problems, how can the demand for code grow infinitely?
The guy who ran that project called it his "$LASTNAME law" that every large company will at some point write their own source control system.
I laughed.
About a year later I went to Google, and within a couple of years Google also did the same thing.
Not at all surprised. Git was just good enough (and cheap enough) the companies didn't need to do this to escape the {phsyical scaling, cost scaling} problems that most commercial SCM systems had.
Less-than-great news for code artisans. You can be brilliant, but can you divine the Right Way to Write It if you're up against an apparatus that starts at "pretty good engineer" then brute-forces its way down an exponential search tree?
The first thing that hits me is, did you consider not going all in on a monorepo? I know it has been the trend in general, but it seems a bit of a red flag when you go from "let's have a monorepo it will make merging easier" to "let's redesign version control because merging is too hard now" without the middle question "should we really have a monorepo?"
As a derivative thought I'm curious if AI tooling is tipping people more towards monorepos now since it gives the agent the whole context of the app in one go? Or is this independent - I can easily check out 5 repos in one directory and run an agent there.
I feel like the advantage of having these models in house can't be understated. Companies with access to inferior models, trying to compete, will be left in the dust when it comes to general development.
These are just some of the things - there may be many more.
If I was an investor I would be on the phone with Dario asking why we are wasting engineering talent on reinventing source control and churning the product with 30k pulls per day.
But it's a sad example, for open-source and for small engineering companies. Instead of infrastructure being more easily shared with the collective, or valued enough to build a business, now companies can tailor even major infrastructure to their own needs, and avoid all that coordination.
I'm glad they publish their notes to encourage others, but I suspect we're entering an epoch where confidentiality matters most.
Anthropic's problem was merging code at scale, but the scale problem I have with SCM is really merging all the related docs; these get lost in a stream of a forest of comments, PR's, and documentation, all of which (like a wiki) goes rapidly out of date before anyone notices because none of it is automatically correlated.
It sounds like Claude loops on itself and must be pushing millions of lines of code changes per day. Even the humans canāt type commit fast enough. Are these people soon to be laid off?
The tradeoff between building something custom and adapting an existing solution is changing. Whether maintaining all that custom software turns out to be equally cheap is another question.
---
Engineers are no less valuable, but their work is changing. All of the engineers who worked on this system would now rate Claude as better at writing code than them. But our engineers still supplied the invaluable understanding of the high-level inter-system components and the design-level tradeoffs, making informed choices around the problems we wanted to solve and the scale we knew we needed to achieve. Very often, we found that the role of the engineers was to tell Claude āno.ā Agents will sometimes violate core system constraints to satisfy a pattern that should never have been allowed in the first place. This was particularly common in our migration. For example, there were multiple systems that tried to brute force looking up when a change merged to main by linearly reading the commit log instead of asking our code review tool. Anthropic also makes extensive use of adversarial review, and, while this is incredibly effective in many ways, it can also be myopic, so skilled engineers needed to steer Claude away from over-engineering parts of the system in response to hypothetical concerns. For example, one such review was overly defensive about trusting another internal service, which resulted in Claude adding excessive validation and complexity. Our team members still review every change request for the system, but they now examine the intent of the change as opposed to reading every line of code.
I read this and wonder what their agents have committed into the repo that people aren't looking at for _that much size_? Do they have a ton of binary assets in there or something??
There's going to be an ugly phase where everybody FOMOs into thinking they need to do the same, only to realize that they're not Anthropic/Google/etc.
Who's beating Anthropic here lol
> 25 million CI checkouts
833 checkouts for every merge? That's quite a few. Even accounting for multiple environments at different stages of production readiness, that's pretty weird.
Here are the public model releases from Anthropic this year:
5th Feb 2026: Claude Opus 4.6
17th Feb 2026: Claude Sonnet 4.6
7th Apr 2026: Claude Mythos Preview
16th Apr 2026: Claude Opus 4.7
28th May 2026: Claude Opus 4.8
9th Jun 2026: Claude Fable 5
9th Jun 2026: Claude Mythos 5
30th Jun 2026: Claude Sonnet 5
24th Jul 2026: Claude Opus 5
1st Sep 2026: Claude Fable 5.1
1st Sep 2026: Claude Mythos 5.1
22nd Sep 2026: Claude Opus 5.5
28th Sep 2026: Claude Sonnet 5.5
7th Oct 2026: Claude Haiku 5.5
I would guess that any public model has been available internally for at least a month before that date, maybe longer.
My best guess is that the uptick in code changes corresponds to the availability of Mythos/Fable-class models inside of Anthropic.
If Anthropic has the best coding model, harness and the best software development methodologies and systems. And then, they scale all that up, with no constrains on their tokens. On top of that, they have access to all kind of ideas their models getting trained on when millions are chatting with it - then in theory they can build any software better than anyone out there. And same goes to science and math..
Their Website/CLI seems keeping the same? What did they do?
I think this is the wrong way to solve problems and goes the wrong way toward solving real problems.
What then?
Maybe I should just bite the bullet and use a different VCS system entirely once I get there, which has most of the Anthropic system's properties (including being much faster) but doesn't need to be horizontally scalable.
Now Anthropic has migrated from Git to Scion. It's a good sign they are on the right track.
Most companies don't use mono-repos like this. While there are some really compelling advantages, the downsides tend to stack up as a company's codevbase grows.
It's mainly just startups and huge companies. For small companies git works fine for mono-repos and large companies can build their own complex code management pipelines for mono-repos.
I have been questioning whether agentic development changes the mono-repo vs multi-repo question.
I just wonder what is the churn rate (and the cost per LOC) here.
From what I recall, Google's repo is closer to 80 terabytes.
This seems like an absurd amount of data / traffic per day. Is there really that much to code that needs changing on a daily basis?
We are offering a bridge to git via a compatibility layer: https://ersc.io/blog/what-comes-after-git
This gives you the ability to ease your transition, as you don't need to swap services and move to something non-git at once. Moving an enterprises' source control is a massive lift, and so building a bridge from here to there is a big focus of what we're doing.
1: That is, if you have a large team + agents working in a monorepo, you will hit git's scaling limits, and need to move to something else. Google and Meta hit this point over a decade ago, but now lots of other companies are too.
> monorepo
Well, there's your problem.
This is absurd, and I expect is a cause of the fact that they seem to use git history as a message queue source of truth between agents. They can do whatever works for them, fine, but I've noticed this has leaked into their training somehow, my agent, despite very strict rules on how to use my git workflow, will do absolutely nonsensical and weird (from a human who has used git for a long time perspective) stuff with git that can only be a byproduct of how they use it.
Hopefully this fixes that, get your weird workflow out of mine.
> So we asked ourselves: if we designed source control with the fact that agents are always online in mind from the start, what would it look like?
And
> In Scion, every commit, rebase, or merge happens on the server
Iāll pass. A big part of the theses of git and, even more so, jujitsu, is that you can build up a history entirely locally, making it reasonable to make tiny commits locally and then rework them, push them, or throw them away.
Sure, in Anthropicās world, itās completely impossible to do anything without access to the inference API, so why not add another mandatory API? But I think this model sucks even in a pervasive AI world, and I donāt want to dig in deeper.
This also sounds like a great way to have yet more security problems where your swarm of agents start cooperating more than you want them to.
By all means, built a better system than git. In many respects, git sucks, and git upstream is not solving the major problems. But donāt lose the actual benefits of git!
For that matter, Iāve seen the crap that the latest models will commit. Why on Earth should oneās VCS force the per-agent-instance system to upload the crap before even contemplating locally whether itās a good idea? Maybe this is why Anthropic has so many commits per day.
(Am I the only one who feels like most of the big AI labs are rapidly gradient-descending toward something that is nowhere near a globally optimal stack?)
P.S. Only 30k commits per day? Thatās not actually very many per employee. Iām suspicious that Anthropicās real issues are that too many of them are giant cross-cutting messes and that git clone is too slow.
And for this they buy disk? Maybe this is irresponsible use of technology.
E energy necessary to integrate changes
m mass of the monorepo
c rate of commits
The amount of photo storage being used is impressive. Automatic backup to icloud and server provisioning to reliably store it all is also impressive. I just wonder if it makes sense.
Oh, it's not available, what a pity. Why write a whole article about a thing no one can have? Maybe the code isn't so great after all? Or it's all just made up and doesn't even exist?
Anyway, this is yet another pointless ad from Anthropic trying to shill their AI on us.
Practically speaking, itās difficult to offer Scion in a form that would be broadly useful to other organizations. It diverges significantly from git, meaning companies would need to migrate every piece of the development lifecycle, and it requires its own code review, code search, and CI. Our internal versions of these components are highly bespoke, so they would be difficult for other organizations to adopt, and we worry that attempting to build a public version that tries to please everyone wouldnāt actually be very useful.
k, thx for sharing. Are you guys sure this isn't a Google blog post? They would be proud!It's also hard to believe all these non-Git alternatives aren't merely an attempt to capture developers into a new proprietary solution.
This could have been done as an open-source fork of Git.
It's designed for container running, but ArtifactFS has some similarity in principle to me. The on-demand materialization. Quote: ArtifactFS is a Git-backed filesystem daemon (FUSE driver) in Go that mounts repositories as normal working trees while avoiding eager blob downloads. https://github.com/cloudflare/artifact-fs
I threw together my own jujutsu powered FUSE subtree project just the other night. This was basically one-shotted, just to see how it does it, to learn more, not really intended for use. But obvious overlap. https://github.com/rektide/jj-workfuse
* This system leverages a metadata log pointing to immutable blobs in object storage. The metadata log can give high write throughput and keeps the control plane consistent. Object storage supports high throughput of the heavier data objects. Overall, it's an excellent choice given VCS systems can typically tolerate the latencies associated with object storage, as is the case with many operations-focused systems for businesses.
* The server/upstream gains scalability by reducing the amount of data it processes when accepting a commit/performing standard VCS operations. No packfile, no file i/o to construct a consistent copy of the repo, just a (usually) small metadata record that can be validated/committed quickly. The client primarily gains scalability along a similar axis (reduce amount of extraneous work for a given goal) where it can request just the pieces of a repository it needs to do its job, when it needs them. For example, agents can begin reading files from a checkout eagerly before the whole updated version of the repo is finished being downloaded, then during a compile step, updates can be fetched only for the module being compiled, etc.
* The system seems well designed from the perspective of considering "when exactly do I need a thing to be done." Git tends to bundle more of this work into synchronous operations when a request comes in, which ensures the system is consistent at the end and doesn't require a persistent process, but is costly. This system makes intentionally different decisions about "what work happens, when" so that the minimum amount of work is done to confirm a write and allow for the state of the repo to be reconstructed consistently, but it defers certain things that are useful but don't need to block a write. The discussion of "derived data" mentions that certain useful structures can be computed in a deferred way, which supports doing work around the time it's needed.
* A large part of the nice operational features of the system is downstream of making core data structures immutable. File contents are immutable, content-addressed blobs. WAL entries are immutable records. As mentioned in the article, this supports excellent caching, and simplicity in reasoning about history/updates. Additionally, the derivation of other useful data can end up as immutable outputs (themselves stored as pointers to objects, if scalability requires it).
* The decision to develop a custom VCS supported a number of other integrations into "agent-first" software development systems. The system makes it easy to enforce access control on specific directories, nice if you want to have agents explicitly denied the ability to edit something. You can also get a sense that it supports some valuable features in the associated code review system, for things like pointing a human at the most important changes in a changeset. You could imagine in the future, some associated highly flexible "owner" rules/notifications where a human can specify "I want to be notified if X system has a significant design change" and that intent can be recorded/managed in a more useful way than "add me as a reviewer if files in /abc directory change."
* I do wonder if content-addressing of the files is broken into chunks or not. Doing "one blob per file" is nice and simple, so maybe that's how it starts. But you also duplicate a lot of data, but maybe the dataset is still trivially small in the scheme of things, and the operational concerns win out. And if files are small anyways, you'd want to avoid splitting them into tiny chunks bc that'd affect the bandwidth you can achieve when downloading (spend more of your time on per-object/request overhead rather than actually transferring data).
Very nice piece of engineering work. I was also impressed by some other parts that received a little less discussion in the post (understandably), but would love to hear more in a future post:
* the testing approach of driving tons of system interactions in a hermetic way is really exciting to see. Feels like agents should help us write really excellent testing infrastructure, and that this would itself be important for gaining confidence to deploy without requiring a read of every line committed.
* Similarly, the mention of the "superfan" agents that closely monitor very specific parts of the system seemed very cool and would help a lot with deploying code you haven't necessarily read closely but seems reasonable architecturally. Though it'd still ofc be best to catch issues prior to merge/release, having extremely good monitoring/rollback capabilities would mitigate a lot of the downsides of high-speed, agent-based software development.
What an unexpected candor! Of course you plagiarize prior art.
No sane person is going to put their sources in this slopware Scion monster.
Can please we all converge to 1 f'ng solution.
how does it compare to Epic's https://lore.org/ ?
I appreciate it's different than git, that type of argument for monorepos is not new.
Agents coding is akin to having a very large developer team, who may or may not be online 24/7.
Anthropic is calling it very specialized, and it's a little baffling.. maybe I'm missing something.
There's no way software development is the same anymore. The previous ceremonies and structures and seats at the table at each stage were human based, and now that's a different make up to the point that different steps and processes are needed.
There's nothing more fundamental to growing software development other than making tools like this available.
My hope is a vibe coded internal mono repo management tool can aim for a v2 with multi-tenancy built in.
Was anyone else looking forward to feature flags as a part of a post like this?
Another issue that I believe Anthropic here didn't state is that git performance _severely_ get degraded when you have hundred of commits landing in parallel. Hence why I fell back to the daily commit approach. You get less granularity but you get stability.
Weāre completely rethinking our priors. This project prompted us to take a very hard look at other workflows designed for human scale. A proper merge queue, for example, doesnāt make much sense at agent scale. What does CI even mean when hundreds or even thousands of changes can merge while tests are running? We havenāt been able to effectively replicate the industry standard of a global repository lock on the merge queue, so instead, weāve invested in post-merge and pre-deploy automated testing. This does mean that concurrent changes can conflict, but Claude can typically fix these conflicts faster than humans can even identify them.
I see, that's why https://status.claude.com/ looks like a Christmas tree on fire.I honestly just cannot. Just tell me what the workflow is. Give me a video that is Twilio-level simple explanation.