Epic Games announces Lore version control system

559 points - today at 2:30 PM

Source

Comments

throw2ih020 today at 2:47 PM
For context, since a lot of people on HN haven't worked on games - this is not intended to compete with Git for general software development. This is a competitor with Perforce for game development.

Git is fine for text based files like code, but it's really bad at stuff like textures, 3D models, audio files, and other non-text files that game developers need to collaborate on. For example, one artist might need to obtain an exclusive lock on some art assets while editing them, because there is no sane way to merge two artists' async edits.

The SOTA in this area is Perforce (https://www.perforce.com/products/helix-core), a proprietary system. From what my gamedev friends tell me, when Perforce works it's great, but it hits enough snags that you need a tools engineer to manage it and occasionally fix issues manually. Git LFS is an alternative, but my gamedev friends all prefer Perforce especially when working on team projects beyond like 3-4 people.

niek_pas today at 2:46 PM
Just today as I pushed some changes to Github, I was thinking how user-unfriendly Git's UI is:

    Enumerating objects: 5, done.
    Counting objects: 100% (5/5), done.
    Delta compression using up to 10 threads
    Compressing objects: 100% (3/3), done.
    Writing objects: 100% (3/3), 290 bytes | 290.00 KiB/s, done.
    Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
    remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
I know all of these things communicate something to the die-hard Git user, but for most people (even most people using Git, I bet) this is just complete gobbledegook. What the hell is "delta compression"? Why do I care how many threads it's using? What is an 'object' and what does it mean when it's 'local'? What does 'pack-reused' mean?

From the documentation, it looks like Lore does a bit better in this regard:

    Pushing 1 fragment(s)
    Pushed 1 fragment(s), 124.00 bytes
    Pushing a3f8c2d1... to branch main
    Pushed revision 1 -> a3f8c2d1... to branch main
tlahtinen today at 4:10 PM
This is a very promising announcement for Unreal game development specifically. For any other purpose I wouldn't care as much.

Perforce definitely needs a challenger. It is not the incumbent because it is particularily simple to use or administer. Git is actually way simpler when it comes to branching operations for example.

The reasons why p4 is often preferred in gamedev have already been mentioned in other comments: large project support, permissions, file locking and so on. Another key reason p4 is the king for Unreal dev is just how well it's supported inside the engine. Not perfect, but it's the best supported VCS because it's what Epic uses. Even the Git plugin is painfully unfinished, because Epic does not internally use it. So with Lore I expect them to give it first class support. I'd recommend Git a lot more if the support in Unreal was better.

(background; I've been in gamedev for almost two decades now, 2-200 person companies, every kind of engine and version control system. I prefer git where I can use it: for Unreal that means small projects and/or tech savvy team members. Pick the tool that is right for the job and the team.)

ksec today at 4:49 PM
Turns out it is not really new but only open sourced it now. From the FQA.

>Lore, formerly called Unreal Revision Control, is the built-in version control system for UEFN (Unreal Editor for Fortnite), where creators have been using it to version their islands. It is also seeing progressive adoption by internal Epic teams, and is being implemented as the backing store for UEFN’s cook pipeline, where it replaces traditional intermediary storage layers—eliminating redundant file transfers and significantly reducing the time between publishing changes and those changes being playable.

Surprised it is in Rust and not Epic C++ or Verse. I wonder why.

MaikuMori today at 6:14 PM
I've been working on something like this for internal use for the last 2 month or so. Very high overlap, CAS, version chains, branching available in storage, but not implemented (not as high value for us), file locks, chunked storage on S3 api. Focused on assets only with more focus on build/export (think gh actions for blender files) and QA processes.

Interesting to see where this goes.

penciltwirler today at 3:29 PM
The premise is that Git-LFS sucks, so we need to build a new data versioning system (in Rust, from scratch). While I mostly agree with this premise, but there are already lots of existing (mature) data versioning systems with the same tricks under the hood:

- Pachyderm (Go): https://github.com/pachyderm/pachyderm

- XetHub (acquired by HuggingFace): https://huggingface.co/blog/xethub-joins-hf

- LakeFS (Go): https://github.com/treeverse/lakeFS

- Oxen (Rust): https://github.com/Oxen-AI/Oxen

I guess with AI, anyone can vibe code a content-addressed, chunk-level deduped, versioning system in Rust these days...

But jokes aside, Lore seems really cool! What's interesting is the realization that different domains/industries have similar problems, but they don't seem to be cross-polinating. In this case AI and Gaming both need a storage system that can version control large binary files at scale. I think there's lots of opportunities to share ideas here, but perhaps the lack of idea sharing (currently) creates opportunity!

jacobgold today at 2:45 PM
I'd trust this project more if it was named Data.
frollogaston today at 3:40 PM
"Full-surface API" is a feature nobody here has mentioned. Is that a dig at how git intentionally has no linkable library? I saw this earlier https://news.ycombinator.com/item?id=48470604
ibejoeb today at 6:01 PM
> lore stage covers adds, edits, and deletes — you use the same command for all three. Stage a deleted file and Lore records the deletion for the next commit. Moves and renames are tracked too, through a dedicated subcommand: lore stage move <from> <to> records the rename so the file keeps its identity and history across the move instead of registering as a delete plus an add.

Oof. So this isn't compatible with any tools that move or rename files. I can't see how this will be acceptable for real-world use.

akurilin today at 5:30 PM
We had to use Perforce (Helix Core Cloud) at my last game studio, and it is the de facto industry standard that most of your creative staff is already familiar with. The programmers don't love it, but they don't rule the roost in games. It's also the safe, verified default for working with Unreal Engine 5.

It does show its years though. We were one of the first users of the Perforce cloud offering, as we were small and didn't want to self-host ourselves, but it was a bit of a rickety experience. You had to register an Azure account in order to be able to access the service, and you had to ask support to modify things like triggers. Coming from the world of GitHub and other SaaS products, you could tell it was an attempt to retrofit an older model into a new skin.

The Git LFS path has some unofficial support as well, but you are on your own when things go poorly. Epic doesn't provide much help there.

Competition in this space is welcome, especially if they're planning to make it fully officially supported by the Engine.

I wrote about why merging files isn't as common in the world of game dev for folks coming from the world of text: https://www.kuril.in/blog/why-game-devs-dont-merge-files/

bentt today at 5:35 PM
There was and still is a very nice competitor in this space called PlasticSCM. They were bought by Unity a few years ago. Unity has not been a good steward. They should have done what Epic is doing and open sourced it. But instead they chose to give it P&L responsibility. Curious what it's contributing to their financials.
iceweaselfan44 today at 3:16 PM
>fully open source >look inside >Lore Desktop Client is available as binaries only, download the installer for your platform here:
samgranieri today at 6:02 PM
I may not be the first to bring this up, but seeing the name of this program reminds me of Data’s evil brother
bel8 today at 2:38 PM
repo: https://github.com/EpicGames/lore

Looks very git-ish. But probably better equipped for large binary files.

    echo "Hello, Lore" > hello.txt

    lore stage hello.txt

    lore status --scan

    lore commit "Initial revision"

    lore push
gavinhoward today at 4:14 PM
As someone who has thought a lot about VCS design [1] [2], the chunking approach is the wrong one and will still waste space.

[1]: https://gavinhoward.com/uploads/designs/yore.md

[2]: My WIP VCS has been named Yore for at least two years; I did not copy Lore's name.

the_duke today at 5:57 PM
Does this support using S3 as the backing store?

That would be very powerful for various use cases.

glouwbug today at 2:41 PM
I’ve always wanted a git with five commands, and maybe with AST based diffing
pkasting today at 4:34 PM
The idea sounds good, even if Epic's recent track record of tools is not inspiring. But the commit messages etc. are very clearly products of vibe-coding. And version control is not the situation where "works 97% of the time" is a good-enough bar.

Passing for now.

swiftcoder today at 5:15 PM
I don't see a workflow for locking assets while they are under modification. This is kind of important for assets? Since we don't really have great merge workflows for meshes/animations/sounds/etc.

I also don't see any sort of GUI client? So the whole art team is going to have to get up close and personal with the CLI

kardianos today at 3:30 PM
I like everything I've read on this site so for, for it is also something I've been wanting.

If the roadmap's "Web client and code review tools" could replace gerrit for me, this would be a easy switch.

Moreover, it looks like they designed both the mutable store and immutable store to be able to easily store their state directly on an s3 like system.

There are a number of features that would greatly speed up CI/CD system operations I belive.

jbverschoor today at 5:35 PM
Too bad it does not support fully local/detached (without server). It would be nice to be able to run it similar to a local repo without remote
speps today at 2:41 PM
They’ve been dabbling in this space within Unreal Engine for a few years. Perforce is the de facto standard in AAA studios from my experience, curious to see what’s going to happen to them.
wky today at 3:48 PM
The link to Architectural Decision Records is empty, but they're present in the repo to look at[0]. Curiously the decision with the most deciders is the implementation of JavaScript bindings[1].

[0] https://github.com/EpicGames/lore/tree/main/docs/developing/...

[1] https://github.com/EpicGames/lore/blob/main/docs/developing/...

wxw today at 5:02 PM
https://epicgames.github.io/lore/explanation/system-design/#...

Helpful page that gets closer to the details

eblanshey today at 3:32 PM
This looks very cool! I maintain a FreeCAD workbench for 3d model version control[0], and it currently uses git as the VCS, simply because that's what I was already using. Thinking long-term, I see it eventually morphing into a broader PDM (Product Data Management) system, perhaps even PLM (Product Lifecycle Management). Lore has a lot of the requirements already built-in, like centralized locking (better than SVN), and it's better suited for for binary files. I implemented the git backend as a protocol/port so it'd be pretty easy to swap it out. I'll be watching Lore closely.

[0] https://github.com/eblanshey/HistoryWorkbench

aayushprime today at 4:54 PM
I must say lore is an awesome name for a version control system. Much better than git in that regards.
BoggleOhYeah today at 3:00 PM
It’s great to finally see a possible alternative to Perforce.
wilg today at 6:10 PM
I've tried very hard to use Git (with LFS) in both Unity and Unreal and its just been a total disaster of problems for me and the team. Hopefully this can help.
goolz today at 5:35 PM
Fantastic. Every time I mix git and games I end up almost having an aneurysm.
Lucasoato today at 3:06 PM
> Git’s content-addressed revision graph is excellent, but it treats binary files as second-class citizens—large files require bolted-on LFS rather than first-class chunked storage, sparse checkouts have sharp edges in offline use, and there is no native multi-tenant isolation.

I'm trying to figure out what Lore can accomplish that git+LFS can't. I've read about big binaries chunking, native interface and permission, is there anything else? Weren't those problems already solvable in the git+LFS ecosystem?

MarvinYork today at 5:43 PM
I wonder how many websites ChatGPT did by now...
nyxtom today at 3:49 PM
I came here hoping Epic Games somehow had launched a reliable alternative to GitHub, but saw their code is hosted on GitHub
hparadiz today at 4:24 PM
As long as Epic Games is anti Linux I will never use any Epic Games product.

Count on it.

rustyhancock today at 4:46 PM
Guess this is announced as part of epics state of unreal and if so this is already off to an amazing start even if this is all there is!
deleted today at 4:54 PM
noopprod today at 4:10 PM
Kind of funny that it's on GitHub no hate.

Missed opportunity for Lorehub.

armchairhacker today at 5:06 PM
Game engine, programming language, VCS
will Epic launch a brand new OS?
yablak today at 3:51 PM
does anyone have a proper comparison of binary control systems like lore, xet, etc? i'd love to see how it handles mixed case workloads.
advisedwang today at 3:55 PM
Nice, this seems sort of like Git-but-for-giant-monorepos. That has been a gap in the opensource VCS market
interpol_p today at 2:55 PM
Their docs seem entirely LLM written. It seems especially obvious in the FAQ. While I'm not against using LLMs for writing assistance, they've left a lot of the unnecessary language and typical stylistic choices in there, which erodes my trust in the project a bit. Perhaps it's a very good game-oriented version control system, but the lack of human attention on the docs makes me wonder how much they care
bachittle today at 2:37 PM
this looks cool for game development, because using Git for projects in Unity and Unreal Engine definitely has it's issues. I'm personally not a fan of Git LFS, especially since GitHub charges you to use it (which makes sense, binaries and assets are big, code is small, relatively speaking).
nicebyte today at 6:03 PM
"Github Repo" "Find Us on GitHub"

????

Surac today at 4:48 PM
Isnt that what SVN is good for?
20k today at 2:49 PM
The incredible laggyness of that website does not inspire confidence. Much of the text selection is also broken, and chrome consumes nearly a full core trying to render.. something?

Its remarkable that anyone thought this website was fit for release, and it gives off strong slop vibes

I also have absolutely zero trust in a product like version control being provided by a for-profit company. It seems like a terrible idea to tie your software stack to Epic Games of all people, given their track record

ryukoposting today at 2:43 PM
Hosted onn GitHub. Heh.
boredatoms today at 4:43 PM
Would lore be good for a tech company monorepo?
ex-aws-dude today at 3:36 PM
Interested in this as perforce is pretty terrible a lot of the time

It’s like anything you do has to talk to the server

Even something as simple as diffing a file will just hang if there are server issues

adamnemecek today at 2:48 PM
This looks really good. I have been using git to store some PDF (tens of GBs) and git is really not well suited for this. No GFS is not a solution.
gbraad today at 2:38 PM
What makes lore better or worth considering... when svn and git never failed me...
applfanboysbgon today at 4:12 PM
I'm in the market for a better VCS designed for gamedev-specific concerns, but reading the system design doc, it's LLM-generated. Not exactly confidence inspiring.
dankobgd today at 2:52 PM
never trust epic
UltimateEdge today at 2:36 PM
Ahah, the second and third links on the page are to GitHub
LoganDark today at 2:43 PM
Interesting to note that this does not seem like a DVCS in the traditional sense because it depends on coordinating with a central server where all repositories will be hosted. I can't tell if servers can pull/push from eachother.
moralestapia today at 2:38 PM
What a waste of a phenomenal domain name.
gonomodagast today at 3:53 PM
Why not just use Alienbrain?
jordand today at 4:51 PM
[dead]
sehw today at 5:56 PM
[dead]
wullbyte today at 2:56 PM
[dead]
throwaway613746 today at 3:46 PM
[dead]
headwayoldest today at 2:35 PM
How long before Epic starts giving away other software and suing git to support lore?
dwroberts today at 3:20 PM
This is just going to become another way to lock developers into UE. Then they will start charging for licenses, same as Unity did for its versioning feature. It might be open source but that doesn’t stop the commercial use of it being charged for.
gregschoeninger today at 5:11 PM
We're also working on an open source large asset versioning tool called "oxen" - https://github.com/Oxen-AI/Oxen

Would love any feedback on it or contributions if people are interested :)