Epic's Lore Version Control System

Lore is a next-generation open-source version control system developed by Epic Games to handle massive projects involving both code and large binary assets. Designed for extreme scalability, it features a centralized architecture that allows for offline work while maintaining a single, cryptographically verifiable source of truth. The system is built on a content-addressed storage layer that utilizes fragment-level deduplication to efficiently manage multi-gigabyte files and millions of revisions. Lore prioritizes a "binary-first" philosophy, treating all data as opaque byte streams and layering text-specific features on top of these core storage primitives. It offers an API-first design with multiple language SDKs, allowing developers to integrate its storage and versioning capabilities directly into custom tools and pipelines. Released under the MIT license, the project aims to establish an open standard for revision control that serves the demanding needs of modern game development and enterprise-scale software engineering.

Note:  This podcast was AI-generated, and sometimes AI can make mistakes.  Please double-check any critical information.

Sponsored by Embersilk LLC

Transcript

SPEAKER_01 0:00

You know, I had this moment of absolute uh just pure agony last week.

SPEAKER_00 0:04

Uh-oh. What happened?

SPEAKER_01 0:06

Well, I was trying to send this massive 4K video file to a friend of mine. It was um an AI automation demo, actually.

SPEAKER_00 0:13

Oh, those files are always gigantic.

SPEAKER_01 0:15

Right. And hey, speaking of AI, a quick thanks to Embersilk for sponsoring our deep dives. Like if you're looking for help with AI training, automation, or you know, just figuring out where agents can make the biggest impact for your business, you really should check out Embersilk.com.

SPEAKER_00 0:30

For sure. They do great work.

SPEAKER_01 0:32

Yeah, they really do. But anyway, so I'm watching this file transfer progress bar, and it is just crawling along for over an hour.

SPEAKER_00 0:40

Oh, I can already see where this is going.

SPEAKER_01 0:41

Yep. It freezes right at 99%, just sits there mocking me. I mean, I eventually just gave up, put the file on a physical thumb drive and dropped it in the mail.

SPEAKER_00 0:50

Ah, the classic sneaker net approach. I mean, it is kind of humbling, right? Yeah. In this era of gigabit fiber, sometimes the fastest bandwidth is still literally a postal truck.

SPEAKER_01 1:00

Exactly. And, you know, that friction, like moving massive chunks of data, that is the daily reality for developers and artists who are building these modern digital worlds.

SPEAKER_00 1:09

Oh, absolutely.

SPEAKER_01 1:10

So today we are taking a deep dive into how Epic Games actually solved this. They built this new open source version control system called lore.

SPEAKER_00 1:20

And it is such a vital solution because the scale of modern game development, or well, really any massive software project, it's just outgrown the traditional tools.

SPEAKER_01 1:30

Right.

SPEAKER_00 1:30

Like anyone listening who works in software knows Git is the undisputed king of version control for text. Sure. But the second you throw massive binary files at it, you know, like high-res 3D textures or compiled audio, it just it hits a brick wall.

SPEAKER_01 1:45

Because Git tries to read everything line by line to track the diffs, right? Exactly. And if you have a 10 gigabyte 3D environment, there are no text lines.

SPEAKER_00 1:51

None at all.

SPEAKER_01 1:52

So if an artist tweaks a single pixel, Git panics and essentially just saves a completely new 10 gigabyte copy of the file.

SPEAKER_00 1:59

Which is wild. And doing that across a team of like hundreds of people making thousands of changes a day, it brings repositories to a grinding halt.

SPEAKER_01 2:08

Just a total nightmare.

SPEAKER_00 2:09

Yeah. So Lore fixes this by shifting to what they call a binary first architecture.

SPEAKER_01 2:15

Okay, so what does that actually mean?

SPEAKER_00 2:16

It means it treats every file as raw, opaque bytes. It doesn't care about text, and it makes absolutely no assumptions about line endings.

SPEAKER_01 2:26

It's like um switching from tracking spelling changes in a dictionary to managing a port full of shipping containers.

SPEAKER_00 2:32

That's a great way to put it.

SPEAKER_01 2:34

Like it doesn't matter if the container is holding teddy bears or car engines, the system just efficiently moves the identical steel boxes.

SPEAKER_00 2:40

Right, exactly.

SPEAKER_01 2:41

But wait, that brings up a mechanical issue, doesn't it?

SPEAKER_00 2:43

How so?

SPEAKER_01 2:44

Well, if I change one single pixel in a 10 gigabyte texture, there is no way Lore is sending the whole 10 gigs again.

SPEAKER_00 2:50

Right.

SPEAKER_01 2:51

So how does it even find a single changed pixel in a sea of opaque bytes? Like how does it isolate that?

SPEAKER_00 2:57

So that is where the engineering gets really, really clever. They use this mechanism called fast CDC.

SPEAKER_01 3:02

Fast CDC.

SPEAKER_00 3:03

Yeah, it stands for content-defined chunking. Instead of looking for uh fixed block sizes or lines of code, the algorithm scans the actual byte patterns. Okay. It scans them to find natural boundaries in the data itself. So it breaks that massive 10 gigabyte file down into much smaller deduplicated fragments.

SPEAKER_01 3:21

Oh, wow. So when the artist changes that one pixel, FastCDC recalculates the boundaries, realizes like 99.9% of the chumps are identical to what's already on the server.

SPEAKER_00 3:33

Yep.

SPEAKER_01 3:33

And then it only transfers the specific new block of bytes that changed.

SPEAKER_00 3:37

Spot on. The network transfer drops from gigabytes to maybe just a few kilobytes.

SPEAKER_01 3:42

That is incredible. And because you're only moving these tiny chunks, I imagine it completely changes how you access the project locally.

SPEAKER_00 3:49

It changes everything. Usually when you clone a large repository, you have to download every historical version of every file before you can even get to work.

SPEAKER_01 3:56

Aaron Powell Which is awful when your repository is terabytes in size.

SPEAKER_00 3:59

Exactly. But LoRa bypasses this by being sparse by default. It uses lazy fetching.

SPEAKER_01 4:04

Okay, lazy fetching.

SPEAKER_00 4:05

Yeah. So when you connect to the project, your local machine only pulls down the specific files you are actively opening or modifying.

SPEAKER_01 4:12

So the rest of the project is effectively just a mirage until you interact with it.

SPEAKER_00 4:16

Precisely. Everything else stays safely on the remote server. You save massive amounts of local storage, and you eliminate the hours of waiting just to sync a project workspace.

SPEAKER_01 4:25

Aaron Powell The concrete impact of this has got to be huge. I mean, it means a global team of, what, 500 artists and developers can collaborate on a single massive game build without every single person needing server farm levels of local storage.

SPEAKER_00 4:39

Yeah, and the best part, Epic Games released this under an MIT license.

SPEAKER_01 4:44

No way.

SPEAKER_00 4:45

Yes. It is a highly permissive open source drop. They solved a massive technical bottleneck for their own Unreal Engine projects and then just handed the architecture to the community.

SPEAKER_01 4:54

That perfectly fuels the intellectually curious spirit of software development. It proves that when we open source our most powerful tools, human progress just accelerates.

SPEAKER_00 5:03

Absolutely. It undeniably lowers the barrier to entry for building incredibly complex digital environments.

SPEAKER_01 5:09

Which leaves you with this really cool thought, I think.

SPEAKER_00 5:12

What's that?

SPEAKER_01 5:12

Well, if we can now perfectly streamline the architecture to cooperatively build massive virtual worlds, you know, down to the individual byte. Right. What other massive data heavy global human endeavors could this infrastructure unlock for us next?

SPEAKER_00 5:30

That is a fantastic problem for the next generation of developers to tackle.

SPEAKER_01 5:33

It really is. Well, if you enjoyed this deep dive, please subscribe to the show. Hey, leave us a five star review if you can. It really does help get the word out. Thanks for tuning in.