Big Intelligence on Tiny Chips

In this episode, we unpack how engineers fit a 28.9M-parameter language model into an $8 ESP32-S3. By using per-layer embeddings and moving most data to flash, the active compute stays in fast SRAM, enabling offline AI at the edge. We explore what it can (and can't) do today—short, simple stories rather than complex instructions—and why this matters for private, decentralized intelligence on everyday devices.

Note:  This podcast was AI-generated, and sometimes AI can make mistakes.  Please double-check any critical information.

Sponsored by Embersilk LLC

Transcript

SPEAKER_01 0:00

I was looking at this uh cheap little electronic gadget the other day. One of those eight dollar toys you pick up at a checkout counter, you know?

SPEAKER_00 0:08

Oh, yeah. The classic impulse buy.

SPEAKER_01 0:10

Exactly. And I was just marveling at the basic circuitry inside. It's wild to think about what these tiny boards can do now.

SPEAKER_00 0:19

Right. They pack a surprising amount of tech.

SPEAKER_01 0:21

But then I read through the stack of technical notes and articles you sent over for our deep dive today, and man, it completely reframed my perspective.

SPEAKER_00 0:29

It is definitely a massive paradigm show.

SPEAKER_01 0:31

Well, like a developer just successfully ran a 28.9 million parameter large language model, completely offline, on a tiny $8 microcontroller, the ESP32S3.

SPEAKER_00 0:44

Yeah, which is just staggering when you think about the physical hardware limits.

SPEAKER_01 0:47

Right. And our mission today is to figure out how developers are basically breaking the rules of silicon to bring AI to the edge. But real quick, this kind of complex hardware and software optimization is exactly what our sponsor, Embersilk, tackles.

SPEAKER_00 1:00

Oh, right, yeah.

SPEAKER_01 1:01

So if you need help with AI training, automation, or uncovering where AI agents could make the most impact for your business or personal life, check out Embersilk.com.

SPEAKER_00 1:10

They do really great work. So looking at the scale of this achievement, you know, before this project, the absolute limit for MIDILES on a microcontroller like this was around 260,000 parameters.

SPEAKER_01 1:23

Wow. So we are talking about something roughly a hundred times larger here.

SPEAKER_00 1:26

Exactly. A hundred times. It means fundamentally changing how the software interacts with the physical chip.

SPEAKER_01 1:33

And to understand how they pulled off that architecture, I mean, we really have to look at the memory bottlenecks. We're talking about fitting a 14.9 megabyte model onto a chip with only 512 kilobytes of fast SRAM memory.

SPEAKER_00 1:47

Right. The math just doesn't work if you followed traditional architecture.

SPEAKER_01 1:49

Yeah, it really doesn't.

SPEAKER_00 1:50

Normally an AI model demands that every single parameter be instantly accessible in that fast memory.

SPEAKER_01 1:56

So I was thinking about this, and tell me if this analogy works. Think of the fast SRAM as a chef's cutting board.

SPEAKER_00 2:01

Okay, I like that.

SPEAKER_01 2:02

Like it's super fast to work on, right? But it's incredibly small. And the chip's flash memory is like a deep freeze in the basement. It's huge, but painfully slow to retrieve things from.

SPEAKER_00 2:15

Yeah, that is a perfect way to look at it.

SPEAKER_01 2:17

So traditional models demand that every single ingredient be on the cutting board at all times.

SPEAKER_00 2:21

Right. Which is why developers usually stay stuck with tiny models. But the developer here used a brilliant workaround. It was actually inspired by Google's Gemma models.

SPEAKER_01 2:31

Oh, really?

SPEAKER_00 2:32

Yeah. Specifically a technique called per layer embeddings.

SPEAKER_01 2:35

Okay, so if they aren't loading the whole model onto the cutting board at once, they must be partitioning it, right?

SPEAKER_00 2:40

Exactly. They realize that in a language model, a huge chunk of the parameters just sit in an embedding table.

SPEAKER_01 2:47

So they keep the active computation in the fast SRAM and just call the reference data from the slower flash memory?

SPEAKER_00 2:54

Precisely. When an LLM processes a language, each token or word maps to a specific vector in that table.

SPEAKER_01 3:01

Got it.

SPEAKER_00 3:02

That means for any given word, like 99% of the embedding table is entirely irrelevant.

SPEAKER_01 3:07

Oh, I see. So you only fetch the specific ingredients you need for that exact recipe.

SPEAKER_00 3:12

You got it. They left about 25 million of those parameters down in the deep freeze. In the slower flash memory.

SPEAKER_01 3:19

That is wild.

SPEAKER_00 3:20

And the chip only pulls the exact rows it needs, which is just about 450 bytes per token, up under the fast SRAM cutting board.

SPEAKER_01 3:28

Well, the actual computing engine stays up there running smoothly.

SPEAKER_00 3:32

Exactly. Because they aren't loading the whole massive table, this giant model costs almost nothing to run. They're getting speeds of about 9.5 tokens per second.

SPEAKER_01 3:42

Quite completely locally.

SPEAKER_00 3:43

Completely local.

SPEAKER_01 3:44

Okay, well I have to push back for a second to keep us grounded here.

SPEAKER_00 3:47

Sure, go ahead.

SPEAKER_01 3:47

If I slap this $8 chip into my smart thermostat today, is it going to start drafting my emails and answering obscure trivia questions?

SPEAKER_00 3:56

Well, we definitely need a reality check on the actual output.

SPEAKER_01 3:59

Right.

SPEAKER_00 3:59

This specific model was trained on the Tiny Stories data set.

SPEAKER_01 4:02

Okay, so what does that mean practically?

SPEAKER_00 4:04

It generates short, simple, and surprisingly coherent stories, but it cannot follow complex instructions. It won't write your code.

SPEAKER_01 4:13

And it definitely doesn't know factual trivia.

SPEAKER_00 4:15

Right. So your thermostat won't be doing your taxes anytime soon.

SPEAKER_01 4:19

Good to know. But I guess the true triumph here isn't really the AI's intellect, is it?

SPEAKER_00 4:23

Not at all. The breakthrough is purely architectural.

SPEAKER_01 4:27

It's a proof of concept.

SPEAKER_00 4:28

Yes. That you can fit a model of this magnitude onto a tiny, incredibly cheap piece of silicon using highly efficient memory management.

SPEAKER_01 4:38

And that is what makes this so incredibly exciting for the future of edge computing.

SPEAKER_00 4:42

It really is. It completely democratizes hardware intelligence.

SPEAKER_01 4:45

Right. We're moving toward a world of cheap, entirely private, offline AI, seamlessly embedded into the physical objects around us without needing a Wi-Fi connection or a massive server farm.

SPEAKER_00 4:57

The potential for positive, decentralized innovation here is massive. We are talking about bringing smart adaptive technology to remote areas, medical devices, you know, everyday tools that genuinely improve human life.

SPEAKER_01 5:10

It's such an inspiring outlook. So if you enjoyed this deep dive into the source material today, please subscribe to the show wherever you listen.

SPEAKER_00 5:17

Yeah, and hey, leave us a five-star review if you can.

SPEAKER_01 5:20

It really does help get the word out. Thanks for tuning in.

SPEAKER_00 5:22

Always a pleasure.

SPEAKER_01 5:23

So I'll leave you with this to ponder. If simple $8 edge chips can hold language models securely offline today using clever memory tricks, what happens to the multi billion dollar cloud computing monopolies tomorrow when our everyday devices no longer need to phone home? The decentralized future of intelligence might just fit right in your pocket.