Why DeepSeek Gave Its Model a Lookup Table (Engram)

Same size, same compute0:00

Yo guys, what is up. David here. Here is an interesting paper done by DeepSeek that I thought I should be covering. DeepSeek's V4.1-Flash has a lookup table inside it, with 196 billion learned numbers of it. As a user, you might not think this is any special, but their engineering might very well be the reason you get better accuracy at better cost. For those curious about their engineering and how they did it, I've took my time reading it for you.

Since this is a paper about DeepSeek, let us use DeepSeek-chan's asset to help out with the presentation. So here's what its paper found. Take two models of exactly the same size, trained on the same text, doing the same amount of work for every word they write. One scores 50.9 on a set of hard reasoning puzzles. The other scores 55.9, and the only difference is the lookup table.

On plain trivia it gains just 0.2, which is strange if you expect a lookup table to improve in recall problems. We will get to that later. That table comes from a paper called Conditional Memory via Scalable Lookup. It's by Xin Cheng, Rui Tian and colleagues, with Wenfeng Liang on the author list, from Peking University and DeepSeek, and it came out in January 2026, months before the model that uses it.

The thing it builds is called Engram.

Parameters, layers and experts1:11

First, let's talk about what a language model is. It reads text and predicts what comes next, one small chunk at a time. To do that it passes the text through a stack of layers, and every layer is one more round of calculation. What it knows is stored in parameters, which are just the numbers it learned during training. That's called parametric knowledge, and it's different from searching the web while it answers.

How many layers there are is called the model's depth. Depth matters because each layer works on what the one before it produced, so a problem with many steps needs many layers to finish. Now say you mention Diana, Princess of Wales. Four words, one person. The paper reproduces an example from earlier research, where you can watch one model do it a layer at a time. After two layers it only has Wales, a country.

By layers 4 and 5 it has a title, and at layer 6 it finally has Diana. The authors' complaint is that nothing in the model just looks it up. They say that lacking a lookup is forcing models to inefficiently simulate retrieval through computation. In other words. DeepSeek's boss is very unhappy that the model needs so much compute and money to name Princess Diana and he wants to do it cheaper.

To see what they changed we need one more idea. A model gets better the more it stores, and gets more expensive the more it uses. A Mixture of Experts, or MoE, is the trick for getting one without the other. You split the model into lots of small specialist blocks, called experts, and for each word only a few of them wake up. This model stores 26.7 billion parameters and uses only 3.8 billion of them for a word.

That gap has a name here, the spare budget. It's everything the model stores and the word doesn't use, and until now all of it went to experts. The authors ask what happens if some of it goes somewhere else.

Compute vs memory2:46

Think about 7 times 8. One student counts it out on her fingers, every single time. The other learned her times table at school, so she just knows it. Both get 56, but the first one spent her working on it. That working is what a model spends its early layers on, for a name or a phrase it has seen a million times. And the multiplication table is what the authors want to give it.

It isn't notes from this conversation, it was learned beforehand and it never changes, and it gives the same answer whatever the question around it is about.

How Engram works3:16

So here's what Engram does to a word. It takes that word and the ones just before it, a short phrase, which is called an n-gram. Yeah, the pronunciation is very close, so most people do not know what you're to when talking. I've wrote down the spelling here. Anyway, it turns the phrase into a row number with a fixed scramble called a hash, and it reads that row, a stored list of numbers about the phrase.

One lookup and no calculation. There are two details that make it work. First, similar spellings are merged before the scramble, capital letters and small ones, accented and plain, and all the kinds of spacing, and that trims the number of different entries by 23%. Second, two different phrases can land on the same row by accident, so each phrase is scrambled several different ways and the results are combined.

And it isn't on every layer. In the big model it sits at two of them, an early one and a middle one. There's a catch with a lookup. The row for Alexander the Great is the same row whether the sentence is about history or about a film. It only sees the last few words, so it knows nothing about the sentence around them. So the model gets a veto.

By the time a word reaches that layer, the model has already read everything before it, and it checks what the table returned against that. If they agree, the memory goes through. If they clash, a small switch called a gate closes it down to nearly nothing. The places the gate opens, in the samples the authors chose to show, are names like Alexander the Great and the Milky Way, and set phrases like By the way.

There's a name for what the experts and the table have in common. In a Mixture of Experts, which experts run depends on the word, so the paper calls that conditional computation. In Engram, which row you read depends on the phrase, so they call it conditional memory. Either way, most of the model sits idle for any given word, and that's what keeps it cheap. So the paper's claim is that the spare budget can be spent two ways, on more specialist blocks to calculate with, or on more rows to look things up in.

Same compute isn't cheaper5:01

Same size and same compute can sound like the model got cheaper to run. It didn't. The total stays the same and the part that does the work stays the same, so the score goes up without the work going down. It's like DeepSeek has found a "free hack" to squeeze out more performance for the same compute. It is not free though, as we will properly note later. The authors note a slowdown in token throughput of 3% under specific model and GPU specification.

Unless you're the kind of vibe coder bottlenecked by time, a 3% slowdown might not be a big deal.

How much memory?5:29

So how much of the spare budget should go to the table? The authors tried it on two small models, moving the budget from experts to table a step at a time. All of it to experts is an ordinary Mixture of Experts. Give a fifth to a quarter of it to the table, and the loss, which is just how wrong the model's guesses are, drops from 1.7248 to 1.7109.

Give the table too much and it gets worse again, so the curve is a U. The best point was about the same in both models. That's a share of the spare budget going to experts, not a share of the experts that wake up. The same number wake either way. And with experts cut down to 40%, the model still matches the ordinary one. Why do we see a U curve in the performance?

The authors say each end is missing something. With all experts, the model has no memory for fixed patterns, so it rebuilds them with depth and computation. With too much table, it runs short of computation, and memory can't replace that. For those who want a resume to work in DeepSeek, the sweep was only at two small sizes, so whether the U holds at much larger scale is open.

They also asked the opposite question. Fix the model, and keep making the table bigger, from about a quarter of a million rows up to 10 million, which is around 13 billion parameters. The loss keeps falling in a straight line every time the table doubles, and it beats a rival way of adding embeddings given the same memory. So more memory keeps helping, but each doubling buys the same small step, and costs more than the last.

The full-size test6:53

Now the real test, at full size. Both models store the same number of parameters, use the same number for a word, and read the same training text. The only change is that the Engram model has fewer experts, 55 instead of 72, and spends what that frees on a 5.7 billion parameter table. A table of facts should help most where a test asks for facts. So, before the experiment. It is a good habit to hypothesize what to expect.

For a lookup architecture, we should expect recall problems to outperform after its introduction. How did DeepSeek test this? Researchers measure a model with sets of questions that have known answers, and each set is called a benchmark. BBH is hard reasoning puzzles that take several steps. MMLU and CMMLU are exam questions across many subjects, the second one in Chinese. ARC is school science questions. HumanEval is writing code that passes tests, and GSM8K and MATH are maths problems, the second at competition level.

TriviaQA is general trivia, and PopQA asks about people and places, from the famous to the obscure, which makes it close to a pure test of what the model remembers. Here are the gains, Engram minus the plain model, on a spread of tests. You can check out these numbers on screen. The biggest is BBH, at 5 points. The exam questions come next, and then coding and maths. The two plain trivia tests are at the bottom.

TriviaQA gains 1.9, and PopQA gains 0.2. The authors say the gains are even bigger in reasoning, code and maths than in the knowledge tests. So the table does help, just not where you'd expect.

Switching the table off8:21

Okay, so here's the question you're probably asking, if it's a table of facts, why doesn't it win at trivia? Inversely, why did it win in hard reasoning benchmarks? Because the facts really are in the table. To prove that is the case, the authors take the trained model and switch the table off, only at the end, when it answers. Meanwhile, they left everything else alone. TriviaQA falls to 29% of its score.

PopQA falls to 44. Reading comprehension keeps 93. And maths falls too, MATH to 36%. The authors warn this is a rough test, because the model was never trained to work without the table. The gains compare Engram with a different model of the same total size. The switch-off compares Engram with itself. So where do the gains come from? The authors' answer is about depth. They say a plain model wastes its early layers rebuilding names and set phrases, depth that could have gone to reasoning.

Give it a table for those, and it gets that depth back. They compare the layers of the two models with a method called CKA, which asks which layer in one model looks most like which layer in the other, and layer 5 of Engram looks most like about layer 12 of the plain model. So the model gets to the same place with fewer layers, and the layers it saves are free for reasoning.

That's a reading of a similarity map and not a measurement of what the layers do, and the paper presents it as the authors' interpretation. In a smaller test, one table at layer 2 worked best and got worse the deeper it went. As for why maths collapses when the table gets removed, the paper doesn't say. My guess is that maths leans on remembering abstract things, like what a symbol stands for and which step we're on.

Long context and caveats9:53

There's one more result a lookup table shouldn't touch. The authors stretched both models to read very long texts, and tested them on finding facts hidden somewhere in a long document. Engram finds 97.0% of several hidden facts at once, against 84.2 for the plain model, and that holds even when the two are compared at the same training loss, so it isn't only a better-trained model. The authors' explanation is that the table handles the short local patterns, and that leaves the model's attention free for the long-range ones.

Done? Not quite. A larger version, Engram-40B, has 18.5 billion more table parameters, and it doesn't win on every test. The authors think it was under-trained, because the gap in training loss was still widening when training stopped. And every number so far is one family of models, at one size, run by the authors themselves.

The cost, and back to the 0.210:39

Then there's the cost. The rows are picked by the words alone, so the model knows which ones it needs before the layer runs, and it can fetch them ahead of time from the computer's ordinary memory instead of the graphics card. The authors tried a table of 100 billion parameters kept there, and the token throughput speed dropped by at most 2.8%. That's on smaller models, on one H800 card, and not on a full Mixture of Experts model from end to end.

So, back to the strange number. A lookup table, and trivia only gains 0.2. The authors saw it too, bigger gains in reasoning than in recall. Their explanation is that the table takes the remembering off the early layers, so the model has more depth left for reasoning. They test that by comparing layers, which is evidence for it and not proof. So next time a model names Princess Diana without breaking a sweat, some of that might not be thinking at all.

It might just be a lookup, and the thinking got saved for something harder.

Sources

1. Cheng et al., "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models" (2026)
2. DeepSeek-AI, Engram repository (Apache-2.0, a demo of the module)
3. DeepSeek-AI, "DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression" (2026)
4. Ghandeharioun et al., "Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models" (ICML 2024)