DeepSeekMoE: Why turning off one expert makes the model stupid

A design that lasted0:00

Yo guys? What is up? David here. We're now at the end of 2026. Both DeepSeek V2 and DeepSeek V3 say in their abstracts that they are built on one design, DeepSeekMoE. This is an architecture that informed DeepSeek the last 2 years of their development. For the AI world, 2 years is a long time. I wouldn't call MoE a Lindy architecture, but it is worth examining what made it so persistent, so that is what we will try to find out today.

The paper is DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. It was posted on 11 January 2024, six days after the first DeepSeek paper, which promised it. It's from DeepSeek with Peking, Tsinghua and Nanjing Universities. The authors say the usual way of building these models faces challenges in making each of its parts learn something of its own. And their paper is the answer to that. Take a small model built from 64 little parts, switch off the one part that is always on, and give its work to another part.

The error goes from 1.81 to 2.41, and that's worse than a plain model with no experts at all.

Experts, routers, two problems1:02

Before we go further, here are the words we need, in plain terms. Inside a language model, the part that stores most of what it knows is called a feed-forward layer. In an ordinary model, which we'll call a dense model, every word goes through all of it. A mixture of experts breaks that layer into several smaller copies. Each one is an expert, and a small scorer called the router picks which experts handle each word.

So if there are 16 experts and the router picks 2, a word only costs the work of 2, even though the model still holds the knowledge of 16. That gives two sizes. The total parameters are everything the model holds, and the active ones are the part that works on each word. The small models in this paper have about 2 billion in total and 0.3 billion active. The active ones set the computing bill, which is counted in FLOPs, the number of arithmetic steps.

The total sets the memory, because every expert has to be stored, even when it's idle. So an idle expert costs you memory, but no computing. Now the part that's easiest to get wrong. Nobody gives an expert a job. There is no expert for law and another for graphic design. All the experts train together, on the same text. What decides who learns what is the router. Each expert carries its own list of numbers that training keeps adjusting, and the router checks how well a word's numbers match each expert's list, and sends the word to the best matches.

A word that goes to an expert trains that expert on that word. After enough words, each expert has become good at the kind of words it was sent. So the slicing is something that happens, it isn't decided. There's a catch the authors name. The router can collapse and keep picking the same few experts, and then those few are overworked and the rest never get enough training. The usual designs, like GShard, have just a few experts, 8 or 16, and send each word to one or two of them.

That causes two problems. The first is knowledge hybridity. With so few experts, each one gets words about many different things, so it can't specialise. The second is knowledge redundancy. Different experts still need the same basics, grammar, common facts, so each one ends up learning them again, and that wastes room.

Smaller experts + a shared expert2:59

Think of a company where every task goes to one of 16 senior staff. Each of them has to be a bit of everything, so nobody is really good at anything. And every one of them has to know the filing system, so the same knowledge sits in 16 heads. To fix that, you'd do two things. You'd split each senior job into four juniors, narrower and cheaper. And you'd put a small front desk that everyone uses, that handles the filing system for all.

Same payroll, same hours, but now we do it at a digital scale. That is the whole paper. Here's the first idea, the juniors. Cut each expert into several smaller ones, and let the router pick as many more. The parameters and the computing per word stay the same on purpose, so no gain can come from a bigger bill. What changes is how finely a word can be matched.

16 experts picking 2 make 120 different pairs. Cut each into 4, so 64 experts, and pick 8, and you get about 4.4 billion different teams. The number of choices goes up, not down. And the hope is that each small expert can then stay narrow. The second idea is the front desk. Take a few experts out of the router's reach and use them for every word. These are the shared experts.

Now the common basics have a home, and the routed experts are free to specialise. To keep the cost the same, the router picks fewer routed ones, exactly as many fewer as there are shared ones. And nobody decides what a shared expert will hold. It's fixed in the design before training, and what it ends up knowing is something we only learn from what breaks when it's gone. The router has one more problem, the collapse from before.

A balance penalty stops it, but the authors keep it small, because pushing hard hurts the model.

The 2B test vs GShard4:30

Now, time for the test. The models train on 100 billion tokens, and have about 2 billion parameters in total. Five designs are compared. A dense model with no experts, three earlier mixture-of-experts designs, Hash Layer, Switch Transformer and GShard, and DeepSeekMoE. The best opponent we want to be comparing is GShard, since it costs the same, with 0.3 billion active parameters. You can check out the loss on screen, lower is better.

DeepSeekMoE gets 1.81, GShard 1.87. The authors call it an overwhelming advantage. To see why a drop from 1.87 to 1.81 is such a big deal, we have to look at what this loss means. It's the cross-entropy loss, and e to the power of the loss gives you perplexity, the number of options the model is effectively guessing between for each token. So GShard is guessing among about 6.5 options, and DeepSeekMoE among about 6.1.

That's about 6% more uncertainty on every token, at the same cost. We also want to mention that those prediction errors we avoided are a result of the model understanding newer and deeper problems. So the next time you see a curve of diminishing returns in deep learning, and you think it's not worth it anymore, remember that the math means something very different to the researchers obsessed with it, and most people greatly underestimate the results.

Then, the authors ask how much room is left. Give GShard 1.5 times the computing, and DeepSeekMoE still ties it on loss. Then, they set a harder bar. The authors build a stand-in for a ceiling, a dense model with all 16 experts always on. Its loss is a little lower, but it costs about 5 times the computing. So the tie in quality comes with a big saving. That it's the best a mixture of experts could do is still just the authors' reading, and they limit it to 2 billion parameters and 100 billion tokens.

With more research we would find out if there are better parameters we can set. Now back to the number from the start. The authors switch off the shared expert and let one more routed expert work in its place, which keeps the cost the same. The loss increased greatly from 1.81 to 2.41, worse than the dense model's 2.06. They read that as the shared expert holding knowledge the routed ones can't pick up.

We can say one thing about it. The cost is the same, so fewer active experts isn't the reason. There is some trained-from-scratch evidence too. Giving GShard one shared expert helps on most tests.

16B and 145B6:45

So the authors scale it up. The 16 billion model trains on 2 trillion tokens, the same data as DeepSeek's dense 7 billion model. That's 40% of the computing. The loss is level, and it's ahead on facts and code. On TriviaQA it scores 64.8 against 59.7. But it's behind on the multiple-choice exams. MMLU is 45 against 48.2. You can check out the rest on screen. The authors give a reason.

The 16 billion model has only about a fifth of the attention parameters of the 7 billion one, and attention is how words look at the other words around them. They point to an earlier DeepSeek study for it. Then they try a bigger one, 145 billion parameters, and they call it a preliminary run. It trains on only 245 billion tokens, and so does the dense 67 billion model they compare it with.

The 145 billion uses 28.5% of that model's computing. You can check out the numbers on screen. It's ahead on loss and on trivia, and behind on MMLU. The authors say it's comparable on the whole, with limits on multiple choice. One more number from that table, and the paper doesn't explain it. GShard's 137 billion model scores 26.3 on MMLU, which is close to what guessing gets on a four-choice test.

So it appears GShard's 137 billion model might not be up to the task for multiple-choice questions.

What was never measured8:03

Here's the catch with the whole paper. The two problems it fixes, hybridity and redundancy, were argued, never measured directly. The closest the paper gets is a knock-out test. Switch off a growing share of each word's top experts, and DeepSeekMoE gets hurt more than a bigger GShard does, which the authors read as less overlap between its experts. That shows each expert matters. It doesn't show what any expert actually contains, so you can't check that the problems were there before the fix.

And that's the risk of this kind of design. Is a new design worth swapping in for a full training run that's known to work, which, at the time, meant LLaMA? Since we are covering an old paper, we do get to find out what happens when other teams tried it.

What OLMoE found8:41

One answer comes from OLMoE, an open mixture-of-experts model from the Allen Institute for AI, that tested the same ideas in its own setup. On finer experts, it agrees. At the same cost, going from 8 experts to 32 improved HellaSwag and MMLU by about 10%, and going to 64 added another 1 to 2%. On the shared expert, it doesn't. Comparing one shared expert plus 31 routed ones against 32 routed ones, at the same cost, the shared version performed slightly worse.

Their reason is that a shared expert removes almost 90% of the combinations. So they don't use one. It's one paper and one setup, so it doesn't overturn this result. But it's the first outside test of the second idea. They also end up measuring what this paper didn't. That is the experts rarely fire together, and many fire mostly on one kind of text, the first one in layer 0 almost only on arXiv papers.

Their reading is that there's little redundancy. But this whole paper's discoveries could be another video for another day.

The answer, and what to try9:39

So, why is DeepSeek known for MoE design? Because the secret was published. Cut the experts into smaller ones, and keep a few of them always on. DeepSeek then built V2 and V3 on it. The two ideas lasted. The balancing penalty didn't, because V3 balances its experts without it. Here's what you can do with this, depending on who you are. If you pick a mixture-of-experts model to run, look for total and active parameters.

For example, V3 has 671 billion in total and 37 billion active. The first is the memory you need, the second is what you pay for. If you train small models, count the combinations before you add a shared expert. It is one line of code, and it takes a minute. Then compare the two at the same cost, like the OLMoE team did. Their code, logs and checkpoints are all open.

And if you follow DeepSeek, V2 is where this design runs at scale, with 236 billion parameters and 21 billion active. Alright, that's it for the video. With AI getting so good these days, I think it is worth creating more videos showing people how AI really works. Even if you are not a researcher yourself, learning these kinds of things helps you be a better user. For those who have been building cool stuff, I hope you find each other in the comments section.

If you know a topic in AI or finance you want me to cover, leave it in the comments as well. I will see you guys next time.

Sources

1. Dai et al., "DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models" (2024)
2. DeepSeek-AI, DeepSeek-MoE repository
3. Muennighoff et al., "OLMoE: Open Mixture-of-Experts Language Models" (2024)
4. DeepSeek-AI, "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model" (2024)
5. DeepSeek-AI, "DeepSeek-V3 Technical Report" (2024)
6. DeepSeek-AI, "DeepSeek LLM: Scaling Open-Source Language Models with Longtermism" (2024)