How to Find the Next Semiconductor Bottleneck (Is It Still Memory?)

The gap0:00

Hello and do you like money? What if I told you that if you can find the next AI bottleneck, it is possible for you to 10x your money, like those who invested in Micron and Sandisk? Obviously, nothing is guaranteed, and even the best-informed investors still aren't always successful. The point I'm trying to make is that there are enough people who make a living or a retirement trying to find the bottleneck in this AI gold rush, from GPU, to memory, to power, to whatever's going to come next.

For responsibility purposes, I can't tell you which stocks to buy or what positions I hold. This is not a financial channel.

However, what I can do is show you the discovery process of how the industry finds its bottleneck, and let you make up your own mind. Even if you don't care about money, this is just my way of getting you to care about the boring physics of semiconductor engineering. Let's get started.

In 2016, Nvidia's flagship data center chip was the Tesla P100. In 2025, it's the Blackwell B300. Over that time, the maths it can do in 16-bit numbers, numbers written with 16 ones and zeros, went up 106 times, while its memory bandwidth, that is how fast it can pull numbers out of memory, went up about 11 times. The reason this matters is that a GPU, the kind of chip that runs AI, doesn't just calculate very fast.

It also needs to read and write its data somewhere.

So the calculation part is the maths it can do, and the read write part is the memory bandwidth. Given that gap in growth rate, eventually we start feeling it every time a chatbot writes us an answer.

How a GPU writes1:37

Let us go through how a GPU produces a token. A token is a word, or a piece of one. If you haven't checked out our previous video on how a data center prints money, we explained how that process works in more detail.

First, the GPU loads the model. A model is a huge list of numbers it learned in training, called its weights, and loading means copying all of them into the GPU's memory. The GPU has two parts that matter here, its memory, where the weights sit, and its maths units, which do the multiplying.

The small memory built right next to the maths units is far too small to hold a whole model, so the weights have to come over from the main memory every time they're needed.

Then it generates. Reading your question happens in one go, but writing the answer is a loop. It writes your answer one token at a time, and for every single token it reads the weights back out of memory and does a multiply and an add with each one. Now you have turned electricity into intelligence. The problem with this process is it's energy intensive. And when it writes for one person, most of that energy goes into the reading, not the maths.

Fetching a weight from memory costs about 200 times the energy of the multiply and add it feeds. To understand the consequence of this, again, check out our previous video.

To see where a chip gets stuck, the metric the industry eventually settled on is how much maths a chip can do for every byte it reads. A byte is 8 bits, and a bit is a single on or off. And an operation is a single multiply or add.

To understand this metric, picture a library. The model is a pile of books. The GPU's memory is the librarian, carrying books to a study table at a fixed speed, and the maths units are the students at the table, who read incredibly fast. So the chip has two speed limits, how fast the books arrive and how fast the students read, and the slower one sets the pace.

On the B300, the students could do about 281 operations in the time it takes to bring them a single byte. That number is the chip's ridge. It's the point where every book trip keeps all the students busy, so the chip stops waiting. A chatbot writing for one person does about 1 operation with each byte.

So the students spend almost all their time waiting for the next book. That wait is the bottleneck. And idle maths units still cost silicon and power, so you're paying for students who just sit there.

Knob 1: faster maths4:06

There are a few ways to get more performance out of this. Faster calculation, faster bandwidth, and a smaller data format. In the library, that's faster students, a faster librarian, or thinner books. Faster students don't help much if the books are late, a faster librarian helps right away, and thinner books mean fewer trips. For faster calculation, you want smaller printed transistors, the tiny switches the maths units are built from.

For faster bandwidth, you want more wires between the chip and its memory, and the issue is you run into the die perimeter problem, which we'll cover in detail later. The die is the slice of silicon a chip is printed on, and its perimeter is its edge. And the most consequential hack Nvidia settled on is to go for smaller data formats. So thinner books it is.

Now back to faster calculation, and why it stopped being the easy knob. It needs smaller transistors, so we can fit more of them on a chip, so it can do more maths. We did it by pushing the lithography machines, the machines that print chips with light, to their limits. Nvidia's H100 fits about 4 times the transistors of the older V100, on a die the same size.

There is one problem with the lithography machines. Each print can be at most this size, about 26 by 33 millimetres. The reason has to do with how the light can be focused, and how fast we want to run our printing machine. ASML's newest machines focus the light more sharply, and the price is a print half that size, so they had to move faster to keep up.

So the machine caps how big one die can be, since it prints each die in a single flash, and every flagship since 2017 has sat right at that limit. We could have another whole video about lithography some other day.

So this knob worked, and that's the gap from the start. Most of it opened in 2017, when the V100 added special maths units called tensor cores, built just for the multiply and add that AI runs on. Since then, at the same 16-bit numbers, the gap has only about doubled.

Faster maths still pays off when there's a lot to calculate at once. Reading your whole question happens in a single pass, and that keeps the students busy. Another example is rendering the graphics in your games, which is what GPUs were invented for in the first place. It's literally named the graphics processing unit. But today, the students are asked to work on a new problem, generating tokens. For a chatbot writing its answer, faster students just wait longer for the next book.

In physical terms, it means we have a memory bottleneck for this problem.

Knob 2: faster memory6:50

To see why memory is slow, I had to start with what a bit physically is. In memory, a bit is like a tiny on off switch. The way it stores memory is by flipping a sequence of these switches on and off. When your memory changes, the switches change too. So this is what's happening when your computer remembers. In DRAM, the kind of memory a GPU's main memory is built from, each switch is really a tiny electric charge held in a cell, charged or empty, a 1 or a 0.

The issue is that a charge leaks, so every cell gets read and written back at least every 64 milliseconds. And to read one, the memory opens a whole row of cells at once, like pulling out a whole drawer to read one card, which is slow and costs power.

Back to the drawer analogy. Storing a bit is cheap. Moving it is what costs. The bits sit in memory and the maths happens in the processor, so every bit a token needs travels down a wire. And a wire behaves like a tiny bucket for charge. Sending a 1 fills it and sending a 0 empties it, and that energy ends up as heat. A longer wire holds more charge, so the further a bit travels, the more it costs.

Each wire also has a top speed. After one switch it has to settle before the next, or the bits smear into each other, and every extra switch a second costs more power. I picture a light switch. Flip it too fast and you ruin it, so you add more switches.

That gives us the recipe for bandwidth. It's the number of wires times how many bits each wire carries a second. So the cheap way to get more bandwidth is more wires, and more wires need their contacts, the tiny metal pads where a wire meets a chip, packed closer together.

The memory on these GPUs is called HBM, high bandwidth memory. It's a stack of DRAM chips, joined by tiny wires that run straight down through them, and it sits right beside the GPU, with both of them resting on a slab of silicon called an interposer. The reason is contacts.

One stack talks over more than a thousand data wires, and every wire needs its own contact.

A normal chip package, the board a chip usually sits on, spaces its contacts far wider apart than silicon can. So on a package those wires alone would need about 6 times the room. Silicon also keeps the wires short, so each one is cheaper to drive. Even this close, most of the energy of a read is spent inside the stack itself, opening the row and carrying the bit to the edge of the memory chip, and only a small part goes on the hop across to the GPU.

That's why a modern AI chip looks like a big die ringed by smaller memory stacks. So the memory sits as close to the maths as it can get, and the room it needs is a strip along the edge of the die. In the library analogy, we put the shelves close to the study table, so the librarian doesn't have to walk as far.

The die's edge9:45

OK. The maths lives on the die's area. But the wires to memory have to cross its edge. Make a square die bigger, and its area grows faster than its edge. This is what I call the die-perimeter problem. Dylan Patel walked through it on Dwarkesh Patel's podcast, and Asianometry has covered the memory wall itself. But it isn't complicated to understand, since it's just basic geometry. And since 2017, the die can't get any bigger, thanks to the lithography printing constraint we just covered.

So Nvidia lined the edge it had with memory stacks, and for the B300 it joined 2 dies into one chip, with 8 stacks around them. Even so, that only about doubled the edge given to memory since the V100. Most of the gain came from each millimetre of edge carrying about 4.4 times more, mostly by running the same wires faster.

Microsoft Research's own suggestion goes the other way. Cut the chip into smaller ones, since quartering a die doubles its edge for the same maths. Interesting proposal. Time will tell if this idea is going to work out.

So how do you escape the die-perimeter problem? If the edge is too short, the other way out is to stop relying on the edge. Bond two chips face to face, and the wires can cross the whole face instead of squeezing through the border.

So instead of building longer houses, you stack them taller. AMD has shipped that since its first processor with a stacked cache, a layer of extra memory bonded right on top of it. The obvious problem is that it's much easier to build longer houses than taller ones. It's difficult, but not impossible.

The next problem is heat. A surface cools faster than a core, and a stack has a core where the heat can't escape. The reason you don't want heat in your chips is that things can literally melt. You may have seen videos of GPU power connectors melting.

But long before that, a hot chip slows itself down to cool off, so you get less processing done.

AMD's first stacked-cache chip runs about a 10% lower base clock than the same chip without the stack, at the same power. The clock is how many steps a chip takes every second, so a lower clock means less maths every second. So stacking the cache cost that chip speed. In its next design, AMD moved the cache underneath, so the hot cores sit right against the cooler, in AMD's words "closer to the cooling solution".

That's why we think the first chip ran slower, the cache on top was trapping the heat.

Memory stacks have the same problem. Micron mentioned that the bottom layers of a stack make the most heat while the cooling sits on top, so the heat has to climb through the whole stack before it gets cooled. Using the tall building analogy, your ground floor is cooking while the top floor gets air conditioning. Sending cold air evenly through the building is a challenge.

Knob 3: smaller numbers12:32

The third knob is smaller numbers. So we're back to the thinner books. Every weight in a model is a number, and so far each one took 16 bits. Write it with half the bits, a format called FP8, and every byte you read carries twice as many weights. Nvidia's FP4 format nearly halves it again.

And the chip does more maths in the smaller format too, so the maths per byte climbs without the memory changing at all. Count things in smaller formats, and that 106 times from the start becomes 637. The chip has to be built for each format, though, which is why FP4 only shows up on the newest chips.

For a chatbot, the win is more obvious. In FP4, a big model shrinks to a bit over a quarter of its 16-bit size, so the same memory has far fewer bytes to read for every word. The price is accuracy. On the model Nvidia tested, FP4 lost 1% or less against FP8. For what we gained, this loss doesn't look too bad.

Reading fewer bytes13:36

The rest of the hacks are software, and they all try to read fewer bytes for every word. The first is batching. Every user's next token needs the same model's weights. In the library analogy, every student wants to read the same books, so one trip by the librarian can serve a whole table of students at once. On the B300 it takes about 281 users sharing each read to bring the chip up to its ridge, where the students are all busy.

The price is that each user waits a little longer for every step. But there's a second thing every token reads. To write the next token, the model looks back at notes it kept on every token before it, and those notes are called the key-value cache, or KV cache. The books are shared, but the notes can't be. They're each student's private notepad on their own conversation, so adding a student adds a new notepad.

On an open model called Qwen, one user with a long conversation fills about 10.7 gigabytes of notepad, so 8 of them overflow the memory of an H100. Batching shares the books, but multiplies the notepads.

The second is mixture-of-experts. We also have a video on this topic if you want to check it out. The idea is that the model is split into many expert blocks, and each token only reads the few it picks. DeepSeek-V3 reads about 37 billion of its 671 billion weights for each token. But when many users pick different experts, most of that saving is gone. The third is speculative decoding, which writes more than one token for each read.

A small, cheap drafter guesses the next few tokens, and the big model checks all the guesses in one pass, so its weights are read once for all of them.

For speculative decoding, imagine you're a newsroom on election night. You have two folders ready, "memes when Kamala wins" and "memes when Donald Trump wins". Those memes are made before the result comes in, and the moment it's confirmed, the right one goes up. So instead of processing the world after the election, you anticipate the election and choose your content later. DeepSeek's version writes about 1.8 times as fast.

The price is the maths spent on guesses that get thrown away. In our meme folder analogy, to beat the news, one folder of memes gets thrown away, never seen by anyone.

The fourth shrinks the notepads. DeepSeek's multi-head latent attention, MLA, squeezes each token's notes into one small compressed list of numbers. It's like writing a whole chessboard down as one short line, and setting the board back up when you need it. The cost is setting the board up again from that note every time you need it. The industry recognizes this problem, so we've also tried experimenting with things like SRAM, which means memory built from transistors right on the chip.

It's very fast, but it holds very little, so a big model has to be spread over many chips. That said, SRAM is still just one idea in a very niche space, and it won't be the focus for this video. For practical purposes, most of the GPUs we know today use DRAM for their main memory.

Everything so far is a speed limit, and real chips fall short of it. The memory alone, on a test that only streams data, gets to about 90% of its rated speed. Writing real answers, independent tests land somewhere between 25 and 70%. One chip serving one user sits near the top, and many chips serving many users sit near the bottom. And when we worked it out on real chips for our GPU video, from a public serving benchmark, they reached only 4 to 29%.

A few things pull down the peak efficiency of a GPU. Memory is fastest when it moves big chunks in one go, and small reads waste that. When a model is split over several chips, they have to stop and sync up with each other after every layer. And the CPU, the regular processor that hands the GPU its next job, sometimes leaves it waiting between steps. I picture a traffic light, where every street waits its turn, and adding more chips doesn't make that wait any shorter.

Even so, the maths is still the part that waits. At any of those shares, a chatbot does far fewer operations per byte than the chip could, so it sits far below the ridge, and the students are still waiting for books.

Still memory?17:47

So when the B300 writes you an answer, it still spends most of its time waiting on memory, and for that job, memory is still the bottleneck.

But the squeeze is already moving. When a model is spread over many chips, one research team found that the time they spend talking to each other barely shrinks as you add more chips. And Nvidia's own link between two chips costs several times the energy for each bit of that short hop to memory. And faster memory costs power. If Nvidia's announced Rubin chip paid the energy reported for each bit of today's memory, its memory alone would draw about 600 watts, by our estimate.

So the bottleneck keeps moving, from GPU, to memory, to power, to whatever's going to come next.

If you want to try the process yourself, take any chip's datasheet, divide its maths by its memory bandwidth, and name the number format every time. That's how much maths it can do for every byte it reads, and the higher it goes, the longer a chatbot leaves it waiting.

Alright, that's it for this video. In future videos, I'll go through more specific bottlenecks in detail. One I'm anticipating is networking and bandwidth, which will take me a long time to research. Thanks for watching, and I'll see you in the next one.

Sources

1. Dylan Patel on the Dwarkesh Podcast (the edge, from 1:21:22)
2. Asianometry, AI's Hardware Problem
3. NVIDIA Tesla P100 datasheet
4. NVIDIA Volta whitepaper
5. NVIDIA, Hopper architecture in depth
6. NVIDIA H100
7. NVIDIA HGX (B300, Rubin)
8. NVIDIA, Inside Blackwell Ultra
9. NVIDIA, NVFP4
10. NVIDIA, nvidia-smi documentation
11. Pope et al., "Efficiently Scaling Transformer Inference"
12. RPU (energy per operation)
13. DistServe
14. Splitwise
15. ASML and Zeiss, High-NA EUV
16. ASML, High NA EUV
17. Microsoft Research, Lite-GPUs
18. RAIDR (DRAM refresh)
19. Horowitz, Computing's Energy Problem
20. Keckler et al., GPUs and the Future of Parallel Computing
21. Ho, Mai & Horowitz, The Future of Wires
22. O'Connor et al., Fine-Grained DRAM
23. UCIe on-package memory (Hot Interconnects 2025 slides)
24. Micron HBM4
25. AMD Ryzen 7 5800X3D
26. AMD Ryzen 7 9800X3D
27. Qwen2.5-72B
28. DeepSeek-V3 Technical Report
29. DeepSeek-V2 (MLA)
30. Speculative decoding
31. Cerebras on SRAM
32. Luo et al., Hopper benchmarks
33. Databricks, LLM inference performance engineering
34. Recasens et al., Mind the Memory Gap
35. SemiAnalysis InferenceX
36. Melting RTX 4090 connectors (PCWorld)
37. Nasdaq historical closes (MU, SNDK, PHLX Semiconductor index)