How to Know If Your Data Centers Print Money?

Which GPU?0:00

Hello guys and what is up? David here. Hypothetically, you were given billions of dollars to build a data center. Which GPU should you pick? The answer seems obvious. It should be Nvidia. After all, it has the biggest market cap. Everyone is talking about them. Everyone is paying an arm and a leg to get their chips. Even China wants Nvidia chips so badly, and US export rules still keep Nvidia's newest chips out.

But other than branding, is there any reason why you want to prefer Nvidia chips at all? Even before that question, we are operating on a premise that Nvidia is the best. What if there exists something better?

If you ask Jensen Huang, his keynote says "Notice NVIDIA is the highest performance in the world." In other words, "Trust me bro". Turns out, the answer is much more complicated than you think. But we will simplify it such that you just care which chips bring you the most money. Lately we've had announcements for chips like Google's TPU, Amazon's Trainium, Cerebras and Huawei's chips. There are so many choices on the market. We can't cover all of them.

So, we read the same independent benchmark for 10 chips that we found numbers for, and got our answer. Here are the spoilers to save you the remaining video. On long prompts, Nvidia's B300 leads when each user is served slowly, and Google's TPU leads when each user is served fast. So the real question isn't which chip is best. It's which chip is best for what you're serving, and how fast.

Reading and writing1:31

Before we can answer what makes a GPU the winner, we first have to understand what is going on inside it. That way, we won't be fooled by published numbers, because reality is conditional, and conditional means published numbers differ from practice. And practice is how you get paid as a data center owner. Here's what a GPU does with your prompt. First, it reads the whole prompt at once.

That process is called prefill, and it's limited by raw compute. Then, it writes the answer one token at a time. That process is called decode, and it's limited by memory, because every new token rereads the model's weights from memory. For the long-prompt model we'll look at, that's about 17 gigabytes read for each token, for a single user.

The next thing to understand is that not everyone prompts the model the same way. Some just want a normal chat, a short question and a short answer. Some want to paste a wall of text as an input, also known as the "long prompt". That's mostly reading, so it leans on compute. And some people want the model to run a long task that generates a ton of tokens in one session.

That's mostly writing, so it leans on memory. Those differences decide a workload. Different workloads have different hardware requirements. On top of that, you also get paid differently.

And there's one more number that decides whether you keep a user engaged. How long they wait for the first word to appear on screen.

The trade-off3:00

Here's the trade-off every GPU lives with. According to Nvidia, you can have throughput or interactivity. Throughput is the most tokens per second per GPU. Interactivity is the most tokens per second per user. On Nvidia's own chart for one model, that's 60,000 tokens a second per GPU at one end, and 1,000 tokens a second per user at the other. They pull in different directions, so a higher number per GPU means a lower number per user, and you have to trade one for the other.

Another important thing to note is that the curve is convex, not concave. What that means is when you try to meet in the middle, at half the top speed per user, you get far less than half the top throughput. There is a physical explanation for why that is the case, but I think it is possible to answer this with an analogy.

Think of a kitchen. Throughput is how many dishes the kitchen serves an hour. Interactivity is how fast one customer's dish arrives. Suppose everyone just asks for the same order. The chef cooks one big batch and finishes for everyone at once. One read of the weights serves the whole table. So the kitchen serves more dishes an hour, but each customer waits a little longer for theirs. Suppose more people sit down, the chef cooks slower, because each customer asks for something different yet wants their order at the same time.

That's why every chart in this video has two numbers that pull against each other.

Now, let us talk about the bottlenecks in a GPU. Each chip carries stacks of HBM, high bandwidth memory, right beside it. Memory matters because it sets a ceiling on how fast tokens can go. And fast is what you want when printing money. Divide the bandwidth by the bytes each step has to read, and you get the fastest any user can go, one token per read. With a few users at once, each step reads even more, because each user's tokens pick different experts inside the model.

That is the physical explanation for our kitchen analogy.

But when you measure real chips against their own ceilings, they reach only about 4 to 29% of it. So memory sets the ceiling, but it doesn't set the speed. There are two ways you can close this gap. The software that schedules the work, where every chip is a special snowflake that needs its own optimization, and the networking between the chips.

That's where Nvidia built its rack. One example would be the NVL72, which ties 72 GPUs into one NVLink domain over copper cables. But networking is also beginning to reach a limit, and you can hear Jensen complaining about it. At high speed per user, "NVLink 72 runs out of steam and simply can't get there".

There's another software trick we later found out that passes the memory ceiling. It's called speculative decoding. Instead of one token per trip to memory, the chip guesses a few tokens ahead and checks them all at once. An analogy for this is an Intern rapidly guessing the next few words. Then, an expert editor reviews a chunk of text written by intern instead of writing every word from scratch by himself, then decide whether to keep the intern's words.

With this trick, for Nvidia's GB200 it nearly doubles the speed per user.

The four questions6:14

Now we understand how GPU works at generating tokens, what is the fair way to compare each chip? You borrow the tests that already exist. Instead of just looking at throughput, researchers look for goodput, the most work a chip does while still meeting a first-word limit and a speed limit. MLPerf, the industry's benchmark, caps the first word at 2 seconds for its chat category. MLPerf doesn't say why it picked 2 seconds.

I'd guess it's simply a wait most users tolerate. Nvidia itself shows curves for each model rather than a single point. And Artificial Analysis already ranks hardware by agents per megawatt, that is how much work a chip does from a fixed power budget, on measured power.

In our case, we took one independent benchmark, SemiAnalysis's InferenceX, and fixed three workloads. Chat, with short questions and short answers. Reasoning, with short questions and long answers. And long prompts, with long documents in and short answers out. Each one at three speeds per user, 20, 50 and 100 tokens a second, with the first word within 2 seconds for 9 users in 10. The reason why we do this is so every chip is judged on the same job.

Otherwise each vendor quotes the setting where it looks best, and that's exactly what fooled us, as you'll see in a minute. For each workload we used the model that ran on the most chips.

So, before your data center starts making money, what are you serving? For chat, of the chips tested, single servers of 8 Nvidia GPUs are the winners at every speed. At a slow 20 tokens per second per user, the B300 makes about 5,650 tokens a second per chip, ahead of the B200. And single 8 GPU servers lead on all four chat models we read, in all but one cell.

Long prompts are a different story. At 20 tokens per user, the B300 leads again. At 50, four chips tie within about 2%, Nvidia's two big racks, the B200, and Google's TPU. At 100, the TPU leads with 404 against the B200's 258.

And reasoning, with long answers? The benchmark can't tell us yet. Too few of its runs meet the first-word cap. But at least, we found out which GPUs passed the test so far.

Question two. How fast must each user be served, and how long may they wait for the first word? This is the one that easily fools headline readers. When we first read the benchmark with no cap on the wait, Nvidia's NVL72 racks led by about 2 times on long prompts, at 50 tokens a second per user. But look at how those racks got there. They ran 4,096 requests at once, and 1 user in 10 waited 41 to 46 seconds for the first word.

So this is like a kitchen that batches the same fried rice really quickly, while some guy who ordered pizza is waiting for hours. Cap that wait at 2 seconds, and the racks drop to about 540 tokens a second per chip, level with the B200 and the TPU. Back in the kitchen, it means the chef is forced to serve one course at a time to every table, so nobody waits forever.

But then she can't stack up the giant orders that made her look so fast. On chat the same cap flips it the other way, and the racks trail the single servers by 3 times or more.

Question three. How many tokens do you get from a megawatt? Jensen's argument is simple. Every data center is power constrained, and a one gigawatt factory will never become two. I think he's right, but only if electricity is your bottleneck, and only if your chips are actually busy.

Nvidia itself says only about 60% of the electricity pulled from the grid turns into useful AI work. It blames cooling and losses in the rack, but it doesn't show the math. We can check it. An average data center spends about a third of its power on overhead like cooling, so about 65% reaches the computers. Across Google's fleet, about 92% does. So that 60% describes an average building, not a good one.

And read the comparisons carefully. Nvidia's 35 times and 50 times compare its racks with its own older Hopper chips. Google does the same, Ironwood against its own previous TPU.

On measured power, at long prompts and 50 tokens per user, the GB300 leads with about 768,000 tokens a second per megawatt, only about 17% ahead of the B300. So using measured power is the better indicator. The H100 drew only 62% of its rated power in the test.

Question four. Has the chip ever been tested independently? Semiconductors are an industry that is sucking up the most capital right now, so there are many people showing beautiful benchmarks then asking you for money. This is also where most chips quietly drop out.

In Google's case, Google still enters MLPerf every round, but after version 5, it has entered with Nvidia systems, not TPUs. Since 2024, no Chinese vendor and no Amazon Trainium appears in any round, and Intel's Gaudi 3 was never entered. Even InferenceX runs its TPU numbers from a private repository, unlike every other chip. As for why most chips don't show up, it doesn't always automatically mean they suck.

Maybe the benchmark is still new. Maybe someone wanted to guard their secrets and surprise us later. Maybe they just want you to find out after you bought and tested things out yourself. The best Chinese number we found is Huawei's own paper, about 2,733 tokens a second per chip for writing the answer, on a different model and precision, and assuming its guesses ahead are mostly right. So it can sit beside our charts, labelled as the vendor's.

If a chip has never been tested on your workload, ask the vendor for the numbers before you plan around it. Or just yolo their entire stock and find out yourself. It is your money.

And the hidden cost of an untested chip isn't just the missing number. If your engineers can't code it, the install takes longer. If it takes you longer to install it, you are missing out on a lot of GPU rents. And then there is another problem with GPU utilization, which is yet another rabbit hole your engineers need to care about as they build the software required to optimize your chips.

The software is inside every number we just showed you. And I suspect this is another reason why Nvidia's CUDA moat is real.

What you pay for12:44

We've talked about memory just now, but we are going to talk about them again. That is because different GPUs come with different memory. In this case, we want those with newer generations of High Bandwidth Memory, and often we pay a premium for it. So why would you pay for faster memory? Because faster memory means faster tokens, and faster tokens are what you sell. At one user, an H100 could do about 300 times more math than its memory can feed it.

So for one fast user, the chip spends almost all its time waiting on memory. On paper, you are always bottlenecked by either FLOPs or memory bandwidth, whichever is slower. Today's best chips feed about 8 terabytes a second each, Nvidia's Blackwell and AMD's MI355X alike. But remember, real chips reach only a fraction of that ceiling. So faster memory only pays once the software keeps the chip busy enough to use it.

OK, we're done talking about token income. But we also shouldn't neglect the costs to own and set up the GPUs. For starters, GPUs are unlike older computers. Many of the newer generations are running into physics problems. First, powering the rack. An 8 GPU Nvidia server is rated at about 14.3 kW. A GB200 rack of 72 GPUs is rated at about 132 kW, about 9 times the power in one box.

So picking the rack over the servers changes your building. Before you can fit a GB200, you may have to rework your data center for that power and heat. That is another round of unexpected cost to be thinking about. Nvidia says the old 54 volt wiring was designed for kilowatt racks, and a 1 megawatt rack at 54 volts would need up to 200 kg of copper.

Then there's the grid. Mark Zuckerberg's own engineers at Meta saw power swings of tens of megawatts across their whole data center while training Llama 3, because thousands of chips compute and pause in step. In plain terms, it means you need to think about balancing your electricity, otherwise you melt your grid and explode your AI factory. The money, the electricity price, the depreciation, those are your own numbers and management capability.

The pick14:59

So, back to your billions. Which GPU do you pick? If you're serving chat, it should be Nvidia after all. Its 8 GPU servers lead at every speed.

If you're serving long prompts fast, then Google's TPU. At a middle speed, you have 4 options, so pick on power, on your building, and on what your engineers can run.

And against AMD, at 50 tokens a second per user, Nvidia beats or ties it on every model the two share. The conclusion? There is no best GPU. There is only the best GPU for your workload, at your speed, on your power.

Here's what you can do with this information. If you are picking chips, ask for numbers at your own workload, with the first-word wait capped. When you compare them, use tokens per megawatt on measured power, not the rating on the box. And if a chip has never been tested independently, ask for your own numbers before you plan around it. If you are an investor, then check if you know what your portfolio companies are buying and what were their workloads.

Alright, that's it for the video. You can show your support with a sub, like, or don't. Just checking if you are paying attention. I will see you guys next time.

Sources

1. SemiAnalysis InferenceX (public data as of Oct 22, 2026)
2. SemiAnalysis, InferenceMAX launch
3. MLCommons, MLPerf inference rules
4. MLCommons, MLPerf power measurement rules
5. MLCommons, MLPerf Llama-2-70B
6. MLCommons, MLPerf inference results v6.1
7. NVIDIA, InferenceMAX results
8. NVIDIA, performance per watt
9. NVIDIA, AI factories
10. NVIDIA GB200 NVL72
11. NVIDIA, 800 V HVDC architecture
12. NVIDIA H100
13. NVIDIA DGX B200 user guide
14. Supermicro GB200 NVL72 rack
15. AMD Instinct MI355X
16. Google, Ironwood TPU
17. Google, data center efficiency
18. Artificial Analysis, AgentPerf
19. DistServe (goodput)
20. Huawei, CloudMatrix384
21. Meta, "The Llama 3 Herd of Models"