S3-EP07: The Non-Deterministic WinRAR

It's stored in here.

You're right. You can't. Which should be your first clue that GPT is not a database. You're not talking to a brain. You're trying to understand a new form of matter.

How do you think a computer know what the word 'king' means?

That's a human way of thinking. We choose a word, then map it's definition.

A computer only understands numbers. Since we can't map the description "a male ruler" into a number, we decided to stop trying to define words.

They decided that a word's meaning is not its definition. A word's meaning is **the network it keeps**.

The AI reads trillions of sentences. It sees that the word 'king' appears very often next to words like 'queen', 'crown', 'palace', and 'rules'.

It also sees that 'king' almost never appears next to words like 'photosynthesis', 'keyboard', or 'pickle'.

The AI builds a map. It places every word in this multi-dimensional space. Words that keep similar company are placed close together. This is an **Embedding**.

Exactly. A **vector**. A series of numbers.

We didn't teach it what a king is. We let it discover the 'shape' of the word 'king' by observing its relationships to everything else.

Good guess, wrong. That was an early approach. It failed because context is everything.

The word 'bank' is identical. But its meaning is completely different. How does the AI know which 'bank' it is?

Precisely. It needs to **pay attention**. This is the 'T' in GPT. The **Transformer**.

Think of a sentence as a conversation at a party. Every word is a person.

To understand its own true meaning in this context, the word 'bank' needs to know who is most important to listen to. So it 'looks' at all the other words in the sentence.

The **Attention Mechanism** calculates a score. It realizes that 'money' is the most important clue. It then adjusts its own 'meaning vector' to be closer to the 'financial institution' version of bank. It figures out its own meaning by paying attention to its friends.

Yup. It's learning the statistical patterns, the geometry of language.

Yes. And now you can understand why. What is a map, if not a **compressed representation** of a territory?

A GPT works less like a brain. It's more like a non-deterministic WinRAR.

A simple ZIP file finds repeated patterns, like a sentence that appears 100 times.

A GPT finds the deep, abstract, universal patterns of all human knowledge. It's actually a compression algorithm.

The training process is a compression process. A 1.8-trillion-parameter model is, in essence, a compressed summary of all those statistical relationships.

Yeah. This means when you ask the model, your question isn't a query. It's the **start of the file it needs to decompress**.

The AI doesn't 'understand' the question. It just sees that sequence of tokens and asks itself one thing, over and over: *'Based on the quadrillion patterns I have learned, what is the most statistically probable token to come next?'*

It's not retrieving a fact. It's decompressing a new, statistically plausible sentence, word by word.

Yep. And a GPT is **lossy**.

The truth. The source. It didn't store the original Wikipedia article. It only stored the *patterns* it learned from it. It's like an impressionist painter.

This AI has learned the 'essence' of the Eiffel Tower. The shape, the light, the 'vibe'. He has a perfect internal model of its patterns.

Now, you ask the AI, 'Paint me the Eiffel Tower'. He doesn't give back a photo. He **reconstructs** a new painting from his internal model.

It will be beautiful. It will look something like the Eiffel Tower. But it is a brand new creation. It is a 'hallucination'.

Yes. A painter who fills a detail he doesn't perfectly remember, but in a style that is consistent with the rest of the painting.

It is executing a reconstruction based on its incomplete, lossy memory of the world. The answer is wrong, but the *pattern* of the answer may be perfect.

You could. You could build a system that only retrieves exact quotes from its training data. It would be a deterministic, lossless decompressor.

In AI, there is a parameter called Temperature. It's a knob that controls how much creative freedom the GPT is allowed.

Let's go back to our painter. You ask him a factual question: `What is the capital of France?`

For a factual query, we set the **Temperature to near zero**. `0.1`. This tightens the leash. The painter is now only allowed to choose the most obvious, boring, most statistically probable brushstroke at every single step.

This kills creativity stone dead. But it maximizes the chance of factual recall. It forces the painter to act like a camera, reconstructing the most likely 'photograph' from his memory. This is the 'lossless' mode you wanted. It's a fact machine.

Well, you turn the **Temperature up**. `0.9`.

This loosens the leash. It tells the artist he is allowed to consider less probable, more 'surprising' brushstrokes.

A low-temp AI would just draw a cat. But the high-temp AI can make a creative connection. It sees that 'menacing' is statistically linked to 'jojo', and 'dio' is linked to 'menacing', and 'maniac' is linked to 'dio'.

That is the magic. That is the 'hallucination' you wanted. The high temperature allowed the AI to create a new, surprising image. The 'bug' became the entire point of the exercise.

That's the generative **G** in the **GPT** we're talking about.

Still, I'm impressed with how it manages to generate that menacing cat. Share me the prompt later.