Post-Selection Inference and Multiple Comparisons Problem

We cured what? The common cold? Death? Or just your boredom?

p = 0.001. That's very significant.

Shez. You tested 50,000 compounds.

If you roll a 1,000-sided die... 50,000 times... how many times will you roll a '1'?

Exactly. You rolled the dice 50,000 times. You got 50 'miracles' purely by chance.

And you picked the top 3 and showed them to me.

That is exactly the crime.

Imagine a cowboy shooting at a barn. He has no aim. He just sprays bullets.

Then, *after* seeing where the bullets hit, he paints a target around them.

You aren't a sniper. You're a fraud. You conditioned the target on the result.

You cannot calculate a p-value on the winners if you ignore the losers.

You are assuming the hypothesis was 'Is X-99 effective?' *before* you started.

But your hypothesis was actually 'Is *Any* of these 50,000 effective?'

If you use a 0.05 threshold... the probability of finding a fake 'cure' in this room is **100%**.

Your p-value is garbage. It's **Post-Selection Inference**.

You used the data to *generate* the hypothesis, then used the *same data* to *prove* it. That is **Double Dipping**.

Enter the **Winner's Curse**.

Imagine everyone has a 'True Skill' (Average Jump Height) + 'Luck' (Wind, Shoes, Breakfast).

The score you see is actually your real Skill plus Luck (Noise). Like: Score = Skill + Noise.

To be the absolute #1 winner out of 50,000, you need two things: High Skill... AND Massive Luck.

The winner is almost always the person who had the *most positive noise*.

Now, make him jump again.

No. His luck ran out. The noise went back to zero.

His score **Regressed to the Mean**.

This compound probably works a *little* bit. But your measurement says '90% efficacy.'

That 90% is likely 40% Reality + 50% Luck.

It will fail. Just like 90% of drugs do in Phase 2. Because you selected the outlier, not the truth.

You can screen. But you have to pay the tax.

Option 1: **The Bonferroni Correction**.

You divide your p-value threshold by the number of tests.

0.05 / 50,000 = 0.000001.

Correct. Bonferroni is brutal. It assumes the worst.

Option 2: **Data Splitting**. The Engineering Way.

You screened 50,000 compounds? Great.

Use the first 50% of the data (samples) to **Select** the winners.

Now, throw away all the data from Set A. Forget it exists.

Test those 3 winners on **Set B**.

In Set B, you are only testing 3 hypotheses. Not 50,000.

So you don't need a crazy correction. 0.05 / 3 is easy to beat.

Exactly. If X-99 was just lucky noise in Set A, it will likely be average in Set B. The lie will reveal itself.

X-99 was the Texas Sharpshooter's target.

Z-01 is a verified hit.

You found one because you stopped lying to yourself about the 49,999 failures.

Now run a replication study on a different day, with different reagents, by a different person.

Because **Batch Effects** are a whole other nightmare, but perhaps we will leave that for another time.