NelworksNelworks

Finance / Season 2 / EP04

Why Backtests Lie

Shez

I'm retiring! I'm buying an island!

Shez

My Python script turned 1,000 into10 Million in the simulation!

Shez

It's a simple Moving Average Crossover. I downloaded a library, plugged in the data, and optimized the window size.

Shez

The perfect window is 14.2 minutes. It never loses!

Kurumi

price = df['close'].

Kurumi

if price > average: buy_at(df['open']).

Shez

Genius strategy! You gotta pay me to license it!

Kurumi

You are using the Close price of the bar to decide whether to buy at the Open of the same bar.

Shez

So?

Kurumi

The Close happens after the Open!

Kurumi

You are deciding to bet on the horse after you watched it win, then traveling back in time to place the bet at the start.

Kurumi

That is Look-Ahead Bias.

Shez

Oh. So... I cheated?

Kurumi

You built a causality violation engine.

Kurumi

Fix it. Use shift(1). Make decisions only on yesterday's data.

Shez

Okay, $1 Million profit. Still good!

Shez

The strategy is still solid!

Kurumi

You tested 5,000 different parameter combinations and picked the best one, right?

Shez

Yeah! Optimization!

Kurumi

Looks like you gotta lose some money to learn this lesson.

Kurumi

You didn't find a strategy. You tailored a suit that fits this specific mannequin perfectly.

Kurumi

You stitched it so tight it can't move. This is Overfitting.

Kurumi

The moment the market moves differently than the past (which it always does), your 'optimized' strategy explodes.

Shez

So... how do I know if it fits a real human?

Kurumi

Out-of-Sample Testing.

Kurumi

Train on 2018-2020. Lock the parameters.

Kurumi

Test on 2021.

Kurumi

If the line goes up in 2021, you might have something.

Shez

It... it died.

Kurumi

It reverted to the mean. You mined noise, not signal.

Shez

So the library lied to me.

Kurumi

The library did exactly what you asked. It found the best curve to fit the past.

Kurumi

But the past is a ghost. You can't trade ghosts.

Kurumi

And even if it worked... look at the fees.

Kurumi

Death by a thousand cuts.

Shez

I'm not giving up!

Shez

In 2 days I will find a strategy that actually makes money! You'll gonna bet with me on that!

Kurumi

Heh. Bet. You really had turned into a degen gambler.

Shez

2 days later...

Shez

Hahahaha! I found a strategy that works! A real one!

Shez

I locked it in this cold storage vault. No one can steal the code. It's safe forever! I can set and forget and print money forever!

Kurumi

Safe? Shez, look at the floor. Your profit is leaking.

Shez

It's melting! But how? The door is locked!

Kurumi

It's not melting because of the heat. It's melting because of Alpha Decay.

Kurumi

In Quant Finance, you don't own the patent. You only own the Lead Time.

Kurumi

The moment you start trading, you are telling the world your secret.

Shez

But I'm using encryption! VPNs! No one sees my code!

Kurumi

They don't need your code. They see your Footprints.

Kurumi

Every order you send moves the price. You can't step without leaving a print.

Kurumi

They read your prints, work out where you're going, and get there first.

Kurumi

Your edge doesn't get stolen. It gets front-run.

Shez

So... by using the strategy, I destroy it?

Kurumi

The Market is an Efficient Learning Machine.

Kurumi

An 'Arbitrage' is a mistake in the market. You are paid to fix it.

Kurumi

Once it's fixed... you stop getting paid.

Shez

This sucks. Fintech companies build a payment app once and charge fees for 50 years.

Shez

Why do we have to reinvent the wheel every Tuesday? I should just quit being a quant and work for fintech instead.

Kurumi

That is why we don't protect the Product (The Strategy).

Kurumi

We protect the Factory (The Research Pipeline).

Kurumi

We assume every strategy will die in 6 months.

Kurumi

So we build infrastructure to test, validate, and deploy new strategies faster than the old ones die.

Shez

It's a treadmill.

Kurumi

It's the Red Queen's Race. You have to run as fast as you can just to stay in the same place.

Kurumi

You want to protect your profit?

Kurumi

Don't hide. Innovate.

Kurumi

Find new data sources. Faster FPGAs. Better math. More creative market exploitation.

Shez

It sounds exhausting.

Kurumi

Passive income is a myth. Every dollar is earned by someone's hard work.

Shez

Okay. New strategy. I'm creating fake immigrant profiles to apply for government grants and income.

Kurumi

That's regulatory arbitrage and it means it's illegal.