A/B Testing Network Effects and SUTVA Violations

Define 'Success.'

Wait. Why did the Control Group's activity spike by 5% too?

It's not magic, Shez. It's **Contamination**.

Your petri dishes are touching.

Meet Alice (Odd ID, Treatment). Meet Bob (Even ID, Control).

Alice gets the 'Double-Coin' offer. She wants coins. What does she do?

Exactly. She invites Bob.

Bob is supposed to be in the 'No Feature' universe.

But suddenly, he gets a notification: 'Alice sent you Coins!'

Now Bob is reacting to the Treatment.

He clicks the link. He engages. Maybe he even gets the coins if your backend is lazy.

Correct. And if the Control goes up, your 'Lift' (Treatment minus Control) gets smaller.

You are actually **Underestimating** your success because you broke the silence.

In this current case? Maybe.

But what if it's a **Competition**?

Imagine we are Uber. We give Driver A a bonus. Driver A drives faster and takes all the customers.

Driver B didn't just stay neutral. Driver B actively **lost** money because Driver A existed.

This is **Negative Spillover**.

You calculate: Treatment (High) - Control (Super Low).

You think the bonus is a miracle. But really, you just cannibalized your own fleet.

Your math assumes **Independence**. Your users are a **Network**.

You violated **SUTVA**.

Stable Unit Treatment Value Assumption. You must have skipped that class. Think of it as the 'Food Coloring in Water' rule.

Standard A/B testing assumes everyone gets their own glass of clear water.

But a Social Network is like a giant punch bowl. If you add food coloring to one part (Treatment), it spreads—everyone's drink changes color.

No. But you can pour Alice and Bob's drinks into the same cup.

Instead of randomizing *Users*, you randomize *Communities*.

New York City = Treatment.

Chicago = Control.

Alice in New York talks to Bob... who is usually also in New York.

The spillover stays inside the bucket. The contamination is more likely to be contained.

Correct. You lose **Statistical Power**.

Since you only have N=2 (Two cities), your error bars are massive. You need lots of clusters (many cities) to make this work.

Then you use the Fourth Dimension.

**Switchback Testing**.

You toggle the feature on and off for the *entire* city at once.

9 AM: Everyone gets the Bonus.

10 AM: Nobody gets the Bonus.

You compare the *aggregate* performance of the Red hours vs. the Blue hours.

That's **Carryover Effect**.

The solution is you delete the data from 9:55 to 10:05. You let the system 'wash out' between cycles.

It's worse than useless. It gives you false confidence.

You think you have a 20% lift, so you roll it out globally.

Then, when everyone has it, the 'novelty' wears off, the competition settles, and your actual revenue stays flat.

And you blame the Engineering team for 'implementation issues.'

If you have a social graph, run a clustering algo (like Louvain). Group tightly connected friends together.

Treat the *Cluster* as the Unit.

But your **Truth** went from 0% to 100%.

Because you're in the same 'Office Cluster' as me.

And I'm in the Treatment group.

I wanted the coins.

Hurry up and accept the invitation so I can get the coins.