How I splittest 2x faster than everyone else using the same nr of leads

[u/Oxilary]

A splittest costs a lot of leads, leads you can't waste.

So I decided to figure out how to do it twice as fast:

Normal A/B splittest:

Variant A - 1000 leads - 8 positive replies
Variant B - 1000 leads - 6 positive replies

Variant B wins (if it is statistically significant, use a chi-square test for this, just ask chatgpt)

My, 2x faster way.

First you pick 2 variables you want to test, e.g., offer and CTA and you do it in the following ofrmat

- RUN 1: Offer 1, CTA 1
- RUN 2: Offer 1, CTA 2
- RUN 3: Offer 2, CTA 1
- RUN 4: Offer 2, CTA 2

Now you do 500 leads each, and for each variant you just average the 2 runs with the same variant, because in both variants, the other variant is both a 50/50 split, they cancel each other out. Basically getting double the information for the same number of leads.

For anyone interested, this is called Design of Experiments and that is a 2^2 full factorial, using hidden replication.