A/B test calculator
Two sponsor creatives, two subject lines, two buttons — one got more clicks. Put in the numbers and find out whether that gap is real or just noise, and how much longer an undecided test needs.
“Impressions” is whatever each variant was shown to — opens for an in-email creative, sends for a subject line. Use the same measure for both.
No winner yet
No clear winner. The gap between “Creative A” and “Creative B” is within what chance alone would produce (p = 0.27).
If the true difference is as big as it looks now, you’d need about 12,743 impressions per variant to detect it reliably — 10,743 more than the smaller arm has so far. If the real gap is smaller than it looks, you’ll need more.
Click rate with 95% interval
Overlapping bars don’t settle it on their own — the test below compares the two directly.
- Difference
- +0.70 pts
- 95%: −0.54 pts to +1.95 pts
- Relative lift
- +18.4%
- Creative A vs Creative B
- p-value
- 0.267
- not below 0.05
- Per variant
- 2,000
- smaller arm · 1,000 needed for a verdict
How it decides
- No rate under 200 impressions. Below that, a click rate is mostly noise, so it isn’t shown at all.
- No winner under 1,000 per variant, however big the gap looks.
- Then a two-proportion z-test. A winner is named only when p is below 0.05 — when a gap this size would turn up less than one time in twenty if the two were really the same.
- Decide your sample size first. Checking repeatedly and stopping the moment it looks significant finds “winners” that aren’t.
Same thresholds and the same two-proportion test as Carrier Crow’s sponsorship report.
Read the guide
Is your sponsor's A/B test result real?
Why 4.1% against 3.8% on a few hundred impressions is noise, and how to tell a real difference between two sponsor creatives from luck.