A/B test calculator: significance and sample size

How to tell whether your A/B test has a winner

An A/B test compares two versions of a page, ad or email to see which converts better. The risk is calling a winner too early: with few visitors, a gap of one or two points can be pure chance. This tool answers the two questions that matter: is the difference you saw real, and how many visitors do you need to see the difference you are looking for.

Everything is computed in your browser. The numbers you type are never sent anywhere.

1. Enter visitors and conversions for A and B. Use the totals for the whole test period. A conversion is the action you measure: a purchase, a signup, a button click.
2. Pick the confidence level. 95% is the industry default: it accepts a 1 in 20 chance of seeing a difference that does not really exist.
3. Pick two-sided or one-sided. Two-sided asks whether there is a difference in either direction and is the safer choice. One-sided only asks whether B is better and must be decided before you look at the data.
4. Read the p-value and the interval. If the p-value is below 1 minus the confidence (0.05 for 95%), the difference is significant. The interval shows the likely size of the lift; if it includes zero, the result is inconclusive.
5. Plan the next test in the sample size tab. Enter the current rate, the smallest lift worth detecting (MDE) and the power. The tool returns how many visitors each version needs.

Everything is calculated here, in your browser. Nothing you type is sent to any server and we use no third-party cookies.

How the calculation works

Significance uses the two-proportion z-test. Assuming A and B convert equally, a single pooled rate is estimated and the distance between the two rates is measured in standard errors. The p-value is the probability of seeing a distance at least that large by chance alone.

The confidence interval of the difference uses the unpooled standard error, the same convention as Stata's prtesti command. The standard normal is computed with the error function, not with rough lookup tables.

Sample size uses the classic Fleiss formula for two proportions, the same one as Stata (power twoproportions). Checked against the Stata manual: rates of 0.001 and 0.015 with 80% power and 5% two-sided need 635 visitors per group.

• RatespA = conversions A ÷ visitors A; pB = conversions B ÷ visitors B
• Pooled ratep = (conversions A + conversions B) ÷ (visitors A + visitors B)
• z statisticz = (pB − pA) ÷ √( p × (1 − p) × (1/nA + 1/nB) )
• Two-sided p-value2 × P(Z ≥ |z|), with Z standard normal
• Confidence interval(pB − pA) ± zc × √( pA(1−pA)/nA + pB(1−pB)/nB )
• Sample per groupn = ( zα × √(2 p̄ (1−p̄)) + zβ × √(p1(1−p1) + p2(1−p2)) )² ÷ (p2 − p1)²

Examples