A/B Test Calculator — Statistical Significance & Sample Size
Check whether an A/B test result is statistically significant, and plan the sample size and duration of the next one.
Control (A)
Variant (B)
Runs entirely in your browser. Nothing you enter is sent to us or stored. Browse all free tools.
Found a problem, or something that could be better? We read every message and we fix things quickly.
This A/B test calculator tells you whether the difference between two conversion rates is statistically significant, or likely to be noise. Enter visitors and conversions for the control and the variant, and it reports the p-value, the confidence level, the relative uplift and a confidence interval for that uplift, which is the number that tells you how big the real effect could plausibly be.
The second tab plans a test before you run it: from your baseline conversion rate and the smallest effect worth detecting, it calculates the visitors needed per variant and how many days that takes at your traffic. The statistics are the standard two-proportion z-test and the sample-size formula used by Evan Miller’s widely cited calculator, and the code is checked against his published examples. It assumes a fixed sample size decided in advance; if you check results daily and stop when they look good, no calculator can make that result reliable.
How to use
- In the significance tab, enter visitors and conversions for control (A) and variant (B).
- Pick a confidence level (95% is standard) and two-sided unless you decided before the test to only look for improvement.
- Read the verdict, p-value and uplift interval. If the interval includes 0, the test has not shown a difference.
- To plan a test, switch to the sample size tab, enter your baseline rate and minimum detectable effect, and add daily traffic to get the duration.
Frequently asked questions
How do you calculate statistical significance for an A/B test?
For conversion rates, the standard method is a two-proportion z-test. Compute each rate, the pooled rate p = (conversions A + conversions B) ÷ (visitors A + visitors B), and the standard error √(p(1 − p)(1/nA + 1/nB)). Then z = (rate B − rate A) ÷ SE, and the two-sided p-value is 2 × (1 − Φ(|z|)). If the p-value is below your significance level, usually 0.05, the result is significant.
What does a p-value of 0.05 mean?
If there were truly no difference between A and B, you would see a gap at least this large about 5% of the time by chance. It does not mean there is a 95% chance B is better, and it says nothing about how large the effect is. That is why this tool also shows the uplift confidence interval.
How many visitors do I need for an A/B test?
It depends on your baseline conversion rate and the smallest effect you care about. At a 3% baseline, detecting a 10% relative lift (3.0% to 3.3%) with 80% power at 95% confidence needs about 51,500 visitors per variant. Detecting a 20% lift needs roughly a quarter of that. Small effects on low conversion rates need a lot of traffic, which is why low-traffic sites should test bigger changes.
Should I use a one-sided or two-sided test?
Use two-sided unless you decided before starting that you only care whether B is better and would treat "B is worse" the same as "no difference". A one-sided test reaches significance with less data, which is exactly why switching to it after seeing the results is a form of cheating. Most testing tools default to two-sided.
Can I stop an A/B test as soon as it is significant?
Not with this kind of test. Checking repeatedly and stopping at the first significant result can raise the false-positive rate from 5% to 20–30% or more. Fix the sample size in advance with the planner, run for whole weeks, and evaluate once. If you need to monitor continuously, use a sequential testing method designed for it.
What is statistical power?
Power is the probability that the test detects a real effect of the size you specified. 80% is the usual standard, meaning a real lift of that size would be missed one time in five. Higher power needs more visitors. A non-significant result from an underpowered test does not show that there is no effect; it shows that the test could not tell.
Reading the result correctly
Look at the uplift interval before the verdict. A result of "+15% uplift, 95% CI −2% to +34%" is not significant, and it also tells you the variant could plausibly be slightly worse or a lot better: you need more data. "+15%, CI +6% to +25%" is significant and tells you the smallest credible effect is still worth having.
• Check for sample ratio mismatch: with a 50/50 split, visitor counts should be close. A large imbalance usually means a tracking or redirect bug, and invalidates the test.
• Run for at least one full week, ideally two, to cover weekday and weekend behaviour and avoid novelty effects.
• Count each visitor once per variant. Counting page views instead of users inflates the sample and makes results look more certain than they are.
• Testing many variants or many metrics multiplies false positives. Use the Bonferroni option or pick one primary metric in advance.
The method used
Significance uses the pooled two-proportion z-test, which for two groups is equivalent to a chi-squared test without continuity correction. The confidence interval for the absolute difference uses the unpooled standard error; the interval for relative uplift uses the delta method on the log of the rate ratio, which stays sensible when rates are small.
Sample size uses n = (z₁₋α/₂ √(2p(1 − p)) + z₁₋β √(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ δ², the same formula as Evan Miller’s calculator. The normal distribution functions are implemented to full double precision, so p-values in the tails are accurate rather than rounded to zero.
The z-test relies on a normal approximation. With fewer than about 30 conversions in either group the tool warns you, because the approximation becomes unreliable.
Related free tools
- CPM Calculator — Work out CPM, ad spend or impressions from the other two, plus CPC, CPA, CTR and conversion rate on one page.
- ROAS Calculator — Calculate ROAS, the break-even ROAS your margin requires, max CPA, LTV, CAC and payback period.
- Email Subject Line Tester — Score a subject line for length, spam trigger words, capitals and punctuation, and preview how it truncates on iPhone and in Gmail.
- UTM Link Builder — Build clean, consistent UTM tracking links so you know exactly which campaigns bring you customers.