A/B Test Statistical Significance Calculator

Check whether your A/B test result is statistically significant. Enter visitors and conversions for each variant; the calculator runs a two-proportion z-test and returns each conversion rate, the relative uplift, the p-value, the confidence level, and a 95% confidence interval on the uplift. Use it after a test finishes, reading significance mid-test and stopping early inflates false positives.

Worked example

Control (A): 1,000 visitors, 100 conversions, a 10.0% rate. Variant (B): 1,000 visitors, 120 conversions, a 12.0% rate and a +20% relative uplift. Looks like a clear win, but the z-test gives p = 0.15: a difference this large would appear about 15% of the time even if the variants were identical, and the 95% confidence interval on the uplift runs from roughly -7% to +47%. Verdict: not significant, the test needs more traffic before you can call it.

How the maths works

The calculator runs a two-proportion z-test: the pooled conversion rate estimates the standard error under the null hypothesis, z is the observed rate difference divided by that standard error, and the two-sided p-value comes from the standard normal distribution. The 95% confidence interval on the uplift uses the unpooled standard error. The optional Bayesian view models each variant's rate as a Beta posterior with uniform priors and reports an approximate probability that B genuinely beats A.

Frequently Asked Questions

What statistical test does this calculator use?

A two-proportion z-test: pooled standard error for the test statistic and a two-sided p-value, with an unpooled 95% confidence interval on the uplift. It is the standard frequentist test for comparing two conversion rates.

What does the p-value mean?

The probability of seeing a difference at least this large if the two variants actually converted identically. A p-value below 0.05 is the conventional threshold for calling a result significant.

What does the Bayesian probability mean?

An approximate probability that variant B genuinely beats variant A, computed from Beta posteriors with uniform priors. It answers the more intuitive question, but treat it as an approximation, not a precise figure.

The result is not significant. What now?

Either the variants genuinely perform similarly, or the test has not collected enough data to tell. Check the confidence interval: if it still spans a range you care about, the test is underpowered - use the sample size calculator before rerunning.

Is 95% confidence always the right bar?

It is the convention, but the right bar depends on the cost of being wrong. Cheap, reversible changes can ship at lower confidence; expensive or risky changes deserve a stricter threshold.

Related tools

Also try our A/B Test Sample Size Calculator and Revenue Per Visitor Calculator. Not sure what to test or fix first? Run our free CRO audit tool for a scored, 6-pillar report on your page in 3-5 minutes. Explore all free CRO tools.