A/B test calculator
Enter visitors and conversions for each variant to see whether the difference is real or chance, how large it is, and how certain. Planning a test instead? The sample size tab shows how many visitors are needed to detect the change you care about, and how many days that takes at your traffic.
A · Control
Conversion rate 3.20%
B · Variant
Conversion rate 3.85%
Conversion rate with 95% interval
Overlapping intervals do not by themselves mean the difference is insignificant; the significance test compares the variants directly.
Variant B wins
B converts +20.3% relative to A, significant at 95% confidence (p = 0.0127).
Relative uplift
+20.3%
Confidence
98.7%
p-value
0.0127
z-score
2.492
Absolute difference · 95% interval below
+0.65 pp
+0.14 pp to +1.16 pp
Test settings
Detects a difference in either direction. The standard choice.
How the result is calculated
Significance uses a two-proportion z-test on the pooled conversion rate, the method used by most testing platforms for conversion metrics. The interval for each variant is a Wilson score interval, which stays accurate at low conversion counts.
Sample size uses the standard formula for comparing two proportions, without continuity correction. It assumes visitors are split evenly and each is counted once.
Decide the sample size before the test starts and read the result once it is reached. Checking daily and stopping at the first significant reading inflates the false-positive rate well beyond the stated confidence level.
Questions
What does statistically significant mean in an A/B test?
That a difference this large would be unlikely if the variants actually performed the same. At 95% confidence, a result is called significant when there is less than a 5% chance of seeing it from random variation alone.
What is a p-value?
The probability of observing a difference at least this large if there were no real difference. A p-value of 0.03 means a 3% chance. It is not the probability that variant B is better.
How many visitors does an A/B test need?
It depends on the baseline conversion rate and the smallest change worth detecting. At a 3% conversion rate, detecting a 10% relative improvement needs about 53,000 visitors per variant at 95% confidence and 80% power. Use the Sample size tab for your numbers.
Should I use a one-sided or two-sided test?
Two-sided in most cases. A one-sided test only asks whether B beats A, so it reaches significance sooner, but it cannot tell you that B made things worse.
How long should an A/B test run?
Until it reaches the planned sample size, and for at least one full week so every weekday is represented. Two full weeks is safer for businesses with weekly cycles.
More free tools
View allQR code generator
Generate static QR codes for links, Wi-Fi, contacts and email, with your logo and colours, exported as SVG or PNG. Codes never expire.
Open toolUTM builder
Build UTM campaign links for GA4 and see which channel each link reports under before it goes live.
Open toolWhatsApp link generator
Create a wa.me click-to-chat link with a pre-filled message, plus a QR code and a website button.
Open toolReady to Dominate AI Search?
Get started now
Allswap