How long should an A/B test run?

Test duration is not a matter of taste. It follows from four things you can know in advance: your baseline conversion rate, the smallest improvement worth detecting, how confident you want to be, and how much traffic reaches the test.

The four inputs

• Baseline conversion rate. What the control currently converts at. Lower baselines need more traffic.
• Minimum detectable effect (MDE). The smallest relative improvement you would actually act on. This is a business decision, not a statistical one — and it is where most teams go wrong, by hoping to detect a 2% lift with traffic that can only resolve 15%.
• Significance level. Conventionally 95%, meaning a 5% chance of declaring a winner that is not one.
• Statistical power. Conventionally 80%, meaning a 20% chance of missing a real effect. Power is routinely ignored and is why so many "inconclusive" tests were simply underpowered.

The rule of thumb, and its limits

A widely used approximation for sample size per variant, at 95% significance and 80% power, is:

n ≈ 16 × p × (1 − p) / δ²

where p is the baseline conversion rate as a decimal and δ is the absolute improvement you want to detect. For a 3% baseline and a desired detection of half a percentage point (to 3.5%), that gives roughly 16 × 0.03 × 0.97 / 0.005² ≈ 18,600 visitors per variant.

Use this to sanity-check feasibility before designing a test, not as a substitute for a proper calculator. It assumes a simple two-variant test with a binary outcome, and it says nothing about the practical constraints below.

The floor nobody should go under: full business cycles

Even when the arithmetic says you have enough traffic in three days, run for at least one full week, and preferably two. Purchase behaviour varies enormously by day of week, and a test that ran Tuesday to Thursday has measured Tuesday-to-Thursday visitors, not your customers.

If your sales cycle means people convert a week after first visiting, the test needs to run long enough for those conversions to land, or you will be attributing them to the wrong period entirely.

Why calling early is so damaging

Conversion rates fluctuate. Early in a test, the observed difference between variants swings wildly, and it will frequently cross the significance threshold by chance alone. If you check daily and stop the first time you see 95%, your actual false positive rate is far higher than 5% — realistically closer to 20–30% depending on how often you look.

This is why the stopping rule has to be fixed before launch. Either commit to a fixed sample size decided in advance, or use a sequential testing method explicitly designed to allow monitoring. What you cannot do is use fixed-horizon statistics and then peek.

What to do when you do not have the traffic

This is the common case, and pretending otherwise is how programs waste a year. Options, roughly in order of usefulness:

• Test bigger changes. A large MDE needs far less traffic. If you can only resolve a 20% effect, test things that could plausibly move it 20%.
• Move up the funnel. Test on a higher-volume step — the page before the conversion often has ten times the traffic.
• Use a proxy metric carefully. A higher-frequency event closer to the change, provided you have evidence it correlates with revenue.
• Accept lower confidence deliberately. For a cheap, reversible change, 80% confidence may be a perfectly rational business threshold — as long as it is a stated decision rather than an accident.
• Stop testing and do research. If you genuinely cannot reach significance on anything meaningful, qualitative research will tell you more per hour than an underpowered test.

Related

• What conversion rate optimization actually is
• How we run the test lifecycle