“How long should we run this test?” is the most common question in experimentation, and the most common wrong answer is “until it’s significant”. The right answer is decided before the test starts, from three numbers you already have.
The short answer
Run a test until it reaches the sample size you calculated in advance, and for at least one full week, rounded up to whole weeks. Then stop and read the result once.
Everything below explains where that sample size comes from, and why the “whole weeks” rule matters.
Step 1: Work out the sample size
The number of visitors a test needs depends on:
- Your baseline conversion rate. Rarer events need more traffic to measure.
- The minimum detectable effect (MDE). The smallest improvement you care about. Halving the MDE roughly quadruples the sample you need.
- Significance and power. The usual defaults are 95% significance and 80% power.
For example, with a 3% baseline and a 10% relative MDE (detecting a move from 3.0% to 3.3%), a standard two-variant test needs about 53,000 visitors per variant, around 106,000 in total. Try your own numbers in our A/B test sample size calculator.
Step 2: Turn sample size into days
Divide the total sample by the daily traffic that will actually enter the test. With 2,000 eligible visitors a day, the example above takes 54 days. With 20,000 a day, it takes 6 days, which you would round up to 7.
Use the traffic that reaches the tested page and meets your targeting, not your total site visits. A test on the checkout page only sees people who reach checkout.
Step 3: Round up to whole weeks
Behaviour changes through the week. B2B tools are busy on weekdays; shops and travel sites often peak at weekends; paydays and email campaigns create spikes. A test that runs Monday to Thursday measures Monday-to-Thursday users. Running whole weeks, a minimum of one and preferably two, gives every day the same weight in both variants.
Why you shouldn’t stop when it turns significant
Checking results every day and stopping the first time the p-value dips below 0.05 is called peeking. Each look is another chance for random noise to cross the line, so the real false-positive rate ends up far higher than the 5% you think you are accepting. Evan Miller’s classic essay How Not To Run an A/B Test explains the problem clearly.
If you genuinely need to stop early, use a method designed for it: sequential testing, which several platforms now offer (Statsig, for example, includes it on its free plan; see our tools comparison).
When the answer is “too long”
If the calculation says three months, don’t run a three-month test and hope. Instead:
- Test a bolder change that could plausibly move the metric by more.
- Pick a more frequent metric higher up the funnel, as long as it predicts the outcome you care about.
- Cut variants. Two variants beat four on a small site.
- Use qualitative research instead: five user interviews or a few hours of session recordings will teach you more than an underpowered test. Start with a minimum viable test.
Checklist
- Sample size calculated before launch, from baseline, MDE, significance and power.
- Duration based on eligible traffic, rounded up to whole weeks (at least one).
- Stop date written down, and results read once at that date.
- Avoid launching during unusual periods (big sales, outages) unless that is what you want to measure.