A/B Test Prioritisation Planner

Decide which test to run first. Choose the page element you want to test and the business goal behind it, then enter your current conversion rate, weekly traffic, and the smallest relative lift you would act on. The planner returns a structured hypothesis, the sample size and whole-week duration the test needs at 95% significance and 80% power, the weekly conversions it can actually move, and a high, medium, or low priority rating with three starter hypotheses for that element. Use it when you have more test ideas than traffic and need to rank them by practicality before building anything. For the detailed numbers, use the A/B Test Sample Size Calculator and A/B Test Duration Calculator.

Worked example

Suppose you want to test the headline on a landing page that converts at 2.5%, receives 5,000 visitors a week, and you would only act on a relative lift of 20% or more (2.5% rising to 3.0%). The planner needs 16,774 visitors per variant, 33,548 in total. At 5,000 visitors a week that is 7 weeks, and the test touches about 125 conversions a week. Seven weeks is within the practical limit but longer than four, so the test is rated medium priority. Run the same idea on a page with 12,000 weekly visitors and the duration drops to 3 weeks with about 300 weekly conversions in play, which lifts it to high priority. These are illustrative inputs, not benchmarks.

How the maths works

The sample size uses the standard two-proportion formula: n = 2 x (z_a/2 + z_b)² x p(1 - p) / (p₁ - p₀)², where p₀ is your baseline rate, p₁ is the baseline multiplied by (1 + MDE), and p is the average of the two. With 95% two-sided significance z_a/2 is 1.96 and with 80% power z_b is 0.84, so the constant 2 x (1.96 + 0.84)² is 15.68. The result is rounded up to whole visitors per variant, doubled for a 50/50 split, then divided by weekly traffic and rounded up to whole weeks (minimum one week). Weekly conversions affected is simply weekly traffic multiplied by the baseline rate.

Priority is a deliberately simple rule built on those two outputs. A test that would run longer than 12 weeks is flagged as impractical and rated low regardless of anything else. Otherwise, a test that concludes within 4 weeks and touches at least 50 conversions a week is high priority; a test that concludes within 8 weeks is medium; anything longer is low. Duration captures how quickly you can learn, and weekly conversions captures how much volume a winning variant could move, which is why neither traffic nor conversion rate alone decides the rating.

The hypothesis, variants, and recommendations are generated from your chosen element and goal using fixed templates, not from your numbers. Treat them as a starting structure to rewrite with your own customer insight. The planner also does not judge whether your chosen change is likely to achieve the lift you entered; a 20% relative lift is ambitious for a small copy tweak and modest for a full layout change, so pick an MDE you genuinely believe the change could deliver.

Frequently Asked Questions

How is test priority determined?

From two things that actually matter: how much conversion volume the test touches each week, and whether it can reach significance in a practical timeframe at your traffic level. High priority means high volume and a conclusion within about 4 weeks.

What makes a good test hypothesis?

A specific change, an expected effect, and a reason: 'By [change], we will [effect], because [insight about your visitors]'. The planner generates this structure for you from your chosen test element and business goal.

How many tests should I run at once?

One per page or funnel step. Simultaneous tests on the same traffic contaminate each other's results unless you have the volume and tooling for proper multivariate testing.

Where do I get the exact sample size and duration?

Use our dedicated A/B test sample size calculator and duration calculator - the planner links to both. They use the same statistics (95% significance, 80% power) with full control over the inputs.

Related tools

Also try our A/B Test Sample Size Calculator and A/B Test Duration Calculator. Not sure what to test or fix first? Run our free CRO audit tool for a scored, 6-pillar report on your page in 3-5 minutes. Explore all free CRO tools.