A/B test performance

A/B test performance measures the difference in outcomes between two or more variants shown to randomly assigned groups of users. The control group sees the existing experience, the treatment group sees the proposed change, and the performance of each is compared on one or more predefined metrics.

The measurement has three core components: the observed difference (lift), the statistical significance of that difference (confidence level), and the practical significance (whether the lift is large enough to matter). A test can show a statistically significant difference that is too small to justify the engineering cost of implementing it, or a large observed difference that is not statistically significant because the sample size was insufficient.

Statistical significance is typically expressed as a p-value or confidence interval. A p-value below 0.05 (95% confidence) is the conventional threshold, meaning there is less than a 5% probability that the observed difference occurred by chance. However, the threshold should be adjusted based on the cost of a wrong decision. High-stakes changes (pricing, core workflows) may warrant 99% confidence, while low-risk changes (copy tweaks, colour variations) may be actionable at 90%.

A/B test performance is not a single metric but a framework for evaluating changes. The specific metrics tracked depend on what the test is trying to improve: conversion rate, revenue per user, engagement, retention, or any other measurable outcome. The discipline is in the methodology: random assignment, adequate sample size, predefined success criteria, and honest interpretation of results.

A metric tree connects individual test results to the business metrics they are intended to influence, creating a structured view of the experimentation programme's overall impact.

This tree shifts the focus from individual test results to the performance of the experimentation programme as a whole. A mature programme runs many tests, accepts that most will not produce significant results, and measures its value by the cumulative impact of the winners that are shipped.

Connecting test results to downstream business metrics like revenue growth rate or retention rate quantifies the ROI of the experimentation programme. This makes it possible to justify investment in experimentation infrastructure and team capacity based on measured business outcomes.

1. 1

   Peeking at results and stopping early

   Checking results before the test reaches its required sample size and stopping when the result looks good inflates false positive rates. Sequential testing methods exist to allow valid early stopping, but the standard fixed-sample test must run to completion.

2. 2

   Testing too many variants without adjusting for multiple comparisons

   Running five variants simultaneously without correcting for multiple comparisons dramatically increases the chance of a false positive. If you test five variants at 95% confidence, the probability of at least one false positive is roughly 23%, not 5%. Use Bonferroni correction or similar methods.

3. 3

   Ignoring practical significance

   A statistically significant result with a tiny effect size may not be worth implementing. Before launching a test, define the minimum detectable effect that would justify the change. If the observed lift is below that threshold, treat it as inconclusive regardless of the p-value.

4. 4

   Measuring the wrong metric

   A test that improves a proxy metric (clicks, pageviews) while degrading the true goal metric (purchases, retention) is a net negative. Define primary and guardrail metrics before the test. Guardrail metrics ensure that optimising one metric does not come at the expense of others.

5. 5

   Insufficient sample size

   Running a test on too small a sample produces noisy results that are unreliable. Calculate the required sample size before launching based on baseline conversion rate, minimum detectable effect, and desired statistical power. If the required sample is larger than available traffic, the test is not feasible at that sensitivity.

KPI Tree lets you model your experimentation programme as a metric tree that connects test velocity, win rate, and effect size to cumulative business impact. Each active test can be tracked as a node with its current lift, confidence level, and projected impact, giving leadership visibility into the experimentation pipeline.

Linking test results to the business metrics they target, such as funnel conversion rate, customer satisfaction score, or average order value, creates accountability between the experimentation team and business outcomes. When a winning test is shipped, its measured impact flows into the tree and contributes to the cumulative programme ROI.

The tree also helps with prioritisation. By modelling the potential impact of proposed tests alongside active ones, teams can allocate traffic and engineering resources to the experiments most likely to produce meaningful results.