Back to Glossary
A/B Testing
Synonyms: Split testing, bucket testing, controlled experiment, variant testing, multivariate testing (close cousin)
Do not index
Definition
A/B testing is a method where you show two versions of something (i.e. a page, button, copy, or flow) to different user groups at the same time to see which performs better.
It's the antidote to "I think this design is better" arguments. Rooted in quantitative user research, it gives you data to make decisions instead of opinions. Think of it as a controlled experiment for your conversion rate optimization efforts.
Use Cases
If you skip A/B testing, you risk launching a "rebrand" that accidentally tanks your revenue. Without data, you’re just guessing, and guessing is expensive.
- In a checkout flow: You’re torn between a single-page checkout and a multi-step progress bar. Instead of arguing in a meeting, you run a test. You find the multi-step version reduces form fatigue and increases completed purchases by 15%.
- Refining a Call-to-Action (CTA): A founder thinks "Sign Up Now" is too aggressive and wants to try "Get Started." You run a split test and discover that "Get Started" actually leads to higher quality leads who stay active longer.
- Optimizing "Above the Fold" content: You aren't sure if a hero image or a short explainer video will drive more sign-ups. By testing both, you discover the video increases time-on-page but the static image actually leads to 20% more immediate conversions.
How it's used in practice
- Define one variable: Change the CTA button color or the headline—not both. Testing multiple changes at once means you won't know what moved the needle.
- Set your success metric first: Clicks? Sign-ups? Time on page? Decide before you launch, not after you see the results.
- Run it long enough: A 2-day test is basically useless. Aim for statistical significance—most tools need at least 1–2 weeks and a few hundred conversions per variant.
- Let data win, not the HiPPO: (Highest Paid Person's Opinion.) This is the whole point.
- Document your losers: Failed tests are gold. They tell you what your users don't want, which is just as valuable.
Beware of the Novelty Effect. Users often click new elements simply because they are new, leading to a temporary spike in engagement. Always look at long-term retention data to ensure your "winning" design actually improved the experience rather than just surprising the user.
