Multivariate Testing (MVT)
Synonyms: MVT, multivariable testing, factorial testing, combination testing, A/B/n testing
Definition
Use cases
- The combo nobody tested: A team tests the headline, ships the winner, then tests the hero image. The best headline and image pair never ran together.
- The 6-week dud: A startup with 3,000 weekly visitors tests 3 headlines × 3 images × 2 buttons. That's 18 versions at about 170 visitors each per week. After 6 weeks, nothing is significant.
- The redesign mix-up: A PM uses MVT to compare the old checkout with a full redesign. The results can't say which flow is better.
How it's used in practice
- Run the math first: Multiply your variants (3 × 3 × 2 = 18) and put your baseline conversion rate into a sample size calculator. If the number is bigger than a month of traffic, run A/B tests.
- A/B first, MVT second: Pick the direction with an A/B test. Then use MVT to tune elements on the winner.
- Cap it at 2 or 3 elements: Pick the ones your analytics or user research flag as weak.
- Preview every combo: White text on a light image makes one combination unreadable and skews the result.
Challenges & limitations
- Traffic hunger: Every extra variant splits visitors further. Low-traffic products rarely finish a test.
- Opportunity cost: One slow MVT takes as long as several clear A/B tests.
- Small-detail bias: MVT pulls focus toward images and button styles, while copy, pricing, and features often move results more.
Free resources
- Multivariate vs. A/B Testing: Incremental vs. Radical Changes (NN/g) — Aurora Harley explains when MVT fits and why it's a poor match for comparing full redesigns.
- The Complex Nature of Running Multivariate Tests (Dynamic Yield) — 5 honest reasons to avoid MVT, from traffic needs to design bias.
- How to Conduct a Multivariate Test (Trevor Calabro) — step-by-step template for running and analyzing MVT in Excel or Google Sheets.
