Consider multivariate testing for website optimization
Consider multivariate testing for website optimization
1. Decide whether it’s acceptable to only show one version of your website to visitors at a time.
If you add customizations that mean every visitor sees a slightly different website, this will skew your A/B testing results.
2. Look at the elements that you want to test, and decide whether you have the time and traffic you’ll need to test all of them.
Testing multiple variations of different elements on a page requires exponentially more traffic than using separate A/B tests. For example, an MVT of three different page elements with two new versions of each requires the same traffic as nine individual A/B tests. As you test more variations, the traffic – and therefore the time – needed goes up exponentially.
3. Weigh the benefits of getting data on how different elements interact and the increased time and traffic requirements compared to A/B testing.
MVT is powerful for determining which combination of elements on a page will maximize conversion, taking into account the interaction between elements. Just because you can do it doesn’t mean you should, though. For example, if you’re in a fast-moving industry, you might find that your data is almost out of date by the time you can apply it.
4. Check that your engineering department has the resources to support the testing and integrate winning variations.
Once your team find a statistically significant winner, they need to ask engineering to code the winner into the base site.