Consider A/B testing for website optimization
Consider A/B testing for website optimization
1. Decide how many versions of your website you want to be able to show to visitors.
If you add customizations that mean every visitor sees a slightly different website, this will skew your A/B testing results.
2. Look at how many ideas you want to test, and decide whether you have the time and traffic to A/B test them one at a time.
You test a single idea at a time with A/B testing. For example, the headline or the call to action. Depending on the amount of traffic your website has, it’s not unusual to wait weeks or even months to see results from A/B tests.
3. Calculate how often, on average, you make changes to the website or advertising that leads people to it.
When you run a promotion, change your ad targeting, or adjust messaging, the visitors you attract to your site may behave differently than your existing visitors. When you run an A/B test, you optimize for the visitors you have today. As your visitors change, your website will remain the same, optimized for the point in time when you ran your A/B test, quietly costing you conversions without you realizing it.
4. Consider whether the loss:reward ratio of losing conversions during A/B testing is worth it.
An A/B test gathers data by serving one experience that will eventually win to 50% of your website visitors and another experience that will eventually lose to the other 50%. While your test is running, you’re missing out on conversions by showing an underperforming variation half the time.
5. Think about whether you need to explore how different page elements interact to boost conversions.
A/B testing will not reveal any insights about interactions between elements on a page, such as how a certain headline might impact which CTA is most effective.
6. Explore the statistics literacy of the team who would be running the tests.
If you don’t know how to interpret statistical results, you may make the wrong decision. It’s easy to incorrectly call a winner or a loser. Marketers do it all the time by checking for statistical significance every day and then acting on it immediately. This is called the “early peeking problem” for most systems that use the typical fixed horizon statistics that many people learn in college.
7. Check that your engineering department has the resources to support the testing and integrate winning variations.
Once marketers find a statistically significant winner, they need to ask engineering to code the winner into the base site.