Prioritize your A/B tests
1. List your hypotheses in a spreadsheet and score them on customer view, customer voice, past experiments, existing evidence in research, and how strong the relationship between what we're optimizing and what the company aims for.
Score each section with scores of high, medium, and low. Create a new column to include and score your hypothesis strength, using the outcomes and data you have.
2. Map each hypothesis to each page of your test and score it from high to low to decide which step of your funnel the hypothesis suits.
For example, some hypotheses in an ecommerce website are not suited for the checkout page, but only for the product page.
3. Calculate the number of users, the conversion rates, and the notable effect to identify the chances of finding a significant outcome in your tests.
Calculate the level of effort required to implement the outcome in your website, and score them from low to high.
4. Prioritize your templates using the minimal detectable effect, by selecting the weeks of testing, % of traffic, the number of variations, and the significance level.
Start with the MDE score that is the lowest in percentage, as shown in the image below. Take into consideration the type of page, and the number of unique visitors that go through the entire experiment (unique visitors). In the early stages of MDE, you assume that you have a similar effect on all the pages.
5. Create a new spreadsheet to include the page templates, weekly visitors, weekly conversions, conversion rate, and add the average effect on your hypothesis.
The average effect is available only after running tests on your templates, and having discovered winners after your tests. Calculate the power for each template of your tests. Include the expected week effect, and assign a prioritization rank using the existing data. Look at the power of a page, and the expected effect of identifying which template has the priority.
6. Create a roadmap by listing the name of your tests, their hypothesis, where the user is located in their journey, and the location or type of page you will be testing.
7. Map your experiments across several weeks and take into consideration the amount of time required for each experiment.
For example, if in your spreadsheet you indicate that the first checkout page needs to run for 3 weeks, then add a second experiment only in the 4th week.