Avoid the narrative fallacy in optimization
1. Focus exclusively on the A/B/n test data to promote rational discussion about the actions that can be taken.
The data available is positive or negative relative to a change, and independent of why you thought you made a change. Focus on the actions you can take, based on your data, to address the larger issues. To maximize efficiency, instead of concentrating on what you wanted to happen, focus on what did happen and what is best for the business moving forward. Avoid focussing on your test ideas as it prevents you from letting the data inform your next steps.
2. Focus on the beta of multiple options, to expand the likelihood of scale and positive outcome.
For example, bring a small, diverse group of individuals together and collect their ideas. This prevents too much focus on a single area, creates a larger range of ideas, and stops the test from being dependent on the value of any single idea.
3. Address a specific issue by optimizing towards a single success metric.
Start every conversation by determining the single measure of success. For example, you can start your conversations with these questions: What is the end goal? Are we trying to improve revenue? Are we trying to improve conversion rates? Establish agreement on the end goal, then center all future conversations and actions around it.
4. Determine if you can act on the results of your test, and evaluate the efficiency of your original test setup.
Avoid theorizing before you have data and instead, discuss what it is possible to do. Resist post-hoc rationalizations as they can cause data to be ignored and prevent rational discussion of the actions that can be taken. Discuss rationally whether or not you can act on your results and move on to the next test. Analyze the efficiency of your test.
5. Choose the best performing option and look for descriptive patterns.
6. Create a system that accounts for your own biases.
Recognize your own biases and note that you can not eliminate them. With the knowledge of your biases in mind, you can develop a system that is focused on efficiency.