Analyze A/B test results
1. Take note of the test date, so you can extract the corresponding data from your analytics platform.
Export only the results from the duration of the experiments. For example, you might need data on users that converted between March 20th 2020 and June 20th 2020.
2. Create a segment for users that have seen the test variation - the experiment group - and users who reached the test goal after seeing the test variation.
Identify whether the test was run on the entire segment or only on a specific type of population; for example, new users.
3. Export data from your testing tool into Google Analytics, assign a User ID to track user activity in your tests, and check that the number of users is the same for every variation.
Record the total users that converted and not the total conversions during the test. If you are using Google Analytics, avoid sampling because it creates more sets of data in the output. Look for obvious mismatches in the user conversion ratio.
4. Track secondary metrics like average cart value and the average quantity of products bought by analyzing micro-conversions such as buttons and add to carts to learn how user behavior changed during the test.
Use the micro-conversion results to change your tests and encourage that type of user behavior. If you can’t find significant observations, either discard your test or implement the variation if it was a winner.
5. Create a two-step segment in Google Analytics with a first step of a unique user seeing the test page on a specific device and a second step of matching a transaction with the same user who saw the test variation.
The segment gives you a user who went through the experiment and converted, having experienced the entire process.
6. Build a custom report in Google Analytics and add the segment for users and users that have converted to identify significant results in your tests.
If you already have a spreadsheet with data from previous tests, automate an export using the Google API.
7. Enter the number of test participants and number of conversions to an A/B test guide calculator to calculate your test's confidence.
Choose a 95% confidence level to show if the results of your tests are significant. If the overlap of the results are big, you cannot conclude that B is better than A. If the test results are inconclusive, it does not mean that your hypothesis was flawed. It means that your variant cannot perform better than the control. You can implement the results without harming your campaign. Don’t search for segments that can show a significant result as they can provide a false-positive result.
8. Implement the winning variant of your test as soon as possible if your results are significant.
If the results are significant, that does not mean that your conversion will increase by X amount once your new variation is live. Analyze your segments to identify which behavior changed and use the findings in your next test.