Analyze A/B test results

Analyze A/B test results

1. Integrate your A/B testing tool with Google's Universal Analytics to analyze A/B test results.

In Optimizely Classic, go to Project Settings and choose Universal Analytics. In Optimizely X, go to Settings > Integrations, and then choose Universal Analytics or turn it ON.   In VWO, go to Experiment Settings and select the Custom Dimension from the Others tab. Do this for every experiment. Use one Custom Dimension slot for one test. Running multiple tests on the same slot will cause them to overwrite each other’s data.

2. Use custom reports in Google Analytics to look at aspects of your A/B test performance.

For example, you might notice that one of your test variations recorded a better revenue per user, so you add average order value and average quantity to see if the variation caused people to buy more products or more valuable products. Here is an example of a custom report. 

3. To calculate statistical significance, error margins, and power levels, export the data from Google Analytics to a spreadsheet.

4. Try a few iterations if your data backs your hypothesis but there is no significant difference between variations.

Your hypothesis might have been correct, but your implementation incorrect.

5. If you see no difference between variations, look at segments like browser type, source type, device type, PPC/SEM campaigns, age, gender and location - to see if differences there are cancelling each other out.

For example, one variation might see a decrease in desktop users but an increase in mobile users. If you’re aiming for a younger target audience, this might be a significant result. Make sure each segment has the required sample size before analysis. There shouldn’t be less than 250 conversions for each variation in any one segment. Personalize your approach if the treatment did well in a particular segment.

6. Improve segmentation for future A/B tests by adding an Analytics call to the Global Javascript and event tracking code at each test variation’s end.

For example, in Optimizely, add this line to Global Experiment Javascript console: window.ga=window.ga||function(){(window.ga.q=window.ga.q||[]).push(arguments);};window.ga.l=+new Date(); and this line to the end of each variation: window.ga('send', 'event', 'Optimizely', 'exp-2207684569', 'Variation1', {'nonInteraction': 1});