Calculate statistical significance for absolute difference in conversion rates
1. Calculate the conversion rates (Pc, Pv) in each of the two test groups by dividing the number of conversions by the total number of observations.
If we use: Ec: number of conversions in the control group Ev: number of conversions in the variant group Nc: total number of observations in the control group Nv: total number of observations in the variant group Pc: conversion rate for the control group Pv: conversion rate for the variant group then calculate: Pc = Ec/Nc Pv = Ev/Nv For example, if the metric of interest is a user-based conversion rate, then the total number of observations in the control group is the number of users enrolled in the control, and the number of conversions is the number of conversions by users in the control group. If there were 10,000 users in the control group and 1,200 checkouts started, then: Pc = 1,200 / 10,000 = 0.12
2. Calculate the difference (Delta) between the two conversion rates by subtracting the conversion rate in the control group from the conversion rate in the variant group.
Following the same notation, the difference in conversion rates would be: Delta = Pv - Pc Following the example above, Pv = 0.12 and Pc = 0.10. Delta = 0.12 - 0.10 = 0.02
3. Compute the pooled proportion using the equation Ppooled = (Ev + Ec) / (Nv + Nc)
For example, if Ev = 1200, Ec = 1000, Nv = 10000 and Nc = 10000, then: Ppooled = (1200 + 1000) / (10000 + 10000) = 2200 / 20000 = 0.11
4. Calculate the standard error (SE) of the difference in conversion rates using the equation SE = SQRT( Ppooled * ( 1 - Ppooled ) * ( 1 / Nv + 1 / Nc ))
SQRT is the square root function available in most software languages. It can be computed using the sqrt() function in Excel, R, and similar software. For example, if Ppooled = 0.11, Nv = 10000 and Nc = 10000, we have: SE = SQRT( 0.11 * (1 - 0.11) * (1 / 10000 + 1 / 10000) ) SE = SQRT( 0.0979 * 0.0002 ) SE = SQRT( 0.00001958 ) SE = 0.004425
5. Calculate the Z score (Z) by using the equation Z = Delta / SE
This is a simple division to express the observed difference Delta in terms of number of standard deviations. Continuing with the example above: Z = 0.02 / 0.0044 = 4.52
6. Use a tool like Excel, R, or the GIGA online z-score calculator to calculate the cumulative distribution function of the standard normal distribution for the value of Z and subtract it from 1 to get the p-value.
Use the NORM.S.DIST() function in Excel with a second parameter set to TRUE. Use the pnorm() function in R. Use the GIGA online z-score calculator for a normal distribution with a mean of 0 and a standard deviation of 1. To make sure your calculation is correct you can check it using this reference: for a Z score of 1.644854, the CDF is ~0.95 and the p-value would be 1 – 0.95 = 0.05.