Test your user state models

1. Choose a user metric that is relevant to your company's success.

Healthy vs unhealthy user state models are the most common. Healthy users are the people exhibiting behavior that indicates they are active and likely to keep using your product or service. For example, common metrics that can define healthy user behavior include days since the last login, number of logins in the past 30 days, session length, interactions, and app screens per session. Alternatively, you can choose other state models such as casual vs core users.

2. Gather a cohort of users and input their data into a spreadsheet.

Include your healthy user metric and others, for example, downloads, and content consumed or created.

3. Use cohort analysis to run correlations and see what tends to correlate with your healthy user metric.

Run a regression analysis using a tool like Amplitude to verify your hypothesis. For example, for an app that is a website uptime monitoring service, like Pingdom, by running regression analysis, you can see that receiving a text 6 or more times, is positively correlated with long-term retention. The Positive and Negative Predictive Values in your analysis help you assess whether this behavior is something to push people towards. In the above case receiving 6 texts has a PPV of 26% and an NPV of 93%. A PPV of 26% means that receiving 6 texts leads to the person being retained 1 in 4 times, which makes for a pretty solid foundation for building further retention. The Proportion Above Threshold number shows how many of your users overall are already exhibiting the behavior. In this case, a low number, such as the 4.3%, shows that there is still room to maneuver with this particular behavior, and you can drive more people towards it.

4. Test your user state model with A/B testing, to find its limitations.

Prioritize the input variable and increase it, possibly at the expense of another variable, and see if those users are more successful as a result of your change.

5. Identify counter metrics early.

Pair every metric with an appropriate counter metric. For example pair: signups with activation, activation with churn, and new paid customers with total revenue. Determine counter metrics to take a holistic approach to growth, and help keep you on track with your user state models.

6. Optimize to increase the metrics that correlate with your healthy user metric.

Optimize so that new users adopt healthy behaviors quickly and healthy user’s stay healthy. You can also optimize to push unhealthy users back to the healthy user state. For example, Plenty of Fish spoke to over 1000 women who married someone they met on their dating site. They found that most women messaged their future partner first. Plenty of Fish’s primary focus was ensuring their users find long-lasting love, so they optimize their site in a way that encourages women to message men first more often.

7. Perform resurrection experiments on at risk, and transitioning out users.

At-risk users are those who are not demonstrating healthy user behavior often as of late. For example, only 1-2 logins in the last 14 days when a healthy user might have logged in 7-10 times. Transitioning out users are similar to at risk users, but they have been exhibiting the unhealthy behavior for a longer period of time. For example, only 1-2 logins in the last 30 days. Users transitioning out are about to be churned users. Apply resurrection messaging and experiments to them before they leave while it is still much easier to reach them.