How we built an experiment to identify leading indicators with a conversion snapshot

A powerful way to use product metrics is to correlate feature adoption with conversion and answer the question:

What actions do successful users take, and what features do they adopt, before they become paying customers?

The reason this is powerful is that we can use that insight to improve the onboarding and make sure more users become successful with our product.

I think we finally found a simple way to identify this using Zapier, Mixpanel, and Google Sheets, and although this is just an experiment, I wanted to share it.

But let’s start with some examples of these leading indicators.

Leading indicators

I’m not sure if they are true, but those I’ve come across are:

• Slack, who found that teams who sent at least 2000 messages were more likely to upgrade to a paid plan.
• Facebook, who found that users who made seven friends within ten days of signing up were likely to stay active.
• Twitter, who found that users become active when they follow 30 accounts, and 10 of them follow back.

In Product Management’s Sacred Seven, the authors hypothesize that Slack got to this number 2000 by comparing the number of messages a team had sent to the % of teams that had upgraded.

Plot that hypothesis into a graph, and we could imagine an S curve like this:

But I doubt that this was how Slack identified this leading indicator.

It doesn’t factor in the time frame and we risk confusing correlation with causation. A team could have upgraded after sending ten messages, used the product for a month, and by the time Slack did this analysis, they would have sent 2000 messages.

As Andrew Chen points out, we don’t want to study fires and conclude that it’s the fire department that is causing them.

It could still be helpful for what it is—an indicator—but it would be much wiser for Slack to capture the number of messages sent at the exact moment a team upgrades.

I’m guessing that’s what they did, and that’s how we’re trying to do it too.

So let’s look at how we’ve set it up.

No code needed

Okay, a tiny bit of code was needed, but Zapier handles all the magic.

When users upgrade their account, we send a notification to Zapier and trigger this flow. We use their webhook trigger for this.

In the payload, we send the unique user ID. It’s the $distinct_id in Mixpanel and this ensures no personal data is shared.

Then, we send a post request to Mixpanel with that user ID, and it returns all the properties on that user. Your engineer will know how to make an API post request like this. Check the template. It’s basic stuff.

Next, Mixpanel will return all the data you have on that user. Below is a sample of some of the feature adoption metrics we track.

Now we have all the data on the user from the exact moment they upgraded, and we can map each of these properties to a column in Google Sheets, where we can start building graphs and models to analyze the data.

The analysis comes next. We need some data to start that. Let’s see how that turns out.

The conversion snapshot

So why go through all this trouble? The core idea that we’re betting on is capturing the data before the conversion and keeping it the same after the upgrade — this is how we avoid the causation vs correlation dilemma with the hypothetical Slack example above.

Surprisingly, product analytics tools like Mixpanel or Amplitude don’t offer the capability to make this moment-in-time snapshot to identify leading indicators. I know Mixpanel tried to solve this problem with their Signal product, but it’s a black box that I don’t trust, and it’s on their enterprise plan.

Besides conversion, there are many use cases for correlating usage with retention or activation as well. So, it would be great if this works out. If it does, someone should probably build a product for this.