Measuring retention? Why is it important?

If you are running an online business, then these 2 goals are definitely on your list:

  1. To onboard as many new users as possible
  2. To convert current users into more frequently returning users

Measuring retention is a topic which raises many questions even on definition-level, whereby potentially incorrect definitions can lead you astray. This article is trying to prevent this kind of problems.

PROBLEM #1: What should be the ideal time frame between two returns?

Why is it good to know this? There’s an instant action item: After how much inactivity should you send your win-back campaigns for the page (e-mail, discount coupons, a bonus month, etc, etc)? If you send it too early, you will be annoying, if you send it too late, you may permanently lose lots of users! But if you know how often your “good users” return, then you will also know when to ping the less active ones.

But the answer is indispensable to more complex metrics as well, to say the least: Customer LifeTime Value.

So what is the ideal timeframe between two returns? Tricky question, because it depends a lot on your business. If you have a music-streaming service (e.g. Spotify), then you would obviously be pleased if they listen to you every day. But if you are developing an app for booking airplane tickets, you have to calculate in that they may only use your app once every 6 months.

Solution #1: Finding the ideal retention time

How can you decide what is the ideal time for you? 4 things can help you with this:

1. Decide what you would like to achieve AND also what you believe to be realistic in terms of your product? Aka. common sense – I know, this is not something that you can hear from me quite often. But still, when it comes to measuring retention, common sense plays an actual role.

2. Check your historical data! You should definitely examine a few given groups who registered on the same day (so to speak: cohorts)! (E.g. Examining the return of those who registered on the 1st of February, 2015.) After that, calculate how many of these returned again – and when (eg. 1 day later, 2 days later, 3 months later, etc). For example, here you can see the monthly breakdown of returning users who registered on the 1st of January, 2015.

The monthly breakdown of returns - measuring retention

The monthly breakdown of returns

It seems this is a dense breakdown, as you can see that the number of users each month is decreasing, but you can’t really see the periodic of return. Let’s look at a weekly breakdown of this same cohort.

The weekly breakdown of returns - measuring retention

The weekly breakdown of returns

Still not the best. The situation is similar here, we don’t see periodic, only a reduction. Let’s go to the daily breakdown.

The daily breakdown of returns - measuring retention

The daily breakdown of returns

Here you can see some periodicity! Roughly every 3 days there’s a peak, so (note that I’m a bit simplifying, but) we can determine that the ideal time between 2 returns is 3 days.

3. If your diagram is not so clear as the chart above – and if you eg. guess to place the return period somewhere between 8 and 20 days, then I would advise to always choose the smallest, yet still realistic timeframe. For one, it’s a great goal for you to aim at getting as frequent returns as possible. Secondly, in terms of measuring retention, you will see: if you go with the 20-days retention, then if you do an AB-test (for instance) focusing on better retention, you have to wait 20 days for the first data point. Right? 8 days waiting time is much more pleasant…

4. Check the market benchmarks! Sometimes it’s the most trivial solution, but we still forget about it. 🙂

Okay, let’s say, you have the ideal time frame! Then comes…

PROBLEM #2: How do I measure the retention rate?

If you have tried doing it before and have dug deeper into it, than a Google Analytics report, then surely you must have raised many questions!

Let’s look at an example where, for the sake of discussion we examine the Daily Retention Rate.

There is one very common mistake one can do calculating the Daily Retention Rate. People usually take Day1 and Day2 and simply check the difference on Daily Active Users. E.g.:

  • if I have 10.000 daily active users on 12/01/2016
  • and I have 9.000 daily active users on 13/01/2016
  • then my daily retention rate is: 90%…

NO! This math is INCORRECT!

Just think about it: what if 10.000 people registered on the 12th of January and none of them ever came back again. Then on the 13th of January an additional 9.000 people registered. Provided that your true goal is to attract Daily Retention, then this is really bad news (as you are throwing money into those who register, and they never come back again).

However your incorrectly defined Daily Retention Rate showed 90%, which would actually be really great news. So pay attention and don’t lie to yourself!

SOLUTION #2A: Measuring Retention Rate Better

A slightly better solution is to calculate the Daily Retention Rate as per below:

I had 10.000 daily active users on 12/01, but out of this I will deduct those who registered on the same day (e.g. 2.000)
– I had 9.000 daily active users on 13/01, but again: out of this I will deduct those who just registered on the same day (e.g. 3.000)
– This way, my daily retention rate is = ((9.000 – 3.000) / (10.000 – 2.000)) = 75%

Of course this still isn’t the real thing, as we are mashing together those returning users who have been with us every day for a year (so they’re very loyal) with those who may have only registered the day before yesterday and accidentally found themselves on the page again (so they are not necessarily committed).

SOLUTION #2B: Measuring Retention Rate At Its Best

The only fully proficient solution I found for measuring the Daily Retention Rate through the years is: cohort analysis. At first instance it is not the easiest visualization method to overview, but if you get your head around it, then you can get the most precise information out of it.

The essence of cohort analysis is that it examines the elements belonging to a cohesive group at the same time (e.g. users who registered on the same day).

This way whatever changes/testing we do on our product, we can separately measure how this affects the return of our loyal users (e.g. the cohort of the 31st of August, 2014), or those who just recently registered. We can set the cohorts on a daily, weekly or monthly basis („users who registered in March”) or alternatively we can define our own timeframe.

Let’s look at a sample cohort analysis:

Sample cohort analysis (In a Mixpanel) - measuring retention

Sample cohort analysis (via Mixpanel)

As you can see, here we are examining 9 adjacent cohorts:
– those who registered on the week of the 26th of January (9.6 million users)
– those who registered on the week of the 2nd of February (9.6 million users)
– and those who registered on the week of the 9th of February
… etc…

From the first row you can see that the cohort of the 26th of January returned at a rate of 84,63% on the first week, 79.08% the second week, 73,52% on the third week, and so on. The rows underneath each other show the same trends, which reinforces that this is generally specific to the users of the given app.

If we compare the rows underneath each other, we can see a slight decrease, which could be seasonality, or it could be a faulty product or the result of a marketing decision.

However way we see it, the report is clear and informative.

Cohort Analysis In Action

This is a hiking e-commerce shop’s example from the Practical Data Dictionary.

“We segmented the Revenue on the below chart based on Payment types. We can see that there was a constant change in 2014 on whether the „simple” First Payments (so namely the first purchase) or the Repeat Purchase (when a previous Customer purchased again) brought in more Revenue.

segmentation - measuring retention

Segmentation (before measuring retention via cohort analysis)

It jumps out that the Revenue generated by New Customers drops in autumn, but returning Customers cover this gap.
In light of this, we created a Cohort analysis for those who made their first Payment in the shop in 2014. We looked at exactly how much was spent and when as a Repeat Purchase. We found this:

cohort analysis - measuring retention

cohort analysis (monthly)

So the Customers from 2014 brought the best Revenue from a Repeat Purchase at the end of the summer and beginning of autumn. In fact, we also know that Customers from February, March, April and May are really strong and spend a lot 4-5 months after they make their first purchase (so July, August, September and October).”

Read more here: Practical Data Dictionary.

Conclusion – how should you measure your returning users?

  1. Define the ideal retention timeframe! It should be as low as possible, but take into consideration your historical data, the market benchmarks and your own expectations!
  2. Try to use the cohort-analysis methodology as much as possible, this will give you the most precise and clear results.

And if you want to be notified first about new content on data36 blog (like articles, videos, handbooks, etc.), sign up for the Newsletter!

Tomi Mester