If you are running an online business, then the 2 below goals are definitely on your list:

  1. To onboard as many new users as possible
  2. To convert current users into more frequently returning users

The measurement of returning users is a topic which raises many questions even on definition-level, whereby potentially incorrect definitons can lead you astray. This article is trying to prevent this kind of problem. 🙂

PROBLEM #1: What should be the ideal time frame between two returns?

Why is it good to know this? There’s an instant action item: After how much inactivity should you send your win-back campaigns for the page (e-mail, discount coupons, a bonus month, etc, etc)? If you send it too early, you will be annoying, if you send it too late, you may permanently lose lots of users! But if you know how often the „well-behaved users” return, then you will also know when to ping the less active ones.

But the answer is indispensable to more complex metrics as well, to say the least: Customer LifeTime Value.

So what is the ideal timeframe between two returns? Tricky question. If you have a live music-streaming service (e.g. Spotify), then you would obviously be pleased if they listen to you every day. But if you are developing an app for booking airplane tickets, you have to calculate in that they may only use your app once every 6 months.

How can you decide what is the ideal time for you? 4 things can help you with this:

1. Firstly, decide what you would like to achieve AND also what you believe to be realistic in terms of your product?
2. Check your current data! You should definitely examine a few given groups who registered on the same day (so to speak: cohorts)! (E.g. Examining the return of those who registered on the 1st of February, 2015.) After that, calculate how many of these returned again – in certain time breakdowns. For example, here you can see the monthly breakdown of returning users who registered on the 1st of January, 2015.

The monthly breakdown of returns

The monthly breakdown of returns

It seems this is a dense breakdown, as you can see that the number of users each month is decreasing, but you can’t really see the periodics of return. Let’s look at a weekly breakdown of this same cohort.

The weekly breakdown of returns

The weekly breakdown of returns

Still not the best. The situation is similar here, we don’t see periodics, only a reduction. Let’s go to the daily breakdown.

The daily breakdown of returns

The daily breakdown of returns

Here you can see some periodicity! Roughly every 3 days there’s a peak, so (now I’m just simplifying, but) we can determine that the ideal time between 2 returns is 3 days.
3. If your diagram is not so clear as the chart above – and if you eg. want to place the return period somewhere between 8 and 20 days, then I would advise to always choose the smallest, yet still realistic number. For one, it’s a great goal for you to aim at getting as frequent returns as possible. Secondly, in terms of measurement, you will see: if you analyze 20 day retention, then if you change anything (or you do an AB-test) in the future in your project, you have to wait 20 days for the data on the impact of return to arrive. 8 days waiting time is much more pleasant…
4. Check the market benchmarks! Sometimes it’s the most trivial solution, but we still forget about it. 🙂

So you have the ideal time frame! Then comes…

PROBLEM #2: How do I measure the retention rate?

If you have tried doing it before and have dug deeper into it, than a Google Analystics report, then surely you must have raised many questions!

Let’s look at an example where, for the sake of discussion we examine the Daily Retention Rate.

The most common mistake people make is calculating the Daily Rentention as a ratio between different days’  number of Daily Active Users. E.g.:

  • if I have 10.000 daily active users on 12/01/2016
  • and I have 9.000 daily active users on 13/01/2016
  • then my daily retention rate is: 90%…

NO! That result is INCORRECT!

Just think about it: what if 10.000 people registered on the 12th of January, 2016, and none of them ever came back again. Then on the 13th of January, 2016 an additional 9.000 people registered. Provided that your true goal is to attract Daily Retention, then this is really bad news (as you are throwing money into those who register, and they never come back again).

However your incorrectly defined Daily Rentention Rate showed 90%, which would actually be really great news.

So pay attention, don’t lie to yourself!

A slightly better solution is to calculate the Daily Rentention Rate as per below:

I had 10.000 daily active users on 12/01, but out of this I will deduct those who recently registered (e.g. 2.000)
– I had 9.000 daily active users on 13/01, but out of this I will deduct those who just registered (e.g. 3.000)
– This way, my daily retention rate is = ((9.000 – 3.000) / (10.000 – 2.000)) = 75%

Of course this still isn’t the real thing, as we are mashing together those returning users who have been with us every day for a year (so they’re very loyal) with those who may have only registered the day before yesterday and accidentally found themselves on the page again (so they are not necessarily committed).

The only fully proficient solution I found for measuring the Daily Rentention Rate through the years is: cohort analysis. At first instance it is not the easiest visualization method to overview, but if you get your head around it, then you can get the most precise information out of it.

The essence of cohort analysis is that is examines the elements belonging to a cohesive group at the same time (e.g. users who registered on the same day).

This way whatever changes/testing we do on our product, we can seperately measure how this affects the return of our loyal users (e.g. the cohort of the 31st of August, 2014), or those who just recently registered. We can set the cohorts on a daily, weekly or monthly basis („users who registered in March”) or alternatively we can define our own timeframe.

Let’s look at a sample cohort analysis:

Sample cohort analysis (In a Mixpanel)

Sample cohort analysis (In Mixpanel)

As you can see, here we are examining 9 adjacent cohorts:
– those who registered the week of the 26th of January (9.6 million users)
– those who registered the week of the 2nd of February (9.6 million users)
– those who registered the week of the 9th of February
… etc,etc

From the first line you can see that the cohort of the 26th of January returned at a rate of 84,63% on the first week, 79.08% the second week, 73,52% on the third week, and so on. The lines underneath each other show the same trends, which reinforces that this is generally specific to the users of the given app.

If we compare the lines underneath each other, we can see a slight deterioration, which could be seasonality, or it could be a faulty product or the result of a marketing decision.

However way we see it, the report is clear and informative.

Conclusion – how should you measure your returning users?

  1. Define the ideal return timeframe! It should be as low as possible, but take into consideration your previous data, the market benchmarks and your own expectations!
  2. Try to use the cohort-analysis methodology as much as possible, this will give you the most precise and clear results.

And if you want to be notified first about new content on data36 blog (like articles, videos, handbooks, etc.), sign up for the Newsletter!

Tomi Mester

ps1. if you want to indulge in the rapture of cohort analysis, I recommend this TopTal article: http://www.toptal.com/data-science/growing-growth-perform-your-own-cohort-analysis

ps2. (As per Schmidt G’s @Ustream advice: ) Would you like to see your own Visitor Retention in Google Analystics? Try this link (it’s only a beta version, but it has quite a few functions already): https://analytics.google.com/analytics/web/#report/visitors-cohort/

share:Tweet about this on Twitter4Share on LinkedIn11Share on Facebook9