How do I know if I have a good retention rate?

Simple, you measure it properly. When I ask most founders or product marketers what their retention rate is their initial instinct is to talk about their monthly or daily active users (MAUs and DAUs). It’s great that they are tracking these metrics of course, but it’s pretty likely that these are not providing an accurate indication of true retention.

In a nut shell, a good retention rate is like the perfect length of string.

Why? It requires a clear frame of reference. The perfect length of string required to tie a bundle of sticks depends on it’s circumference. Similarly, the perfect retention rate of any product depends on how frequently it should be used. We can only answer both of these question once we have some context.

To help explain this further, let’s explore the fundamentals of retention.

The Fundamentals

When it comes to web or mobile applications, a retained user is defined as someone who continues to exhibit a behaviour indicative of ongoing use, over time.

Let’s consider the familiar monthly active users (MAUs) retention metric to verify this framework.

To measure MAUs, the behaviour we are looking for is the act of ‘logging in’. The use over time window has been set to a ‘month’ (30 days). From this we get a group of users who we consider ‘retained’. We then divide the retained group (the numerator) by the total users (the denominator) to get our MAU retention rate. Huzzah, it works.

Retention Rate = retained users / total users

In the formula above, the retained user number is the result of the equation we previously worked out. However, if we include those values in the formula itself, it would look something like this:

Retention Rate = users who exhibit a certain behaviour over time / total users

Let’s push on. The key line here is: ‘exhibit a certain behaviour over time‘. If you can understand this, then you’ll understand what I like to call true retention.

True Retention

Put simply, true retention is the answer to the following question: ‘when the user last had problem ‘x’, did they use our product to solve it? Where problem ‘x’ is the problem your product solves of course (more on this later). Put even more simply, when the user needed to use our product, did they?

As you can see, there are still two parts to this equation. A behaviour, and a time period, yet they are being interpreted slightly differently. The behaviour remains true, it’s knowing whether a user ‘used the product’. However, the time period on the other hand is no longer a specific ‘time window’. Instead, it vaguely refers to the natural period of time it takes for a user to need to use the product.

True retention rate = users who used it when the needed it / total users

True retention is about measuring the completion of a behaviour within the time window that it should naturally take place. The time window is therefore defined as the interval between the user having the problem the first time, and the user having the problem a second time. This could be a short or long period of time.

Retention is About Measuring Problems

Every product solves a problem. Be it getting from A to B (Uber), finding information (Google), or booking accommodation (Airbnb). Our users encounter problems like these in their daily lives. The products we build are focused on solving these problems. The products capacity to do this is directly addressed through it’s core value proposition, more specifically, the primary utility of the product.

A products core utility is ultimately how it creates value, therefore, this is how we should measure retention.

With this in mind, it’s safe to say that our users only need our product when they are faced with the problem that our product solves. Retention is about measuring, surveying, or even guesstimating how frequently our customers are likely to face this problem in their daily lives. In other words, how frequently they need our product. This will be used as our ‘over time’ window for the retention formula.

Let’s use Airbnb as a case study to explore this further.

Airbnb Case Study

The MAU rate of Airbnb is… completely irrelevant. No one goes on holiday every month, so why measure ‘how many people booked accommodation in the past month’? Let’s try and define the true retention rate of Airbnb.

Firstly, we need to know the real rate of holidays that the average person takes each year. According to the ABTA, people take roughly 1.4 holidays per calendar year. We will treat this number as the global average as it sounds about right.

Next, we need to find Airbnb’s total user base so we can work out the total potential number of bookings that could be made. In September 2015, Airbnb has roughly 50 million total users. Let’s assume that by December it had grown to 60 million. Therefore 60 million people taking an average of 1.4 holidays per year would create 84 million trips.

Finally, we need to count the number of users who have ‘booked accommodation’ through Airbnb in the 2015 calendar year. Whilst Airbnb doesn’t reveal their actual bookings, they do reveal the number of nights stayed and average duration of stays. Using recently published Morgan Stanley statistics, we can guesstimate the total number of Airbnb bookings to be roughly 17 million in the 2015 calendar year.

So, we now have our two numbers. Firstly, the total number of accommodation bookings made by Airbnb users on Airbnb in 2015, 17 million. Secondly, the potential number of accommodation bookings made by Airbnb users on any platform in 2015, 84 million. Therefore, the true retention rate of Airbnb in 2015 can be calculated as follows:

17m Airbnb bookings / 84m potential bookings = 20%

The true retention rate of Airbnb is 20%. Moreover, one in five registered Airbnb users booked their 2015 holiday accommodation through Airbnb. Alternatively, four in five Airbnb users did not use Airbnb to book their holiday accommodation in 2015. This is assuming the data I’ve sourced is of course correct.

As you can see, the challenge with looking at the true retention window of Airbnb is that it would take one year for the data to be collected. This long propagation period is the primary reason why most companies look at MAUs. Ultimately, the true look back window may simply be too large. User retention is a long term game though.

More Examples

The key here is to be able to identify the normal usage pattern of your product. The following table provides further examples to help you identify this for your product.

Final Thoughts

The main takeaway message of this article is to determine whether it’s necessary to track a true retention rate alongside your MAU or DAU rates. Some products will not require this. For some, like the last two in the table above, the natural consumption rate may be daily, weekly or monthly anyway. In these circumstances the conventional retention metrics are likely to offer accurate insights.

At Ausmed, we monitor MAUs quite closely. However, we also look at a 90 day retention window. This is because we know that the vast majority of our users undertake at least one educational activity every 90 days. If they have not used our service in the last 90 days, it’s highly likely they are no longer using the product. Therefore, we don’t consider them to be a retained user.

Finally, if you’re working on a product that naturally has long intervals between use, think Airbnb or an annual tax return software as an example, there are ways to reduce the time window. In situations like this, you could consider building features into the product that would encourage more regular use. For Airbnb this might be achieved by promoting local, short stay trips in the user’s home city. For tax return software, this could be achieved by encouraging users to add their tax-deductable expenses as they are incurred (all year round). In both cases, the time window being evaluated for retention would be significantly reduced.

The post What Is a Good Retention Rate? appeared first on Will Egan.

But one test was a stand out success.

In this article, I want to explore that change. It was a feature we called ‘long life login’. It turned out that by automatically logging our users out at the end of a session, we were losing 2% of the active user base every week. Here’s how…

Background

Retention. It’s a hard game. Why? Because the feedback loop isn’t instant. It genuinely takes 90 days to find out how many users are still active 90 days after sign up. Conducting cohort analysis on such a curve can take even longer… years potentially.

Similarly, a single change made in pursuit of greater first month retention takes a month to propagate. You deploy the test, then the lights go out, and 30 days later the results start coming in.

This is all because of time. We can’t speed it up.

Compare this to a normal A/B split test where we can speed up learnings simply by driving more traffic to the test. The question we asked ourselves earlier this year was whether we could do the same for time-based tests? An A/B split test targeting first month retention will take 30 days to resolve. This is fact. But, it turned out there was a faster way to experiment with retention. A way that does provide us with instant feedback. It just requires a different mindset.

New Mindset

When I chat to most companies (startups through to corporates) about retention, they all make the same assumption. That retention is directly linked to the quality of the product itself. This flippant absolving of responsibility of such a key metric is misguided. Sure, it’s true for products like Uber, Snapchat and WhatsApp that stumble across problem solution fit on day one. But, for the other 99.9% of products it’s a compromising belief.

Of course, if the product in question is not genuinely useful and doesn’t actually solve a problem, no amount of retention mastery will save it. But let’s assume you do have a product that solves a problem for some group of users. In this case, it’s likely that the vast majority of your retention will need to be engineered. In other words, product is not the albatross hanging around retention’s neck.

Herein lies the key mindset shift we made, realising that there are two areas to optimise for retention:
1 – Problem based retention, and
2 – Engineering based retention.

In it’s purest terms, a user is driven to use your product by a cognitive or physical trigger they experience in their day-to-day lives. Let’s call this trigger a ‘job to be done’ (Clayton Christensen’s words, not mine). The moment they encounter this ‘job to be done’ their mind starts searching for a product that they can employ, to get the job done. The first product that comes to mind is the one that they believe can best get the job done. Here’s a quick example…

You need to dig a hole. Shovel.

So, let’s assume your user has the problem your product is designed to solve. They whip out their phone, navigate to your app, and open it. This is problem-driven retention in action. At this point the problem based retention engine has worked. It’s done its job. It now hands over a highly motivated user to the engineering based retention engine, which must finish getting the user ‘retained’.

It’s at this very point that the single biggest hinderance to our retention rate was found. The interface between the user and our product… a login form.

Long Life Login

Logging in a pain for most users everyone. Naturally, security is the main consideration here, but truth be told, Facebook has both the most personal data on us, and is the easiest to login. In fact, online banking is really the only service that we appreciate being automatically logged out of.

At Ausmed, we realised pretty quickly that our problem based retention engine was not a problem. Our product is perfectly wrapped around a key job to be done. The moment a user realised they needed our product, they would visit our website or open our app. But, invariably, this strong driver was not leading to sustained retention. We wanted to know more about what was going on here.

To do this, we began tracking behaviour around the login form using event tracking. We defined three events:
1) User attempted to log in
2) User successfully logged in
3) User failed to log in.

The primary question we wanted to answer was ‘what percentage of people who ‘attempted to log in’ yet ‘failed to log in’ did not log in again over the next 7 days?’

In other words, what percentage of the people trying to use our product were being lost due to difficulties logging in? The results were astounding.

Some comments about this graph. Our user base has grown substantially throughout this period, yet the rate has fallen. In March, almost 1 out of 25 active users were being lost every week. Contrast this to December, where only 4 out of 1000 were being lost.

It turned out that an average of 2% of active users were being lost each week. That’s 8% a month. I say average, but some weeks it was as high a 4%. Extrapolated out: if we had 100,000 monthly active users today and did not acquire a single new user for the next 10 weeks, only 81,707 active users would remain. In other words, we were losing 20% of our active users every two months, purely through login.

So, we got to work solving this problem. We made two changes. One that was obvious but also quite technically hard to do, and one that was extremely clever and super easy to do.

How We Fixed Logging-In

1 – We Stopped Logging People Out

Seriously, this will solve the bulk of the problem. In our case the technical changes took some time to solve. Load balancers, caches, token expiry etc.

2 – We Proactively Sent Username Reminders

As we were now tracking events around the login form, we started sending a copy of this event data to Vero. This meant we could begin sending pro-active username reminders (emails). The message in the email was simple: ‘Hey, we notice you’re having difficulty logging in, here’s a friendly reminder of your username. Here’s a (gif-based) demo of how to log in, and here’s a link to reset your password in case need to do that too.’ It worked amazingly well.

This single email resulted in 58% of people who failed to log in, logging in again. It’s incredibly useful and a novel idea—I’ve never seen it done before. Also, it’s worth noticing that not a single person unsubscribed.

For those with a keen eye, the difference between the converted group, and open group is explained by indirect conversion (my next post).

Summary

Nothing else in the past year has improved our retention rate more positively than long life login. Every ten weeks, we were haemorrhaging 20% of our active user base simply by automatically logging them out at the end of a session. Security is important, no doubt. But even a service like Facebook, with so much data on all of us, does not log users out. Neither should you. Proactively helping users who still struggle after you implement long life login is incredibly effective. And finally, I would love to hear your thoughts on this article in the comments box below.

The post User Retention Starts at the Login Form appeared first on Will Egan.