User Retention Starts at the Login Form

In 2016 our monthly active users (MAU) rose from roughly 8% in February to 48% in November. This is an extraordinary result. During this period we tested about 20 changes, from hooks of addiction (inspired by Nir Eyal’s hook model) to more simpler things such as email reminders.

But one test was a stand out success.

In this article, I want to explore that change. It was a feature we called ‘long life login’. It turned out that by automatically logging our users out at the end of a session, we were losing 2% of the active user base every week. Here’s how…

Background

Retention. It’s a hard game. Why? Because the feedback loop isn’t instant. It genuinely takes 90 days to find out how many users are still active 90 days after sign up. Conducting cohort analysis on such a curve can take even longer… years potentially.

Similarly, a single change made in pursuit of greater first month retention takes a month to propagate. You deploy the test, then the lights go out, and 30 days later the results start coming in.

This is all because of time. We can’t speed it up.

Compare this to a normal A/B split test where we can speed up learnings simply by driving more traffic to the test. The question we asked ourselves earlier this year was whether we could do the same for time-based tests? An A/B split test targeting first month retention will take 30 days to resolve. This is fact. But, it turned out there was a faster way to experiment with retention. A way that does provide us with instant feedback. It just requires a different mindset.

New Mindset

When I chat to most companies (startups through to corporates) about retention, they all make the same assumption. That retention is directly linked to the quality of the product itself. This flippant absolving of responsibility of such a key metric is misguided. Sure, it’s true for products like Uber, Snapchat and WhatsApp that stumble across problem solution fit on day one. But, for the other 99.9% of products it’s a compromising belief.

Of course, if the product in question is not genuinely useful and doesn’t actually solve a problem, no amount of retention mastery will save it. But let’s assume you do have a product that solves a problem for some group of users. In this case, it’s likely that the vast majority of your retention will need to be engineered. In other words, product is not the albatross hanging around retention’s neck.

Herein lies the key mindset shift we made, realising that there are two areas to optimise for retention:
1 – Problem based retention, and
2 – Engineering based retention.

In it’s purest terms, a user is driven to use your product by a cognitive or physical trigger they experience in their day-to-day lives. Let’s call this trigger a ‘job to be done’ (Clayton Christensen’s words, not mine). The moment they encounter this ‘job to be done’ their mind starts searching for a product that they can employ, to get the job done. The first product that comes to mind is the one that they believe can best get the job done. Here’s a quick example…

You need to dig a hole. Shovel.

So, let’s assume your user has the problem your product is designed to solve. They whip out their phone, navigate to your app, and open it. This is problem-driven retention in action. At this point the problem based retention engine has worked. It’s done its job. It now hands over a highly motivated user to the engineering based retention engine, which must finish getting the user ‘retained’.

It’s at this very point that the single biggest hinderance to our retention rate was found. The interface between the user and our product… a login form.

Long Life Login

Logging in a pain for most users everyone. Naturally, security is the main consideration here, but truth be told, Facebook has both the most personal data on us, and is the easiest to login. In fact, online banking is really the only service that we appreciate being automatically logged out of.

At Ausmed, we realised pretty quickly that our problem based retention engine was not a problem. Our product is perfectly wrapped around a key job to be done. The moment a user realised they needed our product, they would visit our website or open our app. But, invariably, this strong driver was not leading to sustained retention. We wanted to know more about what was going on here.

To do this, we began tracking behaviour around the login form using event tracking. We defined three events:
1) User attempted to log in
2) User successfully logged in
3) User failed to log in.

The primary question we wanted to answer was ‘what percentage of people who ‘attempted to log in’ yet ‘failed to log in’ did not log in again over the next 7 days?’

In other words, what percentage of the people trying to use our product were being lost due to difficulties logging in? The results were astounding.

Some comments about this graph. Our user base has grown substantially throughout this period, yet the rate has fallen. In March, almost 1 out of 25 active users were being lost every week. Contrast this to December, where only 4 out of 1000 were being lost.

It turned out that an average of 2% of active users were being lost each week. That’s 8% a month. I say average, but some weeks it was as high a 4%. Extrapolated out: if we had 100,000 monthly active users today and did not acquire a single new user for the next 10 weeks, only 81,707 active users would remain. In other words, we were losing 20% of our active users every two months, purely through login.

So, we got to work solving this problem. We made two changes. One that was obvious but also quite technically hard to do, and one that was extremely clever and super easy to do.

How We Fixed Logging-In

1 – We Stopped Logging People Out

Seriously, this will solve the bulk of the problem. In our case the technical changes took some time to solve. Load balancers, caches, token expiry etc.

2 – We Proactively Sent Username Reminders

As we were now tracking events around the login form, we started sending a copy of this event data to Vero. This meant we could begin sending pro-active username reminders (emails). The message in the email was simple: ‘Hey, we notice you’re having difficulty logging in, here’s a friendly reminder of your username. Here’s a (gif-based) demo of how to log in, and here’s a link to reset your password in case need to do that too.’ It worked amazingly well.

This single email resulted in 58% of people who failed to log in, logging in again. It’s incredibly useful and a novel idea—I’ve never seen it done before. Also, it’s worth noticing that not a single person unsubscribed.

For those with a keen eye, the difference between the converted group, and open group is explained by indirect conversion (my next post).

Summary

Nothing else in the past year has improved our retention rate more positively than long life login. Every ten weeks, we were haemorrhaging 20% of our active user base simply by automatically logging them out at the end of a session. Security is important, no doubt. But even a service like Facebook, with so much data on all of us, does not log users out. Neither should you. Proactively helping users who still struggle after you implement long life login is incredibly effective. And finally, I would love to hear your thoughts on this article in the comments box below.