A Product Manager’s Guide to Measuring Things Well
Andrea Sipos is a product person with 10 years of experience and a data-reliant approach. She’s worked as a product manager and leader at Prezi, Skyscanner, and OptiMonk. She puts a huge emphasis on setting up and following the right metrics and making data-informed decisions during the product development process, from initial research to user experience to final design. And I asked her to write about what makes a good metric based on her experience.
What Makes a Good Metric?
As a Product Manager, one of my biggest challenges is to make decisions every day. I need to figure out how to move forward with product development, decide the possible impact of an idea or determine how successful my latest experiment was.
In order to make decisions I rely on metrics heavily. I’m a very data-driven person, trying to get as much input as possible. During my daily work I use quantitative and qualitative data as well.
When it comes to data, the most important thing I’ve learned during my career is that I need to make sure that the number I’m looking at is actually the one I really need. It’s really easy to draw the wrong conclusions due to a poorly chosen metric.
In this post, I would like to share what I believe the most important factors are when defining what we are going to measure in our product.
Any metric can be a good metric
The starting point is that any metric can be a good metric – if it makes sense and fits both the product and the situation. However, you need to make sure that you understand the implications of your choice and you can use your metric without actually misleading yourselves.
There are 3 different factors I take into account:
- The metric is clear and simple
- It reflects users’ behaviour
- It’s actionable, not a vanity metric.
Let’s take a look at these in more detail.
1. A good metric is clear and simple
A metric needs to be easy to understand and communicate to the team. This might sound trivial, but there are some metrics out there which are really complicated to define and follow. If you have a metric which you can’t explain to a stranger in 2 sentences, you might be on the wrong path.
When thinking about what this metric should be, I usually go back to basics and start with the ideal customer journey in my product.
- What is the problem we are solving with our product?
- What do our users do when they use it?
- What do they want to achieve?
A metric which is clear and simple might not be easy to query and calculate. That’s perfectly fine, we then know that there are technical limitations. In that case, you need to find a calculation method which is the closest to the ideal state. If there are too many technical limitations on your ideal metric candidate, be careful – they might cause the metric to be less credible. Nobody wants to base their decisions on a metric which is generated by some black magic under the hood or that they don’t understand clearly. The technical limitations need to be balanced and you might need to go with a less ideal, but more reliable metric.
2. A good metric reflects users’ behaviour
This factor is really important to me, because I’ve experienced how the design of the product can impact a metric and make decisions harder.
Building digital products is not easy. We try to understand what the biggest customer problems are out there, experiment with different solutions and come up with options which are built a certain way. The solutions themselves define how and what we are able to measure. However, the metric which we measure might not be the exact reflection of how users behave in real life.
Imagine if you have a metric which you selected when you started building your product. For example in those early days your product was focused on generating more and more clicks, as you basically sold clicks to your partners. Your main goal was to increase your CTR (click through rate).
But what happens when you decide to pivot your product and focus more on engagement, retention? You start designing a different product with different goals and focus areas. New users come in who will start using the new features and after a while generating clicks is not your main goal anymore. You have to change your main metrics to reflect product usage more accurately.
You need to constantly review if the metric you are working on fits with what you expect your users to do. If there is a misalignment, you need to adjust your metrics.
3. Actionable vs. vanity metrics — absolute numbers vs. ratios
Metrics influence your and your teams’ behaviour and decisions. That’s their job, that’s why we have them. The biggest trap of metrics is that sometimes the metrics which make us proud and excited are not the ones that actually help to make those decisions. We might look at numbers which look great on paper, and help attract new visitors, customers or even investors, but don’t help us to see how good our product is.
Vanity metrics — and the problem with them
Vanity metrics are absolute metrics that make us happy. They reflect quantities and when they increase, we might think we are on the right track. One of the most popular vanity metrics is number of registrations. We see lots of headlines – “XYZ hit 1 million registrations!” Or a similar one: “XYZ has 1.5 million followers on Instagram!” These numbers are definitely impressive.
The problem comes in when we don’t look behind the number of registrations or followers. When I see a number like that, my first question is: ok, how many of them actually use the product? How many of them got value from the product? Today, I can buy 5000 followers for 50 USD for my Instagram account. They’ll be there even if I don’t post any valuable content at all.
Actionable metrics — this should be your focus
When you need to figure out if your product is performing well, you need to look at actionable metrics, which help you identify where the biggest gaps in the flow are. These actionable metrics are usually ratios and not absolute numbers. For instance: out of 1000 registered users, 700 tried the product? Great! Or only 50? Then something is really off in your messaging, onboarding or in the product itself.
Absolute metrics also have a role in decision making, as there are situations when we need to understand the volume, not only the effectiveness. Scaling a product, making sure that there is a proper infrastructure behind it is based on absolute metrics. It does not really matter how well a webshop is converting during Black Friday. If it can’t deal with traffic its sales will be terribly low.
Absolute and actionable metrics together give us a complete overview of what’s going on in the product. Use them at the right time and in the right context.
Summary
Defining a metric can be a really tricky challenge. A great metric can accelerate your products’ performance, while the wrong one can put you on a different track from what you really want.
f you make sure that your metric is clear, reflects product value and is actionable, you will be able to make the right product decisions.
PS. And if you are interested in learning more about what is the right metric for your product or business, I highly recommend the Lean Analytics book by Alistair Croll and Ben Yoskovitz.
Cheers,
Andrea Sipos
