KPIs Gone Wild: Cautionary Tales of Lost Customer Obsession

Welcome to the Scarlet Ink newsletter. I'm Dave Anderson, an ex-Amazon Tech Director and GM. Each week I write a newsletter article on tech industry careers, and specific leadership advice.

Free members can read some amount of each article, while paid members can read the full article. For some, part of the article is plenty! But if you'd like to read more, I'd love you to consider becoming a paid member!

Amazon is a data-driven organization. That means you have goals, KPIs, metrics, and dashboards. Everything that can be measured, tends to be measured.

“What you measure is what you manage.” - Peter Drucker

I heard that quote repeated endlessly, for good reason. Because it’s so incredibly true. For good, and for bad.

In the most basic, naive case, metrics are a fantastic tool.

Imagine you’re a single engineer running a simple website. It has some content, and that content needs to be shown to customers. This sounds like a pretty chill job.

If the content is correct, what can go wrong operationally?

Well, web pages are fairly simple. They can load too slowly. Or they can fail to load. That’s about it.

If you have metrics on page load times, and failure rates of your webpage, you will know if anything goes wrong with your site. Set an alarm on those two metrics, and you’ve achieved perfection. Nothing can go wrong while you sleep.

Even better, if your management team asks you how that fancy content website is working, you can load up the historic page load times and failure rate graphs. This allows you to provide concrete information about how fast things are, and how often they fail. Now you have a data-driven view into the technical quality of your site.

KPIs can save the day.

I’d argue that you should rarely make a change without knowing exactly which KPI / metric is improved.

Years ago, we had one of those “quick” projects pushed by our product management team. If you work in the industry, you’ll know exactly what I mean.

“This is just a super quick thing, shouldn’t take you more than a week or two.”

Understandably, part of the reason a product manager will use this argument is to short-cut the normal funding justifications which would otherwise take place. If they can argue that this is a quick hack, perhaps they won’t need to explain exactly how this project is more important than other ones.

Anyway, this was a change to the content offered to our customers. The product managers said it would improve engagement in our application. And logically, it made sense that it would increase engagement. So we didn’t disagree that this change made sense.

As soon as we began estimating the cost, the product manager pushed back.

“No, you don’t need to instrument this with metrics. You can cancel that line item. We know this is a simple win, we just want to launch this quickly and move on.”

I understand the interest in being scrappy. We all want to spend most of our time adding value, and technically metrics don’t directly add value.

The product manager asked us to address the issue, and move on to their next feature.

Thankfully, particularly in this case, the engineer looking at this feature insisted on doing things properly. They instrumented the before and after engagement with this particular feature.

To the shock of everyone involved (but particularly the product manager), the change negatively impacted engagement. And not by a small amount. It was dramatic.

We investigated because this was counter to our assumptions. What we ended up learning was that many customers used our product in ways we didn’t intend or understand. And this change negatively impacted that percentage of customers.

This reinforced that metrics provide a view into our customers at scale. This provides insights (customer obsession!) into how various cohorts use our products.

It can be tough to understand all your customers. It’s one thing to watch 5 or 10 customers in a lab. And it’s one thing to dogfood a product yourself. But can you understand a million customers at scale? Unlikely, without metrics.

With all that being said, why would I write this article about metrics and KPIs being bad?

What I’m actually saying is that metrics and KPIs are so valuable and important that you’d be a fool to not use them regularly.

However, they can be overused, or abused. And as a critical useful tool, I want to make certain you recognize some of the key ways that this extremely valuable tool can be dangerous...