Using Metrics to Drive Improvement: The Tail Wagging the Dog

Performance metrics help quantify the overall health of an organization or product being developed. Quantitative analysis is a mark of a mature organization. Everyone knows that successful companies collect metrics; therefore, it follows that every company needs to measure things in order to be great. Measurements gauge the efficacy of processes that affect budget, schedule, quality, and safety. Critical business decisions are made based on the results of this data.

Most articles about measurement and analysis opine on the pitfalls of measuring the wrong things, getting buy-in from stakeholders, or how to interpret the numbers collected. Obviously, our ability to make accurate predictions requires proper measurements and good statistical tools. None of that is what this blog entry is about. Instead, I’d like to challenge the very idea that performance metrics themselves ought to drive improvements.¹

Turning the screws

Improvement through tightening of metrics is a common approach for continual improvement. Want more sales? Raise the thresholds for earning high commissions, thus incentivizing salespeople to find more customers. Want to improve product quality? Start a campaign to reduce the number and severity of failure reports. Want to cut operating costs? Make it tougher to get expenses approved. All these solutions will probably succeed in reaching the new measurement goals, but may actually make things worse for the organization.

Seeking improvement by focusing on the metric can lead to perverse incentives and unintended consequences. For example, software companies have tried implementing internal “bug bounties” to improve quality. The idea behind this is obvious: bugs found early in the software development process are cheaper to fix. More bugs found and fixed during development implies less bugs discovered during formal testing. Some companies even offer their developers monetary rewards based on the bugs’ severity. These policies usually don’t last very long. The goal of finding more bugs succeeds because the developers no longer care to be careful when coding. There’s now a financial incentive to do poor work. The company prevailed in improving the metric’s output, but failed to improve quality.*

A counter-argument against my example would be that I’m not measuring enough things or making the measurements too simple. The perverse incentive problem of the internal bug bounty could be mitigated by also monitoring the frequency of bug reports found during the later stages of the software development life cycle. The effectiveness of the bug bounty could be verified by finding there were fewer bugs found during later stage testing. Are those two measurements completely correlated? No, not really. The types of bugs found during early development aren’t always the same as the ones found later on. Factors unrelated to the new profit-motivated bug squashing initiative can distort the results and yield false positives and negatives. Even more measurements and/or added layers of oversight to dissuade corruption could be tacked onto the bug bounty program. However, all these additional solutions to help prop up the initial solution add both complexity and cost. At some point we have to take a step back and ask ourselves how we got into this mess in the first place. What process was broken and how was it determined that a bug bounty was the best way to fix it? What problem was the bug bounty trying to solve? There are no answers to these questions. Nothing was broken. The existing software development process was working fine. The internal bug bounty was a solution in search of a problem. We tried to make quality better, but failed. What was wrong with our approach?

* Bug bounties that exclude internal employees from participating avoid the perverse incentive to introduce bugs for profit. However, even external bug bounties are not immune from unintended consequences.

Systems vs. goals

I reviewed Scott Adams’, “How to Fail at Almost Everything and Still Win Big” back in 2014. He talks about using systems instead of goals in chapter six.²

Throughout my career I’ve had my antennae up, looking for examples of people who use systems as opposed to goals. In most cases, as far as I can tell, the people who use systems do better. The systems-driven people have found a way to look at the familiar in new and more useful ways.

To put it bluntly, goals are for losers. That’s literally true most of the time. For example, if your goal is to lose ten pounds, you will spend every moment until you reach the goal—if you reach it at all—feeling as if you were short of your goal. In other words, goal-oriented people exist in a state of nearly continuous failure that they hope will be temporary. That feeling wears on you. In time, it becomes heavy and uncomfortable. It might even drive you out of the game.

If you achieve your goal, you celebrate and feel terrific, but only until you realize you just lost the thing that gave you purpose and direction. Your options are to feel empty and useless, perhaps enjoying the spoils of your success until they bore you, or set new goals and reenter the cycle of permanent presuccess fail.

··· ··· ··· ··· ··· ··· ··· ···

For our purposes, let’s say a goal is a specific objective that you either achieve or don’t sometime in the future. A system is something you do on a regular basis that increases your odds of happiness in the long run. If you do something every day, it’s a system. If you’re waiting to achieve it someday in the future, it’s a goal.

Continual improvement should be applied to the process (system) not the metric (goal). Identify an aspect of a process that can be improved and improve it. If that process improvement yields better measurement results, then great! However, be vigilant even when these process improvements yield favorable outcomes. Honda used to have an ad campaign where they claimed “their cars sell themselves.” The car salesman in the commercial was despondent because his customers never bothered to haggle. He had no opportunity to use his sales and negotiation skills. Buyers would just walk into the showroom and say, “I’ll take it!” This was obviously an exaggeration, but could you imagine if your company’s product was really that easy to sell? Wouldn’t that be the perfect goal to achieve? No. Absolutely not. Why? If your product is that high in demand and sells that easy, it means your prices are too low. You’re losing money on every sale.

The folly of continual improvement through tighter metrics is that it focuses on improving a number rather than the root causes that affect profitability, quality, and safety.

The principle and role of metrics

Human brains have evolved to distill our complex and multifaceted reality into smaller and simpler models. At best, metrics are a tool to help gain a sliver of insight about something too difficult to observe directly. It makes no difference if we’re talking about measurements relating to finance, manufacturing, or human health. They do not yield truth, but rather provide an abstraction of it. They are not a window on the world, but a shadow.

None of this is meant to imply that metrics are not valuable. They most certainly are. But just like any tool, they must be used properly. Measurements should be derived from project, organizational, or business objectives, and not the other way around.

Conclusion and the path forward

Continual improvement should not be driven by the numbers. It should be driven by improvements to processes. How can we accomplish this in an organization? Identify the systems that are in place and look for bottlenecks, weak spots, and opportunities for increased efficiencies. Task the process owners to come up with a few suggestions to improve their processes regardless of the measurements collected. A return on investment (ROI) should be calculated to determine whether or not the change is worthwhile. (Yes, my dear PHBs... Continual improvement costs money. But think of it as an investment. Like any good investment, it should end up saving money in the long term. Hopefully not too long.) A project plan should be developed to implement the improvement ideas that clear this hurdle. And yes, use metrics to track the modified systems and analyze the results.

References

1. Sanders, Nada R. “Metrics and Frameworks for Assessing Performance.” The Definitive Guide to Manufacturing and Service Operations: Master the Strategies and Tactics for Planning, Organizing, and Managing How Products and Services Are Produced, Pearson Education, 2014, p. 135.
2. Adams, Scott. “Goals Versus Systems.” How to Fail at Almost Everything and Still Win Big: Kind of the Story of My Life, Portfolio/Penguin, 2014, p. 40.