In recent decades, what I call “metric fixation” has engulfed an ever-widening range of institutions: businesses, government, health care, K-12 education, colleges and universities, and nonprofit organizations. It comes with its own vocabulary and master terms. It affects the way that people talk and think about the world and how they act in it. And it is often profoundly wrongheaded and counterproductive.
Metric fixation consists of a set of interconnected beliefs. The first is that it is possible and desirable to replace judgment with numerical indicators of comparative performance based on standardized data. The second is that making such metrics public (transparency) assures that institutions are actually carrying out their purposes (accountability). Finally, there is the belief that people are best motivated by attaching rewards and penalties to their measured performance, rewards that are either monetary (pay for performance) or reputational (rankings).
-- Dr. Jerry Z. Mueller, The Tyranny of Metrics
In his book Other People's Money: The Real Business of Finance, Dr. John Kay confirms the fallacy of the beliefs Dr. Mueller lays out. The societal utility that banks once provided to the communities they served evaporated once bank managers familiar with their client's character and intimate with their client's needs were replaced by bank salespeople hawking financial products to clients on the basis of credit-scoring algorithms. An increase in published corporate financial data has led to a decrease in transparency as the data published is beyond the comprehension of all but the most sophisticated consumers of it. Rewarding people for hitting financial targets created trading for trading's sake and runaway bonuses, culminating in an "I'll be gone, you'll be gone" culture that intensified the 2008 financial crisis. The misplaced beliefs pointed out by Dr. Mueller lead to the undesirable outcomes described by Dr. Kay.
Dr. Mueller goes on: "Not everything that is important is measurable, and much that is measurable is unimportant."
The first part of this statement begs the question: what is important in business? I posit that in most enterprises today, the outcomes are actually less important than the means. Let that sink in for a moment. Companies have lost a lot of tribal knowledge about their systems and even their core business. They need to first regain that knowledge to put themselves on a path to make their legacy systems (a) accessible, then (b) extensible, and eventually (c) malleable again. What most enterprises desperately need is learning and growth, not more software endpoints on the fringe of an impenetrable legacy hairball. What truly matters is being able to systemically achieve outcomes; achieving outcomes in isolated instances is not a proxy measure for the intrinsic ability to do so.
This sounds great, but it is easier said than sold: regaining lost knowledge and developing the ability to do different things with it may be important but it isn't really measurable in any meaningful manner. Because boards are financially - not operationally - focused, learning and growth will never be a board priority because it doesn't appear on any financial statement. Or at least, not in a positive way: "learning" is cost bloat on the income statement, while "knowledge" is not a leverageable asset on the balance sheet. Making learning and growth a long-lived business priority is a leadership challenge that goes beyond reporting "training hours" and "number of people trained". It takes persistent, compelling storytelling that relates how successful outcomes have been directly and indirectly enabled by the journey and application of organizational learning and growth - and therefore how these outcomes have become organizationally systemic, and not accidents of chance.
Proponents of metrics champion causality: that for an action to be important it must yield some sort of measurable result. I've written elsewhere that causality can be difficult to establish, particularly in complex business environments where constant and dramatic changes inside and outside a business will create volatility of an observable metric. But the causality argument can work against prudent decision-making. For example, suppose we expect to achieve a specific cost efficiency in several stages: we first make business process change supported by some crude technology, soon followed by major technology change to more comprehensively automate that process change, and along the way we look at the data for stubbornly high-maintenance customers to weed out. Common sense tells us these are all good things to do and that the combination of these events gives us operational lift. Unfortunately, a spreadsheet analysis would conclude that investing in the comprehensive tech is useless as the bulk of the cost efficiency will be captured by the manual changes supported by crappy technology; vulnerability to things like manual error is a thin justification for allocating capital when capital is held dear. The spreadsheet analysis also concludes that efficiency cannot come at the cost of topline growth; to the spreadsheet analysis, every dollar of revenue is the same, so we keep all customers, no matter how inefficient it may be to serve them. Ironic that the spreadsheet-based decision-making makes a company both more valuable and a worse business at the same time.
The second part of Dr. Mueller's statement - "much that is measureable is unimportant" - points to the idiocy of many metrics. First, there are vanity metrics. I've relayed this case in a previous blog, but I once worked with an insurance company that used a nominally dollar-denominated coin called "business value" to measure the total impact of IT projects. In a single year they reported yielding more business value than the market capitalization of the firm. It's entirely possible they were woefully undervalued by markets, but it's more likely that their "business value" was as worthless as the PowerPoints they were pixelated on. Then there are the tenuous proxy metrics. A universal bank that had caught the Agile bug used the number of teams using Jira and Jenkins as the measure of how many teams had "gone Agile". Never mind what was actually going on in those teams, or the fact that nothing else - quality, throughput, customer satisfaction - was being effectively measured, let alone changing. The boss said we're going Agile, these are Agile tools, so once all of our people are using Agile tools we must be Agile, and the rest will follow.
Pursuing measurable value can create bigger problems if it is used to prioritize local optimization over systemic optimization. Consider a technology that accelerates systems integration and therefore reduces the cost of development of individual projects, but will very likely result in redundant integration activity across multiple project teams, a higher total cost of ownership across the portfolio of software assets, and a higher cost of change when a common system changes. A bankable lower cost today will will win out over potentially higher costs tomorrow. The prior can be measured - and managers rewarded - in the context of beating budgets for specific projects. The latter is absorbed into a business-as-usual budget, where the incremental inefficiency cannot be meaningfully disentangled from all the other incremental inefficiency piled into it. Urgent priorities always crowd out good lifestyle decisions; but metrics that justify the urgent are always more compelling than metrics that prioritize the important.
Dr. Mueller makes several recommendations for overcoming a metrics fixation, among them: "... [A]sking those with the tacit knowledge that comes from direct experience to provide suggestions about how to develop appropriate performance standards. [...] A system of measured performance will work to the extent that the people being measured believe in its worth." To do so recognizes that domain familiarity is necessary to determine the appropriate measurable outcomes. That implicitly means a definition of worth is not something that is going to come out of an abstract analysis of value. An ounce of context is worth a pound of measurements.
"With measurement as with everything else, recognizing limits is often the beginning of wisdom. Not all problems are soluble, and even fewer are soluble by metrics. It’s not true, as too many people now believe, that everything can be improved by measurement, or that everything that can be measured can be improved."
The better that we holistically understand our business and the more imaginative we are about our understanding of it, the better we intrinsically understand what it takes to make it a better business. In human systems, the whole is greater than the sum of the parts because of the intangible elements that humans bring. Consider baseball. The game of baseball has entered a stats-heavy era that has changed how people think about the game, but numbers, as Steven Kettmann put it, "eclipse a nuanced understanding of the game." Numbers provide insight and can help to re-think long held assumptions. But numbers don't tell the full story of the game. "Being alert to the twists and turns of a game is vital, since it’s the glimpses of character that emerge during these unlikely sequences that give baseball its essential flavor." Mr. Kettmann cites the example of a player's anticipation for how a play will develop as the deciding factor in a playoff game, and possibly a series. There is no spreadsheet for human decision-making in the moment.
It will take some time for the dust to settle, but results reported by Kraft Heinz last week have brought 3G Capital's management tactics - heavy cost-cutting deduced from heavy data analysis - into severe question. Those management tactics appeared to be successful for a number of years, until they weren't, and quite abruptly so. That sudden change in fortune has drawn attention to things critical to a business - asymmetric exposure to a single consumer market with subtly changing consumer tastes and an irrelevance of the consumer-products marketing model in people's daily lives - that required more than data to perceive, let alone prepare for.
"Managers agree. 'I watch the game,' said Bruce Bochy, the manager of the World Series champion San Francisco Giants. 'You don’t see me writing down a lot of things or having to look down at stats. They’re important, but there are some things that you can’t see on a spreadsheet.'"
Metrics help us to better understand something that we've learned through experience and observation. But we can never appreciate something through numbers alone: we must have the wisdom of experience and observation. Metrics are sources of data and potentially sources of information, but they are not sources of wisdom.
