Miss Measurement and her demons

What are your KPIs in your work? In your relationships? In life generally? How do others judge you, and how do you judge yourself? If you’re in the game of life, you can’t escape being measured somehow, by someone, in almost everything you do.

People working in different professions get judged by different metrics. Journalists’ work gets translated into metrics like clicks on their headlines, views of their articles, or downloads of their interviews. Academics’ work gets summarised by citations per article, grant money won per year, percentage of students satisfied with their teaching, and so on. Metrics of this sort seduce bosses around the globe and can lull people into a sense that they know where they stand, what they’re worth – even who they are, for those whose identity is based partly on their work – more exactly than ever before. But do performance metrics like these capture the whole story?

Last season on my ABC RN program, The Economists, Peter Martin and I discussed what we called the “tyranny of measurement”. We’re not the first to think of measurement as a potential tyrant. In the middle of the last century, then-US presidential candidate Bobby Kennedy famously expounded upon the many things included in Gross National Product, the forefather of GDP and purported measurer of the total worth of an economy, that were not beneficial at all. He poignantly listed “air pollution, cigarette advertising … locks for our doors and jails for the people who break them … the destruction of the redwoods and the loss of natural wonder in chaotic sprawl … napalm and nuclear warheads, and armoured cars for the police to put down riots in our cities.” 

Conversely, many things of great value, Kennedy noted, were not included in GNP: “The health of our children, the quality of their education, or the joy of their play … the beauty of our poetry or the strength of our marriages, the intelligence of our public debate or the integrity of our public officials. [GNP] measures neither our wit nor our courage, neither our compassion nor our devotion to country … it measures everything in short except that which makes life worthwhile, and it can tell us everything about America except why we are proud to be Americans.”

Apart from making us wish we had more politicians of Kennedy’s calibre, his speech reminds every economist who listens to it that GDP is a flawed measure. Yet GDP, and many other flawed measures of performance at levels from individual worker to country-wide aggregate, are still used worldwide. Why?

Metrics, judgements and effort: likely bedfellows

Many things of great value are very difficult to measure, but people need measurements in order to make decisions. We cannot operate in a vacuum, and so we are drawn, groping, to whatever metrics we can get our hands on. They comfort us. They also protect us from shouldering the adult responsibility of subjective judgement based on experience and instinct, by providing an external sensor reading that we can point to in case anyone later questions our decisions.

In the game of life, you can’t escape being measured somehow, by someone, in almost everything you do.

The downstream effects of this reliance on metrics play out according to that tribal chant of economists: incentives.

When there are rewards from being seen to perform better according to some nominated metric, behaviour gets drawn to increasing it. Said another way, what is measured draws the effort.

We see this on my home ground of academia. The Australian Research Council’s Excellence in Research in Australia (ERA) exercise, begun in 2015 and run every three years, attempts to measure one dimension of university quality using traditional gauges of academics’ work like citations, journal “impact factors” (themselves build on citation counts), and research grant income.

Recognising that this measurement approach does not capture everything that we want from our universities, the ARC this year released new metrics targeting the “engagement and impact” of academics’ work. The hope, presumably, is that by explicitly placing public value on the degree to which academics engage with and impact the society at large, this new measurement system will counteract the effort-drawing effect of the ERA measurement exercise. In a field like economics, for example, leaders of universities who take the E&I exercise seriously might reasonably encourage their academics slightly more towards influencing policy, and slightly less towards other activities, like producing undecipherable articles in elite economic theory journals.

Experts’ (mis)measurement

On our episode, Peter Martin and I talked with Professor Danny Blanchflower of Dartmouth College about his experiences trying to measure non-monetized phenomena that bring people happiness – like a happy marriage, friends, or a clean environment – by estimating how much they are worth in money terms. His work indicates that these things are worth a lot, meaning that when we don’t include them in traditional measures of individual wellbeing like income, we miss a lot of what is really valued by people.

The mismeasurement problem is also evident at higher levels of aggregation. As Blanchflower says, based on his experience advising policymakers in the UK using traditional measures of economic performance like GDP, “you don’t know where you’ve been, you don’t know where are you now, and you don’t know where you’re going”. He recalls how the Great Recession was missed by everyone because of policymakers’ reliance on GDP data that would be downward-revised later, far after intervention could and should have been taken to avert the recession. According to Blanchflower, the most reliable data about the economy in aggregate comes from “the economics of walking about” – asking people and businesses how things are doing, how they are feeling about the economy, and so on – sentiments which are to an extent captured in measures of consumer and business confidence.  An implication at the local level is that management by walking around may be a useful defence against the demons of Miss Measurement.

Returning to the university sector, Blanchflower recommends that instead of asking its academics to produce target numbers of articles in elite journals, university administrators should be asking them a question like “how are you trying to improve humanity’s lot?” This suggestion gives an implicit thumbs-up to the ARC’s new focus on Engagement and Impact.

Taming the demons

Taking in all the evidence, my advice is to judge yourself by the criteria you feel matters – even if they don’t appear in any explicit reckoning or spreadsheet – and try not to get too seduced by the metric of the month, whether you’re a manager or a front-line worker. Hone your ability to sniff out quality using all your senses, and pursue the feelings of a job well done and a life well lived.  Feelings like these are very effective at drowning out the cries of Miss Measurement and her demons.

(If you’d like to get goose bumps from hearing Kennedy’s speech spoken by him and his speechwriter, we played an excerpt on our program which, like all our programs, you can download as a podcast via the ABC Listen app. Stay tuned in April for the third season of The Economists on ABC RN.)

Gigi Foster is Professor of Economics and Director of Education, UNSW Business School.