Measuring the immeasurable – metrics and measurement…

For organisations that seek to move Continuous Improvement (CI) beyond projects and tools and towards a ‘way of being’, measurement – what and how – is often a challenge. Those elements that are seen as critical to adoption are typically those which prove most elusive to pin down with tangible measures and metrics.

When visiting client operations, it’s frequently the case that performance boards, dashboards and CIq reports feature metrics grouped around quality, cost and delivery (output). The creation of a single framework for KPIs is a great thing – it provides consistency and alignment, and creates an easily accessible standard by which we can understand local performance.

However, when operational leaders are asked to describe the elements, they consider to be critical to embedding CI as a sustainable culture, the conversation quickly moves beyond easily measurable numbers and onto abstract behaviours. “People taking personal accountability for improvement,” “collaboration across boundaries” and “holding each other to account” are three of the most frequently cited attributes that operational leaders want to see in pursuit of CI adoption.

So, how can those leaders strike a balance between measuring the short-term imperatives of operational KPIs with the need to influence and embed intangible mind-sets? The answer is to take a broader view of measurement and recognise that it will be a constantly evolving picture that itself must be continuously improved.

Take a broader view of measurement for Continuous Improvement

The organisations that have been most successful in landing a culture of CI have a consistent approach to measurement. They typically separate the outcome metrics from the tangible and intangible inputs – recognising the relationship, but not slavishly seeking a direct causal link to a hard outcome metric. Every successful operation articulates them differently, but they can be grouped into three categories - outcome, operational, observable.

Outcome measurements

These will be directly associated with the various dimensions of performance, like quality, cost, productivity and safety, which are relatively easy to quantify and monitor. However, they’re a result and, like all retrospective measures, can be monitored but can’t be managed without reference to their inputs. For example, case throughput can be monitored but to improve the result, the input activities and processes must be improved. Or, to put it another way, “you don’t fatten a pig by weighing it.”

Operational measurements

These are related to the tangible inputs that collectively deliver the outcome. Assuming a reasonably stable process, these can be managed and improved, and will be the lead indicators of success. The concept of lead and lag metrics is nothing new, yet they are one of the most frequently overlooked areas for CI. Understanding the relationship between operational inputs and the outcome metrics is an infinite game, so operational measures should be continually reviewed and adapted.

Students of economics are very familiar with what has become known as Goodhart’s Law: “When a measure becomes a target, it becomes a redundant measure.” And what is true in macro-economics is equally true in operational environments. Set an outcome target for processes with any level of complexity and there will almost always be an innovative way to game operational requirements to achieve it. Take the example of a sales team, anyone who has run a field sales force knows that Sales Reps are the masters of working a target – holding off closure, carrying over into the next month, the angles are endless and very creative.

Observable measurements

There are the behaviours and underlying mind-sets that result in the observable operational activities. One of the most frequently abused quotations in the field of operational management is “if you can’t measure it, you can’t manage it.” The phrase originates from a 1993 book, The New Economics, by one of the most influential leaders of the Lean movement, W. Edwards Deming. The full quote has a very different emphasis and has implications for anyone seeking to build a sustainable culture of continuous improvement: “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.” Deming went on to expand this idea, identifying that for most operational leaders, the elements critical to the creation of a sustainably high performing, continuously improving operation are those that can’t be measured numerically. He knew that behaviours, culture and mind-sets – the intangible elements – are critical to success. And, as any experienced operational leader knows, you can either spend an appropriate amount of effort managing mind-sets or a disproportionate amount of time dealing with unhelpful behaviours.

Just because we can’t attach a number to something doesn’t mean that it’s impossible to evidence the current state and improvements. This is the domain of qualitative assessment and narrative evidence – familiar territory for social scientists, economists, and psychologists, but often overlooked by operational leaders. If I were asked to comment on the status of my children’s behaviour, I would never resort to numbers but would be confident in articulating progress through the medium of stories and narrative. Periodic, guided conversations (NOT surveys that are a series of closed questions) with small groups across the operation that explore a behavioural theme will quickly reveal progress on complex, intangible questions as such as ‘are we collaborating more effectively for improvement?’ or ‘are people taking accountability for performance?’

Most commonly, I see operational leaders recognise the importance of these intangible elements but have no method of assessment, so they resort to engagement scores. While these are undoubtedly important – particularly in times of economic uncertainty such as we’re experiencing, they’re not very useful in driving sustainable performance improvement and shouldn’t be conflated.

Balance continuous improvement measurement across outcomes, operational and observable

So, in pursuit of a sustainable culture of improvement, measurement must be balanced across all three categories, recognising that as we move upstream from the outcomes, the time to effect increases and requires patience and persistence.