Measurement Mindset: The Critical Thinking Before The Analysis
The following text is sourced from How to Measure Anything: Finding the Value of Intangibles in Business (D. Hubbard, 2014) to act as a book review, in addition to mentioning content from Evaluating Worksite Health Promotion (D. Chenoweth, 2001) and The Price We Pay: What Broke American Health Care-and How to Fix It (Makary, 2019).
It isn’t that everything must be put into dollars or measured through objective and quantifiable variables to be interpreted as meaningful, but we do need to document progress, use tracking as we strive for continuous improvement in terms of effectiveness and efficiency, and use a systematic approach to compare options over time as various opportunities and challenges arise.
Having a program is different than a program having impact. The right thing to do — yes, perhaps — but the right thing to do at the right time for the right people — this is the goal and it is on the path of actually helping others.
Before the analysis of data and the discredit to ourselves from the analysis conversation for if we are not a ‘numbers’ person, we must first know what to measure, and what decision we are supporting with the measurement, rather than skipping to the selection and use of a measurement instrument. Critical thinkers rejoice! Analysis isn’t solely about spreadsheets and data, it’s knowing why we are measuring, what we are measuring, and when we are measuring, then it’s the how to measure and the respective a measurement instrument.
How we think shapes both what we see and do in measurement and thus evaluation. We get to insight by building and using a new lens through asking questions, not just collecting more data and analyzing it without purpose and a pending question. We don’t want to use finite money, time, and effort, to then only realize that all of this data has been collected, but now what to do with it?
Therefore, analysis and evaluation isn’t just about numbers. This article isn’t about measuring the right things correctly (the how), it’s about measuring the right things first (the what, why, and when).
To begin, the fundamental constructs of measurement are reducing uncertainty and information value.
Reducing Uncertainty and Information Value
We must determine what we know now about the uncertain quantity, the amount of risk due to that uncertainty, and the value of reducing the uncertainty further, to compute the value of additional information based on current uncertainty. If it’s something important and something uncertain, there is a cost of being wrong and a chance of being wrong. We can make better decisions when we can reduce uncertainty through measurements and gathering data.
Information has value because it reduces risk in decisions. The value of information is equal to the value of the reduction in risk. For a decision, once we eliminate the chance of loss, any additional measurement has no or reduced value. We measure until we have gathered enough certainty about a pending decision.
Question: Is a Lack of Data the Barrier?
Before we state the lack of data as a barrier for moving forward, we must question why the data is important to gather in terms of what uncertainty toward decision-making is the data helping. What is the value of the information the data is providing? (i.e. the information value). If we have already made a decision to move forward and we are comfortable with the risk in doing so, additional data may not be of further use to us. If the current level of uncertainty about a decision is acceptable, no further measurement or data is justified.
“The problem was not a lack of data but the existence of so much data that wasn’t in a structured, easily analyzed format whereby posed questions to answer and a guiding purpose were established.”
Next in this writing, four practices surrounding measurements through the lens of critical thinking are shared, which includes the titles: Language, Questions, and Definitions, What You Do Know, The Observable, and Thresholds. Then, this writing concludes with four pieces of insight to grow and develop your measurement mindset, which are: What is True, Business Significance Versus Statistical Significance, Not Looking for Perfection, and Balancing Cost.
Four Practices Surrounding Measurements
Language, Questions, and Definitions
Getting the language right, defining what we mean, is a prerequisite to measurement. If we can define the outcome we really want, give examples of it, describe alternatives, and identify how those consequences are observable, then we can design measurements that will measure the outcomes that matter.
Therefore, a problem well stated is a problem half solved. Ambiguous and uncertain language gets in the way of measurement, and purpose for that matter. Once we figure out what they mean and why it matters (i.e. why we want to measure something), the issue in question starts to look a lot more measurable, and the resulting metrics can be more actionable for the target audience. “What do you mean by X?”, “Why do you care?”, “What does perfect look like?”, and “Who is this for?” “What key questions do you want this report to deliver?” Seemingly impossible measurements start with asking the right questions (the use of these “set up” questions), and as alluded to, difficulty defining a decision may sometimes come down to simply identifying whose decision it is.
“The problem is not simple, but well defined.”
At the same time, in health care particularly, language, questions, and definitions can be communicated using patient-centred vocabulary and being mindful of appropriateness. A plethora of surgery measurements can be developed, but the most fundamental question to craft upstream measurement can be “did the patient need the surgery?”
What You Do Know
Assessing what is currently know about a quantity is an important step for those things that do not seem as if they can be measured at all. Mathematically speaking, when almost nothing is known, almost anything will tell something. The lack of having an exact number is not the same as knowing nothing.
“Yes, there are a lot of things you don’t know, but what do you know?”
In a very practical sense, almost any improvement in organizing and presenting the data in an orderly format can be a benefit. Thus, getting organized is in and of itself within our circle of control and a benefit for the measurement mindset.
In addition, crafting a business case can break the problem down and without technically being a measurement based on new observations, reveals something about what is already known, a form of secondary research. For a new or unique situation, we can learn by examining different situations, which may not be perfect, but can be an improvement.
The Observable
The Observable content consists of using a range of possible amounts, an impact pathway, be iterative, and the act of decomposition.
If something matters, it is detectable/observable, in that we have a reason to care about some unknown quantity because we think it corresponds to desirable or undesirable results. Then, if it is observable, it can be detected as an amount, of range of possible amounts. There is nothing we will likely ever need to measure where our only bounds are negative infinity to positive infinity. Using ranges reframes the question from “What do I think this value could be?” to “What values do I know to be ridiculous?”
“The simplicity of the experiment was actually considered a point in favor of the strength of its findings.”
In addition, thinking how one thing affects another (i.e. an impact pathway) and the assumptions underlying as such, can reveal decision and the associated variables for measurement. The things that we care about measuring are also things that tend to leave tracks. “What affects what?” “How strong is the relationship?” “Work through the consequences — what should you see?”
Be iterative — don’t try to eliminate uncertainty in one giant study. When the current uncertainty is great, even a simple observation or assumption can produce a big reduction in uncertainty. People usually have so far to go that even simple methods are a big improvement. Being iterative can come in the form of new observations (i.e. further measurement) through conducting experiments. As such, creating the conditions to observe the questions and variables. “Can the phenomenon be forced to occur under conditions that allow easier observations?”
Decomposition can also be used to help make things more observable. Simple decomposing a variable into the parts that make it up can be an enlightening step. Decomposition involves figuring out how to compute something very uncertain from other things that are a lot less uncertain or at least easier to measure. We can get useful information by simply decomposing a problem and estimating its components. For example, in the workplace health and wellness domain, investing in employees’ can be decomposed in the following scenario:
Thresholds
Thresholds relate to when to stop measuring (i.e. determining a definition of done). Decisions tend to have thresholds where an action is required if the value is below it. “When is X enough to warrant a different course of action?” The point, and mentioned earlier in this writing, is conducting a measurement relative to an important decision threshold.
Note, it is a myth in that when you have a lot of uncertainty, you need a lot of data to tell you something useful. Different things have different thresholds whereby success means a particular threshold is met. You can compute the value of additional information by knowing the “threshold” of the measurement were it begins to make a difference compared to your existing uncertainty. The information value curve is usually steepest at the beginning. For example, the first 100 samples can reduce uncertainty much more than the second 100.
In a generalized manner, when is good enough actually good? At what point is there an acceptable risk acknowledged in order to move forward?
Image from How to Measure Anything: Finding the Value of Intangibles in Business (D. Hubbard, 2014) to illustrate how further resolution reaches diminishing returns quickly.
Four Pieces of Insight
What is True
The following is referenced from Principles: Life and Work (R. Dalio, 2017). To be effective we must not let our need to be right be more important than your need to find out what’s true. We are entitled to our own opinion, but not to our own facts. This is of utmost importance as we draw conclusions and go through data.
Replace your attachment of always being right with the joy of learning what’s true. Be clear on whether you are arguing or seeking to understand.
On a related note, the courage that is needed the most isn’t the kind that drives you to prevail over others, but the kind that allows you to be true to your truest self, no matter what other people want you to be. You will always get an answer, but is it a true answer is the real question.
“We are entitled to our own opinion, but not to our own facts.”
Business Significance Versus Statistical Significance
Statistical significance may not be about whether the measurement was informative or economically justified. There is a difference between reducing uncertainty (relevant to business) and statistical significance — assuming the business decisions are not based on statistically significant results such evidence-based research).
Are you trying to get published in a peer-reviewed journal, or are you just trying to reduce your uncertainty about a real-life business decision?
A statistically significant result can still have an information value of zero for a respective scenario and pending decision. Conversely, situations where a result that would have failed a statistical significance test but significantly reduced the prior state of uncertainty and risk can be of business value.
Not Looking for Perfection
The value is not in achieving perfection but in outperforming the alternative. Measurement is not necessarily synonymous with “exact count”, but can rather means an “uncertainty reduction.” It’s still measurement if it tells you more than you knew before. Measurements are more than you knew before about something that matters.
Further, measurement does not mean a total lack of error. All empirical methods have some error. The question is “compared to what?” “compared to the unaided human?” “Compared to no attempt at measurement at all?” Keep the purpose of measurement in mind: uncertainty reduction, not necessarily uncertainty elimination. The fact that some amount of error is unavoidable but can still be an improvement on prior knowledge is central to how experiments, surveys, and other scientific measurements are performed. The existence of all sorts of errors in an observation is not an obstacle to measurement as long as the uncertainty is less than it was before.
Balancing Cost
As long as the value of information shows a significant information value that is much greater than the cost of a measurement, measurement is justified to continue. In this sense, the cost of measurement is generally small compared to the cost of the decisions the measurement can support with the additional information of high value. As such, if the information value is zero, then any measurement is too expensive. If the outcome of a decision in question is highly uncertain and has significant consequences, then measurements that reduce uncertainty about it have high value and can justify cost that much more.
If someone says a measurement would be too expensive, ask “compared to what?”
Summary
Keeping the information value in mind along with the threshold, the decision, and current uncertainty provides the purpose and context of the measurement. The goal of the report is to empower the audience with the right, and timely, information. Again, critical thinkers rejoice. The how to measure and the instruments come after the right questions.
-
Nathan Kolar
