Fundamentals of Implementing Workplace Measurement: Principles, Pathologies, and Implementation

Introduction

Collecting measurements is a core activity for all data-driven enterprises, engineering endeavors, and scientific inquiries. Measurement objectives are fundamental for effective management and engineering. Meeting these objectives means understanding measurement capabilities and limitations, recognizing that all data collected is driven both by what is being measured as well as how measurement is done, and mastering the properties that define a “good” measurement system while actively avoiding “bad” practices.

Collecting measurements is challenging. Flawed measurement, born from well-intentioned but poorly designed initiatives, can create a form of organizational “debt”. This debt manifests as wasted resources, demotivated teams, and poor strategic decisions based on data that is misleading, inaccurate, or simply irrelevant.

Identifying bad measurement practices—and the statistical fallacies that underpin them—is not incidental. Many organizations are not suffering from a lack of measurement, but from the misapplication of it. The challenge is not to simply begin measuring, but to re-ground the organization’s measurement system in a repeatable and reliable approach. This paper serves as both a diagnostic guide to the pathologies of failed measurement systems and suggests strategies and tactics for effective, reliable, repeatable, and trustworthy measurement.

The Motivation for Measurement

Why Measure Anyway?

Measurement does not directly “satisfy a specific customer” or “create a specific product or service”. We do not measure for the sake of measuring; we measure to gain understanding and insight that informs action.

The value obtained from measuring is indispensable but indirect. A measurement system provides the only possible means for moving beyond subjective “gut feel” or group consensus, instead providing a consistent, repeatable, and objective tool for understanding the work being done. This shared, objective model of reality is the prerequisite for all rational action, comparison, and improvement.

Measurement System Benefits

A measurement system provides three benefits:

1. Objectivity and Consistency: A measurement system must provide a consistent lens for performance measurement. It is the antidote to anecdotal management, where decisions are swayed by the most recent or compelling story. Repeatable, objective data allows for the dispassionate analysis of performance over time.
2. Basis for Improvement: Measurement is the engine of all fact-based improvement frameworks, such as the Deming (Plan-Do-Check-Act) cycle. It provides the basis for improving the work being done by quantifying the “as-is” state (the baseline) and enabling the verification of the “to-be” state (the improvement). Without measurement, “improvement” becomes a subjective claim rather than a verifiable fact.
3. Basis for Comparison: Measurement is the foundation for all rational decision-making. It provides the mechanism for determining if a new process, tool, or activity is truly “better” than the baseline, or if it has simply changed. It allows for the comparison of different activities, projects, or teams using a common, agreed-upon standard.

“What Gets Measured, Gets Done”

A famous aphorism, variously attributed to Peter Drucker, W. Edwards Deming, and Lord Kelvin, states: “What gets measured, gets done”. This statement is the balancing act of management must navigate: without measurement, little will be accomplished, but measuring the wrong thing–or measuring it incorrectly–leads to unintended results. 

Measurement is a powerful tool for focusing organizational energy, signaling strategic priorities, and aligning the behavior of individuals with the goals of the enterprise. By establishing a metric, leadership communicates what it values and the organization works to optimize for that metric.

This same principle is also measurement’s single greatest risk. The power to direct behavior is also the power to unintentionally create perverse incentives. People will always act in a way to make measurement data favorable, whether that action is achieving the desired result or manipulating the system in order to appear to be achieving the result. The power to motivate and the power to create unintended, counter-productive behaviors are not separate.

Definition and Properties of Measures

Our first step is understanding the building blocks of measurement. This section establishes the technical foundations that separate a good measurement system from naive counting.

Practical Definitions

Agile software development practitioners advocate for “napkin metrics” where “rough numbers are good enough”. This Agile philosophy is not an argument for sloppiness, but one for cost-benefit analysis in measurement. A “rough” number is the desired result as long as its implicit “error bars” are small enough that any resulting decision would be identical to the one made with a higher-precision, high-cost number. This approach is excellent advice to guide how we will define measurement as being good for making fact-based decisions, but no better than that. 

Expanding on “good enough”, “cost-benefit”, and :fact-based”, a practical, working definition of a measure is: “A method agreed upon by the parties using it for assigning a value to an artifact or activity in order to compare the artifact or activity to a standard or to other artifacts or activities”.

This definition can be divided into its key components:

• “Agreed upon”: All parties involved in collection and analysis must share a common understanding of the measure’s definition and execution.
• “To compare”: A measure must provide a basis for comparison to something else–a standard or other artifacts or activities–as the basis for decision-making while recognizing that anything being measured “. . . isn’t changing, then who cares?”

Measurement Scales

Before collecting any value, we need to understand that all measurement data falls into one of four scales. These scales establish the properties of the data and what can and cannot be done with it.

Table 1: The Four Scales of Measurement

How Measurement Scales Guide Measurement

Every metric collected using a measurement system has an associated measurement scale. The scale restricts the set of valid mathematical and statistical operations that can be performed on the data. A failure to respect these restrictions, known as a scale violation, can be an expensive error in measurement. This error leads to analyses that are not just wrong, but misleading and statistically meaningless.

For example, “story points” and “defect severity” are classic ordinal scale measurements. They establish a rank (e.g., “critical” is worse than “high”; a 5-point story is “harder” than a 3-point story), but the distance between the ranks is undefined. It is statistically invalid to claim that a “critical” defect is twice as bad as a “high” one, or that two 3-point stories are equivalent to one 6-point story. Treating such ordinal data as if it were ratio data—by adding, subtracting, or, most commonly, averaging it—is a scale violation that produces a nonsensical result.

Table 2: Valid Statistical Operations by Measurement Scale

“Good” Measurement Properties

Some measurement systems perform in a superior way with respect to providing consistent data, avoiding confusion, supporting analysis that leads to decisions, all the while avoiding measurement for measurement’s sake.

Building on the measurement scales, a “good” measure must possess three distinct technical properties: precision, accuracy, and sensitivity. 

• Precision (Repeatability/Reproducible): A measure is precise if repeated measurements on the same subject, under unchanged conditions, produce identical or highly consistent results. Precision is about the consistency of the measurement system itself. Repeatability means examining the different circumstances under which measurements will be taken to make sure the measurement mechanisms will work as intended. The 2000 US presidential election voting system in Florida is a well-known example of an imprecise measurement system: two people attempting to cast the same vote (the “subject”) could have that vote counted differently due to the “hanging chad” mechanism. This was a failure of precision, as the measurement system itself introduced unacceptable variance.
• Accuracy (Validity): A measure is accurate when it measures what it was intended to measure. This is about validity, not consistency. For example, a balance weight scale (sometimes called a balance beam scale) measures an object’s weight on the Earth. However, a measure can be highly precise but completely inaccurate. For example, “lines of code” (LOC) is a notoriously inaccurate measure of “software size,” “effort,” or “complexity”. It is, however, highly precise (it is easy to count LOC repeatably and get the same results). The problem is that LOC fails to capture the multi-dimensional nature of “size” or “complexity,” making it an inaccurate proxy for those concepts.
• Sensitivity (Resolution): A measure is sensitive when meaningful changes in the subject being measured result in corresponding changes to the measurement. Also known as resolution, this property determines how discriminating the measure is. A measure that is insensitive is useless for improvement. For example, a binary (Yes/No) metric for whether a sprint’s planned activities were completed is extremely insensitive. It fails to distinguish between completing 99% of the work (“No”) and completing 10% of the work (“No”). It cannot register the small, incremental gains that are the hallmark of continuous improvement efforts. (Lack of sensitivity in a measurement system can sometimes be remedied by collecting more data.)

Type I and Type II Errors

Finally, because measurement is an act of sampling from a complex reality, all conclusions drawn from it are probabilistic and carry the risk of error. In a statistical hypothesis-testing framework—that is, a mathematical viewpoint on making decisions—these risks are formalized into two error types.

Table 3: The Hypothesis Testing Error Matrix

These are not just academic concepts but are the business risk framework for all data-driven decisions.

• A Type I Error (False Positive) occurs when one concludes there is an effect or difference when, in reality, there is none. For example, a “False Positive” would be investing in an expensive new development tool based on measurement data that falsely indicated it improved productivity. This leads to wasted investment.
• A Type II Error (False Negative) occurs when one fails to detect an effect or difference that is real. For example, a “False Negative” would be abandoning a new employee retention award because the short-term measurement system was not sensitive enough to detect its very real, but small, long-term benefits. This leads to missed opportunities.

A good measurement system must be designed to manage and account for the costs of both error types. The nature of the consequences from making bad decisions versus the cost of the measurement system dictates whether type I or II error is more important. For example, consideration of these consequences is a key driver in a judicial system: is convicting an innocent person charged with murder (false positive–type I) worse or is letting a killer go free (false negative–type II) worse? Judicial systems based on English Common Law, which includes the US judicial system, tend to treat type I errors as worse than type II errors.

Summary of Important Measurement Characteristics

Given a particular management or technical decision to be made, either one-and-done decision or ongoing decisions, the basics of implementing a good measurement system to support the decision(s) are:

• The artifact or activity to be measured must be agreed between the key participants,
• The method for analyzing the data must be valid given the measurement scale (nominal, ordinal, interval, ratio) of the data being collected, 
• The measurement system must be precise (repeatable),
• The measurement system must be accurate (provide valid insight to whatever is being measured),
• The measurement system must be sensitive so that significant changes in whatever is being measured result in changes in the collected metrics and, conversely, inconsequential changes are reflected by inconsequential changes in metrics, 
• The system must balance the consequences of making incorrect choices (false positives or false negatives) against the cost of measurement itself. 

Use Cases of Bad Measurement

“All happy families are alike; each unhappy family is unhappy in its own way,” Leo Tolstoy, Anna Karenina.

Given the properties that we want our measurement system to have, consider what happens when measurement goes poorly.

There is no comprehensive list of all the ways that a measurement program can fail; each failed program–like unhappy families–has its own special way of failing. The pathologies listed below are common examples of failure. Most failed programs exhibit some combination of these pathologies, to a greater or lesser degree. Given a particular decision objective and the need to provide useful data and analysis in support of that objective, efficiently and effectively implementing a measurement system requires both knowledge of the characteristics described previously as well as common measurement mistakes. This section provides examples of these mistakes.

The following examples provide diagnostic archetypes of failed measurement systems:

Case 1: Unactionable data–given the data, the condition(s) indicating that action is needed is unclear

Example

• Decision to be made: Are baselined items being controlled through a configuration management (CM) system?
• Metric: Count how many items are under baseline control in a CM system.
• Diagnosis: This metric is a “zombie metric”—it is technically data, but it provides no information because it is not actionable. Obviously zero items under control is a bad indicator, indicating that the CM system is not being used. But for values of items under control greater than zero, no one knows what a good number of items should be. Without context or a standard, no better course of action can be determined based on this number. It is entertaining, but not useful.

Case 2: Incomplete/insufficient data–either not enough data or not all of the different data needed are collected

• Example
  • Decision to be made: Are enough people being vaccinated for measles to provide “herd immunity”?
  • Metric: Count the number of people getting measles (MMR) vaccines.
  • Diagnosis: This is a univariate measure in a situation that demands a multivariate model. In this case, a simple count–how many people got measles vaccines–is useless without information about how many people were not already vaccinated for measles. Providing a percentage, for example, 30% of eligible people got a measles vaccine, is more useful. However, be mindful of the failure described in case 4.

Case 3: Wrong measurement scale–the measurement system collects data corresponding to a scale that will not support the needed analysis

• Example
  • Decision to be made: Southern Nevada has a water usage goal of 98 Gallons Per Capita Per Day (GPCD) or less by the year 2035. Should further water use restrictions be put in place now in Las Vegas, Nevada?
  • Metric: Number of days that mean water usage was above 98 Gallons Per Capita Per Day (GPCD) or below 98 GPCD.  
  • Diagnosis: Simply knowing that water usage on any day was 98 GPCD or less is a yes/no answer, which is nominal scale. (Counting the number of days with usage at or below 98 GPCD produces a ratio scale measurement when measured across a given time period, such as a year, but this obfuscates that the underlying data are actually nominal.) The metric is insensitive to changes in what is being measured–water usage across a year where 183 days have water usage of 99 GPCD and 182 days have usage of 80 GPCD is quite different from a year with 183 days of 200 GPCD usage and 182 days of 80 GPCD. Collecting data on the actual GPCD usage (ratio scale) has better sensitivity. 

Case 4: Discarding collected data after analysis

• Example
  • Decision to be made: Selecting between different product brands from an online retailer.
  • Metric: Percentage of five-star reviews for a product on the retailer’s website. 
  • Diagnoses: A percentage is a ratio between two numbers–the numerator and the denominator. Not providing at least one of the two numbers used in computing the ratio, either the numerator or the denominator, is misleading. A product with 100% five-star reviews that does not state the total number of reviews fails to fully inform the buyer. For example, suppose there were only 2 reviews (2 out of 2 = 100%) versus 1000 reviews (1000 out of 1000 = 100%). Percentages should always state at least the size of the denominator. 

Case 5: Inter-coder reliability

• Example 1

• • Decision to be made: Whether one software development team is more productive than another team.

• • Metric: The number of agile story points completed by each team per a four-week-long sprint.

• • Diagnosis: The definition of a story point is intentionally not standardized between teams, which means that it has no inter-coder reliability since a “3 point story” could mean entirely different levels of work and complexity between two teams. Someone from one team could judge a story as 3 points and someone from another team could legitimately judge it as 7 points.

• Example 2

• • Decision to be made: A school should be opened or closed during a snow storm.
  • Metric: The predicted total accumulation of snow for a snow storm.
  • Diagnosis: One person with authority to make the open/close decision is from Hawaii and regards any accumulation of snow as hazardous. A parent with a child who attends the school is from Rochester, New York and views storms predicted to deliver less than 12” of snow as business-as-usual. The actual metric (predicted snow accumulation) is the same, but reaching different conclusions based on interpretation of the metric is an inter-coder reliability problem.  

• Case 6: Measurement for measurement’s sake and “easy” data

• Example

• • Decision to be made: None
  • Metric: “Satisfied” versus “unsatisfied” ratings from a survey of current employee’s salary satisfaction when salaries are fixed by law, such as in a government organization.
  • Diagnosis: Organizations will often collect data simply to have data or because obtaining the data is “easy”. (Viewing measure data collection as easy is often seen in organizations where someone gets to decide that measurement should be done but is not responsible for any of the data collection costs.) Having data is viewed as good and therefore collecting data is viewed as good. However, collecting, storing, analyzing, and reporting on data all have a cost. If measurements being collected are not linked to a business or mission goal, then the organization is wasting money collecting the data. See the discussion on defining a good measurement system in Section 5.

Case 8: Correlation versus causation

• Example

• • Decision to be made: Which candidate should the business give presidential campaign contributions to in order to maximize our “political capital”.
  • Metric: Contributions will be determined by the winner of the presidential “straw poll” in Beford Falls, New York based on the fact that Bedfored Falls has aligned with the winner of the national election in every year since 1788.
  • Diagnosis: Correlation does not imply causation. (Although causation does imply correlation!) The winner of a straw poll in Bedford Falls is historically 100% positively correlated with the winner of the presidential election, but is not causative of being elected president. As of the date of this article, then have been 47 US presidential elections. At the 1800 US census, there were about 150 “towns” in the US of 500 or more people. The chances are good that some town is highly positively correlated with “causing” presidential results.

Case 9: Perverse incentives and unintended behaviors–the measurement system is manipulated to appear to meet a desired result

• Example 1

• • Decision to be made: Is a smart phone app ready for public release based on the app’s responsiveness to a user?
  • Metric: Time from button press in the app until the app displays a changed state. (If all response times are under 2 seconds, the app is considered responsive.)

• • Diagnosis:

• • • The smart phone app testing team finds that, by disabling all phone security features and background processes, the app responds to all button presses within 2 seconds. This measurement data is not helpful in making the decision on public release of the app, since users are not typically able–nor desire–to disable security features. The testing team is manipulating the measurement environment to produce misleading results.

• Example 2

• • Decision to be made: Is a smart phone app under standard operating conditions ready for public release based on the app’s responsiveness to a user?
  • Metric: Time from button press in the app until the app displays a changed state while all OEM processes are running. (If all response times are under 2 seconds, the app is considered responsive.)

• • Diagnosis:

• • • The testing team labels all response times over 2.01 seconds as anomalous and excludes this data. The justification for labeling these data points as anomalous is that 2.01 seconds is more than two standard deviations above the mean response time, which in statistical control charts is considered “out-of-control”. The testing team is again manipulating the system by excluding these data points; statistical control chart analysis requires root cause analysis of out-of-control points, not exclusion.

• Example 3

• • Decision to be made: Based on web development team costs versus results achieved, are more or fewer personnel needed on the team?
  • Metric: webproductivity = Hours spent on developmentTask Complexity where “Hours” are from a timesheet (Ratio scale) and “Task Complexity” is an Ordinal scale (1=easy, 2=medium, 3=hard) determined by the developer.

• • Diagnosis:
    • Does anyone ever believe management when they state that measuring cost or productivity is about adding more resources to a team? The metric webproductivity is really a cost metric (hours per complexity). To “look good,” a rational developer must minimize this number. Since organizations generally apply penalties for falsifying recorded work hours on a timesheet, the developer can minimize the webproductivity metric by inflating “task complexity”. Many human beings, knowing that their current and future success depends on measurement data, will choose to manipulate that data when given an opportunity.

Good Measurement

Good measurement systems don’t happen by mistake. A robust, valuable, and reliable measurement system is not an ad-hoc creation but is instead an engineered system. Its implementation follows a deliberate, high-level process.

This process is most effectively structured around the “Goal-Question-Metric-Indicator” (GQMI) framework. GQMI is a widely accepted technique for ensuring that all measurement activity is traceably linked to a strategic business need. The process includes defining goals, defining the measures, developing collection, storage, and analysis mechanisms, and finally, communicating and acting on the results.

Step 1: Define the Business/Mission Goals that Measurement Will Support

A measurement system begins by understanding mission or business information needs and objectives. The GQMI framework deriving measurement objectives from the organization’s top-level goals. The core concept is traceability. 

We build traceability between business/mission goals and measurements by identifying the business or mission need for which we need more information. This need will provide the motivational answer of “Why are we measuring this?” This goal-first step is also how we avoid measurement mistakes like “Measuring Because We Can” and “We Measured But We Cannot Act”. This step is straightforward–without stating anything about measurement or measuring, answer the questions:

• “What questions do we need to answer to know if we are achieving the business/mission goal?”
  • And, for validation, “Is answering the question(s) going to tell us if we are satisfying the mission/business goal?”
• “What do we need to know in order to be able to answer the question(s) implied by the business/mission goal?”

Step 2: Define the Measures

Once we know the questions that need to be answered to understand if we are (or can) meet the business/mission goals, then we can start building the measures to answer those questions. Of course, before we take the expensive step of creating new measures, be sure to check if there are any existing measures—perhaps with a little refinement or in combination with other measures–that will answer the questions. 

If no existing measures can be refined, adapted, or supplemented then we can proceed with defining new measures to answer the questions generated by the business/mission goal. We create these additional measures by developing an operation definition of the new measures. The operational definition is the single most critical artifact in a measurement system.

An operational definition is the specification that provides “sufficient detail to enable the measure to be collected in practice”. Operational definitions help us to avoid the error of the measurement system being manipulated to appear to meet a desired result. It is the mechanism by which a measure becomes “repeatable” (avoiding bias) and “agreed upon” (avoiding inter-code reliability).

Although developing an operational definition involves a lot of work, an operational definition provides sufficient detail to enable the measure to be collected in practice while avoiding wasteful rework and false starts: 

• Defines what data needs to be collected, 
  • Are there any data sampling criteria that must be fulfilled,
• Defines the format of the data to be collected,
• Defines the source of the data,
• Defines the timing, frequency, point in the process, and/or trigger events for data collection,
• Defines who will collect the data,
• Define any data integrity checks to be performed when data is collected,
• Define any supporting tools needed, 
• Define measurement storage, including:
  • How often will data be stored,
  • At what point during the measurement process will data be stored,
  • Who will be responsible for data storage,
  • Determine the timeline for move measurement results from data collection to storage, including any intermediary steps, and
  • Define data integrity checks to be performed when data is move to storage.
• Define the analysis mechanisms, including:
  • What data cleaning steps need to be performed,
    • For example, how will missing or suspect data be handled,
  • What data integrity checks need to be performed, 
  • What computations are to be done in conducting the analysis,
    • How many significant digits are needed,
  • Who will perform the analysis,
  • By what means, in what format, how often, by whom, and to whom will the results of the analysis be communicated
    • For example, what assumptions are made that must be explained to the personnel receiving the analysis with respect to apparent data trends that are not statistically significant or that depends on a curve-fitting technique

There are many potential error cases that an operational definition of a measure exposes and helps us to resolve. For example, consider the step “defines the format of the data to be collected”. If we are collecting time data, such as a timestamp for when an event occurred, then:

• Is the format expressed as HH:MM:SS?
• What are the rules for rounding to the nearest second?
• What time zone?

Likewise, for the source of the data, what clock is providing the timestamp—a central clock, time on a local computer, . . . ?

Developing a response to these definitions, particularly by understanding how the definitions might be vague or inconsistent, will create good operational definitions. Going the extra step of testing the operational definitions by actually collecting data on a limited basis using the operational definitions will help verify that the measurements work as intended.

Step 3: Communicate Results

The final step involves the results of the measurement analysis back into decision-supporting “information” and “insight” that are the answers to the questions developed in step 1. 

Measurement results, and the entire measurement system, are useful if and only if they answer the business/mission goal questions. Answering these questions depends not only of the technically-generated information produced by measurement, but also in expressing the information when needed and in an understandable format to the owners of the business/mission goals. Unless needed to support validation of the measurement analysis, few goal owners care about the technical details of a measure’s operational definition. The key is expressing the measurement results, in response to the business/mission goals, in a way that is (1) as brief as possible, (2) understandable, and that (3) leads to a decision (actionable).

Conclusion: Measurement as a Cyclical Process

Measurement is not a single time, “define and done” task. It is a continuous, cyclical system of inquiry. A healthy measurement program isn’t about accumulating data indefinitely, but instead answering questions and, when and if those particular questions from mission/business goals have been satisfied, moving forward to the next set of information needs. From a macroscopic viewpoint, good measurement systems are driven by the pragmatic Agile software development principle: “if it isn’t changing, who cares?” A measurement program should constantly refine its “Goal-Question-Metric-Indicator” loop to reflect new, emerging business/mission needs. By adhering to a process—from goal-setting to operational definition to analysis to communications—an organization can move from practicing measurement poorly to leveraging it as a core strategic asset for objective understanding and continuous improvement.