The Seven Fatal Flaws of Performance Measurement

By Joseph F. Castellano, Saul Young, and Harper A. Roehm

E-mail Story Print Story

Once the targets are established, most organizations measure the performance of component units by comparing targets to actual performance for certain time periods. Variances from expected results are noted and explanations are required. The popular business press trumpets the efficacy of the above approach, but this methodology has serious flaws. In fact, the design and use of performance measurement systems in most organizations suffer from a number of fatal flaws that can undermine an organization’s ability to use its measurement system to improve processes and make better decisions.

Fatal Flaw 1: Ignoring the Performance Contributions of Interactive System Elements

The first fatal flaw is the failure to view an organization as a system. All organizations are systems: networks of interdependent processes that work together to accomplish the organization’s aims. The goal should be to optimize the overall system, not its component parts. In order to achieve this optimization it is important to recognize the interrelationship between the components within the system; the need to remove all barriers to system optimization; and the impossibility of separating, and usefully measuring, component unit results from overall system results.

The “quality movement” has resulted in most organizations adopting a process view of management. Therefore, top management of most organizations would agree that optimizing the system, understanding process interrelationships, and creating cooperation between component units is both necessary and desirable. Unfortunately, most of these same organizations’ design and use performance measurement systems that treat component units as independent of each other. Such attempts not only ignore the system view of an organization but also are counterproductive.

The important message contained in this first fatal flaw is that all uses of stretch targets and benchmarks for component unit performance measurements ignore the system view of an organization. If virtually all organizations create interdependencies and the overall goal is optimization of the system, then performance measurement systems that focus solely on component unit performance must be deficient.

Fatal Flaw 2: Misunderstanding Variation

The second fatal flaw, and perhaps the root of all of the other problems associated with traditional performance measurement systems, is a failure to address variation. While a complete discussion and explanation of variation is beyond the scope of this article (see Understanding Variation: The Key to Managing Chaos, by Donald Wheeler, SPC Press), it is important to understand what happens when any process measurement fails to take variation into consideration.

There will always be inherent variation in every system component: manpower, machines, methods, materials, and environment. Yet advocates of stretch targets and benchmarking ignore variation both in setting targets and in analyzing results. In a stable process only random (or common cause) variation occurs. This common cause variation generates process capability, which can be measured and graphed in a process behavior chart or a statistical process control (SPC) chart (according to Wheeler). Unfortunately, traditional methods to establish targets or measure results cannot determine process capability.

Because all work is accomplished in processes, it is possible to obtain measurements of key performance indicators. These data serve two very useful purposes. First, they show a distribution of results over time. This is far more meaningful than the single-point comparison of data points common in traditional methods that focus on variances of actual from targeted amounts. Such data also inform management about the behavior of the underlying process from which the data are derived. Through the application of SPC, management can determine whether the process is stable; that is, whether all data points fall between statistically determined upper and lower control limits. The process average can also be calculated over time. Second, SPC data allow management to correctly interpret measurements derived from the process. If the process is stable, then the results derived from that process are predictable and the process has a definable capability. This allows realistic targets and predictions.

If the data collected from the process contain too many points that fall outside the process control limits, then the process is not stable. In such a case, the process has no definable capability. Its future performance is not predictable, and no meaningful numerical goals can be set.

Consider the following example that demonstrates why neither stretch targets nor benchmarking should be used to set numerical goals. Suppose management set a stretch target or benchmarked a best practice for its organization’s on-time delivery (OTD) at a 90% OTD measured every week. Each week a report was generated showing the actual delivery percentage against the target. These weekly reports would likely show both favorable and unfavorable variances. One week’s results might indicate only 82% OTD and the next week OTD might jump to 91%. This pattern would continue week after week, with management having no understanding of the underlying capability of the OTD process.

If the last six months of weekly OTD% had been analyzed using SPC, management might have found weekly variation between a lower control limit (LCL) of 74% OTD and an upper control limit (UCL) of 86% OTD, with a process average of 80% OTD. Setting a stretch or benchmarked target of 90% OTD was clearly beyond the capability of this process.

Also important is the inherent inability of a stable process to achieve some point-specific target each and every reporting period. Variation in any process is inevitable. The best that can be hoped for is that a process is systematically improved to reduce variation and increase average performance. Those who advocate stretch targets and benchmarking tend to ignore the difficulty of achieving such targets regularly.

Fatal Flaw 3: Confusing Signals with Noise

This flaw, which also relates to variation, is the inability of traditional measurement systems to distinguish between signals and noise when analyzing process results. Returning to the example, if during the first reporting period the company achieved an OTD of 81%, this would amount to an unfavorable variance from the target of 90%, demanding an explanation. On the other hand, a process behavior chart would have indicated that this 81% result was consistent with the existing process. If the existing process were stable [all data points were between the UCL (86%) and the LCL (74%)], the result was within the statistically determined limits of the process itself. The variation was random, or what Wheeler calls noise, and signals nothing. Management can either leave the process alone or change it if not satisfied with the results.

What the process behavior chart tells management is not to react to variation in any periodic data point, unless the variation falls outside the process’s control limits. Such a variation would signal that something outside the design of the original process was occurring. The result should be investigated to determine how the system might be changed to avoid (or reproduce) this outcome in the future.

The ability of process behavior charts to give management a methodology to distinguish between signals and noise provides a statistically based method of analysis superior to traditional analysis. Systems that focus on single-point methods of analysis treat every unfavorable result as a negative signal. Untold amounts of time, money, and talent are wasted as process managers attempt to explain negative results, when only random variation may be present.

Fatal Flaw 4: Misunderstanding Psychology

The financial reporting disasters at Enron, Sunbeam, WorldCom, and many others provide ample evidence of what can happen when the figures are distorted to get better results. Yet in the midst of these devastating failures very little attention has been paid to the enormous pressure that can occur when an organization’s performance measurement system is used to align and motivate employee behavior. While the business press has discussed the pressure to “make the numbers” at the companies mentioned above, few have called into question the continued usage of stretch targets and benchmarking.

In a recent example, the SEC charged HealthSouth Corporation and its CEO Richard Scrushy with civil accounting fraud for overstating earnings by $1.4 billion from 1999 to 2002. The SEC charged that the accounting fraud was promulgated so that the company could keep pace with Wall Street’s expectations. According to the SEC’s complaint, the CEO ordered senior accountants to “fix” earnings when HealthSouth’s performance failed to meet Wall Street forecasts. What many fail to realize is that these Wall Street expectations are just another form of stretch targets and benchmarks.

More than two decades ago, W.E. Deming warned about the dangers of using arbitrary goals and targets (New Economics for Industry, Government, and Education). Point 11 of Deming’s famous “14 points” warned management to eliminate such arbitrary targets. Such goals cannot accomplish anything apart from the methods and processes that must be used to carry out work. He often noted that such targets led to distortion of the system and the figures, as fearful workers tried to achieve the numerical goal even though this was beyond their capabilities.

Deming’s warning is largely ignored both in the business press and in most management and accounting textbooks, which advocate the use of benchmarks and stretch targets such as “tight but attainable” standards. Such courses largely overlook process capability analysis and the use of process behavior charts to ensure that targets and goals are not set beyond the capability of existing processes. Many recent financial disasters might have been avoided if top management and Wall Street analysts had heeded Deming’s warning about the “psychology of target setting”: that whenever there is fear you will get figures you cannot trust.

Fatal Flaw 5: Confusing the Voice of the Customer with the Voice of the Process

When top management sets targets, these metrics represent what management, in this case the internal customer, wants the specifications or results to be. This target-setting process, usually in the context of some Management by Objective (MBO) or Management by Result (MBR) initiative, is predicated on management’s determination to align employee behavior and motivation to achieve some predetermined result or best practice. While it is true that meeting such specifications is necessary, management’s specification—the voice of the customer—does not ensure that the underlying process generating the results—the voice of the process—is capable of achieving these results. Unless process behavior charts and capability analysis are used, management cannot determine whether the stretch or benchmarked targets are within the capability of the existing process.

Somehow the prevailing management model, again rooted in an MBO or MBR mind-set, presumes that best efforts, hard work, and the proper incentives can produce whatever specifications thought desirable by management. Targets beyond the capability of the process from which results must come will not be achieved and may instead encourage distortion of the process or the figures.

Fatal Flaw 6: Failure to Support a Process View

If you asked CEOs of publicly traded companies how important it was to manage their firms as systems of interconnected processes, their responses would likely indicate that it was very important. The focus on quality management the past two decades has made converts of companies interested in remaining flexible and responsive in meeting customer needs. Numerous articles and books have chronicled the virtues and promises of process improvement techniques. Furthermore, the growing popularity of the balanced scorecard (BSC) approach attests to the desire of organizations to link their strategies to performance measurement systems that identify the leading and lagging indicators of their metrics across four key perspectives: learning and growth, internal business process, customer, and financial. To achieve these linkages, companies adopting BSC initiatives must be cognizant of the importance of key processes that are vital to achieving their strategy. The good news is that a process management focus seems to have taken hold in many companies. The bad news is that the authors can find very little evidence that a process measurement focus, rooted in process behavior analysis and process capability, has been adopted to support this process management focus.

The BSC process provides an illustration of this problem. A review of the literature describing the importance, methodology, and advantages of BSC is silent on the issue of setting targets and goals embedded in statistically based measurements of process behavior and capability analysis. When discussions turn to how metrics are to be established, for both performance drivers and outcome measures, most discussion centers on stretch targets or benchmarked data. The authors have yet to find an article on BSC that suggests setting scorecard targets by focusing on process measurements that use SPC-based process behavior charts and capability analysis.

The failure to support a process view of a company with a process measurement focus seriously jeopardizes an entity’s ability to evaluate and manage key processes. Whether for BSC or some other process management focus, the failure to set improvement targets or to analyze process results with the use of process behavior charts opens the company to all of the fatal flaws previously discussed.

Process goals and targets (the voice of the customer) cannot realistically be established if the voice of the process is ignored. Traditional methods of target-setting and analysis have no way of informing management whether a variation in results is either signal or noise, or even whether the established target is within the capability of the process. Lack of such information seriously undermines an organization’s efforts to manage from a process focus.

Fatal Flaw 7: Misunderstanding the Real Role of Measurements

The final fatal flaw is a failure to understand the real role of performance measurements. The prevailing view of performance measurement is that its role is to measure outcomes or results against a predetermined set of targets, with the usual approach involving the following steps:

• Set numerical targets—usually stretch or benchmarked metrics.
• Use these targets to align and motivate employee behavior.
• Analyze results within set time periods.
• Evaluate employees on the basis of the results achieved.
• Reward or sanction accordingly.

This process is deceptively simple and fatally flawed for all of the reasons previously discussed. What should be the proper role of measurements? Measurement systems should focus on providing management with the feedback they need to monitor or improve key processes. Good results can only come from good processes. Numerical targets, even those that utilize stretch goals or benchmarks, can be beneficial, provided that these targets are used in the context of capability analysis and process behavior charts. By using them, management can gain valuable insights about process performance—the key to achieving better results. This information not only allows management to distinguish between signals and noise, but also provides vital information necessary for process improvement initiatives and more-informed decision making. Stretch targets and benchmarked metrics, coupled with traditional methods of analysis, do not provide management with meaningful feedback about process performance. Such methods of measurement and analysis waste valuable time, talent, and resources that could be better utilized in improving the firm’s key processes in order to achieve better results.

Flawless Management

The proper role of measurements should be seen in the context of helping employees connect with the overall aim of the organization. Management must gather and analyze information that will help employees become better contributors to the firm’s purpose. In too many organizations just the reverse happens: Measures define what is meaningful instead of letting the work itself define the measures. According to Margaret J. Wheatley (Leadership and the New Science), as employees increasingly focus on the measurements, they disconnect from the larger purpose of the firm and do only what is required and measured. The results that inevitably follow are fatally flawed.