Seven Fatal Flaws of Performance Measurement
Joseph F. Castellano, Saul Young, and Harper A. Roehm
measurement systems are used to establish specific goals,
align employee behavior, and increase accountability. Organizations
often use these systems to set targets for component units
(e.g., individuals, profit centers, divisions, plants). Each
unit is expected to develop its own goals consistent with
overall targets. This process, sometimes called a “roll-up,”
reflects the premise that if all units achieve their targets
then the overall goals will be met. The methods used by most
companies to establish these numerical targets often involve
the use of stretch targets or benchmarking best practices.
the targets are established, most organizations measure
the performance of component units by comparing targets
to actual performance for certain time periods. Variances
from expected results are noted and explanations are required.
The popular business press trumpets the efficacy of the
above approach, but this methodology has serious flaws.
In fact, the design and use of performance measurement systems
in most organizations suffer from a number of fatal flaws
that can undermine an organization’s ability to use
its measurement system to improve processes and make better
Flaw 1: Ignoring the Performance Contributions of Interactive
first fatal flaw is the failure to view an organization
as a system. All organizations are systems: networks of
interdependent processes that work together to accomplish
the organization’s aims. The goal should be to optimize
the overall system, not its component parts. In order to
achieve this optimization it is important to recognize the
interrelationship between the components within the system;
the need to remove all barriers to system optimization;
and the impossibility of separating, and usefully measuring,
component unit results from overall system results.
“quality movement” has resulted in most organizations
adopting a process view of management. Therefore, top management
of most organizations would agree that optimizing the system,
understanding process interrelationships, and creating cooperation
between component units is both necessary and desirable.
Unfortunately, most of these same organizations’ design
and use performance measurement systems that treat component
units as independent of each other. Such attempts not only
ignore the system view of an organization but also are counterproductive.
important message contained in this first fatal flaw is
that all uses of stretch targets and benchmarks for component
unit performance measurements ignore the system view of
an organization. If virtually all organizations create interdependencies
and the overall goal is optimization of the system, then
performance measurement systems that focus solely on component
unit performance must be deficient.
Flaw 2: Misunderstanding Variation
second fatal flaw, and perhaps the root of all of the other
problems associated with traditional performance measurement
systems, is a failure to address variation. While a complete
discussion and explanation of variation is beyond the scope
of this article (see Understanding Variation: The Key
to Managing Chaos, by Donald Wheeler, SPC Press), it
is important to understand what happens when any process
measurement fails to take variation into consideration.
will always be inherent variation in every system component:
manpower, machines, methods, materials, and environment.
Yet advocates of stretch targets and benchmarking ignore
variation both in setting targets and in analyzing results.
In a stable process only random (or common cause) variation
occurs. This common cause variation generates process capability,
which can be measured and graphed in a process behavior
chart or a statistical process control (SPC) chart (according
to Wheeler). Unfortunately, traditional methods to establish
targets or measure results cannot determine process capability.
all work is accomplished in processes, it is possible to
obtain measurements of key performance indicators. These
data serve two very useful purposes. First, they show a
distribution of results over time. This is far more meaningful
than the single-point comparison of data points common in
traditional methods that focus on variances of actual from
targeted amounts. Such data also inform management about
the behavior of the underlying process from which the data
are derived. Through
the application of SPC, management can determine whether
the process is stable; that is, whether all data points
fall between statistically determined upper and lower control
limits. The process average can also be calculated over
time. Second, SPC data allow management to correctly interpret
measurements derived from the process. If the process is
stable, then the results derived from that process are predictable
and the process has a definable capability. This allows
realistic targets and predictions.
the data collected from the process contain too many points
that fall outside the process control limits, then the process
is not stable. In such a case, the process has no definable
capability. Its future performance is not predictable, and
no meaningful numerical goals can be set.
the following example that demonstrates why neither stretch
targets nor benchmarking should be used to set numerical
goals. Suppose management set a stretch target or benchmarked
a best practice for its organization’s on-time delivery
(OTD) at a 90% OTD measured every week. Each week a report
was generated showing the actual delivery percentage against
the target. These weekly reports would likely show both
favorable and unfavorable variances. One week’s results
might indicate only 82% OTD and the next week OTD might
jump to 91%. This pattern would continue week after week,
with management having no understanding of the underlying
capability of the OTD process.
the last six months of weekly OTD% had been analyzed using
SPC, management might have found weekly variation between
a lower control limit (LCL) of 74% OTD and an upper control
limit (UCL) of 86% OTD, with a process average of 80% OTD.
Setting a stretch or benchmarked target of 90% OTD was clearly
beyond the capability of this process.
important is the inherent inability of a stable process
to achieve some point-specific target each and every reporting
period. Variation in any process is inevitable. The best
that can be hoped for is that a process is systematically
improved to reduce variation and increase average performance.
Those who advocate stretch targets and benchmarking tend
to ignore the difficulty of achieving such targets regularly.
Flaw 3: Confusing Signals with Noise
flaw, which also relates to variation, is the inability
of traditional measurement systems to distinguish between
signals and noise when analyzing process results. Returning
to the example, if during the first reporting period the
company achieved an OTD of 81%, this would amount to an
unfavorable variance from the target of 90%, demanding an
explanation. On the other hand, a process behavior chart
would have indicated that this 81% result was consistent
with the existing process. If the existing process were
stable [all data points were between the UCL (86%) and the
LCL (74%)], the result was within the statistically determined
limits of the process itself. The variation was random,
or what Wheeler calls noise, and signals nothing. Management
can either leave the process alone or change it if not satisfied
with the results.
the process behavior chart tells management is not to react
to variation in any periodic data point, unless the variation
falls outside the process’s control limits. Such a
variation would signal that something outside the design
of the original process was occurring. The result should
be investigated to determine how the system might be changed
to avoid (or reproduce) this outcome in the future.
ability of process behavior charts to give management a
methodology to distinguish between signals and noise provides
a statistically based method of analysis superior to traditional
analysis. Systems that focus on single-point methods of
analysis treat every unfavorable result as a negative signal.
Untold amounts of time, money, and talent are wasted as
process managers attempt to explain negative results, when
only random variation may be present.
Flaw 4: Misunderstanding Psychology
financial reporting disasters at Enron, Sunbeam, WorldCom,
and many others provide ample evidence of what can happen
when the figures are distorted to get better results. Yet
in the midst of these devastating failures very little attention
has been paid to the enormous pressure that can occur when
an organization’s performance measurement system is
used to align and motivate employee behavior. While
the business press has discussed the pressure to “make
the numbers” at the companies mentioned above, few
have called into question the continued usage of stretch
targets and benchmarking.
a recent example, the SEC charged HealthSouth Corporation
and its CEO Richard Scrushy with civil accounting fraud
for overstating earnings by $1.4 billion from 1999 to 2002.
The SEC charged that the accounting fraud was promulgated
so that the company could keep pace with Wall Street’s
expectations. According to the SEC’s complaint, the
CEO ordered senior accountants to “fix” earnings
when HealthSouth’s performance failed to meet Wall
Street forecasts. What many fail to realize is that these
Wall Street expectations are just another form of stretch
targets and benchmarks.
than two decades ago, W.E. Deming warned about the dangers
of using arbitrary goals and targets (New Economics
for Industry, Government, and Education). Point 11
of Deming’s famous “14 points” warned
management to eliminate such arbitrary targets. Such goals
cannot accomplish anything apart from the methods and processes
that must be used to carry out work. He often noted that
such targets led to distortion of the system and the figures,
as fearful workers tried to achieve the numerical goal even
though this was beyond their capabilities.
warning is largely ignored both in the business press and
in most management and accounting textbooks, which advocate
the use of benchmarks and stretch targets such as “tight
but attainable” standards. Such courses largely overlook
process capability analysis and the use of process behavior
charts to ensure that targets and goals are not set beyond
the capability of existing processes. Many recent financial
disasters might have been avoided if top management and
Wall Street analysts had heeded Deming’s warning about
the “psychology of target setting”: that whenever
there is fear you will get figures you cannot trust.
Flaw 5: Confusing the Voice of the Customer with the Voice
of the Process
top management sets targets, these metrics represent what
management, in this case the internal customer, wants the
specifications or results to be. This target-setting process,
usually in the context of some Management by Objective (MBO)
or Management by Result (MBR) initiative, is predicated
on management’s determination to align employee behavior
and motivation to achieve some predetermined result or best
practice. While it is true that meeting such specifications
is necessary, management’s specification—the
voice of the customer—does not ensure that the underlying
process generating the results—the voice of the process—is
capable of achieving these results. Unless
process behavior charts and capability analysis are used,
management cannot determine whether the stretch or benchmarked
targets are within the capability of the existing process.
the prevailing management model, again rooted in an MBO
or MBR mind-set, presumes that best efforts, hard work,
and the proper incentives can produce whatever specifications
thought desirable by management. Targets beyond the capability
of the process from which results must come will not be
achieved and may instead encourage distortion of the process
or the figures.
Flaw 6: Failure to Support a Process View
you asked CEOs of publicly traded companies how important
it was to manage their firms as systems of interconnected
processes, their responses would likely indicate that it
was very important. The focus on quality management the
past two decades has made converts of companies interested
in remaining flexible and responsive in meeting customer
needs. Numerous articles and books have chronicled the virtues
and promises of process improvement techniques. Furthermore,
the growing popularity of the balanced scorecard (BSC) approach
attests to the desire of organizations to link their strategies
to performance measurement systems that identify the leading
and lagging indicators of their metrics across four key
perspectives: learning and growth, internal business process,
customer, and financial. To achieve these linkages, companies
adopting BSC initiatives must be cognizant of the importance
of key processes that are vital to achieving their strategy.
The good news is that a process management focus seems to
have taken hold in many companies. The bad news is that
the authors can find very little evidence that a process
measurement focus, rooted in process behavior analysis
and process capability, has been adopted to support this
process management focus.
BSC process provides an illustration of this problem. A
review of the literature describing the importance, methodology,
and advantages of BSC is silent on the issue of setting
targets and goals embedded in statistically based measurements
of process behavior and capability analysis. When discussions
turn to how metrics are to be established, for both performance
drivers and outcome measures, most discussion centers on
stretch targets or benchmarked data. The authors have yet
to find an article on BSC that suggests setting scorecard
targets by focusing on process measurements that use SPC-based
process behavior charts and capability analysis.
failure to support a process view of a company with a process
measurement focus seriously jeopardizes an entity’s
ability to evaluate and manage key processes. Whether for
BSC or some other process management focus, the failure
to set improvement targets or to analyze process results
with the use of process behavior charts opens the company
to all of the fatal flaws previously discussed.
goals and targets (the voice of the customer) cannot realistically
be established if the voice of the process is ignored. Traditional
methods of target-setting and analysis have no way of informing
management whether a variation in results is either signal
or noise, or even whether the established target is within
the capability of the process. Lack of such information
seriously undermines an organization’s efforts to
manage from a process focus.
Flaw 7: Misunderstanding the Real Role of Measurements
final fatal flaw is a failure to understand the real role
of performance measurements. The prevailing view of performance
measurement is that its role is to measure outcomes or results
against a predetermined set of targets, with the usual approach
involving the following steps:
Set numerical targets—usually stretch or benchmarked
these targets to align and motivate employee behavior.
results within set time periods.
employees on the basis of the results achieved.
or sanction accordingly.
process is deceptively simple and fatally flawed for all
of the reasons previously discussed. What should be the
proper role of measurements? Measurement systems should
focus on providing management with the feedback they need
to monitor or improve key processes. Good results can only
come from good processes. Numerical targets, even those
that utilize stretch goals or benchmarks, can be beneficial,
provided that these targets are used in the context of capability
analysis and process behavior charts. By using them, management
can gain valuable insights about process performance—the
key to achieving better results. This information not only
allows management to distinguish between signals and noise,
but also provides vital information necessary for process
improvement initiatives and more-informed decision making.
Stretch targets and benchmarked metrics, coupled with traditional
methods of analysis, do not provide management with meaningful
feedback about process performance. Such
methods of measurement and analysis waste valuable time,
talent, and resources that could be better utilized in improving
the firm’s key processes in order to achieve better
proper role of measurements should be seen in the context
of helping employees connect with the overall aim of the
organization. Management must gather and analyze information
that will help employees become better contributors to the
firm’s purpose. In too many organizations just the
reverse happens: Measures define what is meaningful instead
of letting the work itself define the measures. According
to Margaret J. Wheatley (Leadership and the New Science),
as employees increasingly focus on the measurements, they
disconnect from the larger purpose of the firm and do only
what is required and measured. The results that inevitably
follow are fatally flawed.
F. Castellano, PhD, is a professor of accounting,
Saul Young, PhD, is an associate professor
of operations management, and Harper A. Roehm, DBA,
CPA, is a professor emeritus of accounting, all at
the University of Dayton, Dayton, Ohio.