How Reliable Is Haphazard Sampling?

By Thomas W. Hall, Terri L. Herron, and Bethane Jo Pierce

E-mail Story Print Story

Regardless of which method is used, Statement on Auditing Standards (SAS) 39 indicates that the selection method should be expected to yield a sample that is representative of the population. Similarly, recent guidance on implementing section 404 of the Sarbanes-Oxley Act (SOA) notes that while statistical sampling is not required in these audits, samples testing internal controls should be selected in an unbiased manner (see A Framework for Evaluating Control Exceptions and Deficiencies, version 3, AICPA, December 2004).

To select a sample that satisfies SAS 39’s requirement for representativeness, current standards and related guidance indicate that all population elements must have a chance of selection and that due care must be exercised to avoid selection bias. In circumstances where nonstatistical selection methods are used, auditors must select sample items without regard to their size, shape, location, or other physical features. Also, auditors are warned to avoid distorting samples by selecting only unusual or physically small items, or omitting the first or last items in a population (see AICPA Technical Practice Aids, 2002). Auditors are presumed to be exercising appropriate due care and to be capable of selecting representative samples using nonstatistical selection methods.

In the case of haphazard sampling, the selection process is intended to emulate equal probability sampling, with the effect that all population elements have the same chance of selection. More than 40 years ago, however, noted business sampling experts W. Edwards Deming and Herbert Arkin expressed concerns that nonstatistical methods, including haphazard sampling, are susceptible to unintended selection biases (differences between desired and actual selection probabilities). Two recent research studies confirmed that haphazard sampling is susceptible to selection bias and therefore may not yield representative samples.

Research

The first research study to document selection bias in haphazard samples appeared in 2000 in Behavioral Research In Accounting (vol. 12). In this study, individuals selected samples of vouchers and inventory bins using haphazard selection. Analyses of these samples disclosed selection biases in favor of population elements that were larger, conveniently located, brightly colored, or that had fewer adjacent neighbors. Furthermore, the magnitudes of these selection biases were significant, with some elements selected 57% more often than appropriate for equal-probability sampling.

A second study, which appeared in 2001 in Auditing: A Journal of Practice & Theory (vol. 20), tested whether doubling the size of an audit sample would reliably eliminate the bias inherent in haphazard sampling. The study used populations of vouchers and inventory bins similar to those employed in the BRIA study. Typically, only about 12% of the bias was eliminated, making this approach ineffective as a method for eliminating selection bias in haphazard samples.

Why Haphazard Sampling Is Bias-Prone

The tendency of haphazard sampling to yield biased selections appears to result from subconscious human behavior in the areas of 1) visual perception and 2) the performance of tasks requiring physical effort. Regarding visual perception, research in psychology has long established that individuals see what they consciously direct their attention to, and they subconsciously see other objects that fall into their field of view. This subconscious visual perception process occurs automatically. For example, some individuals passing through a traffic intersection, once through the intersection, cannot remember looking at the light before proceeding through the intersection. For these individuals, the likelihood is that automatic subconscious processes did see the light, identified it as green, and directed continued movement through the intersection, all without their conscious recognition of the process.

This automatic subconscious visual perception process is thought to play a central role in creating biased haphazard selections via the following mechanism. In haphazard sampling, an auditor attempts to select sample items as randomly as possible. That is, population elements are selected with no specific reason for their inclusion. Procedurally, the auditor scans the population listing (or population) and, because no explicit selection strategy is followed (e.g., random or systematic selection), population elements that stand out and draw attention are selected. Even if the auditor conscientiously tries to avoid noticing any features, automatic subconscious visual processes identify these features, and function to bias the selections in favor of population elements that stand out and draw attention. The result of this process is that haphazard sample selections are likely biased by variations in the degree to which various population elements stand out visually and draw attention to themselves.

A second automatic subconscious behavior thought to affect haphazard sample selections is the innate tendency (documented by biology research) of individuals to minimize the energy expenditure in carrying out their physical tasks. This energy-conserving tendency suggests that when haphazard sampling is used, population elements that are easier to access are more likely to be selected than elements that are more difficult to access.

Given these principles from psychology and biology, one should expect haphazard samples to be biased in favor of population elements that draw attention and population elements that are conveniently located. Regarding the ability to draw attention, marketing professionals have long recognized that larger items and brightly colored items are better at attracting attention. Participants in the BRIA and Auditing studies, even though specifically instructed to choose items haphazardly, demonstrated bias in favor of large vouchers and inventory bins, as well as brightly colored inventory bins. Another factor related to the ability to draw attention is the finding from psychology research that objects with few adjacent neighbors tend to stand out and draw attention. Regarding the impact of a convenient location, most individuals who have worked with four-drawer file cabinets find that accessing contents in the top drawer and the front of each drawer requires less effort, hence the selection bias in favor of these items as reported in the BRIA and Auditing studies.

Why Increasing the Sample Size Does Not Eliminate Bias

There are good reasons to expect that increasing the size of a haphazard sample, selected without replacement, will reduce selection bias. Procedurally, the debiasing process operates as follows: First, early in the sample selection process, population elements that are best able to draw attention as well as those with more-convenient locations are selected at higher rates than justified by their population percentage. Second, as the number of sample selections increases, the chance that these overrepresented items will be chosen declines below their population percentage, because the remaining field of valid selections includes fewer and fewer of these population elements. As a result, the overrepresentation of these elements declines as the sample size increases.

This natural debiasing process does not work in audit sampling because a very large number of sample selections must be made before a shortage of attention-drawing or conveniently located elements forces the selection of other elements that are underrepresented in the sample. Based on a mathematical model presented in the 2001 Auditing study, and assuming population percentages and rates of overrepresentation similar to those observed in that study, the sampling fraction (sample size divided by population size) would need to exceed 30% for the natural debiasing process to have a meaningful effect. Because most audit samples fall well below 5%, the natural debiasing process inherent in increased sample sizes never reaches an effective level.

Practice Implications

In both the 2000 BRIA study and the 2001 Auditing study, haphazard samples were found to exhibit meaningful selection biases. The cumulative effect of these biases caused some population elements to appear in samples at rates that were three to eight times the rate of other population elements. From these studies, it is clear that haphazard sampling cannot be expected to reliably emulate equal probability sampling. Rather, haphazard samples appear likely to exhibit multiple, and perhaps unknown, selection biases. In circumstances where overrepresented and underrepresented population elements exhibit different patterns of compliance, value, or error, the result will be biased and unreliable audit assessments.

Given the results of the BRIA and Auditing studies, when a very large percentage (e.g., 30% or more) of the population is examined, haphazard selection seems likely to yield samples with no significant selection bias. When a more typical sample size, 5% or less, is used, however, haphazard sampling may well yield biased and unrepresentative samples. In these circumstances, random selection methods—e.g., simple random sampling, stratified random sampling, or monetary unit sampling—are recommended to ensure compliance with SAS 39’s requirement for representative sampling. Using random selection methods should be relatively easy given the wide availability of desktop generalized audit software.

Auditors that continue to use haphazard selection should employ multiple debiasing procedures and carefully document these procedures in their workpapers. Such procedures might include a combination of: 1) stratification by time period, location, and dollar value; 2) use of a high-value top stratum where all items are audited; and 3) an increase in overall sample size. But auditors should understand that even these procedures will not correct for bias that results from bias-inducing factors that are not well controlled by stratification and practical increases in sample size (e.g., biases due to physical size, color, and number of adjacent neighbors). Ultimately, using random selection may be the more efficient way to avoid the cost and effort of debiasing procedures.

In the unfortunate circumstance that the reliability of a haphazard sample is contested in a court proceeding or regulatory inquiry, auditors should expect to be asked about debiasing procedures, because this is one of many sampling-related questions suggested in a technical manual used by federal judges (see Reference Manual on Scientific Evidence, Federal Judicial Center, 2000). Auditors that lack a good answer to this question may find themselves in a difficult position.

Given the closer scrutiny that auditors are experiencing in the new regulatory environment, and the apparent difficulty individuals have selecting unbiased haphazard samples, auditors seeking a representative sample are advised to consider the use of random selection techniques.