Professional Documents
Culture Documents
While this document is believed to contain correct information, the author, James J. Finn does
not make any warranty, express or implied, or assume any legal responsibility for its accuracy,
completeness, or usefulness. Reference herein to any specific product or publication does not
necessarily constitute or imply its endorsement, recommendation, or favoring by the author. The
views and opinions expressed are those of the author.
I. Guideline Overview.........................................................................................................3
A. Description..........................................................................................................3
B. Scope and Application of the Guideline.............................................................3
C. Purpose of the Guideline.....................................................................................4
II. Sampling and Risk..........................................................................................................5
A. What is sampling?...............................................................................................6
B. What is Sampling Risk?......................................................................................7
III. Sample Bias...................................................................................................................9
A. Risk of Sample Bias ...........................................................................................9
B. How Sample Bias Arises...................................................................................10
1. Bias from Sampling Procedures............................................................10
2. "Crazy Eddies, Inc.", an example of fraud ...........................................12
IV. Use of Sampling in Auditing.......................................................................................15
A. Sampling Methods and Procedures...................................................................15
B. Statistical Sampling ..........................................................................................17
1. General considerations ........................................................................17
2. Specific Considerations for Auditing....................................................20
3. Valid Statistical Sampling Examples ....................................................21
C. Nonstatistical Sampling ...................................................................................27
1. General Considerations:.........................................................................27
2. Specific Considerations for auditing.....................................................28
D. Testing of Controls, Non Inferential sampling.................................................32
1. Intended End Use of a Sample (inferential vs. non inferential) ...........32
2. Sampling Steps for tests of Controls.....................................................36
E. Practical Limitations on Sampling....................................................................37
F. Statistical Inference and Sample Size................................................................38
G. The Rise and fall of Statistical Sampling in Auditing......................................39
V. Effective Statistical Sampling......................................................................................40
A. Probability Theory............................................................................................40
Intellectual Property of James J. Finn
Copyright 2009 ©
For discussion and negotiation purposes only -2-
B. Sampling Method vs. Sample Selection Methods............................................41
1. Techniques for selecting samples..........................................................42
VI. Comparative Analysis of Sampling ............................................................................44
1. Sampling in Financial Reporting Processes .........................................44
2. Statistical Process Control (SPC)..........................................................45
Bibliography.....................................................................................................................45
I. Guideline Overview
A. Description
This guideline surveys the concepts underlying the use of sampling techniques to
strengthen the sufficiency, relevance, and reliability of evidence collected to support internal
audit conclusions and managements testing for internal control and financial reporting procedure
effectiveness. Evidence that is derived from effective sampling techniques would be one way of
fulfilling the requirements of “Practice Advisory 2310-1: Identifying information”, and, since
sampling is based on testing a relatively small number of items, it can be a cost effective
technique. This guideline is intended to provide practical information related to improving
sampling techniques. In addition, this guideline reviews the history of sampling, and provides
examples of erroneous conclusions in auditing caused by intentional sample bias (fraud) or by
unintentional sample bias (incorrect sampling training or techniques). Sampling is viewed in a
comparative manner that provides insights into the use of similar sampling techniques in
industries where sampling is governed by military specifications, and ISO commercial standards.
Sampling, as used in internal auditing, has generally relied on the PCAOB authoritative
guidance contained in AU 350, the AICPA Audit Guide, and the prior AICPA guidance as
provided by SAS-39, which has been amended by SAS-111. These sources are analyzed and
expanded upon to address the practical application of statistical and non-statistical sampling
techniques for internal auditing and management control testing.
The purpose of this guideline is to provide an analysis related to the use of sampling
methods and sample selection techniques for management testing and internal auditing. In
addition, a secondary purpose is to provide a context for using sampling plans in internal
auditing and management testing for compliance programs such as Sarbanes Oxley. The scope
will include researching relevant authoritative auditing guidance, and work by others on using
sampling and probabilities in internal auditing. Also in scope will be a comparison of control-
testing sample plans with similar sample plans as they have been developed and used
commercially in other industries. A focus for this comparison will be on the special application
of attribute sampling referred to as ‘Testing of Controls’, or, in manufacturing quality control,
“Lot Acceptance sampling”. Acceptance sampling plans are widely used in manufacturing for
quality control actions such as ‘sentencing’ product lots as either accepted or rejected base on
assertions of an acceptable quality level. Also in-scope is basic probability concepts related to
sampling, and inferential statistics. The importance of ‘sample risk’ as it relates to the selection
of a ‘sample frame’ is discussed to ensure that the sample frame is representative of the
population, and that sources of potential errors that may be introduced by inadequate sample
sizes are highlighted. A further purpose of this guideline is to move beyond an intuitive
Sections II and III examine, some of the risks associated with sampling, especially
sample bias, that can distort the source of samples (the sample frame), and produce results that
are erroneous. An unknown sample risk from sample bias occurs whenever there is a systemic
bias or preference in selecting the sample that results in drawing items from a population in a
non-random or preferential manner without that being the auditors’ intention. Examples of
sample bias and its impact on audit conclusions or decision-making are presented. Section III
focuses specifically on sample bias and examines some of the common causes of biased
sampling.
Section IV analyzes the uses of sampling within the auditing profession, and provides an
examination of the history related to the use of statistical sampling and non-statistical sampling.
In addition, this section provides an introduction in to “Acceptance” sampling which highlights’
the sometimes-confusing aspects of using small sample sizes to test internal control
effectiveness. Also examined in this section are some of the limitations of sampling techniques,
including the risks involved in projecting or inferring sample features to a population. Section IV
evaluates both statistical and nonstatistical sampling methodologies and compares them in terms
of their relative strengths and weaknesses. It highlights appropriate ways to apply sampling when
auditing financial transactions and processes. This section also examines the auditing
community's acceptance of statistical sampling,i and the observed recent reluctance for applying
the more reliable and involved statistical methods of sampling in day-to-day audit work (See
Section IV, F). Section IV also explores non inferential sampling and how the ‘Testing of
Controls” and the Sarbanes-Oxley Act have impacted the auditor’s use of sampling in ways other
than to project values and percentages to a population.
Section VI compares sampling as used in internal auditing with similar attribute based
“Acceptance Sampling” plans in wide use in the high-tech and other industries, and also briefly
discusses the concepts used in sampling for Statistical Process Control (SPC).
A. What is sampling?
Sampling is selecting less than 100 percent of items from a population that contains the
complete set of the items of interest, and evaluating only the selected items for a pre-determined
value, characteristic, or attribute. The population is frequently either an account with transactions
that have amounts making up the account balance, or a document that has evidence of the
performance or completion of a control procedure. The Audit Standards define audit sampling
as follows:
Audit sampling is the application of an audit procedure to less than 100 percent
of the items within an account balance or class of transactions for the purpose
of evaluating some characteristic of the balance or class.1 This section
provides guidance for planning, performing, and evaluating audit samples.iv
This definition is specific, but is not quite identical to some working definitions used outside of
the audit profession. A definition that may be closer to one used by a business process control
manager, or general business manager could include a reference to inferring or projecting the
measured characteristic of the sample to the population. However, this difference of a reference
requiring projecting results from samples to the population is referenced in different areas of the
audit standards.
Using a sample to infer or project conclusions about a population characteristic is a
technique universally used in many aspects of business, as well as professions and trades. For
example, a criminologist may sample substances obtained during a drug raid to decide if the
entire lot is an illegal substance. The medical profession relies on extensive sampling through
clinical trials in order to determine if new pharmaceutical compounds are safe for human use.
Aircraft engineers sample and test metals and other aircraft and technical components to
determine if they will perform according to design expectations. One definition of sampling that
expresses this purpose of sampling was found in an Internet article related to “Six Sigma”
sampling, which defines sampling as follows:
Sampling risk is a significant component of overall audit risk. Audit risk consists of both
sampling risk and other risks, which may be procedural in nature, or related to other factors
involved in the audit. The auditing standards address audit risk as follows:
Sampling risk occurs because there is a probability that the items selected as a sample
are not representative of the population. As a result, the sampled item’s deviations, or their
percent of attribute successes or failures may not be the same in the sample as they are in the
population. Thus, the auditor evaluating the characteristics of the sample may not reach the same
conclusions they would if they examined the entire population.viii Selection of samples that are
not representative of the population can occur as the result of errors made in the sample
procedures such as the determination of the sample frame and/or the sample size. These
sampling errors can come from multiple sources, but the increased sample risk resulting from
errors is usually caused by not including a sufficient representation of the population’s variability
or attribute proportions in the sample. A sample may be biased systemically and, as a result, not
be representative of the population. A sample may be sized too small and, as a result, not
represent all dispersions or attributes of items in the population. A sample bias may result from
an intentional or unintentional selection of a stratum, or an incorrect portion of the true
Intellectual Property of James J. Finn
Copyright 2009 ©
For discussion and negotiation purposes only -7-
population, or result from a subjective, but inadequate, determination of sample size. Of the
potential sample errors mentioned, sample bias by an incorrect sample frame selection, or by the
determination of an inadequate sample size, can be a very significant source of sampling risk.
The possibility that an auditor may reach an incorrect conclusion because of a sampling error
(sample risk) is addressed in the audit standard as follow
Sampling risk arises from the possibility that, when a test of controls
or a substantive test is restricted to a sample, the auditor's conclusions may be
different from the conclusions he would reach if the test were applied in the
same way to all items in the account balance or class of transactions. That is, a
particular sample may contain proportionately more or less monetary
misstatements or deviations from prescribed controls than exist in the balance
or class as a whole. For a sample of a specific design, sampling risk varies
inversely with sample size: the smaller the sample size, the greater the
sampling risk.ix
Sampling risk is classified in the audit standards by the potential impact of sampling
errors on the conclusions that an auditor may reach. These risks of potentially incorrect
conclusions are divided into two categories. The categories relate to conclusions effecting either
substantive testing or the testing of controls.
It is important to note that, for financial auditors, the impact of assessing control risk either too
low or too high is to affect the determination of the level of substantitive testing required by an
external auditor. Thus, if control risk is assed as being too high, it simply means that more
substantive testing will be applied than may have been needed. However, for control procedures
design, if management testing assesses control risk as either too high or too low, the process
control design can be completely ineffective, or can result in an excessively costly overdesign of
the control procedures. Because of this difference in the end use of the sampling and testing
results, management testing for the purpose of designing and implementing internal control
processes and procedures should be more robust and employ larger sample sizes than would be
adequate for acceptance testing (control testing). The elimination or reduction of sample bias is
so important when the results are to be used for design purposes, that 100% testing, or a
statistically valid sample size should be used wherever possible rather than acceptance sampling
Regardless of the particular techniques employed, all sampling methods base their
efficacy on selecting a representative sample from a larger population of items. The sample must
be selected from a population with the necessary attributes or values of interest. Otherwise, the
sampling results will fail to represent accurately the characteristic values or attributes being
tested for the audit. A sample frame can be misleading whenever it does not include items with
the attribute that needs to be tested, or if it only contains an abnormally small portion of items
with the attribute, or because there is an intentional or unintentional preference in the items
selected to be included in the sample frame. If a sample frame is incorrectly selected, or an
inadequate sample size is used, the entire sample can be flawed and may not be representative of
the population. Even if the sample frame is selected from a relevant population, and the sample
size is adequate, there is still a possible source of sample bias. The potential source of sample
bias is the introduction of any systemic preference when selecting (picking or drawing) the
sample items. All of these sampling variables impact the reliability of the sample when it is used
for inferences about the population.
Whenever a sample frame is selected that is either not as complete as is assumed to be
(e.g. accounts payable vouchers that only include those that were paid by checks, but which
should also have included those that were paid by cash or wire transfers); or is not composed of
the type of items required to test a control attribute (e.g. vouchers as opposed to disbursements),
then the sample frame can introduce an unknown bias, and can be considered inadequate for
projections or inferences related to the controls original intended target population. Also, when
the items available to be selected within a sample frame consist of items with a systemic
departure from the randomness anticipated, or when the method of selecting the samples is
favoring some items over other items in the population, there is a high probability of sample bias.
However, the major risk in biased sampling is that the bias is an unknown bias. Intentional
stratification, however, is a known and purposeful method of sampling bias designed to reduce
sampling risk in a population with a high dispersion (variation of item values), it is not a form of
unknown sampling risk. However, it does redefine the sample frame (population) to a stratum.
Having an unknown or unintended systematic preference when selecting items in the
sample is sample bias, and this is a major risk inherent in the use of sampling. Sample bias can
produce samples, and subsequent testing results, that are not representative of the actual or
intended “target” population. In addition, if a sample frame is biased by consisting only of a
stratum of population items, or, conversely consisting of a whole mix of different items when a
stratum is expected, this can be considered sample bias. It is a biased presentation of items to be
sampled because it is not what the tester is expecting, (i.e.) it is not the target population or target
sample frame. A sample drawn in a biased manner may represent only the samples features, and
not the intended population’s features. Since the sample does not represent the target population,
projections to the population can be incorrect, or, in some way flawed to the extent the sample is
not a “mini” model of the population. Because sampling bias may be unknown or not quantified,
any inference or projection to the population based on the biased sample may lead to an incorrect
However, “acceptance-sampling” sample sizes are smaller by design since they represent
the number of items to be selected without finding a specified number of deficiencies (c). This is
because acceptance sampling is based on the laws of probability (binomial, or hypergeometric)
and uses smaller sample sizes to determine the “chance” that a defect will be found in a given
sample. Attribute acceptance sampling plans are used in auditing when performing probabilistic
sampling for tests of controls to determine an estimated level of control risk.xi When “acceptance
sampling” sample sizes are being used, the results should not be projected as values or
proportions to the population without recalculating a sample size for confidence level and
interval based on the normal probability curve and standard deviation. This will facilitate an
accurate forecast with known statistical limits. Concurrent with the results of the sample size
calculations, an additional source of sampling bias, the “what” or sample frame to be sampled is
simultaneously designed. The “sample frame” is sometimes (incorrectly) assumed to be obvious.
The “what” that is to be sampled is referred to as the sample frame or population, and is the only
population that the sample will be representative of. It is the only population to which evaluation
results can be inferred or projected. Selecting an incorrect sample frame may be one of the less
For the 1985 fiscal year, Crazy Eddie filed with the SEC an annual report on Form
10-K. In that filing, Crazy Eddie reported that it earned pretax income of $ 12.6
million. This figure, however, was fraudulently inflated by $ 2 million, or almost
twenty percent, as a result of the warehouse inventory inflations. When Eddie sold
the stock on behalf of the Relief Defendants, he was fully aware of the fraudulent
inventory inflation and its effect on Crazy Eddie's pretax income.xiii
This initial two million dollar overvaluation of inventory was achieved by misstating the
inventory counts. However, this initial misstatement was small compared to the cumulative inventory
misstatement discovered in 1987. In 1987, when the new management team valued the inventory,
they discovered a $ 60 million shortfall. This is also mentioned in the Court transcript as follows:
This huge shortfall, which was due mainly to counting distortions, has resulted in the auditors
for Crazy Eddies being criticized for not doing their job. The areas of criticism focus on not auditing
all inventory locations simultaneously, and leaving audit count sheets available to be changed
overnight. One of the documented criticisms of the inventory audits is in an article by “Joseph T
Wells” indicating that there was numerous instances of inventory fraud involving counting that could
have been caught.
Rather than climb over boxes in the warehouse, the auditors asked employees
to assist them. Crooked employees volunteered. An employee would stand on
top of a stack of television sets, for example, and call down the count to the
auditors. If there were 10 sets, the worker would claim there were 25. Repeated
many times, this clever trick helped to greatly increase the inventory count.
The message here is obvious: If you're supposed to verify the inventory count,
then you must observe it. xv
Comment: The knowledge that, in many instances, sampling did not have to
conform to a ‘valid statistical sampling’ approach may have removed most of
the corral fences around the use of sampling. In response to the acceptance of
nonstatistical sampling as evidentiary material,xvi some auditors began to
wander from the underlying statistical and probability concepts to an approach
based predominantly on judgment and intuitive risk valuation.
The previous sections II and III focused on sample risk and the element of ‘sample bias.
Bias is a pervasive problem that must be guarded against by the auditor when designing a sample
plan, especially when determining the sample frame and sample size, both of which contribute to
sample risk when unknown or unmanaged. The sample frame should be clearly defined for a
sampling plan, and the sample risk should be quantified and, although not required by the
standards, documented where appropriate.xvii Sampling risk, in many instances can be minimized
as well as quantified by applying probability theory in order to determine a statistically valid
sample size. This capability to quantify sampling risk is not available when relying on a
nonstatistical sample size.
The previous sections also focused on potential and/or actual negative impacts of sample
bias, sample size and sample count misrepresentations on financial reporting, as well as on a
specific example of the misuse of inventory counts to distort financial reporting. In these
examples, sampling methods and procedures could have been an effective internal audit
procedure to detect quantity misstatements or errors. In the next section, sampling methods and
procedures are examined as a method of preventing distortions in results based on sampling.
Define the population (the A specific definition of the pool of all A specific definition of the pool of all
sample frame). items eligible for sampling. For items eligible for sampling. For
example "all customer invoices example "all invoices for customers
processed by the accounting with a bill-to location in the USA".
department located at Boston, MA,
USA".
Determine the sample Alternatives to be considered include Alternatives to be considered include
selection method. Statistical vs. Non-statistical, and Statistical vs. Non-statistical, and
random or judgmental selection. random or judgmental selection.
Determine the sample size. Based on the sample selection Based on the sample selection
method. method.
Perform Audit procedures on Based on audit standards and Based on audit standards and internal
the sample items. internal procedures procedures
Evaluate the sample results Based on audit standards and Based on audit standards and internal
and express a conclusion. internal procedures procedures
A key element of any sampling plan is to “Determine the objectives of the sample.” as
indicated in the upper left corner of the matrix. While important for variables testing, this is
especially important for internal auditors when performing “Attribute Sampling” for control
operating effectiveness:
In order to prevent sampling risk from becoming the weak point in an audit and testing
process, it is essential to do a careful review and matching of the items being sampled to the
objective(s) of the control activity, and to the audit purpose. Comparing the attribute features
required to affect the controls risk-mitigating features to the documents can perform this.
A comparison of the audit objectives to the samples needed to test a control is necessary in order
to ensure that the control evidence being sampled does have the attribute feature required to
B. Statistical Sampling
1. General considerations
Statistical Sampling is a method of sampling where the number of items required for the
sample (the sample size) is determined by mathematics based on probabilities and characteristics
of the population such as variability and proportion. Both mathematical tablesxx and formulas are
available for use by an auditor to determine a statistically valid sample size. In addition to a
sample size based on probabilities, statistical sampling requires selecting the sample randomly
based on an equal probability that each item in the population might be selected. The
mathematics of probability distributions and algorithms take into consideration either the
variance in the population from the mean (standard deviation, sigma) or the percent of the
expected occurrence (success or failure) of the population attribute when determining a valid
statistical sample size. Using a sample that is statistically valid provides the auditor with the
ability to infer or project sample results directly to the population with a quantifiable sample risk
by establishing a confidence level and range of precision (precision interval). Both are available
as components of the calculation of sample size based on probabilities. The sample frame,
confidence level, and precision interval can be determined by the practitioner based on their
acceptability for the intended end use of the audit testing being performed. The auditor as part of
designing the statistical sampling plan establishes the acceptable precision and confidence levels
initially. A 90% confidence level or 95% confidence levels are common confidence levels for
tables and calculations. Since a statistical sample allows the auditor to define how tightly or
precise an inference from the sample to the population will be (precision interval), and since
these key parameters are determined by the auditor, the sampling assumptions, while not
mandatory, should be defined and documented when performing an audit test based on
sampling.xxi
Valid variable statistical sampling procedures should take into consideration a sampling
frame designed to represent the population, and the dispersion between the population mean and
the items value, also referred to as variability or dispersion of the items in the population. By
Intellectual Property of James J. Finn
Copyright 2009 ©
For discussion and negotiation purposes only -17-
measuring or estimating the variability of items in the population an auditor can determine a
value for the formula variable sigma; the auditor may then calculate a sample size that represents
the desired confidence level and “precision” using appropriate formulas. The “precision interval”
is the population’s target-range for the estimate. Statistical sampling is used when a statistic of
the sample (e.g. the ‘mean’) in the case of variables sampling, or the proportion of successes or
failures in the case of attribute sampling is intended to be projected (inferred) to the population
from the sample. This is the branch of statistics referred to as “Inferential Statistics” and is based
on a statistically valid sample randomly selected from the population. The sample size is
determined by applying an appropriate formula (or tables). The calculated sample size takes into
consideration the variability of items in the population (standard deviation or δ) for variables
sampling, or the expected percent of an attribute (portion) in the case of attribute sampling. The
precision or acceptable margin of error inherent in an estimate of a feature to the population is
frequently expressed as being related to the confidence level selection. The use of statistical
sampling in internal auditing is mentioned on page II-115, Section E: Engagement tools, Part II
of the “IIA CIA learning system”xxii which states:
At this point in the discussion of statistical sampling, we have established some of the
underlying structure, which leads to the benefits of a statistical sample. This is the linking of
probabilities to a sample size, and to the projection from the sample to the population. It is this
mathematical transformation using a normal probability distribution that determines these
formulas for calculating sample size. The formulas derived from the application of advanced
mathematics allow an auditor to determine a sample size by using input variables including the
probability distribution factor (Z), and a confidence interval or precision of the error rate of the
population. The result of a calculated sample size is expressed as referred to in the above quote,
as “We are 95% confident that the error rate of the population is 6%, plus or minus 3%.” It is this
ability to quantitatively define the sample-based inference to the populations sample risk in terms
of the confidence level and confidence interval that quantifies the statistical credibility of the
projection from the sample to the population. The formulas for determining sample size and
precision include the affect of the normal probability distribution including a “Z” factor
(standard deviation factor for the probability distribution) in the calculation.
There are two related but different applications for determining a statistical sample size
that have been coved thus far in this guideline. These are determining a sample size for an
average value and its precision range, and a formula for determining the sample size for an
attribute-sampling plan to estimate the percent of an attribute feature and its range or standard
error percent. The latter is also referred to as population percents for a “dummy variable”xxiii.
Determining the sample size for an attribute sample depends on what the sample will be used to
conclude. Specifically, the sample size for evaluating a population’s average and both the upper
and lower limits of the normal curve is frequently larger than a sample intended to determine the
probability that the percent of the feature in the population falls outside an upper or lower one
sided limit.xxiv Thus, there are three basic approaches of interest to determining sample sizes for
1. Sample size for variable sampling to infer a mean value and the range of variance
from the mean (two sided test).
2. Sample size for attribute sampling to infer a percent for a ‘success or failure’ of
an attribute and the range of variance or estimated error from the percentage (two
sided test).
3. Sample size to infer whether an attributes percent of ‘success or failure’ is
probably greater than or less than a selected percent (one sided limit test).
There is also a special application of attribute sampling for ‘testing of controls’, which
should not be used for inferring to a population, will be addressed later in this manuscript in a
section on “Tests of Controls”. However, the first two methods of determining a sample size can
be used to infer two different features of a population. These features are (1) determining a
sample size to infer or project a dollar value or “mean” of a sample to the relevant population -
which is usually an account balance or an account misstatement; or (2) determining a sample size
to infer or project an attributes population proportion or percentage for “success or failure” –
which can be a control attribute such as the dual sign off of checks over some specific dollar
amount. The latter is binomial in that it either exists or does not exist, (I.E. each trial has the
same specific probability of occurrence) thus, the use of the term “success or failure” of a control
attribute.
The formulas for determining a valid statistical sample size are the simplified result of
more complex calculations, which rely, among other advanced techniques, upon determining the
maximum error of the estimates. There is a structured mathematical methodology supporting
both the sample size and the inference precision when a valid statistical sample is used. This is
not so when a nonstatistical sample is used. Before moving on to a comparison of statistical and
nonstatistical sampling it is important to clarify that there are some other sampling techniques
that may be confused with valid statistical sampling. Statistical sampling is frequently confused
with “Random” sampling, or, in some cases with ‘Haphazard’ sampling selection. Statistical
sampling requires a sample size that is calculated based on probabilities and has incorporated
population variability or proportions into its calculation. Thus, even when an auditor goes
through very rigorous procedures to ensure that sampling was done on a random basis, it is not
necessarily a valid statistical sample, see the below reference:
…Any sampling procedure that does not measure the sampling risk is a
nonstatistical sampling procedure. Even though the auditor rigorously selects a
random sample, the sampling procedure is a nonstatistical procedure if the
auditor does not make a statistical evaluation of the sample results” xxv
Thus, randomness alone does not produce a valid statistical sample that can be used for
projecting from the sample to the population. Random sampling is a method of ‘drawing’ or
selecting the sample, but it is not a method of valid statistical sampling, nor is it a method of
quantifying the sample risk. Although it is true that valid statistical samples must be selected or
drawn randomly from the population to benefit from a normal probability distribution, not all
It is clear from the above excerpt that the “Operations Manual” recognizes statistically
valid sampling as the accepted method to use “…when the auditors want to make a statement
about the population from which the sample was selected” xxvii
Simply stated, statistical sampling is a credible sampling method based on probability
mathematics for projecting features from a sample to the population with a quantifiably defined
sampling risk. When using a valid statistical sample, the statistical risks of making a projection
can be quantified and stated in a formal manner. This is not the case for a nonstatistical sample
method where projections from the sample to the population are based on an auditor’s judgment.
However, as mentioned earlier in this manuscript, nonstatistical sampling has its applications and
value, but they should not be confused with the inferential capability and value of statistical
sampling.
This section focuses on the operational use of statistical sampling for auditing. Auditors
can use classical variable statistical sampling to estimate the dollar amount in an account or class
of accounts, or to estimate the difference in dollar amounts between the transaction ‘book
amounts”, and the transactions audited amounts for an account or class of accounts. This is an
application suitable for ‘classical variables’ sampling, which can be used to project a population
‘mean’ or average, and which requires an estimate of variability in the population in order to
calculate a valid statistical sample size. Examples of this application of variables sampling are as
follows:
If an auditor decides to use statistical sampling, the sample size should be calculated
using a formula or a statistical table designed for that purpose (or auditing software with the
capability). Variables sampling can be used whenever the objective of the audit is to verify that
an expected amount or other variable such as an average price, cost, or days-outstanding is in the
appropriate ‘value’ range required by management or required for operation of an internal
control. It is also used to evaluate the significance or materiality of errors made when recording
financial transactions into a journal or general ledger. In this instance, the amounts recorded in a
general ledger account would be compared to sample transactions drawn from a sample frame
containing all the transactions making up the account balances. The average ‘mean’ of the
misstatements can then be inferred to the account balance and a conclusion made regarding the
significance of the errors.
However, it is not always ‘errors’ that constitute the risk for a misstatement. The risk may
be that a corporate pricing policy is not being followed. Valid statistical sampling can provide a
basis for testing the reasonableness of an account such as revenue or accounts receivable
balances that are based on pricing standards. While it cannot verify that all transactions adhered
to the policy, it can provide some confidence to an auditor that the account balances are
reasonable or unreasonable when compared to what would be expected if the pricing policies
were being followed.
3. Valid Statistical Sampling Examples
Thus, the objective of these examples is to demonstrate the relative sensitivity of the
variables used in calculating a sample size used to support an inference (projection) of average
values, or percentages of an attribute feature, with a two-sided precision interval. Although tables
and software are available to determine the appropriate sample size more quickly, and allow
more iteration’s or “modeling” of the variables to select your sample than Example 1, Example 1
allows insight into the variables used to calculate sample sizes. See Exhibit 1 for a calculation of
sample sizes at the 95% confidence level.
In the first calculation of a statistically valid variable sample size, based on a normal
distribution, the “Z” value of 1.96 is selected from column on the far right at Seq.# 4 and placed
in the input cell located at SEQ # 1; "Z" Std deviation factor. The next step is to input a value for
the precision interval, which in this case is selected as .1 or 10%. The final value that is input for
this formula is the value of sigma, which can be determined by a separate calculation based on a
pilot sample or determined judgmentally by the practitioner. This value is important to auditors
since a large sigma, such as is found in financial transactions, will increase the required sample
size substantially. To demonstrate this, a larger sigma based on a greater population unit
variation, will be used for the calculation of Example 1B. However, as can be seen from
Example 1A, the unadjusted sample size for the population of 3000 is 96 items (Seq. # 17), and
the adjusted sample size is 93 (Seq. # 25). These results are for a relatively small sigma of .5.
In the second calculation of a statistically valid variable sample size, based on a normal
distribution, the input values are the same as in Example 1A, However, the final value that is
input for this formula is the value of sigma, which has been increased from .5 to 1.5. This value
is important to auditors since this larger value of sigma, as may be found in financial
transactions, will increase the required sample size substantially. See Example 1B the unadjusted
sample size for the population of 3000 items is 864 (Seq.# 17), and the adjusted sample size is
671 (Seq.# 25). In this instance, the adjustment for population size also has a greater impact on
the adjusted sample size, since the sample size is a larger portion of the total population.
In this first calculation of a statistically valid Attribute sample size, based on a normal
distribution, that could be used to evaluate and infer a percent or portion of a population that has
failed a control attribute, the “Z” value of 1.96 is selected from column on the far right at Seq. #
4 and placed in the input cell located at SEQ # 1; "Z" Std deviation factor. The next step is to
input a value for the precision interval, which in this case is selected as .1 or 10%. The final
value that is input for this formula is the value of “P, the probability of occurrence” (which is a
difference from the variable ‘sigma’ used for variable sampling), and can be determined by a
separate calculation based on a pilot sample or determined judgmentally by the practitioner.
However, the value used in the sample example is 50% since this will result in the largest sample
size if the other parameters remain the same. The most sensitive and important value for
determining the sample size in this example is the desired “Precision Interval” which is the
Intellectual Property of James J. Finn
Copyright 2009 ©
For discussion and negotiation purposes only -24-
accuracy or range of precision that can be estimated. This value is important to auditors since too
large a ‘Precision Interval’ can lead to uncertainty where high-risk financial controls exceed the
estimated or acceptable probability of occurrence for failures. To demonstrate this, a larger value
based on a smaller (tighter) precision interval, will be used for the calculation of Example 2B.
However, as can be seen from the first Example 2A, the unadjusted sample size for this
population of 3000 is 96 items (Seq.# 17), and the adjusted sample size is 93 (Seq.# 25).
Exhibit 2A
Seq. Formula
# Sample size calculation, for Attributes Calculation "Z" factors
Attribute Sample CL Z
In this second calculation of a statistically valid Attribute sample size, based on a normal
distribution, the “Z” value of 1.96 is selected from the column on the far right at Seq.# 4 and
Exhibit 2B
Seq. Formula
# Sample size calculation, for Attributes Calculation "Z" factors
Attribute Sample CL Z
Sample Reliabilit
Size Defects 300 Prob. C=0 y
Population 3000
C. Nonstatistical Sampling
1. General Considerations:
Nonstatistical sampling is an option available to the auditor in which the sample size, in
addition to other sampling considerations, is determined based on the auditors’ judgment. The
audit standards are clear that AU Section 350, Audit Sampling, applies to both statistical
sampling and nonstatistical sampling:
If elements of designing a reliable sample are fulfilled, and a non-statistical sample size is
large enough to include sufficient population variations to be representative of the target
population, then non-statistical sampling could produce results that are similar to the results
obtained by statistical sampling. This is discussed in the auditing standard AU 350:
The sufficiency of audit evidence is related to the design and size of an audit
sample, among other factors. The size of a sample necessary to provide
sufficient audit evidence depends on both the objectives and the efficiency of
the sample. For a given objective, the efficiency of the sample relates to its
design; one sample is more efficient than another if it can achieve the same
objectives with a smaller sample size. In general, careful design can produce
more efficient samples. [Revised, March 2006, to reflect conforming changes
necessary due to the issuance of Statement on Auditing Standards No. 105.]xxx
Intellectual Property of James J. Finn
Copyright 2009 ©
For discussion and negotiation purposes only -28-
The “Sample Frame” for a non-statistical sample may be designed to be similar or even
identical to that used for a statistical sample. A sample frame defines the items in the population
that are available for sampling, and can be similar or the same for either nonstatistical sampling
(judgmental sampling), or statistical sampling. An effort to properly select the items in the
population in a random manner to minimize sample bias can be performed with the same
diligence for either a statistically valid sample or a nonstatistical sample. Either a nonstatistical
sample or a statistical sample may be used to provide sufficient audit evidence. See AU 350
Audit Sampling:
The difference between the two sampling techniques is that the sample risk related to
inferring values or characteristics to the population when using a nonstatistical sample cannot be
quantitatively defined as it can be for a statistically valid sample. The AICPA recognizes both
the potential benefits of nonstatistical sampling and the risks involved when a target population
has a large variation or deviation of items in the population. See on Page 42 of the Audit Guide
for Audit Sampling:
However, considering the variation of items in a large population is difficult unless the
variation is measured and quantified. This variation in value(s) of individual items in the
population or “Standard deviation” is frequently calculated and referred to as “sigma” (δ) when
calculating a classical variable statistical sample. Knowing and understanding the implications of
this value is valuable if an auditor is to infer an account balance from a sample, or is to infer an
error in an account balance based on a sample. Since the nonstatistical sampling method is not a
probabilistically application of sampling, the technique of inferring account amounts (balances or
misstatements) has an element of subjectivity that can be relevant and can contribute to audit
risk. In fact, it is not possible to statistically infer or project an account balance with confidence
using a nonstatistical sample without recognizing these uncertainties related to having an
undefined sampling risk, and a high dependence on subjective judgment.
Using a stratification technique to strengthen the application of a nonstatistical sample is
highlighted in the following paragraph from the AICPA Audit Guide - Audit Sampling, page 42
The effort to separate a population by stratum for sampling improves the data from which
a projection is being developed for the population, and is effective for both statistical sampling
and nonstatistical sampling. However, it still does not ensure a statistically valid sample, and
does not have a quantifiable confidence level or precision interval. This reality is noted in the
AICPA Audit Guide to Audit Sampling, PAR 2.18.
…Any sampling procedure that does not measure the sampling risk is a
nonstatistical sampling procedure. Even though the auditor rigorously selects a
random sample, the sampling procedure is a non-statistical application if the
auditor does not make a statistical evaluation of the audit results.xxxiv
Thus, regardless of other methods used to determine sample size, a sample size is still not
based on probability theory unless a statistical analysis is performed on the sample plan. A
nonstatistical sample size is still determined based on subjectivity or “Judgment.
A focus on attribute sampling is intended to assist internal auditors to improve the testing
of internal financial controls and workflow processes controls related (primarily) to transaction
processing, and to controls where the number of times a control is applied is too high to allow
100% testing of the control attributes. While the need for the application of control attribute
sampling is clear in high volume transaction applications such as disbursements, it may also
apply to areas such as capitalizing Fixed Asset acquisitions, and the movement and valuation of
Inventories as well as controls related to the iterative process of developing standard costs for
inventory valuations. The dividing line between 100% testing and sampling is a practical
consideration (cost vs. benefit) determined by the frequency at which the control is being applied
to the transactions, the availability of documentation needed for testing, and the amount of time
required to test each item. If a control is being applied weekly (a frequency of 52) and the
required documents for testing are in one folder and it takes a minute to test each document, then
100% of the items could be tested. If, however, the required documents are co-mingled in
separate files with other unrelated documents and require sequential sorting and the folders are
filed at different locations, then selecting a nonstatistical sample and applying professional
judgment to determine the testing results may be a viable option. How many to select for a
nonstatistical sample is based on the auditors experience and judgment.
The use of control attribute testing to determine whether or not specific process controls
are being performed effectively has a wide range of application, but it can be confused and used
in place of variable testing, especially when an attempt is made to apply the results of a sample
plan designed for acceptance testing to infer a dollar balance or to extrapolate a dollar
discrepancy that was found during acceptance testing. This may result in a serious testing error,
especially if the sample size was nonstatistical, and determined based on a “Lot Acceptance
Sampling” approach, but is assumed by the auditor to be usable as a “Variables Sample” for
determining dollar values or, more frequently, percents of defects in population controls or
inventory counts. Nonstatistical sampling is prone to this error in usage because the confidence
Although this reference from the “AICPA Audit Guide to Sampling” is in the section for
“Types of Statistical Sampling Plans”, attribute sampling can be performed with both statistical
sampling and nonstatistical sampling plans. The same chapter in the AICPA Audit Guide covers
nonstatistical sampling some what in a previous paragraph in the section “Nonstatistical
Sampling and Statistical Sampling” with the following statement;
When the intended use of the sample is to project significant and relevant characteristics
of the sample to the population the sample should be an “inferential type” sample. An inferential
sample is a valid statistical sample, and should be required in all instances where the sample size
is not cost prohibitive. In order to determine the percent or proportion of defective controls in a
process, attribute sampling can be used to infer the percent of deficient controls by calculating a
statistically valid sample size either by using tables, the appropriate formulas, or statistical
auditing software. If the intended use of the sample is to determine what percent of checks over a
set dollar amount have effective duplicate signatures (the control), assuming a deviation rate
greater than 30% will usually result in a sufficient sample size for evaluation based on a normal
distribution. However, if a maximum sample size is desired, it can be calculated by assuming a
50% deviation rate in the population. This is the maximum sample size because it is the
maximum deviation rate for a binomial distribution. However, if one only wants to know -
“what are the chances that the control is only defective 10% of the time or less” then an
“acceptance” sampling methodology and corresponding sample size can be used. Thus, the
intended end use of the sample, which can be either to determine a confident estimate of a
controls specific percent defective, or merely to determine the “chances” that the deviant controls
are probably in an acceptable range, can be the major factor in the final decision of what
Certainly if 100% of an item is accessible with very little incremental effort over
drawing a sample, this is preferred to either sampling method. However, when an internal auditor
is auditing a transaction-based population with items in the hundreds, thousands, or even
hundreds of thousands of items, 100% manual testing becomes impractical (if not impossible),
and a portion of the population must be selected using sampling. This is where the intended end
use of the sample can become a major factor in deciding between statistical sampling and
nonstatistical sampling, and between one sided and two sided testing, and between classical
sampling or acceptance sampling. If the intended use of the sample’s test result is to support a
conclusion where projected sampling results will be the primary audit evidence, then valid
statistical sampling is preferred in order to provide confidence levels and statistical inferential
credibility to the conclusions. On the other hand, if the intended end use of the sample’s test
results is to be just one minor consideration among many other stronger arguments, then a
nonstatistical or acceptance sample could be adequate.
As can be seen in the previous paragraph an underlying reason for selecting between
statistical or nonstatistical sampling there is whether the auditor intends to use sampling for an
inferential application, or to use sampling to determine the level of risk or probabilities involved
in accepting or rejecting an internal control’s assertion of effectiveness. The latter is a typical end
use of internal audit sampling that is used specifically to evaluate the risk inherent in accepting
internal controls as being effective. Evaluating risks that controls may not be effective, or,
alternatively, of not accepting (rejecting) internal controls that are effective is an objective of
testing of controls.xxxvi This special application of attribute sampling (actually it can also used for
variable sampling, but that is of no concern here) is referred to in authoritative audit guidance as
“Sampling in tests of Controls”. xxxvii Chapter three in the AICPA Audit Guide on Audit Sampling
discusses this application further.
The risk of assessing control risk too high relates to the efficiency of the audit.
The auditors assessed level of control risk based on a sample may lead him or
her to increase the scope of substantive tests unnecessarily to compensate for
the higher level of perceived risk. Although the audit may be less efficient in
this example, it is nevertheless effective. However, the second aspect of
sampling risk in performing tests of controls – the risk of assessing control risk
too low – relates to the effectiveness of the audit. If the auditor assesses control
risk too low, he or she inappropriately reduces the evidence obtained from
substantive tests. Therefore, the discussion of sampling risk in the following
paragraphs relates primarily to the risk of assessing control risk too low.xxxix
Because this testing for the effectiveness of internal controls is a significant factor in
determining the course and development of the subsequent substantive testing, the design of a
sample plan must be as reliable as is practical. The testing for the effectiveness of internal
controls is the foundation for determining the remaining levels and types of testing. However, it
should be clear that the testing of internal controls is not necessarily the major component for an
auditor to form their opinion regarding the quality of the financial reports themselves. The
testing of controls provides evidence related to the overall internal control environment, not the
quality of the financial reports. For this reason, valid statistical sample sizes and inferential
techniques are frequently not used. Rather, testing techniques based on acceptance sampling
concepts are preferred. This is discussed in the audit-sampling guide as follows:
Samples taken for tests of controls are intended to provide evidence about the
operating effectiveness of the controls. Because a test of controls is the primary
source of evidence about whether the controls are operating effectively, the
auditor generally wishes to obtain a high degree of assurance that the
conclusion from the sample would not differ from the conclusion that would be
reached if the test were applied to all transactions. Therefore, in these
circumstances the auditor should allow for a low level of risk for assessing the
control risk too low. Although consideration of risk is implicit in all audit
sampling applications, it is explicit in statistical sampling.xl
If the testing of internal controls indicates a high risk of control ineffectiveness, then a
greater amount of substantive testing may be necessary. However, since this method of
‘acceptance’ testing for internal control effectiveness is based on finding either more or less
deviations then the critical value ‘c’ in the sample than has been calculated as acceptable based
on probabilities, it has the advantage of generally requiring smaller sample sizes than would be
needed for two sided testing inferential testing. It is important to understand that the acceptance
method of sampling and evaluation allows ‘sentencing’ a lot as either good or bad, but does not
tell you anything about the process creating the items in the lot. The effectiveness of smaller
sample sizes allows an auditor to evaluate the risks of accepting an auditee’s internal controls as
being effective in a fast and efficient manner, however, the sample size and the cutoff number
Exhibit 3
Formula A1 Formula A2
n=Z^2(p)(1-
P)/A^2 n=16s^2/L2
n=
Unadjusted
n = Unadjusted Sample size 246 Sample size 256
Numerator 246
1.08195413
Denominator 3
Even less appropriate than projecting a control deficiency rate to the population, is to
take a ‘value’ amount such as the ‘mean’ or deviation of dollar amounts found in the acceptance
samples, and project that statistic to the account balance or class of transactions without
determining the effectiveness of the sample size. The acceptability of a sample for projections
can be determined by using formulas similar to the ones used in the variables sampling exercise
for confidence level and precision interval. A recalculated valid statistical sample size for
Classical Variables sampling could be calculated based on using the acceptance samples standard
deviation (sigma) as a pilot sample.
In order to perform a test for control effectiveness using this reduced sample size
sampling method, the following steps could be followed.
When designing a sample plan for tests of controls, the steps required are the same as
other methods of attribute sampling described in the table included at section IV, A; however,
the method of determining the sample size is the part of the sampling method that is different.
The sample size and critical value can be determined using a binomial nomograph, or special
applications of statistical software, however, appropriate combinations of sample sizes and
critical values can also be determined using tables provided by the AICPAxlii.
In order to use these tables, the auditor must determine the factors needed to ‘pick’ the
correct sample plan. These factors include a judgmental estimate of the expected population
deviation rate, the tolerable population deviation rate, and the acceptable risk of assessing control
risk too low. Because these factors can be specific to the particular circumstances the factors are
determined by the auditor using their experience and judgment and may initially be expressed in
a relative and qualitative manner. The general effect of these factors is discussed in the AICPA
Audit Guidexliii. Once the magnitude of these factors has been determined, it is necessary to view
them quantitatively when using the tables.
Intellectual Property of James J. Finn
Copyright 2009 ©
For discussion and negotiation purposes only -36-
The auditor must first quantify the probability of assessing the Control Risk too low’ in
order to select an appropriate level table of sample sizes. The two most frequent tables used are
those published in the audit guide as tables A1, and A2.These are 95% risk Table A1), and 90%
risk (Table A2). Once the desired table has been selected, it is then necessary to quantify the
‘Expected population deviation rate’ and the ‘Tolerable Rate’ in order to select sample plans
from the tables. An example of this process to select a sample plan is as follows.
• Based on the audit environment, a 5% risk of Assessing Control Risk Too Low is
considered appropriate. This results in using the ‘Table A.1’.xliv
• Based on previous audits, the “Expected Population Deviation Rate” is considered
to be low so a rate of 1% is selected as the row in the table.
• The control being tested is the approval of purchase orders, so the tolerable rate
cannot be too high. A tolerable rate of 5% is selected as the column in the table.
The result of this process is to select a sample size of 93 items, and a critical value of (1) defect.
If subsequent testing of the items sampled results in finding 1 or fewer deviations, the auditor
can conclude that the desired risk of assessing control risk too low is not more than the tolerable
ratexlv. Additional interpretations of the results of sampling are also included in the Audit
Sampling Guidexlvi
Even when an auditor uses a valid statistical sample, the relevance of a projection of the
sample results to the population must be understood and evaluated based on sound judgment. A
projection may be inadequate because of the selection of variables involved in determining the
sample plan. The internal auditor must review and determine the adequacy of the confidence
level assumption i.e. is 90 % or 95% really sufficient or is the population being audited sensitive
enough to require a 99% confidence level. The variables involved for calculating sample size are
different for variable sampling, attribute sampling, and the special form of attribute sampling –
acceptance or npc sampling. Variable sampling plans are applied most frequently to confirm
account balances, or misstatement amounts in dollars, and are therefore most concerned with
values and variability in the population or sigma. Whereas, acceptance-sampling plans are
applied most frequently to determine whether a control attribute is operating effectively, and, are
therefore most concerned with the binomial distribution, or binary conditions – such as whether
reconciliation exists or doesn’t exist. Variable sampling plan formulas require the input of a
value for the variance or “δ” of the population (usually an estimate or result of a pilot sample)
since the inference will be in the form of a population mean and the variance around that mean.
These components of sample size determination must be evaluated knowledgably, and taken into
consideration when performing a projection from samples to populations
This analysis focus on attribute sampling plans since they are the essence of sampling
plans for determining either the proportion of “successes or failures” of controls in the
population. These are the most comprehensive sampling plans by internal auditors or process
managers to test a control attributes success or failure percent in a population. In attribute
sampling plans two of the critical input variables are the ‘expected occurrence rate’ or
probability of occurrence, and the desired ‘Precision”, or acceptable range of probable inference.
The usability of attribute sampling plans and the resulting statistical inference is calculated by
the variables mentioned previously and a selection of the related Confidence Level or tolerable
deviation rate.
Many external auditing firms focused on Statistical (probabilistic) sampling for the
period of time up to and immediately after the issuance of SAS 39; however, that focus has
changed during the decade of the 1990’s to a preference for the use of nonstatistical sampling to
base a judgmental projection of values or control deviation ratings. The use of sampling plans in
Internal Audit and Sarbanes-Oxley testing of internal controls that are not statistically valid for
inference, has become an acceptable methodology because of its simplicity, acceptability, and
cost effectiveness – not because of its inferential statistical reliability. Sarbanes-Oxley
acceptance testing for the risk of control effectiveness has became a popular auditing procedure,
however, the results, in some instances, were projected to the population without verifying the
statistical validity of the sample size. The result can be similar to inferring from a non-statistical
sample. The basis for a projection to a population from a sample that is not statistically valid for
inferential statistics should be defined as such by the auditor. In some instances, the application
of non-statistical sampling is also referred to as “Judgmental Sampling” or “Haphazard
Sampling”.
The emergent use of nonstatistical sampling techniques has been criticized by one of the
authors of SAS No. 39xlvii, the auditing standard that supported nonstatistical samplings’
acceptability as evidence in auditing.
Throughout the 1960s and ’70s, the largest accounting firms devoted extensive
resources to the development and implementation of statistical sampling
procedures. The firms wrote new policies and guidance, developed time-
sharing and batch computer programs, and trained specialized staff. Monetary
unit sampling was developed and became a widespread audit tool. The AICPA
issued Statement on Auditing Procedure (SAP) 54 and published Statistical
Auditing, by Donald M. Roberts. Then, in 1980, the Auditing Standards Board
(ASB) issued SAS 39, Audit Sampling (AU 350). Members of the Statistical
Sampling Subcommittee that wrote SAS 39, which included this author,
expected that the imposition of risk, materiality, and selection requirements
would further establish statistical sampling as a principal audit testing
procedure. In fact, the opposite has occurred, largely because the ASB gave
nonstatistical sampling equal evidentiary weight.xlviii
A. Probability Theory
Even at a very basic level, the selection of a population to be tested, or the effectiveness
of selecting a sample at all (versus inquiry or observation) can be improved by applying
quantitative probability concepts. However, regardless of where the term is used, the usage of the
term ‘probability’ has a wide range of interpretations. One of the interpretations of probabilities
include measuring the percent of an attribute based on the frequency of occurrence (see page 2,l)
to a completely subjective evaluation of entity controls based purely on an educated opinion.
The percentage probabilities based on a frequency of occurrence are the quantitative assessments
of specific control attributes such as date, amount, signature, and existence of a filed document,
whereas a subjective attribute may be “sufficient support”, or ‘adequate review’. Even
determining ‘what’ the population to be sampled should be, can be a point of disagreement or
Intellectual Property of James J. Finn
Copyright 2009 ©
For discussion and negotiation purposes only -40-
uncertainty. However selecting a population and a sample frame can be thought out in a clearer
manner by applying the logic of “sets” (see page 5 footnotes 19). An example of an application
of logical sets would be in the area of AP transaction sampling. There is a large universe of
different types of AP transactions, but if the specific attribute to be tested is whether a ‘receiver
for a document’ is properly verified prior to vouching a vendor invoice, the auditor actually
wants to select only that set of transactions that require a receiver to be created. This is
frequently manufacturing materials, capitalized purchases, and manufacturing MRO items. This
is a subset of the total AP transaction universe, and would constitute the appropriate population
for the sample frame. This is not a small or unimportant problem. Quality assurance evaluations
on previously tested controls specified verifying the receiver was matched to the PO, and yet the
population for selecting the sample frame was defined simply as ‘AP vouchers’; however
‘vouchers’ also included approved documents that did not require or have ‘receivers’.
Thus, the “take away” recommendation on the issue of probability theory for sample
selection is that an internal auditor, who has a solid understanding of probabilities and how they
impact sample sizes and sampling in general, is much more likely to be an effective
communicator of the credibility of sampled test results – even under client pressure. I don’t think
management - without a statistically valid sample would have believed a negative conclusion by
the auditors at “Crazy Eddies”.
Analysis up to this point has focused on dividing sampling methodologies into major
classifications or methods. All sampling methods can be categorized into one of two categories.
These categories are: 1.) Statistically valid sampling; or 2.) Nonstatistical sampling. In a
statistically valid method of sampling the sample size is based on calculations or algorithms
relying on probability theory and rigorous mathematical proofs which provide quantitative
values for measures of confidence level, variability, occurrence rates, and precision for the range
of inferred values. Nonstatistical sampling does not provide these mathematical links to
probability distributions. However, while previous discussions were intended to reduce
confusion regarding the usage of statistical and nonstatistical sampling, there is also another
aspect of sampling where there may be significant confusion in both auditors and clients. This
confusion occurs when they are determining or discussing a ‘sample selection method’. The
sample selection method relates to how the samples are physically acquired or ‘drawn’ from the
population rather than to the category of either statistical or non-statistical sampling plan
determination.
The sample selection method is mandated when using statistical sampling to a method
that provides an equal probability to each population item of being selected (i.e.) a simple
random sample. Generally, the most effective technique for achieving this ‘randomness’ is to use
random number sample selection, or interval sampling beginning with a random starting point.
In addition to these two methods, “cluster” sampling may be accepted in some instances where
there is a multi stage sample selection procedure being used. A multistage selection usually first
selects clusters (or locations) at random, and then samples 100% of all items within the selected
cluster(s), or alternatively, calculates a statistically valid sample size within the random cluster.
Test results are determined for the selected clusters, which are then inferred to all other clusters.
However, the use of cluster sampling for statistically valid sampling programs may not be
acceptable to some testing programs unless there is sufficient reason to believe that all clusters
Intellectual Property of James J. Finn
Copyright 2009 ©
For discussion and negotiation purposes only -41-
have an identical distribution of attributes. In the treatment of cluster sampling, the IRS Treasury
Departments internal audit program does not accept applying the results of one branch to all
branches, in the respect that the results from statistically valid samples selected at one field office
are not acceptable as the basis of conclusions at other field offices.
Nonstatistical sampling, however, has any number of available sample selection methods
that can be applied based on the circumstances or the professional judgment of an auditor.
However, the conclusions related to the sample can only be applied to the sample. Any
projection to the population is actually based on a subjective evaluation by the auditor. In fact, as
mentioned above, a structured random sample selection method may actually be used to select
the samples, even though the end result is still a nonstatistical sample size selection
methodology. Other popular sample selection methods, their appropriate use, and brief
definitions are as follows:
a. Haphazard:
This is a method that approximates the effect of random sampling by selecting samples
without any intended bias and with no discernable pattern. Examples of this method
include selecting document files from a filing cabinet in an unbiased or haphazard
manner, or selecting purchase orders from a listing of all purchase orders with no
apparent bias or preference. When using this sample selection method, the auditors’
intent is to select items that are chosen without intentional bias. However, even when an
auditor applies a haphazard sample selection method based on Excel to generate selection
numbers, or a random number generator pick the items for selection, they are using a
random sample selection method, but may be applying it to a nonstatistical sampling plan
as discussed in the previous paragraph.
b. Interval:
This is a method that can be accepted as a random sample if the beginning point
for the interval is chosen at random and there are no periodic or cyclic recurrences
expected in the population. However, this method may be biased if a listing that is used
Intellectual Property of James J. Finn
Copyright 2009 ©
For discussion and negotiation purposes only -42-
for the representative sample frame has a built in cyclic variation or recurrence. An
example of this problem occurs when the auditor is testing checks over a specific dollar
amount, and the only listing available is a check register that includes all checks over the
specified dollar amount in which the first 5 checks for each weekly check run are for the
same items over the dollar amount such as rent, insurance, employee benefits or other
cyclic payment. As a result of this periodicity, those checks, which are run every check
run, have a higher probability of being selected thus; there is a bias in the selection. When
using this sample selection method, the practitioners’ intent is to select items that are
chosen without a known bias, as a result, it can be applied to statistical sampling, but it is
also frequently used for nonstatistical sampling. In any case, it is best used if the auditors’
judgment is that periodicity is an acceptable bias, or is non-existent. Finally, even though
the auditor may be comfortable applying it to a nonstatistical sampling plan, it may not be
appropriate for all statistically valid sample plans because of the possible bias caused by a
periodic or cyclic re-occurrence.
c. Judgmental:
d. Block:
The sampling technique is named for the manner in which the accounts are
selected to be audited. The probability of an account being included in the
sample is proportional to its size. As a result, large accounts have a higher
probability of being audited. Thus the technique automatically provides audit
evidence to large accounts in a population.li
This method of sampling requires using a set of tables and methods of evaluation that are
specific to the methodology. Details for using this methodology can be found in the sampling
guide for internal auditorslii
The number of transactions being processed in today’s financial payments systems and
financial reporting systems make it almost impossible to manually select and audit 100% of the
transactions for control attributes or dollar amounts. This has provided an impetus for Computer
Assisted Auditing Techniques (CAAT). CAAT software, with well-designed cost effective
sampling programs, is some of the most economically feasible methods to sample transactions
(up to 100%), for control acceptance testing, or account ‘Summing’ assurance. For these
reasons, the technique can be the cornerstones for gathering auditable information on automated
transaction-based processes, and for systems utilizing large database management systems. For
organizations with consolidated subsidiaries, the value of CAAT sampling and testing of
transactions from the subsidiary for accuracy and correctness can be a significant time saver, and
there is audit software available for this purpose. One of the more effective ways to accomplish
an audit of all transactions is to have the transaction details safely copied from the consolidating
closing process to an auditors’ “Sandbox” and use CAAT techniques to verify and sample the
detail transactions. This eliminates the risk of changing transactions in the actual “Production”
database.
As food for thought, consider that in modern high transaction-volume, computer based
accounting systems 100% of the auditable transactions are in the IT system’s databases or files at
one point or another. Given that consideration, Computer Assisted Auditing Techniques that can
apply 100% testing capability for stored transactions makes a lot of sense.
Once a financial reporting process has been described, the reporting and operational risks
analyzed, and the workflow documented, preferably as a flowchart with a supporting
walkthrough narrative, the internal auditor can evaluate the effectiveness of specific control
attributes by sampling transactions that flow through the complete process, or through complete
branches of the reporting process. Generally, the sampling can be done by selecting documents
that evidence whether or not the control activity or attribute is being performed. This may be
Intellectual Property of James J. Finn
Copyright 2009 ©
For discussion and negotiation purposes only -44-
basic, but it is critical if one is basing an audit conclusion related to the process on a sampling
plan. There must be adequate documentation that all steps in the process have been performed.
Generally a valid statistical sampling plan can verify the operational effectiveness of this
complete workflow. If a financial process is organized as a sequence of specific activities
performed in a stable and controllable process, the auditor should be able to sample process flow
documents rather than the individual transaction documents and verify whether or not the entire
process is performing as designed. This could require the process itself to be designed with the
capability to collect the necessary information needed (process flow documents), and to provide
documented evidence that the attributes and process activities were being performed and
recorded in a controlled manner. A possible equivalent of this would be monitoring a
manufacturing process using statistical charts to continuously inform the auditor and the process
manager if the transaction processing steps were functioning in a stable and controlled manner.
This would be a form of statistical process control.
Bibliography
Barbara Apostolou, Sampling: A Guide for Internal Auditors, (Copyright 2004, IIARF)
The Professional Practices Framework, March 2007, (Copyright 2004, The IIA Research
Foundation, IIARF)
Neal B. Hitzig, Statistical Sampling Revisited, CPA Journal, May 2004/Vol. LXXIV No. 5.