Received: 5 January 2018 Revised: 31 March 2018 Accepted: 27 May 2018
DOI: 10.1002/qre.2353
TUTORIAL
Optimized sampling design and rationale for verification
and validation
Abstract
An optimized sampling design that meets customer, design,
or process requirements, while balancing technology limi-
tations, is still a common challenge to engineering commu-
nities. This is especially true in the medical device industry.
Acceptance sampling plans for manufacturing are widely
available, but the appropriate sampling plans for verifica-
tion and validation (V&V) are less well known. This paper
applies established statistical theory to derive sampling
plans appropriate for estimating product reliability during
V&V, where reliability must exceed an established thresh-
old with an appropriate margin of statistical confidence.
The paper provides insight on how to estimate parameters
of interest and interpret acceptance criteria. Operating
characteristic curves are used to examine if a design or pro-
cess is capable of producing future product that meets
design specifications and/or customer requirements in
terms of confidence and reliability. The methodology is
applied to both attribute and variable sampling plans,
including examples showing how to achieve a high proba-
bility of passing the acceptance criteria. Formulas, sample
size tables, and operating characteristic curves are provided
for engineering practitioners to use. The paper aims at pro-
viding a practical quantitative approach and a valid statisti-
cal rationale to assess overall product quality during V&V.
KEYWORDS
confidence, reliability, sampling, validation, verification
1 | INTRODUCTION
Development of an adequate sampling plan with a valid
statistical rationale that balances the customer or design
requirements and technology limitations is a pivotal ele-
ment in verification and validation (V&V), especially for
the medical device industry. Acceptance sampling plans
for manufacturing are widely available, but the
Qual Reliab Engng Int. 2019;35:483–502.
appropriate sampling plans for V&V are less well known.
The development of an adequate sampling plan for V&V
has been challenging and has been frequently cited by
Food and Drug Administration (FDA) warning letters or
Form 483 Observations.
In V&V, a common parameter to estimate is the reli-
ability (R), defined as the proportion of a population that
conforms to the specification, eg, the proportion of
in vitro diagnostic tests that report a valid test result.
Medical device V&V sets the requirement of protecting
the consumer by demonstrating that R ≥ RL at the appro-
priate level of statistical confidence, where RL is the
predetermined minimum acceptable quality level. Verifi-
cation and validation therefore assumes that the require-
ments have not been met unless the testing demonstrates
that they are met. A statistical sample of size N is drawn
from the relevant population of interest; eg, N in vitro
diagnostic test strips are drawn from the product line
and run on a control specimen producing N test results.
R is the proportion of valid test results for the product line
(the population proportion in statistical terminology). RS
is the estimate of this proportion based on the results
observed in the statistical sample. It is important to
understand that R has some true but unknown value in
the population and that RS is an estimate of this parame-
ter. The logic of the sampling plan concerns the accuracy
of the estimate RS based on the size of the statistical sam-
ple. In statistics, the accuracy of an estimate is often
assessed by a confidence interval. The statistical confi-
dence is given by a confidence level defined here as
(1 − β), where typical values of β are 0.025, 0.05, and
0.10, corresponding to 97.5%, 95%, and 90% confidence.
The lower 1‐sided (1 − β) confidence limit of the estimate
of R is determined; the V&V passes its acceptance criteria
when the lower confidence limit of the estimate of R is
equal to or exceeds RL. Another way of stating this
criteria is that the hypothesis R < RL must be rejected
with probability (1 − β), so that we may conclude R ≥ RL
with the stated level of statistical confidence. In this
paper, we speak of 1‐sided lower confidence limits
because the upper confidence limit of the estimate of R
is not relevant.
wileyonlinelibrary.com/journal/qre © 2018 John Wiley & Sons, Ltd. 483
484
Acceptance sampling is often applied to manufactur-
ing, wherein the Operating Characteristic (OC) curve is
used to accept or reject lots with the trade‐off between
Producer's Risk (α) and Consumer's Risk (β). The OC curve
plots the probability of acceptance versus the population
proportion defective, which is 1 − R in our notation. The
probability of acceptance associated with the Producer's
Risk is (1 − α), and the corresponding population propor-
tion defective is the Acceptable Quality Limit (AQL). The
probability of acceptance associated with the Consumer's
Risk is β, and the corresponding population proportion
defective is the Rejectable Quality Level (RQL), also
known as the Lot Tolerance Percent Defective (LTPD).
The OC curve is readily applied to the problem of V&V
by taking the confidence level as (1 − β), the population
proportion defective as (1 − R), and the Rejectable Quality
Level (RQL) as (1 − RL). In the same fashion, we may label
the Acceptable Quality Limit (AQL) as (1 − RA), where RA
is the value of the reliability in the population such that the
probability of acceptance is (1 − α). In practice, α should be
suitably small for the success of the V&V, but the sample
size may be constrained given the practical limitations of
conducting the V&V. Generally speaking, when RA is large
compared with RL, the sample size can be relatively small,
but when RA approaches RL, the sample size must be large.
Minimal sampling plans are often inappropriate because
of the low probability of acceptance when the reliability
in the population is close to the reliability requirement.
Some discussion of notation is unfortunately required.
Here the confidence levels are labeled by the Greek letter
β, because this is the conventional notation for
Consumer's Risk. In conventional statistical notation, con-
fidence levels are labeled by α instead of β, which is just an
exchange of Greek letters. The probability of passing the
acceptance criteria is traditionally labeled as (1 − α) on
the OC curve, where α is the Producer's Risk. In conven-
tional statistical notation, the probability of passing would
typically be called “statistical power” and it would be
labeled as (1 − β), again just an exchange of Greek letters.
A statistician would consider R < RL to be the “null
hypothesis” and would calculate the power of the sample
size under the “alternative hypothesis” that R = RA.
The statistical theories to develop a sampling plan to
estimate reliability with confidence are well established
and have recently been reviewed in the context of V&V
for medical devices.1 However, this reference did not
cover the probability of passing the acceptance criteria
of the V&V protocol. As compared with Ferryanto,1 this
paper covers the probability of acceptance as a function
of the true (but unknown) product reliability and also
includes an improved method for variable sampling as
indicated in Lieberman and Resnikoff2 and Bruhn‐Suhr
and Krumbholz.3
TUTORIAL
Often, the reliability is defined as the proportion of
the population of a continuous variable that must be
between a lower and upper specification to be considered
acceptable. In such cases, the reliability is related to the
mean and the standard deviation of the distribution of
the continuous variable. Typically, when variable sam-
pling plans are considered, the estimate of the reliability
(in the sample) is not explicitly calculated, as it is more
common to report the sample mean and the sample stan-
dard deviation, together with the tolerance interval and
the pass/fail disposition. This is because the statistical
theory of tolerance intervals is well established
(Krishnamoorthy and Mathew,4 Wilks,5,6 ISO 16269‐6,7
Owen,8 and NIST/SEMATECH9); but the statistical the-
ory of estimating the reliability (given the sample mean
and the sample standard deviation) is less familiar to
engineers, although it is on equally strong footing.2,3,10
The technical problem arises when applying the
familiar 2‐sided tolerance interval to a problem with both
a lower and upper specification limit (LSL and USL). If
one requires, for example, as in Ferryanto,1 that both
sides of the tolerance interval must be within the interval
(LSL and USL), then a precise statement regarding the
confidence level can no longer be achieved. In this case,
the V&V may fail against a confidence level that exceeds
the required confidence level by an unspecified amount.
The rigorous method to overcome this deficiency is to
estimate the sample mean and the sample standard devi-
ation and to use these to estimate the reliability in the
sample (RS). The acceptance criterion is then given in
terms of a threshold that RS must meet (or exceed), the
threshold depending on both RL (the predefined quality
level that meets the requirement), and the confidence
level (1 − β). This makes the variable sampling plan sim-
ilar to the attribute sampling plan, in that both plans esti-
mate the same underlying parameter (the reliability). It
also allows the OC curve of each plan to be compared
directly in the same type of plot. In the attribute sampling
plan, RS is calculated by counting the data points that fall
within the interval (LSL and USL), whereas in the vari-
able sampling plan, RS is calculated from the sample
mean and the sample standard deviation. The variable
sampling plan is more efficient than the attribute plan
in terms of the sample size, requiring smaller N to
achieve the same level of confidence.
In addition to using appropriate statistical techniques,
the statistical rationale for the sampling plan requirement
should be based on voice of customer (VOC) and risk
management analyses in accordance with ISO 1497111
together with failure modes and effects analysis (IEC
6081212) and fault tree analysis (IEC 6102513), where the
appropriate confidence and reliability levels are deter-
mined on the basis of the risks. Some discussion of the
TUTORIAL
appropriate levels for confidence and reliability is pro-
vided in Durivage.14
This paper has 5 key elements. The first is to delineate
the statistical principles of estimating reliability with con-
fidence and its application to V&V. The second is to pres-
ent how to design optimized sampling plans using such
statistical techniques. The third is to consider the proba-
bility of passing the acceptance criteria given some
assumption about the true (but unknown) level of R in
the population. The fourth is to estimate the reliability
with confidence for both the attribute and variable sam-
pling plans, in a comparable methodology allowing direct
comparison of OC curves. Finally, examples of typical
applications of the methodology in the medical device
industry are presented in the context of V&V.
2 | CONF IDENCE AND
RELIABILITY METHOD BASED
SAMPLING PLANS
2.1 | Attribute sampling plan
Each measurement in an attribute sampling plan has a
dichotomous result, eg, PASS or FAIL. An example of
the application is for conformance or nonconformance
of product characteristics. For an independent random
sample selected from an infinite population, the sample
size n can be determined from the following equation
based on the binomial distribution, Montgomery15:
C n!
β≥ ∑ ð1−RL Þi RLn−i (1)
i¼0 i! ð n−i Þ!
C n!
Probability of Acceptance ¼ ∑ ð1−RÞi Rn−i (2)
i¼0 i!ðn−iÞ!
where (1 − β) is the required confidence, RL is the speci-
fied or required minimum reliability, R is the true (but
unknown) reliability in the population, C is the number
of allowable failures, and n is the sample size. The case
of sampling from a finite population, as may be appropri-
ate for manufacturing lot release when the lot size is rel-
atively small, is based on the more complicated
hypergeometric distribution, which is not relevant here
because the design or process (capable of making all the
future lots in manufacturing) is the subject of the V&V.
It is worth noting that Equation 1 follows from Equation 2
by setting R = RL and setting the probability of accep-
tance ≤ β. Equation 1 is solved iteratively to find the min-
imum sample size n for the maximum allowable failures,
or the inverse binomial distribution is used for the same
purpose. In this paper, all the equations are programmed
485
and solved with double‐precision floating point using
MATLAB version R2016a, unless otherwise noted.
The sample sizes at designated confidence and reli-
ability are listed in Table 1. For example, the required
minimum sample size is n = 59 when allowing zero fail-
ures to demonstrate 95% reliability (RL = 0.95) with 95%
confidence (β = 0.05).
When C = 0 (zero failures allowed), the above
equation is reduced to
lnβ
n≥ : (3)
lnRL
Equation 3 provides a convenient calculation of the
minimum sample size in circumstances where zero fail-
ures are allowed in the acceptance criteria of the V&V.
For example, to demonstrate 90% reliability with 95%
confidence, the minimum sample size is calculated as
lnð1−0:95Þ= ln0:9≥28:4≈29
ðrounding up to the nearest integerÞ:
2.2 | Variable sampling plan
The variable sampling plan follows the standard statistical
methodology as indicated in Lieberman and Resnikoff,2
Bruhn‐Suhr and Krumbholz,3 and ANSI/ASQ Z1.9‐2003.10
The technique is often perceived as complex because of
the special mathematical functions used in the calculation.
Nonetheless, it is relatively simple to explain. The problem
statement relies on 2 additional input parameters, the LSL
and the USL. Performance is considered acceptable when
the continuous variable of interest has a high probability
of falling within the interval (LSL and USL). The problem
statement can be adapted to the case where either the LSL
goes to negative infinity or the USL goes to positive infinity,
thereby turning the 2‐sided problem statement into a 1‐
sided problem statement. The goal is to estimate the propor-
tion of the population outside of the specification limits, ie,
the proportion that does not conform to the specification.
The maximum allowable estimate of the proportion defec-
tive p* is used in the acceptance criteria shown below:
pL þ pU ≤p* (4)
where pL and pU are the estimated probabilities of noncon-
formity below the LSL or above the USL, respectively. The
probabilities are calculated as follows:
pL ¼ ∅ð−Z L Þ; pU ¼ ∅ð−Z U Þ; and p* ¼ ∅ð−k 1 Þ
where ϕ is the cumulative density function of the standard
normal distribution.
486 TUTORIAL

TABLE 1 Attribute sampling plans for required minimum reliability RL with confidence (1 − β)

Confidence Level 97.5% Confidence Level 95% Confidence Level 90%

Reliability RL Reliability RL Reliability RL
C 0.99 0.95 0.9 0.85 0.8 0.99 0.95 0.9 0.85 0.8 0.99 0.95 0.9 0.85 0.8

0 368 72 36 23 17 299 59 29 19 14 230 45 22 15 11

1 555 110 54 35 26 473 93 46 30 22 388 77 38 25 18
2 720 142 70 46 34 628 124 61 40 30 531 105 52 34 25
3 874 173 85 56 41 773 153 76 50 37 667 132 65 43 32
4 1022 202 100 66 48 913 181 89 59 44 798 158 78 52 38
5 1164 230 114 75 55 1049 208 103 68 50 926 184 91 60 45
6 1303 258 127 84 62 1182 234 116 76 57 1051 209 104 68 51
7 1439 285 141 93 69 1312 260 129 85 63 1175 234 116 77 57
8 1573 312 154 102 75 1441 286 142 93 69 1297 258 128 85 63
9 1705 338 167 110 82 1568 311 154 102 76 1418 282 140 93 69
10 1835 364 180 119 88 1693 336 167 110 82 1538 306 152 100 75
11 1964 390 193 127 94 1818 361 179 118 88 1658 330 164 108 81
12 2092 415 206 136 101 1941 386 191 126 94 1776 353 175 116 86
13 2219 440 218 144 107 2064 410 203 134 100 1893 377 187 124 92
14 2345 465 231 152 113 2185 434 215 142 106 2010 400 199 132 98
15 2470 490 243 160 119 2306 458 227 150 112 2127 423 210 139 104

The quantities ZL and ZU are determined by the sam-
ple mean X and the sample standard deviation (S) of the
continuous variable of interest, as shown in the formulas
below. (The “sample mean” and the “sample standard
deviation” refer to estimates of the mean and standard
deviation calculated from the data collected in a statistical
sample selected from the population of interest.)
statistical confidence level (1 − β), and the sample size
(n) are included in the expression for k1.
To complete the variable sampling plan, Bruhn‐Suhr
and Krumbholz3 provide a formula for the probability of
acceptance as shown below.
 pffiffiffi 
Probability of Acceptance ¼ 1− F k1 n; n−1; δ2 (6)

Here F is the noncentral t cumulative distribution


X−LSL
ZL ¼ ; with n − 1 degrees of freedom and noncentrality param-
S pffiffiffi
eter δ2 ¼ zp2 n, where zp2 is the critical value of the nor-
mal distribution with probability p2 = R (the population

USL−X true but unknown reliability) and where k1 is given by
ZU ¼ :
S Equation 5. The factor k1 is equal to the exact 1‐sided sta-
tistical tolerance limit factor k1.9
The quantity k1 is computed using the inverse cumu- Tables 2–4 list commonly used k1 in medical device
lative distribution function for the noncentral t distribu- V&V. Simplifying the problem statement by setting the
tion, as shown below. USL to infinity allows one to see the exact connection
to the 1‐sided tolerance interval. In this case, pU is zero
t 1−β ðn−1; δ1 Þ and Equation 4 can be solved explicitly, as follows:
k1 ¼ pffiffiffi (5)
n

X−LSL
≥ k1 (7)
pffiffiffi S
Here δ1 ¼ zp1 n is the noncentrality parameter,
which depends on zp1 , the critical value of the normal dis- However, the 2‐sided problem statement (where both
tribution with probability p1 = RL, and the sample size n. the LSL and the USL are finite) is not exactly the same as
Therefore, all the parameters determining the sampling a tolerance interval and Equation 4 must be solved in units
plan, the minimum acceptable reliability (RL), the of probability. Solving the problem in terms of
TUTORIAL 487

TABLE 2 Exact 1‐sided statistical tolerance limit factor and corresponding minimum acceptable estimate of the reliability (1 − p*) for
variable sampling plan at (1 − β) confidence level of 97.5%a

Confidence Level 97.5%

Reliability RL=0.99 Reliability RL=0.95 Reliability RL=0.9 Reliability RL=0.85 Reliability RL=0.8
N k1 1 − p* k1 1 − p* k1 1 − p* k1 1 − p* k1 1 − p*

20 3.5291 0.9998 2.5760 0.9951 2.0792 0.9812 1.7507 0.9600 1.4948 0.9326
30 3.2331 0.9994 2.3508 0.9907 1.8894 0.9706 1.5835 0.9434 1.3445 0.9106
40 3.0780 0.9990 2.2320 0.9872 1.7886 0.9632 1.4941 0.9325 1.2635 0.8968
50 2.9798 0.9986 2.1564 0.9845 1.7243 0.9577 1.4368 0.9247 1.2114 0.8872
60 2.9109 0.9982 2.1032 0.9823 1.6788 0.9535 1.3962 0.9187 1.1744 0.8799
70 2.8593 0.9979 2.0633 0.9805 1.6446 0.9500 1.3656 0.9140 1.1464 0.8742
75 2.8379 0.9978 2.0467 0.9797 1.6304 0.9485 1.3529 0.9120 1.1348 0.8718
80 2.8189 0.9976 2.0319 0.9790 1.6177 0.9472 1.3415 0.9102 1.1243 0.8696
90 2.7861 0.9974 2.0065 0.9776 1.5959 0.9448 1.3219 0.9069 1.1064 0.8658
100 2.7590 0.9972 1.9854 0.9765 1.5777 0.9427 1.3055 0.9042 1.0913 0.8625
125 2.7072 0.9967 1.9450 0.9742 1.5430 0.9386 1.2742 0.8987 1.0626 0.8561
150 2.6701 0.9963 1.9160 0.9724 1.5179 0.9355 1.2516 0.8947 1.0418 0.8513
175 2.6418 0.9959 1.8939 0.9709 1.4988 0.9331 1.2344 0.8915 1.0258 0.8476
200 2.6194 0.9956 1.8763 0.9697 1.4836 0.9311 1.2206 0.8889 1.0131 0.8445
250 2.5857 0.9952 1.8499 0.9679 1.4607 0.9280 1.1999 0.8849 0.9939 0.8399
300 2.5613 0.9948 1.8308 0.9665 1.4440 0.9257 1.1848 0.8820 0.9800 0.8365
350 2.5426 0.9945 1.8161 0.9654 1.4313 0.9239 1.1732 0.8797 0.9692 0.8338
400 2.5277 0.9943 1.8043 0.9645 1.4211 0.9224 1.1639 0.8778 0.9606 0.8317
450 2.5154 0.9941 1.7947 0.9637 1.4127 0.9212 1.1563 0.8763 0.9535 0.8299
500 2.5051 0.9939 1.7866 0.9630 1.4056 0.9201 1.1498 0.8749 0.9475 0.8284
600 2.4887 0.9936 1.7736 0.9620 1.3943 0.9184 1.1396 0.8728 0.9380 0.8259
700 2.4761 0.9934 1.7636 0.9611 1.3856 0.9171 1.1316 0.8711 0.9306 0.8240
800 2.4660 0.9932 1.7556 0.9605 1.3786 0.9160 1.1253 0.8698 0.9247 0.8225
a
All the data in Tables 2–4 are calculated using 16 digits of precision. The k1 values shown in the table are rounded up at the fourth decimal place to align with
ISO 16269‐67. The 1 − p* values shown in the table are also rounded up at the fourth decimal place to be conservative.

probabilities makes the connection between the variable
sampling plan and the attribute sampling plan. The left
side of Equation 4 is the estimate of the proportion defec-
tive (1 − RS) based on the sample mean and the sample
standard deviation. The value of p* on the right side of
Equation 4 is always less than (1 − RL) to allow for the
required level of confidence given a finite sample size.
Tables 2–4 list 1 − p* as a function of the sample size for
combinations of commonly used confidence levels
(1 − β) and minimum required reliability (RL). It is impor-
tant to note that p* plays the same role in the variable sam-
pling plan as C plays in the attribute sampling plan; ie, the
maximum allowable estimate of the proportion defective
in the attribute plan is simply C/N. (For a specific example
comparing the attribute sampling plan and the variable
sampling plan, see the description for Figure 4.)
It is necessary to add some statistical footnotes to the
above methodology for the variable sampling plan. First,
the original methodology2 that is used in ANSI/ASQ
Z1.9‐2003 (R2013)10 is not an exact method, as discussed
in Bruhn‐Suhr and Krumbholz,3 which compares the
original method and the exact method. Second, we have
modified the original methodology following Bruhn‐Suhr
and Krumbholz3 to simplify the interpretation of the
equations. The original methodology uses a minimum
variance unbiased (MVU) estimate of the 3 probabilities
pL, pU, and p*. The modified methodology used in this
paper uses the maximum likelihood (ML) estimate of
the same 3 probabilities. This replacement of the MVU
estimate by the ML estimate is profound in terms of sim-
plifying the equations, because the probabilities pL and
pU are independent of sample size and because the ML
488 TUTORIAL

TABLE 3 Exact 1‐sided statistical tolerance limit factor and corresponding minimum acceptable estimate of the reliability (1 − p*) for
variable sampling plan at (1 − β) confidence level of 95%

Confidence Level 95%

Reliability RL=0.99 Reliability RL=0.95 Reliability RL=0.9 Reliability RL=0.85 Reliability RL=0.8
N k1 1 − p* k1 1 − p* k1 1 − p* k1 1 − p* k1 1 − p*

20 3.2952 0.9996 2.3961 0.9918 1.9260 0.9730 1.6146 0.9468 1.3714 0.9149
30 3.0640 0.9990 2.2199 0.9868 1.7774 0.9623 1.4833 0.9311 1.2531 0.8950
40 2.9410 0.9984 2.1255 0.9833 1.6972 0.9552 1.4121 0.9211 1.1885 0.8827
50 2.8625 0.9979 2.0650 0.9806 1.6456 0.9501 1.3661 0.9141 1.1465 0.8742
60 2.8071 0.9976 2.0222 0.9785 1.6090 0.9462 1.3333 0.9088 1.1165 0.8679
70 2.7654 0.9972 1.9899 0.9767 1.5813 0.9431 1.3084 0.9047 1.0938 0.8630
75 2.7482 0.9971 1.9765 0.9760 1.5697 0.9418 1.2981 0.9029 1.0843 0.8609
80 2.7327 0.9969 1.9645 0.9753 1.5594 0.9406 1.2888 0.9013 1.0758 0.8590
90 2.7061 0.9966 1.9438 0.9741 1.5416 0.9385 1.2728 0.8985 1.0611 0.8557
100 2.6840 0.9964 1.9266 0.9730 1.5268 0.9366 1.2595 0.8961 1.0488 0.8529
125 2.6418 0.9959 1.8937 0.9709 1.4984 0.9330 1.2338 0.8914 1.0252 0.8474
150 2.6114 0.9955 1.8699 0.9693 1.4778 0.9303 1.2153 0.8879 1.0081 0.8433
175 2.5882 0.9952 1.8517 0.9680 1.4621 0.9282 1.2011 0.8852 0.9950 0.8402
200 2.5698 0.9950 1.8373 0.9670 1.4496 0.9265 1.1897 0.8830 0.9845 0.8376
250 2.5421 0.9945 1.8155 0.9653 1.4307 0.9238 1.1726 0.8796 0.9686 0.8337
300 2.5219 0.9942 1.7997 0.9641 1.4170 0.9218 1.1601 0.8770 0.9570 0.8308
350 2.5065 0.9940 1.7875 0.9631 1.4064 0.9202 1.1505 0.8751 0.9481 0.8285
400 2.4941 0.9937 1.7778 0.9623 1.3979 0.9190 1.1428 0.8735 0.9410 0.8267
450 2.4840 0.9936 1.7698 0.9617 1.3910 0.9179 1.1365 0.8722 0.9351 0.8252
500 2.4755 0.9934 1.7631 0.9611 1.3851 0.9170 1.1311 0.8710 0.9301 0.8239
600 2.4619 0.9931 1.7523 0.9602 1.3757 0.9156 1.1226 0.8692 0.9222 0.8218
700 2.4514 0.9929 1.7441 0.9595 1.3685 0.9145 1.1160 0.8678 0.9160 0.8202
800 2.4430 0.9928 1.7374 0.9589 1.3627 0.9136 1.1107 0.8667 0.9111 0.8189

estimate depends on the normal distribution, which is
more familiar than the beta distribution. Monte Carlo
simulations conducted as background material for this
manuscript (unpublished) were not able to distinguish
substantially between the probabilities computed via sim-
ulation of the exact plan and the probabilities computed
via simulations of the approximate plan, either in its orig-
inal form (using the MVU estimates) or in the modified
form, as presented here using the ML estimates of the
probabilities. For example, the exact plan for RL = 0.94,
β = 0.10, N = 34 has p* = 0.02262, as given by example
1 of Bruhn‐Suhr and Krumbholz3 using the MVU esti-
mate, whereas the approximate plan (also using the
MVU estimate) has p* = 0.02337. The probability of
acceptance calculated by Monte Carlo simulation of the
approximate plan is 0.106 for R = RL, as compared with
the probability of acceptance of 0.100 for the exact plan.
Therefore, the approximate plan using the MVU estimate
slightly overestimates the probability of acceptance,
which is why p* is adjusted downward in the exact plan
to achieve the required level of confidence. This can also
be seen by calculating the probability of acceptance for
R = RA = 0.99. The exact plan has a probability of accep-
tance of 0.900 (corresponding to α = 0.10) whereas the
approximate plan has a probability of acceptance of
0.908. Again, the approximate plan using the MVU esti-
mate slightly overestimates the probability of acceptance.
The same calculations via Monte Carlo simulation were
also made for the approximate plan using the ML esti-
mate following the formulas presented in this paper. In
this case, for R = RL, the approximate plan has a probabil-
ity of acceptance of 0.091, as compared with the probabil-
ity of acceptance of 0.100 for the exact plan, whereas for
R = RA, the approximate plan has a probability of
TUTORIAL 489

TABLE 4 Exact 1‐sided statistical tolerance limit factor and corresponding minimum acceptable estimate of the reliability (1 − p*) for
variable sampling plan at (1 − β) confidence level of 90%

Confidence Level 90%

Reliability RL=0.99 Reliability RL=0.95 Reliability RL=0.9 Reliability RL=0.85 Reliability RL=0.8
N k1 1 − p* k1 1 − p* k1 1 − p* k1 1 − p* k1 1 − p*

20 3.0516 0.9989 2.2078 0.9864 1.7653 0.9613 1.4711 0.9294 1.2407 0.8927
30 2.8838 0.9981 2.0799 0.9813 1.6571 0.9513 1.3754 0.9155 1.1542 0.8758
40 2.7932 0.9974 2.0103 0.9778 1.5979 0.9450 1.3228 0.9071 1.1063 0.8658
50 2.7349 0.9969 1.9653 0.9754 1.5595 0.9406 1.2884 0.9012 1.0750 0.8588
60 2.6936 0.9965 1.9333 0.9734 1.5321 0.9373 1.2638 0.8969 1.0525 0.8537
70 2.6623 0.9962 1.9091 0.9719 1.5113 0.9347 1.2451 0.8935 1.0353 0.8498
75 2.6493 0.9960 1.8990 0.9713 1.5026 0.9336 1.2373 0.8921 1.0282 0.8481
80 2.6377 0.9959 1.8899 0.9707 1.4948 0.9326 1.2303 0.8907 1.0217 0.8466
90 2.6177 0.9956 1.8743 0.9696 1.4813 0.9308 1.2182 0.8885 1.0106 0.8439
100 2.6010 0.9954 1.8613 0.9687 1.4701 0.9293 1.2081 0.8865 1.0012 0.8417
125 2.5690 0.9949 1.8363 0.9669 1.4485 0.9263 1.1886 0.8827 0.9833 0.8373
150 2.5459 0.9946 1.8182 0.9655 1.4329 0.9241 1.1744 0.8799 0.9702 0.8341
175 2.5282 0.9943 1.8044 0.9645 1.4209 0.9224 1.1636 0.8777 0.9601 0.8315
200 2.5141 0.9941 1.7934 0.9636 1.4113 0.9210 1.1549 0.8760 0.9521 0.8295
250 2.4930 0.9937 1.7767 0.9622 1.3969 0.9188 1.1418 0.8733 0.9399 0.8264
300 2.4775 0.9934 1.7646 0.9612 1.3863 0.9172 1.1322 0.8713 0.9310 0.8241
350 2.4657 0.9932 1.7553 0.9604 1.3782 0.9160 1.1248 0.8697 0.9241 0.8223
400 2.4562 0.9930 1.7478 0.9598 1.3717 0.9150 1.1189 0.8684 0.9186 0.8209
450 2.4484 0.9929 1.7416 0.9593 1.3663 0.9141 1.1140 0.8674 0.9141 0.8197
500 2.4418 0.9927 1.7365 0.9588 1.3618 0.9134 1.1099 0.8665 0.9103 0.8187
600 2.4314 0.9925 1.7282 0.9581 1.3546 0.9123 1.1033 0.8651 0.9041 0.8171
700 2.4233 0.9924 1.7218 0.9575 1.3490 0.9114 1.0982 0.8640 0.8994 0.8158
800 2.4168 0.9922 1.7167 0.9570 1.3445 0.9107 1.0941 0.8631 0.8956 0.8148

acceptance of 0.887, as compared with the exact plan that
has a probability of acceptance of 0.900. Therefore, the
approximate plan using the ML estimate in this paper
slightly underestimates the probability of acceptance
and, therefore, is conservative.
3 | OPTIMIZED S AMPLI N G DE SI GN
BALANCING CONSUMER
PROTECTION AN D P R OD U CE R 'S
T E C H N O L O G Y LI M I TA T I O N S
An optimized sampling plan captures both the VOCs and
technology limitations commensurate of risks. Therefore,
an optimized sampling plan needs to be (a) adequate and
(b) practical. In the attribute and variable sampling plans
shown in Tables 1–4, each sampling plan is specified by a
confidence level (1 − β), a minimum acceptable reliability
(RL), and a sample size (N). However, the selection of an
appropriate sample size must consider the true (but
unknown) product reliability.
The probability of acceptance is given by Equation 2 for
the attribute sampling plan, or by Equation 6, for the var-
iable sampling plan. The OC curve can be plotted from
Equation 2 or 6. The OC curve is a plot of the probability
of acceptance on the y‐axis versus the population propor-
tion defective (1 − R) on the x‐axis for a given sample size
N at the selected levels for the minimum acceptable reli-
ability RL and statistical confidence (1 − β). The value of
the reliability in the population is a parameter that is con-
sidered unknown during the V&V; hence, it is sometimes
called the “true but unknown” parameter, which we have
denoted as R. Given data from previous studies (design cal-
culation, dry runs, or historical data), it should be possible
to make an initial estimate for the assumed value of this
parameter when choosing the sample size.
490 TUTORIAL

The OC curves for the attribute sampling plan of 95%

reliability and 90% confidence requirement are shown in
Figure 1. Each of the OC curves is labeled by the sample
size, N, and the corresponding maximum allowable fail-
ures, C.
In Figure 1, the OC curves are shown for 95% reliabil-
ity (RL = 0.95) and 90% confidence (β = 0.10) for sample
sizes N = 45, 77, 105, 158, 209, 306, 400, 492, and 807 with
corresponding allowable failures C = 0, 1, 2, 4, 6, 10, 14,
18, and 32. For the attribute plan, all of the OC curves
have a probability of acceptance that is less than or equal
to β when R = RL. In the case of Figure 1, when the prod-
uct true but unknown reliability is equal to the required
reliability of 95%, or the proportion defective is 5%, the
probability of acceptance is 10% for all the OC curves cor-
responding to 95% reliability at 90% confidence.
In the case of Figure 1, the minimum sample size N FIGURE 2 Operating characteristic (OC) curves for attribute
was selected for each level of C, following the convention sampling for 95% reliability (RL) at 90% confidence (1 − β) when
sample sizes are not selected from Table 2 [Colour figure can be
in Table 1. In this case, because the OC curves are labeled
viewed at wileyonlinelibrary.com]
by increasing levels of C, the probability of acceptance
increases with the increase of sample size N. As a more
general example, consider the OC curves for the sample (release 15, 2017, NCSS, LLC). Figure 3 shows that when
size selections shown in Figure 2, where different values the sample size is increased from 76 to 77, the probability
of N may share the same value of C. This is the case for of acceptance jumps from 68.3% to 94.3%. The power
the first 2 curves in Figure 2, where the probability of decreases as N increases (above the minimum required
acceptance decreases when going from N = 50 to N) until C is allowed to increase by one, thereby making
N = 75, because both curves allow the same maximum a saw tooth–shaped power curve. To summarize, Figure 1
number of failures (C = 0). shows the minimum sample size for each level of C (the
This behavior of the power versus sample size curve maximum allowable number of failures), where the prob-
for estimating a binomial proportion is well known16 ability of acceptance is at a local maximum of the saw
and is demonstrated in Figure 3, assuming a true (popu- tooth power curve. For this reason, it is conventional to
lation) reliability of 99.5%, as calculated in PASS software select the minimum sample size for each level of C. (Note,
however, that the number of units selected for evaluation
in the V&V protocol typically exceeds the minimum
required sample size to allow for the possibility that some

FIGURE 1 Operating characteristic (OC) curves for attribute

sampling for 95% reliability (RL) at 90% confidence (1 − β) when FIGURE 3 Probability of acceptance for 95% reliability at 90%
sample sizes are selected from Table 1 [Colour figure can be viewed confidence when the true but unknown (population) reliability is
at wileyonlinelibrary.com] 99.5% [Colour figure can be viewed at wileyonlinelibrary.com]
TUTORIAL
units may give an indeterminate or unevaluable response.
For example, to get a minimum of N = 45 evaluable
in vitro diagnostic test results, the number of human
specimens selected for testing may exceed N = 45 to allow
that some specimens may repeatedly give an error code or
no reportable result.)
Additional OC curves for the attribute sampling plans
corresponding to the reliability and confidence combina-
tions listed in Table 1 are provided in Appendix A for
V&V practitioners to use.
In Figure 4, the OC curves for the variable sampling
plan are shown for 95% reliability (RL = 0.95) and 90% con-
fidence (β = 0.10) for sample sizes N = 50, 75, 100, 150,
200, 300, 400, 500, and 800. For the variable sampling plan,
all of the OC curves have a probability of acceptance of β
when R = RL. In the case of Figure 4, when the product
true but unknown reliability is equal to the required reli-
ability of 95%, or the proportion defective is 5%, the prob-
ability of acceptance is 10% for all the OC curves. The
OC curves in Figure 4 are labeled by the same sample sizes
as the OC curves in Figure 2 to allow a direct comparison
FIGURE 4 Operating characteristic (OC) curves for variable
sampling for 95% reliability (RL) at 90% confidence (1 − β)
[Colour figure can be viewed at wileyonlinelibrary.com]
TABLE 5 Comparisons of sample sizes in variable vs attribute sampling plans for 95% reliability (RL = 0.95) and 90% confidence (β = 0.10)
Minimum Probability Product True
of Acceptance, % Reliability, %
90 99
90 97.7
90 96.8
80 98.1
80 97.8
80 97.1
491
between the variable sampling plan and the attribute sam-
pling plan. Each of the OC curves of the variable sampling
plan corresponds to a unique value of p*, as provided in
Table 4. As previously noted, the value of p* plays the same
role as C/N in the attribute plan. For example, the OC
curve in Figure 2 for the attribute sampling plan of
(N = 100, C = 1) corresponds to a minimum acceptable
reliability estimate of (1 − C/N) = 0.990, as compared with
(1 − p*) = 0.9687 for the N = 100 variable sampling plan to
achieve the same required level of reliability (RL = 0.95) at
the same level of confidence (β = 0.10). It is seen that,
when the sample size is the same, the population reliabil-
ity needed from the variable sampling plan is much lower
than that from the attribute sampling plan to pass at the
same confidence (1 − β) and required reliability (RL). Note
that while the attribute sampling plan for C = 0 imposes a
minimum sample size, for example, N ≥ 45 for 95%
reliability at 90% confidence, there is no minimum
sample size for the variable sampling plan, because the
standard deviation of the variable of interest can be
arbitrarily small.
A variable sampling plan requires a smaller sample
size than an attribute one to demonstrate the fulfillment
of the same confidence and reliability requirement under
V&V. Table 5 compares some of the sampling plan OC
curves based on Figures 1 and 4. It is shown that an attri-
bute sampling plan needs more than double the sample
size of a variable sampling plan to achieve the same prob-
ability of acceptance.
To summarize, when developing an optimized sam-
pling plan for V&V, one must balance consumer protec-
tion and producer's technology limitations. On the basis
of design calculations or historical data, or other initial
engineering estimates, some level of product reliability
must be assumed to represent the population. Then, the
V&V requirement is analyzed to determine if an attribute
or variable sampling plan is appropriate.
An attribute sampling plan has its advantage when
the V&V characteristics are evaluated from binary gauges
because an exact measurement is excessively time‐
Sample Size Additional Samples
Needed for Attribute
Attribute Variable Sampling
N = 105, C = 2 N = 50 55
N = 306, C = 10 N = 150 156
N = 807, C = 32 N = 400 407
N = 158, C = 4 N = 75 83
N = 209, C = 6 N = 100 109
N = 400, C = 14 N = 200 200
492
consuming or expensive or when multiple binary gauges
can be combined into a single binary outcome at the unit
level. To select the appropriate sample size for an attri-
bute sampling plan, the OC curves shown in Figure 1 or
Appendix A are used. For these attribute plans, the accep-
tance criterion is based on comparing the number of fail-
ures observed with the number of failures allowed.
Variable sampling plans are applicable to a single
continuous characteristic and use the sample mean and
sample standard deviation to calculate the reliability in
the sample. When applicable, the variable sampling plans
are encouraged because a much smaller sample size is
required, as compared with an attribute sampling plan,
as shown in Table 5. Operating characteristic curves for
variable sampling, as shown in Figure 4 or Appendix B,
are used to select the appropriate sample size. Equation 4
is used to calculate the reliability from the data
collected during the V&V. The calculated reliability is
compared with the allowable minimum reliability listed
in Tables 2–4 to evaluate the acceptance criteria.
Furthermore, the attribute sampling plan and the var-
iable sampling plan are on equal footing in that both
plans estimate the reliability in a statistical sample
(drawn from a population) and compare this to a mini-
mum acceptable estimate. In the attribute sampling plan,
the estimate of the reliability in the statistical sample is
the proportion (number of passing results)/(total number
of results), which is compared with the minimum accept-
able estimate (1 − C)/N. In the variable sampling plan,
the estimate of the reliability in the statistical sample is
the probability (1 − pL − pU), which is compared with
the minimum acceptable estimate (1 − p*).
4 | I M P L E M E N T A T I O N O F TH E
OPTIMIZED S AMPLI N G DE S I G N
Development of an adequate sampling plan with a valid
statistical rationale for V&V has been challenging for
industry although the statistical theories behind are
widely available. Some examples are provided below to
show how to design an optimized attribute or variable
sampling plan in the context of V&V.
Example 1: Design verification for a medical device is
conducted to demonstrate that a characteristic meets the
specification of 95% reliability at confidence level of 90%.
Four different failure modes are expected from failure
modes and effects analysis, which can be combined into
a single binary attribute. The target probability of accep-
tance is 80%. The initial estimate of the device reliability
calculated from design parameters based on a pessimis-
tic scenario is 98%.
TUTORIAL
Solution: Select Figure 1 that shows OC curves for
95% reliability at confidence level of 90% requirement.
Assume that the true but unknown reliability is 98%. It
follows from Figure 1 that the minimum sample size is
falling between the OC curve of (N = 158, C = 4) with
the probability of acceptance of 78.9% and the OC curve
of (N = 209, C = 6) with the probability of acceptance of
87.1% (rounded down). From Table 1, the sampling plan
(N = 184, C = 5) is likely optimal. From Equation 2, the
probability of acceptance calculated for the (N = 184,
C = 5) plan is 83.4% at the assumed true but unknown
reliability of 98%. Provided that 83.4% probability of
acceptance is a reasonable target (as may be the case
when the assumed true but unknown reliability is based
on a pessimistic scenario), the minimum sample size is
N = 184 with the maximum allowable failures of C = 5
by referring to Table 1.
The exact probability of acceptance (Equation 2) can
be then calculated using Excel using the function
BINOM.DIST (c, n, 1‐true reliability, TRUE). In this case,
BINOM.DIST (c, n, 1‐true reliability, TRUE) = BINOM.
DIST (5, 184, 0.02, TRUE). The V&V is passed when less
than or equal to 5 failures are seen.
Example 2: Engineering dry runs will be performed
prior to a V&V to evaluate if the assumed reliability of
a binary attribute is correct. The quality requirement is
to demonstrate the design meet specification of 95% reli-
ability at confidence level of 90%. The target probability
of acceptance is 90%. The assumed device reliability is
99.4%.
Solution: From Figure 1, it is shown that the mini-
mum sample size is between N = 45 (C = 0) and
N = 77 (C = 1) based on the assumed device reliability
of 99.4%. The larger sample size of N = 77 is selected
for the dry run. After the execution of the dry run, the
result was 2 out of 77 failures. Therefore, the device reli-
ability was estimated to be 75/77 or 97.4% in the dry run.
On the basis of Figure 1 or Table 1, a sampling plan of
(105, 2) was chosen to redo the dry run and the second
dry run passed with 2 out of 105 failures. The device reli-
ability was reestimated to be 178/182 (97.8%) by pooling
the 2 dry runs. On the basis of the best estimate avail-
able, the true but unknown reliability was assumed to
be 97.8% and the sampling plan (N = 282, C = 9) was
selected from Table 1 for the V&V to target a high prob-
ability of acceptance of approximately 90%.
Example 3: Medical device design verification is con-
ducted to determine if a quantitative in vitro diagnostic
result meets the requirement of 95% reliability with
95% confidence for a 1‐sided specification of the USL of
2.2. The target probability of acceptance is 80%. The
TUTORIAL
initial estimate of the device reliability is 98.1%, and the
data are assumed to be normally distributed.
Solution: From Figure B7 in Appendix B, assuming a
true but unknown reliability of 98.1% and targeting the
probability of acceptance of 80%, the sample size chosen
is 100. After the data are collected, the mean value (X ) is
−0.02654, and the standard deviation (S) is 1.059. From
Table 3, the 1‐sided tolerance limit k factor for a sample
of 100 at 95% confidence level and 95% reliability is
k1 = 1.9266, and the minimum acceptable estimate of the
reliability is 0.9730 (=1 − p*) or 97.3%. ZU = (2.2 + 0.02654)/
1.059 = 2.1025, so pU = Φ(−2.1025) = 0.0178. There is no
LSL (so pL = 0), and therefore, the estimated reliability is
1 − pU = 0.9822 (or 98.2%), which is greater than the min-
imum acceptable estimate of 97.3%. Therefore, the design
verification passes.
Example 4: Medical device design verification is con-
ducted to determine if a quantitative in vitro diagnostic
meets the requirement of 95% reliability with 95% confi-
dence for a 2‐sided specification of the USL of 2.5 and
the LSL of −2.5. The target probability of acceptance is
80%. Following an engineering study, the true product
reliability is assumed to be 98.1% and the data are
assumed to be normally distributed.
Solution: From Appendix B, Figure B7, with true reli-
ability of 98.1% (=1 − x‐axis) and the probability of
acceptance of 80% (=y‐axis), the sample size chosen is
100. After test, it is calculated from the data collected
that the mean value (X Þ is −0.02654, and the standard
deviation S is 1.059.
From Table 3, k1 = 1.9266 for sample size n = 100 at
95% reliability with 95% confidence, ZL = (−0.02654 + 2.5)/
1.059 = 2.3357, ZU = (2.5 + 0.02654)/1.059 = 2.3858,
pL = Φ(−2.3357) = 0.00975, pU = Φ(−2.3858) = 0.00852.
pL + pU = 0.0183, which is less than p* = 0.0270
(=1 − 0.9730). Therefore, the verification is passed with
the confirmation that the estimated reliability of 98.3%
(=1 − 0.0183) is greater than the minimum acceptable
estimate of 97.3%.
5 | SUMMARY AND CONCLUSIONS
In a V&V of safety‐critical products, it is necessary to pro-
tect the consumer by demonstrating that the reliability
meets a predetermined minimum quality level at the
appropriate level of statistical confidence. Attribute sam-
pling plans are popular, but minimal plans are often
inappropriate and the engineers responsible for V&V pro-
tocols need to understand the relationship between the
probability of passing the acceptance criteria and the
sample size. When the product true reliability is close to
493
the predefined specification, the required sample size gets
large. Often the reliability is defined as the proportion of
the population of a continuous variable that must be
between the LSL and the USL to be considered accept-
able. In such cases, the reliability is related to the mean
and the standard deviation of the distribution of the con-
tinuous variable and a variable sampling plan is more
efficient than an attribute sampling plan.
This paper aims at providing optimized sampling
design with a valid statistical rationale for V&V, espe-
cially for the medical device industry, to demonstrate
future product quality or processes ability in addition to
comply with regulations. To start, one needs to determine
the appropriate levels of statistical confidence and mini-
mum acceptable reliability that the V&V is going to dem-
onstrate. This determination is based on the VOC and
risk analysis. The manufacturing and technology limita-
tions are then considered by plotting the probability of
acceptance versus the population reliability (OC curve)
and selecting the smallest sample size N that gives a prac-
tical probability of acceptance. The attribute sampling
plan estimates the reliability (a proportion) by counting
the number of instances of a binary variable, whereas the
variable sampling plan estimates the same proportion as
an integral over a probability density that depends on the
mean and standard deviation of a continuous variable.
The variable plan requires that the distribution of the
continuous variable be normal or approximately normal.
When the continuous variable of interest is not normally
distributed, a transformation may be used to make the
transformed variable approximately normal. For example,
the signal in an immunoassay is often log normally distrib-
uted because of the multiplicative effect of the various
components of the system noise. Finally, if the variable is
not normally distributed and there is no suitable continu-
ous transformation, then an attribute sampling plan may
be used by transforming the continuous variable into a
binary variable. In this case, the attribute plan is a non-
parametric statistical technique, free of assumptions about
the distribution of the underlying continuous variable. If
the nonparametric method is used, then it is encouraged
to report descriptive statistics related to the variable data
(eg, mean, standard deviation, median, and interquartile
range) so that these values are readily available in the ver-
ification/validation report. This will help understand the
process, uncover trends, predict future performance, and
support continuous improvement.
In this paper, the statistical theories for estimating reli-
ability by sampling a population are presented. Equations
and tables of commonly used attribute sampling plans
based on the binominal distribution are provided; equa-
tions and tables for the variable sampling plan based on
noncentral t distribution are provided; the probabilities
494
of acceptance associated with the sampling plans are pre-
sented and OC curves are calculated and provided. In the
statistical theory, one must be careful to distinguish
between the reliability of the population (R) sometimes
called the “true but unknown” parameter, the minimum
acceptable reliability based on the product requirements
(RL), and the estimate of the reliability that is calculated
from the statistical sample of the population (RS). The sam-
pling plans guarantee that a prespecified level of statistical
confidence (1 − β) can be achieved to limit the Consumer's
Risk and assure product quality, as appropriate during the
V&V activities of product development or process valida-
tion. This is equivalent to saying that the V&V passes its
acceptance criteria when the lower 1‐sided (1 − β) confi-
dence limit of the estimate of R is equal to or exceeds RL.
In this paper, the method to develop appropriate attri-
bute and variable sampling plans to meet the reliability
and confidence requirement for V&V is demonstrated,
including examples showing how to achieve a high proba-
bility of passing the acceptance criteria. Formulas,
sample size tables, and OC curves for V&V as well as step‐
by‐step examples are provided for engineering practitioners
to use. This paper also provides a statistical rationale for
selecting a sampling plan appropriate for conducting a
V&V. The V&V sampling method presented in this paper
captures both the VOCs and technology limitations. This
is significant because the appropriate sampling plans for
V&V are less well known than for manufacturing, as
evidenced by the fact that many companies have been cited
by Food and Drug Administration warning letters or
Form 483 Observations for lacking a valid statistical
rationale when choosing a sampling plan for V&V.
S. Cheng
K. Kupfer
M. Dixon
S. Shammas
Alere San Diego, Inc, San Diego, CA, USA
Correspondence
S. Cheng, Alere San Diego, Inc, San Diego, CA, USA.
Email: shumin.cheng@quidel.com
R EF E RE N C E S
1. Ferryanto L. Statistical sampling plan for design verification and
validation of medical devices. http://www.ivtnetwork.com/
print/article/statistical‐sampling‐plan‐design‐verification‐and‐
validation‐medical‐devices [6 May 2015]
2. Lieberman GJ, Resnikoff GJ. Sampling plans for inspection by
variables. J Am Stat Assoc. 1955;50:457‐516.
3. Bruhn‐Suhr M, Krumbholz W. Exact two‐sided Lieberman‐
Resnikoff sampling plans. Stat. Pap. 1991;32(1):233‐241.
TUTORIAL
4. Krishnamoorthy K, Mathew T. Statistical Tolerance Regions:
Theory, Applications, and Computation. Hoboken, New Jersey:
John Wiley & Sons, Inc.; 2009. ISBN:978‐0‐470‐38026‐0.
5. Wilks SS. Determination of sample sizes for setting tolerance
limits. Ann. Math. Stat. 1941;12(1):91‐96.
6. Wilks SS. Statistical prediction with special reference to the
problem of tolerance limits. Ann. Math. Stat. 1942;13(4):400‐409.
7. ISO 16269‐6:2014. Statistical interpretation of data—part 6:
determination of statistical tolerance intervals. 2014.
8. Owen DB. Factors for one‐sided tolerance limits and for variables
sampling plans, Monograph No. SCR‐607. Sandia Corp., 1963.
9. NIST/SEMATECH. e‐Handbook of statistical methods. http://
www.itl.nist.gov/div898/handbook/ [30 Oct. 2013]
10. ANSI/ASQ Z1.9‐2003 (R2013): Sampling procedures and tables
for inspection variables for percent nonconforming. 2013.
11. BS EN ISO 14971:2012. Medical devices. Application of risk
management to medical devices. 2012.
12. IEC 60812:2006. Analysis techniques for system reliability—pro-
cedure for failure mode and effects analysis (FMEA). 2006.
13. IEC 61025:2006. Fault tree analysis (FTA). 2006.
14. Durivage M. Risk‐based approaches to establishing sample sizes for
process validation. http://www.asqrd.org/risk‐based‐approaches‐to‐
establishing‐sample‐sizes‐for‐process‐validation/ [16 June 2016]
15. Montgomery DC. Introduction to Statistical Quality Control. Sixth
ed. Hoboken, New Jersey: John Wiley & Sons, Inc; 2009:638.
ISBN:978–0–470‐16992‐6.
16. Chernick MR, Liu CY. The saw‐toothed behavior of power ver-
sus sample size and software solutions: single binomial
proportion using exact methods. Am. Stat. 2002;56(2):149‐155.
Dr S. Cheng is a Certified Manager of Quality/Orga-
nizational Excellence and a Certified Reliability Engi-
neer with ASQ and also a Certified Six Sigma Black
Belt with both Motorola and ASQ. She has practiced
quality, Six Sigma, reliability, and structural analysis
using statistical methods at many major companies
in the medical device industry including Alere, Baxter
Healthcare, Life Technologies, St. Jude Medical, and
Johnson and Johnson. She has also worked in the
automotive, telecommunications, and aerospace
industries. She holds a PhD degree in Engineering
and Applied Science from the Memorial University
of Newfoundland in Canada.
Dr K. Kupfer is a Fellow Biostatistician at Alere, Inc,
and holds a PhD degree in Applied Plasma Physics
and Nuclear Engineering from the Massachusetts
Institute of Technology. He was previously an Assis-
tant Professor of Molecular Biology and Oncology at
Southwestern Medical Center in Dallas, and he has
worked in the pharmaceutical and medical device
industry for approximately 20 years.
TUTORIAL 495

Mr M. Dixon is the Director of Quality Engineering

at Alere, San Diego, and holds a BS degree in Chemi-
cal Engineering from the University of California,
Santa Barbara. He is a certified Six Sigma Black Belt
and has worked in the medical device industry for
approximately 15 years with an emphasis on continu-
ous improvement and new product development in
the rapid diagnostics space.

Mr S. Shammas is a Quality Engineer at Alere San

Diego. He has practiced quality and reliability analysis
at major companies in medical device and diagnostics
industry including Alere, Beckman, and KONE. He
holds a bachelor's degree in Chemistry from the Uni-
versity of Massachusetts.
FIGURE A2 Operating characteristic (OC) curves for attribute
sampling for 95% reliability at 97.5% confidence [Colour figure can
A P P EN D I X A: A TT RI B U T E SA M P LI N G O C be viewed at wileyonlinelibrary.com]
CU RVES WH E N S A M PLIN G SI ZES A RE
S E L E C T E D F R O M T A B L E 2 ( E F F E C TI V E
SAMPLING PLAN)

FIGURE A3 Operating characteristic (OC) curves for attribute

sampling for 90% reliability at 97.5% confidence [Colour figure can
be viewed at wileyonlinelibrary.com]

FIGURE A1 Operating characteristic (OC) curves for attribute

sampling for 99% reliability at 97.5% confidence [Colour figure can
be viewed at wileyonlinelibrary.com]
496 TUTORIAL

FIGURE A4 Operating characteristic (OC) curves for attribute FIGURE A6 Operating characteristic (OC) curves for attribute
sampling for 85% reliability at 97.5% confidence [Colour figure can sampling for 99% reliability at 95% confidence [Colour figure can
be viewed at wileyonlinelibrary.com] be viewed at wileyonlinelibrary.com]

FIGURE A5 Operating characteristic (OC) curves for attribute FIGURE A7 Operating characteristic (OC) curves for attribute
sampling for 80% reliability at 97.5% confidence [Colour figure can sampling for 95% reliability at 95% confidence [Colour figure can
be viewed at wileyonlinelibrary.com] be viewed at wileyonlinelibrary.com]
TUTORIAL 497

FIGURE A8 Operating characteristic (OC) curves for attribute FIGURE A10 Operating characteristic (OC) curves for attribute
sampling for 90% reliability at 95% confidence [Colour figure can sampling for 80% reliability at 95% confidence [Colour figure can
be viewed at wileyonlinelibrary.com] be viewed at wileyonlinelibrary.com]

FIGURE A9 Operating characteristic (OC) curves for attribute FIGURE A11 Operating characteristic (OC) curves for attribute
sampling for 85% reliability at 95% confidence [Colour figure can sampling for 99% reliability at 90% confidence [Colour figure can
be viewed at wileyonlinelibrary.com] be viewed at wileyonlinelibrary.com]
498 TUTORIAL

FIGURE A12 Operating characteristic (OC) curves for attribute FIGURE A14 Operating characteristic (OC) curves for attribute
sampling for 95% reliability at 90% confidence [Colour figure can sampling for 85% reliability at 90% confidence [Colour figure can
be viewed at wileyonlinelibrary.com] be viewed at wileyonlinelibrary.com]

FIGURE A13 Operating characteristic (OC) curves for attribute FIGURE A15 Operating characteristic (OC) curves for attribute
sampling for 90% reliability at 90% confidence [Colour figure can sampling for 80% reliability at 90% confidence [Colour figure can
be viewed at wileyonlinelibrary.com] be viewed at wileyonlinelibrary.com]
TUTORIAL 499

A P P EN D I X B: VA R I A B L E SA M PL I N G O C
CURVES

FIGURE B3 Operating characteristic (OC) curves for variable

sampling for 90% reliability at 97.5% confidence [Colour figure can
be viewed at wileyonlinelibrary.com]

FIGURE B1 Operating characteristic (OC) curves for variable

sampling for 99% reliability at 97.5% confidence [Colour figure can
be viewed at wileyonlinelibrary.com]

FIGURE B4 Operating characteristic (OC) curves for variable

sampling for 85% reliability at 97.5% confidence [Colour figure can
be viewed at wileyonlinelibrary.com]

FIGURE B2 Operating characteristic (OC) curves for variable

sampling for 95% reliability at 97.5% confidence [Colour figure can
be viewed at wileyonlinelibrary.com]
500 TUTORIAL

FIGURE B5 Operating characteristic (OC) curves for variable FIGURE B7 Operating characteristic (OC) curves for variable
sampling for 80% reliability at 97.5% confidence [Colour figure can sampling for 95% reliability at 95% confidence [Colour figure can
be viewed at wileyonlinelibrary.com] be viewed at wileyonlinelibrary.com]

FIGURE B6 Operating characteristic (OC) curves for variable FIGURE B8 Operating characteristic (OC) curves for variable
sampling for 99% reliability at 95% confidence [Colour figure can sampling for 90% reliability at 95% confidence [Colour figure can
be viewed at wileyonlinelibrary.com] be viewed at wileyonlinelibrary.com]
TUTORIAL 501

FIGURE B9 Operating characteristic (OC) curves for variable FIGURE B11 Operating characteristic (OC) curves for variable
sampling for 85% reliability at 95% confidence [Colour figure can sampling for 99% reliability at 90% confidence [Colour figure can
be viewed at wileyonlinelibrary.com] be viewed at wileyonlinelibrary.com]

FIGURE B10 Operating characteristic (OC) curves for variable FIGURE B12 Operating characteristic (OC) curves for variable
sampling for 80% reliability at 95% confidence [Colour figure can sampling for 95% reliability at 90% confidence [Colour figure can
be viewed at wileyonlinelibrary.com] be viewed at wileyonlinelibrary.com]
502 TUTORIAL

FIGURE B13 Operating characteristic (OC) curves for variable FIGURE B15 Operating characteristic (OC) curves for variable
sampling for 90% reliability at 90% confidence [Colour figure can sampling for 80% reliability at 90% confidence [Colour figure can
be viewed at wileyonlinelibrary.com] be viewed at wileyonlinelibrary.com]

FIGURE B14 Operating characteristic (OC) curves for variable

sampling for 85% reliability at 90% confidence [Colour figure can
be viewed at wileyonlinelibrary.com]