' " """"''"
Statistical on Statistical Graphics
Concepts in Metrology
With a Postscript
Harry H. Ku
Statistical Engineering Division Center for Computing and Applied Mathematics National Engineering Laboratory National Bureau of Standards Gaithersburg, MD 20899
"'~' OF :l
eo...u Of ~
S. Department of Commerce C. William Verity, Secretary
National Bureau of Standards Ernest Ambler Director
Library of Congress Catalog Card Number: 88-600569 National Bureau of Standards Special Publication 747 Nati. Bur. Stand. (U.
Spec. Pubi. 747,
u.S. Government Printing Office
For sale by the Superintendent of Documents
u.S. Government Printing Office
Washington, DC .20402
48 pages (Aug. 1988)
COD EN: XNBSAV
.... ..............................: ..... ........ ......... ..... ............................................ ..................................... . . . . . . . . . . . . '. . . . . . . . . . . ...........................................
Statistical Concepts of a Measurement Process Arithmetic Numbers and Measurement Numbers. . . . . Computation and Reporting of Results. . . .
Properties of Measurement Numbers. . . . .
The Limiting Mean Range, Variance, and Standard Deviation. . . . .
Population and the Frequency Curve. . . .
The Normal Distribution Estimates of Population Characteristics. . . . . Interpretation and Computation of Confidence Interval and Limits. . .
Precision and Accuracy "
Index of Precision Interpretation of Precision. . . . Accuracy.
Statistical Analysis of Measurement Data Algebra for the Manipulation of Limiting Means and Variances. . . . .
Propagation of Error Formulas. ... .. . .. . . .. Pooling Estimates of Variances. . . . Component of Variance Between Groups.....................
Comparison of Means and Variances...
Comparison of a Mean with a Standard Value. Comparison Among Two or More Means. . Comparison of Variances or Ranges. . . . . Control Charts Technique for Maintaining Stability and Precision. . . Control Chart for Averages. .. . . . Control Chart for Standard Deviations. . . . Linear Relationship and Fitting of Constants by Least Squares. . . . .
Postscript on Statistical Graphics Plots for Summary and Display of Data. . .
Stem and Leaf............ .............
Plots for Checking on Models and Assumptions. . . .
Adequacy of Model. Testing of Underlying Assumptions. . . . . Stability of a Measurement Sequence. . . . .
References... ........ ........,... ...
...................................................... . ... .". ...........,.............. . ...............................'....,..... .. .... .. ........................... .. .........,....... . ...... .. .
.... . ..
LIst of Figures
1 A sYIllDletrical distribution " (A) The uniform. distribution. (B) The log-normal distribution. . . . . Uniform and normal distribution of individual measurements having the same mean and standard deviation, and the corresponding distribution(s) of arithmetic m.eans of four independent
= 10, 0' = 1 '"
2-4 Computed 90% confidence intervals for 100 samples of size 4
drawn at random from a normal population with
Control chart on.f for NB' 1O gram. . Control chart on 8 for the calibration of standard cells. . . . . Stem and leaf plot. 48 values of isotopic ratios, bromine (79/81). Box plot of isotopic ratio, bromine (79/91) Magnesium content of specimens taken. . ...
Plot ofresiduals after linear fit. .. . .
Plot ofresiduals after quadratic fit. . . .
Normal probability plot of residuals after quadratic fit Djfferences of linewidth measurements from NBS values. Measurements on day 5 inconsistent with others~ Lab A . . .. . . Trend with increasing linewidths - Lab B . .
Significant isolated outliers- Lab
Plot of residuals after linear fit. Measured depth of weld defects
Measurements (% reg) on the power standard at i-year and 3.;monthintervals
LIst of Tables
under normal curve between
m - kO'
2 A brief table of values oft ................................... Propagation of error formulas for some simple functions. . . . . 2-4 Estimate of 0' from the range. . .. . .... . . . .
Computation of confidence limits for observed corrections, NB'
Calibration data for six standard cells. . .
1 Y ~
for reference sample.. . '"
approximation may have to be used. or 7J: are all exact
In metrological work . 1416.
STATISTICAL CONCEPTS OF
A MEASUREMENT PROCESS
numbers are used for different purposes and consequently these numbers have different interpretations. plots. digital
r. National Bureau of Standards.umbers by definition . and Analytical Chemistry.With a
Postscript on Statistical
Harry H. statistics. Statistical Concepts and Procedures. metrology. Since then this chapter has been used as basic text in statistics in Bureau-sponsored courses and seminars. measurement.
Key words: graphics. 3 J2. Arithmetic numbers are exact numbers. For this reason additional sections on statisti-
cal graphics are added as a postscript. i.Statistical Concepts in
Metrology. Thus 7J: may be written as 3. although in expressing some of these numbers in digital form . 1969. depending on our judgment of which is the proper one to
use from the combined point of view of accuracy and convenience. While concepts and techniques introduced in the original chapter remain valid and appropriate. It is therefore important to differentiate the two types of numbers which will be encountered. some additions on recent development of graphical methods for the treatment . including those for Electricity. MD 20899
Statistical Concepts in Metrology " was originally written as Chapter 2 for the Handbook of Industrial Metrology published by the American Society
of Tool and Manufacturing Engineers. VolUlUe I. Precision Measurement and Calibration. 14 or 3. of data would be useful. uncertainty. Ku
Statistical Engineering Division. Gaithersburg. Electronics. By the
. Graphical methods can be used effectively to "explore" information in data sets prior to the application
of classical statistical procedures. 1967. It was reprinted as one of 40
papers in NBS Special Publication 300.
. The last digit in the measured value depends on the instrument used and our ability to read it. One suggestion . and then to round off the final reported to an appropriate number of digits. and the reported value itself should be stated
but numbers obtained by operation under approximately
three measurements on the diameter of
the same conditions. For example .. in the process of computation . is to treat all measurement numbers as
instrument is well-chosen . Reflection on the simple example above will help to convince one of this fact. Computation and Reporting of Results. significant figures " has little meaning in the manipulation of numbers resulting from measurements. The term depends on the .
number of digits to be retained in the final result uncertainties " attached to this reported value. Unfortunately. In fact . 5 units of the last recorded digit. our only concern here is the number of digits in the expression for uncertainty.
The . however . Thus we see that measurement numbers differ by their very nature from arithmetic numbers. uncertainty " will he treated later under . The accuracy of
the result can always be extended if necessary. If the severe rounding would result in loss of information. Precision . 5 unit of the last digit.
396 392 401
Sum 1. the approximations do not differ from the exact values by more than *0.4 .
are not approximations to
exact numbers . we might have obtained 0. appropriate "
A recommended rule is that the uncertainty should be stated to no more
than two significant figures.3963
No general rule can be given for all types of arithmetic operations. the metrologist can usually select an instrument to give him results adequate for his needs as illustrated in the example above. on the other hand .189
Average 0. . we might have been able to record to the fifth digit after the decimal point. therefore .4 0. if a finer instrument .4. By experience . Another recommended procedure is to carry two or three extra figures
throughout the computation ... the last digit given certainly does
by less than not imply that the measured value differs from the diameter ::1:::0.
Measurement numbers . the phrase . If we had used a coarser instrument .
exact numbers in the operations and to round off the final result only.
usual rules of rounding. 009
There is no rounding off here.. both arithmetic numbers and measurement numbers are present and frequently confusion reigns over the number of digits to be kept in
successive arithmetic operations.
a steel shaft with a micrometer may yield the following results:
No.and Accuracy . and 0. In all cases .
Ernest Dorsey (reference 2):
The mean ora family of measurements-of a number of measure-
ments for a given quantity carried out by the same apparatus. By convention . the arithmetic mean by x. should always
be avoided.7 .
or A =
is usually a difference between
as the bias or systematic error of the
measurements. and attained its full development in the hands of Laplace and Gauss. procedure . Otherwise. though different . and to spell out its
components. in symbols
and 7 . In fact .
Properties of Mecsurement Numbers
The study of the properties of measurement numbers . this limiting mean is frequently called the ' true ' value although it bears no necessary relation to the true quaesitum. there exists a limiting mean
*References ' are listed at the end of this chapter. we must require these numbers . and observer-approaches a definite value as the number of
measurements is indefinitely increased.
If the results of measurements are to make any sense for the purpose at hand . An example is:
The apparent mass correction for the nominal 109 weight is
+0. Furthermore . and has equal claims
the same conditions
on its value. or the Theory of Errors . thus providing a basis for the statistical treatment ' and analysis of these numbers in the following major
is the reported
a measure of uncertainty in some vague sense . let us adopt as the postulate of measurement a statement due to N. according to this
postulate. Abbreviated forms such as value and 1: h. 00.
The sentence form is preferred since then the burden is on the reporter to specify exactly the meaning of the term uncertainty. 0420 mg with an overall uncertainty of ::1:0. to behave .' Xn. there as . formally began with Thomas Simpson more than two hundred years ago . if the true value is 7 . Each member of this set is an estimate of the quantity being measured .
This has often confused the unwary. In the theory of errors . As shown in the micrometer example above.
. the numerical values of these
are denoted by Xh X2'.
and the range by
. the difference between the
obtained in the
largest value and the smallest
repeated measurements of a single physical quantity under essentially
yield a set of
measurement numbers. the magnitude of systematic errors from known sources
The Limiting Mean. to the
actual value of the quantity that the observer desires to measure.to the last place affected by the qualification given by the uncertainty statement. Let us call it the limiting mean. where A is defined
approaches as the number of measurements increases indefinitely. In the next subsections some of the important properties of measurement
numbers will be discussed and summarized . as a group in a certain predictable manner. they could not properly be called measurements of a given quantity. Experience has shown that this indeed the case under the conditions stated in italics above. 0087 mg using three
standard deviations as a limit to the
effect of random errors of measurement .
the most frequently used are the variance and the standard deviation. as
dation to build on toward a mathematical model . The probability that a particular
measurement yields a value of which is less than or equal to
proportion of the population that is less than or equal to
proportion of population less than or equal to
*Convention is followed in using the capital
to represent the value that might be
produced by employing the measurement process to obtain a measurement (i.(T
variance. the conditions will not remain constant . and the lower case to represent a particular value of observed.00
The positive square root of the variance . The value of is unknown and usually unknowable . the range may increase but cannot decrease. The result of a single measurement X* can take randomly any of the values belonging to this population.
Population and the frequency Curve. from which estimates can be made and inference drawn . . or
In practice . hence also the bias. and from day to day. we will run into difficulties.
Nevertheless . a random variable). Our model of a measurement process consists then of a defined population of measurement numbers with a
and (T describe
and a standard
deviation (T.~ (Xi m)2
-----. obviously the additional information provided by the measurements in between is lost. and Standard Deviation. and specify it completely in one important special case. The range of
seemingly simple postulate does provide a sound foun-
calculation . With one more measurement . Even for a large number of measurements . or scattering) of our measurements which
on the other hand . such as average deviation and probable error. The value of cannot be obtained since one cannot make an infinite number of measurements. By population is meant the conceptually infinite number of
measurements that can be
generated. We shall call the limiting mean
the location parameter and the standard deviation (T the scale parameter
the population of measurement numbers generated by a particular measurement process. A number of such measures can be constructed. since changes occur from hour to hour .
. the standard deviation is of the same dimensionality as the limiting mean. does not enjoy this desirable property of the arithmetic mean. Variance. Since only the largest and the smallest numbers enter into its
utilize each measurement made with equal weight .
There are other measures of dispersion . and which will approach a definite number as the number of measurements is indefinitely increased. The choice of the variance
as the measure of
and maneuverability Variance is defined
dispersion is based upon its mathematical convenience as the value approached by the
of measurements is indefinitely
average of the sum of squares of the deviations of individual
from the limiting mean as the number
increased . The two numbers
this population of measurements to a large extent . (T is called the standard deviation (of a single measurement). however . as will be seen later on. The relationships between these measures and the standard
deviation can be found in reference I. It will be desirable ' to look for another measure
of the dispersion (spread .
the shaded area to the left of
other distributions are useful in describing . for example . Any curve C' enclosing an area of unity with the abscissa defines the distribution of a particular population. 2-2. Note that the shape of C is not determined by and (J' alone.1.
. curve C in Fig. 2-2A and 2-28. 2. the uniform distribution and the log-normal distribution are given in Figs.
Ag. a measurement process that yields numbers on a continuous scale
the distribution of values of
for the population can be represented by
a smooth curve.
Ag. (A) The uniform distribution (B) The log-normal distribution. The area between C and the abscissa bounded by any two values (xI and is the proportion of the population that takes values between the two values . C is called a frequency curve. A synunetrical distribution.certain populations. Two examples . or the probability that will assume values between
and x2. 2.Similar statements can be made for the probability that
than or equal to or for between and
will be greater
as follows: PfX
For .1. For example. the probability that
can be represented by
the total area between the frequency curve and the abscissa being one by definition.
thirds of the area lies within one 3 percent beyond 2cr of more than 95 percent within and less than 0.
2. Mood .
For . The results spread roughly symmetrically about a central value. or distributed in
accordance with the normal law of error. The area under the curve is one .
The study of a particular theoretical
tation of a
frequency curve of this type leads to the normal curve (Gauss error curve. one version of which and mean If a population has a finite variance is the following independent then the distribution of the sample mean (of
by A. if the measurements follow a normal distribution we can say a great deal about the measurement process. 99.1 which is symmetrical and
bunched together about m.data generated by a measurement process
the following properties are usually observed:
I.can be represented .
Area under normal curve between
Iowa normal distribution approximately.
A measurement process having these two properties would generate a frequencycurve similar to that shown in Fig. (2-0). M. Consequently. If cr
is used as unit on the abscissa .
The normal distribution has been studied intensively during the past century.00
1. The normal curve . Thus about two. Small deviations from this central value are more frequently found
than large deviations. In particular . 2. The question remains: How do we know that this is so from the limited number of repeated measurements on hand? The answer is that we don t! However . McGraw~
*From Chapter 7 Introduction to the Theory of Statistics. to assume that the measurement process generates numbers that fol2.
Some of the important features of the normal curve are:
1. and act as if this were so to rely on the so-called Central Limit Theorem . Hill Book Company.
50. then the area under the curve
between constant multiples of
can be computed from
values of the normal distribution.1.The Normal Distribution.
99. t:xactly by the mathematical
where is the ordinate and the abscissa and
71828 is the base of
Percent area under
curve (approx. From Eq.
Table 2. areas under the curve
for some useful intervals between
are given in and kcr kcr cr of Table 2. in most instances the metrologist
may be willing
1. 1950. as required.'6
. it is evident that the frequency
determined by the two parameters and
curve is completely
95. Measurements having such a normal
frequency curve are said to be normally
distributed. New York.1. It is symmetrical about
These curves are indistinguishable to this scale.
Ag. fact . that the distribution of
sample means of independent measurements will be
normal with the specified limiting meaIl' . . the normal approxi~
mation to the distribution of becomes very good even for
as small as
3 or 4.
according to the theorem. is reduced This last statement is true in general for all by a factor of l/-v'ii! measurement processes in which the measurements are " independent"
It is therefore not a consequence of the Central Limit and for all n. however .measurements) approaches the normal distribution with variance
/n and mean
as the sample size
increases. the sample mean
interest which . Figure
6 -4 -
4. every measurement process must by definition have a finite is the quantity of mean and variance. Third. we may say that 11 normally distributed measurements are independent if these measurements are not correlated or associated in any
A formal definition of the concept of " independence " is out of the scope here. the measure of dispersion . 2. First . the standard deviation of the sample mean .8
1. i. The peaked curve is actually two curves representing the distribution of arithmetic means of four independent measurements from the respective distributions. Theorem. Uniform and normal distribution of individual measurements having the same mean and standard deviation.and
approximately standard deviation
(I/-v'ii for large n.3 shows the uniform and normal distribution having the
same mean and standard deviation. and the corresponding distribution(s) of arithmetic means of four independent measurements. "
and powerful theorem is indeed tailored for measurement processes.6
1. will be approximately
normally distributed for large sample sizes. for a measurement process with a frequency curve that is sym~
metrical about the mean .3. The theorem guarantees. and with small deviations from the mean as compared to the magnitude of the quantity measured. Second .
i.5 and 2-
are shown in
. The standard deviation of the
is sometimes referred to as the standard
measurements (Jim.=1 (Xi xy
~ Ir and
In addition . Thus . Measurements are correlated through time of the day. or has been considered to be of no appreciable effect on the results. Measurements are correlated through rejection of valid data . mean
that the limiting
In the above section it is shown and the variance (1" 2 completely specify a measure-
ment process that follows the normal distribution.~ ~
way. error of the mean . 2.=1
1. a sequence of measurements showing a trend or pattern are not independent measurements. In practice and (1" are not known and cannot be computed from a finite number of measurements. then the result will not be a random sample.
3. I . temperature . The measurements are randomly drawn from this population . is perhaps still the most effective way of detecting trends or correlation. 4. or seasons. The measurement process is established and under control . where
. when the rejection is based on the size of the' number in relation to others of the group. Measurements are correlated through a factor that has not been considered . meaning that the limiting mean and the standard deviation do possess definite values which will not . change over a reasonable period of time. This leads to the use of the sample mean as an estimate of the limiting mean and the square of the computed standard deviation of
the sample . but the constant may overcorrect or undercorrect for
particular samples. the limiting mean of is equal to and of to (1" . and is estimated by
We note that the making of to drawing a sample of size
independent measurements is equivalent
at random from the population of measure-
that the values are of equal weights . 2.
as an estimate of the variance. Two concepts are of importance here: I.
= computed standard deviation
Examples of numerical calculations of
Estimates of Population Characteristics. Suppose out of
which is far apart from the other two is rejected.
and there is no prejudice in the
three measurements the one
selection. . between days weeks . There are many ways by which dependence or correlation creeps into a set of measurement data. For a random sample we can say that is an unbiased estimate of is an unbiased estimate of (1" . e. A standard correction constant has been used for a factor . The traditional way of plotting the data in the sequence they are taken or in some rational grouping. several of the common causes are the following: 1.
From tabulated values of the standardized normal distribution it is known that 95 percent of values will be hounded between . will fluctuate from set to set . . will be centered . we can compute and arrange in a tabular form as follows:
Sample mean Sample standard deviation
In the array of no two will be likely to have exactly the same value... The frequency curve of will be centered about the limiting mean and will have the scale factor In other words aim./fl:)
in the long
The pr:obability that the confidence interval will cover the limiting mean 95 in this case. From the Central Limit Theorem it can be deduced that the will be approximately normally distributed with standard deviation aj. let us
form a quantity similar to
x-m t=I . Nevertheless .. and the quantity
has the properties of a single observation from the " standardized" normal
distribution which has a mean of zero and a standard deviation of one.96 and + 1. The
It is to be borne in mind that
values of the end points of a confidence interval are called confidence limits. The interval
for confidence interval
will be correct 95 percent of the time
L96(alm) to I. and the interval calculated for a particular Xj mayor may not cover
I n the above discussion we have selected
metrical about x. Hence the statement
1. is called the confidence level or confidence coefficient.96 ~
a two-sided interval sym-
intervals the confidence coefficient is usually
denoted by I a.
Interpretation and Computation of Confidence Interval and Limits
run.96. where al2 is the percent of the area under the frequency curve of that is cut off from each tail./fl:
measurements each ../fl:. about
or its equivalent
1... In most cases (J is not known and an estimate of is computed from the same set of measurements we use to calculate x.1.96
is called a
Notice that very little can be said about
with two measurements.x
and our statement will be correct 100
000 816 765
6.725 1. The values of
with the same probability
are listed in Table 2A brief table of values of
case of known
The value of
00 is 1.t
a) percent of the time in the long depends on the degrees of freedom and the probability level. the interval predicted to contain due to both the smaller value of t and the divisor
narrows down steadily. 706(sl"Jn)
t(sl .753 1. Pearson
Cambridge .812 1..In)
under the pen name of " Student " hence the distribution of
Student's distribution.182 776
2.943 895 1. This is indeed the case.447
604 032 707 3. the same as for the
for n larger than 2 .and if we know the distribution of ment as before. S. Hartley.
The distribution of
was obtained mathematically by William S.697
3. Hence the degrees
of freedom here are n ~
Having the table for the distribution of
as before .499
691 687 683
086 042 000 960
750 660 576
Biometrika Tables for Statisticians Vol. In the expression for both
is called the
and s fluctuate from
set to set of
. edited by E. the nth value of Xi is completely determined by and the other (n l)x
values.. we get for a confidence level of 0.n. Intuitively we will expect the value of
larger than that of z for a statement
2.m-c:. The value of
and using the same reasoning
To find a value for
associated with the
we need to know the " degrees of freedom
computed standard deviation s. I . The University Press .132 015
and H. we can make the statement that
"Jn -c:. . 706 303
63. 706(sl. Since is calculated from the same n numbers and has a fixed value. O. In fact the distribution of
we could make the same type of state~ is known for the case of normally
distributed measurements. the following lower and upper confidence limits:
1958. From the table .
the difference between two means . Tables of sample sizes
tions are given in reference 4. one may get an is reasonable and intuitive feeling whether the number of measurements
sufficient for the purpose on hand. but both of these intervals failed to include the line for the mean. expect 95 percent of such intervals to include
Fig. and if we compute a we make a long series of sets of from each set by the prescribed method .4
shows the 90 percent confidence intervals
= 4 from a normal population with from 100 samples of = I. and No. In samples No. 3. is smaller than u2.
Figure 2.4. Since
is a measure of the spread of the frequency
curve about the limiting mean
may be defined as an index of precision. we would confidence interval for
fidence interval computed as a result of or fail to include m. e. From the widths of computed confidence intervals . Three interesting features are to be noted: actually I. The closeness of the mid. .
Thus a measurement process with a standard deviation U.It is probably worthwhile to emphasize again that each particular conmeasurements will either include The probability statement refers to the fact that measurements . For detecting a specified magnitude of interest . 90) computed . Computed 90% confidence intervals for 100 samples of size 4 drawn at random from a normal population with = 10. (J' = 1. even for small
confidence intervals will cover the limiting mean with the specified probability. The number of intervals that include
= 0. or using :. 2. The surprising variation of the sizes of these intervals. is said to be
more precise than aI1other with a standard deviation U2 if U. the four values must have been very close together .
required for certain prescribed condi-
Precision and Accuracy
Index of preeision. and
turns out to be 90
the expected number. . It is true that . by trial and error if
2.points of these intervals to the line for the mean does not seem to be related to the spread. yet the limits may be so far apart as to be of no practical significance. the approximate number of measurements required can be
and solving for
solved by equating the half-width of the confidence interval to this difference using when known . is really a measure of imprecision since the imprecision is (In fact
directly proportional to
Obviously an estimate of the standard deviation of the measurement process based only
on a small number of measurements cannot be considered as convincing
evidence. or in symbols
If (1" is not known .61T .
Thus for a required degree of precision . By the use of the control chart
method for standard deviation
and by the calibration of one s own standard at frequent intervals. n1.. This is accomplished by making more measurements. the laboratory reporting the value must be prepared to give evidence of the precision claimed. the laboratory may eventually claim that the
standard deviation is in fact known and the measurement process is stable
with readily available evidence to support these claims. no amount of detail would cover all the contingencies that may arise. then
could be used as an
estimate of the index of precision. 1961. whenever the terms standard
deviation about the limiting mean ((1"). EI77.
Since a measurement process
numbers as the results of repeated measurements of a single physical quantity under essentially the same conditions . In metrology or calibration work.
but (1"1 2(1"2' Then
for a derived process to have
(1"2. modifiers of the words precision
are recommended by ASTM* to qualify in an unambiguous
Use of the Terms Precision and Accuracy as Applied to the Measurement of a Property of a Material. the precision of the reported value is
an integral part of the result. may vary considerably from sample to sample in the case of a small number of measure-
constant multiples of degrees pf freedom
ments as was shown in Fig. The value of
InterprefClfion of Precision. Many operators measuring the same quantity over a period of time with a number of instruments will yield a precision index a.
and the best estimate we
have of (1" is a
measurements . In fact . the method and procedure in obtaining these numbers must be specified in detail. However.
the respective estimates sand slm may be substituted. 2. ~nd II is proportional to the squares of their respective standard deviations (variances). where the lengths computed from the
of the intervals are
Suppose (1". as subsequently described . Hence . or cover all the factors
that may affect the results of measurement." ASTM Designation . The number
must be considered along with s in indicating how reliable an estimate s is of (1".--
Consider the means of sets of
independent measurements as a new
derived measurement process. It is therefore possible to derive from a less precise measurement process a new process which has a standard deviation equal to that of a more precise process. or standard error of the mean
are used .
needed for measurement processes I. and in practice it is measured by (1" . by taking into consideration the above reservation. The standard deviation of the new process is aim. precision is the main criterion by which
the quality of the work is judged. we need
or we need to use the average of four measurements as a single measurement. In what follows . Thus a single operator in a single day with a single instrument may generate a process with a precisi~)n index measured by (1".
. Logically (1" ' must be larger than
usually considerably larger. n12. the number
and n2. Consequently.
day. However an index of precision based on the former is generally not a justifiable indicator of the quality of the reported value. is a constant . the single-operator
precision is probably appropriate.
In comparing the results of two laboratories .. It is recommended that an index.
J n the last section the basic concepts of a measurement process were
given in an expository manner. the single-operator precision is obviously an inadequate measure to use .j/i
can be considered
as a yardstick with which we can gage the difference between two results
obtained as measurements of the same physical quantity. say.
By making a measurement we obtain a number intended to express quantitatively a measure of " the property of a thing. necessary to the statistical
analysis to be presented in this section . The same publication warns against the use of
if the interpretation of these
the terms " repeatability " and " reproducibility "
terms is not clear from the context. Bias . are " single-operator-machine single-operator. usually in the form of a multiple of the standard deviation (or
estimated standard deviation). are summarized and reviewed below.laboratory. " Measurement numbers differ from ordinary arithmetic numbers .instruments. and if the two results differ by an amount considered to be large as measured by the standard errors . Examples
multi. Hence the selection of an index of precision depends strongly on the
purposes for which the results are to be used or might be used. These concepts. the standard deviation.
The standard deviation () or the standard error
()/. and the other an index of precision . and does not behave in the same
many instances. one the credible bounds for bias . and their meanings are ambiguous unless the components are spelled out in detail. of accuracy be expressed as a pair of numbers . overall bias is of necessity reported in terms of estimated bounds that reasonably include the combined effect of all the elemental biases. the possible sources of biases are known but their magnitudes and directions are not known. and the usual " significant figure treatment is not appropriate.
The term " accuracy " usually denotes in some sense the close-
ness of the measured values to the true value . we may conclude
that the evidence is predominantly against the two results being truly equal. Repeated measurement of a single physical
. taking into consideration
both precision and bias.is meant. these should be reported and discussed in sufficient detail to enable others to use their own judgment on the matter. three separate days. " etc. defined as the difference between the limiting mean and the true value . . If our interest is to compare the results of one operator against another .day. The terms " uncertainty " and " limits of error are sometimes used to express the sum of these two components . Since there are no accepted ways to estimate bounds for elemental
way as the index of precision . on the
Accuracy.. since the precision of each
laboratory must include factors such as multi~'operator. It is common experience that three measurements made within the hour are closer together than three measurements made on . For a thorough discussion realistic evaluation of precision see Section 4 of reference 2. or to combine "them . In
biases . The .
2. .. Since and (Y are usually not known .'. Too wide an interval may merely indicate that more measurements need to be made for the objective desired. averages of independent measurement numbers tend to be normally distributed with the limiting mean and the standard deviation (Y/. and ..
For many measurement processes encountered in metrology. A sequence of measurements showing a trend or pattern are not independent measurements. If.+y m". with
(2. the degrees
The width of a confidence interval gives an intuitive measure of the
uncertainty of the evidence given by the data.y
then it is also
m"... A measurement process is established if this conceptually infinite sequence has a limiting mean and a standard deviation
For paired values of
and Y. where
-L (Xi X)2
The distribution of the quantity
quantity under essentially the same conditions generates a sequence of numbers Xl. Let m".
A number of basic formulas are extremely useful in
dealing with a quantity which is a combination of other measured quantities. I.. we can form the quantity Z. then
m".. From the tabulated values of
fidence intervals can be constructed to bracket
normally (see Table 2. b be constants . Normally distributed measurements are independent if they are not correlated or associated in any way.
and Yare independent .2). namely. these quantities are estimated by calculating and from measurements ... ./l)
distributed) is known. x2.
Algebra for the Manipulation of
Means and Variances
8asic Formulas. and be the respective limiting means of two measured quantities and Y../l
The confidence limits are the end points of confidence intervals defined by
for a given confidence
(probability of being correct in the long run).. the sequence of numbers generated follows approximately the normal distribution specified completely by the two quantities and (Y' Moreover .jn regardless of the distribution of the original numbers. .
and the confidence coefficient
where the value of of freedom
is determined by two parameters .
are correlated in such a way that paired values are likely to be both higher or lower than their respectiv'e' means .13).xy
In particular .10)
as previously stated. is estimated by
mz 0 when Sxy L: (Xi X)(Yi
Since the variance of a constant is zero .
. to the variance .
(2.'s are independent with variance
aX variance bY
and Yare independent and normally distributed .7)
if.. or and Yand is usually denoted by cov xy. If
and is estimated by
L: (Xi X)(Yi . then
a-~ 0'. Since
+ I. The case when two .x a.3) is defined as the covariance of The covariance . then
and Yare not correlated
is defined as:
by (2. The correlation coefficient
a. 3. we have
distributed with limiting
For measurement situations in general . that
for independent variables
and Yare independent. If a high value is Jjkely to be paired with a low tends to be negative.
and and a.xy L:
. 6..Then by formula (2.
and Yare independent
= 0 . 2.
get measurements that are independent
metrologists . Standard errors of the sample mean and the weighted means (of
independent measurements) are special cases of the
(lJn) L: Xi and the x.. V).X
value .usually strive to
. or can be assumed
be the respective variances of
(2.J L: (Xi
Both 5. The limiting mean of Z in (2. Let covariance of and a-~
lie between and and if. and vice Sxy tends
to zero (for large
n). similar .
is normally a-.quantities are dependent because both are
functions of other measured quantities will be treated under propagation of error formulas (see Eq.. then Sxy tends to be
Thus if and positive.9).ji)
by (2.9). and
is an average of
it is logical to compute
In practice .J/(. the quantity of interest is in the functional relationship to compute Sijj
f(m x.. . and
s~. .XI +
o-i.x\ is an average of
the over-all average
WI WI WI and
values . The o-~. . The results of a measurement process can usually be expressed bya number of averages ji.-
+'" ). and consequently the equations hold only as approximations.
(2. s~. The over-all average may be computed to be
The variance of
If the errors of measurements of these quantities are small in comparison with the values measured . may not be of direct interest. this is equivalent to a weighted mean of where the weights are proportional to the number of measurements in each average . These results . however /. Let XI' X2' Xk be a set of averages estimating the same quantity.. then for
~/(k + n).
will have to be used
in the above
Propagation of error formulas.. S etc.y. i. If used as an estimate of
and Yare independent = 0 and therefore the and Yare measured in pairs Sxy (Eq.
is given by
o-~ ro-~ +
l~. the estimated variances si
formulas . and o-~ x\ and X2'
WI XI W2 WI + W2
where the partial derivatives in square brackets are to be evaluated at the
averages of and If last term equals zero. However. m
as an estimate of
It is desired to estimate
+ '" +
+ '" . and the standard errors of these averages Si: s /5.
the weighting factors
are therefore also inversely proportional
to the respective variances of the averages. the propagation of error formulas usually work
surprisingly well. o-t and o-~
that are used in the
will often be replaced in practice by the computed values
The general formula for o-~
s~. This principle can
to more than two variables in the following manner. 2. .4) can be
. .. . +
+ '" +
Approximate formula for
sj. Wiley & Sons.
the last term again
These formulas can be extended to three or more variables if necessary. We will need
UfJ to calculate u~. For convenience . p. mJ
are functionally related to
the covariance is given approximately by
are functionally related.y.
is functionally related to
f(mu. These may be derived from the above formulas as exercises. Propagation
of error formulas for some simple functions
and Yare assumed to be independent.
Statistical Theory with Engineering Applications. Hald. as before .
and Yare independent .2
2. 301. The covariance UflfJ
(2. for example.i
(not directly derived from
I) the formulas)t
*Oistribution of in error for small
is highly skewed and normal approximation could be seriously
by A.3 with Yassumed to be independent. Inc.
b" S~ ). New York. . a few special formulas for commonly encountered functions are listed in Table 2. 1952.
evaluated at and If
that the partial derivatives are to be
The square brackets mean .
For sets of normally distributed measurements where the number measurements in each set is small ..16)
difference of duplicate readings.
master block nl times . thus the
(2.+ .. +
P VI +V2 +.
etc. we may consider the value of to be equal to that of (J" and claim that we know the precision
of the measuring process. arises that there are Estimates of Variances.+ .
(J"x. say less than ten . if
(a) the partial derivatives
when evaluated at the averages are small.. of course . the average range Rcan be computed..'+Vk
Si. Table 2. The standard deviation can be estimated by multiplying the number average range by the factor for
. To get a better evaluation of the precision of the calibration process . y. .. is
Pooling The problem often.
In these formulas . VI
= nl -
of the standard deviation .
As long as the nominal thicknesses of these blocks are within a certain range . (J"
are small compared to
then the approximations are good and
tends to be distributed normally
and normal approximation
(the ones marked by asterisks are highly skewed
could be seriously in error for small
gage block compared with the master block
a gage block may be compared with the resulting in an estimate of the variance si.15)
expression reduces to
+. we would wish to combine these estimates. For large considerable information is lost and this procedure is not recommended. For example . The pooled standard deviation
degrees of freedom. the precision of calibration can be apected to remain the same..4 lists these constants corresponding to the
of measurements in the set. .I . If there are sets of measurements each .
For the special case where
sets of duplicate measurements .
The degrees of freedom for ~he pooled estimate is the sum of the degrees
of freedom of
nk k. The rule is to combine the computed variances weighted by their respective degrees of freedom . . Another
times . or VI + Vk nl + n2
With the increased number of degrees of freedom is a more dependable estimate of (J" than an individual s. giving rise to
+ . Eventually... are available
the above formula reduces to:
where has di
(2. individual estimates.+nk
I)si.. . several estimates of a common variance (J" 2 which we wish to combine into
a single estimate.. and
The pooled estimate
In the example .. an estimate of the
standard deviation can be obtained by multiplying the range of these measurements by a constant.
*Adapted from Biometrika Tables for Statisticians.486 0. u.
of (J' from the range
886 0. it could be assumed that the average of these limiting means will approach a limit which can be called the limiting mean for all the groups. then an estimate of a-~ is
Sj. a.of as individual measurements.Clnce Groups. If in making calibrations there is a difference between groups .
The resolution of the total variance
components attributable to
identifiable causes or factors and the estimation of such components of variances are topics treated under analysis of variance and experimental design.591
the variance of
2 = a-~ + a-.
Vol. For selected treatments and examples see references 5.
and H. we have increased confidence in the value of the estimate obtained.
1. Let us use between groups ."
measurements are available giving averages XI'
X2.". but not necessarily the proper one for dispersions of values belonging to different groups. t .w' The within.4. These limiting means may be thought . Obviouslyao-~ to denote the variance corresponding to the differences groups.is a proper measure
of dispersions of values within the same group. then the limiting meanS of the groups are
In pooling estimates of variBetween VClr. . O. S.
a-i. Let us call this estimate the within.430
Thus for each
Cambridge ." XI
I degrees of freedom .
For the group
measurements in the group. Pearson
1958. thus . The University Press . edited by E. Component of ances from a number of subgroups . the measure of dispersion of the limiting means of the respective groups about the limiting mean for all groups. 6 .group standard deviation a. where
is the average of all
.Table 2. i.group standard deviation . or from set to set . Hartley. the differences of individuals from the respective group In estimating does not include the differences between means are used. say from day to day. and 8.
to be discussed in a
following section .
The control chart for mean with known value. eleven observed corrections of N B'
during May 1963. . We are warned that something may have gone wrong. The average of the observed
corrections does not agree with the accepted value because of certain systematic error ." .
3. The accepted value is incorrect .
In calibration of
weights at the National Bureau of Standards . two series of measurements on the same quantity may be compared.
possible that this par-
ticular set of limits would not cover the accepted value. Table 2. temporary or
seasonal .4040 . . The accepted value is correct.
What if the computed confidence limits for the observed correction do
not cover the accepted value? Three explanations may be suggested:
Comparison of Means and Variances
Comparison of means is perhaps one of the most frequently used techniques in metrology.
The sample means
and the computed standard errors are used to calculate confidence limits on the difference between two means. mately normally distributed. The NB' IO is treated as an unknown and
calibrated with each set of
weights tested using an intercomparison scheme based On a IOO. but not unduly alarmed since such an event will happen purely by chance about once every twenty times. we we will make an
chance alone . or sets of measurements on more than two quantities may be compared to determine homogeneity of the group
of means. we would be extremely reluctant to agree to the third
explanation since we have much more confidence in the accepted value than
the value based only on eleven calibrations.5 lists . By
. Hence the observed correction for NB' IO can be computed for each particular calibration. The " statistic derived from normal distribution theory is used in this procedure since we are assuming either
the measurement process is normal .
It is to be borne in mind in all of the comparisons discussed below
that We are
interested in comparing the limiting means. and we conclude that the
3995. particular to one or several members of this set of data for
which no adjustment has been made. These values include the accepted value observed corrections agree with the
accepted value. the weights to be calibrated are
intercompared with sets of standard weights having " accepted" corrections. the mass of the standard has changed.
Accepted corrections are based on years of experience and considered to be exact to the accuracy required. in choosing
of the time in the long run
= 0. would be the proper tool to use to monitor the constancy of the correction of the standard mass.0. The mean obtained from one measurement process may be compared with a standard value. For instance .0. In our example. However
know that 5 percent
error in our statement.
Comparison of G Mean with
or the ~ample averages are approxi-
2.gm standard weight. Calculated 95 percent confidence limits from the eleven observed corrections are .4040 mg.4041 and -
of . the accepted correction for the
NB' IO gram weight is .0. e.
0. 228 in the table of distribution.3994 0.4022
4075 0.40183 .77623417
= 1.could be zero. 025.
The difference between two
to be measured is the quantity
mx :X and where
and yare averages of a
respectively. for of
then by (2.40183 + 2.0. the confidence limits can be computed for
Suppose we are interested in knowing whether the difference
lower limits include zero . NB' lO gm "'
Observed Corrections to standard 10 gm wt in mg
*Data supplied by Robert Raybold . we conclude that the evidence
mx_ and if the upper and mx_ may take the value
Let us assume that
By Eq. Computation of confidence limits for ob:.
2 = .
mx. This problem can be solved by a technique previously introduced.40183 mg
difference = 0. Metrology Division .Table 2-5. respectively.2.3986 0.4008
63 63 2 63 63 4 63
Comparison Among Two quantities and
Means.fidenceinterval for the mean of the above sample of size 11 a/2 = 0. 000011744
S = 0.228 x 0.4012
4. otherwise .ments of
and Yare independent with
known variances ()~ and (). National Bureau of Standards. 00343 = computed standard deviation of an observed correction about the mean. 8).4015 3992
3973 0. = computed standard error of the mean.4071 0. 228 x 0. 00011744
(0.4053 ~0. Therefore
lI X' +
. 00103 = computed standard deviation of the mean of eleven corrections. 00103 = . 10)
measur(.erved corrections. and the corresponding value of is equal to 2. we could conclude that
zero. (2.2. For a two-sided 95 percentcon.
= 0. 00011744) = 0. 10.
It is desired to compare the means of observed corrections for the two sets of data. but the two can be assumed to be approxie..
and yare measured by
the same process. (2.40183
= 0. TI X 0. -
(n l)s~ (k
. we continue with the
calibration of weights with
II subsequent observed corrections during September and lO gm. the quantity
= 0. 00180
. the quantity
.40782 0.~ 2
This pooled computed variance estimates
mately equal . 000023813
is approximately normally
deviation of one under the assumption If Ux and
distributed with mean
zero and a standard
are not known .0
(2. can be set about
If this interval does not include
is strongly against the hypothesis
zero . 000035482) = 0. and a confidence
000017741 = 0.2. we may conclude that the evidence
As an example . the confidence interval (computed in the same manner as in the
preceding example) has been found to be
is distributed as Student's "
mx-y with .0
(2. For October . then
can be pooled by Eq.
. we conclude that ()~ could be eql!al to
For example . what is the and the question arises: If in fact
probability of getting a value of the ratio as large as the one observed? which are the tables list the values of and Vb. we
conclude that there is no evidence against the hypothesis that the two Note . and Va Let be the estimate of ()~ with s~
estimate of (J"~ with
bution depending on Vb degrees of freedom. the above situation is not likely to happen. or the range. the variance . Therefore
-/n . 086 X 0. degrees of freedom . In practice here we may be interested in whether however . e. ()~ cannot be assumed to be equal to ()~.
absolute difference between and
00271 + 2. if it
in the preceding
Here the degrees of freedom which is exceeded 5 percent . or mx
that we would reach a conclusion of no difference wherever the magnitude of ~ ji (0.
The following additional topics are treated in reference 4. reference 4 . therefore .
similarly. we are usually concerned with whether the imprecision of one
process exceeds that of another process.tk k
~ 0 ~
shows that the
confidence interval includes
zero.Sections 3and 15-
As we have seen .For
= 0. then it is highly
probable that a concl usion of no difference will still be reached. If the computed value of exceeds this tabulated value of
then we conclude that the evidence is against the hypothesis is less . say about 0.
we could compute the ratio of s~ to s~
()~. 1 ~
= 0. Comparison of several means by Studentized range. and
086. processes and
any of the three measures can be used . l. 025
. the precision of a measurement process can be expressed in terms of the computed standard deviation . When mx. the true difference but when the true difference is small .086 x 0. For each pair of values of the upper percentage point of the distribution exceeded with probability
of F. Sample sizes required under certain specified conditions-Tables
the tabulated value of
of the time for
these degrees of freedom is
the preference and convenience of the user.is large .
We could . The ratio Va and Vb' Tables of upper percentage points of s~/s~
has a distri-
are given in most statistical textbooks .Section 33.
ratio of s~ to s~.
we were interested in finding out if the
could reasonably be zero. 00180
= 0. 003 mg. 000011669
000023813 . 00180 = 0. compute the ()~.0. To compare the precision of two
Comparison of variances or ranges. and
= 10 .
()~. or 1. If a detection of difference of this magnitude is of interest . more measurements will
be needed. 00646
.00375 mg) calculated for the particular case.2041
. however observed average corrections are the same .
Section 4In the comparison of means . 00271 mg) is less than the half-width of the confidence interval (2.
is less than 2. between group averages have
.21 give confidence limits on the standard deviation of the process based on computed standard deviation. It follows
poorer than that of May.rol Charts Technique for
Maintaining .20 and A.
is taken to be 3
we know that the chance of a plotted point falling outside of the limits if the process is in control . for Vb
of freedom .
= 0. by day.Since 2.Stability and Precision
A laboratory which performs routine measurement or calibration operaas its daily product .
and the averages are plotted in an orderly sequence.
Control Chart for Averages. these charts usually also indicate where and when the trouble occurred. the value of O"E
= 3. Therefore .
the critical value of 05. if any.
maintaining at 0.five chance of detecting whether
when in fact
from each process.
and standard deviation
by a set of calibrations . The averages of these subgroups are computed. The above reasoning would be applicable to actual cases only if we have chosen the proper standard deviation (T. if a plotted point falls outside these limits.
The basic concept of a control chart is
with limiting mean
in accord with what has been disctlssed thus far.
numbers as products of a manufacturing process to furnish graphical
evidence on whether the measurement process is in statistical control or out of statistical control. These averages will have a mean and a standard deviation 0"/ vn where is the number of measurements within each subgroup. If the standard deviation is estimated
by pooling the estimates computed from each subgroup and denoted by
In the construction of the control chart for averages
is plotted as the
are plotted as control limits. is very small. or corrective measures are called for. a warning is sounded and investigative action to locate the " assignable " cause that produced the departure .
Cont. . These averages are approximately normally distributed.
For small degrees
.II in reference 4 gives the critical values of the ratios of ranges and Tables A.
that a small difference
is not likely to be detected with a
small number of measurements
of. numbers-averages . we conclude that there is no reason to believe
that the precision of the calibration process in
September and October is
is rather large
is 9. obviously differences
. 28. of measurements
Table A. etc. standard deviations and ranges. e.5
No. If it is out of control . 05 the probability of incorrectly concluding that
1. The sequence of numbers produced is divided into " rational" subgroups . The table below gives the approximate number of measurements required to have a four-outis the indicated multiple of (Tb
O"b. A measurement process is assumed. The control chart techniques therefore could be applied to these
tions yields .
. We note that all the averages are within the control limits . as previously stated.group differences is not u. or . 85 . The frequency curve of is limited on the left side by zero . SIGMA) 0 o o. Since the accepted value of the standard deviation was obtained pooling a large number of computed standard deviations for within-sets of calibrations .410.
. A slight shift upwards is also noted between the first 30 points and
the remainder. 2-5. therefore
The distribution of .412. excepting numbers 36 ." values would exceed the limits constructed by using alone. Two alternatives are open to us: (l) remove the cause of the betweengroup variation. could be found for these departures..
. . ----. 2. WEIGHTS RECALIBRATED. . where u~ represents the variance due to differences between groups. and 87. however . including the two sets of data given under comparison of means (points 18 through 28 .--~-----. 63 .
not been taken into consideration. however .
u .-----. and consequently also the degrees of freedom..
depends on the degrees of freedom associated with mo. .. 47 .~ -:::.
LOWER lIMIT=. .. and is not symmetrical about
. . as we have seen before u~ (u. The limits . Where there are between. ---.395. . These computed standard devia-
tions with few degrees of freedom can vary considerably by chance alone
process remains unchanged.404. is a measure of imprecision. As an illustration of the use of a control chart on averages . o
UPPER LlMIT=. If u~ is of any consequence as compared to many of the u.
400. we use again the NB' lO gram data.:
:::. No particular reasons . 6
:E ../n)...: ---------. Control chart on j for NB' 10 gram. and points 60 through 71). For a set of calibrations . The control chart on the computed standard deviations (or ranges) is therefore an indiseven though the precision of the
take it into account as has been previously discussed.20. 6 p. Five in a hundred falling outside of the three-sigma limits is more than predicted by the theory. .. ---------. and has a long " tail" to the right.-------. ChQrt The computed standard deviation .
Control lor StQndQrd f)ev. the number of measurements is usually small .~----.(Jf. the graph indicates that a " between-set" component may be present. A three-sigma limit of 8./n but .ons. (2) if such variation is a proper component of error
the variance of the individual
0 INDICATES CALIBRATIONS WITH COMPUTED STANDARD DEVIATIONS OUT OF CONTROl. 0
0 .g was used based on the " accepted" valueof standard deviation. One hundred observed corrections for NB' lO are plotted in Fig.
The stanstandard cell on ten separate days. 2-
3 of in most textbooks on quality such a table. are given reference 4 gives control.
The constants necessary
limits for averages .. A more comprehensive treatment on control charts is given in ASTM " Manual on Quality Control of Materials " Special Technical
The corresponding current value of standard deviation
ten comparisons will be denoted by made on (1"
of the average of
chart will be
CENTER LlNE= . 111
LOWER LlMIT= O31 (:3..jT(j.
for the construction of three-sigma
computed standard deviations .393)
(. As an example of the use of control charts on the precision of a calibration process ..
Electricity Division . We can therefore pool the standard deviations computed for each cell (with nine degrees of freedom) over a number of cells and take this
value as the current value of the standard deviation of a comparison .J
3? CECC' CALIBR ATE
. the notation employed in quality control work differs in some respect from what is now standard in statistics . National
*IIlustrative data supplied by Miss Catherine Law . if the standard deviation of
is not equal to (1" .
dard deviation of a comparison is calculated from the ten comparisons for each cell and the standard deviation for the average value of the ten com-
parisons is listed in the line marked SDA. Bureau of Standards.jT(j. * Standard cells in groups of four or six are usually compared with an NBS
A typical data sheet for a group of six cells .
(1".. Control chart ons for the calibration of standard cells. and correction factors have to be applied to some of these constants when the computed standard deviation is calculated by the definition given in this chapter.
and ranges . we will use data from NBS calibration of standard cells.
Let us assume that the precision of the calibration process remains the same.s
UPPER LlMIT= 190(3-SIGMA)
Fig. The control
s/.are not symmetrical about
but is equal to (1"
the process is known to be (1"
is a constant associated with the degrees of freedom in s. appears in Table 2. Section 18-
These corrections are explained in the footnote under the table. after all the necessary corrections . 2-6.. These values were plotted as
points 6 through II
24.80 32. 26.3
of reference 4 and apply the corrections as follows:
1. 24. are plotted as the first 32 points in Fig. 33. 25.054 x 1. 35. showed drifts during the period of
calibration. which is based on a value of u estimated with
The report of calibration for these cells (u
perhaps thousands of degrees of freedom and which is shown to be in control. 33.16 24. The main function of the chart is to justify the precision statement on the report of calibration .054 x 0. 24.) . 25. 32.
Calibration data for six standard cells
27. 25.0182342 1. 9227 x 0. 33. 34. 168
0.06 31. 718
33. 24. 24. 114
1.6 far exceed the upper control limit.0182264 1. 2. 24.482
402 0.97 31.30 31.127
0. 25. Microvolts
24.40 24. 31. 114 with 288 degrees of freedom. A fourth point barely exceeded the limit. It is to be noted that the data here were selected to include these three points for purposes of illustration only.031
x 0. 33.
. 2.18 31. 33. 24.43
24.738 0.53 31. All three cells which were from the same source . 262
Lower limit = n n
x 0.6. The standard deviations of the averages of subsequent calibrations are then plotted. volts
233 24. 168
677 33. volts
10. 32. 34.054 x 0.)
For example .96 1. The
standard deviation of the average is 0. 24. 33.6) was constructed using these values of center line and control limits computed from the 32 calibrations. 190
Upper limit =
The control chart (Fig.0182247 1. 23.10
24. the SDA's for 32 cells calibrated between June August 8 .group component is assumed to be negligible. 27. 34. 26. 26.058 366
33. these values are correct to 0. Based on a standard deviation of
Each value is the mean of ten observations made between O. and do not represent the normal sequence of calibrations. 34. 2. 26. 0.. 378
33.36 microvolts relative to volt as maintained by the national reference group.26 31. 23. 32. 1962 . 26.
Corrected Emf' s and standard deviations . 114 = 0.331 26. we find our constants for three-sigma control limits on
in Section 18. I2 microvolts for the
33. 33. 1.117 12) could read:
777 34. 24. 114 = 0.30 34.0182332 1.
23. 25. Three points in Fig.425
we have . as before. i. As a corollary.e.
Here we can express model
our measurements in the form of a mathematical
Y. it follows that and a standard deviation (T.
is a minimum. measurement with a limiting mean zero and € the random error (normal) of By (2. we have selected a value such that for
In using the arithmetic
limiting mean . Yi might be the change in length of a gage block steel observed for
(the intercept) and
+ € (2we can compute a value
n equally spaced temperatures Xi within a certain range. the method also states that the sum of squares of residuals divided by the number of measurements n less the number of
estimated constants p
will give us an estimate of ~ L (Yt
It is seen that the above agrees with our definition of S Suppose Y. and € the random error with For at Xi. Y is the quantity measured. fitted a constant to the data by the method of least squares .Linear Relationship and Fitting of
Constants by Least Squares
mean of n measurements as an estimate of the knowingly or unknowingly. the quantity measured .
(the slope) are two constants to be estimated . or
If we require the sum of squares of the residuals
to be a minimum.19)
where Y stands for the observed values
the limiting mean (a constant).
(1'. and observe We set limiting mean zero and variance Yt.
The method of least squares requires us to use that estimator
that the sum of squares of the residuals is a minimum (among all possible estimators). The quantity of
interest is the coefficient of thermal expansion
for each Xi. .22)
. exhibits a linear functional relationship with a variable which can be controlled accurately.1) and (2-9). example. then it can be shown that
'" L t=1 (x! (Xt X)(Yi
(2. The solution is
The deviations dt =
are called residuals. then a model can
be written as
1. respectively. Confidence interval for a single predicted value of Polynomial and multivariate relationships are treated in Chapter 6
the same reference.
3. and 9. (Yi
The standard errors of and are respectively estimated by Sh and
Sa. J. E. New York.
The following references are recommended for topics introduced in the first section of this chapter:
(2. New York . In addition to the three general references given above. (Xi
. . (X~=-' X)2
s~ L. Youden .2 degrees of freedom since two constants have been estimated from the data.23)
The variance of
can be estimated by
L. An Introduction to Scientific Research McGraw. Eisenhart . 1963. .
for the value of
corresponding to the degree of freedom and the selected
confidence coefficient. No. Confidence intervals for a point on the fitted line. the lower and upper limits of
Isa. 8 .HilI Book Company. Confidence band for the line as a whole.25)
With these estimates and the degrees of freedom associated with
and for the confidence coefficient fidence limits can be computed for selected if we assume that errors are normally distributed.
Instrument Calibration System
Journal of Research of the National Bureau
of Standards Vol. 2.
The following problems relating to a linear relationship between two
variables are treated in reference 4..
Experimentation and Measurement National Science Teacher Association Vista of Science Series No.
Ish. Churchill. the following are selected with special emphasis on their ease of understanding and applicability in the
. are: and Thus .xY
L. for a given 3. B. " Realistic Evaluation of the Precision and Accuracy of
2. Wilson . Chapters 7 . Scholastic Book Series.2
(2. 1952 .
1966. F. Introduction to Statistical Analysis (2nd ed. 1963. E. J. Statistical Theory and Methodology in Science and Engineering. 1958. F.
John Wiley and Sons. 1964. ). American Society for Testing and Materials Manual on Presentation of Data and Control Chart Analysis (STP 15D). D. New York. New York. and Hunter . Inc. Data Analysis. J. S. 1978.
5. Reading. A.
Precision Measurement and Calibration: Statistical Concepts
) Spec. Box. 575 p.
11. D. and Model Building. 407 p. W. .. Planning of Experiments John Wiley and Sons. and F. Pub!.
York. R. Stand. 1957. New York. G. McGraw. v. New York. W. . Areley. 308 p. Bur. . Inc.
1957. Ku. . John Wiley and Sons. . Massey. 1950. DataAnalysis and Regression. .. 588 p. Inc. Fitting Equations to Data. 343 p.. W. Mosteller. N.
7. Natrella . Him.
13.m. J. C. R. Brownlee . 1969.Hill Book Company. H. . H.
New York . R.
15. 1960. New York . K.
Inc. 1976. 1977. Mandel . G.p. 1979. Youden . Statistics for Experimenters
an Introduction to Design. U.
John Wiley .. New York . Statistical Adjustment of Data New York..
12. and Smith. . New York . W. Inc. Applied Regression Analysis John Wiley and Sons. 1970. Dixon. W. H. Massachusetts.
and Procedures Nati.
17. N. 9.
Additional Books on Treatment of Data
Statistical Method in Research and Production
O. E. P. American Society for Testing and Materials ASTM Standard on Precision and Accuracy for Various Applications 1977.
6.. John Wiley & Sons. and Wood. J. . S. Buch Introduction to the Theory of Probability and Statistics John Wiley & Sons .
14. Computer Analysis of Multifactor Data 2d ed.
ed. Draper. . 249 p. 162 p. Washington . Daniel . . and
18. . Inc.
Publishing Co. Hunter.Statistical Methods
John Wiley and
Process Analysis by Statistical Methods. 8. Government Printing Office. . Davies .elblau. New York. G. J. Inc. 464 p. a Second Course Addison-Wesley. D. 1943 261 p. 300. New York . 410 p.
The Statistical Analysis of Experimental Data. and K. Experimental Statistics NBS Handbook 91 . (U. Statistical Methods for Chemists John Wiley & Sons . M.
ciated so.ntro.r the display of small to mo.wer of small computers and the asso.ve article .5) and standard deviatio.f graphical metho.matically present a number o.
to. it has become additio.
. First pro.f areas of rectangles. and
graphs . but it uses digits of data values themselves to sho.ds.ry nature . Plots and graphs have always been po.se relative o.
co.ftware have pushed graphics into. only a few basic techniques to.riginal article remain as valid and appro.derate-sized
data sets.gram and is particularly suited fo.phisticated so.n (Fig. Simplicity (o.f
data may be useful. As John Tukey said the greatest value o.tice
what we never expected to see.w features of the data set instead o.ts. the main desirable attributes o.rk involved.n recent develo. the treatment o. The po.me naturally
through inspectio.nce co.
a stem and leaf plo.me
Over the years since the publication of the abo. " 11J An o.f the histogram .f plots . say? Answers to.n assumptions) are and sho.rmatio.l charts were intro. a postscript.priate as when first written. .f all graphs and plo.utlier? Skewed distributio. 2.
that are particularly useful
Since statistical graphics is a huge subject . a few additio.pments for the treatment o.pular with engineers and scientists . and robustness (no.
all these co.uld be .f numbers reveal little .duced in the o.duced by Walter Shewhart so.lumns o.ntro. plo.r further reading.o.. For this reaso.ts and graphs with
Graphics packages now-a.n . 2are classical examples o.n o.r mo.posed by John W. Co.f pertinent plo.
Stem and Leaf.f a picture is when it fo.me 60 years ago. but t~e~r use has been limited by
the time and wo.t depending o. discussed
Plots for Summary and Display of Data
The stem and leaf plot is a clo. the forefront.ns on statistical graphics
are added as .days
allo.re info.rces us to. It is equally apparent that the concepts and techniques intro.w the user
ease .l charts for the mean (Fig.f measurement data will be .ns o. together with references fo.nal sectio. do. self-explanato. no.n o.pular and most useful tool in business and industry.t retains mo.n from the data than the histo. Co.nstructed).f values? Po. yet the technique remains a po.ts for examination."
apparent that so. if anything. Tukey. and a good statistical package will also auto.delling? What is the data trying to.
0300 1. There is no need for a separate table of data values .0296 1.
Table 1. and 5 to 9 on the second.0299 0299 1.02. range from 1.0293 1. 29.0290 1.
0294 029502 00000086 00029 00008
1. 27 . 0)
Stem and leaf plot.
26. 0261.027683 00000069 00083 00024
00334 566678889 001233344444 5666678999 0022
Unit = (Y -1. or 261 to 305 after coding. listed in Table
27. In this case .0299 1. 30.0269 1.0286 1.0280 1. 1.0284 0270 1.
Fig. Stems are shown on the left side of the vertical line and individual leaves on the right side.Ratios
for reference sample
DETERMINATION I Instrument #4 Instrument
1. 28.0305 1. Is that an outlier? These data will be examined again later.0300 0297
1.0298 1. two lines are given to each stem.0298 1.0291 1.0290 1.0296 1.0279 1.0293 1.they are all shown in the plot! The plot shows . a slight skew towards lower values. 29 .0294 1.02. x 10'. 7 units.0305 .0286 1. because of the heavy concentration of values in stems 28 and 29 .028792 00000041 00064 00018
1.0296 1.0280 1. 1 is a stem and leaf plot of 48 measurements of the isotopic ratio
.0289 1.0274 1. 0 to 9.0286 1. with leaves 0 to 4 on the first line . plotted as 26 11.0287 1.0295 1.0293 1.Fig.0273 1.
Ave. The leaves .0285 1.0288 1.0297 1.0302 0294 1.0261 1. The smallest value separates from the next by 0.
DETERMINATION II Instrument #4 Instrument #1
1.0293 0302 1.0288 1.
(79/81). thus 26/1= 1. Thus 261 is split into two parts .are the last digits of the data values . The stems are 26 .
of 79Bromine to 81Bromine.0261 to 1.029675 00000024 00049 00014
1.0283 1. and 30.0263 1. Values of these 48 data points . 48 values of isotopic ratios. 28.
.The fourth spread (50 percent of data): . are not disturbed by outliers. and the upper are calculated.the median .5(Fu
Ft.. Spread .).suspected
outliers. and u. more of the original features of the batch of data then the two number summary.) is called the fourth spread.
2. the sample of numbers are first ordered from
the smallest to the largest . Location . as explained in expression (2. are not reflected in the average and
standard deviation.. Outlying data points . Two tails from the ends of the box extend
to Z (I) and Z en) respectively. .equal or different tail lengths. and whether it is in the middle of the box. m .0).
with the median line A " box " is then dividing the box into two parts. e.. and Ft.g.5(F
Ft. the int~rval (Fu .
4. To construct a box plot .
. in many ca. The lower cutoff limit
U sing a set of rules . Symmetry/skewness . 3 percent of the data will be in the interval if the distribution is normal and the data set is large).lower and upper cut off limits (99. The box plot (due also to Tukey) presents graphically
a five-number summary which .) contains half of all data points. These two numerical values characterize a nor-
mal distribution . the cutoff limits are also marked. (2).
Pt. We note that m u.
and the upper cutoff limit is
From a box plot one can see certain prominent features of a batch of
1. the lower fourth Ft.
3. Certain features of the data .Box Plot. By definition .Ft. The interval (Fu Ft.
1. skewness and extreme values . If the tails exceed the cutoff limits .ses .
(I). shows . a batch of data is summarized by its average
and standard deviation. (n)'
the median .
1 were actually made on two instruments .
(L) = z(L + 1))/2 if not. bromine (79/91).
(U) + z(U+l)J/2 ifnot.
(M) + Z (M+l))/2
if not. with 24 measurements each. SMALLEST
LOWER CUTOFF LIMIT
+ 1)/2 = (24 + 1)/2 = 12.
is the largest integer
not exceeding m. 5 = 36.
is an integer.
Upper Follrth + 1 (u) if
= 49 ~ 12.
is an integer.'
The 48 measurements of isotopic ratio bromine (79/81) shown in Fig. and for both instruments ate shown in
X(N). 2. for the combined data:
+ 1)/2 = (48 + 1)/2 = 24.
(291 + 292)/2 = 291.
The five numbersumroary for the 48 data point is .
where not exceeding
is the largest integer
(296 + 296)/2 = 296
where not exceeding
is the largest integer
(284 + 285)/2 = 284. Box plots for instrument instrument II ..
(m) if m is an integer. Box plot
of isotopic ratio.
FIg. in meters . think of why this is the case. one fixed and one variable. in other words
Data = model + error.
Magnesium content of specimens taken. we may consider that made up of two parts ..
We use measured data to estimate the fixed part . is less than the lower cutoff for the plot of the combined data . effectively and easily. there'is also a mild indication of
decreasing content of magnesium along the rod. It
results for these two instruments .seems apparent from these two plots that (a) there was a difference between the
Box plots for instruments I and II are similarly constructed. 3.
. The lowest value of instrument I . 261 . but it does not fall below the lower cutoff for instrument I alone.
Plots for Checking on Models and Assumptions
In making measurements .
Measurement = fixed part + variable part
. (the Mean . and (b) the precision of instrument II is better than that of instrument I. for example). 3 is a box plot of the amount of magnesium in different
parts of a long alloy rod. Fig.
Box plots can be used to compare several batches of data . -
Variations of box plots are giyen in 13) and (4).. Ten determinations were made at the selected locations for each specimen. As an exercise. One outlier appears obvious. Le. The specimen number represents the distance . from the edge of the 100 meter rod to the place where the specimen was taken. and use the variable part (perhapssununarized by the standard
deviation) to assess the goodness of our estimate.
and that a second degree equation might fit better:
(deflectiond = bI
(loadi) + b2(loadd2
is linear as in (2.
(deflection d = bI
A line is fitted by the method of least squares and the residuals from the
fit are plotted in Fig.
issue .21). is defined point and the fitted constant
and can be used for plots instead of
The ith residual Ti.
pertinent graphical techniques involving residuals or deviations are
Zi) is the value ofthe function computed with the fitted parameters.
and and then Ti Yi where
IT the relationship between (a bzd
are the intercept and the slope of the fitted straight
line .19).' '
and the ith residual is defined as
F(zd. Then the model is
the i th deviation . the resulting estimate is
or the average of all measurements. In calibrating a load cell .i as used in equation (2. The parabolic curve suggests that this model is inadequate . as in calibration work. the values of
are frequently consid-
ered to be known .
IT we use the method of least squares to estimate m . known deadweights are added in sequence and the deflf:'ctions are read after each additional load. Le.
Adequacy of Model.
the fixed part can be a function of another variable
more than one variable).
Yi Yi (zd
as the difference between the ith data
In general . be a constant
Let the ith data point be
let the fixed part
and let the random error be (. A straight line model looks plausible . respectively. 4. the differences between measured values and known values
will be denoted di. The deflections are plotted against Joads in Fig.
Following is a discussion of some of the issues
involved in checking the adequacy of models and assumptions.i.
deflection vS load. 5.5
LOAD CELL CALIBRATION
X X X
LOAD CELL CALIBRATION
003 004 ~0.-
Plot of residuals after linear fit.
7. yet a pattern may still be discerned upon close inspection. If the distribution is
approximately normal . suggested by this plot.
the assumptions are made that f: represents the random error (normal) and
CT. these . or to all load cells. 6.z:) (J
proportional to depth . assumptions are approximately true.19).
LOAD CELL CALIBRATION
0006 0004 0002
0002 0004 0006
Fig. departures from these assumptions. If the variability of residuals is
The assumption that errors are normally distributed may be checked by doing a normal probability plot of the residuals. the plot should show a linear relationship. In many measurement situations . The increase in variability with depths of dehas a limiting mean zero and a standard deviation fects is apparent from the figure. however .. Departures from these assumptions .
Plot of residuals after quadratic fit. 6.. would invalidate our model and our assessment of uncertainties. Residual plots help in detecting any unacceptable . These patterns can be investigated to see if they are peculiar to this individual load cell . These residuals look random . Hence the assumption of constant the range of F(. fitting of In(yd against known depths is .
In equation (2. Residuals from a straight line fit of measured depths of weld defects (ra" diographic method) to known depths (actually measured) are plotted against the known depths in Fig. resulting in Fig.
This is done and the residuals from this second degree model are plotted against loads . Uncertainties based on residuals resulting from an inadequate model
could be incorrect and misleading. Curvature
in the plot provides evidence that the distribution of errors is other than
. or are
common to all load cells of similar design .(/) .
Fig. 6 showing some evidence of depart ure from normality. Note the change in slope in the middle range.1
Fig. 001 INCHES)
Fig. Measured depth of weld defects vs true depth. 7. however . unless the curvature is substantial.
Plot of residuals after linear fit.
normal. Frequently symmetry of the distribution of
LOAD CELL CALIBRATION
Normal probability plot of residuals after quadratic fit. 8
is a normal probability plot of the residuals in Fig. Inspection of normal probability plot s is not an easy job .ALASKA PIPELINE RADIOGRAPHIC DEFECT BIAS CURVE
20 30 40 50 60
TRUE DEPTH (IN .
represented by der Component of Variance Between Groups .0
~O ~O 8.5O
. instruments .sequence plot differs from
control charts in that no formal rules are used for action.
0. three measurements every three months would yield bett. 19). temperatures . The differences between measured values and known values may
plots of results from three labo-
ratories of measuring linewidth standards using different optical imaging
methods. 1. The difference of 10 measured line widths from NBS values are plotted against NBS values for 7 days.
.Q. Plots of residuals versus days . twelve measurements a year apart. These plots could be of help to those laboratories in 10caHng and correcting causes of these anomalies.
Stability of a Sequence.hour meters from 1978 to 1982. Fig. p.
illS VAlUES f I. It is apparent that measurements made on day 5 were out of control in Fig.J. a normal probability plot. Fig. 11 shows three significant outliers..
12.Lab A. It is a practice of most Measurement experimenters to plot the results of each run in sequence to check whether the measurements are stable over runs. frequently the values of standards are considered
be used for a plot instead of residuals. 2).
In calibration work . Fig.
errors is of main concern. for example . Measurements on day 5 inconsistent with others. .----L-J
0. does not reflect the (discussed in the same variability over a period of time .. 12 plots the results of calibration of standard watt. It is evident (discussed unthat the variability of results at one time . may be used to check whether effects of these factors are indeed negligible. trends over time . and dependence on en~i~onmental conditions are easily seen from a plot of residuals versus such factors. operators .. humidities .0
2. sets . The run. Then a stem and leaf plot of data or residuals serves the purpose just as well as . Fig.un!
10. and 11 are multi~trace
be known. if not better than .. 10 . 9..
Figs. variability information than . say. 10 shows a downward trend of differences with increasing line widths. See . 9. etc. Hence . 9 . Shifts in levels between days or instruments (see Fig. The stability of a measurement process depends on many factors that are recorded but are not considered in the model because their effects are thought to be negligible. Differences of Iinewidth
measurements from NBS values. represented by Ub section).25
2. 0 8.
NIlS VALUES Iflmj
Trend with increasing linewidths.
. and residual plots . B. Many of the examples used were obtained through the use of DATA-
PLOT (6J. J.Wesley. Naturally graphs .
Ag. Reference l7J gives an up. 1985.
(2J Cleveland . Tukey pioneered " Exploratory Data Analysis " (lJ. A.
100. P. Thanks are also due to M. are briefly described in the above paragraphs.date bibliography on Statistical Graphics. Cleveland .
J. Carroll Croarkin for the use of Figs. S. and developed methods to probe for information that is present in data .
.1130. 2 and 3 and Shirley Bremer for editing and typesetting.
About 25 years ago .
J. and plots become one of the indispensable tools. The Elements of Graphing Data Wadsworth Advanced Book and Software . 1977. W.
(3J Chambers .
1130. 06. CiIL IIII/tII\ T I CJG CJE "/EM NI'MT
)( ioo CilLIIII/tII\TlCJG MIEE IIIMHB NI'MT
99. such as stem and leaf plots .
this software system. William S. I wish to express my thanks to Dr. John W. Exploratory Data Analysis
Addision. 9 thru 12 .to. Some of these techniques . and Tukey.. box plots .
Graphical Methods for Data Analysis
Wadsworth International Group
and Duxbury Press . Kleiner .
. 1983. . prior to the application of conventional statistical techniques. . John W. References (lJ through l5J cover most of the recent work done in this area.
. Measurements (% reg) on the power standard at I-year and 3-month intervals. Susannah Schiller for Figs. 12. Filliben .
. Paul F. Computing of Exploratory Data A nalysis Duxbury Press . Basics. David C. Vol. Applications . 419.. . .level Language
Nonlinear Fitting. Frederick .
No. et aI.
l6J Filliben . Understanding Robust and Exploratory Data A nalysis John Wiley & Sons
1983. . William S.423. Mosteller .
August . David C.
Computer Graphics . l5 . and Hoaglin .An Interactive High.
398. 1981. DATAPLOT . '
l4J Hoaglin . Research in Statistical Graphics Journal
of the American Statistical Association .
l5) Velleman . 1981.
for Graphics . June 1987
l7J Cleveland . Data Analysis and
pp. and Tukey. James J. John W.