DESIGN OF EXPERIMENTS

by
R. C. Baker

How to gain 20 years of experience in

one short week!

1
 Role of DOE in Process Improvement
DOE is a formal mathematical method for
systematically planning and conducting scientific
studies that change experimental variables
together in order to determine their effect of a
given response.

DOE makes controlled changes to input

variables in order to gain maximum amounts of
information on cause and effect relationships
with a minimum sample size.
2
 Role of DOE in Process Improvement

DOE is more efficient that a standard

approach of changing one variable at a
time in order to observe the variables
impact on a given response.

DOE generates information on the effect

various factors have on a response variable
and in some cases may be able to determine
optimal settings for those factors.
3
 Role of DOE in Process Improvement

DOE encourages brainstorming activities

associated with discussing key factors that may
affect a given response and allows the
experimenter to identify the key factors for
future studies.

DOE is readily supported by numerous statistical

software packages available on the market.

4
 BASIC STEPS IN DOE

Four elements associated with DOE:

1. The design of the experiment,
2. The collection of the data,
3. The statistical analysis of the data, and
4. The conclusions reached and
recommendations made as a result of the
experiment.

5
 TERMINOLOGY
Replication repetition of a basic
experiment without changing any factor
settings, allows the experimenter to estimate
the experimental error (noise) in the system
used to determine whether observed
differences in the data are real or just
noise, allows the experimenter to obtain
more statistical power (ability to identify
small effects)
6
 TERMINOLOGY

.Randomization a statistical tool used to minimize

potential uncontrollable biases in the experiment by
randomly assigning material, people, order that
experimental trials are conducted, or any other
factor not under the control of the experimenter.
Results in averaging out the effects of the
extraneous factors that may be present in order to
minimize the risk of these factors affecting the
experimental results.

7
 TERMINOLOGY
Blocking technique used to increase the
precision of an experiment by breaking the
experiment into homogeneous segments
(blocks) in order to control any potential
block to block variability (multiple lots of
raw material, several shifts, several
machines, several inspectors). Any effects
on the experimental results as a result of the
blocking factor will be identified and
minimized.
8
 TERMINOLOGY
Confounding - A concept that basically means that
multiple effects are tied together into one parent
effect and cannot be separated. For example,
1. Two people flipping two different coins would
result in the effect of the person and the effect of
the coin to be confounded
2. As experiments get large, higher order
interactions (discussed later) are confounded with
lower order interactions or main effect.

9
 TERMINOLOGY
Factors experimental factors or
independent variables (continuous or
discrete) an investigator manipulates to
capture any changes in the output of the
process. Other factors of concern are those
that are uncontrollable and those which are
controllable but held constant during the
experimental runs.

10
 TERMINOLOGY

Responses dependent variable measured

to describe the output of the process.

Treatment Combinations (run)

experimental trial where all factors are set
at a specified level.

11
 TERMINOLOGY

Fixed Effects Model - If the treatment

levels are specifically chosen by the
experimenter, then conclusions reached
will only apply to those levels.
Random Effects Model If the treatment
levels are randomly chosen from a
population of many possible treatment
levels, then conclusions reached can be
extended to all treatment levels in the
population.

12
 PLANNING A DOE

Everyone involved in the experiment should

have a clear idea in advance of exactly what
is to be studied, the objectives of the
experiment, the questions one hopes to
answer and the results anticipated

13
 PLANNING A DOE

Select a response/dependent variable

(variables) that will provide information
about the problem under study and the
proposed measurement method for this
response variable, including an
understanding of the measurement system
variability

14
 PLANNING A DOE

Select the independent variables/factors

(quantitative or qualitative) to be
investigated in the experiment, the number
of levels for each factor, and the levels of
each factor chosen either specifically (fixed
effects model) or randomly (random effects
model).

15
 PLANNING A DOE
Choose an appropriate experimental design
(relatively simple design and analysis methods are
almost always best) that will allow your experimental
questions to be answered once the data is collected
and analyzed, keeping in mind tradeoffs between
statistical power and economic efficiency. At this
point in time it is generally useful to simulate the
study by generating and analyzing artificial data to
insure that experimental questions can be answered
as a result of conducting your experiment

16
 PLANNING A DOE

Perform the experiment (collect data)

paying particular attention such things as
randomization and measurement system
accuracy, while maintaining as uniform an
experimental environment as possible.
How the data are to be collected is a critical
stage in DOE

17
 PLANNING A DOE

Analyze the data using the appropriate

statistical model insuring that attention is
paid to checking the model accuracy by
validating underlying assumptions
associated with the model. Be liberal in the
utilization of all tools, including graphical
techniques, available in the statistical
software package to insure that a maximum
amount of information is generated

18
 PLANNING A DOE

Based on the results of the analysis, draw

conclusions/inferences about the results,
interpret the physical meaning of these
results, determine the practical significance
of the findings, and make recommendations
for a course of action including further
experiments

19
SIMPLE COMPARATIVE EXPERIMENTS

Single Mean Hypothesis Test

Difference in Means Hypothesis Test with
Equal Variances
Difference in Means Hypothesis Test with
Unequal Variances
Difference in Variances Hypothesis Test
Paired Difference in Mean Hypothesis Test
One Way Analysis of Variance

20
 CRITICAL ISSUES ASSOCIATED WITH
SIMPLE COMPARATIVE EXPERIMENTS

How Large a Sample Should We Take?

Why Does the Sample Size Matter
Anyway?
What Kind of Protection Do We Have
Associated with Rejecting Good Stuff?
What Kind of Protection Do We Have
Associated with Accepting Bad Stuff?

21
 Single Mean Hypothesis Test
After a production run of 12 oz. bottles,
concern is expressed about the possibility that
the average fill is too low.
Ho: = 12
Ha: <> 12

level of significance = = .05

sample size = 9
SPEC FOR THE MEAN: 12 + .1
22
 Single Mean Hypothesis Test
Sample mean = 11.9
Sample standard deviation = 0.15
Sample size = 9
Computed t statistic = -2.0
P-Value = 0.0805162
CONCLUSION: Since P-Value > .05, you
fail to reject hypothesis and ship product.

23
Single Mean Hypothesis Test Power Curve

Power Curve
alpha = 0.05, sigma = 0.15
1

0.8

0.6
Power

0.4

0.2

0
11.8 11.9 12 12.1 12.2
True Mean

24
Single Mean Hypothesis Test Power
Curve - Different Sample Sizes

25
 DIFFERENCE IN MEANS - EQUAL
VARIANCES
Ho:
Ha:
level of significance = = .05
sample sizes both = 15
Assumption: =

Sample means = 11.8 and 12.1

Sample standard deviations = 0.1 and 0.2
Sample sizes = 15 and 15
26
DIFFERENCE IN MEANS - EQUAL VARIANCES
Can you detect this difference?

27
DIFFERENCE IN MEANS - EQUAL
VARIANCES

28
DIFFERENCE IN MEANS - unEQUAL
VARIANCES
Same as the Equal Variance case except
the variances are not assumed equal.

How do you know if it is reasonable to

assume that variances are equal OR
unequal?

29
 DIFFERENCE IN VARIANCE
HYPOTHESIS TEST
Same example as Difference in Mean:
Sample standard deviations = 0.1 and 0.2
Sample sizes = 15 and 15
**********************************
Null Hypothesis: ratio of variances = 1.0
Alternative: not equal
Computed F statistic = 0.25
P-Value = 0.0140071
Reject the null hypothesis for alpha = 0.05. 30
DIFFERENCE IN VARIANCE
HYPOTHESIS TEST
Can you detect this difference?

31
 DIFFERENCE IN VARIANCE
HYPOTHESIS TEST -POWER CURVE

32
 PAIRED DIFFERENCE IN MEANS
HYPOTHESIS TEST
Two different inspectors each measure 10
parts on the same piece of test equipment.
Null hypothesis: DIFFERENCE IN MEANS
= 0.0
Alternative: not equal
Computed t statistic = -1.22702
P-Value = 0.250944
Do not reject the null hypothesis for alpha =
0.05.
33
 PAIRED DIFFERENCE IN MEANS
HYPOTHESIS TEST - POWER CURVE

Power Curve
alpha = 0.05, sigma = 3.866
1

0.8

0.6
Power

0.4

0.2

0
-5 -4 -3 -2 -1 0 1 2 3 4 5
Difference in Means

34
ONE WAY ANALYSIS OF VARIANCE

Used to test hypothesis that the means of

several populations are equal.
Example: Production line has 7 fill needles and
you wish to assess whether or not the average
fill is the same for all 7 needles.
Experiment: sample 20 fills from each of the 9
needles and test at 5% level of sign.
Ho: =
35
 RESULTS: ANALYSIS OF VARIANCE
TABLE

Analysis of Variance
----------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Valu
----------------------------------------------------------------------------
Between groups 1.10019 6 0.183364 18.66 0.000
Within groups 1.30717 133 0.00982837
----------------------------------------------------------------------------
Total (Corr.) 2.40736 139

36
 SINCE NEEDLE MEANS ARE NOT ALL
EQUAL, WHICH ONES ARE DIFFERENT?
Multiple Range Tests for 7 Needles
Method: 95.0 percent LSD
Col_2 Count Mean Homogeneous Groups
--------------------------------------------------------------------------------
N7 20 11.786 X
N2 20 11.9811 X
N1 20 11.9827 X
N6 20 11.9873 X
N3 20 11.9951 X
N5 20 11.9953 X
N4 20 12.11 X

37
 VISUAL COMPARISON OF 7
NEEDLES

Box-and-Whisker Plot

N1
N2
N3
Col_2

N4
N5
N6
N7

11.5 11.7 11.9 12.1 12.3

Col_1

38
 FACTORIAL (2k) DESIGNS
Experiments involving several factors ( k =
# of factors) where it is necessary to study
the joint effect of these factors on a specific
response.
Each of the factors are set at two levels (a
low level and a high level) which may
be qualitative (machine A/machine B, fan
on/fan off) or quantitative (temperature
800/temperature 900, line speed 4000 per
hour/line speed 5000 per hour).
39
 FACTORIAL (2k) DESIGNS
Factors are assumed to be fixed (fixed
effects model)
Designs are completely randomized
(experimental trials are run in a random
order, etc.)
The usual normality assumptions are
satisfied.

40
 FACTORIAL (2k) DESIGNS
Particularly useful in the early stages of
experimental work when you are likely to
have many factors being investigated and
you want to minimize the number of
treatment combinations (sample size) but, at
the same time, study all k factors in a
complete factorial arrangement (the
experiment collects data at all possible
combinations of factor levels).

41
 FACTORIAL (2k) DESIGNS

As k gets large, the sample size will

increase exponentially. If experiment is
replicated, the # runs again increases.
k # of runs
2 4
3 8
4 16
5 32
6 64
7 128
8 256
9 512
10 1024 42
 FACTORIAL (2k) DESIGNS (k = 2)
Two factors set at two levels (normally
referred to as low and high) would result in
the following design where each level of
factor A is paired with each level of factor
B.
Generalized Settings Orthogonal Settings
RUN Factor A Factor B RESPONSE RUN Factor A Factor B RESPONSE
1 low low y1 1 -1 -1 y1
2 high low y2 2 +1 -1 y2
3 low high y3 3 -1 +1 y3
4 high high y4 4 +1 +1 y4
43
 FACTORIAL (2k) DESIGNS (k = 2)
Estimating main effects associated with
changing the level of each factor from low
to high. This is the estimated effect on the
response variable associated with changing
factor A or B from their low to high values.
( y2 y4 ) ( y1 y3 )
Factor A Effect
2 2
( y3 y4 ) ( y1 y2 )
Factor B Effect
2 2
44
 FACTORIAL (2k) DESIGNS (k = 2):
GRAPHICAL OUTPUT
Neither factor A nor Factor B have an effect
on the response variable.

45
 FACTORIAL (2k) DESIGNS (k = 2):
GRAPHICAL OUTPUT
Factor A has an effect on the response
variable, but Factor B does not.

46
 FACTORIAL (2k) DESIGNS (k = 2):
GRAPHICAL OUTPUT
Factor A and Factor B have an effect on the
response variable.

47
 FACTORIAL (2k) DESIGNS (k = 2):
GRAPHICAL OUTPUT
Factor B has an effect on the response variable, but only if
factor A is set at the High level. This is called
interaction and it basically means that the effect one factor
has on a response is dependent on the level you set other
factors at. Interactions can be major problems in a DOE if
you fail to account for the interaction when designing your
experiment.

48
 EXAMPLE:
FACTORIAL (2k) DESIGNS (k = 2)

A microbiologist is interested in the effect

of two different culture mediums [medium 1
(low) and medium 2 (high)] and two
different times [10 hours (low) and 20 hours
(high)] on the growth rate of a particular
CFU [Bugs].

49
 EXAMPLE:
FACTORIAL (2k) DESIGNS (k = 2)
Since two factors are of interest, k =2, and
we would need the following four runs
resulting in
Generalized Settings
RUN Medium Time Growth Rate
1 low low 17
2 high low 15
3 low high 38
4 high high 39

50
 EXAMPLE:
FACTORIAL (2k) DESIGNS (k = 2)
Estimates for the medium and time
effects are

Medium effect = [(15+39)/2] [(17 +

38)/2] = -0.5

Time effect = [(38+39)/2] [(17 + 15)/2] =

22.5

51
 EXAMPLE:
FACTORIAL (2k) DESIGNS (k = 2)

52
 EXAMPLE:
FACTORIAL (2k) DESIGNS (k = 2)
A statistical analysis using the appropriate
statistical model would result in the
following information. Factor A (medium)
and Factor B (time)
Type III Sums of Squares
------------------------------------------------------------------------------------
Source Sum of Squares Df Mean Square F-Ratio P-Value
------------------------------------------------------------------------------------
FACTOR A 0.25 1 0.25 0.11 0.7952
FACTOR B 506.25 1 506.25 225.00 0.0424
Residual 2.25 1 2.25
------------------------------------------------------------------------------------
Total (corrected) 508.75 3
All F-ratios are based on the residual mean square error.

53
 EXAMPLE:
CONCLUSIONS
In statistical language, one would conclude
that factor A (medium) is not statistically
significant at a 5% level of significance
since the p-value is greater than 5% (0.05),
but factor B (time) is statistically significant
at a 5 % level of significance since this p-
value is less than 5%.

54
 EXAMPLE:
CONCLUSIONS
In layman terms, this means that we have
no evidence that would allow us to
conclude that the medium used has an effect
on the growth rate, although it may well
have an effect (our conclusion was
incorrect).

55
 EXAMPLE:
CONCLUSIONS
Additionally, we have evidence that would
allow us to conclude that time does have an
effect on the growth rate, although it may
well not have an effect (our conclusion was
incorrect).

56
 EXAMPLE:
CONCLUSIONS

In general we control the likelihood of

reaching these incorrect conclusions by the
selection of the level of significance for the
test and the amount of data collected
(sample size).

57
 2k DESIGNS (k > 2)
As the number of factors increase, the
number of runs needed to complete a
complete factorial experiment will increase
dramatically. The following 2k design
layout depict the number of runs needed for
values of k from 2 to 5. For example, when
k = 5, it will take 25 = 32 experimental runs
for the complete factorial experiment.

58
 Interactions for 2k Designs (k = 3)
Interactions between various factors can
be estimated for different designs above
by multiplying the appropriate columns
together and then subtracting the average
response for the lows from the average
response for the highs.

59
Interactions for 2k Designs (k = 3)

a b c ab ac bc abc
-1 -1 -1 1 1 1 -1
+1 -1 -1 -1 -1 1 1
-1 +1 -1 -1 1 -1 1
+1 +1 -1 1 -1 -1 -1
-1 -1 +! 1 -1 -1 1
+1 -1 +1 -1 1 -1 -1
-1 +1 +1 -1 -1 1 -1
+1 +1 +1 1 1 1 1

60
 2k DESIGNS (k > 2)

Once the effect for all factors and

interactions are determined, you are able to
develop a prediction model to estimate the
response for specific values of the factors.
In general, we will do this with statistical
software, but for these designs, you can do
it by hand calculations if you wish.

61
 2k DESIGNS (k > 2)

For example, if there are no significant interactions

present, you can estimate a response by the
following formula. (for quantitative factors only)

62
 ONE FACTOR EXAMPLE

Plot of Fitted Model

95

85
GRADE

75

65

55
10 12 14 16 18 20
#HRS STUDY 63
 ONE FACTOR EXAMPLE
The output shows the results of fitting a
general linear model to describe the
relationship between GRADE and #HRS
STUDY. The equation of the fitted general
model is
GRADE = 29.3 + 3.1* (#HRS STUDY)
The fitted orthogonal model is
GRADE = 75 + 15 * (SCALED # HRS)

64
 Two Level Screening Designs
Suppose that your brainstorming session
resulted in 7 factors that various people
think might have an effect on a response.
A full factorial design would require 27 =
128 experimental runs without replication.
The purpose of screening designs is to
reduce (identify) the number of factors
down to the major role players with a
minimal number of experimental runs. One
way to do this is to use the 23 full factorial
design and use interaction columns for
factors.
65
Note that
* Any factor d effect is now confounded with the a*b
interaction
* Any factor e effect is now confounded with the a*c
interaction
* etc.
a
* What
b
is the d*e
c
interaction
d = ab
confounded
e = ac
with????????
f = bc g = abc
-1 -1 -1 1 1 1 -1
+1 -1 -1 -1 -1 1 1
-1 +1 -1 -1 1 -1 1
+1 +1 -1 1 -1 -1 -1
-1 -1 +! 1 -1 -1 1
+1 -1 +1 -1 1 -1 -1
-1 +1 +1 -1 -1 1 -1
+1 +1 +1 1 1 1 1
66
Problems that Interactions Cause!
Interactions If interactions exist and you fail to
account for this, you may reach erroneous
conclusions. Suppose that you plan an
experiment with four runs and three factors
resulting in the following data:

67
Problems that Interactions Cause!
Factor A Effect = 0
Factor B Effect = 0
In this example, if you were assuming that
smaller is better then it appears to make
no difference where you set factors A and B.
If you were to set factor A at the low value
and factor B at the low value, your response
variable would be larger than desired. In this
case there is a factor A interaction with
factor B.
68
Problems that Interactions Cause!

Interaction Plot
10 FACTOR B
-1
9 1
RESPONSE

5
-1 1
FACTOR A

69
 Resolution of a Design
Resolution III Designs No main effects are
aliased with any other main effect BUT some (or
all) main effects are aliased with two way
interactions
Resolution IV Designs No main effects are
aliased with any other main effect OR two factor
interaction, BUT two factor interactions may be
aliased with other two factor interactions
Resolution V Designs No main effect OR two
factor interaction is aliased with any other main
effect or two factor interaction, BUT two factor
interactions are aliased with three factor
interactions.
70
 Common Screening Designs
Fractional Factorial Designs the total
number of experimental runs must be a
power of 2 (4, 8, 16, 32, 64, ). If you
believe first order interactions are small
compared to main effects, then you could
choose a resolution III design. Just
remember that if you have major
interactions, it can mess up your screening
experiment.

71
 Common Screening Designs
Plackett-Burman Designs Two level,
resolution III designs used to study up to
n-1 factors in n experimental runs, where
n is a multiple of 4 ( # of runs will be 4, 8,
12, 16, ). Since n may be quite large,
you can study a large number of factors
with moderately small sample sizes. (n =
100 means you can study 99 factors with
100 runs)

72
 Other Design Issues
May want to collect data at center points to
estimate non-linear responses
More than two levels of a factor no
problem (multi-level factorial)
What do you do if you want to build a non-
linear model to optimize the response.
(hit a target, maximize, or minimize)
called response surface modeling

73
 Response Surface Designs Box-Behnken

RUN F1 F2 F3 Y100

1 10 45 60 11825

2 30 45 40 8781

3 20 30 40 8413

4 10 30 50 9216

5 20 45 50 9288

6 30 60 50 8261

7 20 45 50 9329

8 30 45 60 10855

9 20 45 50 9205

10 20 60 40 8538

11 10 45 40 9718

12 30 30 50 11308

13 20 60 60 10316

14 10 60 50 12056
15 20 30 60 10378

74
 Response Surface Designs Box-Behnken

Regression coeffs. for Var_3

----------------------------------------------------------------------
constant = 2312.5
A:Factor_A = 36.575
B:Factor_B = 200.067
C:Factor_C = 3.85
AA = 9.09875
AB = -9.81167
AC = -0.0825
BB = 0.117222
BC = -0.311667
CC = 1.10875

75
 Response Surface Designs Box-Behnken

Contours of Estimated Response Surface

Factor_C=60.0
60 Var_3
9300.0
55 9500.0
50 9700.0
Factor_B

9900.0
45 10100.0
10300.0
40 10500.0
35 10700.0
10900.0
30 11100.0
10 14 18 22 26 30 11300.0
11500.0
Factor_A 11700.0

76
 CLASSROOM EXERCISE
STUDENT IN-CLASS EXPERIMENT:
Collect data for experiment to determine
factor settings (two factors) to hit a target
response (spot on wall).
Factor A height of shaker (low and high)
Factor B location of shaker (close to
hand and close to wall)
Design experiment would suggest
several replications

77
 CLASSROOM EXERCISE
Conduct Experiment student holds 3 foot
pin the tail on the donkey stick and
attempts to hit the target. An observer will
assist to mark the hit on the target.
Collect data students take data home for
week and come back with what you would
recommend AND why.
YOU TELL THE CLASS HOW TO PLAY
THE GAME TO WIN.

78
CLASSROOM EXERCISE

79
 CLASSROOM EXERCISE

MARKER VERTICAL STANDARD

1ST OBS 2ND OBS 3RD OBS 4TH OBS MEAN
STICK POLE DEVIATION
L L -2.750 -4.500 -4.750 -5.000 -4.250 1.021

H L -12.500 -6.750 -4.625 -4.000 -6.969 3.871

L H 3.000 3.250 3.875 6.250 4.094 1.484

H H 4.625 11.250 12.625 14.000 10.625 4.155

MARKER L = VERTICAL POLE WAS CLOSE TO WALL (MARKER END OF STICK

STICK H=VERTICAL POLE WAS CLOSE TO HAND

VERTICAL L=SHAKING DEVICE LOCATED LOW ON VERTICAL POLE

POLE H=SHAKING DEVICE LOCATED HIGH ON VERTICAL POLE

80
Contour Plots for Mean and Std. Dev.

81