Design of Experiments

Dr. Şeyda Serdar Asan

Today…
• What is experimental design?
• DoE terminology
• Planning experiments and experiment designs
• Helicopter experiment

SSA 1
 Paper Airplanes
• Design an airplane using
the provided design and
materials that will fly
the longest distance.
• Each experimental trial
costs $1000
• Winner will be
determined based on
the distance the
airplane flies and the
total cost of
experiments

DOE defined
• Design of Experiments is a method of
experimenting with the complex interactions
among parameters in a process or product
with the objective of optimizing the process or
product.

SSA 2
 Heat Generated by the CPU
We have three methods of
soldering chips on a circuit board
and we want to know which one
will perform better (generate less
heat) with the CPUs that we are
using.

We are concerned with the

amount of solder left on the circuit
boards with three levels (the small
quantity, medium quantity, or
heavy quantity of solder) and the
heat generated by the CPU, which
is the response variable.

Designed experiments
Independent
Controllable input factors
variables Dependent
x1 x2 … xp variable

Output
Input Process
y

z1 z2 … zp

Uncontrollable input factors

SSA 3
 Design of experiments seeks to
• Determine which variables affect the system.
• Determine how the magnitude of the
variables affects the system.
• Determine how to manipulate the variables to
control the response.
• Determine the optimum levels for the
variables so that variability in response is
small

The experiment design must

determine
• The response variable
• Number of factors to include in the
experiment
• The levels at which each factor will be tested
• The number of trials to be conducted at each
level for each factor
• The settings (conditions) for each trial

SSA 4
 Cake Experiment
• Objective
– To bake a good cake
• Dependent variable
– Taste: scored from 1 (bad) to 10 (good)
• Independent variables
– Mixing Time (min) [1 ‐ 5]
– Temperature (°C) [150 ‐ 200]
– Number of eggs [2 ‐ 6]

DOE Terminology
• Response (dependent variable):
– The variable(s) used to describe the reaction of a
process to variations in control variables (factors).
– The quality characteristic under study.
• Taste: scored from 1 (bad) to 10 (good)

SSA 5
 DOE Terminology
• Factors (independent variables):
– A variable which the experimenter will vary in
order to determine its effect on a response
variable.
• (mixing time, temperature, number of eggs…)
• Level:
– A value assigned to change the factor.
• Temperature; Level 1: 150°, Level 2: 200°

DOE Terminology
• Treatment:
– A set of conditions for an experiment
• factor x level used for a particular run.
• 2 factors x 2 levels  4 treatments
– 5min x 150° x 2eggs

• Run:
– The application of one treatment to one
experimental unit. An experimental trial.

SSA 6
 DOE Terminology
• Effect:
– The result of the experiment. The change in a
response variable produced by a change in the
factor level. What effect did the change produce?
• How well did the solder weld hold?
• How far did the airplane fly?
• How good did the cake taste?

DOE Terminology
• Degrees of Freedom:
– For a factor: the number of levels of a factor
minus 1.
– For an experiment: number of treatments times
number of replications minus 1.
• Interaction:
– Two or more factors that, together, produce a
result different that what the result of their
separate effects would be.

SSA 7
 DOE Terminology
• Noise factor:
– An uncontrollable (but measurable) source of
variation in the functional characteristics of a product
or process. This error term is used to evaluate the
significance of changes in the factor levels.
• Significance:
– A statistical test used to indicate whether a factor or
factor combination caused a significant change in the
response variable.
– It shows the importance of a change in a factor in
either a statistical sense or in a practical sense.

DOE Terminology
• Replication:
– Repeat of the treatment condition.
• Error:
– In an experimental situation, error could represent:
errors in experimentation, errors of measurement,
variation in materials or factors in general, or the
effect of noise factors on the response.
– Experimental error is the variability that is observed
when a treatment combination is repeated, that is,
replicated.

SSA 8
 An experiment defined
Factors: Supplier; press tonnage
Level: Supplier a, b, c;
Press tonnage 20, 25, 30
Effect:Pierce height
Response variable:Pierce height
Treatment:SxP = 3x3 = 9
Df:9-1 =8
Interaction: between s and p
Noise: material properties,
operating temperatures
Significance: did changing
suppliers or press tonnage
significantly affect pierce height?

Basic priciples of experimental design

• Replication
• Randomization
• Blocking

• Balance
• Efficiency
• Fit

SSA 9
 Characteristics of a Good Experiment
Design
• The experiment should provide unbiased
estimates of process variable and treatment
effects (factors at different levels).
• The experiment should provide the precision
necessary to enable the experimenter to
detect important differences.
• The experiment should generate results that
are free from ambiguity of interpretation.

Characteristics of a Good Experiment

Design
• The experiment should point the experimenter in
the direction of improvement.
• The experiment should be as simple as possible.
• Easy to set up and carry out
• Simple to analyze and interpret
• Simple to communicate or explain to others
• The experiment should plan for the analysis of
the results.

SSA 10
 Conducting an Experiment
Plan your experiment!
Successful experiments depend on how well
they are planned.
– What are you investigating?
– What is the objective of your experiment?
– What are you hoping to learn more about?
– What are the critical factors?
– Which of the factors can be controlled?
– What resources will be used?

Steps in conducting an experiment ‐ 1

• Establish the objective of the experiment by
defining the problem
– A well defined problem is half solved!
• Define the objective of the experiment
– We need a SMART definition.
• Specific
• Measurable
• Achievable, Actionable, Agreed
• Realistic, Relevant
• Timely

SSA 11
 Steps in conducting an experiment ‐ 2
• Identify components of the experiment
– How many and which factors will the design
consider?
– How many levels (options) are there for each
factor?
– What are the settings for each level?
– What is the response factor?
– How many trials will you conduct? (treatment,
repetition)

Steps in conducting an experiment ‐ 3

• Select an experimental design for your
experiment
– Single factor
– Full Factorial
– Fractional Factorial
– Other
• Plan your experiment

SSA 12
 Steps in conducting an experiment ‐ 4
• Run your experiment
– Complete the runs as specified by the experiment
at the levels and settings selected.
– Collect data about the results
– Enter the results into analysis program.

Steps in conducting an experiment ‐ 5

• Analyze the data and interpret results.
Use statistical tools to analyze your data and
determine the optimal levels for each factor.
– Analysis of Variance
– Analysis of Means
– Regression Analysis
– Pairwise comparison
– Response Plot
– Effects Plot
– ...
• Act on results

SSA 13
 Experiment Designs
• Trial and Error
• Single Factor Experiment
– one change at a time
• Full Factorial Experiment
– change many things at a time
• Fractional Factorial Experiment
– change many things at a time
• Others (Box‐Jenkins, Taguchi, Shainin, etc.)

Tire failure
Factor Level 1 Level 2
Speed 65 mph 70mph
Temperature 75 F (24) 85 F (30)
Tire Pressure 27 psi 32 psi
Chassis design A B

SSA 14
 DOE
• Single Factor Experiment
– A single factor experiment allows for the
manipulation of only one factor during an
experiment.
• Select one factor and vary it, while holding all other
factors constant.
– The objective in a single factor experiment is to
isolate the changes in the response variable as
they relate to the single factor.

DOE
• Single Factor Experiment Advantage:
– Simple to Analyze
• Only one thing changes at a time and you can see what
affect that change has on the system.

SSA 15
 DOE
• Single Factor Experiment Limitations:
– Time Consuming
• Changing only one thing at a time can result in dozens
of repeated experiments.
– Interactions between factors are not detectable.
• These experiments rarely arrive at an optimum setup
because a change in one factor frequently requires
adjustments to one or more of the other factors to
achieve the best results.
• Single factor changes rarely occur that are not inter‐
related to other factors in real life..

DOE
• Single Factor Experiment Example
– Which combination of factors avoids tire failure?
– To answer this, 8 treatments, with the variables
set at different levels each time, would have to be
run multiple times.

SSA 16
 Single Factor Experiment Example
Problem: What combination of factors avoids tire failure?

Speed Temperature Tire Pressure Chassis Design

65 75 27 A
70 75 27 A

65 75 27 A
65 85 27 A

65 75 27 A
65 75 32 A

65 75 27 A
65 75 27 B

DOE
• Full Factorial Experiment
– A full‐factorial design consists of all possible
combinations of all selected levels of the factors
to be investigated.
– Examines every possible combination of factors at
all levels.

SSA 17
 DOE
• Full Factorial Experiment:
– Advantage: A full‐factorial design allows the most
complete analysis
• Can determine main effects of the factors manipulated
on response variables
• Can determine effects of factor interactions on
response variables
• Can estimate levels at which to set factors for best
result
– Disadvantage: Time consuming and expensive

Full Factorial Experiment Example

What combination of factors avoids tire failure?
Speed Temperature Tire Pressure Chassis design
65 75 27 A
65 75 27 B
65 75 32 A
65 75 32 B
65 85 27 A
65 85 27 B
65 85 32 A
65 85 32 B
70 75 27 A
70 75 27 B
70 75 32 A
70 75 32 B
70 85 27 A
70 85 27 B
70 85 32 A
70 85 32 B

SSA 18
 Office air quality
• In the study on the effect of internal
partitioning on room air quality, three factors
are of interest:
– partition length (PL),
– partition height (PH),
– gap underneath (GU).

Office air quality

SSA 19
 Office air quality

2.0505
2.193
1.437
2.1805
1.949
2.321
1.5865
2.163

Cube Plot

2.1805

SSA 20
 Analyzing the experiment data
• Calculating the main effects:
– The difference in the mean response (result)
between the high level A and the low level of A,
averaged over the levels of the other factors

Main effects – estimated effects

ee(A) = 0.46
ee(B) = -0.29

SSA 21
 Main effects plot

Analyzing the experiment data

• Calculating the interaction effects:
– The difference between the main effect of A at the
high level of B and at the low level of B
OR
– The difference between the main effect of B at the
high level of A and at the low level of A

SSA 22
 Interpreting the interaction effects:

Interaction effect

SSA 23
 Plotting interaction effects AB

Plotting interaction effects AC

SSA 24
 Plotting interaction effects BC

Fractional Factorial Experiment

• Studies only a fraction or subset of all the
possible combinations. Runs selected from the
full‐factorial are not chosen at random.
• A selected and controlled multiple number of
factors are adjusted simultaneously.
– This reduces the total number of experiments.
– This reveals complex interactions between the factors.
– This will reveal which factors are more important than
others.

SSA 25
 Fractional Factorial Experiment
• Many different experiment design exist from
which an appropriate experiment design can
be selected.
– Plackett‐Burman Screening
– Taguchi
– Box‐Jenkins
– Shainin
– Others

Hypothesis and Experiment errors

• Experiment Errors
– lack of uniformity of the material
– inherent variability in the experimental technique

SSA 26
 Hypothesis and Experiment errors
• H0 : the change in the factor level has no
effect on the response variable

• H1: the change in the factor level has an

effect on the response variable

Hypothesis and Experiment errors

• Types of Errors
– Type I Error:  error
• A conclusion that a factor produces a significant effect
on a response variable when, in fact, its effect is
negligible (a false alarm).
– Type II Error:  error
• A conclusion that a factor does not produce a
significant effect on a response variable when, in fact,
its effect is meaningful.

conclusion H 0 true H0 false

H0 not rejected Correct Type II error
H0 rejected Type I error correct

SSA 27
 DOE ‐ Analysis of Variance ‐ ANOVA
• Used to determine whether or not changes in
factor levels have produced significant effects
upon a response variable.
– An ANOVA estimates the variance of the X using
two‐three different methods.
• If the estimates are similar, then detectable differences
between the subgroup averages are unlikely.
• If the differences are large, then there is a difference
between the subgroup averages that are not
attributable to background noise alone.
• ANOVA compares the between‐subgroup estimate of
variance of x with the within subgroup estimate.

Analysing a Factorial Experiment

Bake a Cake
Objective
To prepare a good cake
Dependent variable (Response)
Taste: scored from 1 (bad) to 10 (good)
Independent variables (Factors)
Mixing Time (min) [1 - 5]
Temperature (oC) [150 - 200]
Number of eggs [2 - 6]

SSA 28
 2-level Full Factorial Design

Calculation of Main Effects

Calculate mean responses
5 min = (8+9+8+9) / 4 = 8.5

1 min = (6+7+1+2) / 4 = 4

Effect = 5 min - 1 min

Time Effect = 8.5 - 4 = 4.5

SSA 29
 Plot & Interpret Main Effect

10
1 min = 4
9
8 5 min = 8.5
7
Increasing time
Taste

6
5 from 1 to 5 mins
4 leads to an
3 increase in taste
2 of approximately
1
4.5 units

1 5 Time [min]

Do same for other factors

Main Effects

Time 4.5

Temperature 1.0

Eggs -2.5

Compare size of each effect

SSA 30
 Calculation of Interactions
Calculate 2-by-2 table of
mean responses

Time (1) & Temp (150)

= (6+1) / 2 = 3.5

Time
1 5

Temp
150 3.5
200

Calculation of Interactions
Calculate 2-by-2 table of
mean responses

Time (5) & Temp (150)

= (8+8) / 2 = 8

Time
1 5
Temp

150 3.5 8
200

SSA 31
 Calculation of Interactions

Calculate 2-by-2 table of

mean responses

Time
1 5

Temp
150 3.5 8
200 4.5 9

Plot & Interpret Interaction

Time
10
1 5
9
Temp

150 3.5 8
8
7 200 4.5 9
Taste

6 Temp = 200 oC
Use averages to
5
construct
4
interaction plot.
3
Show effect of tim
2
Temp = 150 oC at both levels of
1
temperature.
1 2 3 4 5 Time [min]
The lines are parallel so effect of increasing time is the same at
both temperatures - NO INTERACTION.

SSA 32