Design of Experiments

Rajiv Gupta
BITS Pilani
 Agenda
• Module 1
– The Need for Experiments
• Module 2
– Sequential Experiment s and PDSA
• Module 3
– Testing the Change
• Module 4
– Design and Analysis of Experiments
• Module 5
– Example of Experiment with One Factor

2
• Begin Module 1
– The Need for Experiments

3
 Process Variation
• Deming’s causes of variation
– Common causes
– Special causes
– 85 % of the problems are caused by the system
(management) and 15% by the workers
• Common causes cannot be explained. This is the
inherent variation in a process. Only management can
take action to change the system.
• Trying to adjust to common causes may increase
variation
• Special causes need to be investigated and corrective
action taken

4
 Control Charts
• In control charts we plotted data points over time to
monitor whether a process was in control/stable
• Only when an abnormality was observed in the
process output that we intervened to investigate the
cause of the abnormality and to correct it
• This method is a passive method because we are not
taking proactive action to bring about a change in
the system. Rather we are reacting to specific
occurrences that are observed.
• Only special causes are addressed
5
Control Charts: X-bar and R Charts

6
 Proactive Action on a Process
• In addition to special causes, there are common
causes of variation which require management
action to bring about improvement in the
system/process
• Such action needs to be proactive where we attempt
to determine what changes will bring about positive
change
• Experiments are needed to evaluate the effect of
making changes to the system so that improvement
can be effected.
7
Reducing Fraction Defectives

8
 Location and Dispersion
(outputs of a process of shooting arrows)

9
 Experiments
• In order to be reasonably confident that a particular
change will bring about a desired/positive outcome,
the experiment needs to be carefully planned
• Design of experiments is intended to provide a
framework for conducting the experiments in a
planned, methodical fashion
• The results of the experiment are interpreted as a
prediction, i.e., how the system is expected to
behave in the future as a result of the changes
investigated through experimentation
10
• End of Module 1

11
• Begin Module 2
– Sequential Experiments and PDSA

12
 The Model for Improvement
What are we trying to accomplish?
How will we know that a change is
Current an improvement?
Knowledge
What changes can we make that will
result in an improvement?

Cycle for learning

and improvement

From “ Quality Improvement through Planned Experimentation” by Ronal D.Moen,

Thomas W. Nolan, Lloyd P. Provost 13
• What are we trying to accomplish?
– Define objectives of the study
• Customer complaints
• Too much rework
• Delays in delivery

14
• How will we know that a change is an
improvement?
– Need to clearly identify the criteria or measures
for improvement
– Multiple measures that could be in conflict
– Need to have a systematic way to prioritize
measures
• Examples could be performance, time, usability,
aesthetics, durability, consistency, etc.

15
• What changes can we make that will result in
an improvement?
– Need to understand the nature of the problem at
hand
– Need to be familiar with new technologies that
could help improve the process or system
– Need to consider alternative materials and
alternative suppliers of the materials
– Need to be innovative

16
 Sequential Building of Knowledge

A P
S D

A P
S D

A P
A P S D
S D

Developing a change Testing a change Implementing a change

17
 Testing Using PDSA Cycles
• During the Plan phase of the testing cycle:
– We need to clearly state how the test is to be performed,
i.e., who will conduct the test, where and when will it be
done, how many replications will be needed, what will be
outcome measures, etc.
– We should look for ways to scale down and decrease the
time required for the initial test so that more knowledge is
gained quickly, which can be used in subsequent tests
– Be innovative in the design of the test without getting too
complicated

18
 Testing Using PDSA Cycles
• During the Do phase of the testing cycle, we actually conduct
the test or experiment. The outcome should provide data for
analysis and gaining knowledge about the effect of the
change.
• If the outcome is not as expected, it might be due to:
– The test not being conducted properly
– The support processes not being adequate
– The result is actually not favorable
• Whatever be the outcome, we get information that helps us in
planning further tests and retests, if required

19
 Testing Using PDSA Cycles
• During the Study phase of the testing cycle, we compare what
our predictions were with regard to the outcome of the
change, to the actual results.
• If the results do not match our prediction, we should attempt
to gain understanding why the prediction was not accurate
• If the test outcome matches the prediction, then our degree
of belief in the change outcome is increased

20
 Testing Using PDSA Cycles
• During the Act phase of the testing cycle, we need to
determine what is the next course of action:
– Is further testing required to increase the degree of belief?
– Should we test other changes to study their effect?
– Should modification be made to the current change and be
retested?
– Should we alter the experiment conditions to retest the
effect of the change?
– Are we ready to roll out the change on a full-scale basis?

21
• End of Module 2

22
• Begin Module 3
– Testing the Change

23
 Testing the Change
• It is important to design the test or experiment carefully as
the outcome and the interpretation of the results will
determine future course of actions
• As discussed earlier, it is recommended that tests be
conducted in sequential cycles so that our degree of belief
increases that the change will result in improvement
• It is also important to ensure that the improvement will be
sustained in a variety of circumstances
• It is also essential to make sure that the outcome of the
experiment is due to the tested change as opposed to any
other reason

24
 Pre-tests and Post-tests
One of the methods to ascertain the effects of the
change is to conduct a measurement before (pre-test),
and a measurement after (post-test). For example
consider a change made in Week 7 and the results of the
pre-test in Week 4 and the post-test in Week 11 are given
below:

Delay

Before the change After the change

From”Quality Improvement through Planned Experimentation” 25
Delay

Change made here

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Week

26
 Alternate Scenario 1

Delay

Change made here

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Week

27
 Alternate Scenario 2

Delay

Change made here

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Week

28
 Alternate Scenario 3

Delay

Change made here

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Week

29
 Alternate Scenario 4

Delay

Change made here

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Week

30
 Testing the Change
• How can we make sure that the change in
output is caused by the change we made?
– Plot a run chart with data from a few weeks
before and after the change
– Conducting the test over a longer time period will
also add to our degree of belief
– Remove the change and see if the process reverts
back to the performance before the change
– Use planned experimentation

31
Basic Principles for Testing Change
• Test on a small scale and build knowledge
sequentially
• Collect data over time
• Include a wide range of conditions in the
sequence of tests

32
• End of Module 3

33
• Begin Module 4
– Design and Analysis of Experiments

34
Enumerative and Analytical Studies
• Enumerative studies
– Are done with a universe that exists at present and static
– The universe consists of tangible units, any number of
which may be sampled and studied
– The aim of an enumerative study is estimation about some
aspect of the universe
– Action may be taken on the universe based on the
estimate through sampling
– Statisticians play a large role in enumerative studies
– Examples of enumerative studies include Census, polling of
people, etc.
35
Enumerative and Analytical Studies
• Analytical studies
– Are done to bring about a change in a product, process, or
system in the future
– Since we are dealing with the future the universe is
dynamic
– The aim of an analytic study is prediction that one of
several alternatives will be superior to the others in the
future
– The role of the subject matter expert is very important in
analytic studies
– Examples could be selection of one vendor from several
for the supply of a given part
36
 Definitions
• Response variable – A variable observed or measured in an
experiment. It is the outcome from the experiment. An
experiment could have more than one response variables
• Factor – A variable deliberately changed in a controlled
manner to observe its impact on the response variable
• Background variable – A variable that can potentially affect a
response variable but is not a factor, e.g., time, lot, operator,
machine, etc.
• Nuisance variable – An unknown variable that can affect a
response variable. It is a background variable typically
unknown at the time of the experiment

37
 Definitions
• Level – A value or qualitative setting of a factor which would
be varied during an experiment
• Experimental unit – The smallest division of the experiment
that can receive a particular set of factor levels, e.g., a batch,
a single part, an hour of production, etc.
• Blocks – Groups of experimental units given similar treatment.
Usually, defined by background variables. The variation within
a block is expected to be less than the variation among blocks

38
 Tools For Experimentation
• Experimental pattern – the arrangement of factor
levels and experimental units in the design
• Planned grouping – forming blocks of experimental
units
• Randomization – the objective assignment of
combinations of factor levels to experimental units
• Replication – Repetition of experiments,
experimental units, measurements, etc.

39
 Experimental Pattern
• The experimental pattern defines how the factors and factor
levels will be studied
• There are a number of experimental patterns, such as
factorial, fractional factorial, and nested
• In factorial design, each possible combination of factors is
examined. So an experiment with 4 factors, each at 2 levels,
will have 24 or 16 factor level combinations to be studied
• In a fractional factorial design only a subset of the possible
combinations of factor levels are examined
• In a nested design the levels of a factor are examined within a
given level of another factor

40
 Full Factorial Design

Pressure 1 Pressure 2
Temp 1 Temp 2 Temp 1 Temp 2
Load 1
Batch 1 Load 2
Load 1
Batch 2 Load 2

41
 Nested Experimental Design

Machine 1 Machine 1

Head 1 Head 2 Head 1 Head 2

pt1 pt2 pt1 pt2 pt1 pt2 pt1 pt2

m1 m2 m1 m2 m1 m2 m1 m2 m1 m2 m1 m2 m1 m2 m1 m2

Four nested factors: Each factor at two nested levels

Machine Machine 1, Machine 2

Head within a machine Head 1, Head 2
Part within a head part 1, part 2
Measurement within a part m1, m2

The measurements are not in random order 42

 Planned Grouping
• How to control the background variables so
that the factor effects are not distorted
• How to use background variables to establish
a wide range of conditions
– We can either keep the background variables
constant
– Or we can measure them
– Or we can use planned grouping to set up blocks

43
 Planned Grouping
• Suppose we want to test the wear on different brands of tires.
A natural grouping/blocking would be a car where 4 brands
could be tested on a car
• Or if we wished to test the wear on the soles of shoes, a
natural grouping would be 2
• When there are several background variables, we can
combine them to form chunks using some extreme values of
the background variables to test their effect on the
measurement of interest

44
 Example of Chunk Variables
• Need to test material from 3 different suppliers. One factor at
3 levels
• Five Background Variables
Background Variable Levels
Machine No. 6 and No. 10
Operator Ram, Sunil, Gopal and Anil
Gauge G-103 and G-233
Saw blade 10 blades available
Time (days) Different days possible

45
4 Blocks can be formed

Block 1 Block 2 Block 3 Block 4

Machine No. 6 No. 10 No. 6 No. 10

Operator Ram Sunil Gopal Anil
Gauge G-103 G-233 G-103 G-233
Saw blade No. 1 No. 2 No. 3 No. 4
Day Day 1 Day 2 Day 3 Day 4

Experiment grouping
Test Block 1 Block 2 Block 3 Block 4
1 B B C A
2 A C A C
3 C A B B

A, B, and C are the 3 suppliers whose samples have to be tested

46
 Randomization and Replication
• Randomization
– The assignment of factor combinations to experimental units and
sequence of testing is randomized
– Helps reduce the variation due to nuisance variables from being
confused with variation due to factors
• Replication
– Repeated measurements of experimental units
– Multiple experimental units for each combination of factors
– Helps minimizing the effect of nuisance variables
– Ideally it is better to get few replications on different days than get all
replications on the same day
– No. of replication is constrained by the cost of the experiment

47
• End of Module 4

48
• Begin Module 5
– Example of Experiment with a Single Factor

49
 Example
The Xbar and R control charts for a critical dimension had been
maintained for 6 months. Recently tools were replaced and the
changes in averages were observed. It was suspected that it was
due to clamp pressure. So it was decided to test the effect of
clamp pressure on the variation in the dimension.

The current clamp pressure is 30 pounds. It was decided to test

the dimension at 3 other clamp pressures, at 20 pounds, 40
pounds, and 80 pounds. Each of these would be tested for one
day. The control chart for the 4 days of experimentation are
shown next.

50
7.3 Control Chart for Averages UCL

-7.3 LCL

Control Chart for Ranges

23 UCL

CL
10

0
1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
Day 1 Day 2 Day 3 Day 4
Pressure 30 Pressure 80 Pressure 20 Pressure 40 51
Summary of control charts:

Day 1: (prior to experiment) Both Xbar and R charts are in control

Day 2: (pressure 80 pounds) The Xbar chart is in control. In the R chart all
points below the CL. No other causes noted. Reduction in variation
likely due to increased clamp pressure

Day 3: (pressure 20 pounds) Variation seems to have increased with one

point out of control Likely due to reduction in pressure

Day 4: (pressure 40 pounds) Bothe charts in control. Slight decrease in

variation

52
 Pressure Average
Day lbs Range
3 20 16
1 30 10
4 40 8
2 80 5
We can conclude that clamp pressure reduces the average
range of the dimension. We should change the clamp pressure
To 80 lbs and monitor the situation to see if the effect is continued.
If the effect continues, we may wish to increase clamp pressure further

53
• End of Module 5

54