Afternoon Session 2 (Summary):1 I. Plackett-Burman designs Benefits. Relationship to fractional factorial designs. Confounding patterns.

Example 1: Direct mail credit card campaign (page 155 in Ledolter and Swersey). Brief summary of more advanced design issues: II. General Factorial Experiment with Factors at 3 or More Levels: Designs and Analysis of the Resulting Data. Example 2: Baking a cake III. Fractional Factorials Designs when Factors are at 3 or More Levels. Example 3: American Express, 18run design in five factors IV. Remarks on the confounding of fractional 3-Level and Mixed-Level Designs V. Concluding Remarks Computer software: JMP (SAS product) and Minitab

Relevant reading in Ledolter/Swersey: Chapters 6 and parts of Chapters 7 and 8.

1

Johannes Ledolter, WU-Wien, Spring 2009. Design of Experiments (Notes 5). Copyright 2009

I. Plackett-Burman Designs A full factorial design for k factors requires N = 2k runs (with k = 10 factors that would mean 1024 runs) In fractional factorial designs the number of runs N is a power of 2 (22 =4, 23 = 8, 24 = 16, 25 = 32, and so on) In Plackett-Burman designs the number of runs N is a multiple of 4 (N = 4, 8, 12, 16, 20, 24, and so on) Plackett-Burman designs fill in the gaps in the run sizes. Fractional designs introduce confounding among the estimated effects (no such thing as a free lunch), and this is also the case with Plackett-Burman designs. Used as screening designs. Study k = 11 factors in N = 12 runs. Study k = 19 factors in N = 20 runs Study k = 23 factors in N = 24 runs Orthogonal designs. Main-effects designs. Give us information about main effects, but we must assume that 2-factor interactions negligible. Resolution III designs Historical comment: World War II; weapon design involving 22 individual components

Example: 20-RUN PLACKETT-BURMAN DESIGN AND 19 FACTORS

Test Cell Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 A + + + + + + + + + + B + + + + + + + + + + C + + + + + + + + + + D + + + + + + + + + + E + + + + + + + + + + F + + + + + + + + + + G + + + + + + + + + + H + + + + + + + + + + Factors I + + + + + + + + + + J + + + + + + + + + + K + + + + + + + + + + L + + + + + + + + + + M + + + + + + + + + + N + + + + + + + + + + O + + + + + + + + + + P + + + + + + + + + + Q + + + + + + + + + + R + + + + + + + + + + S + + + + + + + + + + Response (of 5,000)

Rate 1.04 0.76 0.84 2.68 2.08 1.20 1.22 1.36 1.14 0.60 2.16 0.78 0.80 0.98 0.74 1.98 1.72 0.86 0.94 2.08

52 38 42 134 104 60 61 68 57 30 108 39 40 49 37 99 86 43 47 104

Design Construction:
N 12 20 24 1 + + + 2 + + + 3 + 4 + + 5 + + + 6 + + 7 + + 8 + 9 + 1 0 + + + Factors 1 1 1 1 2 3 - + - - + 1 4 + 1 5 1 6 1 7 + + 1 8 + 1 9 + 2 0 2 1 2 2 2 3

Above table gives us the first run; cyclically rearrange the levels until run N 1 = 19 is reached. Add a run with all minus levels.

EXAMPLE: Mail advertisement for credit card Objective: Increase sign-up rate 19 factors (2 levels each): Which are important? Which are not important?
Factor A B C D E F G H I J K L M N O P Q R S Envelope teaser Return address "Official" ink-stamp on envelope Postage Additional graphic on envelope Price graphic on letter Sticker Personalize letter copy Copy message Letter headline List of benefits Postscript on letter Signature Product selection Value of free gift Reply envelope Information on buckslip 2nd buckslip Interest rate (-) Control General offer Blind Yes Pre-printed Yes Small Yes No Targeted Headline 1 Standard layout Control version Manager Many High Control Product info No Low (+) New Idea Product-specific offer Add company name No Stamp No Large No Yes Generic Headline 2 Creative layout New P.S. Senior executive Few Low New style Free gift info Yes High

Plackett-Burman screening design: 19 factors in 20 runs Lists of addresses. 100,000 test mailing: 20 test cells sent to 5,000 people each Why 100,000? Expect response of about 1 percent. 80 percent chance to uncover factors that shift the response rate by 17 percent. 50-50 chance of detecting effects that shifted the average response by as little as 8.7 percent.
4

Data (see above) Estimation of main effects. Differences of averages at high and low levels. Displayed on chart Significance of main/interaction effects Effects are always a difference of two averages. Each average comes from N individual observations (which can be continuous or which can be 0/1 in the Bernouilli setup). In our example, N = 100,000.

StdDev(Effect) == 4 (1 )
N

StdError(Effect) =

4 (0.01298)(0.98702) = 0.00072 100,000

, or 0.072 percent

Dashed line:

1.96(0.072) = 0.141

Test Results: Main Effects (list 1)

S: Interest rate G: Sticker R: 2nd buckslip I: Copy message J: Letter headline F: Price graphic L: Letter postscript H: Personalization P: Reply envelope O: Value of free gift E: Additional graphic K: List of benefits Q: Info on buckslip B: Return address M: Signature A: Envelope teaser N: Product selection D: Postage C: Official stamp
0.0%

-0.864% -0.556% -0.304% +0.296% -0.192% -0.128% -0.116% +0.104% -0.096% +0.092% +0.092% +0.088% -0.080% +0.076% -0.064% +0.064% -0.052% +0.044% +0.032%
0.1% 0.2% 0.3% 0.4% 0.5% 0.6% 0.7% 0.8% 0.9% 1.0% Significant Effects (above line)

(Effect in percentage points)

Main-Effects Interpretation: Five significant factors S- or Low interest rate: Increasing the rate reduced response 0.864 percentage points. Fairly small change with large effect. Gain from the higher interest rate less than the lost profits. G- or Sticker: Sticker (G-) increases response by 0.556 percentage points. Revenue gain from a sticker was much greater than the cost of having it. R- or No 2nd buckslip: Adding another buckslip reduces number of buyers by 0.304 percentage points. Do not add. I+ or Generic copy message: Simple, direct offer increases response by 0.296 percentage points. Creative team was wrong J- or Letter headline #1: Best wording increases response by 0.192 percentage points. All good headlines are not equal.

Benefits of study Predicted response rate for implied best strategy (2.40%) represented a 15% predicted increase over the previous control 15% jump underestimates true benefits as test also shows which changes to avoid.
7

Additional Comments on Confounding Plackett-Burman designs are resolution III designs. Partial confounding in PB designs as compared to complete confounding in the fractional factorial designs. Each main effect in a PB design is confounded with many 2-factor interactions, but for most of these 2-factor interactions the correlation is rather small; certainly smaller than the correlation among a main effect and its associated 2-factor interactions in a resolution III fractional factorial design (which is either -1 or +1). For this particular 20-run Plackett-Burman design, the correlation between R and SG is given by -0.60. This is by far the largest correlation; the correlations between R and all other 2-factor interactions are considerably smaller ( 0.2 ).
Run Column R Column SG 1 +1 1 2 +1 +1 3 1 +1 4 1 +1 5 1 +1 6 1 1 7 +1 1 8 1 1 9 +1 1 10 1 +1 11 +1 +1 12 +1 1 13 +1 1 14 +1 1 15 1 +1 16 1 +1 17 +1 1 18 +1 1 19 1 +1 20 1 +1

This implies that it may not be the main effect of R (2nd buckslip) that is important, but the interaction between S and G. This simplifies the interpretation: S (interest rate), G (sticker) and the interaction between S and G as the driving forces behind the sign-up rate.

Interaction Plot for Rate

Data Means 2.2 2.0 1.8 1.6 Mean 1.4 1.2 1.0 0.8 0.6 -1 1

G= - 1: With Sticker

G = + 1: Without Sticker

Low

S: Interest Rate

High

II. General Factorial Experiment with Factors at 3 or More Levels: Designs and Analysis of the Resulting Data Example 2: 32 factorial experiment (baking a cake)

Baking temperature (A) and the baking time (B).

Three different temperatures: level 0 represents the time recommended by the instructions on the package, whereas levels -1 and +1 represent temperatures 10% below and 10% above the recommended level.

Three different times: level 0 representing the recommended time setting, and levels -1 and +1 the time settings 10% below and 10% above recommended level.

Three independent replications

Finished cakes are tasted by experts and rated on a 0-6 quality scale.
10

Data: Cake ratings

Temp Time Response -1 1 0, 0, 3 0 1 0, 2, 4 1 1 4, 5, 6 -1 0 2, 3, 4 0 0 3, 6, 6 1 0 1, 2, 3 -1 1 4, 5, 6 0 1 1, 3, 5 1 1 0, 1, 2

Graphical Analysis: Interaction plot of averages.

Interaction Plot for Rating

Data Means 5
Temp -1 0 1

Mean

1 -1 0 Time 1

Interpretation
11

Formal Analysis: ANOVA (Analysis of Variance):

Two-way ANOVA: Rating versus Temp, Time Source Temp Time Interaction Error Total S = 1.453 DF 2 2 4 18 26 SS 2 2 62 38 104 MS 1.0000 1.0000 15.5000 2.1111 F 0.47 0.47 7.34 P 0.630 0.630 0.001

R-Sq = 63.46%

R-Sq(adj) = 47.22%

Interaction component is statistically significant. We stop the formal analysis and interpret the effects through the interaction diagram.

12

III. Fractional Factorials Designs when Factors are at 3 or More Levels

Example: AmEx Multivariate Case Study 5 factors: A: Segment (2 levels): B: Product (3 levels): C: Duration (3 levels): D: BTRate (3 levels): E: GoToRate (3 levels):

LowRevenue, HighRevenue Blue, Cash, Sky 0 for 6, 0 for 12, 0 for 15 3.99, 4.99, 5.99 P + 3.99, P + 6.99, P + 9.99

For a full factorial, one would need (2)(3)(3)(3)(3) = 162 runs (level-combinations).

Way too many runs. No need to be able to estimate highorder interactions (interactions of order 3 and higher; such as A*B*C, A*B*C*D, and A*B*C*D*E).

Hence we are looking for fractions. The design given below, consisting of 18 runs, is one such fraction.

13

Data:
1 2 3 Test 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 S egment Low Rev High Rev Product Blue Cash Sky Product Blue Blue Blue Cash Cash Cash Sky Sky Sky Blue Blue Blue Cash Cash Cash Sky Sky Sky Intro 0% f 6m 0% f 12m 0% f 15m Intro 0% f 6m 0% f 12m 0% f 15m 0% f 6m 0% f 12m 0% f 15m 0% f 6m 0% f 12m 0% f 15m 0% f 6m 0% f 12m 0% f 15m 0% f 6m 0% f 12m 0% f 15m 0% f 6m 0% f 12m 0% f 15m BT 3.99 LOB w/f 4.99 LOB w/f 5.99 LOB BT 3.99 LOB w/f 3.99 LOB w/f 4.99 LOB w/f 5.99 LOB 4.99 LOB w/f 4.99 LOB w/f 5.99 LOB 3.99 LOB w/f 5.99 LOB 4.99 LOB w/f 5.99 LOB 5.99 LOB 3.99 LOB w/f 5.99 LOB 3.99 LOB w/f 4.99 LOB w/f 4.99 LOB w/f 3.99 LOB w/f Merch P+3.99 P+6.99 P+9.99 Merch P+3.99 P+9.99 P+9.99 P+3.99 P+6.99 P+3.99 P+6.99 P+6.99 P+9.99 P+6.99 P+3.99 P+6.99 P+9.99 P+9.99 P+6.99 P+9.99 P+3.99 P+3.99

S egment Low Rev Low Rev Low Rev Low Rev Low Rev Low Rev Low Rev Low Rev Low Rev High Rev High Rev High Rev High Rev High Rev High Rev High Rev High Rev High Rev

Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

A B C D E LettersSent NuAppr %Appr Segment Product Duration BTRate GoToRate -1 0 0 0 0 313039 244 0.077946 -1 0 1 0 2 308983 276 0.089325 -1 0 2 1 2 311986 282 0.090389 -1 1 0 2 1 316433 231 0.073001 -1 1 1 0 1 329002 311 0.094528 -1 1 2 2 2 317774 262 0.082449 -1 2 0 2 0 312444 244 0.078094 -1 2 1 1 1 309410 289 0.093404 -1 2 2 1 0 308508 354 0.114746 1 0 0 1 1 83385 99 0.118726 1 0 1 2 0 85956 157 0.182652 1 0 2 2 1 82631 139 0.168218 1 1 0 1 2 85321 50 0.058602 1 1 1 1 0 85558 132 0.154281 1 1 2 0 0 84458 167 0.197731 1 2 0 0 2 83268 138 0.165730 1 2 1 2 2 79798 111 0.139101 1 2 2 0 1 83052 217 0.261282

14

What is special about this design?

Each factor level combination of any two factors is studied with the same number of runs. We call a design with such property an orthogonal design.

Fortunate consequences of orthogonality:

ANOVA table gives us an additive sum of squared decomposition. Sums of squares of the main effects of the five factors add up to the total sum of squares. Orthogonal design results in factor estimates with the smallest standard error.

Unfortunate consequence of this particular design:

Confounds main effects with 2-factor interactions. Resolution III design. If there are 2-factor interactions, then the main effect estimates are compromised.

15

Analysis of main effects: ANOVA

General Linear Model: %Approved versus Segment, Product, ...
Factor Segment Product Duration BTRate GoToRate Type fixed fixed fixed fixed fixed Levels 2 3 3 3 3 Values -1, 1 0, 1, 2 0, 1, 2 0, 1, 2 0, 1, 2

Analysis of Variance for %Approved, using Adjusted SS for Tests Source Segment Product Duration BTRate GoToRate Error Total DF 1 2 2 2 2 8 17 Seq SS 0.0236490 0.0031593 0.0097985 0.0056130 0.0036698 0.0037538 0.0496434 Adj SS 0.0236490 0.0031593 0.0097985 0.0056130 0.0036698 0.0037538 Adj MS 0.0236490 0.0015797 0.0048993 0.0028065 0.0018349 0.0004692 F 50.40 3.37 10.44 5.98 3.91 P 0.000 0.087 0.006 0.026 0.065

S = 0.0216616

R-Sq = 92.44%

R-Sq(adj) = 83.93%

Main-effects plot and interpretation:

Main Effects Plot for %Approved
Data Means
Segment 0.16 0.14 0.12 0.10 Product Duration

Mean

-1 BTRate 0.16 0.14 0.12 0.10 0 1

1 GoToRate

16

Additional comments: Designs that lead to less confounding

2-level designs: For 5 factors, start with a 2-level design of the form
2 51

with

I = ABCDE. This takes 16 runs, and one is able to estimate all 2-factor interactions. It is likely that fewer than 5 factors will be active (effect sparcity).

Follow-up designs. Follow-up this first experiment with these fewer factors. At that time one can consider 3-level designs if one so wishes. For example, with three remaining factors (one factor at 2 levels and two factors at three levels) the full factorial consists of 18 runs. This would also give us some confirmative information about the results of the first experiment. One would need a total (initial experiment and its follow-up) of 16 + 18 = 34 runs.

Other possibility of resolution V design for the five factors in 54 runs

17

IV. Remarks on the confounding of fractional 3-Level and Mixed-Level Designs Fractional 3k-p Designs: Number of runs in 3k factorial experiments grows rapidly with the number of factors k: 33 = 27; 34 = 81; etc. 3k-p design uses fewer runs, but confounds effects. Special cases:
The 34-2 Design A B C D -1 -1 -1 -1 0 -1 0 1 1 -1 1 0 -1 0 0 0 0 0 1 -1 1 0 -1 1 -1 1 1 1 0 1 -1 0 1 1 0 -1 The 33-1 Design A B C -1 -1 -1 0 -1 0 1 -1 1 -1 0 0 0 0 1 1 0 -1 -1 1 1 0 1 -1 1 1 0

Resolution III designs. Main-effects designs as they confound main effects with 2-factor interactions.

18

Design in 27 runs: 34-1 Design

Factors B -1 -1 -1 0 0 0 1 1 1 -1 -1 -1 0 0 0 1 1 1 -1 -1 -1 0 0 0 1 1 1

A -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1

C -1 -1 -1 -1 -1 -1 -1 -1 -1 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1

D -1 0 1 0 1 -1 1 -1 0 0 1 -1 1 -1 0 -1 0 1 1 -1 0 -1 0 1 0 1 -1

Resolution IV design. Allows us to estimate the four main effects; 2-factor interactions are confounded.

19

Mixed-Level Designs

Resolution V mixed-level designs allow the estimation of all main effects and all 2-factor interactions

Examples 2133 in 27 runs (25 parameters)

Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 A 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 Factors B 0 0 0 1 1 1 2 2 2 0 0 0 1 1 1 2 2 2 0 0 0 1 1 1 2 2 2 C 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 D 1 0 0 0 0 1 0 1 0 0 0 1 0 1 0 1 0 0 0 1 0 1 0 0 0 0 1

20

2134 design in 54 runs (42 parameters)

Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 A -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 B -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 C -1 -1 -1 0 0 0 1 1 1 -1 -1 -1 0 0 0 1 1 1 -1 -1 -1 0 0 0 1 1 1 D -1 -1 -1 -1 -1 -1 -1 -1 -1 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 E -1 0 1 0 1 -1 1 -1 0 0 1 -1 1 -1 0 -1 0 1 1 -1 0 -1 0 1 0 1 -1 Run 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 A 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 B -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 C -1 -1 -1 0 0 0 1 1 1 -1 -1 -1 0 0 0 1 1 1 -1 -1 -1 0 0 0 1 1 1 D -1 -1 -1 -1 -1 -1 -1 -1 -1 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 E 1 0 -1 0 -1 1 -1 1 0 0 -1 1 -1 1 0 1 0 -1 -1 1 0 1 0 -1 0 -1 1

21

2331 design in 16 runs (15 parameters)

Factors B 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1

Run 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

A 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1

C 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1

D 0 1 1 2 1 2 0 1 1 2 0 1 0 1 1 2

22

V. Concluding Remarks

2-level factorial and fractional factorial designs:

Useful search designs if one does not know which of many factors have an effect on the response. Can not learn about nonlinear effects (curvature).

3-level and mixed-level (fractional) factorial designs:

Good option after having reduced the (large) number of possible factors to a smaller set of the vital few. Confounding patterns complicated to untangle.

Computer programs:

Minitab and JMP. Both packages cover most designs discussed in this course.

23