Statistics 203: Introduction to Regression and Analysis of Variance

Fixed vs. Random Effects
Jonathan Taylor

- p. 1/19

2/19 . fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating s s s s Random effects.p. Two-way mixed & random effects ANOVA. 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure . Sattherwaite’s procedure.Today’s class q Today’s class q Two-way ANOVA q Random vs. One-way random effects ANOVA.

j )-“cell”: Yijk = µ + αi + βj + (αβ )ij + εijk . 1 ≤ j ≤ m. m groups ins second and nij samples in (i. 1 ≤ j ≤ m. 1 ≤ i ≤ r. 1 ≤ i ≤ r r x i=1 (αβ )ij = 0. 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure s Constraints: r x i=1 αi = 0 m x j =1 βj = 0 m x j =1 (αβ )ij = 0. 3/19 . fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating s s Second generalization: more than one grouping variable. σ 2 ). . εijk ∼ N (0. Two-way ANOVA model: observations: (Yijk ).p. 1 ≤ k ≤ nij : r groups in first grouping variable.Two-way ANOVA q Today’s class q Two-way ANOVA q Random vs.

In this case. fixed effects q Today’s class q Two-way ANOVA q Random vs. etc. where more than one (identical) measurement is taken on the same individual. the “group” effect αi is best thought of as random because we only sample a subset of the entire population of subjects. long duration. Simplest example: repeated measures.Random vs. In some designs.p. 4/19 . . the categorical variable is “subject”. the categorical variables are well-defined categories: below average fitness. fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating s s s 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure s In ANOVA examples we have seen so far.

Example: if surveying students on different campuses. q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure . “campus” may be a random effect. fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating s s s 2 σµ A “group” effect is random if we can think of the levels we observe in that group to be samples from a larger population. “center” might be thought of as random.p. 5/19 .When to use random effects? q Today’s class q Two-way ANOVA q Random vs. Example: if collecting data from different medical centers.

Questions of interest: what is the “grand mean” sodium content? How much variability is there from brand to brand? “Individuals” in this case are brands.Example: sodium content in beer q Today’s class q Two-way ANOVA q Random vs. 6/19 .p. repeated measures are the 8 bottles. researchers recorded the sodium content of 8 12 ounce bottles. fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating s s s 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure s How much sodium is there in North American beer? How much does this vary by brand? Observations: for 6 brands of beer. .

σµ . Alternatively. 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure s We might be interested in the population mean.One-way random effects model q Today’s class q Two-way ANOVA q Random vs. is it zero? subjects. 1 ≤ i ≤ r. µ· : CIs. 1 ≤ j ≤ n εij ∼ N (0. Yij ∼ µ· + αi + εij . σµ ). is it zero? etc.p. 1 ≤ j ≤ n 2 αi ∼ N (0. σ 2 ). 7/19 . we might be interested in the variability across 2 : CIs. 1 ≤ i ≤ r. fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating s s s s s Suppose we take n identical measurements from r subjects. 1 ≤ i ≤ r.

the observations are no longer independent (even if ε’s are independent). 8/19 .Implications for model q Today’s class q Two-way ANOVA q Random vs. the only parameter to estimate is σ 2 because the covariance matrix is σ 2 I . Yi j ) = σµ s 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure s In more complicated mixed effects models.i + σ 2 δj. In fact 2 δi. In ordinary least squares regression.j . .p. Cov(Yij . but in the covariance as well. this makes MLE more complicated: not only are there parameters in the mean. fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating s In random effects model.

9/19 . fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating Source Treatments Error s s 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure s Only change here is the expectation of SST R which reflects randomness of αi ’s.One-way random ANOVA table q Today’s class q Two-way ANOVA q Random vs. it is easy to see that M ST R ∼ Fr−1.(n−1)r . 2 Under H0 : σµ = 0. M SE SS ”2 “ Pr n SST R = Y Y − ·· i· i=1 Pr Pn 2 SSE = i=1 j =1 (Yij − Y i· ) d f r−1 (n − 1)r E (M S ) 2 σ 2 + nσµ σ2 . ANOVA table is still useful to setup tests: the same F statistics for fixed or random will work here.p.

10/19 . Var(Y ·· ) = rn s Therefore.p. µr ). . . . fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating s We know that E (Y ·· ) = µ· . and can show that 2 nσµ + σ2 . so that Y i· → µi . .Inference for µ· q Today’s class q Two-way ANOVA q Random vs. Y ·· − µ· SST R (r −1)rn 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure ∼ tr−1 s s s Why r − 1 degrees of freedom? Imagine we could record an infinite number of observations for each individual. To learn anything about µ· we still only have r observations (µ1 . . Sampling more within an individual cannot narrow the CI for µ· .

fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating σ 2 µ s From the ANOVA table 2 σµ E (SST R/(r − 1)) − E (SSE/((n − 1)r)) .p.2 Estimating σµ q Today’s class q Two-way ANOVA q Random vs. = n SST R/(r − 1) − SSE/((n − 1)r) = n s Natural estimate: 2 Sµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure s Problem: this estimate can be negative! One of the difficulties in random effects model. . 11/19 .

Company wants to get an idea of daily productivity.Example: productivity study q Today’s class q Two-way ANOVA q Random vs.p. and how it depends on which machine an employee uses. As these employees are not all employees. having each employee work on each machine for a total of n days. and these machines are not all machines it makes sense to think of both the effects of machine and employees (and interactions) as random. Study: take m employees and r machines. fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating s s s 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure s Imagine a study on the productivity of employees in a large manufacturing company. . 12/19 .

fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating s s s s s s Yijk ∼ µ·· + αi + βj + (αβ )ij + εij . σα ). 2 βj ∼ N (0. 2 (αβ )ij ∼ N (0. 1 ≤ j ≤ m. 1 ≤ i ≤ r. σβ ).Two-way random effects model q Today’s class q Two-way ANOVA q Random vs. 1 ≤ i ≤ r. 1 ≤ k ≤ n 2 αi ∼ N (0. σ 2 ). 1 ≤ i ≤ r. 13/19 . . 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure Cov(Yijk . 1 ≤ j ≤ m. 1 ≤ k≤n εijk ∼ N (0. Yi j k 2 2 2 ) = δii σα + δjj σβ + δii δjj σαβ + δii δjj δkk σ 2 . 1 ≤ i ≤ r. 1 ≤ j ≤ m.p. σαβ ). 1 ≤ j ≤ m.

p. 2 To test H0 : σαβ = 0 use SSAB and SSE . q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure . fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating SS ”2 “ P SSA = nm r Y − Y ··· i·· i=1 ”2 Pm “ SSB = nr j =1 Y ·j · − Y ··· ”2 P Pm “ Y Y − Y + Y − SSAB = n r ··· i·· ·j · ij · i=1 j =1 Pm Pn Pr 2 SSE = j =1 i=1 k=1 (Yijk − Y ij · ) d f r−1 m−1 (m − 1)(r − 1) (n − 1)ab E (SS ) 2 + nσ 2 σ 2 + nmσα αβ 2 2 2 σ + nrσβ + nσαβ 2 σ 2 + nσαβ σ2 s s 2 σµ 2 To test H0 : σα = 0 use SSA and SSAB . 14/19 .ANOVA tables: Two-way (random) q Today’s class q Two-way ANOVA q Random vs.

Mixed effects model q Today’s class q Two-way ANOVA q Random vs. For instance. Then. . we might think of a model in which we have a fixed effect for “brewing technique” and a random effect for beer.p. some factors can be thought of as fixed. 15/19 . we might have a study of the effect of a standard part of the brewing process on sodium levels in the beer example. fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating σ 2 µ s s s q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure In some studies. others random.

1 ≤ i ≤ r. 16/19 . σα ). 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure Constraints: m x j =1 βj = 0 r x i=1 (αβ )ij = 0. σ 2 ). βj . 1 ≤ j ≤ m. Yi j δjj 2 σβ + δii k s )= 1 2 − (1 − δii ) m σαβ + δii δkk σ 2 m−1 2 m σαβ . 1 ≤ i ≤ r.p. 1 ≤ i ≤ r. 2 (αβ )ij ∼ N (0. fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating s s s s s s Yijk ∼ µ·· + αi + βj + (αβ )ij + εij . x Cov ((αβ )ij . 1 ≤ j ≤ m are constants. 1 ≤ i ≤ r. 1 ≤ j ≤ m. 1 ≤ k≤n εijk ∼ N (0. 1 ≤ k ≤ n 2 αi ∼ N (0. 1 ≤ i ≤ r. (αβ )i j ) = −σ 2 /m αβ Cov(Yijk . 1 ≤ j ≤ m. (m − 1)σαβ /m).Two-way mixed effects model q Today’s class q Two-way ANOVA q Random vs.

. 17/19 . fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating SS SSA SSB SSAB Pm Pn Pr 2 SSE = j =1 i=1 k=1 (Yijk − Y ij · ) d f r−1 m−1 (m − 1)(r − 1) (n − 1)ab E (M S ) 2 σ 2 + nmσα Pm 2 j =1 βi 2 + nσαβ σ 2 + nr m−1 2 σ 2 + nσαβ σ2 s s s 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure 2 To test H0 : σα = 0 use SSA and SSE .ANOVA tables: Two-way (mixed) q Today’s class q Two-way ANOVA q Random vs. 2 To test H0 : σαβ use SSAB and SSE . To test H0 : β1 = · · · = βm = 0 use SSB and SSAB .p.

fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating s 2 Consider estimating σβ in the two-way random effects ANOVA. 2 To form a confidence interval for σ β we need to know distribution of a linear combination of M S · s. at least approximately.p. 18/19 . .Confidence intervals for variances q Today’s class q Two-way ANOVA q Random vs. A natural estimate is 2 = nr(M SB − M SAB ). σβ s s s 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure What about CI? A linear combination of χ2 – but not χ2 .

2 k 2 2 i=1 ci M Si /dfi where dfi are the degrees of freedom of the i-th M S .α/2 .p.1−α/2 dfT × L LU = 2 χdfT . 19/19 . fixed effects q When to use random effects? q Example: sodium content in beer q One-way random effects model q Implications for model q One-way random ANOVA table q Inference for µ· q Estimating s Given k independent M S ·’s k L∼ i=1 s ci M S i Then dfT L E(L) where dfT = k i=1 ci M Si 2 σµ q Example: productivity study q Two-way random effects model q ANOVA tables: Two-way (random) q Mixed effects model q Two-way mixed effects model q ANOVA tables: Two-way (mixed) q Confidence intervals for variances q Sattherwaite’s procedure “ ∼ χ2 dfT . LL = 2 χdfT . s (1 − α) · 100% CI for E(L): dfT × L .Sattherwaite’s procedure q Today’s class q Two-way ANOVA q Random vs.