Analysis of Variance (ANOVA)

Apr 12 2023

1
Information about the final exam
• Sunday April 23, 7:00 pm start time
• Location: CIRS 1250
• Duration: 2 hours 15 min (aka 135 minutes)

• You will need: calculator, pencil, eraser

• We will provide: all statistical tables and formula sheet,
scantron and exam booklet

2
final exam: Office Hours on Canvas
Official office hours end this Fri

See Canvas for schedule of office hours for next

week. These will be ‘open room’ office hours

If you are struggling PLEASE reach out to myself

or a TA

3
Learning Objectives
• Identify experimental designs for which ANOVA is the
appropriate analysis and discuss the advantages of running
them
• Conduct ANOVAs and apply the steps of hypothesis testing to
ANOVAs (form hypotheses, make decisions about Ho, form
conclusions)
• Conduct post hoc tests after ANOVAs where appropriate
• Calculate and interpret effect sizes for ANOVAs

4
How many conditions/groups/means?
1 sample tested 2 conditions tested More than 2
against μ against each other conditions
tested against
each other
σ is known: Independent samples:
z-test ind samples t-test
Analysis of
σ is not known: Correlated samples: variance
single sample t-test paired samples t-test (ANOVA)

5
Analyzing which variance?
• Variance of DV can be partitioned into some amount that can
be explained by the IV, and some amount that cannot be
explained by the IV

What can account for where each square is on the page?

6
Bobo Doll Experiment
How much of the differences between child
aggression can be explained by which video the
children watched?

Condition 1:
Model rewarded for aggressive behavior
Dependent
Condition 2: variable:
Model punished for aggressive behavior Child
aggression
Condition 3:
Model faced no consequences for aggressive
behavior
7
ANOVAs are great
• No restriction on the number of means tested

8
ANOVAs are great
• Can include multiple independent variables
• Interactions between independent variables

9
 ANOVAs are great
• Why not do multiple t-tests? Type I error problems!

Condition 1:
Model rewarded for aggressive behavior
α = 0.05

Condition 2: α = 0.05
Model punished for aggressive behavior

α = 0.05 Condition 3:
Model faced no consequences for aggressive
behavior

t-tests: 0.05 + 0.05 + 0.05 = 0.15 ANOVA: 0.05

10
How do ANOVAs work?
(1) Are there any differences at all between the levels?
(2) If yes, where are the differences?

Groups that Condition 1

make up the Independent variable
factor are Condition 2 is called a FACTOR
called levels
of the factor Condition 3

11
 How do ANOVAs work?
(1) Are there any differences at all between the levels?

Condition 2 Condition 3

Condition 1

0 0.25 0.5 0.75 1 1.25 1.5 1.75 2 2.25 2.5 2.75 3 3.25 3.5 3.75 4 4.25 4.5 4.75 5 5.25 5.5

Number of Aggressive Behaviors

Means might be different but could these

differences be due to chance?
12
 How do ANOVAs work?
• Compare variability due to factor (systematic) against
variability within factor (error) in the form of a ratio

Condition 2 Condition 3

Condition 1

0 0.25 0.5 0.75 1 1.25 1.5 1.75 2 2.25 2.5 2.75 3 3.25 3.5 3.75 4 4.25 4.5 4.75 5 5.25 5.5

Number of Aggressive Behaviors

High ratio -> higher chance that factor has an effect

13
Variance against variance
• Which experiment has a larger ratio of systematic
variance relative to error variance?

C
14
Variance against variance
• Which experiment has a larger ratio of systematic
variance relative to error variance?

Means are close together

A
Variability within levels is high

Means are far apart

B
Variability within levels is high

C *** Means are far apart

Variability within levels is low
15
F-ratios
Variance due to difference between
factor level means
signal
noise
Variance between groups
F=
Variance within groups

Variance due to differences between

subjects within each factor level

16
ANOVA Hypotheses
Ho: μ1 = μ2 = μ3 .... = μk (all of the means are equal)
Ha: At least one of the population means is not equal to
the others

• Hypotheses for ANOVAs are always non-directional

• If we reject Ho, we won’t be able to conclude *which*
group(s) are different from the rest
17
 ANOVA hypotheses
Which of these group
A distributions probably has
the highest F-ratio?

Which of these group

distributions is most likely to
lead to rejecting Ho?
which case would you be more convinced that
e treatment had a real effect?
18
F-distribution
F=
Variance between groups
Variance within groups

• F-values are ratios (fractions) of variances

• Variances, like means, also have sampling distributions
• Therefore, we need to use sampling distributions (again!)
19
F-distribution
• Like t-distribution, exact shape of the F-distribution depends
on degrees of freedom (df)

dfbetween = K -1
Variance between groups dfbetween = Number of levels - 1
Variance within groups

dfwithin = df1 + df2 + df3 +...dfk

df1 = n1 -1
dfwithin = Number of subjects - 1
20
Participation Question

• You have 3 groups/conditions in your experiment.

Each group has 12 participants. What are your
degrees of freedom between?

A. 2
dfbetween = K -1
B. 3
C. 4

21
Participation Question

• You have 3 groups/conditions in your experiment.

Each group has 12 participants. What are your
degrees of freedom between?

A. 2
dfbetween = K -1
B. 3
C. 4
dfbetween = 3-1= 2

22
Participation Question

• You have 3 groups/conditions in your experiment.

Each group has 12 participants. What are your
degrees of freedom within?

A. 9 dfwithin = df1 + df2 + df3 +...dfk

B. 12
C. 33
D. 36

23
Participation Question

• You have 3 groups/conditions in your experiment.

Each group has 12 participants. What are your
degrees of freedom within?

A. 9 dfwithin = df1 + df2 + df3 +...dfk

B. 12
C. 33
dfwithin = (12 -1)+(12 -1)+ (12 -1) = 33
D. 36

24
F-distribution degrees of freedom

Condition 1 10 participants

Condition 2 6 participants

Condition 3 9 participants

What are the degrees of freedom in this experiment?

dfbetween = K -1 = 3-1 = 2

dfwithin = df1 + df2 + df3 +...dfk = 9 + 5+8 = 22

25
 Table F gives you critical values
F-distribution for the F-distribution

Each column: dfbetween

26
 Table F gives you critical values
F-distribution for the F-distribution

Each group of rows: dfwithin

27
 Table F gives you critical values
F-distribution for the F-distribution

Each row within a group: alpha level (α)

➢ Non-bold text: alpha = 0.05
➢ Bold text: alpha = 0.01

28
 Table F gives you critical values
F-distribution for the F-distribution

Dfbetween = 2
Dfwithin = 22 If α = 0.05, Fcrit = 3.44

29
F-distribution

• All F-values are positive

• The F-distribution is positively skewed
• When dfwithin is higher (i.e., more subjects in the
study), the critical value is lower
30
F-distributions
• In ANOVAs, we only consider
the right tail of the
distribution when
determining the critical value
Variance between groups
F= Variance within groups

• If there is a real difference between groups, then F

will be bigger! No difference = F will be smaller
• Even though we only look at one tail, ANOVAs are
non-directional tests
31
ANOVA assumption of homogeneity of variance
• The populations have similar variances

Condition 2 Condition 3

Condition 1

• Do the groups of subjects come from populations

that have approximately the same variability?

32
ANOVA – mean squares

MSbetween Variance between groups

F=
MS within Variance within groups

In ANOVA terminology, the measure of

variability is called mean square (MS)

SSbetween SSwithin
MSbetween = MSwithin =
dfbetween dfwithin

33
ANOVA – calculations
• Let’s imagine we did a version of the Bobo doll study
with only 15 participants total
Reward Nothing Punishment
7 4 0
9 3 1
5 6 3
8 3 1
6 4 0

7 4 1 Mean

34
ANOVA steps
Decide cut-off (alpha) Decide cut-off (alpha)

Determine (using df) tcrit Determine (using dfs) Fcrit

that corresponds to alpha that corresponds to alpha

Calculate tobt Calculate Fobt

Compare tobt to tcrit Compare Fobt to Fcrit

Make decision about Ho Make decision about Ho

35
ANOVA – Bobo Doll Study
Decide cut-off (alpha)
ANOVAs are always non-directional, # tails not applicable
Let’s use α = 0.05

Determine (using dfs) Fcrit that corresponds to alpha

Reward Nothing Punishment
7 4 0
dfbetween = K -1 9 3 1

= 3-1 = 2 5 6 3
8 3 1
6 4 0

36
ANOVA – Bobo Doll Study
Reward Nothing Punishment
dfwithin = df1 + df2 + df3 +...dfk 7 4 0
9 3 1
= (5-1)+ (5-1)+ (5-1)
5 6 3
=12 8 3 1
6 4 0

37
Finding Fcrit

If α = 0.05, what is Fcrit?

dfbetween = K -1 dfwithin = df1 + df2 + df3 +...dfk

= 3-1 = 2 = (5-1)+ (5-1)+ (5-1) =12

Dfbetween = 2
Dfwithin = 12

If α = 0.05, Fcrit = 3.88

38
ANOVA – Bobo Doll Study
Determine (using dfs) Fcrit that corresponds to alpha

Retain Ho Reject Ho

Fcrit = 3.88
39
ANOVA calculations
MSbetween
F=
MSwithin

SSbetween SSwithin
MSbetween = MSwithin =
dfbetween dfwithin

SSbetween = ån(X - GM ) SSwithin = å(X - X)

2 2

* GM = Grand Mean
40
ANOVA – calculating SSb and SSw
(1) Calculate the grand mean (GM) and the group means

GM =
å X Reward
7
Nothing
4
Punishment
0
N 9 3 1
60
=4
5 6 3
= 8 3 1
15
6 4 0

Group Means 7 4 1

41
ANOVA – calculating SSb and SSw
(2) Calculate SSbetween

SSbetween = ån(X - GM )
2

= 5(7 - 4)2 + 5(4 - 4)2 + 5(1- 4)2

= 5(32 )+ 5(02 )+ 5(32 )
Grand Mean = 4
Reward mean = 7
= 45+ 0 + 45 Nothing mean = 4
= 90 Punishment mean = 1
All the n’s are 5

42
ANOVA – calculating SSb and SSw
(3) Calculate SSwithin Reward Nothing Punishment
7 4 0
SSwithin = å(X - X)
2
9 3 1
5 6 3
= 22 8 3 1
6 4 0

7 4 1

= (7 - 7) + (9 - 7) + (5- 7) + (8- 7) + (6 - 7)
2 2 2 2 2

+(4 - 4)2 + (3- 4)2 + (6 - 4)2 + (3- 4)2 + (4 - 4)2

+(0 -1) + (1-1) + (3-1) + (1-1) + (0 -1)
2 2 2 2 2

43
ANOVA – calculations
MSbetween
F=
MSwithin

SSbetween SSwithin
MSbetween = MSwithin =
dfbetween dfwithin

SSbetween = ån(X - GM ) SSwithin = å(X - X)

2 2

✓ ✓
44
Participation Question

• If I want to reject the null hypothesis, I want

______ to be larger than ________.

A. MSbetween; MSwithin MSbetween

F=
B. MSwithin; MSbetween MSwithin

45
Participation Question

• If I want to reject the null hypothesis, I want

______ to be larger than ________.

A. MSbetween; MSwithin MSbetween

F=
B. MSwithin; MSbetween MSwithin

46
ANOVA – calculating MSb and MSw
(4) Calculate MSbetween and MSwithin

SSbetween = 90 df between = 2
SSwithin = 22 df within = 12

SSbetween SSwithin
MSbetween = MSwithin =
dfbetween dfwithin

90 22
= =
2 12
= 45 =1.83
47
ANOVA – calculations
MSbetween
F=
MSwithin

SSbetween SSwithin
MSbetween = MSwithin =
dfbetween dfwithin
✓ ✓
SSbetween = ån(X - GM ) SSwithin = å(X - X)
2 2

✓ ✓
48
ANOVA – calculating Fobt
(5) Calculate Fobt MSbetween = 45
MSwithin = 1.83
MSbetween 45
F= = = 24.55
MSwithin 1.83
Compare Fobt to Fcrit

Fobt = 24.55 > Fcrit = 3.88

49
Participation Question

• What is the appropriate decision regarding the null

hypothesis?

(A) Reject Ho
(B) Retain Ho

Compare Fobt to Fcrit

Fobt = 24.55 > Fcrit = 3.88

50
ANOVA – Bobo Doll Study
Make a decision about Ho

• Reject Ho!
• Ho: μ1 = μ2 = μ3 .... = μk (all of the means are
equal)

• Conclude that at least one of the means are

different from the others
• Cannot yet conclude which particular means are
different from each other

51
Post-Hoc Tests
Your ANOVA is significant – now what?
Post Hoc = after the event / after the fact

8
6
4
2
0

• Tukey’s HSD (honestly significant difference, aka Qobt)

• Compares pairs of group means to test if they are
significantly different
• Only done after significant ANOVA

52
Tukey’s HSD 8
6

Let’s compare reward and 4

nothing (7 versus 4) 2
0
7 1 4

X1 - X 2 MS within 1.83
HSD = sM = = = 0.605
sM n 5
7- 4
=
0.605 If you have unequal group sizes, there is
a special formula you can use (but will
= 4.96 not need for this class)

HSD = Qobt
53
Tukey’s HSD
Compare HSD/Qobt value to critical q value (use Table G)

Choose the column

that matches the
number of means in
your ANOVA analysis
54
Tukey’s HSD
Compare HSD/Qobt value to critical q value (use Table G)

Choose the group of

rows that matches the
df for MSwithin
55
Tukey’s HSD
Compare HSD/Qobt value to critical q value (use Table G)

Choose the row that

matches your alpha
level
56
Tukey’s HSD

Number of levels (K) = 3

df within = 12
alpha (α) = 0.05

Compare HSD to qcrit

HSD = 4.96 > qcrit = 3.77

• You do not need to worry about the signs (pos/neg)

• Compare the absolute values of HSD/Qobt and Qcrit

57
Tukey’s HSD

Number of levels (K) = 3

df within = 12
alpha (α) = 0.05

Compare HSD to qcrit

HSD = 4.96 > qcrit = 3.77

• Therefore, the mean for the reward condition is significantly

different from the mean for the nothing condition
• Usual procedure: repeat for all pairs of means
58
Effect Size (eta squared)
• η2 is a common measure of effect size for ANOVAs
SSbetween from ANOVA calculation
SSbetween
h =
2

SStotal
SSbetween + SSwithin

• Ranges from 0 to 1.00

• Represents the proportion of variance in the DV that is
accounted for by the IV

59
Participation Question

• Conceptually, η2 is most similar to...

(A) r (correlation coefficient)

(B) r2 (coefficient of determination)
(C) β (standardized regression coefficient)
(D) p (probability value)

60
Participation Question

• Conceptually, η2 is most similar to...

(A) r (correlation coefficient)

(B) r2 (coefficient of determination)
(C) β (standardized regression coefficient)
(D) p (probability value)

61
Suggested Textbook Questions

• Problems at the end of Ch. 15 of the textbook

(p.436-440): 10-11, 19, 20b, 21b, 22a, 23a, 24a, 25a-
b, 26a, 27

• (Not graded)

62