Analysis of Variance (ANOVA) - Summary

Consider the following scenarios:

1. An environmentalist is interested to know if the average amount of pollution varies in several

bodies of water

2. A sociologist is interested in knowing if the average income in a locality varies according to the
upbringing of the residents

3. A consumer looking for a new car is comparing the average gas mileage of several models

These are common scenarios that involve comparing the averages of a specific attribute between several
groups.

When we have only two such groups, we know that the two-sample z-test or t-test can help us arrive at a
statistically-sound conclusion. But what if we have more than 2 groups? One can compare two groups at
a time and repeat the same to cover all the groups one pair at a time. But as there is an error involved
with each test, the cumulative error for multiple tests will be high. Also, conducting multiple tests is
time-consuming when we have many groups.

For such cases, statisticians have developed hypothesis tests that compare the averages between more
than two groups. The group of tests is broadly called Analysis of Variance, or commonly, ANOVA.

ANOVA

ANOVA refers to a set of statistical tests that lets us compare the averages (means) between several
groups, by analyzing the variation within each group and the variation between the groups.

Suppose there are 3 groups in a study. There will be some variation within each group and some
variation between the groups (taking data of all groups together). An ANOVA test compares the
variation between the groups to the variation within the groups to make inferences about the means
of the groups. If the total variation between groups is higher than the variation within groups, then the
chances of the means being significantly different are high.

As these sets of hypothesis tests analyze the variance to make inferences about means, they are named
Analysis of Variance. ANOVA uses F-tests to statistically assess the equality of means of different groups.
A detailed account of how F-distribution and F-tests are used in ANOVA can be found in this article.

One-way ANOVA

There are two main types of ANOVA tests - One-way ANOVA and Two-way ANOVA - based on the
number of independent or explanatory variables involved in the test. The example below will provide
more clarity on the two scenarios:

 One-way ANOVA: Testing the relationship between shoe brands (Nike, Adidas, Puma) and race
finish times in a marathon
  Two-way ANOVA: Testing the relationship between shoe brands (Nike, Adidas, Puma), runner's
age group (Junior, Senior), and race finishing times in a marathon

The simplest form of ANOVA is called a one-way ANOVA. It enables the comparison of means between
three or more groups involving only one independent variable.

The null and alternative hypotheses for a one-way ANOVA test are formulated as:

Null Hypothesis

Alternative Hypothesis

where represents the means of groups

Consider the following graphs:

In the graphs above, a set of box plots is used to represent the distribution of values, with the group
means indicated by a white dot inside the box. These graphs help in the understanding of the hypothesis
test.

 In the first graph (Fig. a), the three populations have the same distribution if the null hypothesis
is true (i.e, the three means are equal). The variance of the combined data is approximately the
same as the variance of each of the populations.

 If the null hypothesis is false (i.e, at least one of the means is not the same), then the variance of
the combined data is larger which is caused by the different means as shown in the second
graph (Fig. b).

Execution in Python

The f_oneway() function from the Scipy library enables us to compute the test statistic and the p-value
for a one-way ANOVA test. The syntax of the function is as follows:

stats.f_oneway(sample1, sample2,....)

where the sample measurements of each group are provided as arrays as the input and the function
returns the computed test statistic and the p-value for the test.

Assumptions of one-way ANOVA

Just like many other statistical tests, the one-way ANOVA has a set of assumptions. Following are the
assumptions of the one-way ANOVA test

 Samples are independent simple random samples

 The group populations are normally distributed

 The group populations have a common variance

The normality assumption can be validated using a test called Shapiro-Wilk’s test, while the common
variance assumption can be validated using a test called Levene’s test.

Shapiro-Wilk’s test

For validating the normality assumption, Shapiro-Wilk’s test is applied to the response variable. The
Shapiro-Wilk test examines if the population from which the sample is drawn follows a normal
distribution or not.

The null and alternative hypotheses can be stated as follows:

As a rule of thumb, we can safely conclude that the normality assumption for ANOVA is validated if the
p-value is greater than the level of significance, i.e, we do not have enough evidence to reject the null
hypothesis.

Execution in Python

The shapiro() function from the Scipy library enables us to perform Shapiro-Wilk's test for normality. The
syntax of the function is as follows:

stats.shapiro(x)

The argument x is an array of sample data that is provided as the input, and the function returns the test
statistic and the p-value as the output.

Levene's test

For validating the common variance assumption, Levene's test is applied to the different groups.
Levene’s test checks whether different samples are drawn from a population with a common variance.

The null and alternative hypotheses for this test can be stated as follows:

As a rule of thumb, we can safely conclude that the common variance assumption for ANOVA is validated
if the p-value is greater than the level of significance, i.e, we do not have enough evidence to reject the
null hypothesis.

Execution in Python

The levene() function from the Scipy library enables us to perform Levene's test for equal variances. The
syntax of the function is as follows:

stats.levene(sample1, sample2,....)
The sample measurements of each group are provided as arrays as the input, and the function returns
the computed test statistic and the p-value for the test.

Important Note:

1. The above tests (Shapiro-Wilk and Levene) are NOT the one-way ANOVA test. These tests are
used to validate the assumptions of the one-way ANOVA test. The satisfaction of the
assumptions improves the reliability of the results of the ANOVA test

2. It is important to note that failure in any of the above tests does not necessarily restrict us
from performing the one-way ANOVA test

Tukey HSD test

A one-way ANOVA test enables us to find whether the difference between the means is significant, but it
does not tell anything about exactly where the differences lie. In other words, the test does not tell us
which pair of groups have a different mean out of the given groups.

After conducting a one-way ANOVA test, if we find that there is a significant difference between the
means of the groups, then one can conduct another test, called Tukey’s HSD test, to find out which pair
of groups have significantly different means. Tukey's HSD test stands for Tukey’s Honest Significant
Difference (HSD) test.

Null Hypothesis

Alternative Hypothesis

Note: Tukey's HSD test is only required when the one-way ANOVA test concludes that the difference
between the means is significant, i.e, the p-value for the one-way ANOVA test is less than the level of
significance.

Execution in Python

The pairwise_tukeyhsd() function from the Statsmodels library enables us to perform Tukey's HSD test.
The syntax of the function is as follows:

pairwise_tukeyhsd(endog, groups, alpha=0.05)

In the above function,

  endog: an array of the response variable

 groups: an array of the groups corresponding to the response variable

 alpha: level of significance for the test (the default value is 0.05)

The function returns relevant data in the form of a table including adjusted p-values for the comparison
of each possible pair from the data.