SOST 3103.

3 Regression Analysis – First Semester 2020/2021 Lecture Note - 03

Residual Analysis
1. For given X’s, the mean value of the disturbance 𝑢𝑖 is zero (Zero mean value of the residual 𝑢𝑖 )
E (ui ) = 0
(
2. The stochastic (disturbance) term 𝑢𝑖 is normally distributed. ui ~ NID 0, 2 )
3. For given X’s, the variance of 𝑢𝑖 is constant or homoscedastistic. (Homoscedasticity or equal variance
of 𝑢𝑖 )
4. For given X’s, there is no autocorrelation in disturbances.
5. If the X’s are stochastic, the disturbance term and the (stochastic) X’s are independent or at least
uncorrelated.
Test whether the Residual term 𝒖𝒊 is normally distributed or not.

The normal probability plot is a graphical technique for assessing whether the data set is
approximately normally distributed.
The data are plotted against a theoretical normal distribution in such a way that the points should form an
approximate straight line. Departures from this straight line indicate departures from normality.
The normal probability plot is a special case of the probability plot. We cover the normal probability plot
separately due to its importance in many applications.

No Residual Residual Cumulative

1 0.63 Probability
2 -1.33 -2.40 0.05
3 -1.77 -1.77 0.15
4 0.19 -1.33 0.25
5 1.26 -0.77 0.35
6 0.82 0.19 0.45
7 -0.77 0.63 0.55
8 1.37 0.82 0.65
9 2.00 1.26 0.75
10 -2.40 1.37 0.85
2.00 0.95

Cumulative probability = (i − 0.5) / n i = 1,2,......n

Anderson Darling test for normality

The Anderson-Darling test developed in 1952 by Theodore Anderson and Donald Darling. The Anderson-
Darling test is an alternative to the chi-square and Kolmogorov-Smirnov goodness-of-fit tests. It is used to
test a given data series is normal. Thus, it tests,

Hypothesis

H0: The data follows a normal distribution

H1: The data do not follow a normal distribution

Test Statistic

The Anderson-Darling statistic is given by the following formula

1 n
AD = −n −  (2i − 1)ln F ( X i ) + ln (1 − F ( X n−i+1 ))
n i −1

Where n = sample size, F(X) = cumulative distribution function for the normal distribution and i = the i th
sample when the data is sorted in ascending order.

Decision rule

If the p-value of these test is less than our chosen significance level ( ) , we can reject the null hypothesis
and conclude that the data do not follow the normal distribution.

Jarque-Bera Test for Normality

Most tests for normality are based either on comparing the empirical cumulative distribution with the
theoretical normal cumulative distribution (Kolmogorov-Smirnov, Anderson-Darling, Chi-Square) or
empirical quantiles with the theoretical normal quantiles (PPCC, Wilk-Shapiro). In contrast, the Jarque-Bera
test is based on the sample skewness and sample kurtosis.

Hypothesis

H0: The data follows a normal distribution

H1: The data do not follow a normal distribution

Teat Statistic

The Jarque-Bera test statistic is defined as:

n  2 1 2
JB =  S + 4 (k − 3)  ~  2
2

with S, K, and n denoting the sample skewness, the sample kurtosis, and the sample size, respectively.
Decision rule

JB   22  H 0 do not reject or if the p-value of these test is less than our chosen significance level ( ) ,
we can reject the null hypothesis and conclude that the data do not follow the normal distribution.

Test whether the Residual term 𝒖𝒊 is homoscedasticity (Constant variance) or not.

When you run a regression analysis, the variance of the error terms must be constant, and they must have a
mean of zero. If this isn't the case, your model may not be valid.

To check these assumptions, you should use a residual versus fitted values plot. The errors have constant
variance, with the residuals scattered randomly around zero. If, for example, the residuals increase or
decrease with the fitted values in a pattern, the errors may not have constant variance.

The points on the plot above appear to be randomly scattered around zero, so assuming that the error terms
have a mean of zero is reasonable. The vertical width of the scatter doesn't appear to increase or decrease
across the fitted values, so we can assume that the variance in the error terms is constant.

But what if this wasn't the case? What if we did notice a pattern in the plot? I created some fake data to
illustrate this point, then created another plot.

There is definitely a noticeable pattern here! The residuals (error terms) take on positive values with small
or large fitted values, and negative values in the middle. The width of the scatter seems consistent, but the
points are not randomly scattered around the zero line from left to right. This graph tells us we should not
use the regression model that produced these results. The residual has a pattern, so we can assume that the
residual series has Quadratic effect.

Check whether the Residual term 𝒖𝒊 is independent or not.

When you run a regression analysis, the error terms must be independent, to check this assumption, you
should use a "residuals vs. order plot". In this section, we learn how to use a "residuals vs. order plot" as
a way of detecting a particular form of non-independence of the error terms, namely serial correlation. If
the data are obtained in a time (or space) sequence, a residual vs. order plot helps to see if there is any
correlation between the error terms that are near each other in the sequence.

So, what is this residual vs. order plot all about? As its name suggests, it is a scatter plot with residuals on
the y axis and the order in which the data were collected on the x axis. Here's an example of a well-behaved
residuals vs. order plot:

The residuals bounce randomly around the residual = 0 line as we would hope so. In general, residuals
exhibiting normal random noise around the residual = 0 line suggest that there is no serial correlation.

Let's take a look at examples of the different kinds of residuals vs. order plots we can obtain and learn what
each tells us.

A time trend

A residual vs. order plot that exhibits (positive) trend as the following plot does:

suggests that some of the variation in the response is due to time. Therefore, it might be a good idea to add
the predictor "time" to the model. That is, you interpret this plot just as you would interpret any other residual
vs. predictor plot. It's just that here your predictor is "time."
Positive serial correlation

A residual vs. order plot that looks like the following plot:

suggests that there is "positive serial correlation" among the error terms. That is, positive serial correlation
exists when residuals tend to be followed, in time, by residuals of the same sign and about the same
magnitude. The plot suggests that the assumption of independent error terms is violated.

Negative serial correlation

A residual vs. order plot that looks like the following plot:

suggests that there is "negative serial correlation" among the error terms. Negative serial correlation exists
when residuals of one sign tend to be followed, in time, by residuals of the opposite sign.

Negative, positive, negative, positive, negative, positive, and so on. The plot suggests that the assumption of
independent error terms is violated.