Torój - OLS Revisited

Introduction OLS
Lecture 1: OLS revisited

Econometric Methods
Andrzej Torój
SGH Warsaw School of Economics – Institute of Econometrics
Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

(1) OLS 1 / 32
Introduction OLS
Outline
1 Introduction
Course information
Econometrics: a reminder
2 OLS: theoretical reminder

Point estimation
Measuring precision
Model quality diagnostics under OLS

(1) OLS 2 / 32
Introduction OLS
Outline
1 Introduction

(1) OLS 3 / 32
Introduction OLS
Course information
Course information
lecturers: Andrzej Torój & Piotr Dybka

my website: http://web.sgh.waw.pl/~atoroj/ (lecture
slides, exercise files, literature, contact)
final grade: homework (50%) + written exam (50%); details:
website

(1) OLS 4 / 32
Introduction OLS
Course information
Course information

website

(1) OLS 4 / 32
Introduction OLS
Course information
Course information

website

(1) OLS 4 / 32
Introduction OLS
Why econometrics?
investigation of relationships
finding parameter values in economic models (e.g. elasticities)
confronting economic theories with data
forecasting
simulating policy scenarios

(1) OLS 5 / 32
Introduction OLS
Why econometrics?
forecasting

(1) OLS 5 / 32
Introduction OLS
Why econometrics?
forecasting

(1) OLS 5 / 32
Introduction OLS
Why econometrics?
forecasting

(1) OLS 5 / 32
Introduction OLS
Why econometrics?
forecasting

(1) OLS 5 / 32
Introduction OLS
Example (1/4)
Student satisfaction survey
Master students of Applied Econometrics at Warsaw School of Economics in
Winter semester 2016/2017 were asked about their satisfaction from studying
to be evaluated from 0 to 100. In addition, their average note from previous
studies and their sex were registered.
1 What kind of data is this? Cross-section, time series, panel?
Frequency? Micro- or macroeconomic?
2 How can we quickly visualise a hypothesised causality from average
note to satisfaction from studying?
3 Does such a relationship seem to be there?
4 How can sex of the respondent potentially affect the satisfaction from
studies or the relationship in question? How can we visualise this?
5 Bottom line, what is the right specification of the linear regression
model?

(1) OLS 6 / 32
Introduction OLS
Outline
1 Introduction

(1) OLS 7 / 32
Introduction OLS
Point estimation
Linear regression model
yi = β0 + β1 x1,i + β2 x2,i 
+ ... +
 βk xk,i + εi =
β0
 β1 
 
1 x1,i x2,i . . . xk,i  β2  + εi = xi β + εi
 
 .. 
 . 
βk
T
Vector of parameters β0 β1 β2 . . . βk is unknown.
Minimize the dispersion of εi around zero, as measured e.g. by
n
ε2i .
P
t=1

(1) OLS 8 / 32
Introduction OLS
Point estimation
Ordinary Least Squares (OLS)
n n
2
ε2i =
P P
S= (yi − β0 − β1 x1,i − β2 x2,i − . . . − βk xk,i ) → min
i=1 i=1 β0 ,β1,...
∂S
FOC: ∂β =0
β0
 
1
   
y1 x1,1 x2,1 ... xk,1
 β1 
 y2   1 x1,2 x2,2 ... xk,2   
Denote: y=

, X = 
 
, β = 
  β2 
.
. .
. .
. .
. .
.

 .   . . . ... .   .
.

 . 
yn 1 x1,n x2,n ... xk,n
βk
and obtain:
−1 T
β = XT X X y

(1) OLS 9 / 32
Introduction OLS
Point estimation
Proof
n
ε2i = εT ε = (y − Xβ)T (y − Xβ) =
P
S =
i=1
= − β T XT y − yT Xβ + β T XT Xβ =
yT y
= yT y − 2yT Xβ + β T XT Xβ
(2. and 3. component were transposed scalars, so they were equal)
∂S ∂yT y 2yT Xβ β T XT Xβ
∂β = 0 ⇐⇒ ∂β − ∂β + ∂β =0
According to the rulesof matrix calculus:
T T
−2yT X T T
+ β T2X X = 0 99K β X X = y X 99K
T
T
X X β=X y
−1 T
β = XT X X y

(1) OLS 10 / 32
Introduction OLS
Point estimation
Example (2/4)

1 Run the regression model with an automated command in R.
2 Write the equation and try to interpret the parameters. Be
careful – it’s tricky! (Why?)
3 Manually replicate the parameter estimates.

(1) OLS 11 / 32
Introduction OLS
Measuring precision
Estimator as a random variable

β̂ is an estimator of the true parameter value β (function of
the random sample choice)
samples, and hence the values of β̂, can be different
estimator as a (vector) random variable has its
variance(-covariance matrix)
β̂0
 
 β̂1 
 
 β̂2 
β̂ = .  Var β̂ =

 . 
 . 
β̂k
 
var β̂0 cov β̂0 , β̂1 cov β̂0 , β̂2 ···
 

 cov β̂0 , β̂1 var β̂1 cov β̂1 , β̂2 ··· 

 

 cov β̂0 , β̂2 cov β̂1 , β̂2 var β̂2 ··· 

 
 . . . . 
 . . . . 

 . . . . 

var β̂k

(1) OLS 12 / 32
Introduction OLS
Measuring precision

β̂0
 
 β̂1 
 
 β̂2 
β̂ = .  Var β̂ =

 . 
 . 
β̂k
 
var β̂0 cov β̂0 , β̂1 cov β̂0 , β̂2 ···
 

 cov β̂0 , β̂1 var β̂1 cov β̂1 , β̂2 ··· 

 

 cov β̂0 , β̂2 cov β̂1 , β̂2 var β̂2 ··· 

 
 . . . . 
 . . . . 

 . . . . 

var β̂k

(1) OLS 12 / 32
Introduction OLS
Measuring precision

β̂0
 
 β̂1 
 
 β̂2 
β̂ = .  Var β̂ =

 . 
 . 
β̂k
 
var β̂0 cov β̂0 , β̂1 cov β̂0 , β̂2 ···
 

 cov β̂0 , β̂1 var β̂1 cov β̂1 , β̂2 ··· 

 

 cov β̂0 , β̂2 cov β̂1 , β̂2 var β̂2 ··· 

 
 . . . . 
 . . . . 

 . . . . 

var β̂k

(1) OLS 12 / 32
Introduction OLS
Measuring precision
Variance-covariance matrix of a random vector
Definition: n o
Var (β) = E [β − E (β)] [β − E (β)]T
For a centered variable, i.e. E (ε) = 0, this definition simplifies:
T

Var (ε) = E εε

(1) OLS 13 / 32
Introduction OLS
Measuring precision
OLS estimator: properties
−1 T
β̂ = XT X X y is an estimator (function of the sample) of
the “true”, unknown values β (population / data generating
process). Under certain conditions (i.a. E (X T ε) = 0
E (εεT ) = σ 2 I ), the OLS estimator is:

unbiased: E β̂ = β
consistent: β̂ converges to β with growing n
efficient: least possible estimator variance (i.e. highest
precision)

(1) OLS 14 / 32
Introduction OLS
Measuring precision
−1 T

precision)

(1) OLS 14 / 32
Introduction OLS
Measuring precision
−1 T

precision)

(1) OLS 14 / 32
Introduction OLS
Measuring precision
Variance of the error term (1)
n
1
1. Variance of the error term (scalar): σ̂ 2 = ε2i
P
n−(k+1)
i=1
Why such a formula if the general formula is
n
1 P
Var (X ) = n−1 (xi − x̄)2 ?
i=1
First of all note that ε̄ = 0 (prove it on your own).
Second, we need to know why 1 turned into (k + 1) in the
denominator.

(1) OLS 15 / 32
Introduction OLS
Measuring precision
By Your intuition, what is the standard deviation in the

following dataset of 3 observation?
Without r
a correction in denominator:
√ h
2 2 2
i q
Var = 3 (3 − 2) + (2 − 2) + (1 − 2) = 23 6= 1
1

(1) OLS 16 / 32
Introduction OLS
Measuring precision
By Your intuition, what is the standard deviation in the

following dataset of 3 observation?
Without r
a correction in denominator:
√ h
2 2 2
i q
Var = 3 (3 − 2) + (2 − 2) + (1 − 2) = 23 6= 1
1

(1) OLS 16 / 32
Introduction OLS
Measuring precision

The intuition behind the standard deviation of 1 is build upon
an implicit, graphical calibration of mean based on the data
sample.
With an adequate correction for thatin denominator:

√
r h i q
2 2 2
Var = 3−1 (3 − 2) + (2 − 2) + (1 − 2) = 22 = 1
1

(1) OLS 17 / 32
Introduction OLS
Measuring precision

The intuition behind the standard deviation of 1 is build upon
an implicit, graphical calibration of mean based on the data
sample.
With an adequate correction for thatin denominator:

√
r h i q
2 2 2
Var = 3−1 (3 − 2) + (2 − 2) + (1 − 2) = 22 = 1
1

(1) OLS 17 / 32
Introduction OLS
Measuring precision
When X is directly observed, the terms like (xi − x̄) consume

one degree of freedom (there is one x̄ estimated before).
When ε is not observed, the terms
εi = yi − β̂0 − β̂1 x1i − ... − β̂k xki consume k + 1) degrees of
freedom (there are k + 1 elements in vector β̂ estimated
before).

(1) OLS 18 / 32
Introduction OLS
Measuring precision
Variance-covariance matrix of the estimator
T
Var β̂ = E β̂ − β β̂ − β =
h i h iT
−1 T −1 T
XT X X y − β · XT X

= E X y −β =
h i h iT
T
−1 T T
−1 T
= E X X X (Xβ + ε) − β · X X X (Xβ + ε) − β =
h −1 T −1 i
T T T
= E X X X ε·ε X X X =
T
−1 T T
T
−1
= X X X E εε X X X =
−1 T 2 −1
= XT X X σ IX XT X =
−1 T −1
= σ 2 XT X X X XT X =
2 T
−1
= σ X X
−1
Empirical counterpart: Var β̂ = σ̂ 2 XT X ≡ [di,j ](k+1)×(k+1)

(1) OLS 19 / 32
Introduction OLS
Measuring precision
Standard errors of estimation
Standard
p p (vector
errors of estimation –
for each parameter):
p
s β̂0 = d1,1 s β̂1 = d2,2 s β̂2 = d3,3 . . .
Calculating S.E.
1. estimate parameters, 2. compute the empirical error terms, 3.
estimate their variance, 4. compute the variance-covariance matrix
of the OLS estimator, 5. compute the SE as a square root of its
diagonal elements.

(1) OLS 20 / 32
Introduction OLS
Measuring precision
Standard errors of estimation
Standard
p p (vector
errors of estimation –
for each parameter):
p
s β̂0 = d1,1 s β̂1 = d2,2 s β̂2 = d3,3 . . .
Calculating S.E.
1. estimate parameters, 2. compute the empirical error terms, 3.
estimate their variance, 4. compute the variance-covariance matrix
of the OLS estimator, 5. compute the SE as a square root of its
diagonal elements.

(1) OLS 20 / 32
Introduction OLS
Measuring precision
t-tests for variable significance
t-Student test
H0 : βi = 0, i.e. i-th explanatory variable does not significantly
influence y
H1 : βi 6= 0, i.e. i-th explanatory variable does not significantly
influence y
Test statistic: t = s β̂β̂i is distributed as t (n − k − 1).
( 1)
p-value<α∗ – reject H0
p-value>α∗ – do not reject H0

(1) OLS 21 / 32
Introduction OLS
Measuring precision
Example (3/4)

1 Compute the fitted values and the error terms.
2 Use this result to estimate the variance of the error term.
3 Estimate the variance-covariance matrix of the β̂ estimates.
4 Derive the standard errors from this matrix.
5 Replicate and interpret the t statistics and the p-values.

(1) OLS 22 / 32
Introduction OLS
R-squared (1)

(1) OLS 23 / 32
Introduction OLS
R-squared (2)

(1) OLS 24 / 32
Introduction OLS
R-squared (3)

(1) OLS 25 / 32
Introduction OLS
R-squared (4)

(1) OLS 26 / 32
Introduction OLS
R-squared (5)

(1) OLS 27 / 32
Introduction OLS
R-squared (6)

(1) OLS 28 / 32
Introduction OLS
R-squared (7)
R 2 ∈ [0; 1] is a share of yt volatility explained by the model in

total yt volatility:
T
(ŷt −ȳ )2
P
T T T
2 2 2
R2 =
P P P t=1
(yt − ȳ ) = (ŷt − ȳ ) + (yt − ŷt ) T
t=1 t=1 t=1 (yt −ȳ )2
P
t=1
Standard goodness-of-fit measure in OLS regressions with a

constant.

(1) OLS 29 / 32
Introduction OLS
Wald test statistic
Wald test
H0 : β1 = β2 = . . . = βk = 0, i.e. no explanatory variable
influences y
H1 : ∃i βi 6= 0, at least 1 explanatory variable influences y
2 /k
Test statistic: F = (1−R 2 R)/(T −k−1)
distributed as F (k, T − k − 1).

(1) OLS 30 / 32
Introduction OLS
Adjusted R-squared
k
R¯2 = |{z}
R2 − 1 − R2

T − (k + 1)
fit | {z }
penalty for overparametrization

(1) OLS 31 / 32
Introduction OLS
Example (4/4)

1 Interpret the F-test result.
2 Replicate the F statistic and its p-value manually.
3 Interpret the R-squared.
4 Replicate the R-squared and adjusted R-squared manually.

(1) OLS 32 / 32

Torój - OLS Revisited

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Torój - OLS Revisited

Uploaded by

Copyright:

Available Formats

Introduction OLS

Lecture 1: OLS revisited

SGH Warsaw School of Economics – Institute of Econometrics

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

2 OLS: theoretical reminder

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

2 OLS: theoretical reminder

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

lecturers: Andrzej Torój & Piotr Dybka

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

lecturers: Andrzej Torój & Piotr Dybka

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

lecturers: Andrzej Torój & Piotr Dybka

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

2 OLS: theoretical reminder

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Linear regression model

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Ordinary Least Squares (OLS)

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Student satisfaction survey

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Estimator as a random variable

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Estimator as a random variable

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Estimator as a random variable

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Variance-covariance matrix of a random vector

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

OLS estimator: properties

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

OLS estimator: properties

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

OLS estimator: properties

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Variance of the error term (1)

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Variance of the error term (2)

By Your intuition, what is the standard deviation in the

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Variance of the error term (2)

By Your intuition, what is the standard deviation in the

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Variance of the error term (3)

With an adequate correction for thatin denominator:

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Variance of the error term (3)

With an adequate correction for thatin denominator:

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Variance of the error term (4)

When X is directly observed, the terms like (xi − x̄) consume

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics

Variance-covariance matrix of the estimator

Andrzej Torój SGH Warsaw School of Economics – Institute of Econometrics