Econometrics I

Professor William Greene

Stern School of Business
Department of Economics

6-1/49 Part 6: Estimating the Variance of b

 Econometrics I

Part 6 – Estimating
the Variance of b

6-2/49 Part 6: Estimating the Variance of b

6-3/49 Part 6: Estimating the Variance of b
 Econometric Dire Emergency

6-4/49 Part 6: Estimating the Variance of b

 Context

The true variance of b|X is 2(XX)-1 . We

consider how to use the sample data to estimate
this matrix. The ultimate objectives are to form
interval estimates for regression slopes and to
test hypotheses about them. Both require
estimates of the variability of the distribution.

6-5/49 Part 6: Estimating the Variance of b

 Estimating 2

Using the residuals instead of the disturbances:

The natural estimator: ee/N as a sample
surrogate for /n
Imperfect observation of i, ei = i - ( - b)xi
Downward bias of ee/N. We obtain the result
E[ee|X] = (N-K)2

6-6/49 Part 6: Estimating the Variance of b

 Expectation of ee

e  y - Xb
 y  X ( X ' X )1 X ' y
1
 [I  X( X ' X) X ']y
 My  M( X  )  MX  M  M
e'e  (M'(M
 'M'M  'MM  'M

6-7/49 Part 6: Estimating the Variance of b

 Method 1:
E[e'e|X ]  E'M |X 
E[ trace ('M |X ) ] scalar = its trace
E[ trace (M'|X ) ] permute in trace
[ trace E (M'|X ) ] linear operators
[ trace M E ('|X ) ] conditioned on X
[ trace M 2I ] model assumption
2 [trace M ] scalar multiplication and I matrix
2 trace [I - X( X'X)-1 X' ]
2 {trace [I] - trace[X( X'X)-1 X' ]}
2 {N - trace[( X'X)-1 X'X ]} permute in trace
2 {N - trace[I]}
2 {N - K}
Notice that E[ee|X] is not a function of X.

6-8/49 Part 6: Estimating the Variance of b

 Estimating σ2
The unbiased estimator is s2 = ee/(N-K).

“Degrees of freedom correction”

Therefore, the unbiased estimator of 2 is

s2 = ee/(N-K)

6-9/49 Part 6: Estimating the Variance of b

 Method 2: Some Matrix Algebra
E[e'e|X ]  2 trace M
What is the trace of M? M is idempotent, so its
trace equals its rank. Its rank equals the number
of nonzero characeristic roots.
Characteric Roots :
Signature of a Matrix = Spectral Decomposition
= Eigen (own) value Decomposition
A = CC' where
C = a matrix of columns such that CC' = C'C = I
 = a diagonal matrix of the characteristic roots
elements of  may be zero

6-10/49 Part 6: Estimating the Variance of b

 Decomposing M
Useful Result: If A = CC' is the spectral
decomposition, then A 2  C2 C ' (just multiply)
M = M2 , so  2  . All of the characteristic
roots of M are 1 or 0. How many of each?
trace(A) = trace(CC')=trace(C'C)=trace( )
Trace of a matrix equals the sum of its characteristic
roots. Since the roots of M are all 1 or 0, its trace is
just the number of ones, which is N-K as we saw.

6-11/49 Part 6: Estimating the Variance of b

 Example: Characteristic Roots of a
Correlation Matrix

6-12/49 Part 6: Estimating the Variance of b

 R = CΛC  i 1 i cici
6

6-13/49 Part 6: Estimating the Variance of b

 Gasoline Data

6-14/49 Part 6: Estimating the Variance of b

 X’X and its Roots

6-15/49 Part 6: Estimating the Variance of b

 Var[b|X]

Estimating the Covariance Matrix for b|X

The true covariance matrix is 2 (X’X)-1
The natural estimator is s2(X’X)-1
“Standard errors” of the individual coefficients are
the square roots of the diagonal elements.

6-16/49 Part 6: Estimating the Variance of b

 X’X

(X’X)-1

s2(X’X)-1

6-17/49 Part 6: Estimating the Variance of b

 Standard Regression Results
----------------------------------------------------------------------
Ordinary least squares regression ........
LHS=G Mean = 226.09444
Standard deviation = 50.59182
Number of observs. = 36
Model size Parameters = 7
Degrees of freedom = 29
Residuals Sum of squares = 778.70227
Standard error of e = 5.18187 <= sqr[778.70227/(36 – 7)]
Fit R-squared = .99131
Adjusted R-squared = .98951
--------+-------------------------------------------------------------
Variable| Coefficient Standard Error t-ratio P[|T|>t] Mean of X
--------+-------------------------------------------------------------
Constant| -7.73975 49.95915 -.155 .8780
PG| -15.3008*** 2.42171 -6.318 .0000 2.31661
Y| .02365*** .00779 3.037 .0050 9232.86
TREND| 4.14359** 1.91513 2.164 .0389 17.5000
PNC| 15.4387 15.21899 1.014 .3188 1.67078
PUC| -5.63438 5.02666 -1.121 .2715 2.34364
PPT| -12.4378** 5.20697 -2.389 .0236 2.74486
--------+-------------------------------------------------------------

6-18/49 Part 6: Estimating the Variance of b

 The Variance of OLS - Sandwiches
If Var[] = 2 I, then Var[b|X] = 2 (X'X) -1
What if Var[]  2 I?
Possibilities: Heteroscedasticity, Autocorrelation, Clustering and common effects.

b =  + (X'X)-1 *  i 1 xi i
n

Var[b|X] = (X'X)-1  Var   i 1 xi i   (X'X)-1

n

 
= A sandwich matrix. = A B A
What does the variance of the sum (the meat) look like?
Leading cases.
(1) Heteroscedasticity
(2) Autocorrelation
(3) Grouped (clustered) observations with common effects.

6-19/49 Part 6: Estimating the Variance of b

 Robust Covariance Estimation
 Not a structural estimator of XX/n
 If the condition is present, the estimator
estimates the true variance of the OLS estimator
 If the condition is not present, the estimator
estimates the same matrix that (2/n)(X’X/n)-1
estimates .
 Heteroscedasticity
 Autocorrelation
 Common effects

6-20/49 Part 6: Estimating the Variance of b

 Heteroscedasticity Robust Covariance Matrix
 Robust estimation: Generality
 How to estimate Var[b|X] = 2 (X’X)-1 XX (X’X)-1 for
the LS b?
 The distinction between estimating
2 an n by n matrix
and estimating the KxK matrix
2 XX = 2 ijij xi xj
 NOTE…… VVVIRs for modern applied econometrics.
 The White estimator
 Newey-West.

6-21/49 Part 6: Estimating the Variance of b

6-22/49 Part 6: Estimating the Variance of b
 The White Estimator
Est.Var[b]  ( X'X ) 1   i1 ei2 x i x i' ( X'X ) 1
n

 
 i1 i
n 2
2
e
Use 
ˆ 
n
nei2 ˆ ˆ )=n

ˆ i = 2 , Ω=diag( ˆ i ) note tr(Ω
ˆ
ˆ  X'X   X'ΩX
 2 1
ˆ   X'X 
Est.Var[b]    
n  n   n   n 
 

 ˆ 
2 X'ΩX 2  X'ΩX 
Does 
ˆ     n   0?
 n   
 
6-23/49 Part 6: Estimating the Variance of b
 Groupwise Heteroscedasticity

Countries
are ordered
by the
standard
deviation of
their 19
residuals.

Regression of log of per capita gasoline use on log of per capita income,
gasoline price and number of cars per capita for 18 OECD countries for 19
years. The standard deviation varies by country. The “solution” is
“weighted least squares.”

6-24/49 Part 6: Estimating the Variance of b

 White Estimator
+--------+--------------+----------------+--------+--------+----------+
|Variable| Coefficient | Standard Error |t-ratio |P[|T|>t]| Mean of X|
+--------+--------------+----------------+--------+--------+----------+
Constant| 2.39132562 .11693429 20.450 .0000
LINCOMEP| .88996166 .03580581 24.855 .0000 -6.13942544
LRPMG | -.89179791 .03031474 -29.418 .0000 -.52310321
LCARPCAP| -.76337275 .01860830 -41.023 .0000 -9.04180473
| White heteroscedasticity robust covariance matrix |
+----------------------------------------------------+
Constant| 2.39132562 .11794828 20.274 .0000
LINCOMEP| .88996166 .04429158 20.093 .0000 -6.13942544
LRPMG | -.89179791 .03890922 -22.920 .0000 -.52310321
LCARPCAP| -.76337275 .02152888 -35.458 .0000 -9.04180473

6-25/49 Part 6: Estimating the Variance of b

 Autocorrelated Residuals
logG=β1 + β2logPg + β3logY + β4logPnc + β5logPuc + ε

6-26/49 Part 6: Estimating the Variance of b

 The Newey-West Estimator
Robust to Autocorrelation
Heteroscedasticity Component - Diagonal Elements
1 n 2
S 0   i1 ei x i x i'
n
Autocorrelation Component - Off Diagonal Elements
1 L
S1   l1  t l1 w le t e t l ( x t x t l  x t l x t )
n

n
l
wl  1  = "Bartlett weight"
L 1
1 1
1  X'X   X'X 
Est.Var[b]=   [S 0  S1 ]  
n n   n 

6-27/49 Part 6: Estimating the Variance of b

 Newey-West Estimate
--------+-------------------------------------------------------------
Variable| Coefficient Standard Error t-ratio P[|T|>t] Mean of X
--------+-------------------------------------------------------------
Constant| -21.2111*** .75322 -28.160 .0000
LP| -.02121 .04377 -.485 .6303 3.72930
LY| 1.09587*** .07771 14.102 .0000 9.67215
LPNC| -.37361** .15707 -2.379 .0215 4.38037
LPUC| .02003 .10330 .194 .8471 4.10545
--------+-------------------------------------------------------------
--------+-------------------------------------------------------------
Variable| Coefficient Standard Error t-ratio P[|T|>t] Mean of X
Robust VC Newey-West, Periods = 10
--------+-------------------------------------------------------------
Constant| -21.2111*** 1.33095 -15.937 .0000
LP| -.02121 .06119 -.347 .7305 3.72930
LY| 1.09587*** .14234 7.699 .0000 9.67215
LPNC| -.37361** .16615 -2.249 .0293 4.38037
LPUC| .02003 .14176 .141 .8882 4.10545
--------+-------------------------------------------------------------

6-28/49 Part 6: Estimating the Variance of b

 Panel Data
 Presence of omitted effects
y it =x itβ + c i + εit , observation for person i at time t
y i = X iβ + c ii + ε i , Ti observations in group i
=X iβ + c i + ε i , note c i  (c i , c i ,...,c i )
y =Xβ + c + ε , Ni=1 Ti observations in the sample

 Potential bias/inconsistency of OLS – depends

on the assumptions about unobserved c.
 Variance of OLS is affected by autocorrelation in
most cases.
6-29/49 Part 6: Estimating the Variance of b
 Estimating the Sampling Variance of b

 s2(X ́X)-1? Inappropriate because

 Correlation across observations (certainly)
 Heteroscedasticity (possibly)

 A ‘robust’ covariance matrix

 Robust estimation (in general)
 The White estimator
 A Robust estimator for OLS.

6-30/49 Part 6: Estimating the Variance of b

 Cluster Robust Estimator
Robust variance estimator for Var[b]
Est.Var[b]
= ( X'X ) 1 Ni=1 ( X iei )(ei X i )  ( X'X ) 1

= ( X'X ) 1 Ni=1 (Ct=1

i
x it eit )(Ct=1
i
x it eit )  ( X'X) 1

 
= ( X'X ) 1 Ni=1 Ct=1
i
Cs=1
i

eit eis x it x is  ( X'X ) 1
e  a least squares residual
(If Ci = 1, this is the White estimator.)

6-31/49 Part 6: Estimating the Variance of b

6-32/49 Part 6: Estimating the Variance of b
6-33/49 Part 6: Estimating the Variance of b
 Alternative OLS Variance Estimators
Cluster correction increases SEs
+---------+--------------+----------------+--------+---------+
|Variable | Coefficient | Standard Error |b/St.Er.|P[|Z|>z] |
+---------+--------------+----------------+--------+---------+
Constant 5.40159723 .04838934 111.628 .0000
EXP .04084968 .00218534 18.693 .0000
EXPSQ -.00068788 .480428D-04 -14.318 .0000
OCC -.13830480 .01480107 -9.344 .0000
SMSA .14856267 .01206772 12.311 .0000
MS .06798358 .02074599 3.277 .0010
FEM -.40020215 .02526118 -15.843 .0000
UNION .09409925 .01253203 7.509 .0000
ED .05812166 .00260039 22.351 .0000
Robust
Constant 5.40159723 .10156038 53.186 .0000
EXP .04084968 .00432272 9.450 .0000
EXPSQ -.00068788 .983981D-04 -6.991 .0000
OCC -.13830480 .02772631 -4.988 .0000
SMSA .14856267 .02423668 6.130 .0000
MS .06798358 .04382220 1.551 .1208
FEM -.40020215 .04961926 -8.065 .0000
UNION .09409925 .02422669 3.884 .0001
ED .05812166 .00555697 10.459 .0000

6-34/49 Part 6: Estimating the Variance of b

 Bootstrapping
Some assumptions that underlie it - the sampling mechanism
Method:
1. Estimate using full sample: --> b
2. Repeat R times:
Draw N observations from the n, with replacement
Estimate  with b(r).
3. Estimate variance with
V = (1/R)r [b(r) - b][b(r) - b]’

6-35/49 Part 6: Estimating the Variance of b

 Bootstrap Application
matr;bboot=init(3,21,0.)$ Store results here
name;x=one,y,pg$ Define X
regr;lhs=g;rhs=x$ Compute b
calc;i=0$ Counter
Proc Define procedure
regr;lhs=g;rhs=x;quietly$ … Regression
matr;{i=i+1};bboot(*,i)=b$... Store b(r)
Endproc Ends procedure
exec;n=20;bootstrap=b$ 20 bootstrap reps
matr;list;bboot' $ Display results

6-36/49 Part 6: Estimating the Variance of b

 Results of Bootstrap Procedure
--------+-------------------------------------------------------------
Variable| Coefficient Standard Error t-ratio P[|T|>t] Mean of X
--------+-------------------------------------------------------------
Constant| -79.7535*** 8.67255 -9.196 .0000
Y| .03692*** .00132 28.022 .0000 9232.86
PG| -15.1224*** 1.88034 -8.042 .0000 2.31661
--------+-------------------------------------------------------------
Completed 20 bootstrap iterations.
----------------------------------------------------------------------
Results of bootstrap estimation of model.
Model has been reestimated 20 times.
Means shown below are the means of the
bootstrap estimates. Coefficients shown
below are the original estimates based
on the full sample.
bootstrap samples have 36 observations.
--------+-------------------------------------------------------------
Variable| Coefficient Standard Error b/St.Er. P[|Z|>z] Mean of X
--------+-------------------------------------------------------------
B001| -79.7535*** 8.35512 -9.545 .0000 -79.5329
B002| .03692*** .00133 27.773 .0000 .03682
B003| -15.1224*** 2.03503 -7.431 .0000 -14.7654
--------+-------------------------------------------------------------

6-37/49 Part 6: Estimating the Variance of b

 Bootstrap Replications

Full sample result

Bootstrapped sample
results

6-38/49 Part 6: Estimating the Variance of b

 Results of C&R Bootstrap Estimation

6-39/49 Part 6: Estimating the Variance of b

 Bootstrap variance for a
panel data estimator
 Panel Bootstrap =
Block Bootstrap
 Data set is N groups of
size Ti
 Bootstrap sample is N
groups of size Ti drawn
with replacement.

6-40/49 Part 6: Estimating the Variance of b

6-41/49 Part 6: Estimating the Variance of b
 Quantile Regression:
Application of Bootstrap
Estimation

6-42/49 Part 6: Estimating the Variance of b

 OLS vs. Least Absolute Deviations
----------------------------------------------------------------------
Least absolute deviations estimator...............
Residuals Sum of squares = 1537.58603
Standard error of e = 6.82594
Fit R-squared = .98284
--------+-------------------------------------------------------------
Variable| Coefficient Standard Error b/St.Er. P[|Z|>z] Mean of X
--------+-------------------------------------------------------------
|Covariance matrix based on 50 replications.
Constant| -84.0258*** 16.08614 -5.223 .0000
Y| .03784*** .00271 13.952 .0000 9232.86
PG| -17.0990*** 4.37160 -3.911 .0001 2.31661
--------+-------------------------------------------------------------
Ordinary least squares regression ............
Residuals Sum of squares = 1472.79834
Standard error of e = 6.68059 Standard errors are based on
Fit R-squared = .98356 50 bootstrap replications
--------+-------------------------------------------------------------
Variable| Coefficient Standard Error t-ratio P[|T|>t] Mean of X
--------+-------------------------------------------------------------
Constant| -79.7535*** 8.67255 -9.196 .0000
Y| .03692*** .00132 28.022 .0000 9232.86
PG| -15.1224*** 1.88034 -8.042 .0000 2.31661
--------+-------------------------------------------------------------

6-43/49 Part 6: Estimating the Variance of b

 Quantile Regression
 Q(y|x,) = x,  = quantile
 Estimated by linear programming
 Q(y|x,.50) = x, .50  median regression
 Median regression estimated by LAD (estimates same
parameters as mean regression if symmetric conditional
distribution)
 Why use quantile (median) regression?
 Semiparametric
 Robust to some extensions (heteroscedasticity?)
 Complete characterization of conditional distribution

6-44/49 Part 6: Estimating the Variance of b

 Estimated Variance for
Quantile Regression

 Asymptotic Theory

 Bootstrap – an ideal application

6-45/49 Part 6: Estimating the Variance of b

 Asymptotic Theory Based Estimator of Variance of Q - REG
Model : yi  βx i  ui , Q( yi | xi , )  βx i , Q[ui | xi , ]  0
Residuals: uˆ i  yi - βˆ x i

Asymptotic Variance:
1
N
 A 1 C A 1 

1 11
A = E[f u (0) xx] Estimated by  i 1 1| uˆi | B xi xi
N

N B2
Bandwidth B can be Silverman's Rule of Thumb:
1.06  Q(uˆi | .75)  Q(uˆi | .25) 
Min  su , 
 
.2
N 1.349
(1- )
C = (1- ) E[xx] Estimated by XX
N
 2
su  XX  .
1
For  =.5 and normally distributed u, this all simplifies to
2
But, this is an ideal application for bootstrapping.

6-46/49 Part 6: Estimating the Variance of b

  = .25

 = .50

 = .75

6-47/49 Part 6: Estimating the Variance of b

6-48/49 Part 6: Estimating the Variance of b
6-49/49 Part 6: Estimating the Variance of b