it is the student's notes on ordinary least square methods

© All Rights Reserved

4 views

it is the student's notes on ordinary least square methods

© All Rights Reserved

- Econometrics
- Session12 Solution
- Reducing Uncertainty Allometric 2001
- Attrition 2
- 2011_Diniz-Filho Et Al_Eigenvector Estimation of Phylogenetic and Functional Diversity
- Ch 16 Binary Dependent Var's
- maldonado & moreno IASC 2011
- Topic 4
- Poverty in Zambia 2
- Demand Estimation
- hChap8
- W8 - AUTOC-6pPg
- Tests of Normality of Regression Errors
- Des Aigoo Ls Bee Final
- Saxena Cs4758 Lecture4
- Econometrics HW V
- 2
- 4614ijcsa01
- Chumacero OLS
- 21Week7Lect3

You are on page 1of 3

Allin Cottrell

1

Let y denote the dependent variable, a n1 vector, and let X denote the nk matrix of regressors (independent

variables). Write b for the k-vector of regression coefficients, and write e for the n-vector of residuals, such

that ei = yi Xi b.

The vector b is the ordinary least squares (OLS) solution if and only if it is chosen such that the sum of

squared residuals,

n

X

SSR =

ei2 ,

i=1

is at a minimum.

Attaining the minimum SSR can be approached as a calculus problem. In matrix notation, we can write the

SSR as

e0 e

(y Xb)0 (y Xb)

(y 0 (Xb)0 ) (y Xb)

y 0 y y 0 Xb (Xb)0 y + (Xb)0 Xb

y 0 y 2b0 X 0 y + b0 X 0 Xb

The manipulations above exploit the properties of the matrix transpose, namely (A + B)0 = A0 + B0 and

(AB)0 = B0 A0 . In addition we rely on the point that y 0 Xb and (Xb)0 y (which can also be written as b0 X 0 y)

are identical by virtue of the fact that they are 1 1: they are transposes of each other, but transposing a 1 1

matrix gives back the same matrix.

Now take the derivative of the last expression above with respect to b and set it to zero. This gives

2X 0 y + 2X 0 Xb

X 0 y + X 0 Xb

X 0 Xb

= 0

= 0

= X0 y

(In general, the derivative of b0 Cb with respect to b is (C + C 0 )b, but X 0 X is a symmetric matrix and the

derivative simplifies to that shown, 2X 0 Xb.)

If the matrix X 0 X is non-singular (i.e. it possesses an inverse) then we can multiply by (X 0 X)1 to get

b = (X 0 X)1 X 0 y

(1)

This is a classic equation in statistics: it gives the OLS coefficients as a function of the data matrices, X and y.

Orthogonality of residuals and regressors. Heres another perspective. As we said above, the regression

residuals are e = y Xb. Suppose we require that the residuals be orthogonal to the regressors, X. We then

have

X0e = 0

0

X (y Xb) = 0

X 0 Xb = X 0 y

which replicates the solution above. So another way of thinking about the least squares coefficient vector, b,

is that it satisfies the condition that the residuals are orthogonal to the regressors. Why might that be a good

thing? Well, intuitively, if this orthogonality condition were not satisfied, that would mean that the predictive

power of X with regard to y is not exhausted. The residuals represent the unexplained variation in y; if they

were not orthogonal to X that would mean that more explanation could be squeezed out of X by a different

choice of coefficients.

Up to this point we have been concerned with the arithmetic of least squares. From an econometric viewpoint,

however, were interested in the properties of least squares as an estimator of some data generating process

(DGP) that we presume to be operating in the economy. Let us write the DGP as

y = X + u

where u is the error or disturbance term, satisfying E(u) = 0.

of the unknown

From this standpoint we consider the least squares coefficient vector b as an estimator, ,

parameter vector . By the same token our regression residuals, e, can be considered as estimates of the

with a question of particular

unknown errors u, or e = u.

In this section we consider the expected value of ,

equal ? In the following section we examine the variance of

interest being: under what conditions does E()

.

Since, as we saw above, the least squares equals (X 0 X)1 X 0 y, we can write

= E[(X 0 X)1 X 0 y]

E()

The next step is to substitute the presumed DGP for y, giving

E()

E[ + (X 0 X)1 X 0 u]

+ E[(X 0 X)1 X 0 u]

(Since itself is not a random variable, it can be moved through the expectation operator.) We then see that

= in other words, the least squares estimator is unbiasedon condition that

E()

E[(X 0 X)1 X 0 u] = 0

What does it take to satisfy this condition? If we take expectations conditional on the observed values of the

regressors, X, the requirement is just that

E(u|X) = 0

(2)

i.e., the error term is independent of the regressors.

So: the least squares estimator is unbiased provided that (a) our favored specification of the DGP is correct

and (b) the independent variables in X offer no predictive power over the errors u. We might say that the errors

must be uncorrelated with the regressors, but the requirement in (2) is actually stronger than that.

Var(X) = E[X E(X)]2 = E[(X E(X)) (X E(X))]

The variance of a random variable X is the expectation of the squared deviation from its expected value. The

the concept

above holds good for a scalar random variable. For a random vector, such as the least squares ,

is similar except that squaring has to be replaced with the matrix counterpart, multiplication of a vector into its

own transpose.

This requires a little thought. The vector is a column vector of length k, and so is the difference E().

Given such a k-vector, v, if we form v 0 v (the inner product) we get a 1 1 or scalar result, the sum of squares

of the elements of v. If we form vv 0 (the outer product) we get a k k matrix. Which do we want here? The

latter: by the variance of a random vector we mean the (co-)variance matrix, which holds the variances of the

elements of the vector on its principal diagonal, and the covariances in the off-diagonal positions.

Therefore, from first principles,

0

= E E()

Var()

E()

2

= and, substituting in

If we suppose that the conditions for the estimator to be unbiased are met, then E()

0

0

1 0

0

1 0

Var() = E (X X) X y (X X) X y

Then, once again, substitute the DGP, X + u, for y, to get

0

0

1 0

0

1 0

Var() = E (X X) X (X + u) (X X) X (X + u)

0

0

1 0

0

1 0

= E + (X X) X u + (X X) X u

0

0

1 0

0

1 0

= E (X X) X u (X X) X u

h

i

= E (X 0 X)1 X 0 uu0 X(X 0 X)1

(3)

(Note that the symmetric matrix (X 0 X)1 is equal to its transpose.) As with figuring the expected value of ,

we will take expectations conditional on X. So we can write

= (X 0 X)1 X 0 E(uu0 )X(X 0 X)1

Var()

(4)

The awkward bit is the matrix in the middle, E(uu0 ). This is the covariance matrix of the error term, u;

note that it is n n and so potentially quite large. However, under certain conditions it simplifies greatly

specifically, if the error term is independently and identically distributed (i.i.d.). In that case E(uu0 ) boils down

to an n n version of the following,

2

0

0

0 2 0 ,

0

0 2

that is, a matrix with a constant value on the diagonal and zeros elsewhere. The constant 2 reflects the

identical part of the i.i.d. condition (the error variance is the same at each observation) and the zeros off the

diagonal reflect the independent part (theres no correlation between the error at any given observation and

the error at any other observation). Such an error term is sometimes referred to as white noise.

Under the assumption that u is i.i.d., then, we can write E(uu0 ) = 2 I, where I is the n n identity matrix.

And the scalar 2 can be moved out, giving

Var()

= 2 (X 0 X)1 X 0 X(X 0 X)1

= 2 (X 0 X)1

(5)

The somewhat nasty equation (4) therefore gives the variance of in general case, while equation (5) gives

the nice and simple version that arises when the error term is assumed to be i.i.d.

The true error variance, 2 ,

How do we actually get a value for (that is, an estimate of) the variance of ?

2

is unknown, but we can substitute an estimate, s , based on the regression residuals:

Pn

e2

2

s = i=1 i

nk

and our estimated variance is then s 2 (X 0 X)1 provided, of course, that we are willing to assume that the error

term is i.i.d.

- EconometricsUploaded byAl
- Session12 SolutionUploaded byapi-3737025
- Reducing Uncertainty Allometric 2001Uploaded bygruporeddperu
- Attrition 2Uploaded byNaimi Smith
- 2011_Diniz-Filho Et Al_Eigenvector Estimation of Phylogenetic and Functional DiversityUploaded bykadu0204
- Ch 16 Binary Dependent Var'sUploaded byEmma
- maldonado & moreno IASC 2011Uploaded byJorge H. Maldonado
- Topic 4Uploaded bysajidalirafique
- Poverty in Zambia 2Uploaded byHarry Ng'ámbi
- Demand EstimationUploaded byMargaret Divya
- hChap8Uploaded byneeraj_kamboj2193
- W8 - AUTOC-6pPgUploaded bykdinner1234
- Tests of Normality of Regression ErrorsUploaded byrunawayyy
- Des Aigoo Ls Bee FinalUploaded bySaina Chuh
- Saxena Cs4758 Lecture4Uploaded byIvan Avramov
- Econometrics HW VUploaded byAnonymous xjWuFPN3i
- 2Uploaded byKunal Rathod
- 4614ijcsa01Uploaded byAnonymous lVQ83F8mC
- Chumacero OLSUploaded byrjalon
- 21Week7Lect3Uploaded byBob
- 1364Uploaded byTriWidyazN
- 1 4 ExercisesUploaded byusman3686
- OutputUploaded byEdo Destrada
- ec444-lec10Uploaded byAllan Mugenya
- PlmUploaded byrerere
- Linear Regression for Machine LearningUploaded byJohn Green
- ASS 2 - DraftUploaded byLeslie Michael Chang
- E viewsUploaded bysaba
- 200110090166_l_9660Uploaded byImel Destiara
- Advanced Econometrics.pdfUploaded byJosé Armando Cobián Álvarez

- Time series solution manualUploaded byA.s. Charan
- Doc1Uploaded byRashidAli
- pak.1200.aliUploaded byRashidAli
- EViewsGuide.pdfUploaded byRashidAli
- Parreńas. Long Distance IntimacyUploaded byRodessa May Marquez
- j.1360-3736.2005.00223.xUploaded byRashidAli
- donor.remUploaded byRashidAli
- Syed Rashid AliUploaded byRashidAli
- indirectutility.pdfUploaded byRashidAli
- 2010.08Uploaded byRashidAli
- Template IJARUploaded byRashidAli
- For Each ArtUploaded byRashidAli
- Demand ExamplesUploaded byRashidAli
- Variance TemplateUploaded byRashidAli
- Indirect Utility exampleUploaded byRashidAli
- fpm-s-08Uploaded byRashidAli
- FPM-S-08Uploaded byUsama Qayyum
- llwp36Uploaded byRashidAli
- Parrenas05Uploaded byRashidAli
- TaxUploaded byRashidAli
- tax (1)Uploaded byRashidAli
- A4B1C351627E42248CD7F310DA8A1352Uploaded byRashidAli
- 15 Types of Regression You Should KnowUploaded byRashidAli
- Regression AnalysisUploaded byRashidAli
- FDI and Economic Growth in PakistanUploaded byRashidAli
- Acceleration PrincipleUploaded byRashidAli
- Lecture1.pdfUploaded byRashidAli
- Econtrics RESOURCEUploaded byrambori

- PLS Path Modeling journal.pdfUploaded byAdam RusNugroho
- US Federal Reserve: ifdp597Uploaded byThe Fed
- MONETARY POLICY AND INFLATION CONTROL IN RWANDA; 2006-2015Uploaded byIJAR Journal
- Unitroot Coint Causality EViewsUploaded byMin Fong Law
- econometrics-project (1).docxUploaded byMuhammad Ali Khoso
- 1-s2.0-S0161893812000725-mainUploaded byZaheer Khan
- Tugas Anreg 1Uploaded bySandy Luke Nugroho
- Macroeconomic Effects of Financial Stress and the Role of Monetary Policy a VAR Analysis for the Euro AreaUploaded byMădă Cionilă
- Applied Bayesian Econometrics for Central Bankers Updated 2017Uploaded byAugusto de Lima
- MGARCH NotesUploaded byGao Jiong
- Free Mit PptUploaded byMohit Pipal
- Lutkepohl Handout 2011[1]Uploaded bypartygerm
- Financial Conditions Indexes for Asian EconomiesUploaded byAsian Development Bank
- Multivariate Laboratory Exercise IIIUploaded byAl-Ahmadgaid Asaad
- urcaUploaded byjan9276
- Dynamic Linkages Between China and US Equity MarketsUploaded byandydo611
- Asymtotic Theory for VARUploaded byAlexis Tocqueville
- Solomon L QP - S1 EdexcelUploaded byabhay
- Solutions for Chapter 8Uploaded byreloadedmemory
- BMR.pdfUploaded byAnonymous eFnAMq6q
- Papers about Extreme Value Distribution by Azais, Wschebor et al.Uploaded byadaniliu13
- Fdi and Current Account Balance in IndiaUploaded byAtiq Ur Rehman Sipra
- An Empirical Study of the Relationships between CO2 Emissions, Economic Growth and Openness.pdfUploaded bySiddharth Venkataraman
- Estimation and Hypothesis Testing of Cointegration Vectors in Gaussian Vector Auto Regressive ModelsUploaded byMenaz Ali
- Recipes for State Space Models in R Paul TeetorUploaded byalexa_sherpy
- JEE Main Sample Paper (5)Uploaded byAnweshaBose
- Twin DeficitUploaded byAnh Nguyen
- Eviews_7.0_Manual.pdfUploaded byBimo Sakti
- 130322_cziraky.pdfUploaded byLui
- genera;ied least square.pdfUploaded bymanishaamba7547