The Likelihood Ratio, Wald, and Lagrange Multiplier Tests: An Expository Note Author(s): A.

Buse Source: The American Statistician, Vol. 36, No. 3, Part 1 (Aug., 1982), pp. 153-157 Published by: American Statistical Association Stable URL: http://www.jstor.org/stable/2683166 . Accessed: 20/01/2014 10:09
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp

.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

American Statistical Association is collaborating with JSTOR to digitize, preserve and extend access to The American Statistician.

http://www.jstor.org

This content downloaded from 137.205.241.10 on Mon, 20 Jan 2014 10:09:20 AM All use subject to JSTOR Terms and Conditions

The LikelihoodRatio, Wald, and Lagrange MultiplierTests: An Expository Note

A. BUSE*
likelihoodmethof maximum Within the framework ods the basic logic of these testsis developed by means of a simplediagram,and the connectionof these tests to the standardlikelihoodratio (LR) testis also estabdevice. Using the inlishedby the same diagrammatic methodwe show sights generatedby our diagrammatic is easily deof these teststatistics thatthe distribution is quadratic. Our exposition rivedif the log-likelihood to theheurcomplement can be viewedas thegeometric of these teststhathas been givenby isticpresentation Silvey(1970, pp. 108-122). It can also be consideredas articles by Breusch to the recentsurvey complementary and Pagan (1980) and Engle (1981). To introduceour approach to these testswe applyit that first to theLR testand givethattesta visualformat helpful.Considera we think is novel and pedagogically ofmaximum in whichthediscussion classroomsituation likelihoodestimationhas been completed and the inhas just explainedthebasic logicoftheLR test. structor that That is to say, he willhave advanced the argument takingthe ratio of likelihoods with and withoutthe imposedis a plausible restrictions of the nullhypothesis the basic equation stating basis fora test.Furthermore, of the teststatistic, distribution the asymptotic LR = 2(logL (0) - logL (0))
_

thisnotegivesan intuitive By means of simplediagrams theLagrangemultiplier, ratio, the likelihood accountof is that It also demonstrated and Wald testprocedures. is quadraticthenthethree function ifthelog-likelihood identicaland have x2disare numerically teststatistics forall sample sizes underthenullhypothesis. tributions KEY WORDS: Likelihood ratio; Lagrange multiplier Chi-squared test; Wald test; Quadratic log-likelihood; distribution.

The level of rigorthat is desirable or attainablein course will depend on the teachingof an econometrics thepurposeof thecourse and thelevel of mathematical it. In of thestudents taking and statistical sophistication econometrics introductory the case of undergraduate rigor courses the level of mathematicaland statistical willnecessarily be limited.At thegraduatelevel,on the otherhand, thereis always a need for a nonspecialist to applicais primarily course in whichthe orientation tions and rigormust give way to the need for broad In coverageof varioustechniquesand theirlimitations. in graduate courses, the either case, but particularly to incorteacherof such a course has a responsibility porate recent developmentsand present them in an accessible form.Yet it is preciselythe most recentdethat are available only in formal,rigorous velopments at thejournallevel. Thus, thereis a shortpresentations termneed to bridgethe gap betweenthe journal literathat ture and the distilledversionsof that literature appear in the textbooks.The purposeof this ultimately literature note is to build such a bridgeto the emerging on the use of the Wald (W) and Lagrange multiplier It shouldbe added here in econometrics. (LM) statistics in thisnote reflects thatthe emphasison econometrics the teachingexperienceof the authorand is not meant to exclude readers fromotherfields. We feel thatthe expositionis applicable to any otherfieldwhere these his testsare used and we invitethe readerto substitute is used. thetermeconometrics fieldwhenever particular since thereare manylevels of aspiration Furthermore, this note for an introductory course in econometrics, mathehave the necessary willassumethatthestudents maticaland statistical backgroundneeded fora course taughtat the level of Kmenta (1971).
of *A. Buse is Professor,Departmentof Economics, University Alberta, Edmonton, Alberta, Canada T6G 2H4. This is a revised of New South versionof a note written in May 1980 at the University Wales while the author held a Leave Fellowshipfromthe SSHRC. to Ron Bewley,David The authoris indebtedforadvice and criticism Giles, Adrian Pagan, and Eric Sowey.

X2(g)'

(1)

forgivendistance0 - 00,the greater ?/2LR.Conversely, thecurvature C(0), thelargerwillbe thedistance?2LR. It is these characteristics thatprovidethe keyto a diaof the Wald test. Instead of conderivation grammatic in log-likelihoods, thistesttakes sidering thedifferences the intuitively withthe appealing approach of working of squared distancebetween0 and 00. Large deviations
'Our use of the notionof curvature is intendedto be informal and intuitive. However,sincethefirst derivative is zero at 0 = 0, C(0) does in factmeasure curvature withrespectto arc length.

willbe thedistance maximum, logL(6), and thelarger

will have made an appearance, where 0 is the unrestricted estimate of the population vector, 0 is the restricted estimate,and g is the numberof restrictions Suppose thatthevector imposedbythenullhypothesis. 0 consistsof onlyone element,so that0 = 00,where00 in a simplenullhypothesis and is thevalue of 0 specified the alternativehypothesisspecifiesthat 0 # 00. If we now plot the log-likelihood thenthe value of function, fromFigure 1 by notingthe ?/2LRcan be read directly values of logL (0) at 0 and 00. will In Figure 1 we note that the distance, ?/2LR, ofthe dependon thedistance,0 - 00,and thecurvature' function, whichwe denote by C(0) and log-likelihood whichis definedby the absolute value of (d2logL)/d02 evaluated at 0 = 0. Given C(0), the largerthe distance between0 and 00,the further willlogL (0o) be from the

?) The AmericanStatistician, August1982, Vol. 36, No. 3, Part 1

153

This content downloaded from 137.205.241.10 on Mon, 20 Jan 2014 10:09:20 AM All use subject to JSTOR Terms and Conditions

Log L (9)
Log L (0) -----

00 r<~

functional restrictions imposed by Ho on the k-vector 0 (k > g), then asymptotically W = (r(6)]'[RI(0) 'R '] '[rQ()] - X2(g), (4) where R is the g x k matrix of partial derivatives ar(0)/&0,evaluated at 0. Large values of W are generated by large deviationsof r(6) away fromzero (the hypothesized value for the restrictions) and the deviationsare weighted by a matrix involving the curvature of the log-likelihood. Equation (4) is a quadraticform in the vectorr(O) and it is clear that(3) is just a special case with r(0) linear (0 - 0o= 0) and R the identity matrix. The Lagrange multiplier testalso involvesthe curvature of the log-likelihoodfunction but the basic idea behind the test focuses on the characteristics of the log-likelihood ofthenullhypothewhentherestrictions sis are imposed. Althoughthe name of the teststatistic derives from a restrictedmaximum likelihood estimationproblemsolved by the Lagrangianmethod,the basic idea of the testis seen most readilyin the equivalentform of the "efficient score" statistic; see Breusch and Pagan (1980, p. 240). We will continueto use the termLagrange multiplier testeven thoughour account is based on the score formulation and despite the fact thatthe score form(Rao 1948) has historical precedent over the Lagrange form(Aitchison and Silvey 1958). has become so firmly imThe Lagrange terminology bedded in the econometrics thatto press for literature the score terminology would be futile.For those who is more workin fieldsin whichthe score terminology commonwe again invitethemto make the appropriate substitution. To proceed withthe score approach we note thatif is true,therestricted thenullhypothesis likemaximum willbe near theunrestricted lihoodestimates estimates. estimates maximize the logSince the unrestricted likelihood,they satisfythe equation S(6) = 0, where S(0) = dlogL/d0, and one can obtain a measureof the oftherestricted failure to reachthemaximum estimates by evaluatingthe extentof the departureof S(0) from estimate. In our zero, where 0 is again the restricted to simpleexample 0 = 00, and since we are indifferent thesignoftheslope, we can beginbysuggesting [S (00)]2 as a teststatistic.2 However, as in the case of the Wald statistic twodata setscan generatethesame slope ofthe butone set has 00closerto themaximum log-likelihood, of the log-likelihood.To get around this difficulty we the squared slope by the curvature, but in again weight thiscase the greaterthe curvature the closer will 00be to the maximum,so that if we weightby the inverse will be generC(00)-', small values of the teststatistic ated if 00 is close to the maximum.The point is illus2Thatthescoreand Lagrangetestare equivalentcan readily be seen by examining the constrained maximization problemassociated with the Lagrange approach. Thus if 4 = logL(O) - X(O - Oo),whereX is the Lagrange multiplier, then the first order conditionson the Lagrangian + yieldS(O) = X and 0 = 00 and hence S(0O) = X. Thus large values of the slope are the same as large values of the Lagrange multiplier and both measure the cost of imposingthe restriction.

Log L (00) L-----'

Figure 1. The LikelihoodRatio Test 0 from00 are taken as evidence that the data do not confirm thenullhypothesis. However,givenour earlier observation about curvatureof the log-likelihood, the squared distance (O - Oo)2 must be weightedby C(0) because two sets of data might produce the same value of (O - Oo)2, yet withone set less favorableto the hypothesis(fromthe perspective of the LR test) because the curvatureof the log-likelihoodis greaterand the value of logL (00) is correspondingly muchfurther away from the maximum. This pointis illustrated in Figure2, where both sets of data generate the same value of o)2but forcase A the greatercurvature 0produces a much smallervalue of logL (00). A Wald statistic can now be definedas W = (^ - 00)2C(^), (2) whichunderHo is asymptotically distributed as x2with one degree of freedom(as is the LR statistic).Large values ofW lead to the rejectionof the nullhypothesis. Equation (2) is not the usual formof the Wald statistic, whichis W = (0 - Oo)21(), (3) where1(0) = E((d2logL)1d02), theinformation matrix, so thattheweighting of the squared distanceis in terms of average curvature. The two statistics are, however, asymptotically equivalentbecause C (0) is a consistent estimator of 1(0). the Wald statistic forthe simple Having constructed case, the studentshould now be less daunted by the general formgiven below. If r(O) = 0 is a vectorof g

Log L (9)
r l I l

Log L (9)

--------

Log LB (90) - - - - - - - -Log LA (I)

Figure 2. The WaldTest

154

?) The AmericanStatistician, August1982, Vol. 36, No. 3, Part I

This content downloaded from 137.205.241.10 on Mon, 20 Jan 2014 10:09:20 AM All use subject to JSTOR Terms and Conditions

tratedin Figure 3, where the data set A has greater generatea smaller curvature at 00and should therefore because 00 is nearerthe maxvalue of the teststatistic imumof its log-likelihood. LM = [S (00)]2C (00) -f, (5) as a x2variablewith asymptotically whichis distributed one degreeof freedom.As in the case of theW statistic fortheLM statistic replacestheactuthestandardform by its expectedvalue so that al curvature
creo que information matrix es estimate de la curvatura

LM = [S(00)]21(0O)-'.

(6)

The generalization to the vectorversionis a straightanalog of (6) thatspecifiesthatasymptotically forward LM = S(6)'I(6)-1S(6) _X2(g) (7) It is clear thatlargevalues of LM leadingto rejectionof will occur when the slope vector the null hypothesis fromzero. S(O) departssubstantially Looking back at Figures 1-3 we can note that the theyrein the kinds of information three tests differ and quire. The LR test requires both the restricted estimatesof the parameter.Since the W unrestricted estimates and theLM test testuses onlytheunrestricted the uses onlythe restricted estimates,computationally LR testis the mostdemanding.There is a directphysical analogythatcan be made foreach of thesetestsin termsof measuringthe verticaldistance between the top and a preassignedpoint on a hill. The LR test theheight at both thedistancebyevaluating determines points. The W testworksfromthe top of the hill and triesto establishthe distanceto a pointlowerdown by distancebetweenthe top and lookingat the horizontal the preassignedpoint and at the rate at whichthe hill the top. Finally,the LM testgoes to curvesaway from how farit the preassigned point and triesto determine the slope of the hilland the is to the top byconsidering rate at whichit is changing.While these physicalanatheydo providea directvisual logies have limitations, for students,and experience suggests representation means of imparting thatsuch analogies are an effective the basic ideas. The careful teacher will, of course, and make suitablecautionary pointout theselimitations in remarks.For example, one should note thatimplicit
Log L (9)
A B

the diagrammatic presentationis the assumptionthat theargument is essentially a "local" one and in thecase of log-likelihoods withmultiplemaximaone has to asclose to the global sume thatthe null is "sufficiently" maximum. The hill-measuring analogycan be used to gain some of the teststatisadditionalinsight into the distribution tics that, up to this point, have been asserted to be x2.We now considerthisquestion. asymptotically as a While we have drawnsymmetric log-likelihoods matter of drafting convenience,symmetry is not essential to our visual argument.If in additionwe assume is quadraticthenwe can exploit thatthe log-likelihood In thiscase the thehill-measuring analogyeven further. W and LM methodsmustgive the same numerical values as the LR method that measures the verticaldistance directly. That thisis so followsfromthe factthat to determine a quadratic threeparameters are sufficient uniquely. In both the W and LM approaches we use threepieces ofinformation to determine the (implicitly) parametersof the quadratic and hence the numerical value ofthevertical distance.Thus in theW case we use theheight of the hillat 0, logL(O), thevalue of thefirst derivative, S(o) = 0, and second derivative, C(o), whereasin the case of the LM we use the heightof the hill at 00, logL(0O), the value of the firstderivative, S(00), and second derivative, C(00). Given thatthe nuare the same, it follows mericalvalues of the statistics This does not, of thattheyhave the same distribution. of the distribution, about the form course,say anything but the quadraticlikelihoodcan again be exploitedfor thispurpose. textbookproblemof testing Considertheelementary ? ,o fora sample the null hypothesis ,u= ,uoagainst pu with known size n drawn froma normal distribution Thus variance,whichforconveniencewe take as unity. Xi - N(,u, 1) and logL(,u) = -(n/2)(1og2rr) a quadraticin ,u. Now
ddlogL
?/21(X, -

EGs

p)2
(8)

- L) = n (X - AL) doL= Y_(X,

-

Log L (Os) _,

-n (9) so thatfrom(9) it is clear thatthe curvature is constant and thisimpliesequalitywiththeexpectedinformation. The maximum likelihoodestimator is, of course,, = and a littlealgebra yieldsthe LR statistic as
dp.2
-

LR = 2[logL (,)
= E(Xi_R-

logL (p.o)]
-X)2

O)2-E(Xi

LR = n(X puo)2.
~S (9) The Wald statistic is givendirectly as
W
=

(10)
(11)

(~

p.0)2C(F)

n(X p.)2
-

Test Figure 3. TheLagrangeMultiplier

and since from(8) dlogL (p.)/dp. = n(X immediately that

p.) we have

?) The AmericanStatistician, August1982, Vol. 36, No. 3, Part1

155

This content downloaded from 137.205.241.10 on Mon, 20 Jan 2014 10:09:20 AM All use subject to JSTOR Terms and Conditions

-' = n (X LM = S (puo)2C(puo)

pLo)2.

(12)

as log-likelihood and therestricted

logL(3) = K-V2[i 'fl-'
.

ourhill-measuring from thevalAs predicted analogy, are identical. ues of all theteststatistics Furthermore,
sincewe knowthatX
-

(21)

squareof a standardized normal and henceis variable distributed as a x2variable with offreedom. one degree Thusinthis particular example arex2 theteststatistics for allsample x2. sizesandtherefore also asymptotically Our secondexample themultidimensional considers extension of theprevious thestanexampleby taking dardeconometric of problem oftesting forthevalidity a setoflinear restrictions. In thecontext ofthelinear weassume model of that thevariance-covariance matrix thedisturbances is known, andthis reduces theproblem
to thequadraticcase. Let y = Xj + u withu - N (0, fl)

n -1), each statistic is the N(puo,

between (20) and (21) willbe used to The difference rewrite but theLR statistic wefirst (21) bynoting define that
u = y -X = u - XC -'R'(RC -'R')(R -r), (22)

for13 from(18). We can wherewe have substituted rewrite therefore (21) as

logL(O)
=

- 2U 'Qn-lu

-112(R 0- r)'(RC-1R ')-1(R',-r),

(23)

wherewe have used the factthatu'Q-'X = 0. The LR

where y is n x 1, X is n x k, 13 is k x 1, andQ is n x n and known. are The null and alternative hypotheses given by
Ho: Rp = r ,B Hl: Rp (13)

as is nowgiven statistic

LR = 2[logL (,B)- logL(I)J = (R pJr)'(RC -'R ')-'(R ,Br). (24)

from follows (4) and is directly The Wald statistic as written

W = (R
-

R isg x k andrisg x 1,both where ofwhich areknown matrices. The log-likelihood, a quadratic in the vector 1, is given by logL(,B)= -(n/2)log(27r)-

r)'(RC-'R')-'(R13

r).

(25)

we notethat from theLM statistic (15) we To obtain have

S(,B) = R'(RC-'R')-'(Rp3 LM = S(13)'C'-S(13) = (R13- r)'(RC-1R'l)-(R13
-

- r)

(26)

l/21oglfl - X 3)

(/2y - X P)'Q-(y

-. Thus, we havesubstituted after (18) for

r). (27)

K - 2(y - X P)'h-'(y - X 3), (14) theconstant K isimplicitly where defined. From (14) we get
=

a13

xtf-'Y - X'Q-1X

(15)

and 32logL -(X'f-hX).

-

(16)

and Givenfl, the Hessian(equation(16)) is constant are again the"curvature" matrix and theinformation equal. In orderto construct theLR statistic we need both the unrestricted and the restricted estimates. The is given from former directly (15) as
,= (X'Q 1X))1X'Qlf y

(17)

p. 285) as

thelatter from whereas can be obtained theusualconstrained maximization problem (see, e.g., Theil 1971,

13 = 13C-C1R'(RC-'R-1)-1(R
with
C =(X'fh1X)
.

3 - r)

(18) (19)

Defining = y - X,B and ui = y - X,Bwe canwrite the u~ unrestricted log-likelihood as

logL(I)
156
=

?iVfhQi

(20)

ifthelog-likelihood a testof linear restrictions is quais not included dratic.The proofof thisproposition andhasthesamestructure becauseitis straightforward In fact, as oursecond for thelinear modelthe example. on theequality ofthestatistics result can be derived by a few substitutions theresults inBreusch using judicious thetimethatthispaperwas under re(1979). During thepaperbyEngle(1981),whoalso viewwe received and showsthatit also holds provesthisproposition oftheparameters when are restricted by onlya subset A generalization thenullhypothesis. to non-linear reon thevector13 strictions is possible(givenfl), butit thatthe restricted likelihood estimaximum requires be obtained from thesecond ofthemethod mate iterate of scoring. be the thefirst iterate must Furthermore, method is unrestricted estimator of 1. Thisestimation thereto a minimum x2approach in which equivalent abouttheunrestricted estimates arelinearized strictions estimation using (18). (Formore followed byrestricted details, see Buse 1981). oftheteststatistics oftheequality The implications Since are straightforward. for tests of hypotheses in the is a quadratic form ,B3 N(13,C'I), each statistic R 13- r so thatappeal to the relevant normal vector is distributed as a theorem givesus thateach statistic

thatLR = W = LM for cases of the generalproposition

bothourexamples can be shown to be special In fact,

shownagain thatLR = W = LM. We have therefore

August1982, Vol. 36, No. 3, Part 1 (? The AmericanStatistician,

This content downloaded from 137.205.241.10 on Mon, 20 Jan 2014 10:09:20 AM All use subject to JSTOR Terms and Conditions

x2(g) variable, where g is the numberof restrictions case would requireestiimposedby Ho. A less artificial matrixof the dismation of the variance-covariance turbances,whichwould destroythe equality between would only be x2(g) and the distributions the statistics In thiscase the linear model generates asymptotically. noted by inequalityW?- LR - LM first the systematic Berndt and Savin (1977), which continues to attract see forexample literature; in the econometric attention Evans and Savin (1980) and Fisherand McAleer (1980). of these inOur results provide an interpretation equalities in termsof the shape of the log-likelihood function.While the directionand order of the inequalities cannot be predictedfromour geometrical it is clear the inequalitiesarise because of arguments, fromthe quadratic the departureof the log-likelihood shape. Whetherone can formalizethe extentof this intothe genesis insight departureand thusgain further termsin a of the inequalities by takinghigher-order is an open Taylorseriesexpansionof the log-likelihood research. question to be pursuedin future leads to another Our approachto these teststatistics forthe W format researchproblem.The conventional and LM test (equations (3) and (6)) uses the informathantheHessian ofthelog-likelihood rather tionmatrix to conjecture (equations (2) and (5)). It is interesting ratherthan the matrix that the use of the information on the powerof effect Hessian could have a significant samples. Insofaras theLR theLM and W testsin finite by the hillin testis based onlyon the data represented in theW and matrix Figure1, theuse oftheinformation because it does not represent LM testsis inappropriate hill of data. It is conof that particular the curvature matrixrather ceivable thatthe use of the information than the Hessian is the cause of the verysubstantial underestimates of Type I errorthathave been observed in the use of the LM statisticfor tests of heteroscedasticity;see, for example, Breusch and Pagan (1979) or Buse (1980). That usingobservedinformation can (the Hessian) as opposed to expected information have a substantial impacton the size of the Wald testis documentedin Efron and Hinckley(1978). However, of ancillary theviewpoint theydealt withtheissue from and we plan a systematic of this investigation statistics used in conjecture for the kinds of models typically econometrics. appropriAlthoughthese issues are not particularly

to find thatour note,itis gratifying ate foran expository device leads to the resimple diagrammatic relatively This, we believe, is in itselfa good search frontier. of its virtuesand potential. recommendation
[ReceivedJune 1981. Revised December 1981. ]

REFERENCES
AITCHISON, J., and SILVEY, S. D. (1958), "MaximumLikelihood Annals of MathSubject to Restraints," of Parameters Estimation 29, 813-828. ematicalStatistics, AmongCriteBERNDT, E. R., and SAVIN, N. E. (1977), "Conflict Linear Regression ria forTestingHypothesesin the Multivariate 45, 1263-1278. Model," Econometrica, BREUSCH, T. S. (1979), "ConflictAmong CriteriaforTestingHy47, 203-208. potheses: Extensionand Comment," Econometrica, BREUSCH, T. S., and PAGAN, A. R. (1979), "A Simple Test for and Random CoefficientVariation," EcoHeteroskedasticity 47, 1287-1294. nometrica, BREUSCH, T. S., and PAGAN, A. R. (1980), "The LagrangeMulin EcotiplierTest and Its Applicationsto Model Specification Reviewof Economic Studies,47, 239-254. nometrics," Some PreBUSE, A. (1980), "Tests forAdditiveHeteroskedasticity: Monte Carlo Results," Paper presentedto the Fourth liminary World Congress of the EconometricSociety, Aix-en-Provence. (1981), "Pooling Time Series and Cross SectionData: A Minimum Chi-Squared Approach," The Manchester School, 49, 229-244. EFRON, B., and HINKLEY, D. V., (1978). "Assessing the Accuracyof the MaximumLikelihoodEstimator:Observed VersusEx65, 457-487. Biometrika, pected Fisher Information," ENGLE, R. F. (1981), "Wald, LikelihoodRatio, and LagrangeMuleds. Testsin Econometrics,"in Handbook of Econometrics, tiplier Amsterdam:NorthHolland. Z. Grilichesand M. Intriligator, EVANS, G. B. A., and SAVIN, N. E. (1980), "ConflictAmong the CriteriaRevisited:The W, LR and LM Tests," Revised versionof a paper presented to the Fourth World Congress of the Econometric Society,Aix-en-Provence. FISHER, G., and McALEER, M. (1980), "Principlesand Methods Models," Discussion Paper No. 400, in the Testingof Alternative Queen's University. New York: MacKMENTA, J. (1971), Elementsof Econometrics, millan. RAO, C. R. (1948), "Large Sample Tests of StatisticalHypotheses ConcerningSeveral ParameterswithApplicationsto Problemsof of theCambridgePhilosophicalSociety, Estimation",Proceedings 44, 50-57. Harmondsworth: PenInference, SILVEY, S. D., (1970), Statistical guin Books. THEIL, H. (1971), Principlesof Econometrics,New York: John Wiley.

(? The AmericanStatistician, August1982, Vol. 36, No. 3, Part 1

157

This content downloaded from 137.205.241.10 on Mon, 20 Jan 2014 10:09:20 AM All use subject to JSTOR Terms and Conditions