THREE ECONOMETRIC METHODOLOGIES: A CRITICAL APPRAISAL! Adrian Pagan University of Rochester Abstract. Three econometric methodologies, associated respectively with David Hendry, Christopher Sims and Edward Leamer have been advocated and prac- ticed by their adherents in recent years. A number of good papers have appeared about each methodology, but little has been written in a comparative vein. This paper is concerned with that task. It provides a statement of the main steps to be followed in using each of the methodologies and comments upon the strengths and weaknesses of each approach. An attempt is made to contrast and compare the techniques used, the information provided, and the questions addressed by each of the methodologies. It is hoped that such a comparison will aid researchers in choosing the best way to examine their particular problem. Keywords. Econometric methodologies; Hendry; Sims; Leamer; extreme bounds analysis; vector autoregressions; dynamic specification. Methodological debate in economics is almost as long-standing as the discipline itself. Probably the first important piece was written by John Stuart Mill (1967), and his conclusions seem as pertinent today as when they were written in the 19th century. He observed that many practitioners of political economy actually held faulty conceptions of what their science covered and the methods used. At the same time he emphasized that, in many instances, it was easier to practice a science than to describe how one was doing it. He finally concluded that a better understanding of scope and method would facilitate the progress of economics as a science, but that sound methodology was no? a necessary condition for the practice of sound methods. ‘Get on with the job’ seems the appropriate message. It is interesting that it was not until the Sth World Congress of the Econometric Society in 1985 that a session was devoted to methodological issues. There are good reasons for this. Until the mid-1970’s it would have been difficult to find a comprehensive statement of the principles guiding econometric research, and it is hard to escape the conclusion that econometricians had taken to Mill’s injunction with a vengeance. Even the debate between ‘frequentists’ and ‘subjec- tivists’ that prevailed in statistics was much more muted in econometrics. It is true that there was a vigorous attempt to convert econometricians to a Bayesian approach by Zellner (1971) and the ‘Belgian connection’ at CORE (see Dréze and Richard, 1983). But this attempt did not seem to have a great impact upon applied research. All of this changed after 1975. Causes are always harder to isolate than effects, (0950-0804/87/01 0003-22 $02.50/0 © 1987 A. Pagan JOURNAL OF ECONOMIC SURVEYS Vol.1, No.1 34 PAGAN but it is difficult to escape the impression that the proximate cause was the predic- tive failure of large-scale models just when they were most needed. In retrospect it seems likely that the gunpowder had been there for some time, and that these events just set it off. Most, for example, will know Ed Leamer’s (1978, p. vi) account of his dissatisfaction with the gap between what he had been taught in books and the way practitioners acted, and it seems likely that many others had felt the same way about the type of econometrics then prevalent. But these misgivings were unlikely to have any impact until there was evidence that there was something to complain about. Since 1975 we have seen a concerted attempt by a number of authors to build methodologies for econometric analysis. Implicit in these actions has been the notion that work along the prescribed lines would ‘better’ econometrics in at least three ways. First, the methodology would (and should) provide a set of principles to guide work in all its facets. Second, by codifying this body of knowledge it should greatly facilitate the transmission of such knowledge. Finally, a style of reporting should naturally arise from the methodology that is informative, succinct and readily understood. In this paper we look at the current state of the debate over methodology. Three major contenders for the ‘best methodology’ title may be distinguished. 1 will refer to these as the ‘Hendry’, ‘Leamer’ and ‘Sims’ methodologies, after those individuals most closely identified with the approach. Generally, each pro- cedure has its origins a good deal further back in time, and is the outcome of a research programme that has had many contributors apart from the named authors above. But the references—Hendry and Richard (1982), Leamer (1978) and Sims (1980a)—are the most accessible and succinct summaries of the material, and therefore it seems appropriate to use the chosen appellations. In- evitably, there has been some convergence in the views, but it will be most useful to present them in polar fashion, so as to isolate their distinct features. 1, The ‘Hendry’ methodology Perhaps the closest of all the methods to the ‘old style’ of investigation is the ‘Hendry’ methodology. It owes a lot to Sargan’s seminal (1964) paper, but it also reflects an oral tradition developed largely at the London School of Economics over the past two decades. Essentially it comprises four ‘steps’. (i) Formulate a general model that is consistent with what economic theory postulates are the variables entering any equilibrium relationship and which restricts the dynamics of the process as little as possible. (ii) Re-parameterize the model to obtain explanatory variables that are near orthogonal and which are ‘interpretable’ in terms of the final equilibrium. Gii) Simplify the model to the smallest version that is compatible with the data (‘congruent’). (iv) Evaluate the resulting model by extensive analysis of residuals and predic- tive performance, aiming to find the weaknesses of the model designed in the previous step.THREE ECONOMETRIC METHODOLOGIES 5 Steps (i) and (ii) Theory and data continually interplay in this methodology. Unless there are good reasons for believing otherwise, it is normally assumed that theory suggests which variables should enter a relationship, and the data is left to determine whether this relationship is static or dynamic (in the sense that once disturbed from equilibrium it takes time to re-establish it). It may help to understand the various steps of Hendry’s methodology if a par- ticular example is studied. Suppose that the investigator is interested in the deter- minants of the velocity of circulation of money. Let m;, be the log of the money supply, pr be the log of the price level and y; be the log of the real income. Theoretical reasoning suggests that, for appropriately defined money, m- pr ¥; should be a function of the nominal interest rate (/;) along any steady state growth path. With i,=log(/), we might therefore write mi — pi— y= 6i; where the starred quantities indicate equilibrium values. Of course equilibrium quantities are not normally observed, leading to the need to relate these to actual values. For time series data it is natural to do this by allowing the relations between the variables m:, pr, ¥: and i; to be governed by a dynamic equation of the form m= 3 agmest ¥ opps Seas 3S di oO int jo imo Jeo The first step in Hendry’s methodology sets p, g, r and sto be as large as prac- ticable in view of the type of data (generally p = r=s=5 for quarterly data), and to then estimate (1). This model, the general model, serves as a vehicle against which all other models are ultimately compared. Now (1) could be written in many different ways, all of which would yield the same estimates of the unknown parameters, but each of which packages the in- formation differently and consequently may be easier to interpret and under- stand. Generally, Hendry prefers to re-write the dynamics in (1) as an ‘error correction mechanism’ (ECM). To illustrate this point, the simple relation Xe = OX) + Dox! + DiXi-1, (2a) where x; is the equilibrium value of x;, has the ECM AX, = (a= 1) (41-1 = X7-1) + DoAX! + (a 1 + bo + bi) X-1 = (a= I) (Xr = x11) + DoAX, (2b) since steady-state equilibrium in (2a) implies x= ax+box+bix or @ + bo + by = 1. Although (2b) is no different to (2a), Hendry prefers it since Ax? and (x;-1— x7-1) are closer to being orthogonal and he is able to interpret its elements as equilibrium (Ax/) and disequilibrium (x;-1 — x7-1) responses. Moving away from this simple representation we can get some feeling for the type of equation Hendry would replace (1) with by assuming that mm, adjusts within the period to p,, making the log of real money m,— p, the natural analogue of x; in 2(a). The equilibrium value is then x7= ys + i, and by appeal6 PAGAN to (2b) it is clear that a re-formatted version of (1) would involve terms such as AY Mig and (21-1 — 1-1) = (OMe = Pema — Yea — Bir 1) = (Mew = Pro — Ir) — 6i;-1. Since (m— p— y),-1 is related to the lagged velocity of circulation, it may be easier to interpret this re-formulated equation. Terms such as (mm — p - y):-1 frequently appear in studies of the demand for money by Hendry and his followers. For example, in Hendry and Mizon (1978), the following equation appears Am — p): = 1.61 + 0.21Ay, — 0.814 + 0.26A(m — p)r-1 — 0.40A p; ~ 0.23(m — p— y)-1 + 0.61i;-4 + 0.141 ~4, where I have replaced log(1 + i,) with — ir. Thus, steps (i) and (ii) demand a clear statement of what the variables in the equilibrium relation should be, as well as a choice of parameterization. Hendry (1986) provides what is currently the most detailed explanation of his second step, but even a perusal of that source leaves an impression of the step being more of an art than a science, and consequently difficult to codify. To some extent the problem arises since Hendry tends to blur steps (ii) and (iii) in his applied work, with the re-formatted equation sometimes seeming to derive from an inspection of the parameter estimates in (1). In those cases (1) is both simplified and re- arranged at the same time. The idea of beginning with a general model as the benchmark against which others might be compared seems only common-sense, but there is little doubt in my mind that it was a minority view in the 1960’s (and may still be). One fre- quently saw (and sees) explicit rejection of this step on the grounds that it was impossible to do because economic variables were too ‘collinear’, with no attempt made to discover if there was even any truth in that assertion for the particular data set being used.? Over many years of looking at my own and students’ empirical studies, I have found the rule of starting with a general model of fundamental importance for eventually drawing any conclusions about the nature of a relationship, and cannot imagine an econometric methodology that did not have this as its primary precept. As will be seen, all the methodologies analysed in this paper ascribe to that proposition. Step (iti) The first two steps in the methodology therefore seem unexceptionable. It is in the third that difficulties arise. These relate to the decision to simplify (1) and the reporting of this decision i.e. how to go from the large model implicit in (1) to one that is easier to comprehend but which represents the data just as well. Nor- mally, in Hendry’s methodology this step involves the deletion of variables from (1), but it could also involve choosing to set combinations of parameters to par- ticular values. For convenience our discussion will centre upon model reduction via variable deletion. To simplify at all requires a criterion function and a deci- sion rule; how to use and report inferences from such information are the difficult issues in this third step.THREE ECONOMETRIC METHODOLOGIES 7 First, the decision stage. It is rare to find a criterion that is not based upon the log likelihood (or its alter ego, in regression models, the sum of squares). Fre- quently, it is something equivalent to the likelihood ratio test statistic, — 2 log (Ls/ Lg) where Ls and Lg are the likelihoods of simplified and general models respectively. For regression models this is approximately the product of the sam- ple size and the proportional change in the residual variance in moving from the general to simplified model. To know what is a ‘big’ change in the likelihood, it is common to select critical values from a table of the chi-square distribution by specifying a desired probability of Type I error. As is well known, one can think of this probability as indicating the fraction of times the simplified model would be rejected when it is true, given that the general model is re-estimated with data from many experiments differing solely by random shocks. Many see this myth as implausible in a non-experimental science such as economics, but myths such as this form the basis of many disciplines e.g. perfect competition in economics. What is important is that any framework within which analysis is conducted lead to useful results. If reliance upon the ‘story’ regularly causes error, it is then time to change it for something else. On the whole, I believe that these concepts have served us well, but there are some suggestions of alternative decision rules that may prove to be more useful. Thus Akaike (1973) and Mallows (1973) derive decision rules that opt for the deletion of a variable in a linear model if the change in the residual variance is less than (2 times the inverse of the sample size.’ Rissanen (1983), looking at the likelihood as an efficient way to summarize all the information in the data, for- mulates a decision rule that the change in residual variance must be less than a function of the sample size and difference in model dimensions. None of these is incompatible with the ‘Hendry’ methodology; to date they have not been used much, but that is a matter of choice rather than necessity. Having made a decision what should be reported? My own attitude, summariz- ed in McAleer ef al. (1985), is that an exact description of the decisions taken in moving from a general to simplified model is imperative in any application of the methodology. Rarely does this involve a single decision, although it would be possible to act as if it did by just comparing the finally chosen simplified model and the original one, thereby ignoring the path followed to the simplified version. This is what Hendry seems to do in various applied studies; he normally only pro- vides the value of a test statistic comparing the two models at each end of the path, with very little discussion (if any) of the route followed from one end to the other. There seem to me to be some arguments against this stance. First, it is hard to have much confidence in a model if little is explained about its origin. Hendry’s attitude seems to be that how a final model is derived is largely irrelevant; it is either useful or not useful, and that characteristic is independent of whether it comes purely from whimsy, some precise theory, or a very structured search (Hendry and Mizon, 1985). In a sense this is true, but it is cold comfort to those who are implementing the methodology or who are learning it for the first time. Reading Hendry’s applied papers frequently leaves only puzzlement about how8 PAGAN he actually did the simplification. In Hendry (1986) for example, the transition from a model with thirty-one parameters to one with only fourteen is explained in the following way (p. 29): ‘These equations.....were then transformed to a more interpretable parameterisation and redundant functions were deleted; the resulting parsimonious models were tested against the initial unrestricted forms by the overall F-test’. It is true that confidence in the simplified model is partly a function of the value of the F-test, but by its very nature this evidence can only mean that some of the deleted variables don’t matter. To see why, consider a general model with three regressors x1, X2 and xs, all of which are orthogonal. Suppose the F statistic for the deletion of x3 is 5 and that for x; is 0.5. Then the F statistic for the joint dele- tion of x2 and x3 is 2.75, and joint deletion is likely, even though it is dubious if x3 should be deleted at all. Thus an adequate documentation of the path followed in any simplification process is desirable, rather than just accompanying any simplification with a vague statement about it. More than that, I do believe in the possibility of situations in which simplication may be done in a systematic way e.g. in choosing dynamics via COMFAC (as in Hendry and Mizon, 1978 or McAleer et al., 1985), polynomial orders within Almon procedures and various types of demand and production restrictions that form a nested heirachy. As far as possible I am in favour of exploiting such well-developed strategies for simplification. Research should also be encouraged with the aim of developing new procedures or methods that require fewer assumptions. Knowledge of the path may be important for another reason. As discussed above the critical value used in the decision rule is taken from the tables of the x? or F distribution. But under the conditions of the story being used, this is only true if the simplification path consists of a single step. When there has been more than one step, the critical values cannot normally be taken from a x? distribution, and it may be misleading if one proceeds as if it can. Some, for example Hill (1986), see this as a major flaw in the methodology, and others feel that the deci- sion rule needs to be modified quite substantially in the presence of such ‘data mining’. When the move from a general to a simplified model can be formulated as a nested sequence, adjustments can be made to obtain the requisite critical value (Mizon (1977) gives an account of this), but in the more common case where this is not possible theoretical analysis has made little progress. Never- theless, numerical methods of Type I error evaluation, such as the bootstrap, do enable the tracing of Type I error for any sequence of tests and specified decision rules. Veall (1986) provides an application of this idea. Tam not certain that it is worthwhile computing exact Type I errors. Ignoring the sequence entirely produces a bias against the simplified model, but that does not seem such a bad thing. Moreover, the ultimate change in the criterion func- tion is independent of the path followed. It is frequently the change in the criterion itself which is of interest, in that it displays the sensitivity of (say) the log likelihood to variation in parameter value for the deleted variables as these range from zero to the point estimates of the general model.