Evaluating Socio Economic Development, SOURCEBOOK 2: Methods & Techniques

Econometric models

[PLEASE NOTE: The following SOURCEBOOK text is designed for presentation as part of
the Internet-based collection of materials for Evaluating Socio Economic Development,
and should be viewed in this context. Introductory remarks are provided on the site
www.evalsed.info]

ECONOMETRIC MODELS

Introduction

Description of the technique

An econometric model is one of a range of tools used to replicate and simulate the main
mechanisms of a regional, national or international economic system. Econometric models are
generally defined by the use that data play in informing the model structure, namely to calculate
the model’s coefficients through a variety of possible estimation methods. In most models that
use the label “econometric”, there is usually a mixture of those coefficients estimated freely by
the data, and those which are fixed, assumed or restricted, due to some limitations on data
quantity or quality. These restrictions or assumptions can often be made according to economic
theory, or sometimes use results from other datasets where the economic mechanisms are
expected to perform in similar ways.

A large number of such models exist, largely because very few are actually built specifically with
the aim of evaluation in mind. More often an existing model, previously built for some other
purpose, is adapted as an evaluation tool as a least-cost option because it would take too many
resources to construct a model from scratch. The models are adapted in order to simulate a
counterfactual situation, and thus to quantitatively evaluate net effects on most of the macro-
economic variables influenced by public actions, eg growth, employment, investment, trade, etc.

Macroeconomic theory is not a particularly stable field, with many competing and diverse
theories, so the different models will not only reflect different uses but also the
viewpoint/ideology of the modeller who constructed them. Generally speaking, the demand-
sides of such models are usually similar, based upon a disaggregation of the expenditure side
of GDP (ie household spending, government spending, investment spending, exports, imports,
and stock-building) with most divergence occurring in the treatment of supply-side effects which
act on the potential output of the economic system.

Purposes of the technique

There are three main purposes for constructing an econometric model, each of which has an
application within the context of evaluation.

1 Understanding
A model can help to shed light on relationships between variables, because in economics, or
more generally the social sciences, the facts don’t speak for themselves. More formally, a
model is a way of testing whether there is evidence for a specific hypothesis, eg variable Y has
a significant influence on variable X.

Within the context of an evaluation, the model will aid an understanding of how the mechanisms
involved in transmitting the effects of a policy (eg CSF) fit together. The model provides a
structure around which the policies effects can be assessed and integrated, and in this way
should make the quantification of these effects more transparent. An econometric model can

Final Materials December 2003 1

 Evaluating Socio Economic Development, SOURCEBOOK 2: Methods & Techniques
Econometric models

make use of the data to provide measures of how well the past performance of the economic
system is explained, eg by use of diagnostic tests and measures of goodness of fit. While such
statistics are not a complete guide to how well a model will perform in an evaluation exercise,
they do provide a method of assessment.

2 Forecasting
Forecasting can be quite distinct from understanding. It is possible to forecast without actually
understanding the processes involved, ie forecasting by association, whereby patterns of data
are associated without having to understand the causal relationships involved.

Most macoeconometric models are based around sets of causal relationships which seek to
both understand/explain and provide some forecast capability. With many evaluations the
effects are forward-looking, ie ex ante. This means that some kind of baseline forecast is
required, against which to assess the effects of the particular policy being evaluated. Achieving
a credible baseline forecast is an important element in the use of econometric models for
evaluation, as an unstable forecast can be an indication of an unstable model structure.

The terms forecast, solution, and simulation are often used as interchangeable, but there are
some subtle differences. The model solution or simulation is generally regarded as raw model
output when the estimated coefficients from the model are combined with future assumptions for
exogenous variables, and in conjunction with the historical data as a starting point, the model
results are pushed forward beyond the sample size. A forecast can often involve some ex-post
adjustment, ending up as a mixture of model results and economist viewpoint. A common
technique used is known as “residual adjustment”, whereby the model residuals (normally zero
in the forecast period) are altered to push the future values of the dependent variable up or
down depending on the desired outcome.

3 Scenarios
Scenarios allow you to ask ‘what if?’ type questions. The process involves constructing an
alternative reality, usually through a different set of assumptions, and then comparing the results
against a baseline (often called a ‘business as usual’) run of the model. This is a different
process to sensitivity analysis, whereby a single variable/assumption is altered and the reaction
of the main model outputs is investigated for signs of instability or anomalous results.

Using econometric models for policy evaluation always involves the construction of a scenario,
ie with and without the policy, to quantify the overall effect in terms of key model outputs. The
complexity with which this scenario is constructed depends on how the policy effects feed into
the model, ie whether they only affect a small set of exogenous variables or whether they might
have an effect on some behavioural relations.

HERMIN, QUEST and E3ME are examples of currently supported econometric models of
different types, that are notable for their wide range of uses in simulating monetary,
convergence and Structural Funds policy impacts in the European Union. The REMI Policy
Insight model has been extensively applied in the US but has only recently been modified for
Europe; a brief mention is nonetheless made for completeness. Box 1 provides further details of
models currently in use.

Final Materials December 2003 2

 Evaluating Socio Economic Development, SOURCEBOOK 2: Methods & Techniques
Econometric models

Box 1: Examples of models in use

1. HERMIN
Each HERMIN model has three broad sub-components (a supply side, an absorption side and an income distribution
side) which function as an integrated system of equations. A conventional Keynesian aggregate demand mechanism
underpins the absorption side of the model. There is some degree of sectoral disaggregation with a supply-side sub-
component helping to determine traded (manufacturing) output as a consequence of national price and cost
competitiveness. Interest and exchange rates are exogenous to the HERMIN model, in line with the general
assumption that the cohesion economies are ‘small’ and ‘open’.
(see Bradley, J. (1997), Aggregate and Regional Impact: The Cases of Greece, Spain, Ireland and Portugal,
published by the Office of the Official Publications of the European Communities for a recent review of model
content)

2 QUEST
QUEST is developed by European Commission Service Directorate and is a multi-country model designed to analyse
the business cycle, the long-term growth of the Member States of the European Union and the interactions of these
states with the rest of the world, especially with the United States and Japan.. The QUEST II version of the model
identifies stock and flow equilibrium variables at macroeconomic level including physical capital, net foreign assets,
money and government debt which are endogenously determined with wealth effects allowed to influence flows of
savings, and production and investment decisions of private households, firms and the government. The supply-side
of the economy in QUEST II is modelled explicitly to conform to a neo-classical aggregate production function setting
potential capacity, with long-run growth rates of this potential determined by the rate of (exogenous) technical
progress and the growth rate of the population. Results of simulations may be presented as deviations from a
baseline scenario.
The model has real interest rates and exchange rates determined endogenously, and this does allow for the possible
‘crowding-out’ effects of Structural Funds on the private sector to be taken into account.
For more details see European Commission (1997), QUEST II. A Multi-Country Business Cycle and Growth Model.
Economic Papers. No. 123. October 1997, also available from:
http://europa.eu.int/comm/economy_finance/publications/economic_papers/economicpapers123_en.htm.

3 E3ME
E3ME, an energy-environment-economy model for Europe, is a multi-sectoral, regionalised, dynamic econometric
model of the EU. It is not a Computable General Equilibrium (CGE) model, but a disaggregated time-series, cross-
section econometric model, that has benefited from some of the techniques used in CGEs relating to calibration on
recent data. The model has been developed for the European Commission under the EU JOULE/THERMIE
programme by a team of partner institutes across Europe led by Cambridge Econometrics. It is designed as a
specifically forward-looking model for assessing energy-environment-economy issues and policies. The model
therefore combines economic, energy and environment components.
(See Commission of the European Communities (1995) E3ME: An Energy-Environment-Economy Model for Europe,
EUR 16715 EN, Official Publications, Luxembourg)

4 REMI Policy Insight Model

The REMI model has until recently been only applied in North America, but within the past year or so some
applications have been carried out on structural funds impacts for the European Commission.

The model is econometric in origin, but the structure is the same for all market-based economies except for
differences in a few key parameters such as the speed of migration response to changes in economic conditions and
the response of wage rates to labour market conditions. The model parameters are estimated over a large sample of
regions and are used for all implementations of the model. By imposing a structure with pre-estimated coefficients,

Final Materials December 2003 3

 Evaluating Socio Economic Development, SOURCEBOOK 2: Methods & Techniques
Econometric models

the REMI model is capable of a much richer representation than would otherwise be available were it to rely purely
on indigenous data sources – this is particularly the case for regional applications.
(see REMI Policy Insight, Model Documentation European Version 5.3), Regional Economic Models Inc., Amherst
MA.)
Frederick Treyz, George Treyz, (2003), Evaluating the Regional Economic Effects of Structural Funds Programs
Using the REMI Policy insight
http://europa.eu.int/comm/regional_policy/sources/docconf/budapeval/index_en.htm

Circumstances in which it is applied

EC (2000) provides an example where HERMIN and QUEST II were used to assess the
contribution of the Community Support Frameworks to economic convergence. The HERMIN
model can be used directly to assess CSF policies. The work was based on simulations using
the HERMIN national models for Greece, Spain, Ireland, Portugal and east Germany. In the
HERMIN models the Structural Funds work through boosting the economy’s long-run supply
potential by:

- improving the physical infrastructure;

- supporting knowledge-based growth (human capital);
- directly supporting the private productive sector.

In EC (2000) the macroeconomic impact of the Structural Funds programme is modelled less
directly by QUEST II as an increase in public capital stock, whose marginal product is assumed
to be 50 per cent higher than that of private capital and which is also assumed to have positive
externalities.

While the QUEST II model has been primarily developed to simulate and analyse changing
macroeconomic financial policies associated with deepening and widening of the EU, it has
been used quite flexibly for policy analysis. It has been used for example to assess the impact
of the Maastricht criteria on growth and employment, the long run effects of fiscal consolidation
and structural reforms in Europe, the impact of monetary policy on the success of government
expenditure cuts, the macroeconomic effects of various tax reforms and VAT harmonisation.
The model has also been used to assess the employment and growth effects of the Trans-
European Transport Networks while (as a described above) models for Greece, Ireland,
Portugal and Spain have been used to look at the macroeconomic effects of the Structural.

E3ME has been designed to provide a forward-looking framework for energy and environment
policy, taking account of economic impacts. It has been flexibly used to assess a range of
structural and financial policy innovations. A recent example of its application (GHK et al. 2003)
was in assessing the contribution of the Structural Funds to sustainable development (SD) in
the three programme periods 1986-1993, 1994-1999 and 2000-2006. As issues of sustainable
development have come to the fore of the policy agenda, DG Regio has focused attention on
sustainability issues within its Structural Fund programmes. This study undertook analysis of the
impact that structural funds spending has on sustainable development. The E3ME modelled
demand-side and supply-side effects by which investment and expenditure in education drive
economic activity and environmental change. The demand-side modelling was used to assess
the effect that SFs have on the levels of taxes, current government expenditure, investment and
the economy. The supply-side modelling assessed the longer-term effects of the SF on
changes in productivity and changes in the accumulation of capital and technology. These

Final Materials December 2003 4

 Evaluating Socio Economic Development, SOURCEBOOK 2: Methods & Techniques
Econometric models

longer-term effects are incorporated in the model through changes in productivity induced by
these expenditures. The analysis compared a baseline (policy on) scenario based on historical
data, against a counterfactual scenario (policy off) where structural funds expenditures have
been eliminated

The REMI model has been used to assess the regional economic effects of structural funds
investments in the Objective 1 portion of Southern Italy, in a joint project by REMI and Irpet for
the European Commission (DG Regio). The outcome of the study included returns (in the form
of GDP and employment multipliers) to various types of investment (equipment, training,
infrastructure) and subsidies are reported for a time horizon over 2000-24. To determine these
kinds of effect, three steps are involved:

1. Examine each of the operational plans in detail. In particular, examine the direct effects on
the economy in the short run (ie mostly demand effects) and long run (ie mostly supply effects).
2. Input these direct effects into the REMI model so that all of the key chains of causality
through which public fund investments can influence the economy.
3. Run the model and calculate the relative effectiveness of the euro expenditure toward
accomplishing the major objectives of the public investments.

The HERMIN model has been developed through collaboration between countries to provide a
tailored approach to facilitate national and cross national comparisons (see Box 2).

Box 2: HERMIN modelling developments

Four cohesions countries – Greece, Ireland, Portugal and Spain – have been involved in a series of collaborations.
CSF impact analysis in the cohesion countries was combined with analysis of the impact of the Single Market. This
was designed to move away from restricted ‘theory of action’ approaches, towards more holistic ‘explanatory’ and
‘global’ understandings of cohesion.

John Bradley, János Gács, Alvar Kangur and Natalie Lubenets (2003), Macro impact evaluation of National
Development Plans: A tale of Irish, Estonian and Hungarian collaborations
http://www.europa.eu.int/comm/regional_policy/sources/docconf/budapeval/work/kangur.doc

The main steps involved

The implementation of an evaluation using an econometric macroeconomic model must be

considered only if the evaluators have adequate knowledge, skills and time. This is because
numerous highly technical steps are involved.

Step 1: Locating a suitable model

It is generally rare for a model to be built from scratch for an evaluation exercise, because such
models are quite resource-intensive. This means that an existing model with broadly
acceptable features must be chosen, and adapted as necessary for the particular policy under
analysis. Depending on the scope of the analysis, eg a single region, a single country, or pan-
European, there will be different numbers of competing models available which will generally
require input from the model proprietor.

Step 2: Adapting the model for evaluation

Final Materials December 2003 5

 Evaluating Socio Economic Development, SOURCEBOOK 2: Methods & Techniques
Econometric models

Once a suitable model is located, its structure must be analysed to see how it would cope with
the transmission mechanisms used by the policy to be evaluated. A schematic diagram (Box 3)
can provide such an overview and help point to areas which may need further elaboration. For
example, it could be that the part of the model dealing with public investment is quite basic,
whereas the policy involves more detailed identification of expenditure flows.

There may also be scope for altering some of the behavioural relationships in the model if it is
believed that a policy would change a fundamental property of the economic system. Such
changes are quite rare, however. More frequent is the need to adjust the model to the most
recent developments observed and to the most recent statistics. In practice, econometric
models must be regularly re-estimated to take into account revisions in basic information and
thus to allow for them to be "adjusted" to the most recent data.

Step 3: Establish a baseline projection

With the model re-estimated and exogenous assumptions reviewed, a baseline projection can
be produced with which to assess the policy impact. The baseline could include the policy to be
evaluated as part of the ‘business as usual’ scenario. This is most common when the policy has
already been in operation for some time and assumptions for future behaviour already taken
into account some of their effects.

Box 3: Flow diagram for typical macroeconomic model mechanisms

Rest of World Activity, Prices

EU Outside Country
EU Transportation Other Countries'
EC Policy, eg CSF an & Distribution Output

Costs and Exports

Productivity Imports

Public Spending Total Demand

Public
Debt Investment and
Household Inputs to Production
Spending (input-output) Output
Taxes

Incomes Employment

Step 4: Estimating primary (exogenous) impacts

To evaluate the effect of a public intervention, ie begin to construct the alternative scenario in
which the policy did not exist, the modeller must consider how to introduce changes into the

Final Materials December 2003 6

 Evaluating Socio Economic Development, SOURCEBOOK 2: Methods & Techniques
Econometric models

model. The evaluation team must then select a few variables or coefficients that are part of the
model and which will be directly affected by the public intervention. These values constitute
"bridges" between the evaluated intervention and the model. The primary impacts of the
Structural Fund interventions may also take the form of investments in productive structures or
in human capital.

Step 5: Running the policy scenario

Having estimated the primary (micro-economic) impacts of the intervention and introduced them
into the model to see how they feed through, the model can be run to calculate the final effect
on model output (ie endogenous) variables. The impact of the intervention can then be
estimated by establishing the difference between the two simulations, ie baseline versus policy-
off model run.

Step 6: Assessing the results

The last phase is that of the formulation and presentation of results. Given the complexity of the
tool and the numerous variables that can be quantified, it is necessary to take specific steps to
present results in a simple and easily accessible way, particularly as the readership may not be
experts in the field of macroeconometric modelling. This phase must therefore be considered
as one of the most important in the implementation of the tool.

An important part of the results assessment is to clearly state the assumptions and choices
made in the evaluation exercise. For the more technically inclined reader it would also be
desirable to include reference to the model structure/ideology, as this is often taken for granted
but can have a big influence, eg whether the model assumes market clearing or allows
disequilibrium to persist in the medium to long term.

A final test of the evaluation results is to evaluate the importance of the alterations made to the
model (Step 4) to assess how sensitive the findings are to small changes in these effects.

Strengths and limitations of the approach

Strengths

The goals of the Structural Funds are defined at the macroeconomic level. The outputs from a
macroeconometric model therefore are generally consistent with requirements, meaning that
this is practically the only tool that can be used to formally ascertain whether European policy
has achieved its aim.

A key strength of an econometric model is its ability to use the data to inform the structure,
which means that robust statistical methods can be used to assess the credibility, eg
diagnostics measuring serial correlation (patterns in model residuals).

The models are usually dynamic, which means they can track annual changes in policy effects
rather than a notional difference between two equilibrium points.

Weaknesses

There is a heavy resource requirement when dealing with econometric models. Good quality
data sets are required to ensure sensible coefficient values, a factor that often limits the scope
of such models. The work involved in constructing such models from scratch usually means

Final Materials December 2003 7

 Evaluating Socio Economic Development, SOURCEBOOK 2: Methods & Techniques
Econometric models

that existing models, which may not be ideally suited to the purpose of evaluation, are adapted
instead.

Most macro-econometric models share a broadly similar demand-side structure based around
the expenditure components of GDP, with further modules for the traded sectors and possibly
input-output tables to cope with inter-industry demand and specific sectoral effects. More
divergence occurs with the treatment of supply-side factors, which measure the potential output
of an economic system, often through the use of a production function. Typically technology is
assumed to be exogenous, although much of the more recent analysis of convergence and the
‘new growth’ literature emphasises the important role of endogenous effects linked to human
capital growth. Because potential output can never be observed there is more debate over how
it should be measured, leading to different structures, different model properties, and ultimately
different evaluation results.

A common criticism of macroeconomic models where aggregate production functions are used
is that they are not consistent with microeconomic theoretical foundations of profit maximisation
and/or cost minimisation by producers. The aggregate relationships are not built up from
consistent demand and cost functions which retain the properties that are desirable in economic
theory, unlike CGE models.

Annotated references

Theoretical foundations

Layard R., S. Nickell and R. Jackman (1991) ,"Unemployment: Macroeconomic Performance

and the Labour Market ", Oxford University Press, Oxford.

Guidance on the application of the method

European Commission (Joint Research Centre) (2002), RTD- Evaluation Toolbox, Assessing
the Socio-Economic Impact of RTD-Policies, August 2002. Particularly Section 3.3:
Econometric Models: Macroeconomic Modelling and Simulation. See
http://epub.jrc.es/docs/EUR-20382-EN.pdf.

Roeger, W. and J. in t’Veld (1997) QUEST II – A multi-country business cycle and growth
model, Economic Papers No 123, European Commission.

Examples of evaluation or studies applying the method

Bradley, J. (1997) Aggregate and Regional Impact: The Cases of Greece, Spain, Ireland and
Portugal (1997) (ISBN No. 92- 827- 8807-5) , published by the Office of the Official Publications
of the European Communities

European Commission (2000), The EU Economy: 2000 Review, Chapter 5 ‘Regional

convergence and catching-up in the EU’, European Economy, No.71.

GHK, PSI, IEEP, CE (2003) The Contribution of the Structural Funds to Sustainable
Development – A Synthesis Report (Volume 1) to DG Regio, EC, chapter 4.

Final Materials December 2003 8

 Evaluating Socio Economic Development, SOURCEBOOK 2: Methods & Techniques
Econometric models

IRPET and REMI, (2003), Assessing the Regional Economic Effects of Structural Funds
Investments, a draft final report for The European Commission DG Regio), Contract No.
2002.CE.16.0.AT.139.

Articles in the evaluation literature that comment on the application of the method.

Hallet, M. and G. Untiedt (2001), The potential and limitations of macroeconomic modelling for
the evaluation of EU Structural Funds illustrated by the HERMIN model for East Germany;
Informationen zur Raumentwicklung No. 4/2001.

Frederick Treyz, George Treyz, (2003), Evaluating the Regional Economic Effects of Structural
Funds Programs Using the REMI Policy insight
http://europa.eu.int/comm/regional_policy/sources/docconf/budapeval/index_en.htm

John Bradley, János Gács, Alvar Kangur and Natalie Lubenets (2003), Macro impact evaluation
of National Development Plans: A tale of Irish, Estonian and Hungarian collaborations
http://www.europa.eu.int/comm/regional_policy/sources/docconf/budapeval/work/kangur.doc

Key terms

CGE - Computable General Equilibrium Models

General equilibrium analysis models an economy to present an integrated picture of the labour
market and goods market relationships Computable General Equilibrium (CGE) models look at
goods and factor markets simultaneously with wages, prices and hence incomes determined
endogenously.

Demand-side effects
Demand-side effects measure the impact of economic change in terms of expenditure and
income effects.

Supply-side effects
Supply-side effects measure the impact of economic change on productivity, wages, and profits.

Exogenous:
Generated by external factors, as opposed to internal factors (endogenous).

Final Materials December 2003 9