Research Synthesis and MetaAnalysis 4A03 2011

'We need not confine ourselves to mere speculation about the socio-economic processes that comprise empirical economic research'
Stanley and Jarrell(1989):'Meta-Regression analysis: A Quantitative Method of Literature Surveys'. Journal of Economic Surveys

Intro ● How do/should we proceed when we have multiple studies on the same topic? Not choose arbritrarily! A number of approaches are possible. ● ● .

interventions)? Goals-1) Integration: Formulating general statements that characterize multiple specific instances related to the focal points Goals-2) Critical Evaluation: Does not necessarily compare studies to each other but to a gold standard.Literature Reviews 1 ● ● Characteristics: Focus(Findings. Evaluate methods in empirical context and evaluate interpretation of findings ● ● ● . Methods. Theories.

Be as exhaustive as possible and justify inclusion and exclusion decisions ● ● ● . Other characteristics: Perspective: You should aim at neutrality Coverage: May vary depending on the approach taken and the available literature.Literature Reviews 2 ● Your case: Focus on findings and methods with goals of integration and critical evaluation.

Implicit in this definition is the notion that seeking generalizations also involves seeking the limits of generalizations. try to resolve conflicts in the literature. and attempt to identify central issues for future research' Research Synthesis 1 ● . Also research syntheses almost always pay attention to relevant theories. critically analyze the research they cover. 'Research syntheses attempt to integrate empirical research for the purpose of creating generalizations.● A research synthesis is a particular kind of literature review: Handbook of Research Synthesis and Meta-Analysis p6.

where(e. when(1993-) etc Stage 3: Inclusion and exclusion criteria are made explicit ● ● ● ● .Research Synthesis 2 ● See Table 1.2 from Handbook of Research Synthesis and Meta-Analysis Stage 1: Question definition Stage 2:Based on systematic literature search Report search terms.g Econlit).

Glass: The statistical analysis of a large collection of results from individual studies for the purpose integrating the findings' Involves describing and explaining the central tendency and variability in the outcomes of similar studies using a statistical approach.Meta-Analysis 1 ● Meta-Analysis is a type of research synthesis dealing with stages 4-6. ● ● .

Combining results can furnish more insight and explanatory power than listing individual results or vote-counting E.g by taking weighted average of summary statistic(e.Meta-Analysis 2 ● Stanley.D(2001) :'Wheat from Chaff: MetaAnalysis as Quantitative Literature review' Journal of Economic Perspectives 15.g treatment effects) to construct mean effect will be more precise. 3. T. Meta-Regression goes further and uses summary statistic as dependent variable in ● ● ● . 131150.

Biggest strength-moves away from subjective choice of good and poor studies by analyst and provides replicable framework for objective analysis ● ● ● . design. design. data affect reported results Can help explain variability and offer reasons why evidence on a certain question is contradictory or very variable.Meta-Regression 1 ● Independent variables are characteristics of the method. data used etc Can identify extent to which particular choice of methods.

Meta-Regression 2 ● ● ● What are the steps in a Meta-Regression? Step 1) Identify relevant studies. data. Criteria for inclusion and exclusion standards applied consistently independent of findings of study. Best practice gives details of computer search to allow replication. or methods do not provide justification for exclusion. ● ● . Computer search of standard databases. Difference in quality.

May need to convert to common scale Usual first overall estimate-combining these to estimate average effect(Stanley p135-136) Technical issues relate to heteroskedasticity due to different degrees of freedom and relation of summary statistic to degrees of freedom. elasticities.g co-efficient estimates. t-stats) Be careful with coefficients if outcomes or regressors are on different scales. ● ● ● .Meta-Regression 3 ● Step 2) Choose summary statistic and reduce evidence to a common metric(e.

Meta-Regression 4 ● Step 3) Choose explanatory variables (Stanley p137-138) Usually there will be much more variability in the summary statistics of interest than would be produced by chance for the same underlying parameter Want to explain this variability using study characteristics # of variables chosen depends on number of studies and variability in study characteristicsWhy? ● ● ● .

Meta-Regression 5 ● ● Suggested regressors: Inclusion/Omission of particular regressors from original model Specification variables Data variables Sample size Author characteristics(particular problem comes about with multiple studies by same author or estimates from same study) Quality Measures(Method variables) ● ● ● ● ● .

Stanley p141-143 Use standard tests for specification testing and out of sample forecasting Example of normalized test-statistics(test-statistics that have p-values matched to standard normal): Purely random variation should have a variance equal to one.Meta-Regression 6 ● Step 4) Conduct the Meta-Regression(standard software for regression analysis) Stanley p139-141 Step 5) Subject Meta-Regression to specification testing and Sensitivity analysis. Example of t-statistics: Including degrees of freedom(or sample size) there should be a positive relationship if the null of zero effect is false and zero if true(in absence of publication bias) Note that discussion in Stanley p142-3 should be discussing absolute ● ● ● ● ● . Can test this and if not rejected provides general specification test that additional regressors will not add explanatory power.

4) Publication Bias-test exist(see Stanley 2005. 3. 309-345) Similar issues also exist with narrative reviews with less explicit criteria and sensitivity analysis ● ● ● ● .Meta-Regression Problems ● 1) Disagreement over regressors(can be dealt with-sensitivity analysis. testing) 2) Weighting of each study equally(overweights those who publish lots-can be dealt with by sensitivity analysis and author effects and explicit weights) 3)Invariant errors-Cannot deal with this! Critical evaluation required. Journal of Economic Surveys 19.

F452477: Active Labour Market Policies ● ● ● . 7. 1-4:Price and Income elasticities for Higher education Card et al(2010) Economic Journal 120. 821-835 :Price elasticities for Tobacco Gemmill et al(2007) Health Economics 16: 27643: Prescription Drug elasticities Gallet (2007) Economics Bulletin 9.Recent Examples ● Gallet and List(2003) Health Economics 12.

Sign up to vote on this title
UsefulNot useful