You are on page 1of 23

COMPARISON OF BAYESIAN AND CLASSICAL META-ANALYSIS

Joe P King Educational Psychology Measurement, Statistics, and Research Design

Meta Analysis
• Overview of Meta-Analysis • Traditional Meta Analysis • Bayesian Meta Analysis • Why are they different? Why is it important? • Comparing methods using experimental data

• Implications
• Conclusions

• This satisfies one of the tenants of science that research must be reproducible and not one study can confirm or deny a theory . • The goal is to take the many studies which have looked at an outcome variable and try to inform on a theory.Introduction • Meta analysis is used in every field of scientific research • Meta analysis allows us to compile many studies to test a theory across many studies.

Introduction • Methods of Meta Analysis • Problem formulation • Data collection and selection of relevant studies • Evaluation of data collected • Interpretation and analysis • Presentation of results .

which should inform on how math instructors teach algebra. and Ronau (2010) analyzed different methods of algebra instruction they found 82 relevant studies. .Example • Study by Rakes. and a total sample size of 22424 students. Valentine. McGatha. 109 independent effect sizes. • Searched electronic journals for relevant articles. calculated effect sizes and in some cases weighted effect sizes due to different sampling techniques or small sample size • In 5 strategies of teaching algebra they found statistical significance for each in at least one model.

& Committee on Prevention of Mental Disorders. & West. 1994). Haggerty. Watt. . materials and technical assistance in the advancement of the program (Coie. 1993. Mrazek.Treatment • Communities that Care • Seeks to prevent anti-social behaviors among the youth in a community and strengthen pro-social behaviors. • Provides community leaders with training.

one city was randomly assigned to receive the treatment. Data was collected on specific adolescent measures. The treatment community in each pair was trained in and delivered the communities that care program for 5 years. . After pairing.Data • This replication summarizes 12 independent studies of • • • • the effects of communities that care 12 pairs of two communities were formed based on comparable demographic data. the other served as the control. here binge drinking and delinquent behavior.

delinquent behaviors were categorized as stealing. and being arrested. delinquent behaviors were carrying a gun to school. beating up someone. shoplifting.Outcomes • Two Measures were Used for this analysis • Delinquent behavior – • Fifth grade. stealing a vehicle. property damage. • Eighth grade. selling drugs. • Binge Drinking - 5 or more drinks at once occasion. and attacking someone with intention of hurting them. and the measurement was how many instances of binge drinking per month .

Traditional Meta Analysis • Effect size was collected for each pair of cities • Used multilevel modeling with two levels • Level 1 was the pairs of cities and accounting for the within experiment variance. . • Level 2 sought to account for the between experiment variance.

• Less Uncertainty in the experiment • Easier to implement .Why Bayes? • Specify Priors (this analysis uses non-informative priors) • Can see the effect of adding each experiment on effect size and precision associated with the estimate.

Bayesian Analysis • Replication of Howard. . & Fleming (2000) • They used Bayesian Calculations to calculate a posterior distribution of 3 studies. Maxwell. • Basics of Bayesian Approach • Prior Distribution – Initial estimate of effect size • Likelihood Principle – Effect size estimate of data • Posterior Distribution – Final Effect Size Combining Prior and Likelihood.

Calculations • What we know • Effect size for each Comparison • Uncertainty within Effect Size • What we will calculate • Precision • Posterior Effect Size for Each City then All Cities • Uncertainty Around the Effect Size .

Precision .

Effect Size .

17 SE 0.31 -0.1400 0.Results – Delinquent Behavior Effect Size Traditional Meta Analysis Bayesian Meta Analysis -0.0025 .

43 SE 0.Results – Binge Drinking Effect Size Bayesian Meta Analysis -0.0011 .

6 -0.2 Effect Size 0.Results for Each Comparison Posterior Distributions for Deliquent Behavior 0.8 -1.0 -0.4 0.0 1 2 3 4 5 6 7 8 9 10 11 12 Comparison Number .4 -0.8 0.6 0.2 -0.

2 -0.4 0.Cumulative Results After Each Iteration Posterior Distributions for Deliquent Behavior 0.3 Effect Size 0.6 0.1 -0.5 0.3 1 2 3 4 5 6 7 8 9 10 11 12 Posterior Iteration .0 -0.1 0.2 0.

0 Effect Size 0.5 -2.0 -0.5 2.5 -1.0 1.5 0.Results for Each Comparison Posterior Distributions for Binge Drinking 2.5 1.0 1 2 3 4 5 6 7 8 9 10 11 12 Comparison Number .0 -1.

Cumulative Results After Each Iteration Posterior Distributions for Binge Drinking 2.0 1.0 -0.5 1 2 3 4 5 6 7 8 9 10 11 12 Posterior Iteration .0 0.5 Effect Size 1.5 2.5 0.

.Summary • Meta Analysis is important to research • Many methods exist yet pose limitations • Bayesian approach is an additional method that shows promise.

Robert Abbott • Dr. Joe Lott • Kelly Jewell • Mom and Dad .Thank you • Dr. Charles Peck • Dr.

& Fleming. J. American Psychologist. The proof of the pudding: an illustration of the relative strengths of null hypothesis. 315-332. Washington. Journal of Laboratory and Clinical Medicine. J. N. V. meta-analysis. J. The science of prevention: A conceptual framework and some directions for a national research program. (2006). 147(1).. 5(3).. 48(10). P. S. (1994). D..References Coie. (2010). J. Meta-analysis: Methods. Watt. E.C. H. weaknesses. 29(1). & West. 1013-1022. R: A Language and Environment for Statistical Computing. & Committee on Prevention of Mental Disorders. Maxwell. R. Mrazek. and political uses.. S. 7-20. Hedges. Reducing risks for mental disorders: Frontiers for preventive intervention research. L. (1993). E.. Educational Evaluation and Policy Analysis. and Bayesian analysis. K. 60-87. G. I. Vienna Austria: R Foundation for Statistical Computing. strengths. Haggerty. Howard. C... . on M. Psychological Methods. S. R Development Core Team. (2000). Noble. & Hedberg. J. (2007). Intraclass Correlation Values for Planning Group-Randomized Trials in Education. National Academy Press.