A Workshop on the Basics of

Systematic Review &

Meta-Analysis
Philip C. Abrami, Robert M. Bernard
C. Anne Wade, Evgueni Borokhovski, Rana Tamim, Gretchen
Lowerison & Mike Surkes
Centre for the Study of Learning and Performance
and CanKnow

Concordia University
What is a Systematic Review?
• A review of a clearly formulated question that
uses systematic and explicit methods to
identify, select and critically appraise relevant
research, and to collect and analyze data from
the studies that are included in the review.
• Statistical methods (meta-analysis) may or may
not be used to analyze and summarize the
results of the included studies.
• Other examples: Narrative review, qualitative
review, vote count, meta-synthesis.

02/25/11 2
 What is Meta-Analysis?
• Meta-Analysis is a set of quantitative
research synthesis techniques and
procedures
• Meta-Analysis uses effect size as a
metric for judging the magnitude of
standardized difference between a
treatment and control condition

02/25/11 3
02/25/11
 Purpose: Explaining
Variability in Effect Size

Study Features Effect Sizes

Shared Variability

Unique Variability Unique Variability

Prediction
02/25/11 5
 10 Steps in Planning and
Conducting a Systematic
Review/Meta-Analysis
2. Determine the research question
3. Develop terms and definitions related to the question
4. Develop a search strategy for identification of relevant
studies
5. Establish criteria for inclusion and exclusion of studies
6. Select studies based on abstract review (agreement)
7. Select studies based on full-text review (agreement)
8. Extract effect sizes (agreement)
9. Develop codebook of study features
10. Code studies (agreement)
11. Conduct statistical analysis and interpretation

02/25/11 6
02/25/11
 10 Steps in a Meta-Analysis

1. Determine the research question

The “big question” that guides the research. It

usually involves asking about the difference between
two conditions (i.e., usually treatment and control) or
the relationship between two measures.

02/25/11 8
 Questions the Researcher
Should Ask
• Does the question have theoretical or practical
relevance (i.e., aids in practice and/or policy
making decisions)?
• Is the literature of a type that can answer the
question?
• Is there a sufficient quantitative research
literature?
• Do the studies lend themselves to meta-analysis?
• Is the literature too large given the resources
available?

02/25/11 9
 Example:
Critical Thinking

Research Question: What instructional

interventions, to what extent, and under what
particular circumstances, impact on the
development and effective use of learner’s
critical thinking skills and dispositions?

02/25/11 10
 10 Steps in a Meta-Analysis

2. Develop terms and definitions related to

the question

This helps refine the research question and inform

the search strategies.

02/25/11 11
 10 Steps in a Meta-Analysis

3. Develop a search strategy for the

identification of relevant studies

This involves the planning/implementation of search

and retrieval for primary studies
(e.g., electronic databases, branching).

02/25/11 12
 Information Retrieval:
A Continuous Process
Preliminary Searches
 Supports beginning steps: Definition of key concepts &
research question
 Use of standard reference tools and broad searches for
review articles and key primary studies
Main Searches
 Identification of primary studies through searches of
online databases, printed indices, Internet, branching, hand-
searches
 Most difficult given a number of challenges
Final Searches
 Occurs towards the end of the Review Process
 Refine search terms and update original searches

02/25/11 13
 Preliminary Searches
Reference Sources:
Purpose: To obtain definitions for the terms; creativity, critical thinking,
decision making, divergent thinking, intelligence; problem solving, reasoning,
thinking.
Sources:
Bailin, S. (1998). Critical Thinking: Philosophical Issues. [CD-ROM] Education:
The Complete Encyclopedia. Elsevier Science, Ltd.

Barrow, R., & Milburn, G. (1990). A critical dictionary of educational concepts:

An appraisal of selected ideas and issues in educational theory and practice
(2nd ed.). Hertfordshire, UK: Harvester Wheatsheaf

Colman (2001). Dictionary of Psychology (complete reference to be obtained)

Corsini, R. J. (1999). The dictionary of psychology. Philadelphia, PA:

Brunner/Mazel

Dejnoka, E. L., & Kapel, D. E. (1991). American educator’s encyclopedia.

Westport, CT: Greenwood Press.

02/25/11 14 (see
……
 Main Searches: Decisions
Selection of Primary Information Retrieval Tools
 Scope of search: Which fields should be searched
(including all related fields)?
 Availability of indexing tools: Which tools do we have
access to at our institution? Are there others who can
perform searches for us?
 Format of indexing tools: What format are they in (e.g.
online, print, web-based)?
 Date: How far back does the indexing go for each tool?
 Language: What is the language of the material that is
indexed? How can we locate non-English material?
 Unpublished work: How can we access dissertations,
reports, & other grey literature?

02/25/11 15
 Examples of Databases
 Education: ERIC, British Education Index, Australian
Education Index, Chinese ERIC, CBCA Education,
Education index, Education: A SAGE Full-text Collection
 Psychology: PsycINFO, PubMed (Medline), Psychology: A
SAGE Full-Text Collection
 Sociology: Sociological Abstracts, Contemporary
Women’s Issues. Sociology: A SAGE Full-text Collection
 Multidisciplinary: EBSCO Academic Search Premier,
ProQuest Dissertations and Theses Fulltext, FRANCIS,
Social Sciences Index, SCOPUS, Web of Science

02/25/11 16
 Example: Critical Thinking
To date, the following databases have been searched:
• AACE Digital Library (now known as EdITLib)
• ABI/Inform Business
• EBSCO Academic Search Premier
• ERIC
• EconLit
• PAIS International
• ProQuest Dissertations and Theses Fulltext
• PsycINFO
• Social Science Index
• Sociological Abstracts

02/25/11 17
 Main Searches: More Decisions

Preparation of Search Strategies

 What are the key concepts to be searched?
 How are these represented in each discipline?
 What are their related terms?
 How are these key concepts represented in the
controlled vocabulary within each database to be
searched? (See handout)

 Note: these decisions need to be made for each indexing tool used.

02/25/11 18
 Main Searches:
Yet More Decisions
Construction of the Search Statements
 What terms should be searched as descriptors or as
“free text”?
 What Boolean operators should be used?
 Where should truncation characters be used? (e.g.
parent* will retrieve parent, parents, parental)
 What limiting features are available to narrow
results? (e.g. use of Publication Type codes)?
 What time period should be searched?

02/25/11 19
 Example: ERIC
Combining Keywords/Descriptors using Boolean
operators:
Searches and records below from: The ERIC Database
#5 #3 and #4 (1520 records)
#4 DTC = 142 or DTC = 143 or control group* (322893 records)
#3 #1 or #2 (7718 records)
#2 critical thinking in DE,ID (7562 records)
#1 thinking skills in DE and critical thinking (1269 records)

02/25/11 20
 Documenting Your Searches
Example from our Codebook:
ERIC (Date: September 21, 2003; AW)
Purpose: To retrieve the first set of abstracts to be reviewed by
team according to the current inclusive/exclusion criteria.
Result: Hit rate of 514/1520
Source code: ERIC1

Searches and records below from: The ERIC Database

(1966-2003, June)
#5 #3 and #4 (1520 records)
#4 DTC = 142 or DTC = 143 or control group* (322893
records)
#3 #1 or #2 (7718 records)
#2 critical thinking in DE,ID (7562 records)
#1 thinking skills in DE and critical thinking (1269 records)

02/25/11 21
 Next Steps

Repeat these steps for each

database to be searched.
(see handout)

02/25/11 22
 Main Searches:
Yet Still More Decisions
Secondary Retrieval Strategies
 Locating the grey (unpublished) literature:
- Using the web, & Dissertations Abstracts
 Branching:
- Scanning the reference section of review articles
 Hand searches:
- Scanning the Table of Contents of key journals and
conference proceedings
 Personal contacts:
- Contacting key researchers in the field

02/25/11 23
 Information Retrieval:
Wrap Up
“Shoestring-budget information retrieval is likely to introduce
bias, and should be avoided.” (IR Policy Brief, 2004)

Importance of information retrieval process

 Not a “one-shot”deal
 Requires expertise in the planning and implementation of
searches
 Library personnel are important members of the team
Use of bibliographic management software
 Reference Manager, EndNotes, RefWorks
Ability to replicate review
 Documentation of entire process, including search strategies
used for each database, decisions taken, etc.

02/25/11 24
 10 Steps in a Meta-Analysis

4. Establish criteria for inclusion

and exclusion of studies

These are the criteria that guide the search for

literature and ultimately determine what studies
are in and out of the review.

02/25/11 25
 Inclusion/Exclusion: Questions

• What characteristics of studies will be used to

determine whether a particular effort was
relevant to the research question?
• What characteristics of studies will lead to
inclusion? exclusion?
• Will relevance decisions be based on a reading of
report titles? abstracts? full reports?
• Who will make the relevance decisions?
• How will the reliability of relevance decisions be
assessed?

02/25/11 26
 10 Steps in a Meta-Analysis

5. Select studies based on

abstract review

This is the initial decision as to what

studies will be retrieved as full-text documents.

02/25/11 27
 10 Steps in a Meta-Analysis

6. Select studies based on

full-text review

This is the second decision as to what studies

will be included in the review.

02/25/11 28
 10 Steps in a Meta-Analysis

7. Extract effect sizes

Effect sizes extraction involves converting

descriptive or other statistical information contained
in studies into a standard metric by which studies
can be compared.

02/25/11 29
 What is an Effect size?

• A descriptive metric that characterizes

the standardized difference (in SD units)
between the mean of a control group and
the mean of a treatment group (educational
intervention)
• Can also be calculated from correlational
data derived from pre-experimental
designs or from repeated measures designs

02/25/11 30
 Characteristics of
Effect Sizes
• Can be positive or negative
• Interpreted as a z-score, in SD units,
units although
individual effect sizes are not part of a z-score
distribution
• Can be aggregated with other effect sizes and
subjected to other statistical procedures such as
ANOVA and multiple regression
• Magnitude interpretation: ≤ 0.20 is a small effect
size, 0.50 is a moderate effect size and ≥ 0.80 is a
large effect size (Cohen, 1992)

02/25/11 31
 Effect Size Extraction

• Effect size extraction is the process of

identifying relevant statistical data in a study and
calculating an effect size based on those data
• All effect sizes should be extracted by two
coders, working independently
• Coders’ results should be compared and a measure
of inter-coder agreement calculated and recorded
• In cases of disagreement, coders should resolve
the discrepancy in collaboration

02/25/11 32
02/25/11
02/25/11
 Example of ES Extraction with
Descriptive Statistics
Study reports: Treatment mean = 42.8 Control Mean = 32.5
Treatment SD = 8.6 Control SD = 7.4
n = 26 n = 31

Procedure: Calculate SDpooled Calculate d and g

SD pooled = ((26 − 1)8.62 ) + (31 − 1)7.4 2 )) / (57 − 2)

Σ∆ ποολεδ = (1849 + 1642.8) / 55 = 3491.8 / 55 = 63.49 = 7.97
42.8 − 32.5 10.3
δ= = = 1.29
7.97 7.97
￦ 3 ￨ ￦ 3 ￨ ￦ 3 ￨
γ = δ ￨1 − = 1.29 1 − = 1.29 ￨1− = 1.27
￨ (4( Ν Ε + Ν Χ )) − 9 ￨
￨ ￨
￨ 4(26 + 31) − 9 ￨
￨ ￨ 219 ￨
￨

02/25/11 35
 Extracting Effect Sizes in the
Absence of Descriptive Statistics

• Inferential Statistics (t-test, ANOVA, ANCOVA,

etc.) when the exact statistics are provided
• Levels of significance, such as p < .05, when the
exact statistics are not given (t can be set at the
conservative t = 1.96) (Glass, McGaw & Smith, 1981; Hedges,
Shymansky & Woodworth, 1989)
• Studies not reporting sample sizes for control and
experimental groups should be considered for
exclusion

02/25/11 36
 Examples of Alternative
Methods of ES Extraction
• Study Reports: t (63) = 2.56, p < .05
2τ 2(2.56) 5.12
d= = = = .6448
δφ 63 7.94
• Study Reports: F (1, 63) = 2.56, p < .05
Convert F to t and apply the above equation:

t= Φ = 1.6; δφ = 63
2τ 2(1.6) 3.2
δ= = = = .4030
δφ 7.94 7.94
02/25/11 37
 Zero Effect Size

ES = 0.00

Control Treatment
Condition Condition

Overlapping
Distributions

02/25/11 38
 Moderate Effect Size

ES = 0.40

Control Treatment
Condition Condition

02/25/11 39
 Large Effect Size

ES = 0.85

Control Treatment
Condition Condition

02/25/11 40
 Mean and Variability

ES+

Variability
Note: Results from Bernard, Abrami, Lou, et al. (2004) RER
02/25/11 41
 10 Steps in a Meta-Analysis

8. Develop a codebook
Study feature coding involves describing the relevant
characteristics for each study (e.g., research
methodology, publication source).The codebook
details the study feature categories and their levels.

02/25/11 42
 Examining Study Features

• Purpose: to attempt to explain variability in

effect size
• Any nominal, ordinal or interval coded study
feature can be investigated
• In addition to mean effect size, variability
should be investigated
• Study features with small ks may be
unstable

02/25/11 43
Examples of Study Features

• Research methodology
• Type and nature of measures
• Direction of the statistical test
• Publication data
• Relevant aspects of the treatment
• Relevant aspects of the control condition

02/25/11 44
 10 Steps in a Meta-Analysis

9. Code studies for study features

Coding study features is perhaps the most time-

consuming and onerous aspect of conducting a
meta-analysis.
However, it is arguably the most important step
because it provides the possibility for explaining
variability in effect sizes.

02/25/11 45
 10 Steps in a Meta-Analysis

10: Analysis and interpretation

Analysis involves invoking a range of standard

statistical tests to examine average effect sizes,
variability and the relationship between study
features and effect size. Interpretation is drawing
conclusion from these analyses.

02/25/11 46
 Questions:
Statistical Analysis
• What techniques will be used to combine results
of separate tests?
• What techniques will be used to assess and then
analyze the variability in findings across studies?
• What sensitivity analyses (i.e., tests of the impact
of such decisions on the results of the review) will
be carried out and how?
• What statistical procedures will be used to test
relationships between study features and effect
sizes (e.g., meta regression)

02/25/11 47
 Homogeneity vs.
Heterogeneity of Effect
Size
• If homogeneity of effect size is
established, then the studies in the meta-
analysis can be thought of as sharing the
same effect size (i.e., the mean)
• If homogeneity of effect size is violated
(heterogeneity of effect size), then no
single effect size is representative of the
collection of studies (i.e., the “true” mean
effect size remains unknown)

02/25/11 48
02/25/11
 Statistics in Comprehensive
Meta-Analysis™
Effect size and 95% confidence interval Test of null (2-Tail)
Number Studies Point estimate Standard error Variance Lower limit Upper limit Z-value P-value
168 0.34 0.01 0.00 0.31 0.36 23.28 0.00

Heterogeneity
Q-value df (Q) P-value
1816.71 167.00 0.00

Interpretation: Moderate ES for all outcomes (g+ = 0.34) in favor of

the intervention condition.
Homogeneity of ES is violated. Q-value is significant (i.e., there is too
much variability for g+ to represent a true average in the population).

Comprehensive Meta-Analysis 2.0 is a trademark of BioStat®

02/25/11 50
 Examining the Study
Feature “Type of Research
Design”

Overall
Effect
g+ = +0.34
Pre-Post Quasi-Exp.
Designs Designs

Post-Only
Designs

02/25/11 51
 Tests of Levels of “Type of
Research Design”
Groups Effect size and 95% confidence interval Heterogeneity
Group N of Studies Point estimate Standard error Lower limit Upper limit Q-value df (Q) P-value

one-group 27 0.16 0.04 0.09 0.24 181.30 26.00 0.00

post only 87 0.38 0.02 0.34 0.42 651.34 86.00 0.00
quasi-exp 54 0.35 0.02 0.31 0.40 957.45 53.00 0.00

Total within 1790.09 165.00 0.00

Total between 26.62 2.00 0.00
Overall 168 0.34 0.01 0.31 0.36 1816.71 167.00 0.00

Interpretation: Small to Moderate ESs for all categories in favor of

the intervention condition.
Homogeneity of ES is violated. Q-value is significant for all categories
(i.e., type of research design does not explain enough variability to
reach homogeneity.

02/25/11 52
 Sensitivity Analysis

• Tests the robustness of the findings

• Asks the question: Will these results stand
up when potentially distorting or deceptive
elements, such as outliers, are removed?
• Particularly important to examine the
robustness of the effect sizes of study
features, as these are usually based on
smaller numbers of outcomes

02/25/11 53
 Selected References

Bernard, R. M., Abrami, P. C., Lou, Y. Borokhovski, E., Wade, A.,

Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How
Does Distance Education Compare to Classroom
Instruction? A Meta-Analysis of the Empirical Literature.
Review of Educational Research, 74(3), 379-439.
Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in
social research. Beverly Hills, CA: Sage.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-
analysis. Orlando, FL: Academic Press.
Hedges, L. V., Shymansky, J. A., & Woodworth, G. (1989). A
practical guide to modern methods of meta-analysis. [ERIC
Document Reproduction Service No. ED 309 952].

02/25/11 54