CRICOS Provider No. 00300K (NT/VIC) I 03286A (NSW) | RTO Provider No.

0373

PSY417: Research Methods and Practice

Week 11 – Synthesizing statistics

Faculty Of Health
Dr. Rebecca Williams
SEM1 2023
 Week # Topic
Readings (textbook) Assessments Due
(Date Beginning)

Week 1
Introduction to the course and review of univariate statistics Chapters 1 & 2
(06/03/2023)

Week 2
(13/03/2023) Non-parametric models Chapters 6 & 7

Online quiz (Opens

Week 3 22/03/2023 at 5:00PM ACST
Logistic regression
(20/03/2023) Chapter 20 and closes 24/03/2023 at
5:00PM)

Week 4
Moderation and mediation Chapter 11
(27/03/2023)

Week 5
Advanced ANOVA: Repeated measures and mixed designs Chapters 15 & 16
(03/04/2023)

(10/04/23) Semester Break

Week 6
ANCOVA and MANOVA Chapters 13 & 17
(17/04/2023)

Week 7 Oral presentation (In class, 15

Oral Presentations
(24/04/2023) minutes)

Week 9
Factor analysis and principal component analysis Chapter 18
(08/05/2023)

Week 10
Structural Equation Modelling
(15/05/2023)

Week 11
Synthesizing statistics// Reading and writing scientific reports
(22/05/2023)

Week 12 Report due (1500 words,

Machine learning for psychology
(29/05/2023) 02/06/2023 by 5:00PM ACST)

(05/06/2023) Revision Week

Final online exam (Opens

14/06/2023 at 5:00PM ACST
(12/06/2023) Centrally organised examination period and closes 16/06/2023 at
5:00PM) 2
Today’s Lecture Outline

• The replication crisis

• Types of reviews
• Systematic review
• Meta-analysis

• Journal club

• Writing scientific reports

3
 What crisis?

In 2015(ish), a research lab failed Replication is vital in psychology

to reproduce the findings from as it is how we distinguish real
100 top psychology studies findings from chance

The cause of the replication

One suggested way to combat
crisis might be shoddy research
the crisis is Open Science
practices
4
TED talk on Open Science as a possible way
to address the crisis

https://www.youtube.com/watch?v=c-bemNZ-IqA 5
Open Science Framework (OSF)

6
Big open databases

7
Big open databases

Epidemiological, cognitive, and imaging data

8
Big open databases

9
Journals that only publish papers describing
openly available datasets

- Journal of Open Psychology Data

- Scientific Data

10
Using openly available data is cheap,
efficient and addresses some of the issues of
the replication crisis

But you can still collect your

own data.

You can even do both (use

large dataset + collect your
own data)

11
Reviewing the literature

• The replication crisis can be addressed with open

research practices and re-using available data.

• Another approach is to review the literature and find

consistencies...

12
Today’s Lecture Outline

• The replication crisis

• Types of reviews
• Systematic review
• Meta-analysis

• Journal club

• Writing scientific reports

13
Performing a literature review
• Non-systematic reviews are often entered into informally, using
subjective methods to find studies and interpret findings.

• The methods underlying non-systematic reviews might not be

explicitly stated, so there is no way to detect bias.

• Think about your own literature review

• What kind of bias do you think might be present?
• Would someone else come to the same conclusion?
14
 Types of literature reviews

15
Source: Grant & Booth (2009).
Systematic review
• Identifies, appraises, and synthesizes all evidence on
a topic or research question.

• The crucial element of a systematic review is that

there is a system
• Methods should be explicitly stated
• The aim should be to objectively determine the state of
knowledge on a particular topic
16
Steps to performing a systematic review
1. Formulate question
2. Develop protocol
3. Conduct search of selected databases
4. Screen results for studies that meet eligibility criteria
5. Critically appraise the quality of the studies for risk of bias
6. Extract and synthesize relevant data from included studies
7. Summarise and interpret evidence
8. Present findings
17
1. Formulate question

• Use PICO or PICOS for

quantitative studies.

• SPIDER is for qualitative

studies.

18
 1. Formulate question

Example research question

using PICO framework

Source: libguides.Murdoch.edu.au/systematic/ 19
2. Develop protocol
• Protocol should include
rationale, objectives,
eligibility criteria, search
terms.

• Use PRISMA flow chart

(Preferred Reporting Items for
Systematic Review and Meta-
Analyses).
20
2. Develop protocol
• Consider prospective registration

• PROSPERO is an open-access prospective

registration platform for planned or on-
going systematic reviews

• Registration improves transparency,

accountability and reduces unplanned
duplication.

• Some journals require registration to

publish.

21
3. Conduct search of selected databases
• Where to search?
• Databases such as PubMed, PsycInfo
• Grey literature
• Reference lists

• What to search for?

• Keywords, Medical Subject Headings (MeSH)

• How to search?
• Use Boolean operators (OR, AND)
• Apply limits (per inclusion criteria)

22
4. Screen results
Does each individual article meet your inclusion criteria?
• Export search results to EndNote library
• Screen titles and abstracts to remove obviously irrelevant
material
• Examine full text of article for detailed screening against
eligibility criteria (need to provide reasons why documents
are excluded)
• Follow the PRISMA diagram
• Always good to have more than 1 person screen results
23
5. Critically appraise the quality of the
studies for risk of bias

Joanna Briggs Institute

(JBI) tool for
systematic reviews
(Aromataris et al.,
2015)

24
6. Extract and synthesize relevant data from
included studies
• The findings of all included studies are combined and
evaluated

• Quantitative information warrants groupings for similar

data, charts, and graphical displays

• Can also do a more formal meta-analysis

25
 6. Extract and synthesize relevant data from
included studies

Forest plot for

systematic reviews
and meta-analyses

26
Source: Forman-Hoffman et al. (2018)
7. Summarize and interpret
• Results section will incorporate most of the systematic
review process
• How it went
• Usually ends of risk of bias

• Discussion section will read just like any other

• Summary
• Interpretation
• Strengths and weaknesses
27
Today’s Lecture Outline

• The replication crisis

• Types of reviews
• Systematic review
• Meta-analysis

• Journal club

• Writing scientific reports

28
Meta-analysis
• Very similar to systematic review, however data are extracted
from included articles

• Data often includes sample size, measures of central tendency,

variability, effect sizes (Cohen’s d)

• Data extraction can be of either the aggregated group results

or of the individual participant data
• Aggregate data is the most common
29
Meta-analysis
• Same steps as a systematic review, however quantitative data
are extracted from included articles

• Data often includes sample size, measures of central tendency,

variability, effect sizes (Cohen’s d)

• Data extraction can be of either the aggregated group results

or of the individual participant data
• Aggregate data is the most common
30
 Meta-analysis
• Data from each study (effect
sizes) are entered into
statistical models, with the goal
of assessing the overall effects.

• Models can be either fixed

effects or random effects
• Random effects includes extra
variance partition to account
for between-study variance

Source: Gurevitch et al. (2018) 31

Today’s Lecture Outline

• The replication crisis

• Types of reviews
• Systematic review
• Meta-analysis

• Journal club

• Writing scientific reports

32
33
Introduction
• Mental wellbeing is linked to health, development, longevity, risk of developing (and
chance of recovery from) mental illness.

• Mental wellbeing is operationalized as:

• Subjective wellbeing (affect and life satisfaction)
• Psychological wellbeing (meaning or purpose, positive relationships)

• Mental wellbeing appears to be independent from states of mental illness. Both

need to be assessed with dedicated scales, and psychological interventions can lead
either to improvements in both mental wellbeing and indicators of illness, or only
one of these domains.

• A visualization of this relationship is shown in the following figure:

34
Introduction

35
Introduction
• This research paves the way for implementation of interventions
and therapeutic approaches that focus on the promotion of
mental wellbeing regardless of the presence of physical or
mental illness.

• Interventions that target symptoms of mental illness and those

that target mental wellbeing can focus on shared processes—for
example, reducing rumination or building a sense of purpose—
but can also focus on differential antecedents or target areas—
for example, guilt or hopelessness for mental illness or personal
growth for wellbeing.
36
Introduction
• However, our current understanding of the impact of the complete landscape of psychological
interventions has traditionally been obstructed for a number of reasons.

• First, the separation of research on practice in wellbeing and mental illness. The main body of
research investigating the impact of psychological interventions on wellbeing stems from the field
of positive psychology, and looks at positive psychological interventions (PPIs). These are defined as
“treatment methods or intentional activities that aim to cultivate positive feelings, behaviours, or
cognitions.”

• PPIs, however, comprise only a very limited subset of psychological approaches and typically do not
address focus areas of traditional psychological interventions, such as rumination and worry. There
is a considerable evidence gap regarding the impact of other psychological interventions.

• Interventions stemming from different therapeutic paradigms can target a variety of underlying
processes related to mental health and wellbeing. 37
Introduction
• Second, there has been less focus on determining the impact of different types
of psychological interventions in building mental wellbeing in clinical
populations. There is a need to synthesize the research on the efficacy of
psychological interventions to build wellbeing of populations that suffer from
physical illness, both acute and chronic, as mental health status has an
important role in disease management and treatment.

• Third, interpreting results of existing independent syntheses poses challenges

as a result of their methodological differences. Mental wellbeing is a broad
concept with differing definitions. It is important to synthesize studies that
have used similar subjective methods to measure wellbeing.

38
Research aims

This systematic review and meta-analysis aimed to create a

clear synthesis of the impact of different types of
psychological interventions on mental wellbeing,
irrespective of their theoretical foundation, and establish the
differential impact of these types of interventions in clinical
and non-clinical populations.

39
Methods
1. Registered in PROSPERO
2. Study selection criteria outlined (5 criteria).
- Randomized controlled designs only
- Specific control groups (assessment only or waitlist; passive or
active control group; treatment-as-usual for physical/mental
illness).
3. Search strategy, study selection and data extraction
- Databases indicated and date of publication cut-off (July 2020).
- Two authors screened all titles and abstracts for eligibility. Inter-
rater reliability (Kappa = 0.85).
- All extracted data independently checked by co-authors.
40
41
Methods
4. Data synthesis and analysis
- Outcome data standardized to Hedge’s G (99% CI, P-values calculated
using dedicated meta-analysis software).
- Separate overall effect sizes were calculated for subjective and
psychological wellbeing.
- Interventions were analysed and presented per population group (non-
clinical, mentally ill, physically ill).
- Overall effect size was calculated using random-effects models.
5. Moderators or subgroup analyses
- Intervention: Type of intervention, mode of delivery, intensity of
intervention.
- Study specific: Different control groups, assessment follow-up, study
quality.
42
Results
Study sample and design characteristics
- 393 studies met inclusion criteria and provided
sufficient information to be included in the
quantitative analysis.
Risk of bias
- Reported for each of the 393 studies.
- Determined using Cochrane risk-of-bias tool
43
44
Results

45
Results
Looking at comparator groups, effect sizes were larger
when studies used a waitlist-control or assessment-only
design. When studies used a control group that included
some form of interaction (for example, a placebo control
group), effect sizes were much lower, with effect sizes in
the general population dropping below the threshold for
a small effect.

46
Discussion
• Mental wellbeing can be significantly improved using a
variety of psychological interventions.

• Multi-component PPIs and mindfulness-based interventions

were consistently associated with positive findings across
populations.

• Moderator analysis indicated that improvement is related to

exposure (more exposure = better treatment outcomes).
47
Today’s Lecture Outline

• The replication crisis

• Types of reviews
• Systematic review
• Meta-analysis

• Journal club

• Writing scientific reports

48
Mack (2018): How to write a good scientific
paper
Introduction
1. Present the field, why it is important, what has been
done.
2. Indicate a gap or raise a research question.
3. Outline the purpose and announce the present
research. Clearly indicate why it is significant.
4. Avoid unnecessary background information,
exaggeration of the importance of the work. 49
Mack (2018): How to write a good scientific
paper
Methods
1. Describe how the results were generated with sufficient detail so that
an independent researcher could reproduce the results sufficiently.
2. Make sure reader can assess internal validity (conclusions are
supported by the results presented) and external validity (conclusions
are properly generalized beyond these specific results).
3. Has the chosen method been justified?
4. Are data analysis and statistical approaches justified? Assumptions and
biases considered?
5. Avoid including results.
50
Mack (2018): How to write a good scientific
paper
Results
1. Present the results of the paper in logical order, using
tables and graphs as necessary.
2. Factual statements supported by evidence. Short and
sweet without excess words.
3. Show how the results help answer the research questions
posed in the introduction.
4. Avoid redundancy- no need to repeat information that is in
the tables or captions.
51
Mack (2018): How to write a good scientific
paper
Discussion
1. Summarize the results (might help to reiterate the research goal).
2. Discuss whether the results were expected or unexpected. Compare to
previous work.
3. Interpret and explain the results – often by comparison to a theory or
model.
4. Discuss any problems or shortcomings.
5. Possible alternate explanations for the results?
6. Avoid presenting discussion that does not relate to any of the results.
52
Mack (2018): How to write a good scientific
paper
Conclusions
1. Provide a very brief summary of the results and discussion.
2. Emphasize the implications of the findings- explain how the
work is significant and provide the key “take-home” messages.
3. Provide the most general claims that can be supported by the
evidence.
4. Provide a future perspective on the work.
5. Avoid introducing new evidence or arguments not found in the
results and discussion. 53
References
• Aromataris, E., Fernandez, R., Godfrey, C., Holly, C., Kahlil, H. & Tungpunkom, P. (2015). Summarizing
systematic reviews: methodological development, conduct and reporting of an Umbrella review approach.
Int J Evid Based Healthc, 13(3), 132-40.
• Forman-Hoffman, V., Middleton, J.C., Feltner, C. et al. (2018). Psychological and pharmacological treatments
for adults with post-traumatic stress disorder: A systematic review update. Comparative Effectiveness
Review, 207.
• Grant, M.J. & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated
methodologies. Health Information and Libraries Journal, 26(9), 91-108. https://doi.org/10.1111/j.1471-
1842.2009.00848.x
• Gurevitch, J., Koricheva, J., Nakagawa, S. & Stewart, G. (2018). Meta-analysis and the science of research
synthesis. Nature, 555, 175-182. doi:10.1038/nature25753
• Mack, C. (2018). How to write a good scientific paper. SPIE: https://spie.org/samples/9781510619142.pdf
• Methley, A.M., Campbell, S., Chew-Graham, C., McNally, R. & Cheraghi-Sohi, S. (2014). PICO, PICOS and
SPIDER: A comparison study of specificity and sensitivity in three search tools for qualitative systematic
reviews. BMC Health Services Research, 14(1), 1-10.
• Page, M.J., McKenzie, J.E., Bossuyt, P.M., Boutron, I., Hoffmann, T.C., Mulrow, C.D., et al. (2021). The PRISMA
2020 statement: an updated guideline for reporting systematicreviews. BMJ, 372(71). doi: 10.1136/bmj.n71
54