Critical Appraisal of the Literature

Dr Brioni Moore

Senior Research Fellow

Curtin Medical School
Learning outcomes
 Understand the key principles of critical appraisal

 Why do we critically appraise literature?

 Understand how critical appraisal procedures are

applied to clinical trial literature.
What is critical appraisal?
 “The process of carefully and systematically examining
research to judge its trustworthiness, and its value and
relevance in a particular context”
Burls A. (What is...? Series) 2nd ed. Oxford, UK: University of Oxford; 2009.
What is Critical Appraisal?

 Critical appraisal evaluates evidence for its:

 Validity (has bias been minimized?)
 Trustworthiness of Results (what were the study findings?)
 Relevance (are the findings relevant to the population of
interest?)
Validity
 Soundness or ‘rigor’ of the data and the extent to
which conclusions are true within the specific context of
the research.
 Clear statement of aims?
 Appropriate methods?
 Relevant participants selected

 Role of bias
Trustworthiness of results
 Ensure that the findings are not skewed by any
potential bias or personal motivations of the researcher
 Clear statements of findings
 Are the results precise?
 What data analysis was conducted?

 Evidence based research

Relevance
 The extent to which the research is useful for others
 Scientific relevance
 Societal relevance
 What is the practical relevance of the findings?
 Were all important outcomes considered?
 Can the results be applied locally?
Why is it important?
 Critical appraisal allows us to:
 Ensures a thorough assessment of the research.
 Recognizes strengths and weaknesses of the research.
 Reduce information overload by eliminating irrelevant or
weak studies.
 Distinguish evidence from opinion, assumptions,
misreporting, and belief.
 Assess the validity of the study.
 Assess the usefulness and clinical applicability of the study
 Recognize any potential for bias.
Core questions for evaluation
Key questions for consideration
1. Is the study relevant to my field?
2. Does the study add anything new to the evidence in the field?
3. What type of research question is being asked?
4. Was the study design appropriate for the research question?
5. Did the methodology address important potential sources of
bias?
6. Was the study performed according to the original protocol?
7. Does the study test a stated hypothesis?
8. Were the statistical analyses performed correctly?
9. Do the data justify the conclusions?
10. Are there any conflicts of interest?
11. Is the study ethical?
Critical appraisal tools
 Over 120 published critical appraisal tools
 Most for experimental studies
 Defined tools for systematic reviews
 Limited number for case studies

 Most common tools for evaluating the quality of

published research.
 Checklists
 Reporting standards
JBI Critical appraisal tool
EQUATOR Network
EQUATOR Network
Where do we start?

Antimicrobial Agents and

Chemotherapy. 2019, 63 (10) e00302-
19.
 Where do we start?
1. Look for a research problem or problem statement
 Usually located in the abstract, title or introduction

2. What is the purpose of the study?

3. Is it a literature review?

4. Is there a hypothesis or research question?

5. What is the sample population of interest?

6. What is the type of research or study design?

Subramanyam R. J Oral Maxillofac Pathol. 2013;17(1):65-70.

The research question

Moore, BR, et al. AAC

(2019); 63 (10) e00302-19.
https://usu.instructure.com/courses/45089/pages/anatomy-of-a-scientific-article
Is the research ethical?
 Was there a need to conduct the trial?

 Was the study approved by an ethics committee?

 Did participants (or their guardians) give informed

consent?

 Was the trial of sufficient size to reach a

meaningful conclusion?

 Was there an independent Data Safety Monitoring

Board (DSMB)?

 What role did the funder(s) play?

 Were there any conflicts of interest?

Study power
 Power and sample size calculations are measures of
how many participants are needed in a study to answer
a research question.
 Clinical studies must include appropriate power calculations,
otherwise research may be deemed unethical.

 Power is the probability of rejecting a null hypothesis

 The higher the power, the more likely you will be able to
detect a significant effect

 The size of the sample influences two statistical

properties
 Precision of estimates
 Power of the study to draw conclusions.
Study methods
 How well was the study conducted?

 You MUST read and critically evaluate the methods

section
 What was the study/trial design?
 What was the recruitment group?
 How did they allocate treatment? Was it randomised?
Is there any randomization bias?
 Was the study blinded? If not, does this introduce any
bias?
 What were the study procedures?
 How did they measure outcomes?

 Is/are the aim/s of the study well described?

Study methods
 Has bias been minimized?
 Effective randomization
 ‘matching’ participants
 Exposure to treatment
 Study blinding?

 Three main types of bias

 Information
 Selection
 Confounding

Pannucci, C. J., & Wilkins, E. G. (2010) Plastic and reconstructive surgery, 126(2), 619–625.
Results
 How many patients were included in the trial,
compared with the planned sample size?

 Did the randomization work?

 Did any participants withdraw from the trial – if so, did

this cause an imbalance on one of the group sizes?
How was this taken into account during analysis?

 What statistics were used to analyse the data?

 What was the primary endpoint?

 What were the main results for secondary endpoints?

Statistics
 A variety of different statistical analyses can be utilized
 What is the research question to be answered?
 Type of data collected.

 Descriptive statistics usually first type of analysis reported

 Mean ± SD or median [range or IQR] depending on whether variables
are continuous
 Number (%) for categorical variables

 Comparative statistics
 T-tests to compare means
 ANOVA for 3 or more means
 X2 analysis for categorical variables

 Interpreting statistical tests

 P-value: <0.05 is statistically significant
 Confidence intervals (95% CI)
 Endpoints
 Generally, should be one primary endpoint (usually efficacy
related) and several secondary endpoints (safety, tolerability,
efficacy)
 Major focus is primary outcome
 Secondary outcomes provide supporting data

 Endpoints should:
 Be defined prior to starting the study
 Capture the most important aspects of the research
 Be relevant
 Appropriate for the population of focus
Critical appraisal results
Summary
 Learning the art of critical appraisal will take time and
practice but is well worth the effort.

 Using available checklists will aid in learning the

process by which you critically appraise scientific
literature.

 Ensuring that literature is accurate, from a reputable

source, accurate and current is integral for ensuring the
quality of your own research assignments.
References
 Burls A. What is…? Series. 2nd ed. Evidence based medicine. Hayward Group Ltd.; 2009.

 Critical Appraisal Skills Programme. Randomised Control Trial Standard Checklist.[online]

Available at: https://casp-uk.b-cdn.net/wp-
content/uploads/2020/10/CASP_RCT_Checklist_PDF_Fillable_Form.pdf

 La Trobe University Library Website: Critical appraisal for health.[online] Available at:
https://latrobe.libguides.com/criticalappraisal/study-design

 Moore BR, et al. 2019. A randomised open-label evaluation of the antimalarial prophylactic
efficacy of azithromycin-piperaquine versus sulfadoxine-pyrimethamine in pregnant Papua
New Guinean women. Antimicrobial Agents and Chemotherapy, 63(10), e00302-19.

 Pannucci, C. J., & Wilkins, E. G. (2010). Identifying and avoiding bias in research. Plastic
and reconstructive surgery, 126(2), 619–625.

 Sabin, C. Session 5: How to critically appraise a paper. UK-CAB[online] Available at:

http://www.ukcab.net/wp-content/uploads/2013/10/How-to-critically-appraise-a-paper-
Caroline-Sabin.pdf

 Umesh, G., Karippacheril, J. G., & Magazine, R. (2016). Critical appraisal of published
literature. Indian journal of anaesthesia, 60(9), 670–673.