CAT

(Critical Appraisal of the Topics)

Warko Karnadihardja, Reno Rudiman

Dept. of Surgery, Hasan Sadikin Hospital
Universitas Padjadjaran
Bandung
Critical Appraisal of the Topics

 Requires certain basic skills : To define the

clinical question as clearly as possible and
translate the question into a statement
 This statement can be used to search
literature, conduct the search, and select the
best articles before applying appropriate rules
of evidence to evaluate the articles

Mc Kibbon, ACP Journal Club, 1994. 120 (Suppl 2): A 10-12

Types of Studies

Cross
Sectional

Case Control Cohort study

Case
T i m e l i n e
Report
Case `Series Clinical Trial
Evidence Based Medicine

Can now be looked at as :

“ The process of life-long, self directed learning
in which caring of patient leads to search for,
critical appraisal and implementation of the
best available evidence in all clinical and
health care related issues with the
incorporation of personal expertise and
patient’s values”

Iqbal Mustafa : EBM Workshop, PERDICI, Jakarta, June 13, 2004

The User’s Guide To The Medical Literature

Three fundamental questions

1. Are the results of the study valid ?
2. What are the results ?
3. Will the results help me in caring for my patients?

Oxman AD et al : JAMA 1993;270: 2093-5

User’s Guides For Selecting Articles Most Likely To
Provide Valid Results

Subject Screening Criteria

Original Studies
 Therapy _____ ?
 Diagnosis _____ ?
 Harm _____ ?
 Prognosis _____ ?
Integrative Studies
 Overview _____ ?
 Practice Guidelines _____ ?
 Decision Analysis _____ ?

Modified from American Medical Association. JAMA 1993; 270(17):2094

User’s Guides for Selecting Articles Most Likely to Provide
Valid Results
Subject Screening Criteria
Original Studies
Therapy Was the assignment of patients to treatments randomized ?
Were all of the patients who entered the trial properly accounted for and
attributed at its conclusion ?
Diagnosis Was there an independent, blind comparison with a reference standard ?
Did the patient sample include an appropriate spectrum of the sort of patients
to whom the diagnostic test will be applied in clinical practice ?
Harm Were there clearly identified comparison groups that were similar with respect
to important determinants of outcome (other than the ones of interest) ?
Were outcomes and exposures measured in the same way in the groups being
compared ?
Prognosis Was there a representative patient sample at a well defined point in the course
of disease ?
Was follow-up sufficiently long and complete ?
User’s Guides for Selecting Articles Most Likely to Provide
Valid Results

Subject Screening Criteria

Integrative Studies
Overview Did the review address a clearly focused question ?
Were appropriate criteria used to select articles for inclusion ?
Practice guidelines Were options and outcomes clearly specified ?
Did the guideline use an explicit process to identify, select and combine
evidence ?
Decision analysis Did the analysis faithfully model a clinically important decision ?
Was valid evidence used to develop the baseline probabilities and
utilities ?

Modified from American Medical Association. JAMA 1993; 270(17):2094

User’s Guides for an Article about Therapy

Are the results of the study valid ?

1. Primary Guides ________ ?
2. Secondary Guides ________ ?
3. What were the results ________ ?
4. Will the results help me in caring for my patients ?
User’s Guides for an Article about Therapy
Are the results of the study valid ?
Primary guides
 Was assignment of patients to treatments randomized ?
 Were all patients who entered the trial properly accounted for and
attributed at its conclusion ?
 Was follow-up complete ?
 Were patients analyzed in the groups to which they were
assigned ?

Secondary guides
 Were patients, health workers, and study personnel blind to
treatment ?
 Were the groups similar at the start of the trial ?
 Aside from experimental intervention, were the groups treated
equally ?
User’s Guides for an Article about Therapy

What were the results ?

 How large was the treatment effect ?
 How precise was the estimate of the treatment effect ?

Will the results help me in caring for my patients ?

 Can the results be applied to my patient care ?
 Were all clinically important outcomes considered ?
 Are the likely treatment benefits worth the potential harm and
costs ?

Modified from American Medical Asociation. JAMA 1994;271(1):60

Statistics for interpreting the importance and precision
of therapeutic results
  Yes No

Exposed a b

Not Exposed c d

Control event rate (CER) = c/c+d

Experimental event rate (EER) = a/a+b
(a) Relative Risk (RR) = EER/CER=(a/a+b)/(c/c+d)
(b) Relative Risk Reduction (RRR) = CER-EER/CER
(commonest reported measure of dichotomous treatment effect)
(c) Absolute Risk Reduction (ARR) = CER-EER
(d) Number Needed to Treat (NNT) = 1/ARR
User’s Guide’s for Evaluating and Applying Results of
Studies of Diagnostic Tests

Are the results of the study valid ?

 Primary guides
 Secondary guides
 What were the results ?
 Will the results help me in caring for my patients ?
User’s Guide’s for Evaluating and Applying Results of
Studies of Diagnostic Tests

Are the results of the study valid ?

Primary guides
 Was there an independent, blind comparison with a reference
standard ?
 Did the patient sample include an appropriate spectrum of
patients to whom the diagnostic test will be applied in clinical
practice ?

Secondary guides
 Did the results of the test being evaluated influence the
decision to perform the reference standard ?
 Were the methods for performing the test described in
sufficient detail to permit replication ?
User’s Guide’s for Evaluating and Applying Results of
Studies of Diagnostic Tests

What were the results ?

 Are the likelihood ratios for the test results presented or are
data necessary for calculation of likelihood ratios provided ?

Will the results help me in caring for my patients ?

 Will the reproducibility of the test result and its interpretation be
satisfactory in my setting ?
 Are the results applicable to my patients ?
 Will the results change my management ?
 Will patients be better off as a result of the test ?

Modified from American Medical Asociation. JAMA 1994;271(5):390

Calculations for Diagnostic Tests
Gold standard
Gold standard
Positive (condition
Negative (condition
present)
not present ) 

Test result True Positive False Positive

Positive

Test result False Negative True Negative

Negative

Stable Properties:
Sensitivity = True Positives/(True Positives + False Negatives)
Specificity = True Negatives/(False Positive + True Negative)
Frequency Dependent Properties:
Positive Predictive Value = True Positive/(True Positive + False
Positive)
Negative Predictive Value = True Negative/(True Negative + False
Negative)
Likelihood Ratios
The likelihood ratio for a test result compares the likelihood of that result in
patients with disease to the likelihood of that result in patients without disease:

  Condition Condition Absent

Present
Test Positive a b

Test Negative c d

Positive LR = (a/a+c)/(b/b+d)
Negative LR = (c/a+c)/(d/b+d)
Causation/Harm/Etiology

Are the results of this study valid ?

Were there clearly defined groups of patients,
similar in all important ways other than exposure
to the treatment or other cause?

Were treatments/exposures and clinical

outcomes measured in the same ways in both
groups (was the assessment of outcomes either
objective or blinded to exposure)?
Was the follow-up of study patients complete and
long enough?
Causation/Harm/Etiology
Is it clear that the exposure preceded the onset of the
outcome ?
Is there a dose-response gradient ?

Is there positive evidence from

“dechallenge-rechallenge” study ?

Is the association consistent from study to

study ?

Does the association make biological

sense ?
Calculations for Causation/Harm/Etiology
Odds Ratios and Relative Risk
  Outcome Outcome
Positive Negative

Exposure a b

No Exposure c d

  Appropriate use Formulae

Relative RR= Interpreting strength of association

Risk  Randomized (a/a+b)/(c/c+d) For randomized trial, strong
controlled association if RR or OR > 1
trials For cohort study, strong association
 Cohort if RR > 3 or OR > 4
studies
Odds Ratio OR=
 Case-control (a/c)/(b/d)=ad/
studies bc
Prognosis
Are the results of this prognosis study valid ?

Was a defined, representative sample of patients

assembled at a common (usually early) point in
the course of their disease?
Was patient follow-up sufficiently long and
complete?

Were objective outcome criteria applied in a

"blind" fashion?

If subgroups with different prognoses are

identified, was there adjustment for important
prognostic factors?
Was there validation in an independent group
("test set") of patients?
Prognosis

Are the results of this prognosis study important?

How likely are the outcomes over

time ?
How precise are the prognostic
estimates ?
Calculating a confidence interval around the measure
of prognosis

Clinical Measure Standard Error (SE) Typical Calculation of

CI

Proportion (as in the rate If p = 24/60 = 0.4 (or

of some prognostic 40%) and n = 60
event, etc.) where: the
number of patients
= n the proportion of where p is proportion
and n is number of = 0.063 (or 6.3%)
these patients who
experience the event patients 95% CI is
=P 40% ± 1.96 × 6.3% or
27.6% to 52.4%
 Limitations of CATs

1. Individual CATs can be wrong. CAT can appear as

drafts, without peer review.
2. Individual CATs contain a single element of the
relevant literature.
3. Individual CATs may have a short shelf life. They
become obsolete as soon as newer, better evidence
becomes available.
 Level 1 of Evidence
Level Therapy/Prevention, Prognosis Diagnosis
Aetiology/Harm

1a SR (with homogeneity*) SR (with homogeneity*) SR (with homogeneity*) of

of RCTs of inception cohort Level 1 diagnostic
studies; CDR† studies; CDR† with 1b
validated in studies from different
different clinical centres
populations

1b Individual RCT (with Individual inception Validating** cohort study

narrow Confidence cohort study with > with good††† reference
Interval‡) 80% follow-up; standards; or CDR†
CDR† validated in a tested within one
single population clinical centre

1c All or none§ All or none case-series Absolute SpPins and SnNouts

††
 Level 2 of Evidence
Level Therapy/Prevention,
Aetiology/Harm
2a SR (with homogeneity* ) of
cohort studies
2b Individual cohort study
(including low quality RCT;
e.g., <80% follow-up)
2c Outcomes" Research;"
Ecological studies
Prognosis Diagnosis
SR (with homogeneity*) of SR (with homogeneity*) of
either retrospective cohort Level >2 diagnostic studies
studies or untreated
control groups in RCTs
Retrospective cohort Exploratory** cohort study with
study or follow-up of good†††reference standards;
untreated control patients CDR† after derivation, or
in an RCT; Derivation of validated only on split-
CDR† or validated on sample§§§ or databases
split-sample§§§ only
Outcomes" Research"  
 Level 3,4,5 of Evidence
Level Therapy/Prevention, Prognosis Diagnosis
Aetiology/Harm
3a SR (with homogeneity*) of   SR (with homogeneity*) of 3b
case-control studies and better studies
3b Individual Case-Control   Non-consecutive study; or
Study without consistently applied
reference standards

4 Case-series (and poor quality
cohort and case-control
studies§§ )
5 Expert opinion without
explicit critical appraisal, or
based on physiology, bench
"research or "first principles
Case-series (and poor
quality prognostic cohort
studies***)
Expert opinion without
explicit critical appraisal,
or based on physiology,
bench research or "first
"principles
Case-control study, poor or
non-independent reference
standard
Expert opinion without explicit
critical appraisal, or based on
physiology, bench research or
""first principles
Grades of Recommendation

A consistent level 1 studies

B consistent level 2 or 3 studies or

extrapolations from level 1 studies
C level 4 studies or extrapolations from
level 2 or 3 studies
D level 5 evidence or troublingly
inconsistent or inconclusive studies of
any level
 Old world EBM world
Source of knowledge Expert opinion analysis of the evidence
Essential skills Clinical Clinical plus ability to appraise evidence

Essential information sources Experts Electronic access to all research

Textbooks  evidence
Selected journals Cochrane Library

Importance of statisticians, epidemiologist, Low High

economists, etc

Importance of gathering new evidence on Low High

patients

Consultant to Juniors Dictatorship Democratic

Importance of keeping up to date Optional Essential

Importance of access to research Low High

evidence

Relationship to patients Expert to pupil Potentially much more equal

BMA/ScHARR

CRITICAL
APPRAISAL
SKILLS
WORKSHOP
Critical Appraisal - What is it and Why is it
important?

Andrew Booth
Senior Lecturer in Evidence Based
Healthcare Information
Why Critical Appraisal?

Theory Practice
 What is critical appraisal?  Why should you get
 How is it done? involved?
 When is it used?  How can you get

 What are some of its involved?

 What resources are there
uses?
to help you?
What is critical appraisal?

“To weigh up the evidence

critically to assess its
validity (closeness to the
truth) and usefulness
(clinical applicability).”
[Adapted from Sackett &
Haynes EBM 1995; 1 : 4-
5].
Background to critical appraisal

 McMaster University
 Clinical Epidemiology
 Problem Based Learning
 Rather than fill students’ heads with facts - cultivate skills
for lifelong learning
 To include searching, filtering, critical appraisal and
“digesting”
 Developed User Guide’s to the Medical Literature
Critical Appraisal Skills Programme

 Criticalappraisal skills
training for the NHS
(Anglia and Oxford)
 Appraises:
 Reliability Validity;
Applicability

http://www.phru.org.uk/
~casp/index.htm
Why is it important?

 Clinicians would have to read

17 articles/day; 365 days per
year to keep up-to-date
(internal medicine)
 Research is of variable
quality
 Only an estimated 1% is
judged clinically relevant
 Which is the 1%?
Why is it important?

A requirement for the “EBM component” of Membership

Exams (e.g. GP, Psych, Family Planning etcetera)
Royal College of Ophthalmologists:
Professional Attitudes and Conduct:
“Through encouragement of personal development, to have
developed a style of care which is :
Scientific (e.g. critical appraisal of the scientific literature,
evidence-based practice and use of information technology and
statistics.) “
How is appraisal done?
 Problem or scenario
 Determine appropriate
source
 Identify relevant article(s)
 Use appropriate checklist
 Assess relative
merits/demerits
 Make overall assessment
(strength of evidence)
 Apply findings (strength
of recommendations)
Principles of appraisal

 Intrinsic not extrinsic factors

 Focussed question [Patient Intervention Outcome
Comparison]
 Structured agenda
 Explicit judgements

 Not an ivory tower exercise - How can I apply

these findings?
When is appraisal used?

 Undergraduate and postgraduate education

 Health technology assessment
 Systematic Reviews (“judgements of methodological
quality”)
 Production of national or local guidelines
 In value-added databases (eg. DARE & NEED)
 Current awareness bulletins and journals

………...and Teaching EBP Workshops

Learning outcomes

Skills learned can be used in 3 important ways:

 Critical appraisal skills can be used to improve clinical decision-
making by the implementation of evidence-based healthcare.
 Evidence can be used to inform decisions about health policy, for
example, in making choices about the prioritisation of
services.
 The approach can be used to empower consumers of the health
services.
Why should you get involved?

 Makes your practice more “scientific”

 Extends existing skills.
 Reduces uncertainty
 Improved profile/prestige
 It’s fun!
Getting started

 Scenario - from real life or invented

 Article - primary or secondary study addressing
the problem in hand
 Checklist - for assessing the study design of the
article [User Guides]
 And optionally, a Crib sheet, digest or
commentary
Appraising a paper

 Threebroad issues need to be considered when

appraising a paper:
 A/ Are the results valid?
 B/ What are the results?
 C/ Will the results help locally?
Systematic reviews - definitions
 Review
A synthesis of results and conclusions of two or
more publications.
 Systematic review
Comprehensive identification and synthesis of
all literature on a given topic.
 Meta-analysis
Statistical technique for combining results of
several studies into a single numerical estimate.
Characteristics

 Summaries
 Comprehensive
 Systematic
 Explicit
 Reproducible
Systematic reviews provide
 Good quality evidence, more
reliable results
 A useful basis for decision making
 Information of greater statistical
significance
 Control over the volume of
available literature
What to look for?

 Review versus systematic review

 Comprehensiveness of search
 focused question
 databases covered
 time period, language
 search strategy
 inclusion, exclusion criteria
 Selection bias addressed
 Assessment of study quality
 Homogeneity/ heterogeneity
What to look for?

 Are the overall results valid?

 How precise are the results?
 Arethe conclusions supported
by the data?

If so,
how do they help my patient care?
Will the results help locally?

 Can the results be applied to my

local population?
 Were all important outcomes
considered?
 Should I change my practice as a
result of this evidence?
Sources of reviews
 Cochrane Collaboration
 Cochrane Library
 NHS Centre for Reviews and Dissemination
(NHS CRD)
 Database of Abstracts of Reviews of
Effectiveness (DARE)
 NHS Health Technology Assessment
 HTA reports
 InterTASC/NICE
 Trent, South & West, West Midlands
Summary

CATs are a tactic for helping clinical learners teach themselves

how to formulate clinical questions; search for the best
evidence; appraise, organise and summarise this evidence;
integrate it with clinical expertise; and practice evidence-
based medicine. When generated by clinical teams, journal
clubs, or in academic half-days, their educational value is
multiplied. Existing CATs can be used as starting points for
seeking and appraising updates in the relevant evidence.
The CAT-maker assists this process by:
1. carrying out the important clinical calculations;
2. storing appraisals (as well as the search strategies that led
to them); and
3. generating files that can be formatted with word-processors,
stored and printed for other team members.