Critical Appraisal

Therapy
J.EKO WAHONO. R.dr.SpS.M.Kes
SMF I.P.Saraf
RSUD Dr.Soetomo
Surabaya

Principal of critical appraisalprimary research

Involves 3 overall question
What is the PICO of the study, and is it closed
enough to your PICO ?
How well was the study done ?
What do the results mean and could they
have been due to chance ?

Critical appraisal Tx
Your question
(PICO)
Is the study question
the same
as your question ?

Study

What do the
result mean ?

How well was

The study done ?

Wheter wearing elastic stockings on long-haul

flights helps to prevent deep vein trombosis
(DVT)

Population/problem =
flights

Intervention
stockings

Comparator/control
Outcome =

passengers on long- haul

wearing elastic compression

no elastic stockings

symptomless DVT
4

Critical appraisal Tx DVT

Clinical question :
In passengers on long-haul flights, does
wearing elastic compression stockings,
compared to not wearing elastic stockings,
prevent DVT ?

Search terms :
Based on the clinical question (PICO)
(flight*OR travel*) AND stockings*AND (DVT
OR trombosis)
5

Critical appraisal Tx DVT

Search result :
PubMed Clinical Queries therapy,broad), 20
hits(referring to 5 studies and several reviews,
including 1 recent Cochrane review)
For exercise :
Scurr et al (2001). Frequency and prevention of symptomless DVT
in long-haul flights:a randomised trial
The Lancet 357:1485-1489

Authorsconclusion :
Wearing of elastic compression stockings during long-haul air
travel is associated with a reduction in symptomless DVT
6

Analyse
How do we know that the results are valid
and real ?
Wearing elastic stockings is an
intervention --- RCT ok
In the real life
straight to the Cochrane systematic review

Question 1 : Is the PICO of the study

close enough to your PICO ?
If you find a study that will answer your
clinical question
Study PICO match your PICO or not !

Example your PICO

P
I
C

= In patients with rheumatoid arthritis

= Does taking anti-inflammatory drugs
= no treatment or simple analgetics

= Increase or reduce fatique

If study PIC = your PIC

but study O # your O
Study O
Your
O
However :

= reduce joint pain

= fatique

The study may report on some measure of

fatique as secondary outcome
Please decide quickly wheter to critize or not

Question 2 :
How well was the study done ?
The quality of an epidemiological study
Internal validity
Free from bias & confounding factors

Bias
The degree to which the result is skewed away
from the truth
Selection bias
Treament bias
Measurement bias

To overcome bias RCT & Blinding

10

Question 2 :
How well was the study done ?
The quality of an epidemiological study
Internal validity
Free from bias & confounding factors

Confounding factors
Patients features & causal factors
To overcome CF
Both Group are closely matched/similar
The management of the group is the same

11

How well was bias & confounding factors

were avoided ?
Check each stage of the study How
fairly were :
the subjects recruitted (the P)
the subjects allocated to groups ( the I and C )
the study group maintained through equal
management and follow up of subjects ( the I
and C )
the outcome measured ( the O )
12

Stucture of a comparative health

care research study
Study
question

Aim

Study methods

Critical appraisal

Fair recruitment
Subject representative of the
target pop

Large enaugh sample

+ randomly

Fair allocation

Randomly allocated
Adjust confounding
(statitical
adjustment/matching)

Fair maintenance

Manage grup equally

Follow up all subjects

Fair measurement
Valid & unbiased outcome
measure

Measure outcome
Blinded
Objective measure

M
B
o
13

Steps in critical appraisal

of primary research - RAMMBO
Recruitment :
Were the subjects representative of the target pop ?

Allocation or adjustment

Was the tx allocation concealed before randomisation and were the

groups comparable at the start of the trial

Maintenance :

Was the comparable status of the study groups maintained through

equal management and adequate follow up ?

Measurement

Were the outcomes measured with

Blinded subjects and assessors, and/or

Objective outcomes ?
14

Recruitment
Were the subjects representative of target population ?

If the subject are not representative

difficult to know to which pop may be applicable

The best way to ensure study group

representative is to :
Recruit potential subjects
Sequentially
At random from population

Only apply exclusion criteria relevant for study

methods
Excluding deaf people from study requiring subjects to listen
the music
15

Recruitment
Were the subjects representative of target population ?

Prefer large study,

because small study group
imprecise estimate of the effects !

Continuous outcomes 50 100 pt

Binary outcomes :
Common event hundreds pt
Rare event thousands pt

16

Recruitment
Were the DVT trial subjects representative of target population ?

Inclusion/exclusion criteria
For RCT
difficult random sampling due to inform consent

Clear idea who they do represent

Describe
the severity,
duration and/or
risk level of the patients recruited

17

Recruitment
Were the DVT trial subjects representative of target population ?

Volunteers
Were recruited by placing advertisements in paper
Passengers
> 50 yo
Economy class
At least 8 hours flight within 6 weeks

Various exclusions

Size of study groups

231 116 received stockings & 116 no
This seem small
As a 10% DVT rate 12 events
18

Allocation
Were the study groups comparable ?
It is vital the groups are matched
except for the interventions ( or
exposure/other indicator)

Ways in which groups could differ

Age
Sex
Smoker/nonsmoker
Disease severity

Random allocation
19

Allocation
were the DVT study groups comparable ?
The paper states
Volunteers were randomised by sealed
envelopre to one of two groups

20

Allocation
Characteristic of DVT study groups
No stockings

Stockings

116

115

Age

62(56-68)

61(56-66)

Females

61(53%)

81(70%)

Varicose veins

41

45

Hb

142

140

During study
Hours flying

22

24

Day of stay

17

16

Number
Pre-study

P < 0.01

21

Maintenance
was the comparable status of the study groups
maintained through equal management and adequate
follow-up ?
Once comparable groups have been set up stay a
that way
Equal management
Unequal tx invalidates result !
In a trial of vit E in preterm infants
Vit E appeared to prevent retrolental fibroplasia ?
It was not !
Control groups 100% O2
Tx groups
not 100% O2
because the babies were removed from O2 for freq
dose of vit E
22

Maintenance
Adequate follow-up
Inevitably, some subjects drop out, change
groups or variously lost to follow up during
study uncomparable groups !
Check :
Subject at start = at the end
Subject are analysed in the groups that they
stated out in ( Intention-to-treat principle )

23

Measurement
Were the outcomes measured with Blinded subjects
and accessorrs and/or Objective measures ?

Measurement bias
Human tendency to unfairly nudge results
Can be overcome by
Blinding
Objective measurement

24

.Measurement
Blinding
Best Double-blind trial
Moderate Single blind
Worst Not blinded
Placebo effect

25

Question 3:
What do the result mean ?
Outcome measures
Binary
Continuous

Are the results real and relevent ?

Assesing change :
P-values (hypothesting)
Confidence interval (estimation)
26

Outcome measures for binary

outcomes
Measure
Meaning
Relative risk RR
Risk of outcome
in the Tx
group/risk in the
control group

how many times more

likely event in the tx
group relative to control
group
RR = 1 no diff between 2
groups

RR < 1 the tx reduces

the risk of event
RR > 1 the Tx increass
the risk of the event

Example
RR = 0.1/0.15
= 0.67
RR < 1
The Tx
decrease the
risk of death

27

Outcome measures for binary

outcomes
Measure

Meaning

Example

ARR
Risk of event in
the control group
risk event in the
tx group

ARR = 0 no diff
ARR + the Tx is
beneficial
ARR - harmfull

ARR = 0.15 0.10 =

0.05 (5%)
The absolute benefit
of Tx is a 5%
reduction in the death
rate

28

Outcome measures for binary outcomes

RRR
Measure

Meaning

Example

RRR
ARR/risk of event
in the control
group

RRR
Reduction in the rate
of event in the Tx
group relative to
control group

RRR =
ARR / Risk of event
control group

29

Outcome measures for binary outcomes

NNT
Measure

Meaning

NNT
= 1 / ARR

NNT =

Example

The number of pts

we need to treat in
order to prevent bad
event

30

P-values
Are a measure of the probability that the
result is purely due to chance

31

Confidence interval (CIs)

More informative > P value
An estimate of the range of value that likely to
include the real value
95% means :
The range of values that have a 95% chance of
including the real value

If the 95% CI for the diff between Tx & Control

group
Small
No overlap the no effect point

The result is real ( that is, with a P-value < 0.05)

32

Confidence interval (CIs)

= point estimate

Diff between
Tx vs Control

= CI
Null hypothesis
(no effect)
A

A : statistically sig result (p < 0.05 ) but low precision

B : statistically sig result (p < 0.05 ) with high precision
C : not statistically sig result (p > 0.05 ) with low precision
D : not statistically sig result (no effect ) with high precision
33

Intervention
= usesful if
The 95% CI includes clinically important tx
effects
Statistically sig
Relates to the size of the effect and the 95%
CI in relation to the Null hypothesis

Clinical importance
Relates to the size of effect and the 95% CI in
relation to a minimum effect that would be
considered to be clinically importantce
34

Clinical importance
= point estimate
= CI
Minimum clinical
Important diff

Null hypothesis
(no effect)
A

A : Diff is statistically sig and clinical importance

B : Diff is not statistically sig but is clinical importance
C : Diff is statistically sig but not clinical importance
D : Diff is not statistically sig and not clinical importance

35