CRITICAL APPRAISAL OF

MEDICAL LITERATURES
Developed by: Kuntjoro Harimurti
Presented by: Bagus Artiko
Critical appraisal of study

• Validity
Assessing risk of bias
• Importance
Magnitude of result and
• Applicability its precision

External validity /
generalizibility
Assessing Risk of Bias (Validity)
Risk of Bias

• The degree to which the result is skewed away from the truth
• Causal inferences from studies can, however, be undermined by flaws
in design, conduct, analyses, and reporting 
• Leading to underestimation or overestimation of the true effect 
 PECOT : the 5 parts of every
epidemiological study

Participants P

Exposure Group E C Comparison Group

O Outcomes
Time
T
 British doctors

smoking status measured

smokers non-smokers

yes
Lung cancer
no 5 years

• Longitudinal (cohort) study

 British doctors

Randomised to aspirin or placebo

aspirin placebo

yes
Myocardial infarction
no 5 years

RCT
 Middle-aged US women

Breast cancer no Breast cancer

positive
mammogram
negative

Diagnostic test accuracy study

Assesing Risk of Bias

PECOT RAMBOMAN
• Participant • Recruitmen
• Exposure/Comparison • Allocation
• Outcome • Maintenance
• Time • Blinding or Objective outcome
Measure
• ANalysis
 Bias in Epidemiology Study: GATE Approach
The PECOT RAMBOMAN
P
Population/ Recruitment Recruitment Bias
Participants
Allocation Allocation Bias

Exposure and Maintenance Maintenance Bias

Comparison E C
Blind
Measurement Bias
Outcomes Objective
yes
O Measurements
Time no
T ANalyses Confounding Bias
Critical appraisal of a
diagnostic accuracy study
 Before ordering a test ask:
What will you do if the test is positive?
What will you do if the test is negative?
If the answers are the same, then don’t do the test.

Myriam Hunink and Paul Glasziou

Decision Making in Health and Medicine (2001)
Defining the clinical question: PICO or PIRT
• Patient/Problem
How would I describe a group of patients similar to mine?

• Index test
Which test am I considering?

• Comparator… or …Reference Standard

What is the best reference standard to diagnose the target
condition?

• Outcome….or….Target condition
Which condition do I want to rule in or rule out?
Diagnostic Accuracy Studies
Series of patients

Index test

Reference standard

Compare the results of the index

test with the reference standard,
blinded
 Appraising diagnostic studies: 3 easy steps
•Appropriate spectrum of patients?
•Does everyone get the reference
standard?
Are the results valid?
•Is there an independent, blind or
objective comparison with the reference
standard?

What are the results?

Will they help me look

after my patients?
Biases in Diagnostic Accuracy Studies…
 1. Appropriate spectrum of patients?
Ideally, test should be performed on a group of
patients in whom it will be applied in the real
world clinical setting
Spectrum bias:
study uses only highly selected
patients…….perhaps those in
whom you would really suspect
have the diagnosis
 2. Do all patients have the reference
standard?
Ideally all patients get the reference standard test
Verification bias:
only some patients get the reference
standard…..probably the ones in
whom you really suspect have the
disease
 3. Independent, blind or objective comparison
with the reference standard?

Ideally, the reference standard is independent,

blind and objective
Observer bias:
test is very subjective, or
done by person who
knows something about
the patient or samples
Diagnostic Study Example
 1. Spectrum

2. Index test

3. Reference standard
4. Blinding
 Appraising diagnostic tests
•Appropriate spectrum of patients?
•Does everyone get the reference
standard?
Are the results valid?
•Is there an independent, blind or
objective comparison with the gold
standard?
•Sensitivity, specificity
What are the results?
•Likelihood ratios
•Positive and Negative Predictive Values

Will they help me look

after my patients?
Diagnostic Study Example
Sensitivity and Specificity
 The 2 by 2 table

Disease
+ -
True False
+ positives positives

Test
False True
- negatives negatives
 The 2 by 2 table: Sensitivity
Disease Proportion of people

+ - WITH the disease who

have a positive test result.
90 a
+ True
positives
So, a test with 90%
sensitivity….means that
the test identifies 90 out
Test 10 c of 100 people WITH the
disease

- False
negatives

Sensitivity = a / a + c Sensitivity = 90/100

 The 2 by 2 table: Specificity
Disease
+ - Proportion of people
25 WITHOUT the disease who
b have a negative test result.
+ False
positives So, a test with 75%
Test 75 d
specificity will be
NEGATIVE in 75 out of
True 100 people WITHOUT
- negatives the disease

Specificity = d / b + d Specificity = 75/100

 The Speed bump Example
Disease: Appendicitis

+ - There were 34 people

who had appendicitis…
33
+ 21 54 the speed bump test was
positive in 33 of them

Test: Pain over

speed bump 1 9 10 There were 30 people
- who did not have
appendicitis… the speed
bump test was negative
34 30 64 in 9 of them

Sensitivity = 33/34 = 0.97 (97%) Specificity = 9/30 = 0.30 (30%)

 Ruling In and Ruling Out
High Sensitivity A good test to help in Ruling Out disease
High sensitivity means there are very few false negatives – SnNOUT
so if the test comes back negative it’s highly unlikely the
person has the disease
High Specificity A good test to help in Ruling In disease
High specificity means there are very few false positives – so if the SpPIN
test comes back positive it’s highly likely the person has the disease

Disease: Appendicitis Disease

+ - +
a b
-
33 +
+ 21 True False
positives positives
Test
c d
Test: Pain over
speed bump - False True
1 9 negatives negatives
-
Sensitivity = 97% Specificity = Sensitivity = a/a+c Specificity = d/b+d
30%
Predictive Values
 Positive and Negative Predictive Value
Disease PPV = Proportion of people

+ - with a positive test who

have the disease.
a b
PPV = a / a + b
+ True
positives
False
positives

Test c d
NPV = d / c + d
- False True
negatives negatives NPV = Proportion of
people with a negative test
who do not have the
disease.
 The Speed bump Example
Disease: Appendicitis

+ - PPV = 33/54 = 61%

33
+ 21 54

Test: Pain over NPV = 9/10 = 90%

speedbump 1 9 10
-
34 30 64
 Predictive Value: Natural Frequencies

Your father went to his doctor and was

told that his test for a disease was
positive. He is really worried, and
comes to ask you for help!

After doing some reading, you find that for men of his age:
The prevalence of the disease is 30%
The test has sensitivity of 50% and specificity of 90%

“Tell me what’s the chance I have this disease?”

Prevalence of 30%, Sensitivity of 50%, Specificity of 90%

Sensitivity
Disease +ve = 50% 22 people test

30 15 positive………
of whom 15
have the
disease
100 Testing +ve

Disease -ve
70
False
7 So, chance of
disease is
15/22 = 68%
positive rate
= 10%
Likelihood Ratios
 Likelihood ratios

Probability of clinical finding in patients with disease

LR =
Probability of same finding in patients without disease

Example:
If 80% of people with a cold have a runny nose
and 10% of people without a cold have a runny nose,
then the LR for runny nose is:
80%/10% = 8
 Likelihood ratios

Positive likelihood ratio (LR+)

How much more likely is a positive test to be found in a person
with the disease than in a person without it?
LR+ = sens/(1-spec)

Negative likelihood ratio (LR-)

How much more likely is a negative test to be found in a
person without the disease than in a person with it?
LR- = (1-sens)/(spec)
 What do likelihood ratios mean?

LR=1 LR>10 = strong

LR<0.1 = strong positive test
negative test No diagnostic result
result value
 Likelihood Ratio
LR Interpretation

> 10 Large and often conclusive increase in the likelihood of disease

5 – 10 Moderate increase in the likelihood of disease
2–5 Small increase in the likelihood of disease
1–2 Minimal increase in the likelihood of disease

1 No change in the likelihood of disease

0.5 - 1.0 Minimal decrease in the likelihood of disease

0.2 - 0.5 Small decrease in the likelihood of disease
0.1 - 0.2 Moderate decrease in the likelihood of disease
< 0.1 Large and often conclusive decrease in the likelihood of disease
 %
Fagan nomogram
Bayesian reasoning

Post-test odds for

disease after one
test become pre-
Pre test 5% test odds for next
test etc.
?Appendicitis:
McBurney tenderness LR+ = 3.4 Post test ~20%

Speed bump test LR- = 0.1

Post-test odds =
Pre-test odds x Likelihood ratio
Post test ~0.5%

%
Kelebihan LR

• Sedikit terpengaruh oleh prevalensi

• Dapat segera menghitung post test probability untuk penyakit
 Appraising diagnostic tests
•Appropriate spectrum of patients?
•Does everyone get the gold standard?
Are the results valid? •Is there an independent, blind or
objective comparison with the gold
standard?

•Sensitivity, specificity
What are the results?
•Likelihood ratios
•Positive and Negative Predictive
Values
•Can I do the test in my setting?
Will they help me look •Do results apply to the mix of patients I
after my patients? see?
•Will the result change my management?
•Costs to patient/health service?
 Will the test apply in my setting?

• Reproducibility of the test and interpretation in my setting

• Do results apply to the mix of patients I see?
• Will the results change my management?
• Impact on outcomes that are important to patients?
• Where does the test fit into the diagnostic strategy?
• Costs to patient/health service?
Critical Appraisal of Prognostic Study
Cohort  different lecture
CRITICAL APPRAISAL
OF STUDY ABOUT THERAPY
RCT  different lecture
Critical Appraisal of
Etiologic/Harm Study
Relevant Terms for Etiologic/Harm Study

•Association
•Causation
Association
 Two variables appear to be related by a mathematical relationship.
 A change of one appears to be related to the change in the other.
 Necessary for a causal relationship to exist, but association alone
does not prove that a causal relationship exists.
Causation
 Combination of necessary and sufficient factors, the presence of
which, alone or in combination, at some time inevitably result in an
incidence of interest.
 A necessary factor/cause is a risk factor that must be, or have been,
present for a specified outcome to occur.
 A sufficient factor/cause is the minimal or combination of risk factors
that inevitably results in a specified outcome
 Bradford Hill A. The environment and disease: association or causation?

Proc R Soc Med 1965;58:295-300.

1. Is there evidence from true experiments in humans?

2. Is the association strong?
3. Is the association consistent from study to study?
4. Is the temporal relation appropriate (did the postulated
cause precede the postulated effect)?
 Bradford Hill A. The environment and disease: association or causation?

Proc R Soc Med 1965;58:295-300.

5. Is there a dose-response gradient (does more of the

postulated effect follow more of the postulated cause)?
6. Does the association make epidemiological sense?
7. Does the association make biological sense?
8. Is the association analogous to a previously proved causal
association?
 Evans AS. Causation and disease: the Henle- Koch postulates revisited".

Yale J Biol Med 1976; 49: 175–95.

a. Prevalence of the disease should be significantly higher in those

exposed to the risk factor than those not.
b. Exposure to the risk factor should be more frequent among those with
the disease than those without.
c. In prospective studies, the incidence of the disease should be higher in
those exposed to the risk factor than those not.
d. The disease should follow exposure to the risk factor with a normal or
log-normal distribution of incubation periods.
e. A spectrum of host responses along a logical biological gradient from
mild to severe should follow exposure to the risk factor.
 Evans AS. Causation and disease: the Henle- Koch postulates revisited".

Yale J Biol Med 1976; 49: 175–95.

f. A measurable host response should follow exposure to the risk

factor in those lacking this response before exposure or should
increase in those with this response before exposure. This response
should be infrequent in those not exposed to the risk factor.
g.In experiments, the disease should occur more frequently in those
exposed to the risk factor than in controls not exposed.
h.Reduction or elimination of the risk factor should reduce the risk
of the disease.
i. Modifying or preventing the host response should decrease or
eliminate the disease.
j. All findings should make biological and epidemiological sense.
Etiologic/Harm/Causation Studies

Selection of patients

Define exposure status

Follow-up

Determination of outcomes
Etiology - Harm - Causation

 Evidence levels:
Randomised clinical trial > cohort/ clinical trial >
case-control > cross-sectional > single case

15
Etiologic research: What study design?

Design of two observational studies to distinguish

between cause and effect:
1. Cohort study
2. Case-control study
Cohort Study

• Follow-up studies  subjects selected on presence or

absence of exposure & absence of disease at one
point in time. Disease is then assessed for all subjects
at another point in time.
• Typically prospective but can be retrospective,
depending on temporal relationship between study
initiation & occurrence of disease.
• Individuals selected by exposure status and future
occurrence of disease measured
 Cohort Study
disease +
determinant +
cohort disease -
without
disease
outcome disease +
determinant -
disease -

time
start study disease-outcome

Example of a research question:

Is hypertension a risk factor for myocardial infarction?
Prospective Cohort

START HERE Outcome

Exposure
+
Free of
**
outcome
-
+
*
-
to t1
 START HERE
Historical/RetrospectiveCohort

Exposure Outcome
+
Free of
**
outcome
-
+
*
-
to t1
Case-Control Study
• To establish association between exposure or risk
factors & disease.
• Unlike cohort studies, members of population with
the disease are selected into the study at the outset
and risk factor information is collected
retrospectively (Cases Group). A second group of
individuals who do not have the disease (Controls
Group).
Case-control study

determinant +
disease +
determinant - (patients)

determinant +
disease –
determinant - (controls)

time
start study
Cohort Study

• Advantages
• Cause is measured before effect
• Not very sensitive to selection- and information bias
• Appropriate for rare determinant
• Can study several outcomes
• Disadvantages
• Selective withdrawal / loss to follow-up
• Expensive and time consuming
• Not appropriate for rare outcome
Case-Control study
• Advantages
• Efficient and relatively cheap
• Appropriate for rare outcome
• Can study several determinants
• Disadvantages
• Cause is measured after effect
• Very sensitive to selection- and infobias
• Not appropriate to study several outcomes
Controlling Confounding

• Confounding is a distortion (inaccuracy) in the estimated

measure of association that occurs when the primary exposure
of interest is mixed up with some other factor that is associated
with the outcome.
• Confounding variables are factors affecting both the risk
factor and the outcome. They may affect the exposed and
unexposed groups differently and lead to a bias in the
conclusions.
Controlling Confounding

In the design In the analysis

• Restriction of the study • Restriction of the analysis
• Matching • Stratification
• Multivariable methods
 Appraising etiologic/harm studies:
3 easy steps

How good was the

VALIDITY study?

IMPORTANCE What are the results?

Can I apply the results

APPLICABILITY to my patients

http://guides.mclibrary.duke.edu/c.php?g=158201&p=1036072
VALIDITY

Is this a cohort study or case control study?

COHORT STUDY CASE CONTROL STUDY

Aside from the exposure of interest, did the Did the cases and controls have the same
exposed and control groups start and finish risk of exposure?
with the same risk for the outcome?

http://guides.mclibrary.duke.edu/c.php?g=158201&p=1036072
 COHORT Aside from the exposure of interest, did the exposed and control groups start
STUDY and finish with the same risk for the outcome?

1. Were patients similar for prognostic factors that are known to be

associated with the outcome (or did statistical adjustment level the
playing field)?

The two groups, those exposed to the harm and those not exposed, must
begin with the same prognosis. The characteristics of the exposed and non-
exposed patients need to be carefully documented and their similarity (except
for the exposure) needs to be demonstrated.
The choice of comparison groups has a significant influence on the credibility
of the study results. The researchers should identify an appropriate control
population before making a strong inference about a harmful agent. The two
groups should have the same baseline characteristics.  If there are differences
investigators should use statistical techniques to adjust or correct for
differences. http://guides.mclibrary.duke.edu/c.php?g=158201&p=1036072
COHORT Aside from the exposure of interest, did the exposed and control groups start
STUDY and finish with the same risk for the outcome?

2. Were the circumstances and methods for detecting the outcome

similar? 
In cohort studies determination of the outcome is critical.  It is
important to define the outcome and use objective measures to avoid
possible bias.  Detection bias may be an issue for these studies, as
unblinded researchers may look deeper to detect disease or an
outcome.

3. Was follow-up sufficiently complete? 

Patients unavailable for complete follow-up may compromise the
validity of the research because often these patients have very different
http://guides.mclibrary.duke.edu/c.php?g=158201&p=1036072

outcomes than those that stayed with the study.

Did the cases and controls have the same risk of being exposed in the CASE CONTROL
past?
STUDY

1. Were cases and controls similar with respect to the indication or

circumstances that would lead to exposure?
The characteristics of the cases and controls need to be carefully documented
and their similarity needs to be demonstrated. The choice of comparison groups
has a significant influence on the credibility of the study results. The researchers
should identify an appropriate control population that would be eligible or likely
to have the same exposure as the cases.

2. Were the circumstances and methods for determining exposure similar for
cases and controls? 
In a case control study determination of the exposure is critical.  The exposure in
the two groups should be identified by the same method. The identification
should avoid any kind of bias, such as recall bias. Sometimes using objective
data, such as medical records, or blinding the interviewer can help eliminate
bias. http://guides.mclibrary.duke.edu/c.php?g=158201&p=1036072
 IMPORTANCE
1. How strong is the association between exposure and outcome?
* What is the risk ratio or odds ratio?
* Is there a dose-response relationship between exposure and outcome?

COHORT STUDY CASE CONTROL STUDY

Outcome YES Outcome NO Outcome YES Outcome NO

Exposure YES a b Exposure YES a b

Exposure No c d Exposure No c d

Odds Ratio (OR) =  (a / c) / (b / d)

Relative Risk (RR) = a /(a + b) / c/(c + d)
is the risk of the outcome in the exposed group divided
by the risk of the outcome in the unexposed group:
RR = (exposed outcome yes / all exposed) / (not exposed
outcome yes / all not exposed)
is the odds of previous exposure in a case divided by
the odds of exposure in a control patient:
OR = (exposed - outcome yes / not exposed -
outcome yes) / (exposed - outcome no / not exposed
- outcome no)

http://guides.mclibrary.duke.edu/c.php?g=158201&p=1036072
IMPORTANCE
2. How precise was the estimate of the risk?
* What is the confidence interval for the relative risk or odds ratio?

Confidence Intervals are a measure of the precision of the results of a study.

For example, “36 [95% CI 27-51]“, a 95%CI range means that if you were to
repeat the same clinical trial a hundred times you can be sure that 95% of the
time the results would fall within the calculated range of 27-51. Wider intervals
indicate lower precision; narrow intervals show greater precision.

Confounding Variable is one whose influence distorts the true relationship

between a potential risk factor and the clinical outcome of interest.

http://guides.mclibrary.duke.edu/c.php?g=158201&p=1036072
APPLICABILITY
1. Were the study subjects similar to your patients or population?
Is your patient so different from those included in the study that the results may not
apply?

2. Was the follow-up sufficiently long?

Were study participants followed-up long enough for important harmful effects to be
detected?

3. Is the exposure similar to what might occur in your patient?

Are there important differences in exposures (dose, duration, etc) for your patients?

4. What is the magnitude of the risk?

What level of baseline risk for the harm is amplified by the exposure studied?

5. Are there any benefits known to be associated with the exposure?

What is the balance between benefits and harms for patients like yours?

http://guides.mclibrary.duke.edu/c.php?g=158201&p=103607
TERIMA KASIH