Evaluating

Diagnostic Tests
Payam Kabiri, MD. PhD.
Clinical Epidemiologist
Tehran University of Medical Sciences
Seven question to evaluate the
utility of a diagnostic test
 Can the test be reliably performed?
 Was the test evaluated on an appropriate
population?
 Was an appropriate gold standard used?
 Was an appropriate cut-off value chosen
to optimize sensitivity and specificity?
Seven question to evaluate the
utility of a diagnostic test
 What are the positive and negative
likelihood ratios?
 How well does the test perform in specific
populations?
 What is the balance between cost of the
disease and cost of the test?
Which one of these test is the best
for SLE Dx?
Test Sensitivity% Specificity%
ANA 99 80
dsDNA 70 95
ssDNA 80 50
Histone 30-80 50
Nucleoprotein 58 50
Sm 25 99
RNP 50 87-94
PCNA 5 95
Diagnostic Tests Characteristics
 Sensitivity
 Specificity
 PredictiveValue
 Likelihood Ratio

5
 Validity of Screening Tests
True Disease Status
Results of + -
Screening Test
+ a b

- c d

Sensitivity: The probability of testing

positive if the disease is truly present

Sensitivity = a / (a + c)
6
 Validity of Screening Tests
True Disease Status
Results of + -
Screening Test
+ a b

- c d

Specificity: The probability of screening

negative if the disease is truly absent

Specificity = d / (b + d)
7
 Two-by-two tables can also be used for
calculating the false positive and false
negative rates.
 The false positive rate = false positives /
(false positives + true negatives). It is also
equal to 1- specificity.
 The false negative rate = false negatives /
(false negatives + true positives). It is also
equal to 1 – sensitivity.
 An ideal test maximizes both sensitivity
and specificity, thereby minimizing the
false positive and false negative rates.
 Validity of Screening Tests
Breast Cancer
Physical Exam + -
and Mammo-
graphy + 132 983

- 45 63650

Sensitivity: a / (a + c)
Sensitivity =
Specificity: d / (b + d)
Specificity =
10
 Validity of Screening Tests
Breast Cancer
Physical Exam + -
and Mammo-
graphy + 132 983

- 45 63650

Sensitivity: a / (a + c)
Sensitivity = 132 / (132 + 45) = 74.6%

Specificity: d / (b + d)
Specificity = 63650 / (983 + 63650) = 98.5%
11
2 X 2 table
Disease
+ -

Positive
+
predictive
Test value

Sensitivity
Natural Frequencies Tree
Population
100
In Every 100 People, 4 Will Have The Disease

Population
100

Disease + Disease -
4 96

If these 100 people are representative of the population at

risk, the assessed rate of those with the disease (4%)
represents the PREVALENCE of the disease – it can also be
considered the PRE-TEST PROBABILITY of having the disease
OF THE 4 PEOPLE WITH THE DISEASE, THE TEST
WILL DETECT 3

Population
100

Disease + Disease -
4 96

Test + Test - In other words, the

sensitivity is 75%
3 1
AMONG THE 96 PEOPLE WITHOUT THE DISEASE, 7
WILL TEST POSITIVE

Population In other words, the

specificity is 93%
100

Disease + Disease -
4 96

Test + Test - Test + Test -

3 1 7 89
AMONG THOSE WHO TEST POSITIVE, 3 IN 10 WILL
ACTUALLY HAVE THE DISEASE

Population This is also the

POST-TEST PROB-
100 ABILITY of having
the disease

Disease + Disease -
4 96

Test + Test + POSITIVE

PREDICTIVE
3 7 VALUE = 30%

Test - Test -
1 89
AMONG THOSE WHO TEST NEGATIVE, 89 OF 90 WILL
NOT HAVE THE DISEASE

Population
100

Disease + Disease -
4 96

Test + Test +
3 7

NEGATIVE Test - Test -

PREDICTIVE
VALUE = 99% 1 89
CONVERSELY, IF SOMEONE TESTS NEGATIVE, THE
CHANCE OF HAVING THE DISEASE IS ONLY 1 IN 90

Population
100

Disease + Disease -
4 96

Test + Test +
3 7

Test - Test -
1 89
PREDICTIVE VALUES AND CHANGING PREVALENCE

Population
1000

Disease + Disease -
4 996

Prevalence reduced by an order

of magnitude from 4% to 0.4%
PREDICTIVE VALUE AND CHANGING PREVALENCE

Sensitivity and Population

Specificity 1000
unchanged

Disease + Disease -
4 996

Test + Test +
3 70

Test - Test -
1 926
POSITIVE PREDICTIVE VALUE AT LOW PREVALENCE

Population
Previously, PPV
1000
was 30%

Disease + Disease -
4 996

Test + Test + POSITIVE

PREDICTIVE
3 70 VALUE = 4%

Test - Test -
1 926
NEGATIVE PREDICTIVE VALUE AT LOW PREVALENCE

Population
Previously, NPV
1000
was 99%

Disease + Disease -
4 996

Test + Test +
3 70

NEGATIVE Test - Test -

PREDICTIVE
VALUE >99% 1 926
Prediction Of Low Prevalence
Events
 Even highly specific tests, when applied to
low prevalence events, yield a high number
of false positive results
 Because of this, under such circumstances,
the Positive Predictive Value of a test is low
 However, this has much less influence on
the Negative Predictive Value
 Relationship Between
Prevalence and Predictive Value
1
Difference between
PPV and NPV
Predictive Value

0.8
relatively small

0.6

0.4
Difference between
0.2 PPV PPV and NPV
relatively large
NPV
0
0.05 0.2 0.4 0.6 0.8 0.95
Pre-test Probability (Prevalence)

Based on a test with 90% sensitivity and 82% specificity

Relationship Between
Prevalence And Predictive Value
Predictive Value

Prevalence

Based on a test with 75% sensitivity and 93% specificity

 Performance of A Test With
Changing Prevalence
A : Sensitivity =
Specificity = 0.9
POST-TEST PROBABILITY

LR+ = 9.0

B : Sensitivity =
Specificity = 0.7
LR+ = 3.0

C : Sensitivity =
Specificity = 0.5
LR+ = 1.0
2 X 2 table
 Sensitivity

FALSE
NEGATIVES

Sensitivity
The proportion of people with the diagnosis (N=4) who are
correctly identified (N=3)
Sensitivity = a/(a+c) = 3/4 = 75%
 Specificity
FALSE
POSITIVES

Specificity
The proportion of people without the diagnosis (N=96) who
are correctly identified (N=89)
Specificity = d/(b+d) = 89/96 = 93%
Value of a diagnostic test depends
on the prior probability of disease
 Prevalence  Prevalence
(Probability) = 5% (Probability) = 90%
 Sensitivity = 90%  Sensitivity = 90%
 Specificity = 85%  Specificity = 85%
 PV+ = 24%  PV+ = 98%
 PV- = 99%  PV- = 49%
 Test not as useful
when disease unlikely
 Test not as useful
when disease likely
31
 A Test With Normally
Distributed Values
Assessing the performance of
Test cut-off the test assumes that these
two distributions remain
constant. However, each of
them will vary (particularly
through spectrum or selection
bias)
% of Group

NON-DESEASED

DISEASED

Negative Positive
Degree of ‘positivity’ on test
 Performance of A Diagnostic
Test
NON-CASES CASES
FALSE
NEGATIVES Test cut-off
FALSE
POSITIVES
% of Group

NON-DESEASED

DISEASED

Negative Positive
Degree of ‘positivity’ on test
 Minimising False Negatives: A
Sensitive Test
NON-CASES CASES
Cut-off shifted to minimise
Test cut-off false negatives ie to
optimise sensitivity

CONSEQUENCES:
- Specificity reduced
% of Group

-A Negative result from a

seNsitive test rules out the
NON- diagnosis - snNout
DESEASED
DISEASED

Negative Positive
Degree of ‘positivity’ on test
 Minimising False Positives: A
Specific Test
Cut-off shifted to
minimise false positives
Test cut-off ie to optimise specificity

CONSEQUENCES:
- Sensitivity reduced
% of Group

-A Positive result from a

sPecific test rules in the
diagnosis - spPin
NON-DESEASED

DISEASED

Negative Positive
Degree of ‘positivity’ on test
Receiver Operating Characteristics (ROC)

Non-diseased Diseased

Threshold

Evaluation Result Value

Or
Subjective Judgment Of Likelihood That Case Is Diseased
 Non-diseased Diseased
Centers Centers

Threshold

Test result value

or
subjective judgment of likelihood that case is diseased
more typically:

Non-diseased Diseased
Centers Centers

Cutoff point
more typically:

Non-diseased Diseased
cases cases

FP rate
more typically:

Non-diseased Diseased
Centers Centers

TP rate
Non-diseased
Centers

TPF, sensitivity
Threshold
less aggressive
mindset
Diseased
Centers

FPF, 1-specificity
Non-diseased
cases

TPF, sensitivity
moderate
mindset
Threshold

Diseased
cases

FPF, 1-specificity
Non-diseased
cases

more

TPF, sensitivity
aggressive
mindset

Threshold

Diseased
cases

FPF, 1-specificity
Non-diseased
cases
Entire ROC curve

TPF, sensitivity
Threshold

Diseased
cases

FPF, 1-specificity
 Entire ROC curve
e
il n

TPF, sensitivity
c e
an
ch

FPF, 1-specificity
 Check this out:
http://www.anaesthetist.com/mnm/stats/ro
c/Findex.htm
Likelihood Ratios
Pre-test & post-test probability
 Pre-test probability of disease can be
compared with the estimated later probability
of disease using the information provided by
a diagnostic test.
 The difference between the previous
probability and the later probability is an
effective way to analyze the efficiency of a
diagnostic method.
 It tells you how much a positive or negative
result changes the likelihood that a patient
would have the disease.
 The likelihood ratio incorporates both the
sensitivity and specificity of the test and
provides a direct estimate of how much a test
result will change the odds of having a
disease
 The likelihood ratio for a positive result
(LR+) tells you how much the odds of the
disease increase when a test is positive.

 The likelihood ratio for a negative result

(LR-) tells you how much the odds of the
disease decrease when a test is negative.
Positive & Negative Likelihood
Ratios
 We can judge diagnostic tests: positive
and negative likelihood ratios.

 Like sensitivity and specificity, are

independent of disease prevalence.
 Likelihood Ratios (Odds)

 The probability of a test result in those

with the disease divided by the
probability of the result in those without
the disease.
 How many more times (or less) likely a
test result is to be found in the disease
compared with the non-diseased.

52
Positive Likelihood Ratios
 This ratio divides the probability that a
diseased patient will test positive by the
probability that a healthy patient will test
positive.
 The positive likelihood ratio
+LR = sensitivity/(1 – specificity)
False Positive Rate
 The false positive rate = false positives /
(false positives + true negatives). It is also
equal to 1- specificity.
 The false negative rate = false negatives /
(false negatives + true positives). It is also
equal to 1 – sensitivity.
Positive Likelihood Ratios
 It can also be written as the
true positive rate/false positive rate.
 Thus, the higher the positive likelihood
ratio, the better the test (a perfect test has
a positive likelihood ratio equal to infinity).
Negative Likelihood Ratio
 This ratio divides the probability that a
diseased patient will test negative by the
probability that a healthy patient will test
negative.
 The negative likelihood ratio
–LR = (1 – sensitivity)/specificity.
False Negative Rate
 The false negative rate = false negatives /
(false negatives + true positives).
 It is also equal to 1 – sensitivity.
Negative Likelihood Ratio
 It can also be written as the
false negative rate/true negative rate.
 Therefore, the lower the negative
likelihood ratio, the better the test (a
perfect test has a negative likelihood ratio
of zero).
Positive & Negative Likelihood
Ratios

 Although likelihood ratios are independent

of disease prevalence, their direct validity
is only within the original study population.
Probability of Disease
 Pre-test probability of disease = disease
prevalence
 Post-test probability of disease =
 If normal, c/(c+d)
 If negative, a/(a+b)

Disease absent, Disease present,

gold standard gold standard
Test result positive True positives (a) False positives (b)

Test result negative False negatives (c) True negatives (d)

60
 Bayes Theorem

Post-test Odds =
Likelihood Ratio X Pre-test Odds
Using Likelihood Ratios to Determine Post-
Test Disease Probability

Pre-test Pre-test
probability odds of
of disease disease Post-test Post-test
odds of probability
disease of disease
Likelihood
ratio

62
Pre-test & post-test probability
 “Post-test probability” depends on the
accuracy of the diagnostic test and the
pre-test probability of disease
 A test result cannot be interpreted without
some knowledge of the pre-test probability
Where does “pre-test
probability” come from?

 Clinical experience
 Epidemiological data
 “Clinical decision rules”
 Guess
what is the likelihood that this
patient has the disease?

 A disease with a prevalence of 30% must

be diagnosed.
 There is a test for this disease.
 It has a sensitivity of 50% and a specificity
of 90%.
 Likelihood Ratios
Sensitivity Specificity From: J Clin End & Metab. 2006;
91(11):4295-4301.
FNA Biopsy 88% 82%

Sensitivity
1 – Specificity
= 0.88 / (1 – 0.82)
= 4.89

This means that Anne’s positive FNA biopsy will

be approx. 5 times as likely to be seen with,
as opposed to without, thyroid cancer.
 Prevalence of 30%
Sensitivity of 50%
Specificity of 90%

22 positive
Disease +ve 30 15 15
tests in
total of
which 15
100 have the
disease
Disease -ve
70 63 About 70%

70 – 63 = 7
Likelihood
Population
100

Disease +
4 The likelihood that
someone with the
disease will have a
Test +
positive test is ¾ or
3 75%
This is the same as
Test - the sensitivity
1
Likelihood II
Population
100

Disease -
96
The likelihood that
someone without
the disease will Test +
have a positive test 7
is 7/96 or 7%
This is the same as Test -
the (1-specificity)
89
 Likelihood Ratio
Likelihood of Positive Test Given
The Disease
Likelihood Ratio =
Likelihood of Positive Test
in the Absence of the Disease

Sensitivity 0.75
= = = 10.7
1- Specificity 0.07

A Likelihood Ratio of 1.0 indicates an uninformative test (occurs

when sensitivity and specificity are both 50%)
The higher the Likelihood Ratio, the better the test (other
factors being equal)
 Diagnostic Odds Ratio
Potentially useful as an
overall summary
measure, but only in
conjunction with other
measures (LR,
sensitivity, specificity)

3
The Diagnostic Odds Ratio is DOR  7
1
the ratio of odds of having the 89
diagnosis given a positive test 
0.429
 38.2
to those of having the 0.011

diagnosis given a negative test

Is there an
easier way?
Likelihood Ratio And Pre- And
Post-test Probabilities
For a given test with a
given likelihood ratio, the
post-test probability will
depend on the pre-test
probability (that is, the
prevalence of the condition
in the sample being
assessed)
Sensitivity Analysis of A
Diagnostic Test
Value 95% CI
Pre-test 26% to
35%
probability 44%
Sensitivity Analysis of A
Diagnostic Test
Value 95% CI
Pre-test
35% 26% to 44%
probability
Likelihood
5.0 3.0 to 8.5
ratio

Applying the 95% confidence

intervals above to the
nomogram, the post-test
probability is likely to lie in the
range 55-85%
Applying A Diagnostic Test In
Different Settings
 ThePositive Predictive Value of a test will vary
(according to the prevalence of the condition in the
chosen setting)
 Sensitivity
and Specificity are usually considered
properties of the test rather than the setting, and are
therefore usually considered to remain constant
 However, sensitivity and specificity are likely to be
influenced by complexity of differential diagnoses and a
multitude of other factors (cf spectrum bias)
 Likelihood Ratios (Odds)

 This is an alternative way of describing the

performance of a diagnostic test. Similar to S
and S, and can be used to calculate the
probability of disease after a positive or
negative test (predictive value). Advantage of
this is that it can be used at multiple levels of
test results.

77
What is this second fraction?
 Likelihood Ratio Positive
 Multiplied by any patient’s pretest odds
gives you their posttest odds.
 Comparing LR+ of different tests is
comparing their ability to “rule in” a
diagnosis.
 As specificity increases LR+ increases and
PPV increases (Sp P In)

78
Clinical interpretation of post-
test probability
Probability of disease:
Don't Do further
treat for diagnostic Treat for
disease testing disease

0 1

Testing Treatment
threshold threshold
If you are here, Test
will help you to go
Disease toward one end of
Disease
ruled out this probability,
ruled in
either 0 or 1 to get
79 the final decision.
Values of Positive and Negative
Likelihood Ratios (LR)

 LR Poor-fair Good Excellent

Positive
likelihood 2.1-5 5.1-10 10>
ratio

Negative
likelihood 0.5-0.2 0.19-0.1 0.1<
ratio
 Likelihood Ratios & You
 Allows us to determine the accuracy with which a
test identifies the target disorder
 As the LR becomes larger, the likelihood of the
target disease increases:
Likelihood ratio Interpretation
>10 Strong evidence to rule in disease
5-10 Moderate evidence to rule in disease
2-5 Weak evidence to rule in disease
0.5-2 No significant change in the likelihood of disease
0.2-0.5 Weak evidence to rule out disease
0.1-0.2 Moderate evidence to rule out disease
<0.1 Strong evidence to rule out disease
Advantages of LRs
 The higher or lower the LR, the higher or lower
the post-test disease probability
 Which test will result in the highest post-test
probability in a given patient?
 The test with the largest LR+
 Which test will result in the lowest post-test
probability in a given patient?
 The test with the smallest LR-

82
Advantages of LRs
 Clear separation of test characteristics
from disease probability.

83
Likelihood Ratios - Advantage
 Provide a measure of a test’s ability to rule
in or rule out disease independent of
disease probability
 Test A LR+ > Test B LR+
 Test A PV+ > Test B PV+ always!
 Test A LR- < Test B LR-
 Test A PV- > Test B PV- always!

84
 Predictive Values
Alternate formulations:Bayes’ Theorem

PV+ =
Se  Pre-test Prevalence
Se  Pre-test Prevalence + (1 - Sp)  (1 - Pre-test Prevalence)

High specificity to “rule-in” disease

PV- =
Sp  (1 - Pre-test Prevalence)
Sp  (1 - Pre-test Prevalence) + (1 - Se)  Pre-test Prevalence

High sensitivity to “rule-out” disease

85
 Clinical Interpretation: Predictive Values

PV+ And PV-1 Of Electrocardiographic Status2

For Angiographically Verified3 Coronary Artery
Disease, By Age And Sex Of Patient
Sex Age PV+ (%) PV- (%)

F <40 32 88
F 40-50 46 80
F 50+ 62 68

M <40 62 68
M 40-50 75 54
M 50+ 85 38

1. Based on statistical smoothing of results from 78 patients referred to NC

Memorial Hospital for chest pain. Each value has a standard error of 6-7%.
2. At least one millivolt horizontal st segment depression.
3. At least 50% stenosis in one or more main coronary vessels.

86
If Predictive value is more
useful why not reported?
 Should they report it?
 Only if everyone is tested.
 And even then.
 You need sensitivity and specificity from
literature. Add YOUR OWN pretest
probability.

87
So how do you figure pretest
probability?
 Start with disease prevalence.
 Refine to local population.
 Refine to population you serve.
 Refine according to patient’s presentation.
 Add in results of history and exam (clinical
suspicion).
 Also consider your own threshold for testing.

88
Pretest Probability: Clinical
Significance
 Expected test result means more than
unexpected.
 Same clinical findings have different
meaning in different settings
(e.g.scheduled versus unscheduled visit).
Heart sound, tender area.
 Neurosurgeon.
 Lupus nephritis.

89
What proportion of all patients
will test positive?
 Diseased X sensitivity
+ Healthy X (1-specificity)
 Prevalence X sensitivity +

(1-prevalence)(1-specificity)
 We call this “test prevalence”
 i.e. prevalence according to the test.
Some Examples
Diabetes mellitus (type 2)
 Check out this:
Some Examples from
Essential Evidence Plus
Disease Link Address

Diabetes Mellitus http://www.essentialevidenceplus.com/content/eee/127

(type 2)
Deep Vein http://www.essentialevidenceplus.com/content/eee/28
Thrombosis
Arrhythmia (Atrial http://www.essentialevidenceplus.com/content/eee/13
Fibrillation & Flutter)

http://www.essentialevidenceplus.com/
Which one of these test is the best
for SLE Dx?
Test Sensitivity Specificity LR(+)
ANA 99 80 4.95
dsDNA 70 95 14
ssDNA 80 50 1.6
Histone 30-80 50 1.1
Nucleoprotein 58 50 1.16
Sm 25 99 25
RNP 50 87-94 3.8-8.3
PCNA 5 95 1
Was it clear enough !
Key References

Sedlmeier P and Gigerenzer G. Teaching Bayesian

reasoning in less than two hours. Journal of Experimental
Psychology: General. 130 (3):380-400, 2001.
Knotternus JA (ed). The Evidence Base of Clinical
Diagnosis. London: BMJ Books, 2002.
Sackett DL, Haynes RB, Guyatt G, and Tugwell P. Clinical
Epidemiology : A Basic Science for Clinical Medicine.
Boston, Mass: Little, Brown & Co, 1991.
Loong TW. Understanding sensitivity and specificity with
the right side of the brain. BMJ 2003: 327: 716-19.
 ‫ ! بزنید‬Email ‫اگر میل داشتید‬
payam.kabiri@gmail.com