Pendalaman Uji Diagnosis Biostat PPDS 2020

Pendalaman
Statistik Deskriptif:
Diagnostik
ARIA KEKALIH
Program Pendidikan Dokter Spesialis
FKUI 2020
Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

Wilma Heemsbergen, Het Nederland Kanker Institute
Marcelo Pagano, Harvard School of Public Health
Anuraj H.Shankar, Harvard School of Public Health
AKNOWLEDGEMENT Prof. Dr. dr. Rianto Setiabudy, SpFK, FKUI
Prof. Dr. dr. Sudigdo Sastroasmoro, Sp.A(K), FKUI
Dr.dr. Joedo Prihartono, MPH, FKUI
Dr.dr. T. Mirawati Soedibjo, SpMK

GARIS BESAR
Konsep Uji Diagnostik: Sensitivitas, Spesifisitas, Nilai Prediksi
Akurasi Kurva ROC
Aplikasi pada penelitian kedokteran

Diagnostic tools
Diagnostic tools are used to: classify / distinguish:
- sick people and healthy people (special case: screening);
- patients with good prognosis and bad prognosis;
- patients with different stages of the disease.
The results of the diagnostic procedures are used to:

- decide whether further diagnosis is needed;
- determine which treatment is optimal;
- estimate the further course of the disease.
-…
The accuracy of a test to discriminate diseased cases from normal cases
is essential, and the subject of many studies.
Gold standard
• the test that most accurately determines the presence of a disease.
• It may not actually be used in a clinical setting, because of expense, time, degree if
invasiveness or expertise, but it is used to determine the sensitivity and specificity
of other tests.
• it is important to realize that most times, not even the gold standard is perfect.
In general, "gold standard" refers to the best available means of doing something.
The "gold standard" refers to the most validated test that makes a diagnosis. The new
test is compared to the most accurate test in use at the time.

Each cut-off is a trade-off between the %
false negative and % false positive cases.
Statistics
Sensitivity (%): probability that a test result will be positive when the disease is present
(true positive rate) = a / (a+b)
Specificity (%): probability that a test result will be negative when the disease is not
present (true negative rate) = d / (c+d)
Positive predictive value (%): probability that the disease is present when the test is
positive = a / (a+c)
Negative predictive value (%): probability that the disease is not present when the test is
negative. = d / (b+d)
Example
Exercise Stress Test (EST) for the diagnosis of Coronary Artery Disease (CAD)
CAD
y n
EST + 815 55.6% 115 7.8% 930 63.4%
- 208 14.2% 327 22.3% 535 36.5%
1023 69.8% 442 30.1% 1465/100%
TPF = 815/1023=0.797
Sensitivity = 79.7%
TNF = 327/442=0.740
Specificity = 74.0%
PPV = 815/930=88%
NPV = 327/535=61%
Purpose of diagnostic tests :
1. Diagnosis or exclusion of disease

- The tests must be : sensitive and specific
2. Screening of disease among asymptomatic persons.
Additional test will be needed to confirm early diagnosis.
The test are useful when:
- Prevalence of disease is “high”
- Significant morbidity/mortality without treatment
- Effective therapy is available
- Early treatment gives better outcome.
Pendidikan
Dr. Aria Kekalih – Magister PendidikanSPKP FKUI FKUI
Kedokteran
Purpose of diagnostic test (contd.)
3. Part of the treament

- To monitor disease/ treatment progress
- To identify complication
` - To monitor drug level
- Determine prognosis
- To confirm indeterminate tests.
- Characterization of causative microorganism
(e.g. drug resistance, genotype)
Important : reproducibility
4. Epidemiology study
Pendidikan
Kedokteran
Structure
They have :
- Predictor variable (the test results)
- Outcome variable (presence or absence of disease)
Pendidikan
Kedokteran
The test results as the predictor variable
- dichotomous (positive or negative)
- categorical (++++, +++, ++, + or -)
- continous ( …. Mg of glucose/ ml) --- cut off point
The disease as outcome variable

- Absence or presence of disease
- Determined by gold standard
Pendidikan
Kedokteran
2x2 table
Disease status
Test result Breast cancer benign nodule
Positive 65 30
Negative 35 70
Analysis with x2 p< 0.001 Statistically the positive

result is highly correlated with presence of disease.
But this test cannot well differentiate absence or presence of
disease.
Pendidikan
Kedokteran
Sensitivity
= the proportion of subjects with the disease who have
positive test
Indicating how good a test is at identifying the diseased
Specificity
= the proportion of subjects without the disease who
have a negative test
Indicating how good a test is at indicating the non-
diseased
Pendidikan
Kedokteran
- The probability that a person with a positive result
actually has the disease.
------ Positive predictive value (PV+)
- The probability that a person with a negative result

actually doesn’t have the disease
------ Negative predictive value (PV-)
Pendidikan
Kedokteran
Disease status
Test result Breast cancer benign nodule
Positif 65 30
Negative 35 70
True positive : the test is positive & he has the disease

False positive : the test is positive & he doesn’t have the disease
True negative : The test is negative & doesn’t have the disease
False negative : The test is negative & he has the disease
Pendidikan
Kedokteran
DISEASE
YES NO TOTAL
TEST YES TRUE FALSE TP + FP

RESULT POSITIVE POSITIVE
NO FALSE TRUE FN + TN
NEGATIVE NEGATIVE
TOTAL TP + FN FP + TN TP + FP + FN +
TN
Pendidikan
Kedokteran
GOLD STANDARD
POSITIVE NEGATIVE TOTAL
TEST POSITIVE A B A+B

RESULT (45) (10) (55)
NEGATIV C D C+D
E (5) (40) (45)
TOTAL A+C B+D A+B+
(50) (50) C+D
Sensitivity = A : ( A+C) = 90%

Specivicity = D : (B + D) = 80%
Positive predictive value = A : (A + B) = 82%
Negative predictive value = D : (C + D) = 89%
Pendidikan
Kedokteran
Disease status
Test result Breast cancer benign nodule Total
Positif 65 30 95
Negative 35 70 105
Total 100 100 200
Sensitivity = ?
Specivicity = ?
PV+ = ?
PV- = ?
Pendidikan
Kedokteran
Disease status
Test result Breast cancer benign nodule Total
Positif 65 30 95
Negative 35 70 105
Total 100 100 200

PV+ = A : (A + B) = 68.4%
PV- = D : (C + D) = 66.7%
Pendidikan
Kedokteran
Steps in diagnostic test research
1. Identify why a new diagnostic test is necessary
- How good is the present available diagnostic test? Any
weakness?
Can a new method overcome the weakness of the old one?
2. Determine the main purpose of the new test.

- To screen? --- high sensitivity
- To confirm diagnosis? --- high sensitivity and specificity
- To exclude? ---- high specificity
3. Select subject population.

- Screening / Case finding/ Diagnosis
- Location
- Sample number
- Inclusion criteria
Pendidikan SPKP FKUI
4. Select gold standard
- Best available diagnostic test
5. Do the test
- Blinded
6. Data analysis and report
- Sensitivity, Specificity, PV+, PV-.
With confidence interval
- ROC for continous data
Pendidikan
Kedokteran
GOLD STANDARD
- Standard method to determine presence or absence
of disease
- Ideally : always positive for diseased person, and
always negative for non-diseased person ---- rare, if
any ----- use the best method available
- One or combination of methods
- Its sensitivity and specificity should not lower than
the new method.
Pendidikan
Kedokteran
Cut Off
- When data are in ordinal (categorical) or numeral
(continous) We have to decide the point that
differentiate “normal” and “abnormal”.
- Depends on the purpose of the test, need high
sensitivity or high specificity.
- E.g : for screening : high sensitivity. To decide whether
a patient need a high-risk surgery : high specificity.
Pendidikan
Kedokteran
Receiver operator curve
A graph that show the “bargain” between sensitivity and specificity when
we determine a cut-off point.
Increase sensitivity - decrease specificity, vice versa.
Points closer to diagonal line – worse result
Selection of cut-off point depends on the purpose of the test.
x
1.0
x D
C
S
e 0.8
n
s x
B
i 0.6
t
i
v x
A
0.4
i
t
x
y 0.2
0 0.2 0.4 0.6 0.8– MagisterPendidikan

Dr. Aria Kekalih 1.0 SPKP
Pendidikan FKUI FKUI
Kedokteran
1 - Specificity
Receiver operator curve (ROC) for serum alanine aminotransferase (ALT)
Level (U/L) in the diagnosis of hepatitis
1.0
25
S 50
e 0.8
n
s
i 0.6 100
t
i x
v 0.4
200
i x
t
y 0.2 x 400
0 0.2 0.4 0.6 0.8 1.0

1 - Specificity Pendidikan
Kedokteran
The value of diagnostic test also depends on prevalence of
the disease in the population being tested.
Prevalence decrease
less likely that someone with a positive test is
actually has the disease
the more specific a test must be in order to be
clinically useful.
Pendidikan
Kedokteran
GOLD STANDARD
TEST POSITIVE A B A+B

RESULT (45) (10) (55)
NEGATIVE C D C+D
(5) (40) (45)
TOTAL A+C B+D 100

(50) (50)
Prevalence = 50%
PV+ = A : (A + B) = 82%
PV- = D : (C + D) = 89%
Pendidikan
Kedokteran
GOLD STANDARD
TEST POSITIVE 18 16 34
RESULT
NEGATIVE 2 64 66
TOTAL 20 80 100
Prevalence = 20%
Specivisity = D : (B + D) = 80%
Nilai duga positif = A : (A + B) = 55%
Nilai duga negatif = D : (C + D) = 97%
Pendidikan
Kedokteran
Limitations
1. Random error
- by chance
- quantifiable ---- confidence interval.
2. Systematic error
2.1. Sampling bias :
- When the study sample is not representative of the
target population in which test will be used.
2.2. Measurement bias :
- Increase when the person determine the test result
have already known the outcome of gold standard.
- borderline result --- determine in advance how to
treat this result.
2.3. Reporting bias
- Unpromising results usually go unreported.
------ enough samples, so negative results can be
meaningful and reported. Pendidikan
Kedokteran
ROC Curve
A Receiver Operating Characteristic Curve describes the whole set of
potential combinations that are possible (False Positive Fraction – True
Positive Fraction).
Sensitivity is plotted in function of (100-

Specificity) for different cut-off points. Each
point represents a sens/spec pair
corresponding to a particular cut-off.
A test with perfect discrimination (no overlap)
has a ROC plot that passes through the upper
left corner (100% sensitivity, 100% specificity).
Therefore the closer the ROC plot is to the
upper left corner, the higher the overall
accuracy of the test.
AUC (area under the curve) summarizes the
discriminating power (0.5 – 1.00).
AUC 0.5: discriminating power is as good as
throwing a coin (50% chance for correct
answer).
Likelihood ratio
Positive likelihood ratio: ratio between the probability of a positive test result given the
presence of the disease and the probability of a positive test result given the absence of
the disease, i.e.
= True positive rate / False positive rate = Sensitivity / (1-Specificity)
Negative likelihood ratio: ratio between the probability of a negative test result given the
presence of the disease and the probability of a negative test result given the absence of
the disease, i.e.
= False negative rate / True negative rate = (1-Sensitivity) / Specificity

Association vs. Classification
A factor that is significantly associated with a disease (significant HR/OR),
is not by definition a good marker to classify this disease.

(Figure 4)
Evaluation of the diagnostic value of 2

tumor markers: separately, combined.

Example

Questions
Read the (relevant parts of the) paper. Answer the following questions.
What is the optimal cut-off with the lowest probability of false positives.
What is the optimal cut-off with the lowest probability of false negatives
Calculate for both cut-offs in a screening population of 10,000 people with 10 cancer cases:
- how many persons receive a positive test result (indicating cancer), and how many of them will
be definitely diagnosed with cancer after additional conclusive tests.
- what is the number of persons with a false positive result,
- what is the number of persons with a false negative result.
Answer the same questions for a hospital

population of 1,000 people with 500 cases.

Example
The Lancet
2002;
359: 572-77
(360: 169)

Prediction Models
Prediction model: construct a model to predict a certain outcome by
(a number of) predictive variables (e.g. the outcome cancer is
predicted based on a tumor marker level).
There is always a risk of overfitting the data.
We are interested in the model with the greatest predictive validity

(i.e. expected to perform optimally in future samples).
Therefore validation of the model is needed:

1- within the same data set: by resampling (e.g. cross validation).
2- in an external data set.

Prediction Models
Clin Cancer Res 2008;14: 5977. Taylor et al. Validation of

Biomarker based risk prediction models.
Also interesting: Int J Radiat Oncol Biol Phys. 2009 1;74(2):355-62.

Development and external validation of prognostic model for 2-year
survival of non-small-cell lung cancer patients treated with
chemoradiotherapy. Dehing-Oberije et al.

Exercise
• A pharmaceutical lab is trying to develop a rapid assay for detecting HIV
infection. The delay in obtaining results for traditional tests reduces their
effectiveness because many patients don't return to learn the results.
The challenge is to develop a test that provides results in 10 to 15
minutes and is as accurate as the traditional tests. The results of the
assay are eight deepening shades of red, with deeper shades indicating
greater likelihood of infection. The test is fast, but is it accurate?
• To help answer this question, a laboratory trial was conducted on 2000
blood samples, half of which were infected with HIV and half of which
were clean. The results are collected in hivassay.sav. See the
topic Sample Files for more information. Use ROC Curve to determine at
what shade the physician should assume that the patient is HIV-positive.

Coordinates of the Curve
Test Result Variable(s): Assay result
Positive if Greater Than or Equal Toa Sensitivity 1 - Specificity
.00 1.000 1.000
1.50 .997 .058
2.50 .995 .040
3.50 .993 .024
4.50 .988 .017
5.50 .978 .015
6.50 .973 .012
7.50 .965 .003
9.00 .000 .000
The test result variable(s): Assay result has at least one tie between
the positive actual state group and the negative actual state group.
a The smallest cutoff value is the minimum observed test value minus
1, and the largest cutoff value is the maximum observed test value
plus 1. All the other cutoff values are the averages of two consecutive
ordered observed test values.

Summary
1. A diagnostic test study determines the usefulness of a
test in the diagnosis of a disease. Good tests are
those that distinguish the diseased from the non-
diseased, and are safe, quick, simple, painless, reliable,
and inexpensive.
Randomized blinded trial and usual clinical practice as
model is important in diagnostic test study.
2. In diagnostic test study there is a predictor variable
(test result) and an outcome variable (the disease,
determined by gold standard).
The goal is to describe how strong the association is,
in terms of its sensitivity and specificity.
Pendidikan
Kedokteran
3. The investigator should determine the sensitivity,
specificity, predictive value of positive test,
predictive value of negative test.
A cutoff point must be determined for calling a result
“positive”.
4. Studies of diagnostic tests are subject to several
biases; the most important are sampling bias,
measurement bias, and reporting bias.
5. Steps in planning diagnostic test study : a) Identify
why a new diagnostic test is necessary; b) Determine
the main purpose of the new test; c) Select subject
population; d) Select gold standard; e) Apply the test
and the gold standard bilndly; f) Analyse and report
data in terms of sensitivity, specificity and predictive
value.
Pendidikan
Kedokteran
TERIMA KASIH
Pendidikan
Kedokteran

Pendalaman Uji Diagnosis Biostat PPDS 2020

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Pendalaman Uji Diagnosis Biostat PPDS 2020

Uploaded by

Copyright:

Available Formats

Pendalaman

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

The results of the diagnostic procedures are used to:

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

1. Diagnosis or exclusion of disease

3. Part of the treament

The disease as outcome variable

Test result Breast cancer benign nodule

Analysis with x2 p< 0.001 Statistically the positive

- The probability that a person with a negative result

Test result Breast cancer benign nodule

True positive : the test is positive & he has the disease

TEST YES TRUE FALSE TP + FP

TEST POSITIVE A B A+B

Sensitivity = A : ( A+C) = 90%

Test result Breast cancer benign nodule Total

Total 100 100 200

Test result Breast cancer benign nodule Total

Total 100 100 200

Sensitivity = A : ( A+C) = 65%

2. Determine the main purpose of the new test.

3. Select subject population.

0 0.2 0.4 0.6 0.8– MagisterPendidikan

0 0.2 0.4 0.6 0.8 1.0

TEST POSITIVE A B A+B

TOTAL A+C B+D 100

Sensitivity is plotted in function of (100-

= True positive rate / False positive rate = Sensitivity / (1-Specificity)

= False negative rate / True negative rate = (1-Sensitivity) / Specificity

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

Evaluation of the diagnostic value of 2

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

Answer the same questions for a hospital

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

There is always a risk of overfitting the data.

We are interested in the model with the greatest predictive validity

Therefore validation of the model is needed:

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

Clin Cancer Res 2008;14: 5977. Taylor et al. Validation of

Also interesting: Int J Radiat Oncol Biol Phys. 2009 1;74(2):355-62.

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

Dr. Aria Kekalih – Magister Pendidikan Kedokteran FKUI

You might also like