Assessing the Validity and Reliability of Diagnostic and Screening Tests

Adora F. Mendoza – Abat, M.D., CFP

Screening
• The application of a test to people who are as yet asymptomatic for the purpose of classifying them with respect to their likelihood of having a particular disease • Not diagnostic tests but preliminary tests seeking to identify those who are most likely to have the disease

Criteria of Disease Suitable for Screening Program
• 1. The disease should be serious • 2. Early treatment must be beneficial in terms of reducing mortality and morbidity comparable to late treatment • 3. The prevalence of preclinical disease must be high in the population screened

Biologic Variation of Human Populations (Bimodal Curve)
25 20 15 10 5 0 0 3 6 9 12 15 18 21 24 27

Induration # of Subj.

Biologic Variation of Human Populations (Unimodal Curve)
140 120 100 80 60 40 20 0 <110 120 140 160 >180 S. BP # of Men

Validity of Screening Tests the test to • Validity – the ability of

distinguish between who has a disease and who does not • 2 Components: • 1. Sensitivity – the ability of the test to identify correctly those who have the disease • - The proportion of diseased people who were correctly identified as such by the test

Factors that Increase Sensitivity
• 1. Lowering the cut-off and making criteria for disease definition less stringent • 2. Parallel Testing – the administration of 2 screening tests at the same time and persons with a positive result on any of these tests are considered positive

When Should Sensitivity Be Increased
• 1. When the disease is serious and a definitive treatment exists • 2. When the disease can be spread • 3. When subsequent diagnostic evaluations of positive screening tests are associated with minimal costs and risks

Validity of Screening Tests
• 2. Specificity – the ability of the test to identify correctly those who do not have the disease • - proportion of non-diseased people who are correctly called negative by the test • Note : In order to quantitatively assess the sensitivity and specificity of a test, we must have another source of truth with which to compare the test results

Factors that Increase Specificity
• 1. Making the criteria for disease definition more stringent or increasing the cut-off level • 2. Series Testing – an initial screening test is administered and then, only persons who are positive on this preliminary test will undergo a second additional screening procedure

When Should Specificity Be Increased
• 1. When the costs or risks associated with further diagnostic techniques are substantial • 2. When the stigma associated with having the disease is severe

Test Dse. No Tota Dse. l + 80 20 100 180 800 820

Concept of the Sensitivity and Specificity of Screening Exams
• Sensitivity = 80 x 100 • 100 • = 80%

Tota 100 900 1 l 000

• Specificity = 800 x 100 • 900 • = 89%

Comparison of the Results of a Dichotomous Test with Actual Disease Status
Test Results W/ Dse. W/O Dse.

Sensitivity = TP x 100 TP + FN Specificity = TN x 100 TN + FP

+

-

Have dse. & have (+) test = true (+) or (TP) Have dse. but have (-) test = false (-) or (FN)

No dse. & have (+) test = false negative (FP) No dse. and have (-) test = true (-) (TN)

Issues of False Positives
• All people who screened positive are brought back to a more sophisticated and more expensive tests • The anxiety and worry induced in persons who have been told that they have tested positive

Importance of Problems in False Negatives
• Depends on the nature and severity of the disease being screened for, the effectiveness of available intervention measures, and whether the effectiveness is greater if the intervention is administered early in the natural history of the disease

Tests of Continuous Variables
Diabetics High Non-Diabetics

Blood Sugar

Low

Test

Test

Screening Using a Dichotomous Test
Test Disease Positive (+) a (True positives) Negative b (False Positives) a + b (all people with positive tests) Total

(-)

c (False negatives)

d ( True negatives)

c + d ( all people with negative tests)

Two-Stage Screening
• First Stage – less expensive, less invasive, or less uncomfortable test • - screened positive are recalled for further testing • Second Stage – more expensive, more invasive which may have greater sensitivity and specificity • - reduces the problem of false positives

Two Stage Screening Program • Test 1 (Blood Sugar)
• • • • Assume: Disease Prevalence = 5% Population = 10, 000 Test 1 (Blood Sugar) : Sensitivity = 70% Specificity = 80% Disease (+) 350 150 500 Total (-) 1,900 7,600 9,500 2,250 7,750 10,000

Test Results (+) (-) Total

Two Stage Screening
• Test 2 (OGTT) Test Dse. -Dse Total • Given: + 350 1 2250 Sensitivity = 90% 150 900 7750 7 Specificity = 90% Tota 500 600 10000 • Solution : 9 l 500 Net Sensitivity Test Dse. -Dse Tota = (315/500)(100)= 63% + 315 190 l 505 Net Specificity 35 1 1 = [ (7600+1710) / 9500] [100] Tota 350 710 745 1 2 l 900 250 = 98%

Predictive Value of a Test
• The proportion of people correctly identified by the test to have a disease • Positive Predictive Value – the proportion of patients who test positive and who have actually the disease in question • - divide the number of true positives by the total number who tested positive (true positives + False

• Negative Predictive Value – the probability that those who tested negative do not have the disease • - divide the number of true negatives by all those who tested negative (true negatives + false negatives)

Predictive Value of a Test
Positive Total
Test Results Disease 80 20 100 No Dse. 100 800 900 Total 180 820 1,000

Negative

• Positive Predictive Value = (80/180) x 100 = 44% • Negative Predictive Value = (800/820) x 100= 98%

Ex.; Sensitivity = 99% Specificity = 95% Dse. Test Sick Not Totals Predictive Prevalence Results 99 Sick 1% + 495 594 Value 1 9405 9406 5% Totals 100 9900 10000 + 495 475 970 5 9025 9303 Totals 500 9500 10000

Relation of Disease Prevalence to Predictive Value

Ex.; Sensitivity = 99% Specificity = 95% Dse. Test Sick Not Totals Predictive Prevalence Results Sick Value 1% + 99 495 594 (99/594) x 100 = 17% 1 9405 9406 (9405/9406) x + 100=100% 475 5% 495 970 (495/970) x 100 = 51%Totals 100 9900 10000 5 9025 9303 (9025/9303) x 100 = 97%

Relation of Disease Prevalence to Predictive Value

Test
-

Relation of Predictive Value to the Specificity of the Test
Tota + 250 250 l 500 250 250 500 + • • • • • • • • Prevalence = 50% Sensitivity = 50% Specificity = 50% Predictive Value? Prevalence = 20% Sensitivity = 50% Specificity = 50% Predictive Value?

Tota 500 500 1 l 000 Test + Tota + 100 400 l 500 100 400 500 Tota 200 800 1 l 000

Relation of Predictive Value to the Specificity of the Test Test + Tota
l
+ 180

400 580 400 420 80 Tota l 180

Test + -

20 + 100

• • • • • • • •

Prevalence = 20% Sensitivity = 90% Specificity = 50% Predictive Value? Prevalence = 20% Sensitivity = 50% Specificity = 90% Predictive Value?

720 870 100

Relation of Predictive Value to the Specificity of the Test

Interpretation
• Sensitivity = 90% -90% of those with the disease will be labeled as positive by the test • Specificity = 90% - 90% of those without the disease will be labeled as negative by the test Positive Predictive Value = 75% 75% of those who test positive will really have the disease Negative Predictive Value = 75% 75% of those who test negative will really

Reliability (Repeatability of Tests)
• Regardless of the specificity and sensitivity, if the test results cannot be reproduced, the value and usefulness of the test are minimal.

Factors that Contribute to Variation in Test Results
• 1. Intrasubject Variation – variation between individual subjects • - In evaluating any test result, it is important to take into consideration the conditions under which the test was carried out, including the time of day. • 2. Interobserver Variation – variation between those reading the test results

Relation Between Validity and Reliability

Exercise 1
• Calculate for the sensitivity, specificity, positive predictive value, and negative predictive value • Interpret your answers
THROAT CULTURE

Test Results +
-

Total

With Disease 27 10 37

W/O Dse. 35 77 112

Total 62 87 149

Exercise 2
• Complete the table with needed data • Solve for the positive predictive value and interpret your answer • State the relationship of predictive value and specificity using the positive predictive value • Solve for the negative predictive value and interpret your answer

Relationship of Specificity to Predictive Value
Ex. Prevalence = 10% ; Sensitivity = 100% Specificity Test Sick Not Totals Predictive 70% + Results Sick Value Totals 10000 95% + Totals 10000

Sign up to vote on this title
UsefulNot useful

Master Your Semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master Your Semester with a Special Offer from Scribd & The New York Times

Cancel anytime.