otherwise it can result in seriously mistaken
Base Rate Fallacy and
COVID-19 Testing –
Considerations for Policy
Mainak Seni
May 2020
A classic application of one of the great
theorems of probabilityii is the so called base-
rate fallacy – a result that common people find
counterintuitiveiii, but which is well known to
most practitioners and students of statistics.
When cast in the context of epidemiology it
takes the form:
For a rare disease, many (often, the majority) of
the positive results of diagnostic tests, even
accurateiv ones, are actually false positives.
For example, if a disease is actually present in
only 1 out of every 1000 people in the
populationv, then even with a “99.9% accurate
test”, half of the people whose test results say
that they have the disease actually do not. More
surprisingly, if the test is even slightly less
accurate, 99%, then the proportion of false
positives out of all test-positives (the False
Discovery Rate, FDRvi) is more than 90%; and
with 98% accuracy, it is over 95%! Fig 1 shows
the nature of the relationshipvii between the FDR
and the prevalence rate when the prevalence is
low.
The only “escape” from this fallacy is a perfect
test (“100% accurate”)viii that, like all perfect
things, is hard to come by.
Somewhat assuringly (perhaps) the results are
closer to common intuition if the disease is
more prevalent. For example, if 1 in 5 people in
the population have the disease, then the
proportion of false positives even with only the
“95% accurate” test is a little more than one-
sixth. And, it reduces to the intuitive 1 in 20
when the prevalence is 1 out of 2.
For an infectious, and potentially rapidly
growing, pandemic like COVID-19, policy
makers need to be aware of this fallacy as
page 1 of 3
decisions. Likewise, it is also important to try to
communicate this, curious yet possibly
consequential, piece of mathematical reasoning
to the widest section of citizens.
As the pandemic spreads the prevalence rate
starts at very low values and then increases. A
successful control strategy will mean that the
prevalence remains low (less than 1% of the
population). But, this will also mean that the
majority of people identified as positive by any
less-than-perfectix test will be actually free of
the virus. One needs to very carefully evaluate
the health and economic costs, both to the
individual concerned and to the society as a
whole, of isolation / quarantine of merely “test-
positive” individuals, in the absence of either
clinical symptoms of disease or strong index of
suspicion.
Moreover, if most “test-positive” individuals are
asymptomatic, to infer from there that most
carriers of the virus are asymptomatic will be
fraught with risk of error; it is most likely they
are not infected at all (i.e., are false positives)
and, hence, asymptomatic.
Decision making is further complicated, thus
requiring even more careful consideration, by
the facts that a) the exact prevalence rate is
unknown, b) the prevalence rate may be widely
different across sub-populationsx, and c) it may
increase rapidly with time, particularly if control
strategies are ineffectivexi, causing markedly
decreased FDR.
Fig 1: FDR vs Prevalence for “rare” event
i Director and Head of Analytics, Business Brio, Kolkata

ii Bayes’ Theorem. Although the formal probability statement is a textbook application of this famous theorem, the
informal sample proportion statement (like how it is stated here) can easily be derived from first principles by
elementary arithmetic; see vii below

iii This is a case of a type of bias in judgment that, some cognitive psychologists have argued, arises out of
insensitivity to prior probability of outcomes. Refer to page 1 of:
Judgment under Uncertainty: Heuristics and Biases
Author(s): Amos Tversky and Daniel Kahneman
Source: Science, New Series, Vol. 185, No. 4157, (Sep. 27, 1974), pp. 1124-1131
https://www.jstor.org/stable/1738360?origin=JSTOR-pdf

iv Accuracy may be misleading and so usually specificity and sensitivity are considered separately; here “high
accuracy” is meant as simultaneously high specificity and sensitivity. In this paper, for simplicity “99% accuracy”
denotes specificity = sensitivity = 99%, etc., though in reality specificity and sensitivity of a test are usually not the
same. For low prevalence, however, it is the specificity that is more influential on FDR, though

v To put this in the context of COVID-19, as of 21 May 2020, Qatar has among the highest case prevalence as a % of
the whole population at 1.34%; on that same date the rates for the US and India are 0.49% and 0.0086%
respectively [Source: https://www.worldometers.info/coronavirus/]

vi False Discovery Rate = Prob(Actual Infected = 0 | Test Result = 1)

Actual Infected status must be judged independently by some “Gold Standard” criteria (may be, another more
sensitive test)
from ibid v: Rounding up India’s prevalence rate to 0.01%, and with a test accuracy of 99.9%, the FDR is more
than 90%
Table: Computed FDR for Some Countries

Test “Accuracy”
Prevalence 99.90% 99.00% 98.00%
Qatar 1.34% 6.90% 42.70% 60.04%
US 0.49% 16.90% 67.20% 80.6
India 0.0086% 92.10% 99.10% 99.60%

It should be noted that these quoted “case prevalence rates” are simply computed by dividing the number of
positive cases (usually from tests) with the country’s population and, the testing strategy and coverage vary among
countries and with time

vii
( 1−specificty )×( 1− prevalence )
FDR=
sensitivity× prevalence+ ( 1−specificty )×( 1− prevalence )

1
=
sensitivity prevalence
1+ ×
1− prevalence 1−specificity

1
=
sensitivity 1
1+ ×
1−specificity 1
−1
prevalence
The total cases that are identified as positive (the denominator in the first expression above) is the sum of two
components: a) those that are actually positive and correctly identified as such, and b) those that are actually
negative but erroneously identified as positive.
Low prevalence means that there are many more negative cases than positive ones in the population. Specificity is
the measure of how well the test can correctly identify the real negative cases. A test with slightly imperfect
specificity will have a small false positive rate (as a % of real negatives) but since the number of real negatives are
very high, this will still translate to a significant number of false positives.
Sensitivity, on the other hand, measures how well the test can correctly identify the real positive cases. Now,
because of low prevalence, the number of real positive cases will be small to begin with. This not only makes
 component b) dominant over component a) and hence results in high FDR (by definition) but also makes the FDR
sensitive to small changes in the specificity, as the high number of real negatives accentuates the small false
positive rate. The fallacy becomes pronounced whenever the false positive rate is larger than the (low) prevalence
rate. Also, ibid iv, vi

viii Strictly speaking a test with perfect (= 100%) specificity irrespective of (even poor, but non-zero) sensitivity. It is,
however, not easy to empirically establish perfect specificity (to say nothing of the technical difficulties of
developing such a test in the first place): Even if 100 known disease free individuals are all correctly identified by
the test as negative then also the 95% confidence interval for the specificity is (96.4%, 100%)
For example of a real time RT-PCR for SARS-CoV-2 test specificity see
https://www.fda.gov/media/136231/download

ix Less-than-perfect in the sense of specificity < 100%

x It is crucial to consider the prevalence rate in the sub-population that is being tested; If only symptomatic and/or
high-exposure-risk (e.g., medical providers or close contacts of otherwise confirmed cases) people are being tested
then the prevalence rate among them might be a lot higher than what would be the case if the general population are
being tested at random. Or, conversely, if the general population, regardless of symptoms and index of suspicion,
are being tested at random, then there will be much higher FDR; this can be a reasonable argument for not doing
general testing in the early stages of the pandemic

xi Or prevalence may decrease if testing policy changes from symptomatic-and-high-risk-only-testing to more

inclusive general testing causing the FDR to increase