Note: This copy is for your personal non-commercial use only.

To order presentation-ready
copies for distribution to your colleagues or clients, contact us at www.rsna.org/rsnarights.

ORIGINAL RESEARCH
Overinterpretation and
Misreporting of Diagnostic

n SPECIAL REPORT
Accuracy Studies:Evidence
of “Spin”1
Eleanor A. Ochodo, MBChB, MIH
Purpose: To estimate the frequency of distorted presentation
Margriet C. de Haan, MD
and overinterpretation of results in diagnostic accuracy
Johannes B. Reitsma, MD, PhD
studies.
Lotty Hooft, PhD
Patrick M. Bossuyt, PhD Materials and MEDLINE was searched for diagnostic accuracy studies
Mariska M. G. Leeflang, PhD Methods: published between January and June 2010 in journals with
an impact factor of 4 or higher. Articles included were pri-
mary studies of the accuracy of one or more tests in which
the results were compared with a clinical reference stan-
dard. Two authors scored each article independently by
using a pretested data-extraction form to identify actual
overinterpretation and practices that facilitate overinter-
pretation, such as incomplete reporting of study methods
or the use of inappropriate methods (potential overinter-
pretation). The frequency of overinterpretation was esti-
mated in all studies and in a subgroup of imaging studies.

Results: Of the 126 articles, 39 (31%; 95% confidence interval

[CI]: 23, 39) contained a form of actual overinterpreta-
tion, including 29 (23%; 95% CI: 16, 30) with an overly
optimistic abstract, 10 (8%; 96% CI: 3%, 13%) with a
discrepancy between the study aim and conclusion, and
eight with conclusions based on selected subgroups. In
our analysis of potential overinterpretation, authors of
89% (95% CI: 83%, 94%) of the studies did not include
a sample size calculation, 88% (95% CI: 82%, 94%) did
not state a test hypothesis, and 57% (95% CI: 48%, 66%)
did not report CIs of accuracy measurements. In 43%
(95% CI: 34%, 52%) of studies, authors were unclear
about the intended role of the test, and in 3% (95% CI:
0%, 6%) they used inappropriate statistical tests. A sub-
group analysis of imaging studies showed 16 (30%; 95%
CI: 17%, 43%) and 53 (100%; 95% CI: 92%, 100%) con-
tained forms of actual and potential overinterpretation,
respectively.

1
From the Department of Clinical Epidemiology, Biosta- Conclusion: Overinterpretation and misreporting of results in diag-
tistics & Bioinformatics (E.A.O., P.M.B., M.M.G.L.) and nostic accuracy studies is frequent in journals with high
Department of Radiology (M.C.d.H.), Academic Medical
impact factors.
Centre, University of Amsterdam, Meibergdreef 9, 1105
AZ Amsterdam, the Netherlands; Julius Center for Health
Sciences and Primary Care, University Medical Center
q
RSNA, 2013
Utrecht, Utrecht, the Netherlands (J.B.R.); and Dutch
Cochrane Centre, Academic Medical Centre, University of Supplemental material: http://radiology.rsna.org/lookup
Amsterdam, Amsterdam, the Netherlands (L.H.). Received /suppl/doi:10.1148/radiol.12120527/-/DC1
March 17, 2012; revision requested April 23; revision
received August 21; accepted September 12; final version
accepted October 15. Address correspondence to E.A.O.
(e-mail: e.a.ochodo@amc.uva.nl).

q
RSNA, 2013

Radiology: Volume 267: Number 2—May 2013 n radiology.rsna.org 581

SPECIAL REPORT: Overinterpretation and Misreporting of Diagnostic Accuracy Studies
R
eporting that distorts or misrepre-
sents study results to make the in-
terventions look favorable is called
overinterpretation, which is also referred
to as “spin” (1). Authors may use exag-
gerated language, present an overly op-
timistic conclusion in the abstract com-
pared with that in the main text, or draw
favorable conclusions from results of
selected subgroups (2,3). Overinterpre-
tation may also be introduced because
of methodologic shortcomings such as
failure to specify a study hypothesis or
make a sample size calculation or by the
choice of statistical tests that produce
the desired results (1,3–6). These forms
of misleading representation of the re-
sults of scientific research may compro-
mise decision making in health care and
the well-being of patients.
Overinterpretation has been shown
to be common in randomized con-
trolled trials. Boutron and colleagues
(7) identified overinterpretation in re-
ports of randomized clinical trials with
a clearly identified primary outcome
that showed statistically insignificant
results. More than 40% of the reports
had distorted interpretation in at least
two sections of the main text.
Advances in Knowledge
n Overinterpretation occurs in
about three in 10 diagnostic ac-
curacy studies in journals with an
impact factor of 4 or higher.
n The most frequent form of over-
interpretation is an overly opti-
mistic conclusion in the abstract,
which occurs in about one in
four studies.
n The most common practices that
facilitate overinterpretation (mis-
reporting of results) are not
reporting a sample size calcula-
tion (n = 112 of 126, 89%) and
not stating a test hypothesis (n =
111 of 126, 88%).
n The proportion of overinterpreta-
tion and misreporting of results
in the subgroup of imaging
studies is similar to that of diag-
nostic accuracy studies as a
whole.
582
Overinterpretation may also exist in
diagnostic accuracy studies. Authors of
such studies evaluate the ability of a test
or marker to correctly identify subjects
with the target condition. The clinical
use of tests based on inflated conclu-
sions may cause physicians to make in-
correct clinical decisions, thereby com-
promising patient safety. Exaggerated
conclusions could also lead to unneces-
sary testing and avoidable health care
costs (8). The purpose of this study was
to estimate the frequency of distorted
presentation and overinterpretation of
results in diagnostic accuracy studies.
Materials and Methods
This study was based on a systematic
search of the literature for diagnostic
accuracy studies and the evaluation
of the studies for actual and potential
overinterpretation.
Literature Search
Two authors (E.A.O. and M.M.G.L.,
with 2 and 8 years of experience, re-
spectively) independently searched
MEDLINE in January 2011 for diagnos-
tic accuracy studies published between
January and June 2010 in journals
with an impact factor of 4 or higher.
We focused on these journals because
Lumbreras et al (9) found that overin-
terpretation of the clinical applicabil-
ity of molecular diagnostic tests was
more likely in journals with higher im-
pact factors. This impact factor cut-off
was based on a previously published
analysis of accuracy studies (10). We
limited our search to the most recent
studies indexed in MEDLINE.
The search was a combination of a
previously validated search strategy for
diagnostic accuracy studies (“sensitivity
AND specificity.sh” OR “specificit*.tw”
OR “false negative.tw” OR “accuracy.tw”,
where “.sh” indicates subject heading,
“.tw” indicates text word, and an asterisk
(*) is a wildcard) (11) and a list of 622
international standard serial numbers
of journals with an impact factor of 4 or
higher obtained from the medical library
of the University of Amsterdam. Appen-
dix E1 (online) includes details of the
search strategy.
radiology.rsna.org
Ochodo et al
Eligible for inclusion were primary
studies in which the authors evaluated
the diagnostic accuracy of one or more
tests compared with a clinical reference
standard. Excluded were non-English
studies, animal studies, and studies in
which accuracy measurements were not
reported.
One author (E.A.O.) independently
identified potentially eligible articles by
reading titles and abstracts. To ensure
that no articles were missed, a second au-
thor (M.M.G.L.) independently screened
a random sample of titles and abstracts
of 1000 articles from all those that the
search strategy yielded. We aimed to
find a random sample of the potentially
eligible articles as outlined in the analysis
and results sections. A summary of the
search process is outlined in Figure 1.
Definition of Overinterpretation in
Diagnostic Accuracy Studies
Diagnostic accuracy studies vary by
study question, design, and type and
number of tests evaluated (12,13). We
used a definition of overinterpretation
based on common features that could
apply to a wide range of tests.
We defined overinterpretation as
reporting of diagnostic accuracy stud-
ies that makes tests look more favor-
able than the results justify. We fur-
ther distinguished between actual and
potential forms of overinterpretation.
We defined actual overinterpretation
Published online before print
10.1148/radiol.12120527 Content code:
Radiology 2013; 267:581–588
Abbreviations:
CI = confidence interval
STARD = Standards for Reporting of Diagnostic Accuracy
Author contributions:
Guarantors of integrity of entire study, E.A.O., P.M.B.,
M.M.G.L.; study concepts/study design or data acquisition
or data analysis/interpretation, all authors; manuscript
drafting or manuscript revision for important intellectual
content, all authors; approval of final version of submitted
manuscript, all authors; literature research, E.A.O., J.B.R.,
L.H., M.M.G.L.; statistical analysis, E.A.O., P.M.B.; and
manuscript editing, all authors
Conflicts of interest are listed at the end of this article.
See also the editorial by Levine et al in this issue.
n Radiology: Volume 267: Number 2—May 2013
SPECIAL REPORT: Overinterpretation and Misreporting of Diagnostic Accuracy Studies Ochodo et al

Figure 1
Figure 1: Flowchart of search results.
as explicit false-positive interpretation
of study results and potential overin-
terpretation as practices that facilitate
overinterpretation such as incomplete
reporting of applied study methods and
assumptions or the use of inappropriate
methods. Incomplete reporting of data
may hinder objective appraisal of an ar-
ticle and may mislead readers to think
that test results are favorable (3,4,14).
This definition of overinterpre-
tation was based on items extracted
from published literature (1–3,5,6), the
Standards for Reporting of Diagnostic
Accuracy (STARD) (8), and experi-
ence of content experts on the team.
We first listed the potential items that
could introduce overinterpretation in
diagnostic accuracy studies on the basis
of the experience of content experts on
the team. We then searched MEDLINE
for key literature on poor reporting
and interpretation of scientific reports
that had been published up to Janu-
ary 2011. We identified key articles on
misrepresentation of study findings in
randomized control trials (7), molecu-
lar diagnostic tests (9), and in scientific
research generally (1–3,5,6,15,16).
From these articles, we extracted a list
of potential items that could help iden-
tify overinterpretation in diagnostic ac-
curacy studies.
We then designed and pretested a
data collection form by listing the items
that may introduce overinterpretation on
10 diagnostic accuracy studies published
in 2011, which were independently eval-
uated by five authors. These studies
were not included in the final analysis.
This process of identifying overinterpre-
tation is outlined in Figure 2.
Data Extraction
We extracted data on study characteris-
tics and items that can introduce overin-
terpretation by using the pretested data-
extraction form (Appendix E2 [online]).
We looked for items that can introduce
overinterpretation in the abstract (with
special focus on the results and conclu-
sion section), and in the main text (intro-
duction, methods, results, and conclusion
sections). The actual forms of overinter-
pretation that we extracted included the
following:
An overly optimistic abstract.—This
was considered to be present when the
abstract only reported the best results
but the main text had an array of results
or when stronger test recommendations
or conclusions were reported in the ab-
stract than in the main text. For the lat-
ter, we evaluated the language that was
used to make the recommendations. If
the authors used affirmative language
in the abstract but they used condi-
tional language in the main text to make
recommendations or conclusions, we
scored the article as having a stronger
abstract. Affirmative language included
words such as “is definitely,” “should be,”
“excellent for use,” and “strongly recom-
mended”; conditional language included
words such as “appears to be,” “may be,”
“could be,” and “probably should.”
Favorable study conclusions or test
recommendations based on selected
subgroups.—We scored this as overin-
terpretation when multiple subgroups
were reported in the methods or results
section, but the recommendations were
only based on one of these subgroups.
Caution must be used in the evaluation of
results of subgroup analysis because they
may have a high false-positive rate due to
the effect of multiple testing (17,18).
Discrepancy between study aim and
conclusion.—A conclusion that does
not reflect the aim of the report may
be indicative of flawed results (3,5). We
evaluated the main text of the article to
determine if the conclusion was in line
with the specified aim of the study.
The potential forms of overinter-
pretation included the following:
Not stating a test hypothesis.—We
evaluated whether a statistical test hy-
pothesis was stated, such as the hypo-
thesis that one test was superior to an-
other or that a measure of diagnostic
accuracy would surpass a prespecified
value. The minimally acceptable accu-
racy values (null hypothesis) and antic-
ipated performance values (alternative
hypothesis) of a test under evaluation
depend on the clinical context. These
performance values can be obtained
from pilot data, prior studies, or, in
cases of novel studies, from experts
who may give estimates of clinically
relevant performance values. A priori
specification of a hypothesis limits the
chances of post hoc or subjective judg-
ment about the accuracy and intended
role of the test (4,19). The anticipated
or desired performance measurements
also guide sample-size calculations.
Not reporting a sample size calcula-
tion.—We assessed whether the sample
size required gave a study the power to
allow estimation of test accuracy with
sufficient precision and whether the
method used to calculate the sample size
was reported. Without knowledge of the
calculation used to determine sample
size, readers cannot know if the sample
size is sufficient to estimate the accuracy
measurements with precision (19).
Not stating or unclearly stating the
intended role of the test under evalua-
tion.—We assessed whether the role of
the test was clearly defined in the main
text. Before evaluation and recommenda-
tion of a test, its intended role must be
clearly defined. A new test may be used
to replace the existing test or may be
used before (triage) or after (add-on) an

Radiology: Volume 267: Number 2—May 2013 n radiology.rsna.org 583

SPECIAL REPORT: Overinterpretation and Misreporting of Diagnostic Accuracy Studies Ochodo et al

Figure 2

Figure 2: Flowchart of development of definition of overinterpretation. ∗ = For interventional studies, P values are used to evaluate overinterpretation (7). In diagnostic
tests, overinterpretation can occur when a low reference value or poor comparator is chosen, which can result in reporting of nonsignificant differences as significant.
Because reference values and comparators were not stated in our sample tests, scoring was difficult.

existing test (20,21). The preferred accu-
racy value of a test depends on its role.
Not prespecifying groups for sub-
group analysis a priori in the methods
section.—We assessed whether the
subgroups presented in the results
section were prespecified at the start
of the study. Failure to prespecify sub-
groups can lead to post hoc analyses
motivated by initial inspection of data
and may give room for manipulation of
results to look favorable (17,18,22).
Not prespecifying positivity thresh-
olds of tests.—For continuous tests,
we determined whether the optimal
threshold or cut-off value at which a
test result is either considered positive
or negative was prespecified before the
start of the study. Stating a threshold
value after data collection and analysis
may give room for manipulation to max-
imize a test characteristic (4,23,24).
Not stating CIs of accuracy mea-
surements.—We determined if the con-
fidence intervals (CIs) of the accuracy
estimates were reported. CIs enable
readers to appreciate the precision of
the accuracy measurements. Without
these data, it is difficult to assess the
clinical applicability of the tests (25–27).
Using inappropriate statistical
tests.—We evaluated the tests of sig-
nificance used to compare the accu-
racy measurements of the index test
and reference standard or those that
compared the accuracy measurements
of multiple tests. In diagnostic accuracy
studies, the test of significance to be
used depends on the role of the test
under evaluation and whether the tests
are performed in different groups of pa-
tients (unpaired design) or in the same
patients (paired design) (28).
Two authors scored each article in-
dependently. The abstract and main text
sections (introduction, methods, results,
discussion, and conclusion) were read.
One author (E.A.O.) scored all the se-
lected articles. The other five authors
(M.C.d.H., J.B.R., L.H., P.M.B., with
4, 18, 13, and 25 years of experience,
respectively, and M.M.G.L.) scored the
same articles in predetermined pro-
portions. Disagreement was resolved

584 radiology.rsna.org n Radiology: Volume 267: Number 2—May 2013

SPECIAL REPORT: Overinterpretation and Misreporting of Diagnostic Accuracy Studies Ochodo et al

by consensus or by consultation with a Table 1

third party when needed.
Summary of Study Characteristics
Analysis Characteristic All Studies (n = 126)

We calculated the proportion of ar- Median journal impact factor, median* 5.491 (4.014–16.225)
ticles with actual and potential over- Sample size, median (range) 150.5 (12–20 765)
interpretation, and the level of inter- Type of test evaluated
rater agreement in scoring both. We Clinical diagnosis (history and physical examination) 13 (10) [5, 16]
anticipated that 40% of articles would Imaging test 53 (42) [33, 51]
have a distorted presentation or overin- Biochemical test 13 (10) [5, 16]
terpretation (7), so we calculated that Molecular test 14 (11) [6, 17]
evaluating 100 articles would produce a Immunodiagnostic test 14 (11) [6, 17]
two-sided 95% CI of 30%-50% by us- Other 19 (15) [9, 21]
ing the exact (Clopper-Pearson) method Study design
Single test 64 (51) [42, 60]
(29,30).
Comparator test 62 (49) [40, 58]
We analyzed all the included studies
Study design
and the subset of imaging studies to es-
Cross sectional 102 (81) [74, 88]
timate the frequencies of actual and po-
Longitudinal (with follow up) 17 (13) [7, 19]
tential overinterpretation by using SAS
Diagnostic randomized design 1 (1) [0, 2]
version 9.2 (SAS Institute, Cary, NC). Case control 2 (2) [0, 4]
Unclear 5 (4) [0, 6]
Sampling method
Results
Consecutive series 60 (48) [39, 56]
Search Results Random series 6 (5) [1, 8]
Our initial search yielded 6978 articles. Convenience sampling 23 (18) [11, 25]
Multistage stratified sampling 1 (1) [0, 2]
After the titles and abstracts were read,
Unclear 36 (28) [21, 37]
6558 articles were deemed to be ineli-
Number of groups from which patients were sampled
gible. Of the remaining 420 potentially
One group 87 (69) [61, 77]
eligible articles, we randomly selected
Different groups 30 (24) [16, 31]
140 articles for evaluation by using
Unclear 9 (7) [3, 12]
STATA version 10.0 (Stata, College Sta-
tion, Tex) with the code “sample 140, Note.—Unless otherwise indicated, data are number of studies, with percentage in parentheses and 95% CIs in brackets.
count.” We included more articles than * Numbers in parentheses are the range.
indicated by our sample size calculation
to compensate for any false-positive ar-
ticles in the random selection. After the Table 2
full texts of these 140 articles were as-
Actual Overinterpretation in Diagnostic Accuracy Studies
sessed, 14 studies were excluded (Fig
Form of Actual Overinterpretation All Studies (n = 126) Imaging Studies (n = 53)
1). A total of 126 studies were included
in the final analysis. Overly optimistic abstract 29 (23) [16–30] 13 (25) [13–37]
Stronger conclusion in abstract 22 (17) [11–24] 9 (17) [9–30]
Characteristics of Included Articles Selective reporting of results in abstract 7 (6) [2–10] 4 (8) [3–20]
Study conclusions based on selected subgroups 8* (10) [5–19] 3† (7) [2–21]
Details of the study characteristics are
Discrepancy between aim and conclusion 10 (8) [3–13] 2 (4) [0–9]
outlined in Table 1. In summary, the me-
Articles with one or more forms of actual 39 (31) [23–39] 16 (30) [17–43]
dian impact factor of all the included ar-
overinterpretation
ticles was 5.5 (range, 4.0–16.2) and the
median sample size was 151 (range, 12– Note.—Data are number of studies, with percentage in parentheses and 95% CIs in brackets. Studies can have more than one
20 765). Of all the tests evaluated in the form of actual overinterpretation

included articles, imaging tests formed * Eighty articles included analysis of subgroups (n = 80).
†
Forty-one imaging articles included analysis of subgroups (n = 41).
the largest group (n = 53 of 126, 42%).

Agreement
Interrater agreement for scoring both Actual Overinterpretation overinterpretation (Table 2). The most
actual and potential overinterpretation Of 126 articles, 39 (31%; 95% CI: frequent form of overinterpretation
are outlined in Appendix E3 (online). 23%, 39%) contained a form of actual was overly optimistic abstract (n = 29

Radiology: Volume 267: Number 2—May 2013 n radiology.rsna.org 585

SPECIAL REPORT: Overinterpretation and Misreporting of Diagnostic Accuracy Studies Ochodo et al

Table 3 of 126; 23%; 95% CI: 16%, 30%). Of

these, 22 (17%; 95% CI: 11%, 24%)
Potential Overinterpretation in Diagnostic Accuracy Studies articles had stronger test recommen-
Forms of Potential Overinterpretation All Studies (n = 126) Imaging Studies (n = 53) dations or conclusions in the abstract
than in the main text, and seven (6%;
Sample size calculation not reported 112 (89) [83, 94] 48 (91) [82, 99] 95% CI: 2%, 10%) articles had selec-
Test hypothesis not stated 111 (88) [82, 94] 47 (89) [80, 98] tively reported favorable results in the
CIs of accuracy measurements not reported 72 (57) [48, 66] 26 (49) [35, 63]
abstract.
Role of test not stated or unclear 54 (43) [34, 52] 17 (32) [19, 45]
Of the 53 included imaging studies,
Groups for subgroup analysis not prespecified in the 25/80* (31) [21, 42] 8/41† (20) [8, 36]
16 articles (30%; 95% CI: 17%, 43%)
methods section of main text
contained a form of actual overinter-
Positivity thresholds of continuous tests 22/63‡ (35) [24, 48] 6/25§ (24) [10, 45]
pretation (Table 2). Similar to the re-
not reported
Use of inappropriate statistical tests 4 (3) [0, 6] 3 (6) [0, 12]
sults of the overall studies, the most fre-
Overall proportion of articles with one or more forms 125 (99) [98, 100] 53 (100) [92, 100] quent form of actual overinterpretation
of potential overinterpretation was an overly optimistic abstract, in 13
articles (25%; 95% CI: 13%, 37%).
Note.—Data are number of studies, with percentage in parentheses and 95% CIs in brackets. Studies can have more than one
form of potential overinterpretation.
Potential Overinterpretation
* Eighty articles included analysis of subgroups.
†
Forty-one imaging articles included analysis of subgroups. Details of the potential forms of over-
‡
Sixty-three articles included evaluation of continuous tests. interpretation are outlined in Table 3.
§
Twenty-five articles included evaluation of continuous tests. Of the 126 included articles, only 14
(11%) reported a sample size calcula-
tion. Only 15 (12%) articles reported
an explicit study hypothesis. All imaging
Table 4 studies (n = 53) contained a form of
potential overinterpretation. Of these,
Examples of Actual Overinterpretation
only five (9%) included articles that re-
Actual Overinterpretation ported a sample size calculation. Only
six (11%) articles reported an explicit
1. An abstract with a stronger conclusion: (31)
Conclusion in main text:
study hypothesis. Examples of overin-
“Detection of antigen in BAL using the Mvista antigen appears to be a useful method (…) Additional terpretation are provided in Table 4.
studies are needed in patients with pulmonary histoplasmosis.”
Conclusion in Abstract: Discussion
“Detection of antigen in BAL fluid complements antigen detection in serum and urine as an objective
Our study results showed that about
test for histoplasmosis”
three of 10 studies of the diagnostic
2. Conclusions drawn from selected subgroups: (32)
accuracy of biomarkers or other med-
A study evaluates the aptness of F-desmethoxyfallypride (F-DMFP) PET for the differential diagnosis of
ical tests published in journals with an
Idiopathic Parkinsonian Syndrome (IPS) and non-IPS in a series of 81 patients with a clinical
diagnosis of Parkinsonism. The authors compared several F-DMFP PET indexes for the
impact factor of 4 or higher contained
discrimination of IPS and non-IPS and reported only the best sensitivity and specificity estimates. overinterpreted results, and 99% of
They concluded that F-DMFP PET was an accurate method for differential diagnosis. studies contained practices that facili-
3. Disconnect between the aim and conclusion of the study: (33) tate overinterpretation. The most com-
The study design described in this paper aimed to evaluate the sensitivity and specificity of the mon form of actual overinterpretation
IgM anti-EV71 assay. However the conclusion is not on accuracy rather it focuses on other is an overly optimistic abstract (ap-
measurements of diagnostic performance. proximately one in four) with stronger
Aim of study: conclusions or test recommendations
“The aim of this study was to assess the performance of detecting IgM anti-EV71 for early diagnosis than those in the main text (approxi-
of patients with HFMD.” mately one in five). In terms of prac-
Conclusion: tices that facilitate overinterpretation,
“The data here presented show that the detection of IgM anti-EV71 by ELISA affords a reliable or potential overinterpretation, most
convenient and prompt diagnosis of EV71. The whole assay takes 90 mins using readily available of the studies did not report an a pri-
ELISA equipment, is easy to perform with low cost which make it suitable in clinical diagnosis as ori formulated test hypothesis, did not
well as in public health utility.” include a corresponding sample size
Note.—BAL = bronchoalveolar lavage, PET = positron emission tomography, IgM = Immunoglobulin M, EV71 = enterovirus 71, calculation, and did not include CIs for
HFMD = hand, foot, and mouth disease, ELISA = enzyme-linked immunosorbent assay. accuracy measurements. In a closely re-
lated study, Lumbreras and colleagues

586 radiology.rsna.org n Radiology: Volume 267: Number 2—May 2013

SPECIAL REPORT: Overinterpretation and Misreporting of Diagnostic Accuracy Studies
(9) evaluated overinterpretation of the
clinical applicability of molecular diag-
nostic tests only. Of the 108 evaluated
articles, 61 (56%) had overinterpreta-
tions of the clinical applicability of the
molecular test under study.
The defining strengths of our study
were that we analyzed a sample of di-
agnostic accuracy studies evaluating
a wide range of tests and we defined
overinterpretation in terms of com-
mon features that apply to most tests.
To limit subjectivity, we systematically
searched for diagnostic studies with a
validated search strategy. Scoring of
the articles was done by two authors in-
dependently by using a pretested data-
extraction form.
The forms of overinterpretation
that we found in our study may have
several implications for diagnostic re-
search and practice. One of the most
important consequences might be that
diagnostic accuracy studies with opti-
mistic conclusions may be highly cited,
leading to a cascade of inflated and
questionable evidence in the literature.
Subsequently, this may translate to the
premature adoption of tests in clinical
practice. In a recently published review,
Ioannidis and Panagiotou (34) reported
that highly cited biomarker studies of-
ten had inflated results. Of the included
highly cited studies in their review, 86%
had larger effect sizes than those in the
largest study and 83% had larger effect
sizes than those in a corresponding
meta-analysis evaluating the same bio-
marker. Of note, the larger studies and
meta-analysis received fewer citations
(35).
Our study was largely limited to
what was reported. For instance, there
was no guarantee that, because sub-
groups or thresholds were listed in the
methods, they were indeed prespeci-
fied. An alternative is to look at study
protocols, but unlike in randomized
trials, protocols of diagnostic accuracy
studies are not always registered. An-
other limitation of our study was the
considerable variation in the interra-
ter agreement for scoring. The overall
scoring of articles was difficult because
many articles were poorly reported.
Many authors had not used the STARD
Radiology: Volume 267: Number 2—May 2013
guidelines in preparing their manu-
scripts. The suboptimal use of STARD
has also been documented in previous
reports (36–41).
Comprehensively evaluating over-
interpretation in diagnostic studies
depends on the context in which the
test is used. For instance, overinterpre-
tation can occur when positive recom-
mendations are made for the clinical
use of tests even if the accuracy mea-
surements do not justify this. Because
of the wide range of tests evaluated in
our study, it was difficult to determine
a standard cutoff measure to define
low and high accuracy measurements.
Preferred accuracy measurements dif-
fer and are dependent on the type of
test used, the role of the test, the tar-
get condition, and the setting in which
the test is being evaluated and on the
accuracy of other methods available.
A sensitivity of 80% may be “definitely
useful” in one situation, but it may be
useless in another situation.
In addition, not reporting CIs may
be regarded as either actual or po-
tential overinterpretation depending
on the context. For instance, report-
ing very high point estimates such as
99% without CIs on the basis of a small
sample size such as 10 cases may be
regarded as actual overinterpretation.
On the other hand, not reporting CIs
in cases of moderate or high estimates
with large sample sizes or in compara-
tive evaluations of two tests when one
is statistically superior, can be regarded
as potential overinterpretation.
To curb the occurrence of overin-
terpretation and misreporting of re-
sults in diagnostic accuracy studies, we
recommend that journals continually
emphasize that the manuscripts sub-
mitted must be reported according to
the STARD reporting guidelines. This
may also diminish the methodologic
conditions that may lead to overinter-
pretation. Readers largely depend on
abstracts to draw conclusions about
an article, and sometimes when full
texts are not available, they may make
decisions on the basis of abstracts
alone (7,42,43). Hence, reviewers
must be more stringent when reading
abstracts of submitted manuscripts
n radiology.rsna.org
Ochodo et al
to ensure that the abstracts are fair
representations of the main texts. We
hope that highlighting the forms of
overinterpretation will enable peer re-
viewers to correctly identify overly op-
timistic reports of diagnostic accuracy
studies and encourage investigators to
be clearer in designing, more transpar-
ent in reporting, and more stringent in
interpreting test accuracy studies.
Acknowledgments: We thank René Spijker,
MSc (Dutch Cochrane Centre, University of
Amsterdam) for assisting in the development
of the search strategy of this project. We also
thank Gordon Guyatt, BSc, MD, MSc, FR-
CPC (McMaster University, Canada) and Paul
Glasziou, PhD, FRACGP, MRCGP (Bond Uni-
versity) for their comments on an earlier ver-
sion of this manuscript. Mr Spijker, Dr Guyatt
and Dr Glasziou received no compensation for
their contributions.
Disclosures of Conflicts of Interest: E.A.O.
No relevant conflicts of interest to disclose.
M.C.d.H. No relevant conflicts of interest to
disclose. J.B.R. No relevant conflicts of interest
to disclose. L.H. No relevant conflicts of inter-
est to disclose. P.M.B. No relevant conflicts of
interest to disclose. M.M.G.L. No relevant con-
flicts of interest to disclose.
References
1. Fletcher RH, Black B. “Spin” in scientific
writing: scientific mischief and legal jeop-
ardy. Med Law 2007;26(3):511–525.
2. Horton R. The rhetoric of research. BMJ
1995;310(6985):985–987.
3. Marco CA, Larkin GL. Research ethics:
ethical issues of data reporting and the
quest for authenticity. Acad Emerg Med
2000;7(6):691–694.
4. Bossuyt PM, Reitsma JB, Bruns DE, et
al. The STARD statement for reporting
studies of diagnostic accuracy: explana-
tion and elaboration. Ann Intern Med
2003;138(1):W1–W12.
5. Zinsmeister AR, Connor JT. Ten common
statistical errors and how to avoid them.
Am J Gastroenterol 2008;103(2):262–266.
6. Scott IA, Greenberg PB, Poole PJ. Cau-
tionary tales in the clinical interpretation
of studies of diagnostic tests. Intern Med J
2008;38(2):120–129.
7. Boutron I, Dutton S, Ravaud P, Altman DG.
Reporting and interpretation of randomized
controlled trials with statistically nonsignif-
icant results for primary outcomes. JAMA
2010;303(20):2058–2064.
8. Bossuyt PM, Reitsma JB, Bruns DE, et al.
Towards complete and accurate reporting of
587
SPECIAL REPORT: Overinterpretation and Misreporting of Diagnostic Accuracy Studies
studies of diagnostic accuracy: The STARD
Initiative. Ann Intern Med 2003;138(1):40–
44.
9. Lumbreras B, Parker LA, Porta M, Pol-
lán M, Ioannidis JP, Hernández-Aguado I.
Overinterpretation of clinical applicabil-
ity in molecular diagnostic research. Clin
Chem 2009;55(4):786–794.
10. Smidt N, Rutjes AW, van der Windt DA, et
al. Quality of reporting of diagnostic accu-
racy studies. Radiology 2005;235(2):347–
353.
11. Devillé WL, Bezemer PD, Bouter LM. Pub-
lications on diagnostic test evaluation in
family medicine journals: an optimal search
strategy. J Clin Epidemiol 2000;53(1):65–
69.
12. Knottnerus JA, Muris JW. Assessment
of the accuracy of diagnostic tests: the
cross-sectional study. In: Knottnerus JA,
ed. The evidence base of clinical diagnosis.
London, England: BMJ Publishing Group,
2002; 39–59.
13. Irwig L, Bossuyt P, Glasziou P, Gatsonis
CA, Lijmer JG. Designing studies to ensure
that estimates of test accuracy will travel.
In: Knottnerus JA, ed. The evidence base
of clinical diagnosis. London, England: BMJ
Publishing Group, 2002; 95–116.
14. Chalmers I. Underreporting research is
scientific misconduct. JAMA 1990;263(10):
1405–1408.
15. Ioannidis JP. Why most published re-
search findings are false. PLoS Med
2005;2(8):e124.
16. Young NS, Ioannidis JP, Al-Ubaydli O. Why
current publication practices may distort
science. PLoS Med 2008;5(10):e201.
17. Wang R, Lagakos SW, Ware JH, Hunter DJ,
Drazen JM. Statistics in medicine—report-
ing of subgroup analyses in clinical trials. N
Engl J Med 2007;357(21):2189–2194.
18. Lagakos SW. The challenge of subgroup
analyses—reporting without distorting. N
Engl J Med 2006;354(16):1667–1669.
19. Pepe MS, Feng Z, Janes H, Bossuyt PM,
Potter JD. Pivotal evaluation of the accu-
racy of a biomarker used for classification
or prediction: standards for study design. J
Natl Cancer Inst 2008;100(20):1432–1438.
588
20. Fritz JM, Wainner RS. Examining diagnostic
tests: an evidence-based perspective. Phys
Ther 2001;81(9):1546–1564.
21. Bossuyt PM, Irwig L, Craig J, Glasziou P.
Comparative accuracy: assessing new tests
against existing diagnostic pathways. BMJ
2006;332(7549):1089–1092.
22. Montori VM, Jaeschke R, Schünemann HJ,
et al. Users’ guide to detecting misleading
claims in clinical research reports. BMJ
2004;329(7474):1093–1096.
23. Ewald B. Post hoc choice of cut points in-
troduced bias to diagnostic research. J Clin
Epidemiol 2006;59(8):798–801.
24. Leeflang MM, Moons KG, Reitsma JB, Zwin-
derman AH. Bias in sensitivity and specific-
ity caused by data-driven selection of optimal
cutoff values: mechanisms, magnitude, and
solutions. Clin Chem 2008;54(4):729–737.
25. Harper R, Reeves B. Reporting of precision
of estimates for diagnostic accuracy: a re-
view. BMJ 1999;318(7194):1322–1323.
26. Habbema JDF, Eijekmans R, Krijnen P,
Knottnerus JA. Analysis of data on the accu-
racy of diagnostic tests. In: Knottnerus JA,
ed. The evidence base of clinical diagnosis.
London, England: BMJ Publishing Group,
2002; 117–144.
27. Altman DG. Why we need confidence inter-
vals. World J Surg 2005;29(5):554–556.
28. Hayen A, Macaskill P, Irwig L, Bossuyt P.
Appropriate statistical methods are re-
quired to assess diagnostic tests for replace-
ment, add-on, and triage. J Clin Epidemiol
2010;63(8):883–891.
29. Clopper CJ, Pearson ES. The use of confi-
dence or fiducial limits illustrated in the case
of the binomial. Biometrika 1934;26(4):404–
413.
30. Hintze JL. PASS 11 (Power Analysis and
Sample Size). Kaysville Utah: NCSS, 2011.
31. Hage CA, Davis TE, Fuller D, et al. Diagno-
sis of histoplasmosis by antigen detection in
BAL fluid. Chest 2010;137(3):623–628.
32. la Fougère C, Pöpperl G, Levin J, et al.
The value of the dopamine D2/3 receptor
ligand 18F-desmethoxyfallypride for the
differentiation of idiopathic and nonidio-
pathic parkinsonian syndromes. J Nucl Med
2010;51(4):581–587.
radiology.rsna.org
Ochodo et al
33. Xu F, Yan Q, Wang H, et al. Performance
of detecting IgM antibodies against entero-
virus 71 for early diagnosis. PLoS ONE
2010;5(6):e11388.
34. Ioannidis JP, Panagiotou OA. Comparison
of effect sizes associated with biomarkers
reported in highly cited individual articles
and in subsequent meta-analyses. JAMA
2011;305(21):2200–2210.
35. Bossuyt PM. The thin line between hope
and hype in biomarker research. JAMA
2011;305(21):2229–2230.
36. Paranjothy B, Shunmugam M, Azuara-
Blanco A. The quality of reporting of diag-
nostic accuracy studies in glaucoma using
scanning laser polarimetry. J Glaucoma
2007;16(8):670–675.
37. Bossuyt PM. STARD statement: still
room for improvement in the reporting
of diagnostic accuracy studies. Radiology
2008;248(3):713–714.
38. Wilczynski NL. Quality of reporting of di-
agnostic accuracy studies: no change since
STARD statement publication—before-and-
after study. Radiology 2008;248(3):817–823.
39. Fontela PS, Pant Pai N, Schiller I, Den-
dukuri N, Ramsay A, Pai M. Quality and
reporting of diagnostic accuracy studies
in TB, HIV and malaria: evaluation using
QUADAS and STARD standards. PLoS ONE
2009;4(11):e7753.
40. Areia M, Soares M, Dinis-Ribeiro M.
Quality reporting of endoscopic diagnostic
studies in gastrointestinal journals: where
do we stand on the use of the STARD
and CONSORT statements? Endoscopy
2010;42(2):138–147.
41. Selman TJ, Morris RK, Zamora J, Khan
KS. The quality of reporting of primary test
accuracy studies in obstetrics and gynae-
cology: application of the STARD criteria.
BMC Womens Health 2011;11:8.
42. Pitkin RM, Branagan MA, Burmeister LF.
Accuracy of data in abstracts of published
research articles. JAMA 1999;281(12):1110–
1111.
43. Beller EM, Glasziou PP, Hopewell S, Altman
DG. Reporting of effect direction and size
in abstracts of systematic reviews. JAMA
2011;306(18):1981–1982.
n Radiology: Volume 267: Number 2—May 2013