You are on page 1of 6

Essay

How to Make More Published Research True


John P. A. Ioannidis1,2,3,4*
1 Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California, United States of America, 2 Department of Medicine, Stanford
Prevention Research Center, Stanford, California, United States of America, 3 Department of Health Research and Policy, Stanford University School of Medicine, Stanford,
California, United States of America, 4 Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, California, United States of America

The achievements of scientific research


are amazing. Science has grown from the Summary Points
occupation of a few dilettanti into a vibrant
global industry with more than 15,000,000 N Currently, many published research findings are false or exaggerated, and an
people authoring more than 25,000,000 estimated 85% of research resources are wasted.
scientific papers in 19962011 alone [1]. N To make more published research true, practices that have improved credibility
However, true and readily applicable major and efficiency in specific fields may be transplanted to others which would
discoveries are far fewer. Many new benefit from thempossibilities include the adoption of large-scale collabo-
proposed associations and/or effects are rative research; replication culture; registration; sharing; reproducibility
false or grossly exaggerated [2,3], and practices; better statistical methods; standardization of definitions and analyses;
translation of knowledge into useful appli- more appropriate (usually more stringent) statistical thresholds; and improve-
cations is often slow and potentially ineffi- ment in study design standards, peer review, reporting and dissemination of
cient [4]. Given the abundance of data, research, and training of the scientific workforce.
research on research (i.e., meta-research) N Selection of interventions to improve research practices requires rigorous
can derive empirical estimates of the examination and experimental testing whenever feasible.
prevalence of risk factors for high false- N Optimal interventions need to understand and harness the motives of various
positive rates (underpowered studies; small stakeholders who operate in scientific research and who differ on the extent to
effect sizes; low pre-study odds; flexibility in which they are interested in promoting publishable, fundable, translatable, or
designs, definitions, outcomes, analyses; profitable results.
biases and conflicts of interest; bandwagon N Modifications need to be made in the reward system for science, affecting the
patterns; and lack of collaboration) [3]. exchange rates for currencies (e.g., publications and grants) and purchased
Currently, an estimated 85% of research academic goods (e.g., promotion and other academic or administrative power)
resources are wasted [5]. and introducing currencies that are better aligned with translatable and
reproducible research.

Effective Interventions
We need effective interventions to im- analysis plan: the error rate would decrease reviewers typically have only modest CVs
prove the credibility and efficiency of to zero simply because no research would and most of the top influential scientists dont
scientific investigation. Some risk factors ever be done. Thus, whatever solutions are review grant applications and dont get
for false results are immutable, like small proposed should be pragmatic, applicable, funded by government funds, even in the
effect sizes, but others are modifiable. We and ideally, amenable to reliable testing of United States [6], which arguably has the
must diminish biases, conflicts of interest, their performance. strongest scientific impact at the moment
and fragmentation of efforts in favor of Currently, major decisions about how than any other country (e.g., in cumulative
unbiased, transparent, collaborative re- research is done may too often be based on citations). Non-meritocratic practices, includ-
search with greater standardization. How- convention and inertia rather than being ing nepotism, sexism, and unwarranted
ever, we should also consider the possibility highly imaginative or evidence-based [515]. conservatism, are probably widespread [7].
that interventions aimed at improving For example, there is evidence that grant Allegiance and confirmation biases are
scientific efficiency may cause collateral
damage or themselves wastefully consume
resources. To give an extreme example, Citation: Ioannidis JPA (2014) How to Make More Published Research True. PLoS Med 11(10): e1001747.
doi:10.1371/journal.pmed.1001747
one could easily eliminate all false positives
simply by discarding all studies with even Published October 21, 2014
minimal bias, by making the research Copyright: 2014 John P. A. Ioannidis. This is an open-access article distributed under the terms of the
questions so bland that nobody cares about Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any
(or has a conflict with) the results, and by medium, provided the original author and source are credited.
waiting for all scientists in each field to join Funding: The Meta-Research Innovation Center at Stanford is funded by the Laura and John Arnold
forces on a single standardized protocol and Foundation. The funders had no role in the decision to publish or in the preparation of the manuscript.

Competing Interests: JPAI is a member of the Editorial Board of PLOS Medicine. The author has declared that
no competing interests exist.
Essays are opinion pieces on a topic of broad
interest to a general medical audience. * E-mail: jioannid@stanford.edu
Provenance: Commissioned; externally peer reviewed

PLOS Medicine | www.plosmedicine.org 1 October 2014 | Volume 11 | Issue 10 | e1001747


powerful in scientific processes [8,9]. For ogy from a spurious field [17] to a highly have been debated, like the potential of
healthcare and clinical practice, while credible one [18]. Such practices could be multiple analysts performing contradicting
evidence-based medicine has grown stron- applied to other fields of observational analyses, difficulties with de-identification
ger over time, some argue that it is research and beyond [19]. of participants, and the potential for
currently in crisis [10] and evidence- Replication has different connotations parties to introduce uncertainty for results
based terminology has been usurped to for different settings and designs. For basic that hurt their interests, as in the case of
promote expert-based beliefs and industry laboratory and preclinical studies, replica- diesel exhaust and cancer risk [33].
agendas [11]. We have little experimental tion should be feasible as a default, but Dissociation of some research types from
evidence on how peer review should be even in those cases, there should be an a specific conflicted sponsors or authors has
done and when (e.g., protocol-based, man- priori understanding of the essential fea- been proposed (not without debate) for
uscript-based, post-publication) [5,12,13] tures that are needed to be replicated and designs as diverse as cost-effectiveness analy-
or on how research funds should be how much heterogeneity is acceptable ses [34], meta-analyses [35,36], and guide-
allocated [14,15]. Many dominant scientif- [20]. For some clinical research, replica- lines [37]. For all of these types of research,
ic structures date back to the Middle Ages tion is difficult, especially for very large, involvement of sponsors with conflicts has
(e.g., academic hierarchies) or the 17th long-term, expensive studies. The prospect been shown to spin more favorable conclu-
century (e.g., professional societies, journal of replication needs to be considered and sions.
publishing), but their suitability for the incorporated up front in designing the Adoption of more appropriate statistical
current growth of science is uncertain. At research agenda in a given field [12]. methods [38], standardized definitions and
the same time, there is an obvious tension Otherwise, some questions are not ad- analyses and more stringent thresholds for
in hoping for decisions to be both more dressed at all or are addressed by single claiming discoveries or successes [39] may
imaginative and more evidence-based; it studies that are never replicated, while decrease false-positive rates in fields that have
may be the case that the bureaucracy and others are subjected to multiple unneces- to-date been too lenient (like epidemiology
practice of science require different people sary replications or even redundant meta- [40], psychology [41,42], or economics [43]).
with different skill sets, and it may even be analyses combining them [21]. It may lead them to higher credibility, more
that a system too focused on eliminating Registration of randomized trials [22] akin to that of fields that have traditionally
unfair discrimination also eliminates the (and, more recently, registration of their been more rigorous in this regard, like the
reasonable discrimination required to make results [23]) has enhanced transparency in physical sciences [44].
wise choices. While we could certainly clinical trials research and has allowed Improvements in study design standards
introduce changes that made science worse, probing of selective reporting biases [24,25], could improve the reliability of results [45].
we could also purposefully introduce ones even if not fully remedying them. It may For example, for animal studies of interven-
to make it better. show redundancy and allow better visualizing tions, this would include randomization and
One option is to transplant into as many of the evolution of the total corpus of research blinding of investigators [27]. There is
scientific disciplines as possible research in a given field. Registration is currently increasing interest in proposing checklists for
practices that have worked successfully proposed for many other types of research, the conduct of studies to be approved
when applied elsewhere. Box 1 lists a few including both human observational studies [46,47], making it vital to ensure both that
examples that are presented in more detail [26] and nonhuman studies [27]. checklist items are indeed essential and that
here. Sharing of data, protocols, materials, claims of adherence to them are verifiable.
Adoption of large-scale collaborative and software has been promoted in several Reporting, review, publication, dissem-
research with a strong replication culture -omics fields, creating a substrate for ination, and post-publication review of
[16] has been successful in several bio- reproducible data practices [2831]. Pro- research shape its reliability. There are
medical fields: in particular, in genetic and motion of data sharing in clinical trials currently multiple efforts to improve and
molecular epidemiology. These techniques may similarly improve the credibility of standardize reporting (e.g., as catalogued
have helped transform genetic epidemiol- clinical research [32]. Some disadvantages by the EQUATOR initiative [48]) and
multiple ideas about how to change peer
review (by whom, how, and when) and
Box 1. Some Research Practices that May Help Increase the dissemination of information [25,4951].
Proportion of True Research Findings Finally, proper training and continuing
education of scientists in research methods
N Large-scale collaborative research and statistical literacy are also important [47].
N Adoption of replication culture
Stakeholders
N Registration (of studies, protocols, analysis codes, datasets, raw data, and
results) As we design, test, and implement
N Sharing (of data, protocols, materials, software, and other tools) interventions on research practices, we
N Reproducibility practices need to understand who is affected by and
N Containment of conflicted sponsors and authors shaping research [5,52,53]. Scientists are
only one group in a larger network
N More appropriate statistical methods
(Table 1) in which different stakeholders
N Standardization of definitions and analyses
have different expectations. Stakeholders
N More stringent thresholds for claiming discoveries or successes may cherish research for being publish-
N Improvement of study design standards able, fundable, translatable, or profitable.
N Improvements in peer review, reporting, and dissemination of research Their expectations are not necessarily
N Better training of scientific workforce in methods and statistical literacy aligned with one another. Scientists may
continue publishing and getting grants

PLOS Medicine | www.plosmedicine.org 2 October 2014 | Volume 11 | Issue 10 | e1001747


Table 1. Some major stakeholders in science and their extent of interest in research and its results from various perspectives;
typical patterns are presented (exceptions do occur).

Extent of interest in research results

Publishable Fundable Translatable Profitable

Scientists +++ +++ +


Industry sales and marketing +++
Industry R & D +++ +++
Private investors, including hedge funds ++ +++
Public funders open (e.g. NIH, NSF) ++ +
Public funders closed (e.g. military) +++
Not-for-profit funders/philanthropists ++ +++
Journal editors +++ +
For-profit publishers + +++
Professional and scientific societies +
Universities + +++ +
Not-for-profit research institutions +++ +++ + +
Supporting non-scientific staff +++
Hospitals and other professional facilities offering services related to science + +++
Other financial entities that are affected by these services (e.g. insurance) +++
Governments and state/federal authorities ++
Consumers of products and services +++

doi:10.1371/journal.pmed.1001747.t001

without making real progress, if more targets. Conversely, the industry is defen- the system, producing prolifically mediocre
publications and more grants are all that sive about data sharing from clinical trials and/or irreproducible research; controlling
matters. If science is supported primarily [30], which occurs at a point in the peer review at journals and study sections;
by private investors who desire patents product development when re-analyses enjoying sterile bureaucracy, lobbying, and
and profit, this may lead to expedited may correctly or incorrectly [58] invali- maneuvering; and promoting those who
translation and discoveries that work (or date evidence supporting drugs in which it think and act in the same way.
seem to work) but also barriers against has already invested heavily. There are also opportunities in grasping
transparency and sharing of information. Dynamics between different stakeholders the importance of the key currencies. For
Corporate influence may subvert science are complex. Moreover, sometimes the example, registration of clinical trials
for the purposes of advertising, with same person may wear many stakeholder worked because all major journals adopted
papers in influential journals, prestigious hats; e.g., an academic researcher may also it as prerequisite for publication [60], a
society meetings, and a professorate sys- be journal editor, spin-off company owner, major reference currency in the reward
tem of opinion leaders becoming branches professional society officer, government chain. Conversely, interesting post-publi-
of their marketing department [11,54]. advisor, and/or beneficiary of the industry. cation review efforts such as PubMed
The geography of scientific production Commons [61] have so far not fulfilled
changes rapidly; e.g., soon there will be Research Currencies their potential as progressive vehicles for
more English language papers from China evaluating research, probably because
than from Europe and the US [55]. Publications and grants are key curren- there is currently no reward for such
Research efforts are embedded in wider cies in science (Table 2). They purchase post-publication peer review.
societies, which have provided scientific academic goods such as promotion and
developments that differ according to time other power. Academic titles and power Modifying the Reward System
period and location. What can be done to add further to the wealth of their
enhance the capacity of science to flourish possessor. The exact exchange rate of The reward system may be systematically
and to assess and promote this capacity currencies and the price of academic goods modified [62]. Modifying interventions may
across cultures that may vary in attitudes [59] may vary across institutional microen- be anywhere from fine-tuning to disruptive.
toward skepticism, inquisitiveness, and vironments, scientific disciplines and cir- Table 2 compares the status quo (first
contrarian reasoning? Different stakehold- cumstances, and are also affected by each column) against two potential modifications
ers have their own preferences about when microenvironments fairness or unfairness of the reward system, with Change 2 being
reproducibility should be promoted or (e.g., nepotism, cronyism, or corruption). more prominent than Change 1.
shunned. Pharmaceutical industry teams Administrative power, networking, and The current system values publications,
have championed reproducibility in pre- lobbying within universities, inbred profes- grants, academic titles, and previously accu-
clinical research [56,57] because they sional societies, and academies further mulated power. Researchers at higher ranks
depend on pre-clinical academic investi- distort the picture. This status quo can have more papers and more grants. Howev-
gations accurately pinpointing useful drug easily select for those who excel at gaming er, scholars at the very top of the ladder (e.g.,

PLOS Medicine | www.plosmedicine.org 3 October 2014 | Volume 11 | Issue 10 | e1001747


Table 2. An illustration of different exchange rates for various currencies and wealth items in research.

Different examples of reward systems

Current Change 1 Change 2

CURRENCIES
Publication (per unit) Win 1 No value No value
Replicated publication (per unit) Win 1 Win 2 Win 2
Successfully translated publication (per unit) Win 1 Win 5 Win 5
Refuted publication (per unit) Win 1 Lose 1 Lose 1
Sharing data, protocols, analysis codes (per unit) No value Win 2 Win 2
Contribution to peer-review (per unit) No value Win 2 Win 2
Contribution to education/training (per unit) No value Win 1 Win 1
Grant funding (per one R01) Win 5 Win 5 Lose 5
OTHER WEALTH ITEMS
Assistant professor, title in good university Win 3 Win 3 No value
Associate professor, title in good university Win 10 Win 10 No value
Tenured professor, title in good university Win 20 Win 20 No value
Team leader/director
Per 1 doctoral student/post-doc Win 2 Win 2 Lose 2
Administrative power, networking, lobbying Win up to 200 No value Lose up to 200

doi:10.1371/journal.pmed.1001747.t002

university presidents) have modest, mediocre, The currency values shown in Table 2 are publications is primarily differentiated de-
or weak publication and citation records [63]. for illustrative purposes, to provoke thought pending on their replication and translation
This might be because their lobbying about the sort of rewards that bias the process status. Value is given to sound ideas and
dexterity compensates for their lack of such of scientific work. Such currency values will results that are replicated and reproducible
credentials, and their success comes at the vary across microenvironments and specific [74] rather than publication per se. Further
expense of other worthier candidates who fields and situations. A putative currency value is given to publications that lead to
would bring more intellectual rigor and value value of 1 for a publication unit (e.g., a first- things that work, like effective treatments,
to senior decision making; equally, it could be or senior-authored paper in a highly respect- diagnostic tests, or prognostic tools that
because they excel at the bureaucratic work able journal in the field), 5 for a sizeable demonstrably improve important outcomes
necessary to keep the mind-boggling aca- investigator grant (e.g., an R01 in the US), in clinical trials. Additional value is ob-
demic machine going, and their skills enable and 2 for a post-doctoral fellow means that a tained for sharing and for meaningful
more scientifically gifted colleagues to con- scientist would find equivalent value in participation in peer review and education-
centrate on research. The current system publishing five such papers as first or senior al activities of proven efficacy. A peer
does not reward replicationit often even author as in getting an R01 as a principal reviewer or an editor occasionally may
penalizes people who want to rigorously investigator, or in publishing two such papers contribute the same value as an author.
replicate previous work, and it pushes as in getting a post-doctoral fellow to work for The second example of a proposed
investigators to claim that their work is highly her. Moreover, what constitutes a publication modification shown in Table 2 carries even
novel and significant [64]. Sharing (data, unit may also vary across fields: in fields in greater changes to the reward system. Besides
protocols, analysis codes, etc.) is not incenti- which people publish sparingly, a single the changes adopted in the first example,
vized or requested, with some notable article may be enough to define a publication obtaining grants, awards, or other powers are
exceptions [6567]. With lack of supportive unit, while in fields in which it is typical for considered negatively unless one delivers
resources and with competition (competitors people to put their names in hundreds of more good-quality science in proportion.
will steal my data, my ideas, and eventually papers, often with extreme multi-authorship, Resources and power are seen as opportuni-
my funding), sharing becomes even disin- ten such papers may be needed for an ties, and researchers need to match their
centivized. Other aspects of scientific citi- equivalent publication unit. Inflationary output to the opportunities that they have
zenship, such as high-quality peer review, trends like redundant and salami publication been offeredthe more opportunities, the
are not valued. Peer review can be a [70] and unwarranted multi-authorship have more the expected (replicated and, hopefully,
beneficial process, acting as a safety net made the publication currency lose relative even translated) output. Academic ranks have
and a mechanism for augmenting quality. value over time in many disciplines. Adjust- no value in this model and may even be
It can also be superficial, lead to only ments for multi-authorship are readily feasi- eliminated: researchers simply have to main-
modest improvements of the reviewed ble [71,72]. Knowledge of individual contri- tain a non-negative balance of output versus
work, and allow for the acceptance of butions in each paper would allow even opportunities. In this deliberately provocative
blatantly wrong papers [68,69]. That it is so better allocation of credit [73]. scenario, investigators would be loath to
little valued and rewarded is not calculated In the first example of a proposed obtain grants or become powerful (in the
to encourage its benefits and minimize its modification of the reward system shown current sense), because this would be seen as
harms. in Table 2, the purchasing power of a burden. The potential side effects might be

PLOS Medicine | www.plosmedicine.org 4 October 2014 | Volume 11 | Issue 10 | e1001747


to discourage ambitious grant applications stantial improvements are almost cer- ally, they should be evaluated experi-
and leadership. tainly feasible. The fine-tuning of mentally. The achievements of science
Such trade-offs clarify that when it comes existing policies and more disruptive are amazing, yet the majority of re-
to modifying the structure of scientific careers, and radical interventions should be search effort is currently wasted. Inter-
as when modifying pathophysiology in an considered, but neither presence nor ventions to make science less wasteful
attempt to fight illness, interventions can do absence of revolutionary intent should and more effective could be hugely
harm as well as good. Given the complexity be taken as a reliable surrogate for beneficial to our health, our comfort,
of the situation, interventions should have actual impact. There are many different and our grasp of truth and could help
their actual impacts fairly and reliably scenarios for the evolution of biomedical scientific research more successfully
assessed. research and scientific investigation in pursue its noble goals.
general, each more or less compatible
Moving Forward with seeking truthfulness and human Author Contributions
The extent to which the current well-being. Interventions to change the Wrote the first draft of the manuscript: JPAI.
efficiency of research practices can be current system should not be accepted Wrote the paper: JPAI. ICMJE criteria for
improved is unknown. Given the exist- without proper scrutiny, even when they authorship read and met: JPAI. Conceived the
ing huge inefficiencies, however, sub- are reasonable and well intended. Ide- ideas and concepts discussed: JPAI.

References
1. Boyack KW, Klavans R, Sorensen AA, Ioannidis 18. Panagiotou OA, Willer CJ, Hirschhorn JN, ported meta-analyses and other meta-analyses of
JP (2013) A list of highly influential biomedical Ioannidis JP (2013) The power of meta-analysis the same drugs: systematic review. BMJ 333: 782.
researchers, 19962011. Eur J Clin Invest 43: in genome-wide association studies. Annu Rev 36. Gtzsche PC, Ioannidis JP (2012) Content area
13391365. Genomics Hum Genet 14: 441465. experts as authors: helpful or harmful for
2. Ioannidis JP (2008) Why most discovered true 19. Khoury MJ, Lam TK, Ioannidis JP, Hartge P, systematic reviews and meta-analyses? BMJ 345:
associations are inflated. Epidemiology 19: 640 Spitz MR, et al. (2013) Transforming epidemiol- e7031.
648. ogy for 21st century medicine and public health. 37. Institute of Medicine (2011) Clinical Practice
3. Ioannidis JP (2005) Why most published research Cancer Epidemiol Biomarkers Prev 22: 508516. Guidelines We Can Trust. Washington, D.C.:
findings are false. PLoS Med 2: e124. 20. Bissell M (2013) Reproducibility: The risks of the National Academies Press.
4. Contopoulos-Ioannidis DG, Alexiou GA, Gou- replication drive. Nature 503: 333334. 38. Nuzzo R (2014) Scientific method: statistical
vias TC, Ioannidis JP (2008) Life cycle of 21. Siontis KC, Hernandez-Boussard T, Ioannidis JP errors. Nature 506: 150152.
translational research for medical interventions. (2013) Overlapping meta-analyses on the same 39. Johnson VE (2013) Revised standards for statis-
Science 321: 12981299. topic: survey of published studies. BMJ 347: tical evidence. Proc Natl Acad Sci U S A 110:
5. Macleod MR, Michie S, Roberts I, Dirnagl U, f4501. 1931319317.
Chalmers I, et al. (2014) Biomedical research: 22. Zarin DA, Ide NC, Tse T, Harlan WR, West JC, 40. Young SS, Karr A (2011) Deming, data, and
increasing value, reducing waste. Lancet 383: et al (2007) Issues in the registration of clinical observational studies: a process out of control and
101104. trials. JAMA 297: 21122120. needing fixing. Significance 8: 116120.
6. Nicholson JM and Ioannidis JPA (2012) Research 23. Zarin DA, Tse T, Williams RJ, Califf RM, Ide NC 41. Pashler H, Harris CR (2012) Is the replicability
grants: Conform and be funded. Nature 492: 3436. (2011) The ClinicalTrials.gov results databaseup- crisis overblown? Three arguments examined.
7. Wenneras C, Wold A (1997) Nepotism and date and key issues. N Engl J Med 364: 852860. Persp Psychol Sci 7: 531536.
sexism in peer-review. Nature 387: 341343. 24. Dwan K, Gamble C, Williamson PR, Kirkham 42. Simmons JP, Nelson LD, Simonsohn U (2011)
8. Nickerson RS (1998) Confirmation bias: A JJ, Reporting Bias Group (2013) Systematic False-positive psychology: undisclosed flexibility in
ubiquitous phenomenon in many guises. Rev review of the empirical evidence of study data collection and analysis allows presenting
Gen Psychol 2: 175220. publication bias and outcome reporting bias - anything as significant. Psychol Sci 22: 13591366.
9. Mynatta CR, Dohertya ME, Tweneya RD (1977) an updated review. PLoS ONE 8: e66844. 43. Ioannidis JP, Doucouliagos C (2013) Whats to
Confirmation bias in a simulated research 25. Chan AW, Song F, Vickers A, Jefferson T, know about the credibility of empirical econom-
environment: An experimental study of scientific ics. J Economic Surveys 27: 9971004.
Dickersin K, et al. (2014) Increasing value and
inference. Quarterly J Exp Psychol 29: 8595. reducing waste: addressing inaccessible research. 44. Fanelli D (2010) Positive results increase down the
10. Greenhalgh T, Howick J, Maskrey N, Evidence Hierarchy of the Sciences. PLoS ONE 5: e10068.
Lancet 383: 257266.
Based Medicine Renaissance Group (2014) Evi- 45. Poste G (2012) Biospecimens, biomarkers, and
26. Dal-Re R, Ioannidis JP, Bracken MB, Buffler PA,
dence based medicine: a movement in crisis? BMJ burgeoning data: the imperative for more rigorous
Chan AW, et al. (2014) Making prospective
348: g3725. research standards. Trends Mol Med 18: 717722.
registration of observational research a reality.
11. Stamatakis E1, Weiler R, Ioannidis JP (2013) 46. Landis SC, Amara SG, Asadullah K, Austin CP,
Sci Transl Med 6: 224cm1.
Undue industry influences that distort healthcare Blumenstein R, et al. (2012) A call for transparent
27. Macleod M (2011) Why animal research needs to
research, strategy, expenditure and practice: a reporting to optimize the predictive value of
improve. Nature 477: 511.
review. Eur J Clin Invest 43: 469475. preclinical research. Nature 490: 187191.
12. Chalmers I, Bracken MB, Djulbegovic B, Gar- 28. Stodden V, Guo P, Ma Z (2013) Toward 47. Collins FS, Tabak LA (2014) NIH plans to
attini S, Grant J, et al. (2014) How to increase reproducible computational research: an empiri- enhance reproducibility. Nature 505: 612613.
value and reduce waste when research priorities cal analysis of data and code policy adoption by 48. Simera I, Moher D, Hoey J, Schulz KF, Altman
are set. Lancet 383: 156165. journals. PLoS ONE 8: e67111. DG (2010) A catalogue of reporting guidelines for
13. Rennie D, Flanagin A (2014) Research on peer 29. Donoho DL (2010) An invitation to reproducible health research. Eur J Clin Invest 40: 3553.
review and biomedical publication: furthering the computational research. Biostatistics 11: 385388. 49. Nosek BA, Bar-Anand Y (2012) Scientific utopia:
quest to improve the quality of reporting. JAMA 30. Peng RD (2011) Reproducible research in I. Opening scientific communication. Psycholog-
311: 10191020. computational science. Science 334: 12261227. ical Inquiry 23: 217223.
14. Danthi N, Wu CO, Shi P, Lauer M (2014) 31. Peng RD, Dominici F, Zeger SL (2006) Repro- 50. Glasziou P, Altman DG, Bossuyt P, Boutron I,
Percentile ranking and citation impact of a large ducible epidemiologic research. Am J Epidemiol Clarke M, et al.(2014) Reducing waste from
cohort of national heart, lung, and blood institute- 163: 783789. incomplete or unusable reports of biomedical
funded cardiovascular R01 grants. Circ Res 114: 32. Doshi P, Goodman SN, Ioannidis JP (2013) Raw research. Lancet 383: 267276.
600606. data from clinical trials: within reach? Trends 51. Al-Shahi Salman R, Beller E, Kagan J, Hemminki
15. Ioannidis JP (2011) More time for research: fund Pharmacol Sci 34: 645647. E, Phillips RS, et al. (2014) Increasing value and
people not projects. Nature 477: 529531. 33. Montfortin C (2006) Weight of the evidence or reducing waste in biomedical research regulation
16. NCI-NHGRI Working Group on Replication in wait for the evidence? Protecting underground and management. Lancet 383: 176185.
Association Studies, Chanock SJ, Manolio T, miners from diesel particulate matter. Am J Pub- 52. Khoury MJ1, Gwinn M, Dotson WD, Schully SD
Boehnke M, Boerwinkle E, et al. (2007) Replicat- lic Health 96: 271276. (2012) Knowledge integration at the center of
ing genotype-phenotype associations. Nature 34. Kassirer JP, Angell M (1994) The journals policy genomic medicine. Genet Med 14: 643647.
447(7145): 655660. on cost-effectiveness analyses. N Engl J Med 331: 53. Al-Shahi Salman R, Beller E, Kagan J, Hemminki
17. Ioannidis JP1, Tarone R, McLaughlin JK (2011) 669670. E, Phillips RS, et al. (2014) Increasing value and
The false-positive to false-negative ratio in epidemi- 35. Jrgensen AW, Hilden J, Gtzsche PC (2006) reducing waste in biomedical research regulation
ologic studies. Epidemiology 22: 450456. Cochrane reviews compared with industry sup- and management. Lancet 383: 176185.

PLOS Medicine | www.plosmedicine.org 5 October 2014 | Volume 11 | Issue 10 | e1001747


54. Krumholz SD, Egilman DS, Ross JS (2011) 61. Witten DM, Tibshirani R (2013) Scientific re- 68. Bohannon J (2013) Whos afraid of peer review?
Study of Neurontin: titrate to effect, profile of search in the age of omics: the good, the bad, and Science 342: 6065.
safety (STEPS) trial. A narrative account of a the sloppy. J Am Med Inform Assoc 20: 125127. 69. Hopewell S, Collins GS, Boutron I, Yu LM, Cook
gabapentin seeding trial. Arch Intern Med 171: 62. Ioannidis JP, Khoury MJ (2014) Assessing value in J, et al. (2014) Impact of peer review on reports of
11001107. biomedical research: The PQRST of appraisal and randomised trials published in open peer review
55. Van Noorden R (2014) China tops Europe in reward. JAMA 312: 483484. doi:10.1001/ journals: retrospective before and after study.
R&D intensity. Nature 505: 144145. jama.2014.6932. BMJ 349: g4145.
56. Begley CG, Ellis LM (2012) Drug development: 63. Ioannidis JP (2010) Is there a glass ceiling for 70. Schein M, Paladugu R (2001) Redundant surgical
Raise standards for preclinical cancer research. highly cited scientists at the top of research publications: tip of the iceberg? Surgery 129:
Nature 483: 531533. universities? FASEB J 24: 46354638. 655661.
57. Prinz F, Schlange T, Asadullah K (2011) Believe it 64. Nosek BA, Spies JR, Motyl M (2012) Scientific 71. Hagen NT (2008) Harmonic allocation of au-
or not: how much can we rely on published data Utopia: II. Restructuring incentives and practices thorship credit: source-level correction of biblio-
on potential drug targets? Nat Rev Drug Discov 10: to promote truth over publishability. Persp metric bias assures accurate publication and
712. Psychological Sci 7: 615631. citation analysis. PLoS ONE 3: e4021.
58. Christakis DA, Zimmerman FJ (2013) Rethinking 65. Hayden EC (2014) Cancer-gene data sharing 72. Aziz NA, Rozing MP (2013) Profit (p)-index: the
reanalysis. JAMA 310: 24992500. boosted. Nature 510: 198. degree to which authors profit from co-authors.
59. Young NS, Ioannidis JP, Al-Ubaydli O (2008) 66. Krumholz HM, Gross CP, Blount KL, Ritchie PLoS ONE 8: e59814.
Why current publication practices may distort JD, Hodshon B, et al. (2014) Sea change in 73. Yank V, Rennie D (1999) Disclosure of researcher
science. PLoS Med 5: e201. open science and data sharing: leadership by contributions: a study of original research articles
60. Laine C, Horton R, DeAngelis CD, Drazen JM, industry. Circ Cardiovasc Qual Outcomes 7: in The Lancet. Ann Intern Med 130: 661670.
Frizelle FA, et al. (2007) Clinical trial registration: 499504. 74. Wagenmakers EJ, Forstman BU (2014) Reward-
looking back and moving ahead. JAMA 298: 93 67. Editorial (2014) Data sharing will pay dividends. ing high-power replication research. Cortex 51:
94. Nature 505: 131. 105106.

PLOS Medicine | www.plosmedicine.org 6 October 2014 | Volume 11 | Issue 10 | e1001747

You might also like