Professional Documents
Culture Documents
Effective Interventions
We need effective interventions to im- analysis plan: the error rate would decrease reviewers typically have only modest CVs
prove the credibility and efficiency of to zero simply because no research would and most of the top influential scientists dont
scientific investigation. Some risk factors ever be done. Thus, whatever solutions are review grant applications and dont get
for false results are immutable, like small proposed should be pragmatic, applicable, funded by government funds, even in the
effect sizes, but others are modifiable. We and ideally, amenable to reliable testing of United States [6], which arguably has the
must diminish biases, conflicts of interest, their performance. strongest scientific impact at the moment
and fragmentation of efforts in favor of Currently, major decisions about how than any other country (e.g., in cumulative
unbiased, transparent, collaborative re- research is done may too often be based on citations). Non-meritocratic practices, includ-
search with greater standardization. How- convention and inertia rather than being ing nepotism, sexism, and unwarranted
ever, we should also consider the possibility highly imaginative or evidence-based [515]. conservatism, are probably widespread [7].
that interventions aimed at improving For example, there is evidence that grant Allegiance and confirmation biases are
scientific efficiency may cause collateral
damage or themselves wastefully consume
resources. To give an extreme example, Citation: Ioannidis JPA (2014) How to Make More Published Research True. PLoS Med 11(10): e1001747.
doi:10.1371/journal.pmed.1001747
one could easily eliminate all false positives
simply by discarding all studies with even Published October 21, 2014
minimal bias, by making the research Copyright: 2014 John P. A. Ioannidis. This is an open-access article distributed under the terms of the
questions so bland that nobody cares about Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any
(or has a conflict with) the results, and by medium, provided the original author and source are credited.
waiting for all scientists in each field to join Funding: The Meta-Research Innovation Center at Stanford is funded by the Laura and John Arnold
forces on a single standardized protocol and Foundation. The funders had no role in the decision to publish or in the preparation of the manuscript.
Competing Interests: JPAI is a member of the Editorial Board of PLOS Medicine. The author has declared that
no competing interests exist.
Essays are opinion pieces on a topic of broad
interest to a general medical audience. * E-mail: jioannid@stanford.edu
Provenance: Commissioned; externally peer reviewed
doi:10.1371/journal.pmed.1001747.t001
without making real progress, if more targets. Conversely, the industry is defen- the system, producing prolifically mediocre
publications and more grants are all that sive about data sharing from clinical trials and/or irreproducible research; controlling
matters. If science is supported primarily [30], which occurs at a point in the peer review at journals and study sections;
by private investors who desire patents product development when re-analyses enjoying sterile bureaucracy, lobbying, and
and profit, this may lead to expedited may correctly or incorrectly [58] invali- maneuvering; and promoting those who
translation and discoveries that work (or date evidence supporting drugs in which it think and act in the same way.
seem to work) but also barriers against has already invested heavily. There are also opportunities in grasping
transparency and sharing of information. Dynamics between different stakeholders the importance of the key currencies. For
Corporate influence may subvert science are complex. Moreover, sometimes the example, registration of clinical trials
for the purposes of advertising, with same person may wear many stakeholder worked because all major journals adopted
papers in influential journals, prestigious hats; e.g., an academic researcher may also it as prerequisite for publication [60], a
society meetings, and a professorate sys- be journal editor, spin-off company owner, major reference currency in the reward
tem of opinion leaders becoming branches professional society officer, government chain. Conversely, interesting post-publi-
of their marketing department [11,54]. advisor, and/or beneficiary of the industry. cation review efforts such as PubMed
The geography of scientific production Commons [61] have so far not fulfilled
changes rapidly; e.g., soon there will be Research Currencies their potential as progressive vehicles for
more English language papers from China evaluating research, probably because
than from Europe and the US [55]. Publications and grants are key curren- there is currently no reward for such
Research efforts are embedded in wider cies in science (Table 2). They purchase post-publication peer review.
societies, which have provided scientific academic goods such as promotion and
developments that differ according to time other power. Academic titles and power Modifying the Reward System
period and location. What can be done to add further to the wealth of their
enhance the capacity of science to flourish possessor. The exact exchange rate of The reward system may be systematically
and to assess and promote this capacity currencies and the price of academic goods modified [62]. Modifying interventions may
across cultures that may vary in attitudes [59] may vary across institutional microen- be anywhere from fine-tuning to disruptive.
toward skepticism, inquisitiveness, and vironments, scientific disciplines and cir- Table 2 compares the status quo (first
contrarian reasoning? Different stakehold- cumstances, and are also affected by each column) against two potential modifications
ers have their own preferences about when microenvironments fairness or unfairness of the reward system, with Change 2 being
reproducibility should be promoted or (e.g., nepotism, cronyism, or corruption). more prominent than Change 1.
shunned. Pharmaceutical industry teams Administrative power, networking, and The current system values publications,
have championed reproducibility in pre- lobbying within universities, inbred profes- grants, academic titles, and previously accu-
clinical research [56,57] because they sional societies, and academies further mulated power. Researchers at higher ranks
depend on pre-clinical academic investi- distort the picture. This status quo can have more papers and more grants. Howev-
gations accurately pinpointing useful drug easily select for those who excel at gaming er, scholars at the very top of the ladder (e.g.,
CURRENCIES
Publication (per unit) Win 1 No value No value
Replicated publication (per unit) Win 1 Win 2 Win 2
Successfully translated publication (per unit) Win 1 Win 5 Win 5
Refuted publication (per unit) Win 1 Lose 1 Lose 1
Sharing data, protocols, analysis codes (per unit) No value Win 2 Win 2
Contribution to peer-review (per unit) No value Win 2 Win 2
Contribution to education/training (per unit) No value Win 1 Win 1
Grant funding (per one R01) Win 5 Win 5 Lose 5
OTHER WEALTH ITEMS
Assistant professor, title in good university Win 3 Win 3 No value
Associate professor, title in good university Win 10 Win 10 No value
Tenured professor, title in good university Win 20 Win 20 No value
Team leader/director
Per 1 doctoral student/post-doc Win 2 Win 2 Lose 2
Administrative power, networking, lobbying Win up to 200 No value Lose up to 200
doi:10.1371/journal.pmed.1001747.t002
university presidents) have modest, mediocre, The currency values shown in Table 2 are publications is primarily differentiated de-
or weak publication and citation records [63]. for illustrative purposes, to provoke thought pending on their replication and translation
This might be because their lobbying about the sort of rewards that bias the process status. Value is given to sound ideas and
dexterity compensates for their lack of such of scientific work. Such currency values will results that are replicated and reproducible
credentials, and their success comes at the vary across microenvironments and specific [74] rather than publication per se. Further
expense of other worthier candidates who fields and situations. A putative currency value is given to publications that lead to
would bring more intellectual rigor and value value of 1 for a publication unit (e.g., a first- things that work, like effective treatments,
to senior decision making; equally, it could be or senior-authored paper in a highly respect- diagnostic tests, or prognostic tools that
because they excel at the bureaucratic work able journal in the field), 5 for a sizeable demonstrably improve important outcomes
necessary to keep the mind-boggling aca- investigator grant (e.g., an R01 in the US), in clinical trials. Additional value is ob-
demic machine going, and their skills enable and 2 for a post-doctoral fellow means that a tained for sharing and for meaningful
more scientifically gifted colleagues to con- scientist would find equivalent value in participation in peer review and education-
centrate on research. The current system publishing five such papers as first or senior al activities of proven efficacy. A peer
does not reward replicationit often even author as in getting an R01 as a principal reviewer or an editor occasionally may
penalizes people who want to rigorously investigator, or in publishing two such papers contribute the same value as an author.
replicate previous work, and it pushes as in getting a post-doctoral fellow to work for The second example of a proposed
investigators to claim that their work is highly her. Moreover, what constitutes a publication modification shown in Table 2 carries even
novel and significant [64]. Sharing (data, unit may also vary across fields: in fields in greater changes to the reward system. Besides
protocols, analysis codes, etc.) is not incenti- which people publish sparingly, a single the changes adopted in the first example,
vized or requested, with some notable article may be enough to define a publication obtaining grants, awards, or other powers are
exceptions [6567]. With lack of supportive unit, while in fields in which it is typical for considered negatively unless one delivers
resources and with competition (competitors people to put their names in hundreds of more good-quality science in proportion.
will steal my data, my ideas, and eventually papers, often with extreme multi-authorship, Resources and power are seen as opportuni-
my funding), sharing becomes even disin- ten such papers may be needed for an ties, and researchers need to match their
centivized. Other aspects of scientific citi- equivalent publication unit. Inflationary output to the opportunities that they have
zenship, such as high-quality peer review, trends like redundant and salami publication been offeredthe more opportunities, the
are not valued. Peer review can be a [70] and unwarranted multi-authorship have more the expected (replicated and, hopefully,
beneficial process, acting as a safety net made the publication currency lose relative even translated) output. Academic ranks have
and a mechanism for augmenting quality. value over time in many disciplines. Adjust- no value in this model and may even be
It can also be superficial, lead to only ments for multi-authorship are readily feasi- eliminated: researchers simply have to main-
modest improvements of the reviewed ble [71,72]. Knowledge of individual contri- tain a non-negative balance of output versus
work, and allow for the acceptance of butions in each paper would allow even opportunities. In this deliberately provocative
blatantly wrong papers [68,69]. That it is so better allocation of credit [73]. scenario, investigators would be loath to
little valued and rewarded is not calculated In the first example of a proposed obtain grants or become powerful (in the
to encourage its benefits and minimize its modification of the reward system shown current sense), because this would be seen as
harms. in Table 2, the purchasing power of a burden. The potential side effects might be
References
1. Boyack KW, Klavans R, Sorensen AA, Ioannidis 18. Panagiotou OA, Willer CJ, Hirschhorn JN, ported meta-analyses and other meta-analyses of
JP (2013) A list of highly influential biomedical Ioannidis JP (2013) The power of meta-analysis the same drugs: systematic review. BMJ 333: 782.
researchers, 19962011. Eur J Clin Invest 43: in genome-wide association studies. Annu Rev 36. Gtzsche PC, Ioannidis JP (2012) Content area
13391365. Genomics Hum Genet 14: 441465. experts as authors: helpful or harmful for
2. Ioannidis JP (2008) Why most discovered true 19. Khoury MJ, Lam TK, Ioannidis JP, Hartge P, systematic reviews and meta-analyses? BMJ 345:
associations are inflated. Epidemiology 19: 640 Spitz MR, et al. (2013) Transforming epidemiol- e7031.
648. ogy for 21st century medicine and public health. 37. Institute of Medicine (2011) Clinical Practice
3. Ioannidis JP (2005) Why most published research Cancer Epidemiol Biomarkers Prev 22: 508516. Guidelines We Can Trust. Washington, D.C.:
findings are false. PLoS Med 2: e124. 20. Bissell M (2013) Reproducibility: The risks of the National Academies Press.
4. Contopoulos-Ioannidis DG, Alexiou GA, Gou- replication drive. Nature 503: 333334. 38. Nuzzo R (2014) Scientific method: statistical
vias TC, Ioannidis JP (2008) Life cycle of 21. Siontis KC, Hernandez-Boussard T, Ioannidis JP errors. Nature 506: 150152.
translational research for medical interventions. (2013) Overlapping meta-analyses on the same 39. Johnson VE (2013) Revised standards for statis-
Science 321: 12981299. topic: survey of published studies. BMJ 347: tical evidence. Proc Natl Acad Sci U S A 110:
5. Macleod MR, Michie S, Roberts I, Dirnagl U, f4501. 1931319317.
Chalmers I, et al. (2014) Biomedical research: 22. Zarin DA, Ide NC, Tse T, Harlan WR, West JC, 40. Young SS, Karr A (2011) Deming, data, and
increasing value, reducing waste. Lancet 383: et al (2007) Issues in the registration of clinical observational studies: a process out of control and
101104. trials. JAMA 297: 21122120. needing fixing. Significance 8: 116120.
6. Nicholson JM and Ioannidis JPA (2012) Research 23. Zarin DA, Tse T, Williams RJ, Califf RM, Ide NC 41. Pashler H, Harris CR (2012) Is the replicability
grants: Conform and be funded. Nature 492: 3436. (2011) The ClinicalTrials.gov results databaseup- crisis overblown? Three arguments examined.
7. Wenneras C, Wold A (1997) Nepotism and date and key issues. N Engl J Med 364: 852860. Persp Psychol Sci 7: 531536.
sexism in peer-review. Nature 387: 341343. 24. Dwan K, Gamble C, Williamson PR, Kirkham 42. Simmons JP, Nelson LD, Simonsohn U (2011)
8. Nickerson RS (1998) Confirmation bias: A JJ, Reporting Bias Group (2013) Systematic False-positive psychology: undisclosed flexibility in
ubiquitous phenomenon in many guises. Rev review of the empirical evidence of study data collection and analysis allows presenting
Gen Psychol 2: 175220. publication bias and outcome reporting bias - anything as significant. Psychol Sci 22: 13591366.
9. Mynatta CR, Dohertya ME, Tweneya RD (1977) an updated review. PLoS ONE 8: e66844. 43. Ioannidis JP, Doucouliagos C (2013) Whats to
Confirmation bias in a simulated research 25. Chan AW, Song F, Vickers A, Jefferson T, know about the credibility of empirical econom-
environment: An experimental study of scientific ics. J Economic Surveys 27: 9971004.
Dickersin K, et al. (2014) Increasing value and
inference. Quarterly J Exp Psychol 29: 8595. reducing waste: addressing inaccessible research. 44. Fanelli D (2010) Positive results increase down the
10. Greenhalgh T, Howick J, Maskrey N, Evidence Hierarchy of the Sciences. PLoS ONE 5: e10068.
Lancet 383: 257266.
Based Medicine Renaissance Group (2014) Evi- 45. Poste G (2012) Biospecimens, biomarkers, and
26. Dal-Re R, Ioannidis JP, Bracken MB, Buffler PA,
dence based medicine: a movement in crisis? BMJ burgeoning data: the imperative for more rigorous
Chan AW, et al. (2014) Making prospective
348: g3725. research standards. Trends Mol Med 18: 717722.
registration of observational research a reality.
11. Stamatakis E1, Weiler R, Ioannidis JP (2013) 46. Landis SC, Amara SG, Asadullah K, Austin CP,
Sci Transl Med 6: 224cm1.
Undue industry influences that distort healthcare Blumenstein R, et al. (2012) A call for transparent
27. Macleod M (2011) Why animal research needs to
research, strategy, expenditure and practice: a reporting to optimize the predictive value of
improve. Nature 477: 511.
review. Eur J Clin Invest 43: 469475. preclinical research. Nature 490: 187191.
12. Chalmers I, Bracken MB, Djulbegovic B, Gar- 28. Stodden V, Guo P, Ma Z (2013) Toward 47. Collins FS, Tabak LA (2014) NIH plans to
attini S, Grant J, et al. (2014) How to increase reproducible computational research: an empiri- enhance reproducibility. Nature 505: 612613.
value and reduce waste when research priorities cal analysis of data and code policy adoption by 48. Simera I, Moher D, Hoey J, Schulz KF, Altman
are set. Lancet 383: 156165. journals. PLoS ONE 8: e67111. DG (2010) A catalogue of reporting guidelines for
13. Rennie D, Flanagin A (2014) Research on peer 29. Donoho DL (2010) An invitation to reproducible health research. Eur J Clin Invest 40: 3553.
review and biomedical publication: furthering the computational research. Biostatistics 11: 385388. 49. Nosek BA, Bar-Anand Y (2012) Scientific utopia:
quest to improve the quality of reporting. JAMA 30. Peng RD (2011) Reproducible research in I. Opening scientific communication. Psycholog-
311: 10191020. computational science. Science 334: 12261227. ical Inquiry 23: 217223.
14. Danthi N, Wu CO, Shi P, Lauer M (2014) 31. Peng RD, Dominici F, Zeger SL (2006) Repro- 50. Glasziou P, Altman DG, Bossuyt P, Boutron I,
Percentile ranking and citation impact of a large ducible epidemiologic research. Am J Epidemiol Clarke M, et al.(2014) Reducing waste from
cohort of national heart, lung, and blood institute- 163: 783789. incomplete or unusable reports of biomedical
funded cardiovascular R01 grants. Circ Res 114: 32. Doshi P, Goodman SN, Ioannidis JP (2013) Raw research. Lancet 383: 267276.
600606. data from clinical trials: within reach? Trends 51. Al-Shahi Salman R, Beller E, Kagan J, Hemminki
15. Ioannidis JP (2011) More time for research: fund Pharmacol Sci 34: 645647. E, Phillips RS, et al. (2014) Increasing value and
people not projects. Nature 477: 529531. 33. Montfortin C (2006) Weight of the evidence or reducing waste in biomedical research regulation
16. NCI-NHGRI Working Group on Replication in wait for the evidence? Protecting underground and management. Lancet 383: 176185.
Association Studies, Chanock SJ, Manolio T, miners from diesel particulate matter. Am J Pub- 52. Khoury MJ1, Gwinn M, Dotson WD, Schully SD
Boehnke M, Boerwinkle E, et al. (2007) Replicat- lic Health 96: 271276. (2012) Knowledge integration at the center of
ing genotype-phenotype associations. Nature 34. Kassirer JP, Angell M (1994) The journals policy genomic medicine. Genet Med 14: 643647.
447(7145): 655660. on cost-effectiveness analyses. N Engl J Med 331: 53. Al-Shahi Salman R, Beller E, Kagan J, Hemminki
17. Ioannidis JP1, Tarone R, McLaughlin JK (2011) 669670. E, Phillips RS, et al. (2014) Increasing value and
The false-positive to false-negative ratio in epidemi- 35. Jrgensen AW, Hilden J, Gtzsche PC (2006) reducing waste in biomedical research regulation
ologic studies. Epidemiology 22: 450456. Cochrane reviews compared with industry sup- and management. Lancet 383: 176185.