INSIGHTS
B O OKS et al .
REPRODUCIBILITY
On rigor and
replication
A journalist shines a harsh
spotlight on biomedicine’s
reproducibility crisis
By Leonard P. Freedman
I
n Rigor Mortis: How Sloppy Science Cre-
ates Worthless Cures, Crushes Hopes, and
Wastes Billions, a sensationalistic title
belies a carefully crafted book about data
reproducibility and scientific rigor in
biomedical research. Although the book
breaks no new ground, at a time when the
so-called “irreproducibility crisis” has stoked
passionate debate among scientists, bewil-
dered the public, and even launched new
disciplines (e.g., metaresearch), an accessible
overview of the problem is most welcome.
In a folksy style reminiscent of his re-
porting for National Public Radio, Richard
Harris begins most chapters with a human-
interest vignette, then ties the anecdote to
a pervasive problem in science. In a chap-
ter entitled “A Bucket of Cold Water,” for
example, the reader is introduced to Tom
Murphy, a 56-year-old father of three who
participated in a hopeful ALS clinical trial
that ultimately failed. Scores of human
clinical trials conducted with promising
ALS candidate drugs have ended unsuccess-
fully, Harris goes on to explain, invariably
because of flawed preclinical study designs
and an overreliance on poor mouse models
for the disease.
Harris shows both sides of the reproducibil-
ity debate, noting that many eminent mem-
bers of the research establishment would like
to see this new practice of airing the scientific
community’s dirty laundry quietly disappear.
He describes how, for example, in the after-
math of their 2012 paper demonstrating that
only 6 of 53 landmark studies in cancer biol-
ogy could be reproduced (1), Glenn Begley and
Lee Ellis were immediately attacked by some
in the biomedical research aristocracy for
their “naÔveté,” their “lack of competence” and
their “disservice” to the scientific community.
Those of us who have succeeded as aca-
The reviewer is at the Global Biological Standards Institute,
Washington, DC 20036, USA. Email: lfreedman@gbsi.org
34 7 APRIL 2017 • VOL 356 ISSUE 6333
Tight research budgets and flawed preclinical studies have compromised efforts to cure ALS, writes Harris.
demic researchers have done so in part by em-
bracing incentives that are embedded in our
academic culture—pitching ambitious pro-
posals to funding agencies and publishing as
often as we can in journals with high impact
factors, hoping to put forward a winning pack-
age to hiring or tenure committees.
Although “publish or perish” was the old
dogma, “show me the money” is the reality fac-
ing today’s academic scientists. “What we have
is a Darwinian winnowing,” Henry Bourne, a
professor emeritus in the Department of Cel-
lular and Molecular Biology at the University
of California San Francisco, tells Harris. “We
take them [researchers] if NIH gives them a
grant. And we don’t if they don’t.”
This so-called meritocracy, and its inher-
ent hypercompetitiveness, has likely fueled
the irreproducibility fire more than any other
factor, especially in today’s chronic state of
insufficient science funding. In a chapter
cleverly called “A Broken Culture,” Harris de-
scribes a cutthroat environment that tempts
many researchers to cherry-pick their data for
publication. Or, worse still, drives some to fab-
ricate the perfect result.
Although Nobel laureates and National
Academy members have weighed in soberly
with well-intentioned commentaries and sug-
gestions for new approaches, the status quo
for success in research has not fundamentally
changed. Ultimately, Harris argues, for im-
pactful reproducibility initiatives to be volun-
tarily embraced by the research community,
the perverse incentives driving current defini-
tions of academic success must be reassessed.
Gone are the good old free-wheeling days
of the gene jockeys, eyeballing the relative in-
tensities of bands on a gel or dots on a plate
(guilty as charged). More data generated
means more evaluation and quantitation. As
biomedical research goes big, lessons learned
from other disciplines point to the need for
a more deliberate effort when it comes to
designing, executing, and analyzing experi-
Published by AAAS
Downloaded from http://science.sciencemag.org/ on April 6, 2017
Rigor Mortis
How Sloppy Science Creates
Worthless Cures, Crushes
Hopes, and Wastes Billions
Richard Harris
Basic Books, 2017. 288 pp.
ments. One approach, commonplace in vir-
tually every industry and discipline except
biology, is the broad application of standards
and best practices. Validated cell lines and an-
tibodies, two examples that Harris describes
in detail in a chapter entitled “Trusting the
Untrustworthy,” would have an immediate
positive impact on reproducibility from ex-
periment to experiment and from lab to lab.
Harris devotes the book’s final chapters
to other important responses to irreproduc-
ibility, such as requiring open access of data
and methods, as well as offering new train-
ing modalities for students that emphasize
study design and statistics. However, he
insightfully observes that many newly in-
troduced initiatives and guidelines, notably
by major publishers and funders, are “fre-
quently ignored” by the research commu-
nity and will continue to be until there are
meaningful incentives in place.
Rigor Mortis effectively illustrates what
can happen when a convergence of social,
cultural, and scientific forces, as well as ba-
sic human motivation, conspires to create a
PHOTO: ITAR-TASS PHOTO AGENCY/ALAMY STOCK PHOTO
real crisis of confidence in the research pro-
cess. The book’s ominous title and subtitle
notwithstanding, Harris shines a glimmer of
light on a community beginning to awaken to
its predicament, revealing how many of the
same thoughtful, creative people who were
attracted to a career in science may also be
instrumental in fixing it. j
REFERENCE
1. C. G. Begley, L. M. Ellis, Nature 483, 531 (2012).
10.1126/science.aam8039
sciencemag.org SCIENCE
 On rigor and replication
Leonard P. Freedman (April 6, 2017)
Science 356 (6333), 34. [doi: 10.1126/science.aam8039]

Editor's Summary

Downloaded from http://science.sciencemag.org/ on April 6, 2017

This copy is for your personal, non-commercial use only.

Article Tools Visit the online version of this article to access the personalization and
article tools:
http://science.sciencemag.org/content/356/6333/34

Permissions Obtain information about reproducing this article:

http://www.sciencemag.org/about/permissions.dtl

Science (print ISSN 0036-8075; online ISSN 1095-9203) is published weekly, except the last week
in December, by the American Association for the Advancement of Science, 1200 New York
Avenue NW, Washington, DC 20005. Copyright 2016 by the American Association for the
Advancement of Science; all rights reserved. The title Science is a registered trademark of AAAS.