Why it's time to publish research “failures”

elsevier.com/connect/scientists-we-want-your-negative-results-too

Publishing bias favors positive results; now there's a movement

to change that

By Lucy Goodchild van Hilten Posted on 5 May 2015

Share story:

After hours of testing, retesting,

Source: xkcd.com
checking and confirming, Dr. Jones
closes her lab book; her experiments
have not disproved the null hypothesis
and they don’t support her new theory.
She leaves her negative results behind,
moving on to work on something new in
the hope that she’ll have a positive
outcome next time.
Meanwhile, what happened in her lab
will stay in her lab. No point in
publishing negative results, after all.

Or is there?

Publication bias affects the body of

scientific knowledge in different ways,
including skewing it towards statistically
significant or “positive” results. This means that the results of thousands of experiments that
fail to confirm the efficacy of a treatment or vaccine – including the outcomes of clinical trials –
fail to see the light of day.

Now, a movement is gaining momentum to counter this bias.

Recently, the World Health Organisation (WHO) announced its position on publishing the
results of clinical trials, calling for the main findings to be submitted for publication in a peer
reviewed journal within 12 months of study completion, and for all previously unreported
results – including negative findings – to be published.

1/7
Meanwhile, new journals like New Negatives in Plant Science, approaches like Registered
Reports and initiatives like the All Trials campaign are trying to balance the literature by
encouraging researchers to publish their negative results.

To understand the challenge they face, it helps to understand why research and publication
are so focused on positive results. The answer is not as simple as you might think.

In his 1963 collection Conjectures and Refutations: The Growth of Scientific Knowledge,
science philosopher Karl Popper wrote: “Refutations have often been regarded as establishing
the failure of a scientist, or at least of his theory. It should be stressed that this is an inductivist
error. Every refutation should be regarded as a great success. … Even if a new theory …
should meet an early death, it should not be forgotten; rather its beauty should be
remembered, and history should record our gratitude to it.”

Perhaps the best-known example of a scientific refutation occurred in 1897 when Albert A.
Michelson and Edward W. Morley published the results of their experiment to detect the
relative motion of matter through the “luminiferous aether.” Their paper “On the relative motion
of the Earth and the luminiferous ether,” in the American Journal of Science, described what
has come to be known as the most famous failed experiment ever. The authors summarized
their disappointing findings:

This new explanation… failed to account for the fact proved by experiment that the aberration
was unchanged when observations were made with a telescope filled with water.

A few years later, Einstein developed his theory of special relativity, motivated in part by
Michelson and Morley’s lack of evidence for the luminiferous ether. This is widely recognized
as the starting point for the “second Scientific Revolution,” and it contributed to Michelson
being awarded the 1907 Nobel Prize in Physics – a rare accolade for a “negative” result. But
why is it such an unusual example?

The power of negative results

In their editorial in Disease Models & Mechanisms, Dr. Natalie Matosin and her colleagues
from the University of Wollongong and the Schizophrenia Research Institute in Sydney,
Australia argue that “Science is, by its nature, a collaborative discipline, and one of the
principal reasons why we should report negative results is so our colleagues do not waste their
time and resources repeating our findings.”

The majority of negative results serve this purpose – to add to our knowledge and function as
a collaboration tool. But some, like the Michelson-Morley experiment, turn out to have huge
significance.

When he looked back over his work to chronicle his career, Prof. Dr. Anthony Cerami,
translational medicine pioneer and the Hermann Boerhaave Visiting Professor of Medicine at
Leiden University, said he was surprised to see how significant his negative results had been:

2/7
 Many of the biggest discoveries of my career were the results of failure of another research
project. ... Failure strikes a negative tone, but it appeared in my personal history that it was an
essential experience on the path to important discoveries.

Prof. Cerami describes one of these failures. He and his colleagues had come up with a
number of compounds with the potential to fight trypanosome infection, a parasite that affects
cattle. They travelled to Kenya to test one particular compound in cattle, which promptly died
within minutes of administration.

The “treatment” soon became the joke of the town, but Prof. Cerami saw the value in the
results, noticing that there must be something other than the parasite causing wasting in the
cattle. He was right, and the protein they eventually isolated was tumor necrosis factor (TNF).
He continued to work on the protein, eventually revealing its important role in inflammatory
diseases including rheumatoid arthritis.

Publication bias: a publishing problem?

Despite their potential, negative results are repeatedly relegated to the lab books, the drawers
and the trash bins. This is not a new phenomenon – research published in Controlled Clinical
Trials in 1987 showed that statistically significant clinical trial results were three times more
likely to be published than those supporting the null hypothesis.

Perhaps surprisingly, the researchers concluded that rather than being a result of editorial
decisions, this was because scientists were failing to write up and submit papers describing
their negative results.

When Matt Shipman, public information officer at North Carolina University, wrote about
publishing negative results, he saw a flurry of social media activity – many researchers thought
it was important, but some weren’t so keen on the idea. Peter Dudek was one of the people
who responded on Twitter: “If I chronicled all my negative results during my studies, the thesis
would have been 20,000 pages instead of 200.”

The academic community has developed a culture that overwhelmingly supports statistically
significant, “positive” results. Researchers themselves strive for these results and rush to
publish them, leaving the “failed” attempts in the dust. How can this culture be shifted towards
valuing negative results?

The role of publishing in changing perceptions

Publishing is a major factor in academic success, so the publishing industry is in a position to
change this academic culture and publication bias. In recent years, publishers and journals
have been encouraging more submissions of negative results, particularly for clinical trials.

This is an area in which publication bias is being tackled from different angles. Launched by
Sense About Science in 2013, the All Trials campaign calls for “all past and present clinical
trials to be registered and their full methods and summary results reported.” According to the
3/7
campaign website, “Millions of volunteers have participated in clinical trials to help find out
more about the effects of treatments on disease, yet that important ethical principle about
reporting has been widely ignored.”

The WHO’s announcement is a big win for the All Trials campaign, whose response explains
the importance of publishing all results:

This is a strong statement from the WHO. It has been welcomed by patients, doctors and
researchers worldwide. The best available evidence shows that around half of all the clinical trials
that have ever been carried out have never reported results. … It means that information about the
medicines we use every day is at risk of being lost forever.

To promote and enable the ongoing publication of new clinical trials data, many journals are
encouraging submissions describing negative results. One journal published by Elsevier is
taking it a step further: Cortex has launched a section called Registered Reports, which
commits to publishing the outcome of a trial before it has been completed.

“Our aim with Registered Reports is to enhance the transparency and reproducibility of
science by reviewing study protocols before experiments are conducted,” said Prof. Chris
Chambers, Registered Reports Editor of Cortex and one of the concept’s founders, in the
launch announcement. “If we think the protocol has merit we will commit, in advance, to
publishing the outcomes. Armed with this provisional acceptance of their work, authors can
perform the research safe in the knowledge that the results themselves will not determine the
article's publication.”

A plant science journal dedicated to negative results

It’s not just the clinical trials literature that is missing negative results – any experimental
discipline that works on the basis of a hypothesis runs the risk of this bias.

“All these results exist, but they are not

Emma Granqvist, PhD
published,” said Dr. Emma Granqvist,
Publisher of Elsevier’s New Negatives in
Plant Science, who was a researcher in
molecular plant science. “Researchers are
saying that if they submit these results,
journal editors will reject their manuscripts
because they’re not interesting or high
impact.”
New Negatives in Plant Science aims to
address this by publishing only negative results. “Some people disagree with having a journal
purely for negative results, saying it’s the wrong approach – that all journals should publish
these results,” Dr. Granqvist said. “Ideally, this would be the case, but it’s not realistic. A shift
like this takes time – it’s about changing attitudes. This is the first step, a spotlight on the fact
that negative results are largely missing from the literature.”
4/7
The language around results has an effect, too, Dr, Granqvist added. “By calling these results
‘negative,’ they are immediately less appealing – especially to the researchers themselves,
who usually neglect to publish them.

“This isn’t an issue in every discipline; in mathematics, for example, counter-proofs and proofs
by contradiction are all considered logical and are not as readily labeled as a failure,” she
explained. “Perhaps we can take some inspiration from fields like this and start to value all
results more equally.”

Related stories
"Why science needs to publish negative results" (Editors Update, March 2015)
"AllTrials campaign and petition support greater clinical trial transparency" (Elsevier
Connect, June 2014)

Positively negative journals

An increasing number of journals proactively publish negative results. Here are a few of them:

New Negatives in Plant Science. Publishes hypothesis-driven, scientifically sound

studies that describe unexpected, controversial, dissenting, and/or null (negative) results
in basic plant sciences.
Journal of Negative Results. Provides an online medium to publish peer-reviewed, sound
scientific work in ecology and evolutionary biology that is scientifically rigorous but does
not rely upon arbitrary significance thresholds to support conclusions.
Journal of Negative Results in BioMedicine. Provides a platform for the publication and
discussion of unexpected, controversial, provocative and/or negative results in the
context of current tenets.
Journal of Pharmaceutical Negative Results. Publishes original, innovative and novel
research articles resulting in negative results; publishes theoretical and empirical papers
that report the negative findings and research failures in pharmaceutical field.
The All Results journals. The main objective of the four journals – Chem, Nano, Biol and
Phys – is to recover and publish negative results, valuable pieces of information in
science.

Elsevier Connect Contributor

After a few accidents, Lucy Goodchild van
Hilten discovered that she’s a much better
writer than a scientist. Following an MSc
in the History of Science, Medicine and
Technology at Imperial College London,
she became Assistant Editor of
Microbiology Today. A stint in the press
5/7
office at Imperial saw her stories on the Lucy Goodchild van
front pages, and she moved to Hilten
Amsterdam to work at Elsevier as Senior
Marketing Communications Manager for
Life Sciences.
She’s now a freelance writer at Tell Lucy.
Tweet her @LucyGoodchild.

Share story:

Related Stories
AllTrials campaign and petition support greater clinical trial
transparency
By Síle Lane | Posted on 23 Jun 2014

Elsevier supports the growing movement to require clinical trials to

be registered and their results reported

Solutions

Researchers

About Elsevier

How can we help?

Select location/language
Global - English

Copyright © 2018 Elsevier, except certain content provided by third parties

Cookies are used by this site. To decline or learn more, visit ourCookies page.

Terms and Conditions Privacy Policy Sitemap

6/7
7/7