Science will stick with peer review

ANALYSIS
By Paul Rincon and Jonathan Amos
BBC News science reporters

It was late on a Wednesday night in February 2004 that Dr

Hwang Woo-suk and his entourage swept into the Grand Hyatt
Hotel in Seattle to meet the BBC.
Science will examine itself in
The South Korean scientists had just flown in to the US city to
announce an astonishing breakthrough: they had cloned 30 human the light of the Hwang affair
embryos and managed to extract stem cells from one of them.

Their "advance" was set to take us into a new era; a new type of medicine beckoned that had the potential
to roll back the degenerative processes that rack the body as it ages.

At that stage, Hwang was largely unknown; certainly those outside his field or in the general public would
not have heard of him.

But as he sat there in the office patiently repeating his comments for
all of the BBC's TV, radio and online outlets, it was easy to be [Peer review] is a bit like
comfortable with this man; his clarity, his purpose and his passion democracy: it's a lousy
were all impressive - and persuasive. system but it's the best one
we have
We now know we were interviewing a fraud.
Liz Wager, publications
Not only was that landmark 2004 work highly dubious, Hwang's consultant
research published a year later on cloned personalised stem cell
lines was also built on fabricated data.

"This conduct cannot but be seen as an act that attempted to fool the whole scientific community and the
public," said Professor Chung Myung-hee, head of the Seoul National University (SNU) panel
investigating the affair.

Getting away with it

We all have so many questions. Why did he do it? How did he expect to get away with it? And, was there
anything that could have, or should have, been done to pick up the great con much earlier.

The last question, quite naturally, is being directed at Science magazine, which published the 2004 and
2005 manuscripts; and at the process of "peer review" which it, and
other leading journals, use to check papers before they publish them.

That process is supposed to ensure that any study's methodology is

sound and that interpretation of data does not go beyond what can be
reasonably justified.

Science magazine is continuing its own internal review but its Editor
in Chief, Dr Donald Kennedy, is doubtful there are any systematic
flaws in the peer review process that made the Korean fraud any
easier. The Korean's Seattle
announcement on cloning was
big news
"We've had a couple of papers in Science in the last four or five years that plainly involved scientific
misconduct, ultimately discovered on investigation and publicised," he told reporters last month.

"We were asked questions about those papers and I said, editorially, each time that there is no way that the
peer review system can be made proof against misrepresentation of data."

It has been suggested that on particularly contentious or high profile research, reviewers should do more
than just read through manuscripts to check they add up; could materials also be submitted for independent
analysis and verification?

Extra checks?

"You can only assess the science in terms of what is in the paper, and the data looked accurate," Dr Stephen
Minger, director of King's Stem Cell Biology Laboratory, UK, told the BBC News website.

"I know at least one person who reviewed the original paper, and they were convinced the data was real;
and short of going to the lab, and physically inspecting the data yourself, and saying 'I want to see the cells,
I want to do the DNA analyses myself' - you just cannot do that physically."

On that, Dr Kennedy is in agreement. Insisting replication of data by an independent group be part of the
referring process would be a nightmare, he believes.

"I think that to install a procedure by which replication by a third

party was required for acceptance of a manuscript would impose We think it would be
simply an enormous load on our readership and the scientific harder for people to
community," he said.
plagiarise work once you can
do extensive word searches
Colin McGuckin, professor of regenerative medicine at Newcastle
University, UK, commented: "We don't want to become and access more material
bureaucratic, because that will hold up everything. But sometimes free on the internet
as a scientist, your biggest critics will be in the organisation you
work at. They're the ones you should look to for advice." Robert Terry, Wellcome Trust
The Newcastle stem cell expert said there should be internal review procedures within universities and
scientific institutions to evaluate research before it is submitted to academic journals for peer review.

Freelance publications consultant Liz Wager agrees there needs to be improved governance of scientific
research by universities and institutions.

"This kind of thing has to be policed at the departmental and institutional level; they actually know what's
going on. They need to create an environment in which a whistleblower can feel safe."

She added: "Once the paper reaches a journal, it can be one that is on another continent. The journal can't
investigate, speak to lots of other people involved or look at the lab notes.

"Peer review is good at calming down over-optimistic claims and improving the presentation, but the
evidence shows it is really bad at picking out very major fraud."

Wider scrutiny

Of course, when fraudulent scientific papers enter the larger literature, there is always the possibility that
they will be exposed when other researchers try to replicate, or repeat, the findings themselves.
But Professor McGuckin says the community should not be satisfied
to leave it to this: "When you have something that is such a
momentous breakthrough, we should find someone else in the world
to do it, too. And then we can be sure that it works," he said.

It is a point recognised by Dr Elaine Ostrander. The canine scientist

from the US National Human Genome Research Institute was asked
by the journal Nature to check the validity of Hwang's cloned dog,
Snuppy. Snuppy (right) has been
confirmed as the first dog
She believes certain papers, because of their importance, may in
clone
future be subjected to a far more vigorous form of peer review.

"In hindsight, you would say, 'yes' - this sort of validation should be done on this kind of a scientific
breakthrough," she told the BBC News website, "especially in the case of the dog where there are well-
developed and recognised resources out there such as microsatellite markers for paternity testing."

Some scientists say that one of the benefits of the "open access" business model for journals - where
scientific papers are free for all to read in a web-based database - could be beneficial for picking up
plagiarism and possibly other forms of misconduct.

A great many scientific journals are subscription-based, so that readers have to pay to view research.

"We think it would be harder for people to plagiarise work once you can do extensive word searches and
access more material free on the internet. You'll be able to spot where someone has lifted their work much
more easily," says Robert Terry, senior policy adviser at the UK medical charity, the Wellcome Trust.

Oversight body

When the US government set up the Office of Research Integrity (ORI) in 1992, to investigate cases of
misconduct in federally funded research, it saw an initial rise in
allegations of malfeasance.

This probably was not down to a real increase; it instead suggested

that some significant proportion of scientific malpractice goes
undetected. And some scientists think more oversight in other
countries would be beneficial.

"Governments giving licences to work on stem cells could perhaps

be better at monitoring how the work is coming out from their own
countries before it goes international," says Colin McGuckin.

But, as Donald Kennedy suggests, there appear to be few options for

fundamental changes to the peer review process that would make it
harder for fraudulent papers to enter the scientific literature.

"Thousands of papers are reviewed every week, and peer review

works usually," says Ms Wager.
The fall from grace was
"There aren't any alternative models to peer review. It's a bit like inevitable
democracy: it's a lousy system but it's the best one we have.
"There are always cases that seem to get through, especially in areas where everyone wants the results to be
true."

The leading British geneticist and author Professor Steve Jones commented: "The odd thing about this is
that this was such a high profile claim that people were bound to try to repeat his work sooner or later and
would not be able to do it; so he would be found out.

"Let's remember that his cloned dog seems to be real, so he's got a lot of scientific credibility, and you can't
blame the scientific community for having taken the rest of his results on face value.

"Maybe they should have been feeling more cynical, but again that terrible illness called optimism is out
there all the time."