Sci Eng Ethics (2016) 22:169–188

DOI 10.1007/s11948-015-9625-5

ORIGINAL PAPER

Ensuring the Quality, Fairness, and Integrity of Journal

Peer Review: A Possible Role of Editors

David B. Resnik • Susan A. Elmore

Received: 31 October 2014 / Accepted: 22 January 2015 / Published online: 30 January 2015
Ó Springer Science+Business Media Dordrecht (outside the USA) 2015

Abstract A growing body of literature has identified potential problems that can
compromise the quality, fairness, and integrity of journal peer review, including
inadequate review, inconsistent reviewer reports, reviewer biases, and ethical
transgressions by reviewers. We examine the evidence concerning these problems
and discuss proposed reforms, including double-blind and open review. Regardless
of the outcome of additional research or attempts at reforming the system, it is clear
that editors are the linchpin of peer review, since they make decisions that have a
significant impact on the process and its outcome. We consider some of the steps
editors should take to promote quality, fairness and integrity in different stages of
the peer review process and make some recommendations for editorial conduct and
decision-making.

Keywords Peer review
Quality
Fairness
Integrity
Ethics
Reliability
Bias

Editors
Publication

Introduction

Nearly all scientific journals make publication decisions based on the recommen-
dations of peer reviewers. The editors of Philosophical Transactions of the Royal
Society of London instituted the first peer review procedures in the seventeenth
century, but peer review did not become a common practice in scientific publishing

D. B. Resnik (&)
National Institute of Environmental Health Sciences, National Institutes of Health, Box 12233, Mail
Drop CU 03, Research Triangle Park, NC 27709, USA
e-mail: resnikd@niehs.nih.gov

S. A. Elmore
National Toxicology Program, National Institute of Environmental Health Sciences, National
Institutes of Health, Research Triangle Park, NC 27709, USA

123
170 D. B. Resnik, S. A. Elmore

until the late 19th century (Lee et al. 2012). Since the early 20th century, peer
review has been used to evaluate grant applications, make personnel decisions, and
award scientific prizes (Resnik 2011). Journal peer review performs several
functions in science. First, it serves as a type of quality control by helping to ensure
that published research meets standards of methodology and ethics (Smith 2006).
Peer review is a form of scientific gate-keeping. Second, it serves as a form of
quality improvement because authors usually make positive changes in their
manuscripts in response to useful comments from reviewers and editors (Armstrong
1997). Third, peer review serves as a form of education if authors learn more about
how to do good science and write good papers through their interactions with
reviewers and editors (Shamoo and Resnik 2015). Peer review is far from perfect—
low quality papers that are rejected by one journal may eventually be published by
another, and reviewers may fail to make suggestions that improve papers or teach
authors about doing good science—but most researchers agree that peer review is an
indispensable part of scientific inquiry (Merton 1973; Smith 2006; Resnik 2011; Lee
et al. 2012; Ho et al. 2013).
A growing body of literature has identified potential problems that can compromise
the quality, fairness, and integrity of journal peer review. Some of these include
inadequate, inconsistent or biased review, and ethical transgressions by reviewers
(Resnik et al. 2008; Lee et al. 2012). In an online survey of scientists from top
universities from around the world, less than half of 1,340 respondents agreed that peer
review is fair or scientific, and 60 % said that reviewers are often incompetent (Ho
et al. 2013). While the empirical data indicates that problems can occur in peer review,
in many cases the evidence for them is weak or inconclusive (Lee et al. 2012).
Scientists and journal editors have suggested or implemented several reforms to
address these problems, such as double-blind or open review procedures (Resnik 2011;
Lee et al. 2012). Under the more traditional anonymous or single-blind peer review
process, used by the majority of scientific journals, the reviewer’s name is not
disclosed to the author. The identities of the authors and reviewers are concealed from
each other in a double-blind review whereas in an open peer review process no
identities are concealed. While the double-blind or open review proposals may or may
not help to improve peer review, they overlook the key players in the entire process—
the editors. Editors play a crucial role in peer review because they manage the review
process, and peer review cannot function well unless editors adhere to the highest
standards of science and ethics. This article will summarize some of the problems with
the peer review system, discuss proposed reforms, and describe the role that journal
editors should play in safeguarding and improving peer review.

Inadequate Review

Some studies have found that reviewers often fail to catch simple errors or
methodological flaws in submitted manuscripts (Lee et al. 2012). This may be due to
poor choice of reviewers. Reviewers that provide inadequate assessments may be
inexperienced in the review process or may lack in depth knowledge of the type of
study under review. Reviewers should be experts in the field and their numbers may

123
Ensuring the Quality, Fairness, and Integrity 171

be few, depending on the area of study. Articles may also be complex, incorporating
research from different areas of expertise, so no one expert reviewer would be able
to provide a thorough and knowledgeable review (Shamoo and Resnik 2015).
One study sought to evaluate the effects of reviewer characteristics and
publication language on manuscript assessment. The researchers sent two
inauthentic but realistic papers with obvious methodological flaws, such as
inappropriate statistical tests, to 180 reviewers. Reviewers who caught all of the
errors received a grade of 4 on a 1 to 4 scale. The 156 reviewers who returned
reviews received an average score of 1.4. Only 25 % identified wrong sampling unit
as a methodological flaw, and 34 % made no comments about the methodology
(Nylenna et al. 1994). Another study aimed to determine the frequency of detection
of major and minor errors at a general medical journal, the types of errors missed,
and the impact of training on error detection. The conclusions were that, under the
conditions of this study, training appeared to have only a minimal impact on
reviewer performance (Schroter et al. 2008).
Many commentators have observed that peer reviewers seldom detect data
fabrication, falsification or other types of misconduct (LaFollette 1992; Resnik
2011; Shamoo and Resnik 2015). Some well-known fraudulent papers that made it
through the peer review system include a study on the generation of human
embryonic stem cells via therapeutic cloning (Hwang et al. 2005), experiments
demonstrating how to derive pluripotent stem cells from somatic cells (Obokata
et al. 2014), and a paper linking the measles vaccine to autism (Wakefield et al.
1998). While the rate of misconduct in published papers is not known, it is thought
to be low. A meta-analysis of 18 surveys found that an average of 1.97 % of
scientists admitted to fabricating or falsifying data at least once in their careers
(Fanelli 2009). It is not fair to blame reviewers for failing to detect misconduct in
submitted manuscripts, however, since they usually do not have access to the
materials needed to detect fabrication or falsification, such as the original research
records. Reviewers usually only examine the submitted manuscript and occasionally
some supporting documents or data (Lee et al. 2012). Clever, intentional
manipulation of data can be extremely difficult to detect, especially if the reviewer
is relying solely upon his or her own reading of the manuscript.
Some journals have begun using computer programs to check submitted
manuscripts for suspected plagiarism. The programs compare manuscripts to a
database of published articles (Butler 2010). One publisher, Taylor and Francis,
screened all articles submitted to three journals during a six month period for
plagiarism. One journal had a 23 % rate of suspected plagiarism (including self-
plagiarism) (Butler 2010). Other journals have begun using computer programs to
check digital images for deceptive image manipulation. These journals have
developed guidelines for image manipulation and require authors to submit original
images for review, so the editors can determine whether changes that have been
made to the original images are appropriate (Lee et al. 2012). Two journals that
implemented these procedures found that 20-25 % of authors failed to follow image
manipulation guidelines and that 1 % of submitted manuscripts had problems that
were so significant that the editors decided to contact the authors’ institution for
suspected misconduct (Cromey 2013).

123
172 D. B. Resnik, S. A. Elmore

Inconsistent Review

Lack of consistency is another problem with peer review. Some scientists have
complained that review is so inconsistent that getting a paper published is more a
function of luck than talent (Lee et al. 2012). Whether a paper is accepted or
rejected may depend less on the quality of one’s work and more on the reviewers
who agree to evaluate the manuscript. Because reviewers are invited, and not
assigned, to review manuscripts, even if they are members of an editorial review
board, it can be difficult at times to find the desired number of reviewers and this
may affect the level of reviewer expertise. It is also not unusual for reviewers to
have diametrically opposed opinions of the same manuscript (Lee et al. 2012). Since
reviewers may have different interpretations of how the review criteria apply to
scientific manuscripts, some variation should be expected (Lee 2013). However,
significant divergence in reviewer assessments of manuscripts can indicate
inadequate reviewer expertise and/or problems with the clarity of review standards
that can undermine the quality, fairness, and integrity of review. In multidisciplinary
research, reviewers from different areas of expertise may have divergent opinions of
manuscripts based on divergent review standards (Lee et al. 2012). So this would
not reflect deficiencies in expertise, but rather differences in areas of expertise
pertaining to different sections or data reviewed by each expert.
One of the reasons why reviewer reports are often inconsistent is that reviewers
are usually asked to evaluate subjective characteristics of the research, such as its
significance, novelty, or overall quality (Lee et al. 2012). Reviewers may have
different opinions concerning significance, novelty, or overall quality because they
have different levels of experience, theoretical or methodological assumptions or
ideas about the direction the field should take. For example, a reviewer who accepts
a controversial hypothesis concerning the treatment of prostate cancer might find a
paper that reinforces this hypothesis to be significant, but a reviewer who rejects this
hypothesis might not. Because subjective assessments of papers are a key part of the
review process, it may be difficult to reduce inconsistency in peer review (Park et al.
2014).
In one of the earliest studies on consistency in peer review, the researchers
resubmitted twelve articles to the same psychology journals that had refereed and
published them 18 to 32 months earlier. The articles were the same, except the
names and affiliations had been changed. Only 3 out of the 38 editors and reviewers
(7.9 %) detected the resubmission. Nine articles went through the review process
again. 16 out of 18 reviewers recommended rejecting these articles based on
‘‘serious methodological flaws’’ and the editors concurred (Peters and Ceci 1982).
While this study produced some interesting (and potentially embarrassing) results,
its main flaw was the small sample size.
Inter-rater reliability (IRR) for journal peer reviews is the degree to which two or
more independent reviews of the same scientific document agree (Cicchetti 1991).
A number of large studies on IRR for peer reviewers have been published. One
study examined 5,881 reviewer reports for the Journal of General Internal Medicine
from 2004 to 2008 and found that IRR for the overall decision (reject vs. accept/
revise) was barely better than chance (Kravitz et al. 2010). Another study examined

123
Ensuring the Quality, Fairness, and Integrity 173

IRR for papers submitted to two neuroscience journals. For one journal, agreement
was very low (Cohen’s Kappa statistic = 0.08, where 0.0 = equals agreement no
better than chance and 1.0 = perfect agreement). For the other journal, agreement
was also poor (Cohen’s Kappa = 0.28) (Rothwell and Martyn 2000). However,
another study of 554 reviews of 206 manuscripts submitted to a general medical
journal between 2008 and 2009 found that while IRR was low as measured by the
Cohen’s Kappa statistic (\ 0.2), it was higher when using Gwet’s Kappa (0.63). The
authors concluded that IRR was high enough for the purposes of editorial decision-
making (Baethge et al. 2013). Another study examined reviewer assessments of 495
abstracts submitted to a scientific meeting from 2001 to 2004 and found significant
agreement about the overall quality of the submissions. IRR was higher for
methodological criteria, such as statistical considerations, than for opinion-based
criteria, such as overall significance of the research (Rowe et al. 2006). Finally, a
meta-analysis of 48 journal articles published between 1966 and 2008 that examined
reviewer agreement found that mean IRR was low (mean Kappa = 0.17)
(Bornmann et al. 2010).
As shown here, accurate interpretation of various journal IRR data may be
challenging. The overall low IRRs among different journals may be due to a variety
of factors such as the difficulty that editors have in finding sufficient reviewer
expertise for submitted manuscripts, the complexity of manuscript data and thus the
need for different expert reviewers, or that the data used to determine the IRR is
inappropriate.
The Journal of the American Child and Adolescent Psychiatry tested various
methods to improve the IRR of reviewers’ ratings of submitted manuscripts over a
1-year period with 296 pairs of ratings studied. Their findings determined that,
under the conditions tested, the IRR could be improved by using a multi-item rating
scale, using training manuals, obtaining the ratings of solid, concrete items rather
than judgments or opinions, and averaging the scores of multiple reviewers
(Strayhorn et al. 1993).

Biased Review

Bias is a third problem with the peer review process and is considered a violation of
impartiality in the evaluation of a submission. Because peer review is conducted by
people who may have different ideas about what constitutes good science, no review
will be completely free from methodological or theoretical biases (Lee et al. 2012).
However, certain types of strong biases can undermine the quality, fairness, and
integrity of peer review. Some examples of what may influence reviewer bias
include a preference for positive versus negative findings, gender of author, gender
of reviewer, institutional affiliations, author nationalities, controversial or innova-
tive research, author recommended reviewers, reviewer or editor conflict of interest,
and authors’ reputation.
There is a well-documented bias toward publication of positive findings as
opposed to negative ones. One of the earliest studies of this type of bias examined
the fate of 285 research projects approved by a human subjects ethics committee

123
174 D. B. Resnik, S. A. Elmore

between 1984 and 1987 and found that studies showing a statistically significant
difference between treatment groups in clinical trials were much more likely to be
published than those showing no difference (Easterbrook et al. 1991). Another study
with a similar design found that clinical trials reporting statistically significant
positive results were much more likely to be published than trials with negative
results, and their time to publication was also shorter (Stern and Simes 1997). A
systematic review and meta-analysis of twenty studies on positive biases found
strong evidence for an association between statistically significant positive results
and publication (Dwan et al. 2013).
Although the evidence supports a bias toward publishing positive results, it is not
clear that the peer review system is responsible for this bias. For example, one study
found that the bias toward publication of positive findings was due to decisions
made by authors, not editors or reviewers (Dickersin et al. 1992). A study of 745
manuscripts submitted to the Journal of the American Medical Association between
1996 and 1999 found that manuscripts reporting positive results were not more
likely to be published than those reporting negative results (Olson et al. 2002).
However, the results of this study may not be generalizable, since it focused on one
high-impact journal. Furthermore, it may be the case that authors refrain from
submitting papers with negative results because they expect that they will be
rejected (Olson et al. 2002). Clearly, more research on the causes of the bias toward
publication of positive findings is needed.
A number of studies have identified gender biases in grant peer review
(Wenneras and Wold 1997; Bornmann et al. 2007; Lee et al. 2012), but evidence for
gender bias in journal peer review is weak. One study sent an identical manuscript
to 126 male and female reviewers from behavioral science journals. The only
difference in the manuscripts was that half had a first author with a male name and
the other had one with a female name. 65 reviews (51.6 %) were returned. Female
reviewers accepted significantly more female-authored papers than male reviewers
and they accepted significantly more female-authored than male-authored papers
(Lloyd 1990). A significant limitation of this study was its small sample size. A
larger study with a similar design sent the same paper (with male vs. female first
author names) to biology students and faculty to review. 989 responses were
suitable for data analysis. The study found that the gender of the first author name
had no effect on the review assessment or the overall recommendation, although
female post-graduate students who reviewed papers had significantly higher
rejection rates, suggesting that they may apply different standards to peer review
(Borsuk et al. 2009).
A 2001 study that provided evidence of gender bias reported that the percentage
of female first authors on papers significantly increased after Trends in Ecology and
Evolution adopted a double-blind review system, in which neither authors nor
reviewers are informed of each other’s identity (Grod et al. 2008; Budden et al.
2008). However, some commentators on this study objected to its statistical analysis
and presented data challenging the notion that peer review is gender biased
(Whittaker et al. 2008). Another study of 2,680 papers from five journals found no
significant difference in the acceptance rates of female first-authored papers and
male first-authored papers (Tregenza 2002), and a study of 713 papers submitted to

123
Ensuring the Quality, Fairness, and Integrity 175

the Journal of Neurophysiology in 2007 reported similar results (Lane and Linden
2009). An article reviewing the causes of the underrepresentation of women in
science concluded that there is no evidence of gender bias in journal peer review
(Ceci and Williams 2011).
Some studies have examined biases related to the institutional affiliations and
nationalities of authors. One study examined abstracts submitted to the American
Heart Association’s Scientific Sessions from 2000 to 2005. From 2000 to 2001, the
organization used an open review system, so that information concerning the
authors’ names and institutional affiliations was shared with reviewers. In 2001, the
organization instituted a blind review policy, so the reviewers were not told the
authors’ names or affiliations from 2002 to 2005. 27,311 abstracts were submitted
from 2001 to 2002, and 39,964 abstracts were submitted from 2002 to 2005. The
study found that, during open review, there was evidence of possible bias in favor of
authors from the U.S. and other English-speaking countries, prestigious academic
institutions, and U.S. government agencies. There was evidence of possible bias
against authors from private industry but no evidence of gender bias. Blinded peer
review partially reduced evidence of reviewer biases (Ross et al. 2006). A study of
17,205 abstracts submitted to a scientific meeting between 1992 and 1995 found that
North American authors were more likely to have their abstracts accepted than
authors from other regions (Timmer et al. 2001), and a study of 2,680 papers
submitted to five journals found that authors from English-speaking countries were
more likely to have their papers accepted than authors from non-English-speaking
countries. Another study found that journals were more likely to accept papers from
authors who have the same nationality as the journal, with the strongest effect being
in favor of U.S. authors (Ernst and Kienbacher 1991). A retrospective review of 258
major papers and 147 brief reports published in a medical journal in 1994 found that
there was no bias in favor of papers from more prestigious institutions although
there was some bias in favor of brief reports (Garfunkel et al. 1994). However, one
must also consider that apparent bias in favor of or against authors of a particular
nationality or affiliation may be due to differences in scientific rigor or writing
proficiency between various countries and/or institutions. Moreover, if the science
appears sound but the manuscript is unacceptable due to poor writing style or
excessive grammatical errors, then a journal may reject with a recommendation to
resubmit once the editing has been completed. So a true account of the rejection rate
for any particular journal may be difficult to determine.
Several commentators have criticized the grant peer review process for being
conservative, claiming that reviewers are biased against research that is controver-
sial, innovative, interdisciplinary, or otherwise challenges their theoretical frame-
works, methodologies, or assumptions (Hull 1988; Chubin and Hackett 1990;
Shamoo and Resnik 2015). A few studies have found that journal peer review may
also be conservative, but this claim is difficult to assess (Lee et al. 2012). One study
randomly assigned 398 reviewers to receive either a paper on a conventional therapy
for obesity or one on an unconventional therapy. The papers were identical except
for the type intervention. 141 reviewers returned useable assessment forms. The
study found significant evidence of bias in favor of the conventional therapy,
suggesting that reviewers may be biased against unorthodox research (Resch et al.

123
176 D. B. Resnik, S. A. Elmore

2000). Other studies have found that reviewers tend to favor papers that fit their
theoretical assumptions (Mahoney 1977; Hull 1988).
Various studies have examined other types of biases in peer review. One study
that assessed the reviews from 156 reviewers from a Scandinavian journal found
evidence of a bias toward manuscripts written in English compared to the same
manuscripts written in the author’s native language (Nylenna et al. 1994). Two
observational studies comparing reviewers suggested by authors vs. editors found
no difference in the quality of review, although author-suggested reviewers were
more likely to recommend publication (Schroter et al. 2006; Rivara et al. 2007).
Some studies have produced evidence that female reviewers are less likely to
recommend acceptance of manuscripts than male reviewers (Gilbert et al. 1994;
Borsuk et al. 2009; Wing et al. 2010).
An important potential source of bias that has not been well-studied is reviewer
or editor conflict of interest. Reviewers or editors may have financial interests, such
as stock ownership, intellectual property, or consulting arrangements with private
sponsors, which are related to the research that they are asked to review (Gasparyan
et al. 2013; Shamoo and Resnik 2015). While there is a considerable body of
evidence showing that authors’ financial interests can create bias in research
(Sismondo 2008; Shamoo and Resnik 2015), there are no published studies on the
impact of reviewers’ or editors’ financial interests. It may difficult to study this topic
because many journals to do not require editors or reviewers to disclose conflicts of
interest and journals that have a disclosure policy may not enforce it or have a way
to verify the reviewer’s stated lack of conflict. It may therefore be difficult to
determine whether reviewers or editors have undisclosed conflicts of interest that
could impact review.
Another potential source of bias that has not been well-studied pertains to the
authors’ reputation. Several decades ago, sociologist of science Robert Merton
(1973) hypothesized that science is subject to a bias which he dubbed the Matthew
Effect after the passage from the Holy Bible, King James Version: ‘‘For unto every
one that hath shall be given, and he shall have abundance: but from him that hath
not shall be taken even that which he hath (Matthew 25:29).’’ The basic idea here is
that researchers with well-established reputations tend to increase their funding and
publications, while those without well-established reputations do not. While many
scientists have complained that an ‘‘old boys’ network’’ shapes grant review in
science (McCullough 1989; Shamoo and Resnik 2015), the influence of the author’s
reputation on journal publication decisions has not been well-studied (Lee et al.
2012).

Unethical Review

It is important for reviewers to adhere to ethical standards to protect the integrity

and trustworthiness of peer review. Authors who submit manuscripts to a journal
must be able to trust that reviewers will not use their ideas, methods, or data without
permission and will treat their manuscripts as confidential (Shamoo and Resnik
2015). Journal editors must be able to trust that reviewers will disclose conflicts of

123
Ensuring the Quality, Fairness, and Integrity 177

interest and conduct reviews professionally. The Committee on Publication Ethics

(2014) has developed ethical standards for authors, editors, and reviewers. There is
some anecdotal evidence that reviewers sometimes act unethically, but there is little
empirical data to support this claim (Nature Editors 2001; Rennie 2003; Smith 2006;
Resnik 2011). One study administered a survey on perceptions of unethical practices
in peer review to biomedical scientists (Resnik et al. 2008). Respondents said that
comments from reviewers have included personal attacks, that a reviewer had
delayed the review so that he/she could publish an article on the same topic, that a
reviewer breached confidentiality, and that a reviewer used ideas, data, or methods
without permission (Resnik et al. 2008). An online survey also found that scientists
have some concerns about ethical problems in peer review such as personal attacks
from reviewers, reviewer breach of confidentiality, and reviewers requiring
unnecessary references to their own research (Ho et al. 2013). One limitation of
both of these studies is that they asked respondents to provide information about
their experiences with journal peer review but they did not confirm whether alleged
ethical transgressions actually occurred. The low response rate is a limitation of the
larger survey, since scientists who experienced problems with peer review may have
been more likely to respond. Nevertheless, these surveys indicate that reviewers
sometimes violate ethical norms and that there is a perception among scientists that
this does occur with some frequency.

Proposals for Reforming Journal Peer Review

Editors, scientists, and commentators have discussed various proposals for reforming
the journal peer review process to address the problems that have been identified.
Proposals range from minor tweaks to radical revisions. A modest proposal for
reforming the system is to increase the number of reviewers used to evaluate
manuscripts (Neff and Olden 2006). Most journals use two reviewers but some use more.
The thinking behind this proposal is that adding additional reviewers will increase the
probability of converging on the correct assessment of the manuscript (Neff and Olden
2006). While using more reviewers could help promote consistency and reduce bias, it
might not, since additional reviewers might reinforce each other’s biases or increase the
divergence of opinions without convergence. Since this proposal has not been well-
studied, we cannot say for certain whether adding additional reviewers improves peer
review (Newton 2010). However, it is the editor’s responsibility to ensure, to the best of
his/her ability, that reviewers’ edits and comments are not biased but rather provide the
author with a detailed report of the strengths and weaknesses of the paper so that the
author may make appropriate revisions.
Another modest proposal is to train reviewers in review standards and techniques
to improve peer review (Lee et al. 2012). Some scientists have published papers on
how to review a manuscript (Salasche 1997; Benos et al. 2007; Lovejoy et al. 2011)
and most journals provide reviewers with some amount of instruction. While
training sounds like a good idea, there is little published evidence that it works. One
study randomly assigned 609 reviewers to receive one day of face-to-face training in
review standards and techniques, self-study training materials with the same content

123
178 D. B. Resnik, S. A. Elmore

as the face-to-face training, or no training (control). The reviewers received three

manuscripts to review containing some deliberately introduced errors. Reviewer
performance was measured at baseline and after receiving the training (or no
training). The study found that both intervention groups caught more errors than the
control group, and that the self-study training materials had a minor impact on
reviewer performance, but that these were slight improvements in reviewer
performance that faded over time (Schroter et al. 2004). Other studies have found
that training and mentoring have little impact on reviewer performance (Callaham
et al. 1998; Callaham and Tercier 2007; Houry et al. 2012). Although these studies
have not shown evidence of a positive effect of training on reviewer performance,
there are no studies that show a negative impact. We believe that the use of training
materials, either self-study or face-to-face, should not be discouraged, since training
can help the novice reviewer understand the methodology of manuscript review as
well as encourage consistency in the preview process.
A more radical proposal is to use double-blind review. As noted above, most
scientific journals use a single-blind system. Some have argued that a double-blind
system, in which authors and reviewers are both anonymous, would improve the
peer review process. 58.2 % of respondents in a survey of peer review favored a
system in which authors are anonymous (Ho et al. 2013). Some journals have
adopted a double-blind system in an attempt to improve their peer review process
(Resnik 2011; Lee et al. 2012). Several studies have conducted randomized
controlled trials to answer the question of whether double-blinding improves the
quality of review. While two small studies (McNutt et al. 1990; Fisher et al. 1994)
found that blinding reviewers improves the quality of review, several larger ones
found that it does not (van Rooyen et al. 1998; Justice et al. 1998; Godlee et al.
1998). Moreover, there is evidence that reviewers can frequently identify authors
from their references and topics, especially in small fields in which most scientists
know each other or are familiar with each other’s work (Lee et al. 2012). One study
found that blinded reviewers could correctly identify first authors 42 % of the time
(van Rooyen et al. 1998). Other studies have yielded similar results (Fisher et al.
1994; Cho et al. 1998; Justice et al. 1998; Baggs et al. 2008).
One of the potential dangers of double-blinding is that it may not provide
adequate management of reviewer conflict of interest (COI). Most journals ask
reviewers to disclose COIs related to the review. A reviewer who is not told the
authors’ identities may see no reason to disclose a COI, even if the reviewer can
identify the authors of the paper. One way to deal with this problem would be for
editors to ask reviewers to decline an invitation to review if they can identify the
authors and they have a COI.
Although double-blinding may not work all the time, when it does work it may
help to promote the ideal that publications be judged on the quality of the work with
no bias for or against the author. In our opinion, a double-blind system would
probably reduce biases related to reputation, gender, nationality, and institutional
affiliation but it probably would not reduce other biases or address defective or
inconsistent review, or ethical transgressions by reviewers.
The most radical proposal for reforming peer review is to remove blinders from
authors and reviewers so the entire process is open, and some journals have already

123
Ensuring the Quality, Fairness, and Integrity 179

implemented such an open review process (Smith 1999; Lee et al. 2012). Open
review could reduce ethical transgressions by holding reviewers accountable for
their conduct (Resnik 2011). Reviewers might be more likely to review manuscripts
more carefully under an open system, since their identities would be revealed to the
authors. They also might be less likely to make personal attacks in their reviews,
breach confidentiality, or use information without permission. Authors might also
regard an open process as fairer than a blinded one because there would be more
transparency.
So far the evidence for the effectiveness of open review is inconclusive. Several
randomized controlled trials of open review have found that open review does not
significantly improve the quality of peer review (van Rooyen et al. 1999; Godlee
et al. 1998; van Rooyen et al. 2010). However, one study found reviewers who
participated in open review performed better reviews and were more courteous
(Walsh et al. 2000). One of the biggest drawbacks of an open review system is the
potential that reviewers may decline to perform reviews because they want the
protection from potential reprisal afforded by anonymity (Lee et al. 2012). Also,
reviewers may provide a less-than-honest opinion of the work if they know that their
identities will be revealed to the authors. Junior scientists might fear reprisals from
more senior researchers if they make negative or critical comments about their
papers (Smith 1999) and authors may feel compelled to be more responsive to
senior reviewers than junior ones. A narrow field with a limited number of experts
may require colleagues to review each other’s work, and they may not want to
provide negative reviews for each other under an open review system. Some studies
have shown that most reviewers prefer to remain anonymous (Lee et al. 2012). For
example, 64.4 % of respondents in a study of scientists’ views on the peer system
said that reviewers’ identities should not be disclosed to authors (Ho et al. 2013). In
a study comparing blinded vs. un-blinded peer review, potential reviewers were
more likely to decline to participate in the study if their identities would be revealed
to the authors (van Rooyen et al. 1999). Another study of open review found that
55 % of potential reviewers refused to participate in a study in which their identities
would be shared with authors (van Rooyen et al. 2010). The main reason why
experts often decline to participate in open review is that they fear retaliation from
authors (Ho et al. 2013).
Some commentators have recommended that journals adopt radically open
systems to enhance transparency and accountability. Under this proposal, the entire
peer review process would be open to the public so that scientists could critique it.
Review reports, including identifying information, would be published along with
papers. Readers of the journal would be allowed to critique the papers, review
reports, and the editorial decision-making process (Wicherts et al. 2012). One
advantage of a radically open system is that it would make it possible to discover
undisclosed conflicts of interest (Benos et al. 2007), which can be difficult to detect.
A radically open system might also improve the quality and integrity of review by
holding reviewers accountable to the scientific community, but qualified reviewers
might refuse to participate out of fear of reprisal. Another problem with radically
open review is that it would make information available to the public, such as data
that could be used to identify human subjects, which should be kept confidential.

123
180 D. B. Resnik, S. A. Elmore

Some journals have also begun using hybrid systems in which the initial review is
double-blinded, but then invited open peer commentaries on the article are posted
once it is published. Authors can read and respond to the commentaries in the same
issue of the journal. In theory, hybrid systems could help to reduce bias while
encouraging free exchange of ideas. However, the impact of hybrid systems on peer
review has not been studied (Lee et al. 2012).

The Role of Editors in Promoting Quality, Integrity, and Fairness in Review

Regardless of the outcome of additional research or attempts at reforming the

system, it is clear that editors are the linchpin of peer review, since they make
decisions that have a significant impact on the process and its outcome (Newton
2010). Editors invite experts to review manuscripts, communicate with reviewers
and authors, provide guidance for reviewers, read reviewer reports, ensure that
policies regarding conflict of interest and confidentiality are upheld, and provide
opinions to the editor-in-chief concerning acceptance, revisions, or rejection
(Chubin and Hackett 1990). Editors can compensate for the flaws inherent in the
peer review system or they can exacerbate them.
Journals may have different personnel organizations, depending on their size and
affiliation. Larger journals may have a team of full-time professional editors. Other
journals, such as those affiliated with a professional society, may have an outside
volunteer editorial board composed of associate editors to assign reviewers and
oversee individual manuscripts, and a larger group of volunteer reviewers, with a
variety of expertise, who review papers on specialized topics. This helps to ensure
that they have enough expertise to handle a variety of reviews. But in all cases, the
editor-in-chief (chief editor or executive editor) is the editorial leader with final
responsibility for all operations and policies and ultimately decides whether or not a
submitted manuscript will be accepted or rejected.
In the following sections, we offer some opinions about the steps that editors
should take to minimize and mitigate the problems identified in previous sections
and improve the quality, integrity, and fairness of peer review. These opinions are
based on our professional experience as journal editors. We will organize our
discussion of editorial responsibilities around the different stages of the peer review
process, from selecting reviewers to dealing with post-publication issues.

Selecting Reviewers

Editors have an opportunity to significantly impact the quality, fairness, and

integrity of the peer review process when they select reviewers. Editors may use
different methods for choosing potential reviewers, such as checking the paper’s
reference list for subject matter experts or utilizing a database containing the names
and expertises of people who have agreed to review papers for the journal. If editors
use reviewers suggested by authors, they should include some independent
reviewers to ensure balance. Editors should avoid stacking the deck for or against

123
Ensuring the Quality, Fairness, and Integrity 181

a manuscript. To ensure that manuscripts receive a fair hearing, editors should seek
an appropriate balance between different types of reviewers with different
qualifications.
Editors may have information concerning an individual’s potential bias for or
against a particular theoretical orientation, methodology, research group, discipline,
nationality, or institution. Editors may also know whether some reviewers are more
demanding or lenient than others. If a submission crosses disciplinary boundaries,
editors should invite reviewers from different disciplines to ensure that it receives a
fair and adequate review. If a manuscript is highly innovative or controversial,
editors should consider inviting reviewers who they believe are open to new ideas
(Armstrong 1997). Of course, inexperienced editors may not be familiar with
reviewer penchants and biases, but they should gain experience as they continue to
manage manuscript review.
If editors invite reviewers who are not well-qualified to review the paper, then
their reviewer reports may be inadequate. In many research fields, those who are
best qualified to review a paper may also have biases for or against the paper.
Editors should balance fairness and reviewer competence when choosing reviewers.
In some cases, it may be necessary to use reviewers who may have a bias in order to
obtain a high-quality review. It may be necessary to invite reviewers with special
qualifications beyond the topic of the paper to ensure that the paper is properly
reviewed. For example, a paper that includes digital images of the structure of a
protein may need to be reviewed by someone with expertise in digital imaging, and
a paper that uses an innovative method for analyzing the results of a clinical trial
may need to be reviewed by a statistician.

Providing Guidance for Reviewers

Editors should promote the fairness, quality, and integrity of peer review by
providing reviewers with appropriate guidance. When editors provide guidance for
reviewers, they should inform them of the criteria used to evaluate manuscripts
(such as originality, significance, rigor, etc.), journal review policies (including
ethics and conflict of interest), the journal’s policies on authorship criteria, and the
timetable for returning reviews. Editors should remind reviewers of deadlines to
prevent tardiness and they should consider inviting a different reviewer if a reviewer
has not completed an assignment on time. It is important to note, however, that
reviewing a paper can take a great deal of time and reviewers may be busy with
other projects, so editors should take this into account when communicating with
them. Editors should also be on the alert for possible ethical transgressions by
reviewers and take steps to address them if they occur.
When a reviewer discloses a conflict of interest, editors must decide whether to
invite another reviewer or to solicit input from the reviewer despite the conflict. An
editor might decide to seek input from a reviewer with a disclosed conflict of
interest because he or she judges that the conflict is not likely to significantly impact
the reviewer’s judgment and there are very few people who are qualified or willing
to review the manuscript.

123
182 D. B. Resnik, S. A. Elmore

Evaluating Reviewer Reports

When editors read reviewer reports, they should assess them for quality, fairness,
thoroughness, and professionalism. Ideally, editors should have enough knowledge
of the relevant area of research so they can form their own independent judgment of
the merits of the manuscript, which will allow them to assess reviewer reports.
Editors should take an active role in the review process by engaging with reviewers
and critically evaluating their reports (Newton 2010). To do this, they should read
the manuscript carefully and have some familiarity with the research. When editors
receive a report, they should determine whether it is accurate, thorough, fair, and
professional. They should assess reviewer comments to decide whether they are
useful and appropriate. Editors should ask reviewers to clarify ambiguous
comments. An editor may decide to ignore or downgrade a reviewer report if he
or she judges that a review is inadequate, incompetent, and/or biased. If a reviewer
makes a personal attack in a report, the editor should delete those remarks from
comments that are passed on to the authors. However, editors should be careful
when deleting reviewer portions of comments, since this is a form of censorship,
and deciding what counts as inappropriate is a matter of interpretation. Larger
journals may use multiple associate editors and multiple reviewers that review
papers on specialized topics to ensure that they have enough expertise to handle the
review. If editors lack sufficient expertise in a domain of research they are
overseeing, they should assign the manuscript to another editor, or they should
consult with other editors or editorial board members who have the requisite
expertise.
When editors make decisions concerning a manuscript, they should weigh and
consider the reviewer reports and be mindful of their own biases for or against a
paper, to the extent that this is possible (Newton 2010). For example, editors should
be careful to avoid basing a judgment of a paper on an author’s or an institution’s
reputation (Lee et al. 2012). When reports are conflicting, editors may invite another
reviewer to break the tie or make a decision based on their overall assessment of the
manuscript and the reports. Journals with very high standards for publication may
require that all of the reviewer reports are positive, whereas journals with lower
standards may decide to accept an article even if some reports are negative. In
difficult cases, editors should consider holding a conference with each other to
decide upon the outcome of review. Editors may also share other reviewer reports
with the reviewers after the review process is complete to provide them with
different perspectives on the paper and help them learn more about peer review. As
an example, some journals blind copy the reviewers on the decision letter emails to
the authors so that the reviewers will have such access to all reviews.

Communicating Decisions to Authors

Editors should promote fairness and integrity when they communicate with authors
about editorial decisions. If authors do not agree with editorial decisions or reviewer
reports, they should be given the opportunity to respond to reviewer comments and

123
Ensuring the Quality, Fairness, and Integrity 183

explain why one or more reviewer recommendations are inaccurate, unfair or

unnecessary. Such discussions may go back and forth between authors, editors, and
reviewers, and other experts or chief editors may need to be consulted. Chief editors
have the final responsibility for deciding whether requested changes should be made
and whether manuscripts should be accepted for publication.
When editors convey their decisions to authors, they must decide upon the
conditions for acceptance. For example, an editor may decide not to require an
author to include a reference recommended by a reviewer if he or she deems it to be
unnecessary. Editors should provide instructions to authors for revising the
manuscript, distinguishing between revisions that are necessary and those that are
optional. Editors should require authors to submit a letter detailing the revisions that
have been made when they submit a revision. They may also require the authors to
submit a tracked changes version of the manuscript with the revision. They can also
address any questions or concerns authors may have about the review process or the
editorial decision.

Post-publication Editorial Decisions

Editors should continue to promote the quality, integrity, and fairness of the peer
review process after an article is published. When a paper appears in print or online,
chief editors must be prepared to respond to commentaries on the paper from other
scientists. Chief editors may decide to publish a commentary or letter that criticizes
a paper and give the authors an opportunity to respond in a concurrently published
letter. If there are serious problems with a paper, such as errors that undermine the
validity of the data, fabrication or falisification of data, or plagiarism, chief editors
must decide whether to require that the authors print a correction or retraction of the
paper. Editors may sometimes need to decide whether to retract a paper even if not
all of the authors agree to a retraction. Retracting a paper without the consent of all
of the authors opens the editors and the journal to legal liability, since the authors
may claim that their reputation has been unfairly damaged (Williams and Wager
2013). Editors must decide how to balance liability concerns against the need to
maintain the integrity of the published record. Some editors may choose to publish
an editorial expression of concern if a paper with a serious problem is not retracted
or corrected. In cases of suspected misconduct that cannot be confirmed or denied
by the editor alone, the editor may contact the author’s institution about the
allegation. The institution may or may not respond to the allegation. If institutional
officials conclude that misconduct has occurred they may ask the editors to retract
the paper. To promote the integrity of the published record, chief editors and their
publishing companies should develop procedures for clearly marking retracted or
corrected papers and linking them to research databases.
Chief editors should develop policies and procedures pertaining to peer review
and publication, such as policies on public availability of supporting data and
information, conflict of interest (for authors, reviewers, and editors), authorship,
prior publication of data or text, misconduct (including data fabrication or
falsification or plagiarism), retractions and corrections, human and animal research

123
184 D. B. Resnik, S. A. Elmore

protections, and clinical trial registration. The Committee on Publication Ethics

(2014) and the International Committee of Medical Journal Editors (2014) have
developed guidelines for editorial policies. Editorial policies can help authors and
reviewers better understand the expectations for publication within a journal.
Editors should ensure that statistics are compiled on their review process so they
can make improvements. They could keep track of how often individuals are asked
to review manuscripts, the value of their recommendations, and how often they
return a report on time. Editors may also decide to assess reviewer agreement and
potential bias in their review process. Some journals keep thorough statistics on the
peer review process and others have begun to move in this direction (Lee et al.
2012). The electronic submission systems that most journals have adopted can make
it easier to keep track of information related to the review process.

Conclusion

Evidence shows that several different problems—inadequate, inconsistent, or biased

review, and ethical transgressions by reviewers—can have a negative impact on
journal peer review. Since most researchers agree that peer review is an essential
part of scientific research, the question is not whether to abandon peer review but
how to improve it (Smith 2006; Lee et al. 2012). It is not known whether
implementing reform proposals, such as reviewer training or double-blind or open
review, will effectively address these problems. More research is therefore needed
on peer review’s problems and potential solutions. Since editors have a major
impact on the review process and its outcome, they will continue to play a key role
in any reform efforts, and they should take appropriate steps to promote quality,
consistency, fairness, and ethics in peer review.

Acknowledgments David B. Resnik is Associate Editor of Accountability in Research. Susan A.

Elmore is Editor and Editor-in-Chief of Toxicologic Pathology. This research is supported by the National
Institute of Environmental Health Sciences (NIEHS), National Institutes of Health (NIH). It does not
represent the views of the NIEHS, NIH, or U.S. government.

Conflict of interest The authors disclose no conficts of interest.

References

Armstrong, J. S. (1997). Peer review for journals: Evidence on quality control, fairness, and innovation.
Science and Engineering Ethics, 3(1), 63–84.
Baethge, C., Franklin, J., & Mertens, S. (2013). Substantial agreement of referee recommendations at a
general medical journal–A peer review evaluation at Deutsches Ärzteblatt International. PLoS ONE,
8(5), e61401.
Baggs, J. G., Broome, M. E., Dougherty, M. C., Freda, M. C., & Kearney, M. H. (2008). Blinding in peer
review: The preferences of reviewers for nursing journals. Journal of Advances in Nursing, 64(2),
131–138.
Benos, D. J., Bashari, E., Chaves, J. M., Gaggar, A., Kapoor, N., LaFrance, M., et al. (2007). The ups and
downs of peer review. Advances in Physiology Education, 31(2), 145–152.

123
Ensuring the Quality, Fairness, and Integrity 185

Bornmann, L., Mutz, R., & Daniela, H. D. (2010). Reliability-generalization study of journal peer
reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE, 5(12),
e14331.
Bornmann, L., Mutza, R., & Daniela, H. D. (2007). Gender differences in grant peer review: A meta-
analysis. Journal of Informetrics, 1(3), 226–238.
Borsuk, R. M., Aarssen, L. W., Budden, A. E., Koricheva, J., Leimu, R., Tregenza, T., & Lortie, C. J.
(2009). To name or not to name: the effect of changing author gender on peer review. BioScience,
59(11), 985–989.
Budden, A. E., Tregenza, T., Aarssen, L. W., Koricheva, J., Leimu, R., & Lortie, C. J. (2008). Double-
blind review favours increased representation of female authors. Trends in Ecology & Evolution,
23(1), 4–6.
Butler, D. (2010). Journals step up plagiarism policing. Nature, 466(7303), 167.
Callaham, M. L., & Tercier, J. (2007). The relationship of previous training and experience of journal
peer reviewers to subsequent review quality. PLoS Med, 4(1), e40.
Callaham, M. L., Wears, R. L., & Waeckerle, J. F. (1998). Effect of attendance at a training session on
peer reviewer quality and performance. Annals of Emergency Medicine, 32(3 Pt 1), 318–322.
Ceci, S. J., & Williams, W. M. (2011). Understanding current causes of women’s underrepresentation in
science. Proceedings of the National Academy of Sciences of the United States of America, 108(8),
3157–3162.
Cho, M. K., Justice, A. C., Winker, M. A., Berlin, J. A., Waeckerle, J. F., Callaham, M. L., & Rennie, D.
(1998). Masking author identity in peer review: What factors influence masking success? PEER
Investigators. Journal of the American Medical Association, 280(3), 243–245.
Chubin, D. E., & Hackett, E. J. (1990). Peerless science: Peer review and U.S. Science Policy. Albany,
NY: State University of New York Press.
Cicchetti, D. V. (1991). The reliability of peer review for manuscript and grant submissions: A cross-
disciplinary investigation. Behavioral and Brain Sciences, 14(1), 119–135.
Committee on Publication Ethics. (2014). Code of Conduct. http://publicationethics.org/resources/code-
conduct. Accessed 31 Oct 2014.
Cromey, D. W. (2013). Digital images are data: And should be treated as such. Methods in Molecular
Biology, 931, 1–27.
Dickersin, K., Min, Y. I., & Meinert, C. L. (1992). Factors influencing publication of research results.
Follow-up of applications submitted to two institutional review boards. Journal of the American
Medical Association, 267(3), 374–378.
Dwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J., & Reporting Bias Group. (2013). Systematic
review of the empirical evidence of study publication bias and outcome reporting bias—An updated
review. PLoS ONE, 8(7), e66844.
Easterbrook, P. J., Berlin, J. A., Gopalan, R., & Matthews, D. R. (1991). Publication bias in clinical
research. Lancet, 337(8746), 867–872.
Editors, Nature. (2001). Editorial: Bad peer reviewers. Nature, 413(6852), 93.
Ernst, E., & Kienbacher, T. (1991). Chauvinism. Nature, 352(6336), 560.
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-
analysis of survey data. PLoS ONE, 4(5), e5738.
Fisher, M., Friedman, S. B., & Strauss, B. (1994). The effects of blinding on acceptance of research
papers by peer review. Journal of the American Medical Association, 272(2), 143–146.
Garfunkel, J. M., Ulshen, M. H., Hamrick, H. J., & Lawson, E. E. (1994). Effect of institutional prestige
on reviewers’ recommendations and editorial decisions. Journal of the American Medical
Association, 272(2), 137–138.
Gasparyan, A. Y., Ayvazyan, L., Akazhanov, N. A., & Kitas, G. D. (2013). Conflicts of interest in
biomedical publications: Considerations for authors, peer reviewers, and editors. Croatian Medical
Journal, 54(6), 600–608.
Gilbert, J. R., Williams, E. S., & Lundberg, G. D. (1994). Is there gender bias in JAMA’s peer-review
process? Journal of the American Medical Association, 272(2), 139–142.
Godlee, F., Gale, C. R., & Martyn, C. N. (1998). Effect on the quality of peer review of blinding
reviewers and asking them to sign their reports: A randomized controlled trial. Journal of the
American Medical Association, 280(3), 237–240.
Grod, O. N., Budden, A. E., Tregenza, T., Koricheva, J., Leimu, R., Aarssen, L. W., & Lortie, C. J.
(2008). Systematic variation in reviewer practice according to country and gender in the field of
ecology and evolution. PLoS ONE, 3(9), e3202.

123
186 D. B. Resnik, S. A. Elmore

Ho, R. C., Mak, K. K., Tao, R., Lu, Y., Day, J. R., & Pan, F. (2013). Views on the peer review system of
biomedical journals: An online survey of academics from high-ranking universities. BMC Medical
Research Methodology, 13, 74.
Holy Bible, King James Version. (1991). New York: Random House.
Houry, D., Green, S., & Callaham, M. (2012). Does mentoring new peer reviewers improve review
quality? A randomized trial. BMC Medical Education, 12, 83.
Hull, D. (1988). Science as a process. Chicago: University of Chicago Press.
Hwang, W. S., Roh, S. I., Lee, B. C., Kang, S. K., Kwon, D. K., Kim, S., et al. (2005). Patient-specific
embryonic stem cells derived from human SCNT blastocysts. Science, 308(5729), 1777–1783.
International Committee of Medical Journal Editors. (2014). Recommendations. http://www.icmje.org.
Accessed 31 Oct 2014.
Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., & Rennie, D. (1998). Does masking author
identity improve peer review quality? A randomized controlled trial. PEER Investigators. Journal of
the American Medical Association, 280(3), 240–242.
Kravitz, R. L., Franks, P., Feldman, M. D., Gerrity, M., Byrne, C., & Tierney, W. M. (2010). Editorial
peer reviewers’ recommendations at a general medical journal: Are they reliable and do editors
care? PLoS ONE, 5(4), e10072.
LaFollette, M. (1992). Stealing Into Print: Fraud, plagiarism, and misconduct in scientific publishing.
Berkeley, CA: University of California Press.
Lane, J. A., & Linden, D. J. (2009). Is there gender bias in the peer review process at Journal of
Neurophysiology? Journal of Neurophysiology, 101(5), 2195–2196.
Lee, C. J. (2013). A Kuhnian critique of psychometric research on peer review. Philosophy of Science,
79(5), 859–870.
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2012). Bias in peer review. Journal of the American
Society for Information Science and Technology, 64(1), 2–17.
Lloyd, M. E. (1990). Gender factors in reviewer recommendations for manuscript publication. Journal of
Applied Behavior Analysis, 23(4), 539–543.
Lovejoy, T. I., Revenson, T. A., & France, C. R. (2011). Reviewing manuscripts for peer-review journals:
A primer for novice and seasoned reviewers. Annals of Behavioral Medicine, 42(1), 1–13.
Mahoney, M. J. (1977). Publication preferences: An experimental study of confirmatory bias in the peer
review system. Cognitive Therapy and Research, 1(2), 161–175.
McCullough, J. (1989). First comprehensive survey of NSF applicants focuses on their concerns about
proposal review. Science, Technology and Human Values, 14(1), 78–88.
McNutt, R. A., Evans, A. T., Fletcher, R. H., & Fletcher, S. W. (1990). The effects of blinding on the
quality of peer review. A randomized trial. Journal of the American Medical Association, 263(10),
1371–1376.
Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago:
University of Chicago Press.
Neff, B. D., & Olden, J. D. (2006). Is peer review a game of chance? BioScience, 56(4), 333–340.
Newton, D. P. (2010). Quality and peer review of research: An adjudicating role for editors.
Accountability in Research, 17(3), 130–145.
Nylenna, M., Riis, P., & Karlsson, Y. (1994). Multiple blinded reviews of the same two manuscripts.
Effects of referee characteristics and publication language. Journal of the American Medical
Association, 272(2), 149–151.
Obokata, H., Wakayama, T., Sasai, Y., Kojima, K., Vacanti, M. P., Niwa, H., et al. (2014). Stimulus-
triggered fate conversion of somatic cells into pluripotency. Nature, 505(7485), 641–647.
Olson, C. M., Rennie, D., Cook, D., Dickersin, K., Flanagin, A., Hogan, J. W., et al. (2002). Publication
bias in editorial decision making. Journal of the American Medical Association, 287(21),
2825–2828.
Park, I. U., Peacey, M. W., & Munafò, M. R. (2014). Modelling the effects of subjective and objective
decision making in scientific peer review. Nature, 506(7486), 93–96.
Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: the fate of published
articles, submitted again. Behavioral and Brain Sciences, 5(2), 187–195.
Rennie, D. (2003). Misconduct and journal peer review. In F. Godlee & T. Jefferson (Eds.), Peer review
in health sciences (2nd ed., pp. 118–129). London: BMJ Books.
Resch, K. I., Ernst, E., & Garrow, J. (2000). A randomized controlled study of reviewer bias against an
unconventional therapy. Journal of the Royal Society of Medicine, 93(4), 164–167.
Resnik, D. B. (2011). A troubled tradition. American Scientist, 99(1), 24–28.

123
Ensuring the Quality, Fairness, and Integrity 187

Resnik, D. B., Gutierrez-Ford, C., & Peddada, S. (2008). Perceptions of ethical problems with scientific
journal peer review: An exploratory study. Science and Engineering Ethics, 14(3), 305–310.
Rivara, F. P., Cummings, P., Ringold, S., Bergman, A. B., Joffe, A., & Christakis, D. A. (2007). A
comparison of reviewers selected by editors and reviewers suggested by authors. Journal of
Pediatrics, 151(2), 202–205.
Ross, J. S., Gross, C. P., Desai, M. M., Hong, Y., Grant, A. O., Daniels, S. R., et al. (2006). Effect of
blinded peer review on abstract acceptance. Journal of the American Medical Association, 295(14),
1675–1680.
Rothwell, P. M., & Martyn, C. N. (2000). Reproducibility of peer review in clinical neuroscience. Is
agreement between reviewers any greater than would be expected by chance alone? Brain, 123(Pt
9), 1964–1969.
Rowe, B. H., Strome, T. L., Spooner, C., Blitz, S., Grafstein, E., & Worster, A. (2006). Reviewer
agreement trends from four years of electronic submissions of conference abstract. BMC Medical
Research Methodology, 6, 14.
Salasche, S. J. (1997). How to ‘‘peer review’’ a medical journal manuscript. Dermatological Surgery,
23(6), 423–428.
Schroter, S., Black, N., Evans, S., Carpenter, J., Godlee, F., & Smith, R. (2004). Effects of training on
quality of peer review: Randomised controlled trial. British Medical Journal, 328(7441), 673.
Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., & Smith, R. (2008). What errors do peer
reviewers detect, and does training improve their ability to detect them? Journal of the Royal Society
of Medicine, 101(10), 507–514.
Schroter, S., Tite, L., Hutchings, A., & Black, N. (2006). Differences in review quality and
recommendations for publication between peer reviewers suggested by authors or by editors.
Journal of the American Medical Association, 295(3), 314–317.
Shamoo, A. E., & Resnik, D. B. (2015). Responsible conduct of research (3rd ed.). New York: Oxford
University Press.
Sismondo, S. (2008). Pharmaceutical company funding and its consequences: A qualitative systematic
review. Contempory Clinical Trials, 29(2), 109–113.
Smith, R. (1999). Opening up BMJ peer review: A beginning that should lead to complete transparency.
British Medical Journal, 318(7175), 4–5.
Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal
Society of Medicine, 99(4), 178–182.
Stern, J. M., & Simes, R. J. (1997). Publication bias: evidence of delayed publication in a cohort study of
clinical research projects. British Medical Journal, 315(7109), 640–645.
Strayhorn, J., McDermott, J. F., & Tanguay, P. (1993). An intervention to improve the reliability of
manuscript reviews for the Journal of the American Acacemy of Child and Adolescent Psychiatry.
Journal of the American Acacemy of Child and Adolescent Psychiatry, 150(6), 947–952.
Timmer, A., Hilsden, R. J., & Sutherland, L. R. (2001). Determinants of abstract acceptance for the
Digestive Diseases Week–A cross sectional study. BMC Medical Research Methodology, 1, 13.
Tregenza, T. (2002). Gender bias in the refereeing process? Trends in Ecology & Evolution, 17(8),
349–350.
van Rooyen, S., Delamothe, T., & Evans, S. J. (2010). Effect on peer review of telling reviewers that their
signed reviews might be posted on the web: Randomised controlled trial. British Medical Journal,
341, c5729.
van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on
quality of reviews and on reviewers’ recommendations: A randomised trial. British Medical Journal,
318(7175), 23–27.
van Rooyen, S., Godlee, F., Evans, S., Smith, R., & Black, N. (1998). Effect of blinding and unmasking
on the quality of peer review: A randomized trial. Journal of the American Medical Association,
280(3), 234–237.
Wakefield, A. J., Murch, S. H., Anthony, A., Linnell, J., Casson, D. M., Malik, M., et al. (1998). Ileal–
lymphoid–nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in
children. Lancet, 351(9103), 637–641.
Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: a randomised controlled
trial. British Journal of Psychiatry, 176, 47–51.
Wenneras, C., & Wold, A. (1997). Nepotism and sexism in peer-review. Nature, 387(6631), 341–343.
Whittaker, R. J., et al. (2008). Journal review and gender equality: A critical comment on Budden et al.
Trends in Ecology & Evolution, 23, 478–479.

123
188 D. B. Resnik, S. A. Elmore

Wicherts, J. M., Kievit, R. A., Bakker, M., & Borsboom, D. (2012). Letting the daylight in reviewing the
reviewers and other ways to maximize transparency in science. Frontiers in Computational
Neuroscience, 6, 20.
Williams, P., & Wager, E. (2013). Exploring why and how journal editors retract articles: Findings from a
qualitative study. Science and Engineering Ethics, 19(1), 1–11.
Wing, D. A., Benner, R. S., Petersen, R., Newcomb, R., & Scott, J. R. (2010). Differences in editorial
board reviewer behavior based on gender. Journal of Women’s Health, 19(10), 1919–1923.

123