Peer review and its contribution to manuscript quality: an Australian perspective 307

Learned Publishing, 21, 307–318

doi:10.1087/095315108X323884
Peer review and its contribution to manuscript quality: an Australian perspective
Yanping Lu
LEARNED PUBLISHING VOL. 21 NO. 3 JULY 2008 RESEARCH ARTICLE

Peer review and

its contribution to
I
ntroduction
The practice of editorial peer review
began in 1665 with the founding of the
first scientific journals: the Journal des
manuscript
Sçavans and the Philosophical Transactions of
the Royal Society,1 and has achieved near uni-
versal application over the centuries.2 The
central aim of editorial peer review, as most
quality: an
members of the research community will
agree, is to help editors select the best papers
for publication3–5 and to improve manu-
scripts in terms of scientific quality and writ-
Australian
ten presentation.6–8 Entangled with these
practical aims are its symbolic significance as
‘a mechanism of scientific self-regulation
that preserves the autonomy of science and
perspective
as a symbol of professional accountability
that insures democratic control of science’.9 Yanping LU
So strongly has peer review been seen as © Yanping Lu, 2008
the way of establishing standards in the
judgement of science that to question it is ABSTRACT. Journal reviewers’ understanding and
tantamount to questioning science itself.9 expectations of peer review, their incentives to take
Anyone questioning peer review can expect on the task, and the reasons why they sometimes
to encounter resistance from editors, review- declined were explored through a questionnaire
ers, and even authors.2 Empirical knowledge survey, with particular attention to potential
about peer review has grown over time, par- differences between education, physics, and
ticularly about specific editorial procedures; chemistry. Eighty-four senior researchers from 27
however, little is known about the way in Australian universities, who had served as reviewers
which reviewers approach reviewing, and in education, physics, and chemistry, returned a
their understanding and expectations of the completed questionnaire. There were significant
process, including the impact of peer review variations in reviewers’ expectations and
in improving the quality of manuscripts.10,11 understanding of reviewing, mostly related to
Similarly, there are many assumptions about seniority rather than discipline. They valued peer
disciplinary differences in various aspects of review as a way of maintaining the quality of
peer review12,13 but little empirical evidence. science publications, and were generally satisfied
In this article, the author used a ques- with the current system; their impression of peer
tionnaire survey to explore a group of experi- review’s effectiveness was significantly correlated
enced reviewers’ expectations and under- with their own experience. They saw reviewing as a
standing of some key aspects of peer review, professional obligation and part of their personal
paying particular attention to the differences professional development. The most frequently
between peer-review practices in education mentioned reasons for declining to review were lack
and those in physics and chemistry. of expertise and lack of time. Yanping Lu

LEARNED PUBLISHING VOL. 21 NO. 3 J U LY 2 0 0 8


308 Yanping Lu
Literature on the contribution of peer
review to improving quality
In the past two decades, some attempts have
been made to test the effect of peer review in
improving the quality of manuscripts. Sev-
eral studies have compared the initial
submissions with their published versions and
provided evidence that peer review helped
improve a manuscript in terms of its pre-
sentation and readability,14–17, reporting of
statistics,18 accuracy of content,19 and over-
the proportion all quality of the manuscript.20–22 Studies of
of favourable
reviewers’ comments revealed that most
reviewers’ comments, even those with a highly critical
comments is far tone, showed the reviewers’ commitment to
helping authors.4,23,24 Some authors also
more influential evaluated the helpfulness of the reviewers’
than the comments that they received.25–28 While
characteristics the majority of the responding authors,
especially those responding to a recent inter-
of authors and national survey,28 favoured the peer-review
manuscripts system, it should be noted that the voice of
unsuccessful authors was absent in these
studies. There was also evidence that many
authors experienced pressure to make revi-
sions to conform to editors’ or reviewers’
personal preference, or to make changes that
they felt were wrong.25,29 Even with review-
ers’ helpful comments, many authors still
struggled in revising their manuscripts,
which implied that reviewers’ contribution
to improving a manuscript was somewhat
limited.4,30,31 Jefferson and colleagues11 con-
ducted a systematic review of articles that
reported studies on peer review published in
biomedical journals up to June 2000. Of the
135 articles that they identified, only 19
Table 1. Distribution of literature on peer review (excluding books and book chapters)
Discipline Research literature
n % (n = 385)
Business, economics, management 32
Sociology, social works, humanities 31
Education 11
Science, information, mathematics 15
Medicine, biology 176
Physics, chemistry 11
Psychology 49
Cross-disciplinary, general 60
Total 385
LEARNED PUBLISHING VOL. 21
17414857, 2008, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1087/095315108X323884 by Nigeria Hinari NPL, Wiley Online Library on [13/09/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
studied the effect of any aspect of the
peer-review process, such as blinding identi-
ties of reviewers or authors, provision of
submission checklists, and training review-
ers. The majority of the studies identified by
Jefferson et al. focused on specific editorial
procedures; only two assessed the effect of
peer review on the quality of manu-
scripts.15,21 Both studies revealed beneficial
effects, but they may have only limited
generalizability, due to atypical settings.
There are also problems with reviewers.
There is ample evidence that editors’
decisions were closely related to the recom-
mendations of reviewers, and the proportion
of favourable reviewers’ comments is far
more influential than the characteristics of
authors and manuscripts.13,22,23,32–37 Ensur-
ing the quality of reviewers and reviewing is
therefore crucial. However, reviewers were
frequently criticized for their tardiness in
returning reviews or for returning reviews of
poor quality.38–40
The disciplinary focus of studies on peer
review
Existing empirical and anecdotal evidence
on editorial peer review is patchy in its cov-
erage of major disciplines. In the pursuit of a
systematic review of the literature, the
researcher developed a database of all rele-
vant literature. This database contained
more than 1,200 items that appeared in jour-
nals or books in all key disciplines from 1960
to the present, including 385 research arti-
cles, 650 essays and commentaries that
were not based on original data but also
Essays, comments, and editorials
n % (n =650+199)
8.3 31 3.7
8.1 52 6.1
2.9 31 3.7
3.9 51 6.0
45.7 261 30.7
2.9 37 4.4
12.7 68 8.0
15.6 318 37.5
100 849 100
NO. 3 J U LY 2 0 0 8

Peer review and its contribution to manuscript quality: an Australian perspective
contributed to the debate, and 199 editorials
(Table 1). Twenty books were also identified;
13 of these were focused exclusively on peer
review and the remainder were about scien-
tific communication and journal publication
in general, but also covered the topic of peer
review to a substantial extent. Most of the
authors of those works based their discus-
sions or studies on such fields as medicine,
psychology, business, and economics; there
was considerably less coverage from educa-
tion or physics and chemistry. Drawing on
this lack of research in education, physics,
and chemistry, the researcher decided to
focus her investigation on these three fields.
The questionnaire survey
This paper reports some of the findings of a
Ph.D. study undertaken by the researcher.
The study consisted of six phases: a system-
atic review of the literature; a mapping of
editorial policies of all existing English-
language scholarly journals in education; a
pre-interview survey; interviews; a post-
interview survey; and an analysis of review-
ers’ comments. This paper draws on the
findings of the pre-interview survey. Whilst
peer review is something that involves all
members of the research community, the
exercise of power is often restricted to a
dominant few. Academic élites who, in the
current context, are defined as long-standing
researchers and reviewers, are predominant
shapers of the peer-review process. It is for
this reason that the study’s target sample
was professors, associate professors, and
senior lecturers who had served as journal
reviewers in education, physics, and chemis-
try in 37 Australian universities (excluding
the author’s own university, the University
of Newcastle, to avoid bias).
By browsing university websites, a list of
930 academics was generated. Physics and
chemistry were treated as one group – physi-
cal sciences – for three reasons: (i) both are
laboratory-based; (ii) physics journals often
embrace chemistry in their scope, and vice
versa; and (iii) the total population of senior
academics in physics and chemistry is
approximately the same as that in education,
and combining physics and chemistry there-
fore facilitates comparison with education.
LEARNED PUBLISHING
17414857, 2008, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1087/095315108X323884 by Nigeria Hinari NPL, Wiley Online Library on [13/09/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
309
Every fourth name in the list (232 individu-
als in all) was sent a four-page survey (see
Appendix). The survey comprised 44 ques-
tions, divided into five sections: (1) back-
ground information about the reviewers; (2)
their opinions of peer review as a method of
research assessment in their field; (3) their
experiences as reviewers and authors; (4)
their level of success in grant application
and journal publication; and (5) an open-
ended question to allow respondents to com-
ment freely. A total of 84 usable responses
from 27 universities were received; this rep-
resents a response rate of 36.2%, which is
within the common range for mail sur-
veys.41,42 Pearson’s chi-square was calculated
to test for systematic non-response bias by
comparing the number of respondents to
that of non-respondents for each discipline.
The resultant chi-square χ2(1) =0.72, P =
0.337 indicated no evidence of non-response
bias.
The full questionnaire is given in the
Appendix.
Results
Demographic information
whilst peer
review is
Table 2 summarizes information on the
demographic and academic background of
something that
the reviewers. Most of the respondents were involves all
experienced researchers and reviewers. All members of the
of the 84 respondents had a Ph.D. degree; 37 research
(44%) had achieved their doctorate more
than 20 years ago. There were 26 professors, community, the
30 associate professors, and 28 senior lectur- exercise of
ers; 37 (44%) were from the ‘Group of Eight’ power is often
– Australia’s leading universities. No respon-
dent was younger than 35 years old; 28 restricted to a
(33.3%) were between 35 and 49, and 56 dominant few
(66.7%) were 50 or older. In the field of edu-
cation, 25 males and 22 females responded
and, in physical sciences, 34 males and 3
females. Thirty-nine (46.4%) of the respon-
dents had served as editor for peer-reviewed
journals; all had been journal reviewers for
at least 4 years and 61 (72.6%) had reviewed
for more than 10 years. They reviewed an
average of 14.2 manuscripts per year. Seven
respondents (8.3%) had served on the
assessment panels of the Australian Research
Council (ARC), the major national research
VOL. 21 NO. 3 J U LY 2 0 0 8
 17414857, 2008, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1087/095315108X323884 by Nigeria Hinari NPL, Wiley Online Library on [13/09/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
310 Yanping Lu

Table 2. Demographic information about respondents

Discipline Total
Education Physical sciences
n % n % n %
Total 47 100 37 100 84 100
Age** Under 50 8 17.0 20 54.1 28 33.3
50 or over 39 83.0 17 45.9 56 66.7
Sex** Male 25 53.2 34 91.9 59 70.2
Female 22 46.8 3 8.1 25 29.8
Academic title* Professor 10 21.3 16 43.2 26 31.0
Associate professor 16 34.0 14 37.8 30 35.7
Senior lecturer 21 44.7 7 18.9 28 33.3
Institution* Group of Eight 16 34.0 21 56.8 37 44.0
Other universities 31 66.0 16 43.2 47 56.0
Served as editors 25 53.2 14 37.8 39 46.4
Editorial board members 35 74.5 22 59.5 57 67.9
Members of ARC panels 4 8.5 3 8.1 7 8.3
External assessors for the ARC** 15 31.9 29 78.4 44 52.4
Assessors of other funding bodies 9 19.1 13 35.1 22 26.2
Members of internal research committee 34 72.3 27 73.0 61 72.6

*Significance at the 0.05 level, using Cramer’s V-test.

**Significance at the 0.01 level.

funding body of Australia; 44 (52.4%) had Reviewers’ expectations and perspectives of

been external assessors for the ARC, 14
(16.7%) of them for more than 10 years, and
they assessed an average of 7.6 grant appli-
cations per year. In addition, 22 (26.2%)
respondents served on the assessment panel
of other research funding bodies, and 61
in general, (72.6%) sat on institutional research com-
reviewers’ mittees.
overall Table 2 presents a breakdown of the demo-
graphic information by discipline. Education
impression academics tended to take up posts after work
of the experience. Compared to their physics and
effectiveness of chemistry counterparts, they were less senior
peer review was in terms of academic status, more active as
editors and editorial board members but less
highly positive so in the sector of grant assessment, and a
greater proportion of them were from uni-
versities outside the Group of Eight. The
data also indicates that physics and chemis-
try are male-dominated at senior level.
Cramer’s V-test was conducted for all these
items and significant differences were found
between discipline difference and age, sex,
academic title, institution affiliation, and
experience as ARC assessors.
peer review for journals
The respondents were asked five questions
about their expectations and perspectives of
journal peer review. Responses were rated
on a six-point Likert scale that ranged from
‘1: Not at all’ to ‘6: Very substantially’. Table
3 presents the sample means and standard
deviations of the responses to these ques-
tions and the outcomes of relevant statistical
tests that show moderate to significant rela-
tions.
In general, reviewers’ overall impression of
the effectiveness of peer review was highly
positive, with reviewers from physics and
chemistry and those who had reviewed for a
longer time rating this item significantly
higher than others. The average rating for
the second question, ‘To what extent should
peer review aim to improve manuscript qual-
ity?’, was very high (mean = 5.45). Female
reviewers demonstrated a significantly higher
expectation than males (female 5.72 com-
pared to male 5.07, P < 0.05). In contrast,
editors and more experienced reviewers

LEARNED PUBLISHING VOL. 21 NO. 3 J U LY 2 0 0 8

 17414857, 2008, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1087/095315108X323884 by Nigeria Hinari NPL, Wiley Online Library on [13/09/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Peer review and its contribution to manuscript quality: an Australian perspective 311

Table 3. Reviewers’ expectations and perspectives of peer review: means and correlations
Six-point Correlation Compare means
Likert
scale Age Years being a Sex Discipline
reviewer
Statistical test Mean (SD) Kendall’s τ t-test (2-tailed)
What is your impression of the 5.00 τ = 0.212* t(82)=1.79 t(82) = –2.23
effectiveness of peer review? (0.826) P = 0.027 P = 0.077 P = 0.029
To what extent should peer review 5.45 t(82)=2.70
aim to improve manuscript quality? (0.916) P = 0.008
To what extent can peer review 4.88
improve manuscript quality? (1.087)
How often do editors follow 4.98 τ = 0.165 τ = 0.228* t(81) = 2.15 t(81) = –2.27
reviewers’ recommendations? (0.604) P = 0.078 P = 0.019 P = 0.035 P = 0.026
Do reviewers bring different 4.93
expectations to grant and journal (1.045)
article review?

(reviewed for 10 years or longer) were less whether they had ever refused to act as a
supportive of the idea that it was up to peer reviewer, either for some papers or for some
review to improve quality. To clarify this journals. Those answering ‘yes’ were asked
point, a reviewer wrote that ‘too many to give two reasons for their refusal. Fifty- although
authors used peer review to improve their four (64.3%) responded ‘yes’, and gave a
papers . . and often submitted papers that total of 100 reasons (Table 4). In 37 cases reviewers
were not up to the standard of submission’. reviewers chose not to review some papers generally
The third question ‘To what extent can because they were outside their field of expected peer
peer review improve manuscript quality?’ expertise; in 23 cases they did not have
was rated somewhat lower than the second enough time; and 4 indicated conflict of review to
one, which suggests that, although reviewers interest because they knew the authors. Sim- improve
generally expected peer review to improve ilarly, in 14 cases the reviewers declined to manuscript
manuscript quality, they were less certain of review for some journals due to lack of
its ability to achieve this. Whether reviewers expertise; in 10 cases because of time pres-
quality, they
felt that editors followed their recommenda- sure; and in 12 because they thought the were less
tions (question 4) was significantly related quality of the journal was poor or they had certain of its
to reviewers’ level of seniority, in terms of been treated badly by the journal in the past. ability to
age and years of reviewing; physicists and The second question asked reviewers to
chemists rated it significantly higher than give two reasons why they did agree to achieve this
education academics (P < 0.05). Sixty-one
(72.6%) of the journal reviewers also had Table 4. Reasons for declining to review
the experience of assessing research projects
for grant application. They generally brought Reasons Frequency (%)
different expectations to the review of grants For paper For journal Total
and journal articles (mean = 4.93), but no
significant difference was found by disci- Lack of 37 (57.8%) 14 (38.9%) 51 (51%)
expertise
pline, seniority, or sex.
Lack of 23 (35.9%) 10 (27.8%) 33 (33%)
time
Reviewers’ experiences of peer review Other 4 (6.3%) 12 (33.3%) 16 (16%)
Questions in section 3 sought responses on Total 64 (100%) 36 (100%) 100 (100%)
aspects of the reviewers’ experiences of peer answers
review. The first question asked reviewers offered

LEARNED PUBLISHING VOL. 21 NO. 3 J U LY 2 0 0 8


312 Yanping Lu
Table 5. Reasons for agreeing to review
Reasons
Professional responsibility and obligation
Personal professional development
Maintaining quality of published papers
Being interested or expert in the topic
Contributing to the peer-review system
Total answers offered
review. A total of 134 reasons was given;
these were grouped into five categories (Table
5). Each reason was coded once per individ-
ual, so the total number of reasons exceeded
the total number of respondents. The
reviewers regarded their contribution firstly
reviewers did as part of their professional responsibility,
not generally which was mentioned 48 times (35.8%).
regard Some reviewers regarded peer review as a
good opportunity for professional develop-
reviewing as ment, in the sense of ‘keeping abreast of
a means to current developments in the field’, ‘getting a
better feeling of what papers are accepted’,
improve the
and gaining ‘something to put on CV’. Some
quality of reviewers agreed to contribute their time
manuscripts or and expertise with the aim of keeping the
to maintain the peer-review system going and to ‘share
the load with other academics’. While the
quality of reviewers deemed reviewing as ‘an expecta-
journal tion of my academic job’, they did not
publications generally regard it as a means to improve the
quality of manuscripts or to maintain the
quality of journal publications, reasons
which were mentioned by only 12 of them
(9%).
The next three questions solicited infor-
mation about guidelines for reviewers.
Nearly 80% of the journals for which respon-
dents reviewed almost always provided them
with guidelines. All but two reviewers
almost always followed the guidelines, and
65 (77.4%) felt the guidelines helped them
generate better reviews, with the relation-
ship between their following the guidelines
provided and ultimately generating better
reviews significantly positive, r = 0.35, P =
0.02. There was no significant difference in
reviewers’ perspectives by discipline.
Next, reviewers were asked whether they
had been paid by any journal for reviewing;
LEARNED PUBLISHING VOL. 21
17414857, 2008, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1087/095315108X323884 by Nigeria Hinari NPL, Wiley Online Library on [13/09/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Frequency %
48 35.8
29 21.6
12 9.0
18 13.4
27 20.1
134 100.0
no one said ‘yes’. The last question of Sec-
tion 3 asked reviewers whether grant or
journal reviewing was the more important
activity for them. Among the 58 reviewers
who responded to this question, 35 review-
ers thought journal peer review was more
important, as opposed to 7 who thought
grant peer review was more important. Six-
teen felt the two were equally important
activities.
Respondents’ overall level of success as
researchers
The next section of the questionnaire looked
at respondents’ own level of success in terms
of gaining grants and publishing their
research outcomes (Table 6). For the period
2001–5, 67 (79.8%) of the respondents won
one or more competitive grants from the
ARC or the Australian National Health
and Medical Research Council (NHMRC);
40 (47.6%) won competitive grants from
other national funding bodies; 45 (53.6%)
received institutional research grants; and
83 (98.8%) published articles in peer-
reviewed journals. The success rates self-
reported by individual respondents (e.g. pub-
lications against submissions) were high for
most, ranging from 38 to 100%. As shown in
Table 6, on average, academics from physics
and chemistry reported higher rates of suc-
cess in publication (93.6%) than those in
education (90.9%), and in winning grants
from the ARC and NHMRC (64.6% com-
pared to 44.7% – statistically significant, P
< 0.05) and other national funding bodies
(70.8% compared to 67.1%). In contrast,
education academics won significantly more
grants from their own institutions (95.1%
NO. 3 J U LY 2 0 0 8

Peer review and its contribution to manuscript quality: an Australian perspective
Table 6. Respondents’ level of success
Education
No. of Average success
respondents rate (%)
ARC and NHMRC grants 30 44.7
Other national grants 20 67.1
Internal grants 20 95.1
Publications 46 90.9
compared to 80.3%, P < 0.05). In addition,
Pearson coefficient correlations were calcu-
lated to test for the impact of personal
success on their impression of the peer-
review system. Respondents’ overall impres-
sion of the effectiveness of peer review was
significantly related to their success rates in
gaining grants from the national funding
bodies of ARC and NHMRC (r = 0.45, P <
0.001) and successful publication (r = 0.24,
P < 0.05).
Reviewers’ comments
Lastly, respondents were invited to make
any additional comments or suggestions
about peer review; 46 of them shared their
thoughts. The reviewers generally valued
peer review as ‘the key to maintaining qual-
ity’ and supported its function in providing
an imprimatur. Conversely, some reviewers
maintained that subjectivity was inevitable
in peer judgement; the process sometimes
took an unnecessarily long time and sub-
stantially slowed the publication process;
some noted that, as authors, they had
encountered ‘very irresponsible reviewers’
and some reviews were ‘disappointingly brief’
and ‘surprisingly unhelpful’. One reviewer
commented:
The peer-review process works well when
reviewers are concerned to promote schol-
arship and knowledge in an area of study.
The peer-review process does not work
well when it is used by reviewers to ex-
clude promotion of alternative views other
than their own or when reviewers are not
sensitive to the feelings of authors and
write inappropriate and sometimes offen-
sive reviews.
LEARNED PUBLISHING
17414857, 2008, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1087/095315108X323884 by Nigeria Hinari NPL, Wiley Online Library on [13/09/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
313
Physics and chemistry Comparisons
No. of Average success t-test (2-tailed)
respondents rate (%)
37 64.6 t(65) = –2.328 P = 0.023
20 70.8 t(38) = –0.287 P = 0.776
25 80.3 t(43) = 2.089 P = 0.043
37 93.6 t(65) = –0.972 P = 0.334
Another reviewer described his experience
as an author, feeling disadvantaged by the
anonymous reviewing system:
On a number of occasions, I have submit-
ted a paper to a journal, had it rejected,
submitted it again to a different journal
and had it accepted. This is not explained
by the rejecting journals having extremely
high standards, but by unprofessional ref-
ereeing. The anonymous review, which I
guess was imported from science, has
major flaws in my field, which is badly
affected by ideological wars. Current
practice of anonymous reviewers and some reviewers
non-anonymous authors is inappropriate,
and encourages reviewers to hide behind
maintained
that subjectivity
their anonymity.
Several reviewers expressed their concern was inevitable
about the lack of induction into the review in peer
process – academics should ‘have the neces- judgement
sary skills to peer review but these skills are
rarely (if ever) discussed’. There was also
criticism of publishers in that they ‘made use
of academics without paying them but yet
charged academics and libraries extortionate
fees to access works whose quality was
ensured by academics’.
Many respondents pointed out that it was
increasingly difficult to find time to review
promptly and thoroughly. Some stated they
always tried to give timely and useful feed-
back because they understood ‘the review
process can take too long’. Again, some
reviewers emphasized the heavy workload
and time pressure they faced and appealed
for greater recognition of their contribution
by employing institutions, as ‘an important
aspect of academics’ work’. A reviewer sug-
gested a possible solution to the shortage of
time:
VOL. 21 NO. 3 J U LY 2 0 0 8

314 Yanping Lu
There is a lot of duplication and time
wasted through submissions to different
journals, each conducting its own peer re-
view. A central agency conducting peer
review that would make it available to
journals would improve this situation.
Discussion
This study has its limitations. For example,
the sample was small and entirely limited to
successful Australian universities, and the response
researchers and rate was not high enough for safe generaliza-
experienced tion. This study did not attempt to study less
experienced and less powerful members of
reviewers were
the research community or those from other
generally disciplines. Yet the responses still conveyed
satisfied important messages. While the respondents’
overall attitude towards the effectiveness of
with the
peer review differed significantly by disci-
effectiveness of pline, the expectations and understandings
the current that they brought to reviewing did not.
system of Their approach to reviewing was primarily
related to their level of seniority as academ-
editorial peer ics and as reviewers, rather than to their
review discipline.
The successful researchers and experi-
enced reviewers in this study were generally
satisfied with the effectiveness of the current
system of editorial peer review, irrespective
of their discipline, sex, or academic status.
Most respondents rated ‘should improve
manuscript quality’ highly as the main func-
tion of peer review (though there was a lack
of consistency), but this was not their pri-
mary reason for undertaking the job of
reviewing (Table 5). The system of peer
review was put in place by journals in the
17th century in order to assist editors in the
selection of manuscripts for publication.1,43
Numerous editorials in various journals indi-
cate that journal editors regard peer review
as the best tool for ensuring the scientific
quality of publications – a tool for both
gatekeeping and quality control. This study
confirmed the idea that reviewers think so
too.
Do academic authors expect peer review
to improve the quality of their own manu-
scripts? It would seem that many do. Several
studies have shown that authors generally
expect their submissions to be improved in
the peer review process.8,28,44,45 Authors
LEARNED PUBLISHING VOL. 21
17414857, 2008, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1087/095315108X323884 by Nigeria Hinari NPL, Wiley Online Library on [13/09/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
were reported as being dissatisfied with the
process because they thought the comments
they received from reviewers or editors were
unhelpful.29,46 However, from the outset
journals have placed the ultimate responsi-
bility for the quality of the manuscript
squarely with the author. This divergence in
expectation can lead to considerable frustra-
tion for authors, reviewers, and editors.
The reviewers’ reasons for agreeing or
declining to review conformed to the find-
ings of a recent study in medicine. In a
survey of 551 medical journal reviewers, Tite
and Schroter47 found that the most frequent
reasons for agreeing to review were: contri-
bution of the paper to the field; relevance of
the topic to own field of interest; opportu-
nity to learn from the paper; and sense of
professional duty, while the main reasons for
declining to review were all related to lack of
time: conflict with other workload; having
to review too many manuscripts for other
journals; and tight deadlines for completing
the review.
The reviewers’ responses to the question
of why they declined to review raised a ques-
tion that was not anticipated when the
survey was developed. A majority of the
reviewers indicated that they had received
some manuscripts for which the subject was
outside their field of expertise (51%, see
Table 4). This would seem to be a common
problem; author responses as reported in
several previous studies suggested that edit-
ors might not necessarily know who would
be a competent reviewer in a particular
area,25,27,29,48,49 which, according to Calla-
ham and Tercier,50 is a ‘crucial limitation in
the peer review process’.
The reviewers in this study took the task
of reviewing seriously. Their contribution in
reviewing is all the more precious in view of
the fact that they were seldom rewarded
explicitly, either by their institutions or by
the journals. However, when time pressure is
predominant, as suggested by Tite and
Schroter,46 what could the author expect
from the reviewer other than a tick in the
box of ‘Accept’ or ‘Reject’ and probably a
line or two pointing out the most obvious
flaws of the article?
At present, peer review is supposed to
identify high-quality research. However, so
NO. 3 J U LY 2 0 0 8

Peer review and its contribution to manuscript quality: an Australian perspective
far no agreement has been reached on objec-
tive ways to measure improvement in
manuscript quality, nor is it possible to dis-
tinguish between the quality improvement
brought about by peer review and that
resulting from subsequent editing. As a
result, it is not possible objectively to deter-
mine the effect of peer review on manuscript
quality. Peer review still has to be regarded
as an under-studied process with uncertain
outcomes.
References
1. Zuckerman, H. and Merton, R.K. 1971. Patterns of
evaluation in science: institutionalization, structure
and functions of the referee systems. Minerva, 9:
66–100.
http://dx.doi.org/10.1007/BF01553188
2. Jefferson, T., Wager, E. and Davidoff, F. 2002. Measur-
ing the quality of editorial peer review. Journal of the
American Medical Association, 287: 2786–90.
http://dx.doi.org/10.1001/jama.287.21.2786
3. Faupel, C. 2002. Editor’s comments. Sociological
Inquiry, 72: 1–3.
http://dx.doi.org/10.1111/1475–682X.00001
4. Gosden, H. 2003. ‘Why not give us the full story?’
Functions of referees’ comments in peer reviews of sci-
entific research papers. Journal of English for Academic
Purposes, 2, 87–101.
http://dx.doi.org/10.1016/S1475–1585(02)00037–1
5. Miser, H.J. 1998. The easy chair: journal editing as I
see it. Interfaces, 28: 115–23.
6. Abby, M., Massey, M.D., Galandiuk, S. and Polk,
H.C. 1994. Peer review is an effective screening pro-
cess to evaluate medical manuscripts. Journal of the
American Medical Association, 272: 105–7.
http://dx.doi.org/10.1001/jama.272.2.105
7. Horrobin, D. F. 1990. The philosophical basis of peer
review and the suppression of innovation. Journal of
the American Medical Association, 263: 1438–41.
http://dx.doi.org/10.1001/jama.263.10.1438
8. Sweitzer, B.J. and Cullen, D.J. 1994. How well does a
journal’s peer review process function? A survey of
authors’ opinion. Journal of the American Medical Asso-
ciation, 272: 152–3.
http://dx.doi.org/10.1001/jama.272.2.152
9. Chubin, D.E. and Hackett, E.J. Peerless Science: Peer
Review and US Science Policy. Albany, State University
of New York Press, 1990, p. 4.
10. Fletcher, R. and Fletcher, S. 1997. Evidence for the
effectiveness of peer review. Science and Engineering
Ethics, 3: 35–50.
http://dx.doi.org/10.1007/s11948-997-0015-5
11. Jefferson, T., Wager, E. and Davidoff, F. 2002. Effects
of editorial peer review: A systematic review. Journal
of the American Medical Association, 287: 2784–6.
URL: http://jama.ama-assn.org/cgi/content/full/287/
21/2784
12. Cole, S., Simon, G. and Cole, J. R. 1988. Do journal
rejection rates index consensus? American Sociological
Review, 53: 152–6.
13. Hargens, L.L. 1988. Scholarly consensus & journal
rejection rates. American Sociological Review, 53:
139–51.
LEARNED PUBLISHING
17414857, 2008, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1087/095315108X323884 by Nigeria Hinari NPL, Wiley Online Library on [13/09/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
315
14. Biddle, C. and Aker, J. 1996. How does the peer
review process influence AANA Journal article read-
ability? Journal of the American Association of Nurse
Anesthetists, 64: 65–8.
15. Pierie, J.-P.E.N., Walvoort, H.C. and Overbeke,
A.J.P.M. 1996. Readers’ evaluation of effect of peer
review and editing on quality of articles in the
Nederlands Tijdschrift voor Geneeskunde. The Lancet,
348: 1480–3.
http://dx.doi.org/10.1016/S0140-6736(96)05016-7
16. Metoyer-Duran, C. 1993. The readibility of published,
accepted, and rejected papers appearing in College &
it is not possible
Research Libraries. College & Research Libraries, 54: objectively to
517–25.
17. Roberts, J.C., Fletcher, R.H. and Fletcher, S.W. 1994. determine the
Effects of peer review and editing on the readability of
articles published in Annals of Internal Medicine. Jour-
effect of peer
nal of the American Medical Association, 272: 119–21. review on
http://dx.doi.org/10.1001/jama.272.2.119
18. Cobo, E., Selva-O’Callagham, A., Ribera, J.-M., manuscript
Cardellach, F., Dominguez, R., and Vilardell, M. 2007.
Statistical reviewers improve reporting in biomedical
quality
articles: a randomized trial. PLoS One, 3: 1–8.
http://dx.doi.org/10.1371/journal.pone.0000332
19. Purcell, G.P., Donovan, S.L. and Davidoff, F. 1998.
Changes to manuscripts during the editorial process:
characterizing the evolution of a clinical paper. Jour-
nal of the American Medical Association, 280: 227–8.
http://dx.doi.org/10.1001/jama.280.3.227
20. Campanario, J.M. 1996. Have referees rejected some
of the most-cited articles of all times? Journal of the
American Society for Information Science, 47: 302–10.
http://dx.doi.org/10.1002/(SICI)1097–4571(199604)4
7:4<302::AID-ASI6>3.0.CO;2-0
21. Goodman, S.N., Berlin, J., Fletcher, S.W. and Fletcher,
R.H. 1994. Manuscript quality before and after peer
review and editing at Annals of Internal Medicine.
Annals of Internal Medicine, 121: 11–21.
http://www.annals.org/cgi/content/full/121/1/11
22. Smigel, E.O. and Ross, H.L. 1970. Factors in the edi-
torial decision. The American Sociologist, 5: 19–21.
23. Beyer, J.M., Chanove, R.G. and Fox, W.B. 1995. The
review process and the fates of manuscripts submitted
to AMJ. Academy of Management Journal, 38:
1219–60.
24. Jauch, L.R. and Wall, J.L. 1989. What they do when
they get your manuscript: a survey of Academy of
Management reviewer practices. Academy of Manage-
ment Journal, 32: 157–73.
25. Bedeian, A.G. 2003. The manuscript review process:
the proper roles of authors, referees, and editors. Jour-
nal of Management Inquiry, 12: 331–8.
http://dx.doi.org/10.1177/1056492603258974
26. Rowlands, I. and Nicholas, D. 2006. The changing
scholarly communication landscape: an international
survey of senior researchers. Learned Publishing, 19:
31–55.
http://dx.doi.org/10.1087/095315106775122493
27. Thyer, B.A. and Myers, L.L. 2003. An empirical eval-
uation of the editorial practices of social work jour-
nals. Journal of Social Work Education, 39: 125–40.
Available at http://www.accessmylibrary.com/coms2/
summary_0286–23466319_ITM
28. Mark Ware Consulting. Peer Review in Scholarly
Journals: Perspective of the scholarly community – An
International Study. London, Publishing Research
Consortium, 2007. Available at http://www.publishing
VOL. 21 NO. 3 J U LY 2 0 0 8

316 Yanping Lu
r e s e a r c h . n e t / d o c u m e n t s / Pe e r R e v i e w Fu l l P R C
Report-final.pdf
29. Bradley, J.V. 1981. Pernicious publication practices.
Bulletin of the Psychonomic Society, 18: 31–4.
http://www.psychonomic.org/search/view.cgi?id=124
78
30. La Forge, J. and Coelho, R.J. 1998. Writing for publi-
cation: advice from authors. Journal of Applied Rehabil-
itation Counselling, 29: 35–9.
31. Swan, A. 1999. What authors want: the ALPSP
research study on the motivations and concerns of
contributors to learned journals. Learned Publishing,
12: 170–2.
http://dx.doi.org/10.1087/09531519950145742
32. Bakanic, V., McPhail, C. and Simon, R.J. 1987. The
manuscript review and decision-making process.
American Sociological Review, 52: 631–42.
33. Callaham, M.L., Baxt, W.G., Waeckerle, J.F. and
Wears, R.L. 1998. Reliability of editors’ subjective
quality ratings of peer reviewers of manuscripts. Jour-
nal of the American Medical Association, 280: 229–31.
http://dx.doi.org/10.1001/jama.280.3.229
34. Fogg, L. and Fiske, D.W. 1993. Foretelling the judg-
ments of reviewers and editors. American Psychologist,
48: 293–4.
http://dx.doi.org/10.1037/0003-066X.48.3.293
35. Gilliland, S.W. and Cortina, J.M. 1997. Reviewer and
editor decision making in the journal review process.
Personnel Psychology, 50: 427–52.
http://dx.doi.org/10.1111/j.1744-6570.1997.tb00914.x
36. Henderson, A. 2001. Undermining peer review. Soci-
ety, 38: 47–54.
http://dx.doi.org/10.1007/s12115-001-1040-7
37. Murphy, T.M. and Utts, J.M. 1994. A retrospective
analysis of peer review at Physiologia Plantarum.
Physiologia Plantarum, 92: 535–42.
http://dx.doi.org/10.1111/j.1399–3054.1994.tb03021.x
38. Ellison, G. 2002. The slowdown of the economics
publishing process. Journal of Political Economy, 110:
947–93.
http://dx.doi.org/10.1086/341868
39. Mason, P.M., Steagall, J.W. and Fabritius, M.M. 1992.
Publication delays in articles in economics what to do
about them. Applied Economics, 24: 859–74.
http://dx.doi.org/10.1080/00036849200000054
40. Tight, M. 2003. Reviewing the reviewers. Quality in
Higher Education, 9: 295–303.
http://dx.doi.org/10.1080/1353832032000151157
41. Arleck, P.L. and Settle, R.B. The survey research hand-
book. Irwin, IL, Homewood, 1985.
LEARNED PUBLISHING VOL. 21
17414857, 2008, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1087/095315108X323884 by Nigeria Hinari NPL, Wiley Online Library on [13/09/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
42. Kerlinger, F.N. and Lee, H.B. Foundations of Behavioral
Research, 4th edition. Fort Worth, TX, Harcourt,
2000.
43. Burnham, J.C. 1990. The evolution of editorial peer
review. Journal of the American Medical Association,
263: 1323–9.
http://dx.doi.org/10.1001/jama.263.10.1323
44. Garfunkel, J.M., Lawson, E.E., Hamrick, H.J. and
Ulshen, M.H. 1990. Effect of acceptance or rejection
on the author’s evaluation of peer review of medical
manuscripts. Journal of the American Medical Associa-
tion, 263: 1376–8.
http://dx.doi.org/10.1001/jama.263.10.1376
45. Laband, D.N. 1990. Is there value-added from the
review process in economics? Preliminary evidence
from authors. Quarterly Journal of Economics, 105:
341–52.
46. Wilson, D.G. 1999. A bill of rights for authors?
Learned Publishing, 12: 259–63.
http://dx.doi.org/10.1087/09531519950145652
47. Tite, L. and Schroter, S. 2007. Why do peer reviewers
decline to review? A survey. Journal of Epidemiology
and Community Health, 61: 9–12.
http://dx.doi.org/10.1136/jech.2006.049817
48. Epstein, W. M. 1990. Confirmational response bias
among social work journals. Science, Technology, &
Human Values, 15: 9–38.
http://dx.doi.org/10.1177/016224399001500102
49. Nicholas, D., Jamali, H.R., Huntington, P. and
Rowlands, I. 2006. On the tips of their tongues:
authors and their views on scholarly publishing.
Learned Publishing, 19: 193–203.
http://dx.doi.org/10.1087/095315106777877494
50. Callaham, M.L. and Tercier, J. 2007. The relationship
of previous training and experience of journal peer
reviewers to subsequent review quality. PloS Medicine,
4: 32–40.
http://dx.doi.org/10.1371/journal.pmed.0040040
Yanping Lu
Centre for the Study of Research Training &
Impact
Faculty of Education & Arts
NUPSA 2007 University of Newcastle,
Australia
Email: yanping.lu@newcastle.edu.au
NO. 3 J U LY 2 0 0 8
17414857, 2008, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1087/095315108X323884 by Nigeria Hinari NPL, Wiley Online Library on [13/09/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Appendix
17414857, 2008, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1087/095315108X323884 by Nigeria Hinari NPL, Wiley Online Library on [13/09/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License