See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/248962982

Examiner comment on the literature review in Ph.D. theses

Article  in  Studies in Higher Education · June 2007

DOI: 10.1080/03075070701346899

CITATIONS READS
39 7,689

4 authors, including:

Allyson Holbrook Sid Bourke

The University of Newcastle, Australia The University of Newcastle, Australia
81 PUBLICATIONS   1,070 CITATIONS    52 PUBLICATIONS   1,489 CITATIONS   

SEE PROFILE SEE PROFILE

Hedy Fairbairn
The University of Newcastle, Australia
15 PUBLICATIONS   279 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Instrumental Music Pedagogy View project

PhD Study - Dispositions Towards Learning in Masters Students View project

All content following this page was uploaded by Hedy Fairbairn on 25 August 2015.

The user has requested enhancement of the downloaded file.

This article was downloaded by: [University of Newcastle (Australia)]
On: 22 October 2014, At: 21:54
Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered
office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Studies in Higher Education

Publication details, including instructions for authors and
subscription information:
http://www.tandfonline.com/loi/cshe20

Examiner comment on the literature

review in Ph.D. theses
a a a a
Allyson Holbrook , Sid Bourke , Hedy Fairbairn & Terry Lovat
a
University of Newcastle , New South Wales, Australia
Published online: 12 Jun 2007.

To cite this article: Allyson Holbrook , Sid Bourke , Hedy Fairbairn & Terry Lovat (2007) Examiner
comment on the literature review in Ph.D. theses, Studies in Higher Education, 32:3, 337-356, DOI:
10.1080/03075070701346899

To link to this article: http://dx.doi.org/10.1080/03075070701346899

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the
“Content”) contained in the publications on our platform. However, Taylor & Francis,
our agents, and our licensors make no representations or warranties whatsoever as to
the accuracy, completeness, or suitability for any purpose of the Content. Any opinions
and views expressed in this publication are the opinions and views of the authors,
and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content
should not be relied upon and should be independently verified with primary sources
of information. Taylor and Francis shall not be liable for any losses, actions, claims,
proceedings, demands, costs, expenses, damages, and other liabilities whatsoever
or howsoever caused arising directly or indirectly in connection with, in relation to or
arising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Any
substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,
systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Conditions of access and use can be found at http://www.tandfonline.com/page/terms-
and-conditions
 Studies in Higher Education
Vol. 32, No. 3, June 2007, pp. 337–356

Examiner comment on the literature

review in Ph.D. theses
Allyson Holbrook*, Sid Bourke, Hedy Fairbairn and Terry
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

Lovat
University of Newcastle, New South Wales, Australia
Taylor
Studies
10.1080/03075070701346899
CSHE_A_234583.sgm
0307-5079
Original
Society
302007
32
Allyson.Holbrook@newcastle.edu.au
AllysonHolbrooka
00000June
and
in
for
Article
Higher
(print)/1470-174X
Francis
Research
2007 Education
Ltd
into Higher
(online)
Education

The review of literature, so central to scholarly work and disciplined inquiry, is expected of the
Ph.D. student, but how far along the road are they expected to travel? This article investigates the
expectations of ‘the literature’ in research and scholarship at Ph.D. level from the examiner and
assessment perspective. The analysis draws on the examiner report data for 501 candidates (1310
reports) across five Australian universities. On average about one-tenth of an examiner report is
devoted to the literature and examiners provide detail about coverage, types of errors and the nature
of use of the literature. It was the latter type of comment about coherent and substantive use of the
literature that provided the most information about ‘expectation’. Examiners identified ‘working
understanding’, ‘critical appraisal’ of the body of literature, ‘connection of the literature to findings’,
and ‘disciplinary perspective’ as key indicators of performance in the candidate’s use of the litera-
ture. While examiners appeared to anticipate that all these elements should be present in scholarly
work (and identified them in the best theses), they were prepared to accept less for a barely passable
thesis, but pressed for at least some demonstration of critical appraisal.

Introduction
In academe scholarly activity is inextricably interlinked with ‘research’. Peer review is
the main process by which the quality and contribution of research is judged, and is
an activity undertaken by academics as part of their work. The way in which academ-
ics work is captured in the concept of ‘scholarship’ (Brew, 2001), and this includes
their ‘professional approach’, and the qualities of ‘meticulousness and rigour associ-
ated with academic inquiry and reporting’ (p. 45). The skills and understandings
required for disciplined scholarly inquiry are acquired through undertaking research
in a defined field or fields, usually through a research degree such as a Ph.D. Even so,
these capacities, especially the ability to evaluate research and write in a coherent way

*Corresponding author: Centre for the Study of Research Training & Impact, Faculty of Education
& Arts, University of Newcastle, Newbold Building, Corner Gavey & Frith Street, Mayfield, NSW,
Australia. Email: allyson.holbrook@newcastle.edu.au

ISSN 0307-5079 (print)/ISSN 1470-174X (online)/07/030337–20

© 2007 Society for Research into Higher Education
DOI: 10.1080/03075070701346899
 338 A. Holbrook et al.

about it, are not easily attained or accessed, and they require time and immersion in
research activity to form and develop. Supervisors of research students need to assist
them to read the professional literature, and to demonstrate, ‘as explicitly as possible,
how the scholarly community exercises its judgements, and the more implicit aspects
of scholarly activity’ (Delamont et al., 1997, p. 105). Candidates need to know how
peer review works and to demonstrate this in the production of a thesis that is ‘judged
on its ability to acknowledge the existing field adequately as well as on its ability to
add something new’. The examiner is also assessing the potential of the candidate as
a ‘peer’, i.e. their readiness to enter the academy (Yates, 2004, p. 81). A credible
doctoral performance in evaluating the literature, identifying the strengths and weak-
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

nesses of previous studies, and their contribution as well as one’s own, is to demon-
strate a capacity for peer review. To ‘merit publication’ is to satisfy one’s peers. The
Quality Assurance Agency (QAA) in the UK notes the identifying criterion for
doctoral work is that it leads to the creation and interpretation of new knowledge
through original research, sufficient to ‘satisfy peer review’ and ‘merit’ publication
(QAA, 2001). Doctoral examination has been seen to be less a form of educational
assessment than a form of peer review (Phillips, 1994, p. 137).
The review of literature, so central to scholarly work and disciplined inquiry, is
expected of the doctoral student, but how far along the road are they expected to
travel in this scholarly endeavour? What exactly are they expected to achieve? This
article addresses this question through the analysis of the examination reports on
Ph.D. doctoral theses across disciplines in five Australian universities.
The main question addressed is: What are the expectations of the literature in
research and scholarship at doctoral level, and how are these evidenced in thesis
examination? Questions specific to this are: Relative to other elements in examiner
reports on the Ph.D., how much emphasis is given to the literature and what forms
does it take; and What are the qualities that examiners identify as indicators of strong
and poor use of the literature in a thesis, and is this reflected in their recommendation?
After an analysis of the literature and a section on method, this article moves on to
an analysis that identifies how much examiners say about the literature, how closely
this relates to the final recommendation for the thesis, and what can be identified
about their expectations of the treatment of the literature.

The role of the literature review in doctoral candidature

While ‘the literature’ may feature in several places, and in various forms of presenta-
tion throughout a thesis or dissertation (Bruce, 1994, p. 218), there are some common
conventions and expectations, and the most traditional of these is summarised by Ely
et al. (1997):
there comes a time, usually in the drafting of the working text, when a researcher discusses
how ‘the literature in the field’ and other sources of information led to understanding …
The language of theory, in fact, often stands like parentheses at either end of academic
research reports: a theoretical framework is proposed at the beginning and a theoretical
discussion synthesizes findings and their significance at the end. (p. 225)
 Examiner comment on Ph.D. theses 339

In its most straightforward manifestation this would translate, particularly in the

sciences and social sciences, to an early chapter devoted to a thorough review of rele-
vant literature, and a final discussion in a chapter that revisits the literature. In large
part the expectations of the form of the thesis are shaped by the roles ascribed to the
literature in scholarly work in the different disciplines. Ultimately candidates will
need to demonstrate that they have made decisions about what literature to include,
and to ‘derive and to explore ideas through the literature rather than passively report-
ing it’ (Delamont, et al., 1997, p. 66).
The demands of a Ph.D. thesis, particularly the demand that it make a significant
contribution to a field or discipline, will call for a scholarly analysis of a body of
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

research. For many candidates scholarly critical appraisal will prove difficult to
conceptualise and challenging to accomplish. Research students cannot be automat-
ically expected to enter their candidature with the skills, or even the understandings,
in place to undertake a scholarly review (Bruce, 1994; Delamont et al., 1997).
In a phenomenographic study of research students’ early experiences of the litera-
ture review, Bruce (1994) found that the students (n = 41) had to ‘attain significant
shifts in understanding’ in order to ‘conceive’ of the literature review as a ‘tool for
demonstrating’ their grasp of theory, and that this condition was not much in
evidence at the early stages of candidature (p. 227). Achieving these higher levels of
understanding is allied very closely to awareness of, and an enculturation in, the
expectations placed on the academic researcher. Delamont et al. (1997) noted that
‘very few’ students ‘have thought about … what the finished product is for’, or have
recognised ‘that displaying the ability to synthesize the literature is one of the skills
they will be judged on’, and that ‘many examiners see that synthesis is a crucial part
of a good thesis’ (p. 59).
While there are many suggestions in discipline-specific methods texts, general
thesis ‘guides’ and on websites about how to proceed with collecting and organising
the literature, and what strategies to adopt in its use, there is little empirical research
about learning and skills development at the research degree level. The expectations
about the literature, for example, ‘synthesis’, demonstration of critical ‘judgement’
and ‘theoretical application’, have not been supported by studies that determine if
those expectations are followed through consistently in assessment. This article
attempts to address the gap.
Empirical research that focuses specifically on research degree examination is
sparse. The majority of studies that look at viva proceedings are from the UK, and
those that focus on written reports are from Australia—a feature which reflects the
major difference in process between nations. In Australia, typically there is no oral
examination or viva as a component of Ph.D. examination. There have been studies
that have drawn on interviews or surveys with examiners, supervisors and students,
observed examination proceedings, and collected the text of examiner reports in an
attempt to elicit what examiners are looking for in a thesis (Nightingale 1984;
Pitkethly & Prosser, 1995; Johnston, 1997; Delamont et al., 1997; Mullins & Kiley,
2002; Winter et al., 2000; Denicolo, 2003; Trafford, 2003; Tinkler & Jackson,
2004). It has not been typical for these studies to be large-scale or to span disciplines
 340 A. Holbrook et al.

or institutions systematically. In many studies samples have been based on conve-

nience, for reasons such as difficulty in gaining access to viva proceedings (Trafford,
2003). Also, many institutions do not keep or have accessible documentation. Some
are single institution studies (Pitkethly & Prosser, 1995; Johnston, 1997; Winter
et al., 2000). The most extensive examination study focused on examination process
in the UK and the role of the viva, which spanned 20 institutions (Tinkler & Jackson,
2004). There are also several articles that systematically scope or review the issue
(e.g. Morley et al., 2002; Lawson et al., 2003).
Of the studies that analyse the text of written reports on Ph.D. theses (Pitkethly &
Prosser, 1995; Johnston, 1997), and one that examined reports on Master’s-level
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

research (Hansford & Maxwell, 1993), only the last of these identified the literature
review and conceptual framework as substantial themes. In this study, 34% of exam-
iners made critical comment on the literature review. The two ‘most common criti-
cisms’ were the ‘failure to use recent literature and the inability to critically assess the
existing literature’ (p. 179). The study drew on 125 students in education in one insti-
tution over a four-year period. The authors noted that, if their study had ‘endeav-
oured to identify the nature of examiner comment’ relating to high-quality theses,
their ‘belief’ was that a number of such comments would refer to aspects of ‘theoret-
ical or conceptual frameworks’ (p. 181). Yates (2004), also from the field of educa-
tion, drew on her experience as an assessor of top quality theses for a national prize
to note that the best theses exhibit ‘mastery’, not only of techniques and presentation,
but also in work that convincingly demonstrates that the candidate ‘knows what the
field has been saying and doing and that can convincingly argue for an approach that
goes beyond that’ (p. 75).
While guides for Ph.D. candidates, institutional guidelines for examiners and
research methods texts all identify that the literature review is pivotal to disciplined
inquiry in established fields of research, the existing studies that attend to Ph.D.
examination, in whole or part, do not provide substantive evidence of its role and rela-
tive importance in examiner judgement.

Approach
This article reports on a section of a study that investigates doctoral examination in
Australia through the use of examiner reports (Holbrook & Bourke, 2004; Holbrook
et al., 2004a, b). The analysis draws on the examiner report data for 501 candidates
(1310 reports) across five Australian universities (two from the research-intensive
‘Group of Eight’ universities, two from a group calling themselves the ‘Innovative
Research’ universities, and one other). Institutions were asked to provide the exam-
iner reports on their most recent 100 thesis examinations, thus representing a cross-
section of fields of study. Two institutions used two examiners and the remainder
used three for all but a few candidates. Almost all examiners were external to the insti-
tution, and 50% of examiners were recruited from outside Australia.
The examiner reports range, on average, between two and three pages in length in
standardised format. The reports were coded into categories which were then entered
 Examiner comment on Ph.D. theses 341

into QSR N6 software. The analyses are based on the proportions of total standard-
ised text units coded in each category, instances of code occurrence and coded text
unit intersection (i.e. overlapping categories). The core coding categories capture all
the text content of the reports. The five core coding categories (parent nodes) consist
of a hierarchical structure of sub-categories. The act of coding occurs at this sub-
category level. The two broad core categories that capture examiner comments on
literature review are those designated ‘assessable areas covered’ and ‘evaluative
elements’. In most cases the coding for these categories overlaps.
The ‘assessable areas covered’ category captures all comment about the possible
outcomes, subject matter and presentation of the thesis under examination—the
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

substantive elements of the thesis and the project at its heart. The topics that are typi-
cally addressed by examiners include the scope and significance of the study, the liter-
ature review, the methods used and the substance of the findings (including
interpretation and analysis of the data), and also the presentation and communicative
competence of the candidate. There are no absolute surprises in the broad areas
identified here, and they might be found in any guide to writing a thesis. They have
also been identified under similar headings in other text analyses of report content
(Hansford & Maxwell, 1993; Johnston, 1997).
The ‘evaluative elements’ category captures all comment that contains evaluation
and judgement, including different types of instructive comment, as well as text iden-
tifying positive and negative judgements about various aspects of the thesis, and the
candidate’s capacity to conduct research. This range of categorisation has not been
identified in previous studies.
Text on the literature review was coded in three sub-categories. Those comments
that were clearly positive or negative in tone were identified. The first sub-category is
‘coverage’, which incorporates comment on the essentially ‘front end’ compilation
and organisation of the literature. It picks up whether or not the candidate is suffi-
ciently informed and prepared from the available literature in pertinent fields to
‘engage’ in research and to form research questions. The second sub-category iden-
tifies comment on inaccuracy or errors. These are errors in citation, or in the refer-
ences, and are primarily straightforward mistakes in presentation rather than
understanding. The third sub- category is ‘use and application’ (including candidate
understanding of the literature). This category captures comment about how the
literature is used to identify and support significance and contribution, enhance and
support interpretation of the findings, and underpin argument.
When justifying their recommendation, examiners are typically asked, in one form
or another, to comment on the candidate’s coverage, understanding and critical
appraisal of the literature of the discipline, and/or the candidate’s ability to critically
discuss their research findings in the context of that literature. Some Australian
universities very clearly spell out the importance of the literature review by asking that
the examiners respond to specific questions, for example: ‘Does the candidate show
sufficient familiarity with, and understanding and critical appraisal of the relevant
literature?’ Other institutions couch the importance of the literature review in terms
of the candidate’s ability to position their work in the broader context of their chosen
 342 A. Holbrook et al.

discipline. Here, the importance of the literature review is implied with wording to
examiners such as the examiner ‘should consider’ if the thesis ‘relates the topic of
research to the broader framework of the discipline(s) within which it falls’. Several
institutions, however, provide only very broad guidelines to examiners, with no
specific questions to consider on any aspect of the thesis. The evidence suggests,
however, that even if the literature was not specified as a separate criterion, examiners
still devote attention to it, not least because in most disciplines a significant section
of a thesis will be devoted to the literature to position the study within the discipline
or field.
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

The types of comments on literature

Examiners on average devote about 10% of the examination report to the way the
candidate has reviewed the literature. Most of the comments relate to coverage of the
literature, while approximately 2% of comments refer to literature use and applica-
tion, and the most extensive of those comments are formative or instructive in nature
and are attempting to guide the candidate to improve the thesis.
Table 1 provides basic information concerning the extent and nature of examiner
comment about the literature included in each thesis under examination. The three
categories of comment described above—literature coverage, inaccuracies and use and
application are shown in the table. While all inaccuracies noted by the examiners were
clearly negative comments, comments on both coverage and use and application may
have been positive, negative or neutral in tone. Where the initial categories have been
broken down, only the occurrences and proportions of positive and negative
comments that summarise the qualities of the treatment of the literature are presented
in the tables. Sections of text that identify examiner instructions to the candidate and
more extensive commentary are included in the extended analysis reported in the next
section of the article.

Table 1. Literature review categories: occurrences, and range and mean proportions of examiner
report text (%)

Categories Occurrencea Range Mean proportion of textb

Coverage 71.3 0–50 6.3

Positive tone 48.4 0–25 2.3
Negative tone 11.5 0–24 0.4
Inaccuracies
(Negative only) 27.9 0–51 1.6
Use and application 28.7 0–52 2.0
Positive tone 14.2 0–29 0.7
Negative tone 5.0 0–17 0.1

aProportion of examiner reports which included at least one occurrence of the category, with a possible range
for each category of 0–100%.
bProportion of text coded in each category.
 Examiner comment on Ph.D. theses 343

Table 2 shows the correlations of these same three literature review categories with
two measures of thesis quality. The first quality measure is the examiner’s overall
recommendation (on a 5-point scale) made in all 1307 examiner reports, and the
second relates to membership of one of two extreme groups of theses—theses of high
quality and theses of marginal quality. High-quality theses were those which received
a recommendation of ‘accept the thesis without any amendment’ from all examiners
and from the university committee awarding the degree. Marginal theses were those
where at least half the examiners recommended that the thesis be ‘revised and resub-
mitted for further examination’ or failed, and the university committee at least
required significant changes to the thesis before its acceptance
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

Literature coverage. As can be seen in Table 1, comment on coverage of the literature

review was present in more than 70% of examiner reports, and in one case comprised
half the report. This type of comment was the most extensive of all the categories of
comment relating to the literature review, accounting for an average of more than 6%
of the total content of reports. Very frequently the examiner will provide further infor-
mation about specific references, thus explaining the amount of text captured by the
category.
It is clear in Table 1 that there is more text of a positive tone than negative tone
related to literature coverage in the examiner reports. This can be seen in terms of the
occurrence of positive comment in almost half of the reports, but negative comment
in only a little more than 10% of reports. It is also evident in the proportion of positive
tone comment, being almost five times that of negative tone comment on literature
coverage.
A quite typical positive comment on coverage would be:

Table 2. Review of literature: correlations of each category with examiner recommendation (all
theses) and with membership of either the high quality or marginal thesis group

Correlation with examiner Correlation with group

Categories recommendation (n = 1307) membership (n = 149)

Coverage −.05 (NS) −.23**

Positive tone .22*** .15 (NS)
Negative tone −.20*** −.19*
Inaccuracy
(Negative only) −.14** −.32***
Use and application −.01 (NS) .05 (NS)
Positive tone .16*** .21*
Negative tone −.04 (NS) −.14 (NS)

Notes:
1. Correlation coefficient significance levels: *** = < .001, ** = < .01, * = < .05, NS = not significant. A
positive coefficient indicates the better theses had more comment in this category, and a negative
coefficient that the better theses had less.
2. There were 91 reports in the high quality group and 58 in the marginal group.
 344 A. Holbrook et al.

The review of relevant literature is systematic, very clearly presented, and contains much
important information.

Another comment imbued with a positive tone of this type is:

She has reviewed a considerable body of literature in the area and provided an excellent
structure to make sense of the area and set up her thesis. The major researchers and theo-
rists are included in the literature review. (401110320)

On the negative side, one examiner noted of a thesis: ‘The now quite rich literature
on the topic of [X] … to enhance biological control is represented by only two arti-
cles’, demonstrating concern about inadequate coverage.
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

The correlation of literature ‘coverage’ with examiner recommendation for all

theses was not significant (Table 2), presumably reflecting the mix of positive and
negative comments included in overall coverage. Despite this, however, the overall
literature coverage category was significantly correlated with the quality measure
distinguishing between theses in the high quality and marginal groups. This reflects
that examiners concerned about marginal theses frequently wrote more about litera-
ture coverage.
As would be expected, the correlations of the positive and negative tone with exam-
iner recommendation for all theses were much higher than for the overall coverage
measure, and were significant. Also, in relation to coverage, comments with a nega-
tive tone were significantly related to the thesis being in either the high quality or
marginal group.

Literature inaccuracy. Inaccuracy here refers to any minor or substantial comment

by the examiner that the references are omitted, misreported, or used inconsistently
in referencing, citation or quotation. Inaccuracies of this type attracted comment in
more than a quarter of all examiner reports. Comments about inaccuracy ranged up
to slightly more than half of one examiner’s entire report, but more normally this
type of comment tended to be short, accounting on average for less than 2% of the
content of reports. In some cases the literature inaccuracy may have been simply a
wrong publication date or a misspelled name of an author, and thus not considered
by the examiner as crucial in assessing the overall quality of the thesis. However, the
correlation of the proportion of text devoted to literature inaccuracy with examiner
recommendation for all theses was significant. The proportion of text devoted to
literature inaccuracy was much more strongly correlated with the thesis being in
either the high quality or marginal group, indicating that more substantial inaccu-
racy in this area was very important for thesis quality at the extremes. Misreading or
poor understanding of references were not included here but are captured in litera-
ture ‘use’.

Literature use and application. Although commentary on literature use and applica-
tion was considerably less commonly found in examiner reports than comment on
literature coverage, it was present in more than a quarter of all reports, and accounted
 Examiner comment on Ph.D. theses 345

for an average of 2% of each report. More than half of one report consisted of
comment of this type. The inclusion of both positive and negative commentary within
this overall category meant that it was not significantly related to either of the thesis
quality measures.
Positive tone about literature use and application in reports appeared in almost
three times as many reports as negative tone and, on average, it also accounted for
much more text. The proportion of positive comments was significantly correlated
with the examiner recommendation for all theses and for the two extreme groups in
the direction anticipated. However, the correlation between negative comments on
literature use and application and examiner recommendation was not significantly
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

related to either the recommendation for all theses or the extreme groups, perhaps
reflecting the relatively small proportion of negative commentary overall on this
aspect of the literature review.
The above analysis of the positive and negative tone of comment was restricted to
summary comments that were wholly positive or negative. The proportion of nega-
tive comment on coverage predicted the examiner recommendation for all theses
and for the top quality and marginal theses. A similar point can be made about
inaccuracy.

An extended analysis of how examiners comment on ‘use’ of the literature

A high proportion of examiner comment (about one-quarter on average) is ‘forma-
tive’ in nature, that is, the examiner is providing comment to assist the candidate to
improve the quality of the thesis, or the publications that arise out of it. Another 10%
is devoted to prescriptive ‘fix-it’ types of comments. Summative comments by exam-
iners on the whole tend to be loaded toward the positive, but this is particularly so
with reference to literature review (Holbrook et al., 2004a). The comments about
literature use and application pinpoint some core attributes that examiners expect to
find in the thesis. Extended analysis of comments about this category yielded two
further groups of sub-themes or categories. One of these captures aspects of coherent
presentation of the literature. The second and larger grouping captures examiner
comment about how the candidate used the literature substantively.

Coherent use
The importance of coherent argument emerges in relation to all aspects of conven-
tional doctoral theses, not just the use of the literature (Winter et al., 2000; Mullins
& Kiley, 2002; Tinkler & Jackson, 2004). With respect to coherent use of the litera-
ture, the examiners focused on two areas. First, examiners commented on evidence
of a candidate’s ability or inability to select literature and position it in a way that
advances an argument. At times this type of comment reflected examiner preference,
i.e. they might prefer to see all the literature in one chapter or, conversely, spread
throughout. The comments in this sub-category are predominantly critical, as the
following examples indicate:
 346 A. Holbrook et al.

First, sometimes it is impossible to see the wood for the trees. Second, sometimes the
quantity of ancillary materials drawn upon detracts from adequate attention to the key
concept itself. (500310050)
X’s … thinking seems not to be strongly enough focused on her project to recognize useful,
and reject non-useful, previous research. And worse, she continually refers to research
which will not be used in her project. (508311700)
This chapter is a significant weakness of the thesis. The review consists of a long sequence
of short independent summaries of past studies, in apparently random order. (205101530)
In some parts of the thesis … the literature is reported around concepts which become
isolated from an underlying discussion framework. (210002920)
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

Secondly, in a further small number of instances the focus is on the misuse or under-
use of references in support of the argument. Primarily the examiner feels there is
none, or too little, ‘supporting evidence’ from the literature:
if I am right, this is extremely misleading and needs to be corrected or at the least have
some literature references which would support the statement. (400210040)
A tendency is to start sections with general summaries whose claims are unsupported by
evidence. Once the work of the chapter really begins, arguments are much more cogent
but readers should be convinced from the outset that there is more on offer than rhetorical
flourish. Certainly for those familiar with the general field most claims are of the kind
we can take for granted, but still they can be easily backed up with a few citations.
(406712020)

Inadequacies in reporting literature do draw attention to the possibility of plagiarism.

From the five institutions mentions of plagiarism or suspected plagiarism were found
in five reports. The charge of plagiarism is made in one of them, and the examiner
believed it serious enough to warrant resubmission. In the other cases the candidate
was given the benefit of the doubt.

Substantive use
Guides for students note that there is usually a literature chapter that positions the
research questions, and then use of the literature to connect with the findings. Yet, in
addition to adequate and appropriate coverage of the pertinent literature, what is also
anticipated is substantial familiarity with, and systematic treatment of, the literature,
plus critical engagement and sustained depth of immersion throughout the thesis.
When examiners find these qualities they identify and praise them. This is unusual in
the assessment language of academics, where assessors typically articulate what is
wrong when marking assignments but not what is right or praiseworthy (Lea & Street,
1998; Lillis & Turner, 2001).
The use and application of the literature is at the heart of scholarship—of belonging
to the academy. It requires intellectual acumen and a deep appreciation and under-
standing of disciplinary traditions and new directions. The acknowledgement of the
candidate who achieves this is also an acknowledgement of their academic creden-
tials. The top theses receive this level of positive comment, while marginal ones
 Examiner comment on Ph.D. theses 347

receive the opposite. How examiners articulate these qualities, positively and nega-
tively, is an invaluable resource for research training, and they are set out in the tables
that follow.
The extended analysis reported here draws on every comment about the use of the
literature made by examiners. Examiners identified ‘working understanding’, ‘critical
appraisal’ of the body of literature, ‘connection of the literature to findings’, and
‘embeddedness of disciplinary perspective’ as indicators of performance in the candi-
date’s use of the literature. Each comment in whole or in part had a negative or posi-
tive emphasis.
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

A working understanding. One step up from an acceptable level of literature ‘cover-

age’ is a solid knowledge and understanding of the literature—an understanding suffi-
cient to allow for ordering and synthesis (see examples in Table 3). This is sometimes
encapsulated by examiners and Ph.D. assessment guidelines as ‘familiarity’ with the
literature, and can extend to ‘mastery’ as evidenced by excellent synthesis linked to
new insights. As is evident from the entries in the negative column, the candidate must

Table 3. Comments illustrating examiner expectations about understanding the literature

Positive Negative

Clearly, X is familiar with the literature
on this topic and has excellent
understanding of it. (506511350)
[there is] clear evidence that the
literature has been fully consulted and
understood. (508411740)
I must say that X has very thoughtfully
categorised and analysed the body of
work … One important aspect that a
Ph.D. should demonstrate is mastery
of the relevant literature in both
awareness and its potential application.
(120110390)
This chapter discusses and integrates a
number of findings that have not been
reviewed in this detail in the past.
Moreover, the chapter provides some
new insights into the field that, to my
knowledge, have not been published
before. (302000390)
I didn’t find evidence of an advanced level of
understanding of … theory as applied in … [X area] …
which is expected of a PhD candidate in this field.
(203801080)
The lasting impression is that the materials have not
been as fully worked into a satisfactory synthesis as they
could have been, although the ingredients are generally
appropriate. (209002650)
… you will need to take a deep breath and say: OK,
now that I’ve got most of the literature down on paper,
what does it all mean? Which is not to say that you
haven’t done a good job with the literature review—
you’ve done what you’re supposed to do in Chapter 1
of a dissertation, which is to show your committee that
you know the literature and therefore can place your
work in an appropriate context. However this chapter
is not synthetic, which is needed for a publishable
review. (309101830)
 348 A. Holbrook et al.

‘synthesise’, not just provide the ‘ingredients’. They need to at least ‘work’ with the
literature to develop their own conclusions.

Critical appraisal. Examiners expect the candidate to be able to do more than

acknowledge the literature and pull it together. Even in a good, technically compe-
tent synthesis, they also expect that the candidate will weigh up the literature and
subject it to critical appraisal, ideally to lead to a new or interesting perspective.
The lack of such appraisal casts doubt on the depth of the candidate’s understand-
ing of the sources, on their expertness in their chosen field, and certainly their
scholarliness.
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

There were a substantial number of comments about critical appraisal and, in

contrast to the other sub-categories of use and application, most were about prob-
lems. There were twice as many instances identified in the extended analysis of
comments that pointed out a lack of critical appraisal than there were positive ones
acknowledging its application. Where examiners commented on the use of literature,
many noted the absence of such appraisal. Often these negative comments went into
some depth to explain what was missing. The negative comments in Table 4 note not
only the lack of critique, but lack of discrimination in the use of the literature. There
is a clear indication that examiners identify a critical perspective as an essential
precursor to positioning a study and identifying its contribution in an informed and
scholarly way.

Connection with findings. There were many comments that specifically referred to
linking the literature with the findings, and positive comments outnumbered negative
ones by a ratio of two to one. Of those in Table 5, the majority make fairly direct
points about there being, or not being, links made. Others extended to the acuity and
facility of the candidate in bringing the literature to bear in their discussion section,
or at other points in a thesis where conclusions were drawn.

Disciplinary perspective. There is a strand of expectation, sometimes explicit but

frequently implicit, and not only exclusively connected with the literature, that the
candidate is applying a perspective grounded in their immersion in the literature. In
the simplest form the comment may say that a candidate’s arguments or analyses
are ‘well-grounded in the field’, or conversely, that a candidate ‘has not engaged
adequately with the body of material relevant to the core of the thesis’. We know
that scholars develop a form of ‘methodological self-consciousness’ accompanied by
‘endless processes of selection’ (Clough & Nutbrown, 2002, p. 31). The point
made by Delamont et al. (1997) about ‘deriving and exploring ideas’ through the
literature applies here (p. 66). As one student undertaking an ethnographic project
wrote:
I discovered that I had come to understand the function of the literature review … This
was not simply a process of building on previous research or identifying a research deficit,
 Examiner comment on Ph.D. theses 349

Table 4. Comments that illustrate examiner expectations about critical appraisal of the literature

Positive Negative

There was also a clear and critical apprehension of
the various bodies of literature and theoretical
framings around the field of literacy as it is
currently and historically constituted. (151911190)
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014
It is rare to find a reference in the thesis that
has been critically appraised. This is
particularly frustrating when references with
conflicting findings are quoted in successive
sentences. Often, it is not made clear which
references represent primary research and
which are themselves commentaries or
views. As a reader, I was left in no position

to judge how the various conclusions,

[the] candidate’s discussion was furthermore
superbly critical, in the sense that it always
regarded individual insights within a broader
scholarly scope, including diverging opinions and
possible alternative hypotheses. (402510740)
There is no question about the candidate’s ability
to provide critical appraisals and to link these to
sociological knowledge. This thesis demonstrates
commitment to a scholarly exegesis of the
theoretical frameworks presented here, while
maintaining a critical perspective. (404511370)
The author has provided a genuine and unique
perspective that calls into question the atheoretical
approach to this field of research that has persisted
for well over the past five decades. (191210640)
especially when in conflict, were derived.
Importantly, I was given no insight as to
how the candidate weighted up and
prioritized the conflicting sources of
information. (405011510)
The published accounts are cited
uncritically. The first problem is that when
there are disagreements in the major
conclusions, the candidate has not
examined the conditions, procedures and
results to try to figure out where the basis of
the apparent discrepancies might lie. … It is
important that the candidate can distinguish
among data, author’s interpretation of their
data, author’s speculation beyond their
data. (206401910)

but a continuous process of using the thinking of others to clarify one’s own thinking.
(Lilly, 2002, p. 189)

In a small number of reports the examiners note that the candidate’s treatment of the
literature, of itself, produces an advance in the field. Disciplinary perspective denotes
a genuine grasp of theory, and the candidate’s ever-present awareness of the literature
and evidence of engagement with it, particularly in producing a project that does
make a significant and original contribution to the field. In the best research writing
 350 A. Holbrook et al.

Table 5. Comments that illustrate examiner expectations about connections between the
literature and the findings of a study

Positive Negative

A very good effort is also made at tying the
results back into the existing literature.
(501610320)
she has demonstrated a comprehensive
knowledge and understanding of the
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014
I was also a bit amazed that only relatively few
references were given in the discussion and
conclusion sections, which are typically the
parts of the work in which the results are
compared to the earlier work. (150712200)
She does not however develop her discussion
much beyond a reiteration of major findings

underlying theoretical concepts, which has
enabled the results to be fully discussed and
their significance identified. (507711580)
It is also important to note the seamless
weaving of the literature into the discussion of
the data. … the use of the literature to identify
issues in the data and to then explore tensions
is a further strength of this study. (401110330)
Fragments of data are interwoven with a
discriminating literature in a cogent fashion,
such that telling insights derived from the
empirical part of the candidate’s own work are
not left hanging in a theoretical vacuum, but
are helpfully earthed to the previous work of
others. This literary style betrays a creditable
academic tradition, in which the otherwise
novel current work nevertheless acknowledges
a raft of earlier thinkers. (170711340)
which have already been presented in the
previous chapters. She needs to engage with
her findings and the literature at a deeper,
more theoretical/analytical level. (151211180)
Use of references in the final discussion was
poor; this was a reflection of the inadequate
nature of the discussion. (190811870)
It would be interesting to push the boundaries
of the study further to include the ways in
which writers identify key players in the field,
differentiate between their positions, and
indicate points of commonality. While
implicit in the present literature review, the
relative theoretical positioning of the cited
authors could have been foregrounded more.
(401610480)

academics anticipate that the literature will be synthesised, weighed up, and woven
into the very fibre of the argument.
In the best theses disciplinary knowing is embedded and instantly recognisable by
academic peers. Some examiners, when this is absent in a thesis, will seek some correc-
tion by directly referring to the need to foreground the literature, particularly in the
justification for the research and discussion of findings (see Table 6), but such correc-
tions would never achieve the level of ‘embeddedness’ of perspective and the concep-
tual clarity evident in quality academic work. Novice researchers may well come to
recognise the quality of disciplinary knowing, but will be unable to duplicate it. What
are the outcomes if this is the case? The question is addressed in the following section.

Conclusions and discussion

The study reported here explores the emphasis and the approach taken by Ph.D.
examiners in discussing the candidate’s handling of the literature in 501 doctoral

Table 6. Comments that illustrate examiner expectations about applying a disciplinary
perspective
Positive
I was particularly impressed with the review of
the literature, the distillation of the key issues
and the placement of the present findings in
context. (507511550)
She demonstrated that she understood the
essential contributions of the different papers
and was able to apply this knowledge to her
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014
own work. Furthermore, she was able to
derive important clinical information from the
… data and apply this information to the
development of treatment paradigms.
(407112130)
The candidate has used the literature review
to good effect in at least two ways. She has
used some praiseworthy aspects of the studies
in the literature reviewed to strengthen her
research design, for example, the types of
instrumental probe to be utilised. At the same
time she has used perceived weaknesses in the
studies, such as the lack of a theoretical
infrastructure upon which to base certain
analytical decisions, to introduce the
theoretical model on which she bases her
analyses and descriptions of the data.
(209202700)
In her use of these sources, X transforms the
knowledge ‘sedimented’ in previous
literatures not only by reconceptualizing the
three individual analyses but also by setting
them in conceptual contact with each other.
Moreover, she implicitly exemplifies the goal
set for her research subjects in her own
elegant citation practices throughout the
thesis. (401610490)
… the candidate is an independent thinker.
On the basis of the literature review and what
she has learned from reflecting over the
difficulties and limitations of earlier studies,
the candidate has formulated a model of …
(207702300)
Examiner comment on Ph.D. theses 351
Negative
X has not engaged adequately with a body of
material relevant to the core of the thesis.
(500810150)
Any future publication from the thesis would
greatly benefit from the theoretical discussion
being better integrated into the body of the
work. (402410710)
Clearly the author can examine and review a
complex body of literature. However, a
literature review must carry a major part of the
argument in the thesis, the argument that
convinces the reader of the need to investigate
the to-be-specified research questions. I think
that some additions need to be made to these
chapters to more directly focus the reader’s
attention on how contemporary research can be
argued to provide a basis for the investigations.
(151410680)
So often a thesis such as this is set into related
but unconnected segments. Literature reviews
whilst exploring every facet of the related
literature should pivot around the central
argument of the thesis and flow the reader
through the justification for the research. This
literature review, whilst thorough in its
exploration … does not bring it together as a
rationale for the research. (408412540)
… There is no attempt to situate the argument
within the (growing) literature on [x] …
Reading and rereading the thesis, I felt there
should be more of an introduction into the
conceptual problems in [the method] … In my
opinion, the quality of an academic thesis would
have been improved by such a discussion.
(409712930)
 352 A. Holbrook et al.

theses across the full range of disciplines. On average about one-tenth of an examiner
report is devoted to the literature. There was more comment on literature coverage
than literature use and application. And, when these two categories are considered
together, the commentary was positive rather than negative by a ratio of six to one.
Both positive and negative comments on these two aspects of the literature were
indicative of thesis quality, in the expected directions. More specifically, the positive
and negative comments on literature coverage, and positive comments on use were
significantly related to examiner recommendation, whereas negative comments on
use were rarer and not related to recommendation.
The range of comments on literature use and application fell into two broad group-
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

ings, ‘coherent use’ and ‘substantive use’, with the emphasis weighted toward
substantive use. Coherence was essentially linked to making a cogent argument, but
substantive use picked up aspects that provided the substance of the argument, from
grounding the purpose of the study through to identifying and evaluating its contri-
bution. Only 29% of examiners provided any instances of literature use and applica-
tion, but the comments were particularly descriptive, are predictive of good quality
theses, and have potential to inform research training.
Comments on substantive use identified four distinct forms of treatment of a body
of literature, that also indicated a hierarchy of expectation: (1) ‘working understanding’
of the literature—demonstrating a familiarity that resulted in at least a reasonable
level of analysis and synthesis; (2) ‘critical appraisal’ of the literature—the scholarly
activity of weighing up the body of work; (3) direct ‘connection’ of literature to the
candidate’s findings; and (4) the demonstration of ‘disciplinary perspective’. Disci-
plinary perspective constitutes an embeddedness within, and an all-encompassing
awareness of and engagement with, the literature. It frames a quality of ‘knowing’ that
emerges from deep immersion in both a research study and academic discourse and
culture. As one examiner notes of a candidate:

She seems not to have been able to combine previous research with her own where this
would show signs and degrees of originality. It is also not made clear just where the candi-
date agrees with and where she varies from the ideas she reviews: she fails to position
herself in the group of scholars which she seeks to join. (508311700)

The four uses indicate interlinked and essentially cumulative levels of academic
expectation. For example, a student might prove very familiar with the literature, but
not necessarily achieve a good measure of critical appraisal. Critical appraisal is a
precursor to achieving meaningful connection between the findings and the literature,
and a disciplinary perspective indicates a quality of ‘knowing’, the possession of which
can enable the owner to transform what is already known into something new.
In order to develop knowledge about doctoral learning and identify the most appro-
priate strategies to develop candidate skills and understandings, firm links need to be
established between what is expected, taught and assessed (National Research Council,
2001). Supervisors need to be able to build on more than personal experience, shared
anecdote and ideal versions of what a thesis ‘should be’ to help a candidate with the
literature or any other aspect of their thesis. Supervisors and students need a clear idea
 Examiner comment on Ph.D. theses 353

of standards and how they are applied. Figure 1 provides a schema to represent what
we have learned about examiner expectation of the literature review from the nature
of examiner comments on literature coverage and use, coupled with the relationship
of those comments to examiner recommendation.
Coverage of the literature on its own (i.e. marshalling relevant material) is not
Figure 1. Examiner expectations about coverage and use of ‘the literature’ in Ph.D. theses

enough to secure a pass for a Ph.D. There needs to be some treatment or use of the
literature that signifies and demonstrates understanding. Marginal theses often exhib-
ited coverage problems as well as deficiency in ‘use’ even at the most basic levels.
Examiners expect to find evidence of a working understanding that at least yields
simple and accurate levels of analysis. They also actively seek some critical appraisal
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

of the body of literature, particularly in order to position the study and set the scene
for discussion. It is at this level that examiners appear to acknowledge ‘scholarliness’.
If a Ph.D. is seen to be a ‘starting point’, then candidates who show the ability to

LITERATURE USE

CONNECTION
TO FINDINGS
CRITICAL
APPRAISAL

WORKING DISCIPLINARY
UNDERSTAND PERSPECTIVE
-ING

COVERAGE

SCHOLARLINESS

THESIS PASS ZONE OF ENTRY

ZONE TO ACADEME

Figure 1. Examiner expectations about coverage and use of ‘the literature’ in Ph.D. theses
 354 A. Holbrook et al.

synthesise, appraise and apply the literature are achieving a passing level and show
promise in scholarship (identified as ‘thesis pass zone’ in Figure 1).
Examiners remarked about aspects of the candidate’s treatment of the literature
that showed how immersed and embedded they were, their knowledgeability in their
area or field, so much so that this was evident in the thesis from start to finish. It
would be difficult for the student to link the literature to the findings effectively if this
was not so (hence the foregrounding of this perspective in Figure 1). Examiners
expected the student to make the links primarily to indicate and argue contribution.
However, from the comments it was possible for a student to make links between the
literature and findings while employing critical appraisal, and yet not demonstrate the
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

embeddedness of disciplinary perspective. All three would be evident in someone who had
satisfied the highest standards of scholarship for their thesis. They are in the zone of entry
to academe.
In summary, while examiners appeared to anticipate that all the elements in
Figure 1 should be present in scholarly work (identifying them in top performing
theses and often remarking if they were not evident in theses at a lower level), they
were prepared to accept less for a barely passable thesis, but pressed for at least some
demonstration of critical appraisal. Some candidates produce a thesis, engage in the
procedures that produce a front-end literature review chapter and yet do not evidence
scholarship. A student may also be technically proficient in their research, their study
may have been deemed significant, but they would not be ready to engage in activities
such as peer review or publication. Where a thesis is at the margins of, or below a pass
level, the examiners will inevitably call for some ‘improvement’ to the candidate’s
treatment of the literature.
We know from the literature on academic writing in both undergraduate and post-
graduate study that mastery of academic discourse is made all the more challenging
by a set of mostly invisible assumptions about how academic writing should be struc-
tured. Recent linguistic research has found that research Master’s and Ph.D. candi-
dature calls on very different levels of language skills and use (Hyland & Tse, 2004).
It is to be hoped that the examples above provide a tangible framework for discussion
for Ph.D. candidates and their supervisors to identify what is expected at the Ph.D.
level in relation to the coverage and use of academic literature. The findings also
present a new point of entry for discussion as to what differentiates a Ph.D. from
other research degrees.
In many discussions about the purpose and role of the Ph.D., the question that is
raised is whether or not a doctorate is a starting point or an end point. Are researchers
with new doctorates still in training? On the basis of this study the answer would be
that many are still in training, while some have clearly and unambiguously ‘arrived’
as academics. A way to re-address the question is to focus on what constitutes an
unambiguous academic entry point, and here one could say that a litmus test is the
demonstrated potential to undertake peer review in their area, as attested to by their
critical mastery of the literature in their discipline. Examiners are recognising this in
the way a candidate handles the literature and its presentation throughout the thesis.
If the candidate combines good coverage with perceptive and productive use of the
 Examiner comment on Ph.D. theses 355

literature, it is also very likely they are applying a disciplinary and a disciplined
perspective. When examiners identify these qualities they react very positively—so
positively that it is clear that such qualities are not taken for granted, because at core
they represent the essence of scholarship, and to achieve this requires a significant
cognitive shift. Identifying the point and nature of that shift would advance postgrad-
uate pedagogy significantly.

References
Brew, A. (2001) The nature of research: inquiry in academic contexts (London, RoutledgeFalmer).
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

Bruce, C. S. (1994) Research students’ early experiences of the dissertation literature review,
Studies in Higher Education 19(2), 217–229.
Clough, P. & Nutbrown, C. (2002) A student’s guide to methodology (London, Sage).
Delamont, S., Atkinson, P. & Parry, O. (1997) Supervising the PhD: a guide to success (Buckingham,
Open University Press).
Denicolo, P. (2003) Assessing the PhD: a constructive view of criteria, Quality Assurance in Educa-
tion, 11(2), 84–91.
Ely, M., Vinz, R., Downing, M. & Anzul, M. (1997) On writing qualitative research (London,
Falmer Press).
Hansford, B. C. & Maxwell, T. W. (1993) A Masters degree program: structural components and
examiners’ comments, Higher Education Research and Development, 12(2), 171–187.
Holbrook, A. & Bourke, S. (2004) An investigation of PhD examination outcome in Australia
using a mixed method approach, Australian Journal of Educational & Developmental Psychology,
4, 153–169.
Holbrook, A., Bourke, S., Lovat, T. & Dally, K. (2004a) Investigating PhD thesis examination
reports, International Journal of Educational Research, 41(2), 98–119.
Holbrook, A., Bourke, S., Lovat, T. & Dally, K. (2004b) Qualities and characteristics in the writ-
ten reports of doctoral thesis examiners, Australian Journal of Educational & Developmental
Psychology, 4, 110–129.
Hyland, K. & Tse, P. (2004) Metadiscourse in academic writing: a reappraisal, Applied Linguistics,
25(2), 156–177.
Johnston, S. (1997) Examining the examiners: an analysis of examiners’ report on doctoral theses,
Studies in Higher Education, 22(3), 333–347.
Lawson, A., Marsh, H. & Tansley, T. (2003) Examining the examiners, Australian Universities
Review, 46(1), 32–36.
Lea, M. R. & Street, B. V. (1998) Student writing in higher education: an academic literacies
approach, Studies in Higher Education, 23(2), 182–199.
Lillis, T. & Turner, J. (2001) Student writing in higher education: contemporary confusion, tradi-
tional concerns, Teaching in Higher Education, 6(1), 57–68.
Lilly, J. (2002) Researching the ineffable—that which cannot be expressed in words, in: Geoffrey
Walford (Ed.). Doing a doctorate in educational ethnography. Studies in Educational Ethnography
Volume 7 (Oxford, Elsevier), 187–210.
Morley, L., Leonard, D. & David, M. (2002) Variations in vivas: quality and equality in British
PhD assessments, Studies in Higher Education, 27(3), 263–272.
Mullins, G. & Kiley, M. (2002) ‘It’s a Ph.D., not a Nobel Prize’: how experienced examiners
assess research theses, Studies in Higher Education, 27(4), 369–386.
National Research Council (2001) Knowing what students know: the science and design of educational
assessment (Washington, DC, National Academy Press).
Nightingale, P. (1984) Examination of research theses, Higher Education Research and Develop-
ment, 3(2), 137–150.
 356 A. Holbrook et al.

Phillips, E. M. (1994) Quality in the PhD: points at which quality may be assessed, in: R. G.
Burgess (Ed.) Postgraduate education and training in the social sciences (London, Jessica Kings-
ley), 124–146.
Pitkethly, A. & Prosser, M. (1995) Examiners’ comments on the international context of Ph.D.
theses, in: C. McNaught & K. Beattie (Eds) Research into higher education: dilemmas, directions
and diversion (Melbourne, Higher Education Research and Development Society of Australa-
sia), 129–136.
Quality Assurance Agency for Higher Education (2001) The National Qualifications Framework.
National Committee of Enquiry into Higher Education. Available online at: www.qaa.ac.uk/
cmtwork/nqf (accessed July 2005).
Tinkler, P. & Jackson, C. (2004) The doctoral examination process: a handbook for students, examiners
and supervisors (Maidenhead, Open University Press).
Downloaded by [University of Newcastle (Australia)] at 21:54 22 October 2014

Trafford , V. (2003) Questions in doctoral vivas: views from the inside, Quality Assurance in Educa-
tion, 11(2), 113–121.
Winter, R., Griffiths, M. & Green, K. (2000). The ‘academic’ qualities of practice: what are the
criteria for a practice-based Ph.D.? Studies in Higher Education, 25(1), 25–37.
Yates, L. (2004) What does good education research look like? Situating a field and its practices (Maid-
enhead, Open University Press).

View publication stats