See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/240729253

The Place of Inter-Rater Reliability in Qualitative Research: An Empirical Study

Article  in  Sociology · August 1997

DOI: 10.1177/0038038597031003015

CITATIONS READS
746 9,027

4 authors, including:

David Armstrong John Weinman

King's College London King's College London
223 PUBLICATIONS   8,550 CITATIONS    429 PUBLICATIONS   35,708 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Foucault Studies Special Issue: Biopolitical tensions after pandemic times View project

A new history of identity View project

All content following this page was uploaded by David Armstrong on 28 May 2014.

The user has requested enhancement of the downloaded file.

Sociology August 1997 v31 n3 p597(10) Page 1

The place of inter-rater reliability in qualitative research: an empirical

study.
by David Armstrong, Ann Gosling, Josh Weinman and Theresa Martaeu

Assessing inter-rater reliability, whereby data are independently coded and the codings compared
for agreement, is a recognised process in quantitative research. However, its applicability to
qualitative research is less clear: should researchers be expected to identify the same codes or
themes in a transcript or should they be expected to produce different accounts? Some
qualitative researchers argue that assessing inter-rater reliability is an important method for
ensuring rigour, others that it is unimportant; and yet it has never been formally examined in an
empirical qualitative study. Accordingly, to explore the degree of inter-rater reliability that might
be expected, six researchers were asked to identify themes in the same focus group transcript.
The results showed close agreement on the basic themes but each analyst ’packaged’ the themes
differently.

Key words: inter-rater reliability, qualitative research, research methods.

© COPYRIGHT 1997 British Sociological Association consistency of findings from an analysis conducted by two
Publication Ltd. (BSA) or more researchers. However, the concept emerges
implicitly in descriptions of procedures for carrying out the
Reliability and validity are fundamental concerns of the analysis of qualitative data. The frequent stress on an
quantitative researcher but seem to have an uncertain analysis being better conducted as a group activity
place in the repertoire of the qualitative methodologist. suggests that results will be improved if one view is
Indeed, for some researchers the problem has apparently tempered by another. Waitzkin described meeting with two
disappeared: as Denzin and Lincoln have observed, research assistants to discuss and negotiate agreements
’Terms such as credibility, transferability, dependability and disagreements about coding in a process he
and confirmability replace the usual positivist criteria of described as ’hashing out’ (1991:69). Another example is
internal and external validity, reliability and objectivity’ afforded by Olesen and her colleagues (1994) who
(1994:14). Nevertheless, the ghost of reliability and validity described how they (together with their graduate students -
continues to haunt qualitative methodology and different a standard resource in these reports - ’debriefed’ and
researchers in the field have approached the problem in a ’brainstormed’ to pull our first-order statements from
number of different ways. respondents’ accounts and agree them. Indeed, in
commenting on Olesen and her colleagues work, Bryman
One strategy for addressing these concepts is that of and Burgess (1994) wondered whether members of teams
’triangulation’. This device, it is claimed, follows from should produce separate analyses and then resolve any
navigation science and the techniques deployed by discrepancies, or whether joint meetings should generate
surveyors to establish the accuracy of a particular point a single, definitive coded set of materials.
(though it bears remarkable similarities to the
psychometric concepts of convergent and construct Qualitative methodologists are keen on stressing the
validity). In this way, it is argued, diverse confirmatory transparency of their technique, for example, in carefully
instances in qualitative research lend weight to findings. documenting all steps, presumably so that they can be
Denzin (1978) suggested that triangulation can involve a ’checked’ by another researcher: ’by keeping all collected
variety of data sources; multiple theoretical perspectives to data in well-organized, retrievable form, researchers can
interpret a single set of data; multiple methodologies to make them available easily if the findings are challenged
study a single problem; and several different researchers or if another researcher wants to reanalyze the data’
or evaluators. This latter form of triangulation implies that (Marshall and Rossman 1989:146). Although there is no
the difference between researchers can be used as a formal description of how any reanalysis of data might be
method for promoting better understanding. But what role used, there is clearly an assumption that comparison with
is there for the more traditional concept of reliability? the original findings can be used to reject, or sustain, any
Should the consistency of researchers’ interpretations, challenge to the original interpretations. In other words,
rather than their differences, be used as a support for the there is an implicit notion of reliability within the call for
status of any findings? transparency of technique.

In general, qualitative methodologies do not make explicit Unusually for a literature that is so opaque about the
use of the concept of inter-rarer reliability to establish the importance of independent analyses of a single dataset,

- Reprinted with permission. Additional copying is prohibited. - GALE GROUP

Information Integrity
Sociology August 1997 v31 n3 p597(10)
The place of inter-rater reliability in qualitative research: an empirical
study.
Mays and Pope explicitly use the term ’reliability’ and,
moreover, claim that it is a significant criterion for
assessing the value of a piece of qualitative research: ’the
analysis of qualitative data can be enhanced by organising
an independent assessment of transcripts by additional
skilled qualitative researchers and comparing agreement
between the raters’ (1995:110). This approach, they claim,
was used by Daly et al. (1992) in a study of clinical
encounters between cardiologists and their patients when
the transcripts were analysed by the principal researcher
and ’an independent panel’, and the level of agreement
assessed. However, ironically, the procedure described by
Daly et al. was actually one of ascribing quantitative
weights to pregiven ’variables’ which were then subjected
to statistical analysis (1992:204).
A contrary position is taken by Morse who argues that the
use of ’external raters’ is more suited to quantitative
research; expecting another researcher to have the same
’insights’ from a limited data base is unrealistic: ’No-one
takes a second reader to the library to check that indeed
he or she is interpreting the original sources correctly, so
why does anyone need a reliability checker for his or her
data?’ (Morse 1994:231). This latter position is taken
further by those so-called ’post-modernist’ qualitative
researchers (Vidich and Lyman 1994) who would
challenge the whole notion of consistency in analysing
data. The researcher’s analysis bears no direct
correspondence with any underlying ’reality’ and different
researchers would be expected to offer different accounts
as reality itself (if indeed it can be accessed) is
characterised by multiplicity. For example, Tyler (1986)
claims that a qualitative account cannot be held to
’represent’ the social world, rather it ’evokes’ it- which
means, presumably, that different researchers would offer
different evocations. Hammersely (1991) by contrast
argues that this position risks privileging the rhetorical over
the ’scientific’ and argues that quality of argument and use
of evidence should remain the arbiters of qualitative
accounts; in other words, a place remains for some sort of
correspondence between the description and reality that
would allow a role for ’consistency’. Presumably this latter
position would be supported by most qualitative
researchers, particularly those drawing inspiration from
Glaser and Strauss’s seminal text which claimed that the
virtue of inductive processes was that they ensured that
theory was ’closely related to the daily realities (what is
actually going on) of substantive areas’ (1967:239).
In summary, the debates within qualitative methodology on
the place of the traditional concept of reliability (and
validity) remain confused. On the one hand are those
researchers such as Mays and Pope who believe reliability
should be a benchmark for judging qualitative research;
- Reprinted with permission. Additional copying is prohibited. -
Page 2
and, more commonly, those who reject the term but allow
the concept to creep into their work. On the other hand are
those who adopt such a relativist position that issues of
consistency are meaningless as all accounts have some
’validity’ whatever their claims. A theoretical resolution of
these divergent positions is impossible as their core
ontological assumptions are so different. Yet this still
leaves a simple empirical question: do qualitative
researchers actually show consistency in their accounts?
The answer to this question may not resolve the
methodological confusion but it may clarify the nature of
the debate. If accounts do diverge then for the modernists
there is a methodological problem and for the
postmodernists a confirmation of diversity; if accounts are
similar, the modernists’ search for measures of
consistency is reinforced and the postmodernists need to
recognise that accounts do not necessarily recognise the
multiple character of reality.
The purpose of the study was to see the extent to which
researchers show consistency in their accounts and
involved asking a number of qualitative researchers to
identify themes in the same data set. These accounts were
then themselves subjected to analysis to identify the
degree of concordance between them.
Method
As part of a wider study of the relationship between
perceptions of disability and genetic screening, a number
of focus groups were organised. One of these focus
groups consisted of adults with cystic fibrosis (CF), a
genetic disorder affecting the secretory tissues of the
body, particularly the lung. Not only might these adults with
cystic fibrosis have particular views of disability but theirs
was a condition for which widespread genetic screening
was being advocated. The aim of such a screening
programme was to identify ’carriers’ of the gene so that
their reproductive decisions might be influenced to prevent
the birth of children with the disorder.
The focus group was invited to discuss the topic of genetic
screening. The session was introduced with a brief
summary of what screening techniques were currently
available and then discussion from the group on views of
genetic screening was invited and facilitated. The ensuing
discussion was tape recorded and transcribed. Six
experienced qualitative investigators in Britain and the
United States who had some interest in this area of work
were approached and asked if they would ’analyse’ the
transcript and prepare an independent report on it,
identifying, and where possible rank ordering, the main
themes emerging from the discussion (with a maximum of
five themes). The analysts were offered a fee for this work.
GALE GROUP
Information Integrity
Sociology August 1997 v31 n3 p597(10)
The place of inter-rater reliability in qualitative research: an empirical
study.
The choice of method for examining the six reports was
made on pragmatic grounds. One method, consistent with
the general approach, would have been to ask a further six
researchers to write reports on the degree of consistency
that they perceived in the initial accounts. But then, these
accounts themselves would have needed yet further
researchers to be recruited for another assessment, and
so on. At some point a ’final’ judgement of consistency
needed to be made and it was thought that this could just
as easily be made on the first set of reports. Accordingly,
one of the authors (DA) scrutinised all six reports and
deliberately did not read the original focus group transcript.
The approach involved listing the themes that were
identified by the six researchers and making judgements
from the background justification whether or not there were
similarities and differences between them.
Results
The focus group interview with the adults with cystic
fibrosis was transcribed into a document 13,500 words
long and sent to the six designated researchers. All six
researchers returned reports. Five of the reports, as
requested, described themes: four analysts identified five
each, the other four. The sixth analysts returned a lengthy
and discursive report that commented extensively on the
dynamics of the focus group, but then discussed a number
of more thematic issues. Although not explicitly described,
five themes could be abstracted from this text.
In broad outline, the six analysts did identify similar themes
but there were significant differences in the way they were
’packaged’. These differences can be illustrated by
examining four different themes that the researchers
identified in the transcript, namely, ’visibility’, ’ignorance’,
’health service provision’ and ’genetic screening’.
Visibility. All six analysts identified a similar constellation of
themes around such issues as the relative invisibility of
genetic disorders, people’s ignorance, the eugenic debate
and health care choices. However, analysts frequently
differed in the actual label they applied to the theme. For
example, while ’misperceptions of the disabled’, ’relative
deprivation in relation to visibly disabled’, and ’images of
disability’ were worded differently, it was clear from the
accompanying description that they all related to the same
phenomenon, namely the fact that the general public were
prepared to identify - and give consideration to - disability
that was overt, whereas genetic disorders such as CF
were more hidden and less likely to elicit a sympathetic
response.
Further, although each theme was given a label it was
more than a simple descriptor; the theme was placed in a
- Reprinted with permission. Additional copying is prohibited. -
Page 3
context that gave it coherence. At its simplest this can be
illustrated by the way that the theme of the relative
invisibility of genetic disorders as forms of disability was
handled. All six analysts agreed that it was an important
theme and in those instances when the analysts attempted
a ranking, most placed it first. For example, according to
the third rarer:
The visibility of the disability is the single most important
element in its representation. [R3]
But while all analysts identified an invisibility theme, all
also expressed it as a comparative phenomenon:
traditional disability is visible while CF is invisible.
The stereotypes of the disabled person in the wheelchair;
the contrast between visible, e.g. gross physical, and
invisible, e.g. specific genetic, disabilities; and the special
problems posed by the general invisibility of so many
genetic disabilities. [R2]
In short, the theme was contextualised to make it
coherent, and give it meaning. Perhaps because the
invisibility theme came with an implicit package of a
contrast with traditional images of deviance, there was
general agreement on the theme and its meaning across
all the analysts. Even so, the theme of invisibility was also
used by some analysts as a vehicle for other issues that
they thought were related: a link with stigma was
mentioned by two analysts; another pointed out the
difficulty of managing invisibility by CF sufferers.
Ignorance. Whereas the theme of invisibility had a clear
referent of visibility against which there could be general
consensus, other themes offered fewer such ’natural’
backdrops. Thus, the theme of people’s ignorance about
genetic matters was picked up by five of the six analysts,
but presented in different ways. Only one analyst
expressed it as a basic theme while others chose to link
ignorance with other issues to make a broader theme. One
linked it explicitly with the need for education.
The main attitudes expressed were of great concern at the
low levels of public awareness and understanding of
disability, and of great concern that more educational effort
should be put into putting this right. [R2]
Three other analysts tied the population’s ignorance to the
eugenic threat. For example:
Ignorance and fear about genetic disorders and screening,
and the future outcomes for society. The group saw the
public as associating genetic technologies with Hitler,
eugenics, and sex selection, and confusing minor gene
GALE GROUP
Information Integrity
Sociology August 1997 v31 n3 p597(10)
The place of inter-rater reliability in qualitative research: an empirical
study.
therapy alterations with alterations of the whole person.
[R6]
But this did not mean that the need for education or the
potential eugenic threat was absent from those analysts
who did not link the theme with ignorance. For example,
one analyst thought that education deserved to be labelled
as a separate theme, another placed it in the context of
’the parenting dilemma’:
The dilemmas for prospective parents of genetic
screening: an ignorant public needs education, those
involved need information, protection and confidentiality.
[R5]
And a third in the context of choice and genetic screening:
The participants . . . see [screening] as a way of allowing
choice, especially if it goes hand in hand with better
education about genetics. [R1]
Health service provision. The theme of health service
provision for the genetically ’disabled’ was another
common one but this was even more widely contexualised
than ’ignorance’. Health care was mentioned in some form
by four of the analysts, all in the context of a wider
resource allocation debate. One analyst simply recognised
the problem of limited health care resources.
The cost to the health service for CF patients is high and is
recognised as such. With the health service debate
dominated by cost the CF patients are keenly aware that
their position is precarious. [R5]
Two others set these demands in a wider moral context.
Theme: MEETING THE COST OF MEDICAL SUPPORT.
The costs of providing for disability; the need for adequate
provision through the NHS; the moral responsibilities of the
able-bodied majority and the disabled minority; etc . . . and
of concern that government was generally reluctant to
invest sufficiently in medical treatment. [R2]
The need for adequate and non-discriminatory provision of
treatment and facilities for people with disabilities under
the NHS, to come if necessary from increased NI
contributions. Unfair to penalise the ’innocent party’ - the
child born with the disability. [R6]
The fourth analyst placed health care needs within a social
welfare model. But this analyst also identified a cultural
context for the focus group transcript that presumably
reflects on the sense of strangeness that the analyst found
in the discussion.
- Reprinted with permission. Additional copying is prohibited. -
Page 4
In terms of social welfare, there was quite an acute, and
arguably, very British, framing of themes within a
discourse of what kind of support should be provided and
how it should be paid for. [R4]
Genetic screening. Two analysts made no mention of the
NHS and health care in general. However, they, like all the
other analysts commented on issues of genetic screening
and therapy - though each used the theme in different
ways. Three linked screening explicitly with increasing
individual choice. For example:
The participants are in favour of generally available carrier
screening. They see it as a way of allowing choice,
especially if it goes hand-in-hand with better education
about genetics. [R1]
A fourth analyst also discussed genetic screening in the
context of choice, but perhaps because the notion of
screening was embedded in a wider theme (genetic
research and therapy), also linked it with other issues.
The attitudes expressed towards genetic science are a bit
muted, and I found it hard to assess attitudes towards, for
example, gene therapy. On the one hand, there was
criticism from one male speaker in particular of ’public
misunderstanding’ of genetic science. On the other hand,
attitudes towards genetic screening tended to dissolve into
an awareness of choice and diversity. [R4]
A fifth analyst also placed screening in the context of
choice but packaged the link with the dilemmas introduced
by such a choice, in particular resonating with themes of
ignorance and education (as described above).
The parenting dilemma: The dilemmas for prospective
parents of genetic screening: an ignorant public needs
education, those involved need information, protection and
confidentiality. The personal dilemmas for those found to
be gene carriers, the decision to go ahead with a
pregnancy or to abort is clearly a difficult decision but, in
the final analysis, it should be left to the parents. [R5]
In similar fashion, the sixth analyst chose to use the
screening theme to make a statement about eugenics- a
theme that several of the other analysts had addressed in
relationship to the ’education’ theme.
GENETIC ENGINEERING AND EUGENICS: The general
unease and misapprehension surrounding modern genetic
screening and genetic treatment, due principally to the
historical association of genetics with Nazi eugenics. [R2]
In short, analysts tackled the ’core’ theme of genetic
GALE GROUP
Information Integrity
Sociology August 1997 v31 n3 p597(10)
The place of inter-rater reliability in qualitative research: an empirical
study.
screening in different ways. In a sense it was impossible
for ’genetic screening’ to stand on its own as a theme, like,
for example, ignorance, as the explicit purpose of the
focus group discussion had been to look at the problem of
screening in the context of genetic disability. Instead,
rather than state the obvious, analysts had to make
choices of how to present this important facet of the
discussion. The theme of choice was a common one but
even so this could link with other issues such as education
which other analysts had chosen to treat as separate
themes. And then one analyst who had already identified
an explicit theme of ignorance and education needed to
make a different link - in this case with eugenics.
Discussion
This study was necessarily limited using only a single
transcript as its dataset. This was a limit imposed by the
need to restrict the amount of reading so as to obtain the
co-operation of the researchers in question. It is possible
that different results would have been obtained had a
wider range of material been presented, though the main
findings would seem to be independent of the amount of
material that was scrutinised. Moreover, the transcript was
not based on an interview with a single respondent but
with a focus group voicing a number of different
perspectives on the problem. Criticism can also be levelled
at the failure to involve the analysts at all stages of the
research, as is advised by many writers on qualitative
methodology (Burgess 1984), but this too was impractical.
The study attempted to begin to provide an answer to the
question of reliability in qualitative research. By assessing
the extent of consensus on the basic building blocks of this
form of research, namely the raw themes that emerge from
relatively unselected data, it is possible to see the range of
bias. The six researchers who took part in this study were
all highly experienced qualitative researchers (several with
national and international reputations): it might be
expected that if any group was to show agreement it would
be these experienced researchers; the corollary is that any
idiosyncrasies identified in this study are likely to be
considered magnified if a group of less experienced
researchers had been recruited.
The main finding is that there is indeed a degree of
consensus in the identification of themes between the
different analyses but that the ’packaging’ of these themes
showed a number of different configurations. Social
representation theory has argued that people’s
representations are embedded in a network of other
related representations (Moscovici 1981). Thus an
interview elicits not only the representation under
consideration but the map of other representations in
- Reprinted with permission. Additional copying is prohibited. -
Page 5
which it is located. The results of this study would suggest
that an exactly analogous process occurs in qualitative
research. The analysts involved in this study all chose to
embed the themes they identified in a wider context of
other themes. Such contexts might have reflected
geography (analysts were drawn from Britain and the
United States), or discipline (they include anthropology,
psychology and sociology), or personal differences in
experience or views, but the number of analysts was too
few to draw any definite conclusions.
It is important to separate out the level at which
concordance might be expected. In qualitative research
the raw data initially yields the basic themes, though
clearly these might be used to construct many different
’stories’. In exactly the same way, a quantitative dataset
may yield a number of different ’stories’ depending on how
it is analysed and the interpretation placed on those
findings by the researcher. Even so, at a purely descriptive
level there should be some concordance about, say 45 per
cent of the sample being male or that 73 per cent had
visited a doctor in the last year. What is not clear is
whether there would be concordance among qualitative
researchers on the basic ’themes’ contained within a data
set irrespective of how these might be interwoven into a
’story’ if the material were being placed in a more formal
paper.
What then are the implications for ’inter-rater reliability’ in
qualitative research? For the modernists, concerned to
establish some degree of ’accuracy’ in representation, it
does seem that the technique of triangulation - at least by
using different researchers - is limited by the processes
inherent in qualitative data analysis. As the analysts used
in this study showed, all analysis is a form of interpretation
and interpretation involves a dialogue between researcher
and data in which the researcher’s own views have
important effects. Of course, this inherent subjectivity is
freely acknowledged in qualitative research, indeed it is
often cited as a hallmark of this approach unlike the
spurious claims to objectivity of more traditional
quantitative methods. Nevertheless, subjectivity does not
necessarily mean singularity: the views of the analysts in
this study were socially patterned, and this determined
their interpretations. In other words, while quantitative
methods are rightly criticised for encoding the researchers’
biases in apparently objective ’numbers’, the same can be
said for qualitative method. An interview transcript might
represent ’raw’ data but the basic themes that are
extracted have already been ’contaminated’ by the
researcher.
In some ways this argument lends support to the
postmodernists in their claim that all accounts are unique
GALE GROUP
Information Integrity
Sociology August 1997 v31 n3 p597(10) Page 6

The place of inter-rater reliability in qualitative research: an empirical

study.
in that they represent the differing perspectives of different
observers. None the less, the findings of this study did not
find completely divergent interpretations but a
concordance at a level of situating themes within a wider
framework.
Acknowledgements
Forgas (ed.) Social Cognition: Perspectives on Everyday
Life. London: Academic Press.
OLESON, V., DROES, N., HATTON, D., CHICO, N. and
SCHATZMAN, L. 1994. ’Analyzing together: recollections
of a team approach’, in A. Bryman and R. G. Burgess
(eds.) Analyzing qualitative data. London: Routledge.

We would like to thank the Medical Research Council for
funding the project from which this study is drawn. Theresa
Marteau is funded by the Wellcome Trust.
References
TYLER, S. A. 1986. ’Post-modern Ethnography: From
Document of the Occult to Occult Document’, pp. 122-40
in J. Clifford and G. E. Marcus (eds.) Writing Culture: The
Poetics and Politics of Ethnography. Berkeley: University
of California Press.

BRYMAN, A. and BURGESS, R. G. (eds.) 1994. Analyzing
qualitative data. London: Routledge.
BURGESS, R. G. 1984. In the Field: An Introduction to
Field Research. London: Allen and Unwin.
DALY, J., MACDONALD, I. and WILLIS, E. 1992. ’Why
Don’t You Ask Them? A Qualitative Research Framework
for Investigating the Diagnosis of Cardiac Normality’, pp.
189-206 in J. Daly, I. MacDonald and E. Willis (eds.)
Researching Health Care: Designs, Dilemmas, Disciplines.
London: Tavistock.
DENZIN, N. K. 1978. The Research Act: A Theoretical
Introduction to Sociological Methods. New York: McGraw
Hill.
DENZIN, N. K. and LINCOLN, Y. S. 1994. ’Introduction:
Entering the Field of Qualitative Research’, pp.1-18 in N.
K. Denzin and Y. S. Lincoln (eds.) Handbook of Qualitative
Research. London: Sage.
VIDICH, A. J. and LYMAN, S. M. 1994. ’Qualitative
Methods: Their History in Sociology and Anthropology’, pp.
23-59 in N. K. Denzin and Y. S. Lincoln (eds.) Handbook
of Qualitative Research. London: Sage.
WAITZKIN, H. 1991. The Politics of Medical Encounters.
New Haven: Yale.
Biographical note: DAVID ARMSTRONG is Reader in
Sociology as applied to Medicine in the Department of
General Practice; ANN GOSLING is a Research Associate
and THERESA MARTEAU Professor of Health Psychology
in the Psychology and Genetic Research Group; and
JOHN WEINMAN is Professor of Psychology as applied to
Medicine in the Unit of Psychology, all at UMDS, London.
Address: Dr David Armstrong, Department of General
Practice, UMDS, 5 Lambeth Walk, London SE11 6SP.

GLASER, B. and STRAUSS, A. 1967. The Discovery of

Grounded Theory: Strategies for Qualitative Research.
Chicago: Aldine.

HAMMERSELY, M. 1991. Reading Ethnographic

Research: A Critical Guide. London: Longman.

MARSHALL, C. and ROSSMAN, G. B. 1989. Designing

Qualitative Research. London: Sage.

MAYS, N. and POPE, C. 1995. ’Rigour and Qualitative

Research’. British Medical Journal 311:109-12.

MORSE, J. M. 1994. ’Designing Funded Qualitative

Research’, pp. 220-35 in N. K. Denzin and Y. S. Lincoln
(eds.) Handbook of Qualitative Research. London: Sage.

MOSCOVICI, S. 1981. ’On Social Representation’, in J.P.

- Reprinted with permission. Additional copying is prohibited. - GALE GROUP
Information Integrity
View publication stats