See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/6914582

Reliability and validity in research

Article in Nursing standard: official newspaper of the Royal College of Nursing · July 2006
DOI: 10.7748/ns2006.07.20.44.41.c6560 · Source: PubMed

CITATIONS READS

245 7,703

2 authors, including:

Helena Mary Priest

Staffordshire University
80 PUBLICATIONS 1,860 CITATIONS

SEE PROFILE

All content following this page was uploaded by Helena Mary Priest on 08 April 2021.

The user has requested enhancement of the downloaded file.

art & science CLINICAL RESEARCH
Reliability and validity in research
Roberts P et al (2006) Reliability and validity in research. Nursing Standard. 20, 44, 41-45.
Date of acceptance: April 6 2006.
SummarY
This article examines reliability and validity as ways to demonstrate
the rigour and trustworthiness of quantitative and qualitative
research. The authors discuss the basic principles of reliability and
validity for readers who are new to research.
Authors
Paula Roberts and Helena Priest are senior lecturers, School of
Nursing and Midwifery, Keele University, Keele, Staffordshire;
Michael Traynor is professor of nursing, Middlesex University,
London. Email: p.m.roberts@keele.ac.uk
Keyyyords
Qualitative research; Quantitative research; Reliability; Validity
These keywords are based on the subject headings from the British
Nursing Index. This article has been subject to double-blind review.
For related articles and author guidelines visit our online archive at
www.nursing-standard.co.uk and search using the keywords.
'THE HALLMARK OF science is the pursuit of
truth and the limitation of error. As such, science is Reliability describes how far a particular test,
an attitude of mind rather than a set of procedures. procedure or tool, such as a questionnaire, will
The defining characteristic of that attitude is a
commitment to subject any claim to rigorous
evaluation and the conscientious seeking out of
evidence that might contradict or modify that
claim' (Murphy and Dingwall 2003). Although
presenting an idealised view of science, this
statement is a good starting point for a discussion
that aims to demystify two related examples of
research terminology: reliability and validity.
Reliability and validity are ways of
demonstrating and communicating the rigour of
research processes and the trustworthiness of
research findings. If research is to be helpful, it
should avoid misleading those who use it. If a
hospital decides to replace one treatment with
another safer and more effective treatment, then
managers, clinicians and patients can justifiably
expect the decision to be based on good, rather
than flawed, evidence. This trustworthiness
NURSING STANDARD
The synthesis of art and science is lived by
the nurse in the nursing act JOSEPHINE c PATERSON
E DUC ATI 0N
depends on a number of research features: the initial
research question, how data are collected including
when and from whom, how they are analysed, and
what conclusions are drawn.
Although Murphy and Dingwall (2003) are
right in stating that following procedures alone is
not sufficient to produce trustworthy results, this
is not to say that procedures can be ignored.
When reading published research or designing
research projects, it is important to consider
issues of reliability and validity from the outset.
The authors provide an overview of the basic
principles of reliability and validity in relation to
quantitative and qualitative nursing research. In
so doing, they aim to assist those readers starting
out in reading or conducting research to
understand and use these frequently encountered
terms correctly. Readers are directed to other
useful resources that can provide more detail
than is possible within the scope of this article.
Of the two terms, reliability is probably the
simpler to understand and demonstrate.
produce similar results in different circumstances,
assuming nothing else has changed.
Validity is a subtler concept. It is about the
closeness of what we believe we are measuring to
what we intended to measure. For example, a
strategic health authority might want to know if it
is worth paying more to a university to train
nurses to degree rather than diploma level. They
want to know: are graduate nurses as competent
to practise as diploma holders? We design a
questionnaire, test its reliability, get both types of
nurse to complete it, and tell the authority that
degree and diploma level nurses are equally
competent. However, someone notices that many
ofthe questions were about sociological theory,
advanced statistics and the history of science, and
they point out that this is not what we meant by
'competent to practise'. Reliability is a necessary
but insufficient condition for validity, and
July 12 :: vol 20 no 44 :: 2006 4 1

art & science research methods
although the questionnaire may have been
reliable, it was not valid.
Quantitative and qualitative research
Quantitative research is the conduct of
investigations primarily using numerical
methods, whereas qualitative research tends to
use exploratory approaches and produce textual
data rather than numbers or measurements.
Quantitative or statistical methods have been
used in research since the early nineteenth century
and Florence Nightingale was one ofthe pioneers.
The Royal Statistical Society was founded in 1834,
although famous examples of data collection and
presentation go back another 150 years.
Concerned with questions about how likely their
results are to be misleading, statisticians have
devised procedures for expressing likelihoods and
for estimating how accurate their measurements
are. Being able to report on these procedures and
their outcomes can help to demonstrate the rigour,
accuracy and, therefore, the usefulness of such
work. Unsurprisingly, the basis of these
procedures is mathematical.
However, qualitative researchers find, when
searching for similar procedures, that there is no
simple equivalent. For complex reasons, social
scientists also began to feel that they needed to
address issues like rigour and trustworthiness.
Grounded theory, for example, sought to address
these issues (Glaser and Strauss 1967).
Qualitative researchers felt challenged to develop
the same notions of reliability and validity as their
colleagues who used quantitative methods. The
problem was that some qualitative researchers
had actively rejected assumptions about reality
and our ability to represent it accurately and
objectively. For individuals working within this
school of thought, tests to assess accuracy of
measurement are problematic. Hence, there is
debate about 'equivalents' to these concepts in
qualitative research. Despite the variety of
schemes devised by different researchers, the vital
question that all address is: 'How can I assure the
user of my work that it is trustworthy?'
There are many approaches to the generation
and analysis of qualitative data, including
phenomenology, grounded theory, qualitative
content analysis and narrative analysis (Priest et
a/2002). The purpose in each case is to
summarise the data using a variety of coding
procedures. Many proceed from particular
philosophical or political standpoints and these
tend to have well-elaborated positions on issues
like truthfulness. In general, qualitative
42 July 12 :: vol 20 no 44 :: 2006
researchers consider that whichever approach is
used, there is a need to determine how far the
research findings are believable, accurate and
useful (Creswell 1998).
It is sometimes claimed, however, that
traditional tests of rigour are not relevant in the
qualitative paradigm - qualitative research should
be judged against a different set of standards, using
alternative terminology, and evaluated using an
alternative set of strategies to quantitative
research. In the light ofthis claim, qualitative
researchers have devised an array of procedures
for demonstrating validity and reliability.
Reliability in quantitative research
Research methods resulting in the production of
numerical data of relevance to nurses include
experiments, for example, a randomised
controlled trial ofthe effectiveness of a new
clinical treatment, and surveys, for example,
patient satisfaction questionnaires. Further
information on these approaches can be found in
Cormack (2000). Researchers have developed
particular measures of reliability, some
associated with specific statistical tests.
Essentially, any research tool should provide
the same information if used by different people
(inter-rater reliability), or if it is used at different
times, for example, on Friday morning and again
on Sunday afternoon (test-retest reliability). The
internal consistency of research tools needs to be
assessed. Internal consistency is the relationship
between all the results obtained from a single test
or survey. If we ask people ten questions about
job satisfaction, do they answer every question in
a similar way, or are there a few questions where
the replies seem to be unrelated to the others?
Internal consistency of items such as
individual questions in a questionnaire can be
measured using statistical procedures such as
Cronbach's alpha coefficient (Cronbach 1951),
randomly splitting all the responses to a question
into two sets, totalling the scores on the two sets,
and working out the correlation between the two
sets. This is known as a 'split-half test. A more
sophisticated way of doing this is to create all
possible split halves and determine the average
correlation between all of them. Cronbach's
alpha (1951) is an estimate of the average of all
split-half estimates of reliability.
Reliability is the proportion of variability in a
measured score that is due to variability in the
true score (rather than some kind of error). A
reliability of 0.9 means 90 per cent ofthe
variability in the observed score is true and 10 per
cent is due to error. A reliability of 80 to 90 per
cent is recommended for most research purposes.
Methods of estimating the reliability of
measurements do have some limitations; for
NURSING STANDARO
example, test-retest reliability is potentially
flawed if respondents' previous experiences in
the first testing influence responses in the second
testing (Carmines and Zeller 1979). Moreover,
intervening events between the two
administrations may account for differences
between the two sets of results (Bryman and
Cramer 2004) and contribute to flaws in external
validity (Robinson Kurpius and Stafford 2005).
Validity in quantitative research
Validity describes the extent to which a measure
accurately represents the concept it claims to
measure (Punch 1998). There are two broad
measures of validity - external and internal.
External validity addresses the ability to apply with
confidence the findings ofthe study to other people
and other situations, and ensures that the
'conditions under which the study is carried out are
representative ofthe situations and time to which
the results are to apply' (Black 1999). The sample
of participants drawn from the population of
interest must be representative of that population
at the time ofthe study. Finally, representative
samples should be drawn with reference to relevant
variables in the study, such as gender and age.
Internal validity addresses the reasons for the
outcomes of the study, and helps to reduce other,
often unanticipated, reasons for these outcomes.
Three approaches to assessing internal validity
are content validity, criterion-related validity,
and construct validity (Eby 1993, Punch 1998).
Content validity is the weakest level of
validity, and is concerned with the relevance and
representativeness of items, such as individual
questions in a questionnaire, to the intended
setting. It is particularly important to measure
this if the study is designed to ascertain
respondents' knowledge within a specific field,
or to measure personal attributes such as
attitudes (Eby 1993). It can be achieved through
conducting a pilot study with people who are
similar to the intended study participants. Such
relevance can be supported by literature reviews
and documentary evidence, where available.
Criterion-related validity is a stronger form
of validity, established when a tool such as a
questionnaire can be compared to other similar
validated measures of the same concept or
phenomenon (Eby 1993). However, where no
other measures exist, this will not be possible.
Construct validity involves demonstrating
relationships between the concepts under study
and the construct or theory that is relevant to
them. Eor example, the group of drugs known
as antibiotics (construct) comprises several
distinct drugs (concepts, for example,
penicillin, erythromycin and oxytetracycline),
which are individual drugs but related to one
NURSING STANOARO
another in that their primary purpose is to treat
bacterial infection.
There are several ways of demonstrating
construct validity, one of which is factor analysis.
Factor analysis refers to a number of statistical
procedures used to determine characteristics that
relate to each other (Bryman and Cramer 2004).
Factor analysis is particularly useful for examining
the relationships between large numbers of
variables, disentangling them and identifying
clusters of variables that are closely linked together
(Burns and Grove 2005). In the antibiotic example,
factor analysis might identify a cluster of drugs
useful for treating respiratory infections, a cluster
useful for treating skin infections and another
group with a number of different uses (broad
spectrum antibiotics). For more detailed discussion
of different methods to conduct factor analysis,
and how statistical computer packages can assist
the process, see Bryman and Cramer (2004).
Reliability in qualitative research
In qualitative research, reliability can be thought
of as the trustworthiness of the procedures and
data generated (Stiles 1993). It is concerned with
the extent to which the results of a study or a
measure are repeatable in different circumstances
(Bryman 2001). Thus, we need to confirm
findings by revisiting data in different
circumstances. For example, to overcome any
researcher bias in the interpretation of data and as
an auditing measure, interview data may be sent
to an independent researcher to verify how much
agreement there is about findings and analysis - a
form of inter-rater reliability (Weber 1990).
Additionally, keeping detailed notes on decisions
made throughout the process will add to the
project's auditability and, therefore, reliability.
Qualitative content analysis is a particularly
reliable approach to handling data. Specific
codes are created to describe the data, such as
statements from interview transcripts, and can
be confirmed by revisiting previously coded data
periodically to check for stability over time
(Roberts 1999). Additionally, using
computerised data analysis packages, such as
NVivo (QSR), can enhance reliability (Roberts
and Woods 2000) by applying the rules built into
the programme (Robson 1994). Furthermore,
line by line numbering of interview transcripts
inbuilt in such software may assist inter-rater
reliability. However, caution needs to be
exercised in that an over-emphasis on
standardisation may separate the data from its
context so much that it almost becomes
meaningless (Burton 2000).
Other methods for increasing reliability
include ensuring technical accuracy in recording
and transcribing. While some suggest that
July 12 :: vol 20 no 44 ;: 2006 4 3

art & science research methods
tape-recorded interviews and interview
transcripts can help improve reliability, important
non-verbal aspects of communication are
sometimes omitted from transcripts (Perakyla
1997), In one approach to qualitative research,
conversation analysis, an attempt is made to
capture the subtle nuances of communication by,
for example, using standardised transcription
notation, which records pauses, emphases and
interruptions (Atkinson and Heritage 1984).
Intensive engagement with the data - moving
backwards and forwards between the data and
our interpretation of it - and making firm hnks
between our interpretations and the data by, for
example, using verbatim examples of
participants' comments in written accounts ofthe
findings, can all increase reliability and
readability. However, it is important that the
selection of illustrative quotations does not
introduce bias by 'cherry picking' the most vivid
examples from the research. They should reflect
the range and tone of responses generated.
Validity in qualitative research
Validity is assessed in terms of how well the
research tools measure the phenomena under
investigation (Punch 1998), Apotential difficulty
in achieving validity in qualitative research is
researcher bias, arising out of selective collection
and recording of data, or from interpretation
based on personal perspectives (Johnson 1997). In
the case of interviews, a common method of data
collection in qualitative research, the validity of
the interview data needs to be considered.
While the interviewer should assume that self-
reporting is accurate and therefore valid (Appleton
1995, Burns and Grove 2005), distortions can arise
through the process of analysis and interpretation,
'Interested' researchers who are familiar with the
field may overlook certain nuances and
ambiguities of data because of their implicit
understanding ofthe research setting. Being
familiar with the setting, its people and processes, is
both advantageous and potentially problematic.
Such insights can be useful in authenticating
responses and findings, but familiarity may also
obscure any ambiguous issues that others, from
outside the field, might question.
Where the researcher is personally familiar
with the subject or environment being studied,
the researcher can try to be non-reactive and
achieve analytical distance. The researcher thus
attempts to minimise bias in the data collection,
interpretation and presentation of findings. In
descriptive phenomenological research, based on
44 July 12 :: vol 20 no 44 :: 2006
the work of Husserl (1982), such bias is avoided
through the use of 'bracketing', whereby
researchers attempt to suspend their experience,
judgement and beliefs (Cutcliffe and McKenna
1999). Although bracketing is often difficult, if
not impossible, to achieve, the credibility of
findings is increased if researchers make explicit
their presuppositions and acknowledge their
subjective judgements (Ashworth 1997a,
1997b). It is becoming increasingly usual for
qualitative researchers to reflect openly on their
own ability, or otherwise, to be unbiased, and to
consider the effects of this on the research and on
themselves as researchers. This process,
sometimes known as 'reflexivity', will appear in
the final written account ofthe research process
and can serve as a further measure of validity.
The reduction of bias can also be facilitated by
respondent validation. This refers to the practice
of researchers sharing interpretations and
theorising with the research participants, who
can check, amend and provide feedback as to
whether they are recognisable accounts
consistent with their experience (Bryman 2001).
Although often described as a test of validity
(Lincoln and Guba 1985), the usefulness of
respondent validation may also lie in providing
researchers with an opportunity to think again
about their interpretations, A range of factors
determines whether or not participants agree
with the researcher's conclusions, such as what is
at stake for them if researchers present their
actions in an unfavourable light? Or do they
interpret the exercise as an invitation to be
supportive and agreeable to the researchers?
Good research often aims to get beyond
participants' own understandings of their
situations. One common strategy is to illustrate
analyses with what has been described as 'low
inference descriptors' (Johnson 1997), that is,
using examples of participants' verbatim accounts
within the written account of the findings to
demonstrate that findings are grounded in the
data. The practice of 'cherry picking' only
particularly vivid examples should be avoided.
Triangulation is another way of enhancing the
validity of qualitative research. Triangulation
describes the combination of two or more
theories, data sources, methods or researchers in
the study of a topic (Halcomb and Andrew 2005,
Williamson 2005), and assists with the
consistency, comprehensiveness and robustness
ofthe study. Examples include cross-case analysis
and comparisons across data from different
groups of participants, and cross checking with
documentary evidence and published literature
(Halcomb and Andrew 2005, Williamson 2005).
Other approaches to increasing validity include
prolonged engagement in the research site
(Erlandson etal 1993), checking for deviant cases
NURSING STANDARD
by requesting negative descriptions of the topic
under investigation (Streubert Speziale and
Carpenter 1995) and regular supervision and peer
review on analysis and findings by researchers.
Some qualitative researchers claim that
complete objectivity or detachment in the research
process is impossible (Guba and Lincoln 1981,
Stiles 1993). However, it is helpful if the research
process is transparent so that readers can trace the
decision processes relating to personal orientation
and context, theory, methodology and analysis
throughout the life ofthe study. Morse and Field
(1996) andKoch (1994) have recommended that
an audit trail should be evident in the study to
demonstrate rigour. This is commonly achieved
through the maintenance of a research diary, in
which the decision trail is recorded.
Conclusion
Using either established or more novel
approaches to assessing the reliability and
validity of research is one way of producing
useful and trustworthy research findings. In
determining the reliability and validity of
research, reducing error is of prime concern.
However, while adhering as closely as possible to
a set of procedures to pursue truth and limit error
is important, an attitude that seeks to ensure
rigour in research is equally important.
All measures possess some residual bias or
unreliability or inaccuracy (Punch 1998). While
efforts can be made to minimise such risks,
particularly systematic errors, they are
acknowledged as a limitation in all types of
research. While researchers should use as many
approaches as possible to ensure reliability and
validity, there remains the possibility that flaws
may occur at the design, measurement or analysis
stage, resulting in a less than perfect study.
However, paying attention to the procedures
outlined in this article will help to produce
knowledge that is robust enough to use in
planning or introducing change NS

References
Appleton JV (1995) Analysing
qualitative interview data:
addressing issues of validity and
reliability. Journal of Advanced
Nursing. 22, 5,993-997
Ashworth PD (1997a) The variety
of qualitative research. Part one:
introduction to the problem. Nurse
Education Today. 17 3, 215-218.
Ashworth PD (1997b) TVie variety
of qualitative research. Part two:
non-positivist approaches.
Nurse Education Today. 17 3,
219-224.
Atkinson J M , Heritage J (1984)
Structures of Social Action: Studies
in Conversation Analysis Cambridge
University Press, Cambridge.
Black TR (1999) Doing Quantitative
Research in the Social Sciences.
Sage, London.
Bryman A (2001) Social Research
Methods. Oxford University Press,
Oxford.
Bryman A, Cramer D (2004)
Quantitative Data Analysis with
SPSS 12 and 13: A Guide for Social
Scientists. Routledge, London.
Burns N, Grove SK (2005) The
Practice of Nursing Research:
Conduct, Critique, and Utilization.
Fifth edition. WB Saunders,
Philadelphia PA.
Burton D (2000) Research Training
for Social Scientists. Sage, London.
Carmines EG, Zelier RA (1979)
Reliability and Validity Assessment.
Sage, London.
Cormack D (2000) The Research
Process in Nursing. Fourth edition.
Blackwell, Oxford.
Creswell J W (1998) Qualitative
Inquiry and Research Design.
Choosing among Five Traditions
Sage, Thousand Oaks CA.
Cronbach LJ (1951) Coefficient
alpha and the internal structure of
tests. Psychometrika. 16, 3, 297-334.
Cutcliffe JR, McKenna HP (1999)
Establishing the credibility of
qualitative research findings: the
plot thickens. Journal of Advanced
Nursing. 30, 2, 374-380.
Eby iVI (1993) Validation: choosing
a test to fit the design. Nurse
Researcher. 1, 2, 27-33.
Eriandson DA, Harris EL,
Skipper BL, Allen SD (1993) Doing
Naturalistic Inquiry. A Guide to
Methods. Sage, Newbury Park CA.
Glaser BG, Strauss AL (1967) The
Discovery of Grounded Theory.
Aldine, Chicago I L
Guba EG, Lincoln YS (1981)
Effective Evaluation. Jossey Bass,
San Francisco CA.
Halcomb E, Andrew S (2005)
Triangulation as a method for
contemporary nursing research.
Nurse Researcher. 13, 2, 71-82.
Husserl E (1982) Ideas pertaining
to a pure phenomenology and to a
phenomenological philosophy.
Nijhoff, The Hague. (Original work
published 1913).
Johnson RB (1997) Examining the
validity structure of qualitative
research. Education. 118,2,282-292.
Koch T (1994) Establishing rigour
in qualitative research: the decision
trail. Journal of Advanced Nursing.
19, 5, 976-986.
Lincoln YS, Guba EG (1985)
Naturalistic Inquiry. Sage, Newbury
Park CA.
Morse J M , Field PA (1996)
Nursing Research. The Application
of Qualitative Approaches. Second
edition. Chapman and Hall, London.
Murphy E, Dingwall R (2003)
Qualitative Methods and Health
Policy Research. Aldine De Gruyter,
New York NY.
Perakyla A (1997) Reliability and
validity in research based on tapes
and transcripts. In Silverman D (Ed)
Qualitative Research. Theory,
Method and Practice. Sage,
Thousand Oaks CA, 201-220.
Priest HM, Roberts PM,
Woods LP (2002) An overview of
three different approaches to the
interpretation of qualitative data.
Part 1: theoretical issues Nurse
Researcher 10,1, 30-42.
Punch KF (1998) Introduction to
Social Research. Sage, London.
Roberts PM (1999) The develop-
ment of NEdSERV: quantitative
instrumentation to measure service
quality in nurse education. Nurse
Education Today. 19, 5, 396-407
Roberts PM, Woods LP (2000)
Alternative methods of gathering
and handling data: maximising the
use of modern technology. Nurse
Researcher. 8, 2, 84-95.
Robinson Kurpius SE,
Stafford ME (2005) Testing and
Measurement A User-Friendly
Guide. Sage, London.
Robson C (1994) Analysing
documents and records. In
Bennett N, Glatter R, Levacic R (Eds)
Improving Educational Management
Through Research and Consultancy.
Paul Chapman, London, 237-247
Stiles WB (1993) Quality control in
qualitative research. Clinical
Psychology Review. B , 6, 593-618.
Streubert Speziale H, Carpenter D
(1995) Qualitative Research in
Nursing: Advancing the Humanistic
Imperative. LippincotL Philadelphia PA.
Weber RP (1990) Basic Content
Analysis. Second edition. Sage,
Newbury Park CA.
Williamson GR (2005) Illustrating
triangulation in mixed-methods
nursing research. Nurse Researcher
12, 4, 7-18.

NURSING STANDARD July 12 :: vol 20 no 44 :: 2006 45

View publication stats