194 CLINICAL METHODS

Lynn Benfield BSN RN CCRN Staff NurselPrecepior Pres research on the topic area Thus a study has a if
byterian Hospital , , clinical focus specialists. in that, clinical area or in
Address , - a related ,
reprint requests :
area might be asked to review data col
lection instruments Instruments that evolve froma -
10 Janice K .
specific theoretical or conceptual framework are
University North Carolina CharlotleJanken PhD RN best reviewed by experts on the theory or concept
Such reviews .
. of at ,
College Nursing are designed to assess the content strumentvalidity
ofpossessesaninstrumentsufficient that is whether .
of , .
the
number and ypes of

Charloise 28224 , - in
NC , .
Copyright 0 1992byIV.B Saunders Company

,
0897189719254.000755 .00 ..

InstrumentReview Getting the Nunnallyto 1978 Because, of the complex nature

//
items represent
- the desired domain of content
Most From a Panel of Experts of many nursing topics experts from several dis
). for

: ciplines useful as content reviewers -

( , may be ,

data collection instruments For example in a

nursing study of eating disorders among adoles
Linda Lindsey Davis olescent, psychology .

and clinical nutrition

cents expert reviewers might be sought , as well-

-
DECISIONS as pediatric and mental health nursing from ad
ments canABOUTinfluencedataresearchcollectioutcomesninnes tant on review panels Reviewers .
Other types of content expertise ,

with expertise -
also are impor

whether an investigator's purpose is to explore

describe or explain phenomena of interest Nurs such as developing . in

ing research instruments originate from a variety , - the structural aspects of instrument construction
sources clinical observations or hunches . of and using rating scales and in -
, of instrument items with the statistics ,
existing
dices or matching the desired level of measurement
literature on a topic of interest conceptual or theo to be used in
: ,
retical models or adaptation of preexisting instru able assistance in instrument reviews In general
ments used in earlier studies , - -

Thus techniques -

that ,
improvedata ,collection instruments continue to be studytesting conceptsthestudyhypothesistheory can also provide valu
or problem .

a topic of interest for both experienced as well as expert reviewers should have expertise with the

. , which ,
the topic content of the instrument or expertise
governs

beginning nurse researchers One method that withfecsthe structural format, of an instrument Hav
many investigators will use to evaluate an instrument,
ment is a panel of experts The purposes of - imizes the likelihcoodnstructionofhavingtechniquesinstrumentswhichthataf- -
this on a -
article are to discuss .
criteria for selecting experts ing both types of expertise ;
review panelare

max
for a review panel anddiscuss. strategies that panel of exp rts should have at least two reviewers
to can well et

improve the outcomes of the expert panel method Waltz .

 construc d - as well as content valid A

for instrument evaluation who are exper s in the content area to be measured
.

1991 and at least one expert who is

CHOOSING INSTRUMENT REVIEWERS ., ),
selection

The .

of individuals review knowledgeable about instrument construction

It is often difficult to find reviewers who possess

both
to and cri (

tiqueonthedataindividual'scollectionexpertiseinstrumentswith istheusuallytopic basedtobe - possess onlyoneoftheseabilities .

both types of expertise If individual reviewers

sudied alc He or 19 is an investigator

it , Sons an .in view panel at different points ,
vestigator must make n selecti g a review pa el is may find it useful to work with subsets of the re -
in o
;

. . in the inst ument

gat r wished to devel p an instrument f a -
how to def ne the necessary expertise f pa el review pr cess F r example supp se an investi
members An expert panel member can be defi ed mailed
by
certa ncharacter st cs A experthas d cume ted surveytheirselfidividuals ,
. with
chr nic illnesses n
care practices The investigat w uld
.
who
achieved profess onal certification a related eed to c
ve e a panel that included eviewe s
clinical exper ence w th the target p pulati were

experts c ncepts and the ies f self

topic area presented professio al papers the
area; at state reg o alora d ati al pro care a d chr . ic illness as well as eviewewhos -
meet ngspubl shed papers on the t
fess onal

top c
pic

, . / - were k wledgeable ab ut the devel pment and

area
;
resources
lesti repg rt regi nal
survey measu es
i

self
CUNICAL METHODS 195

tigator would want to use reviews from the first group ical definitions of concepts as well as a list of
of experts as the basis for adding deleting or refining
topic items on chronic illness and self which instrument items are proposed to measure
care to the instrument Responses , -
from the , each of these concepts By having information on
second the conceptual or theoretical basis of the instru
subset of experts would be used to determine the
mostitemsffectiveformat and structure for instrument ment reviewers .
. also can be asked to judge -
. , how
STRATEGIES FOR PREPARING EXPERTS
well they believe the concepts have been repre
TO REVIEW
sented in the instrument For example a content
An instrument review can be fragmented or mis reviewer who knows items 9 to 19
directed if expert reviewers are not oriented to - -
on this hypo
the
thetical instrument have been ,
to-
conceptual basis of the study Every reviewer sure the division . developedmea
should have a working knowledge of the study
of labor differentiation -
including definitions hypotheses. to be tested , and
on a
the general goals for instrument usage In addition to providing
reviewers with copies of ,the instru nursing unit is better able to critique the desired
domain of content of these 10 items
ments Waltzet.1991, recommend. investhat. In another variation of this ( )
approach to estab

lishing congruence between .

theoretical and - -

op

erational definitions for this hypothetical study of

clinical nursing units each reviewer is given a
copy of instrument ,-

and
initi ns plus a separate list of
objec ives purpos
itemsto def
with - be ,

reviewed Expert
the

, information

,
instrument purpose as well as a list of pertinent match each item with the appropriate objectivel

reviewers

tigators provide reviewers ( ) a . are then asked to

, be asked to match each it m on e Clinical Nurs
ing one , -

instrument A sample of a oneand backe Measure with

d fiUnition For example a reviewer could

sudy sary pins optist retiris purpose for/

differentiation formalizati n or decentralization

In . of the three definitions

page attachment

ground to provide a comprehensive review of the this way an investigator can determine whether
of and -

type philosophical
theoretical orientation

a hypothetical instrument designed to provide this ,

reviewers judge items to be appropriate :

expert

in -
. When attached to the Clinical Nursing Unit in - dicators definitions ,
, As an additional dimension of .
for reviewers is found in Table 1 of the study constructs
content valida

tion reviewers () be to .

strument Table I provides reviewers with theoret should askedrate each item
Table Instrument Objectives Concepts Definitions and Organization of Items on the Clinical
 Nursing Unit Measure
Theoretical, Instrument kers
Differentiati Definition,
Number and categories of different health care Items 9 to 19 : Respondents are asked to check
Concept

workers on the nursing unit from a list it types of workers how many
provide patient care services in the setting One
point will be given for each ,
-, .
higher the the the .
score greater checkedunit

differentiation ,

Items 20 80 39 : item The

written policies exist which govern this patient
Formalization Existence of written policies which limit the . Respondents are asked whether
care activity on the unit One point will be

,
nature and scope of nursing practice on the given for each checked . ".
yes
The
higher the
the the "
score, greater unit level formalization
1 . lems40 10 49 :
Decentralization Amountofnursing participation in patient care Respondents will be asked how

decisions on the nursing unit likely is it that nurses on the unit would
participate in making this type of .
decision Each

item will have a likert scale with scores ranging

Scores
from likely to highly likely). .

(5

not ( 1 ) Higher scores represent

50

can range from 10 the nursing unit

more decentralization -..

Antin intes penity it in to diestin helene minting wes governing nursing work
 196 CLINICAL METHODS

for its relevance in representing the topic of in er item ratings of experts n rder to make decisions
est To quantify this rating process expert review-
f the items As
. rating scales Fig step in this pr ce nterrater agreement in
should asked item exclusi

rating scale the

about
rvieers
inclusi
useftherating scale should
. be
ers be
dec ntralization it ms to use 2, n

ure 1 is an example of afor,- - assessed Figure

on the - content relevance shratingws fther aclassification30 of item
clinical nursing unit "
measure Relevance rating scales insure that re - item instrument
expert reviewers Interrater agreement
viewers. assess each item separatelyof. and/ quantify /- is
calculated by adding the number of agreements
that assessment
Whenever possible reviewers should receive in - the .
formation on the plan for scoring and interpreting oramong2byboth ratersexpertplusreviewallitersmsallratitemsd3 orrated4byI

two
instrument . ,
By having information on meth
od items both raters and , (
s() . - ) dividing by the total number of

instrument construction xperts can determ ne .

items
. range from 0to 1.
for item scoring and interpretation Possible scoresIf theforesultaninterratinteragreementateragreecan
priate level of measurement for testing ,

specific

ment score is unacceptably low lessthe than - ),

whether the proposed format will yield the appro
planned
rating

- scale

70
sino know theres way miles not resires expert reviewers may not be using ( .
comparable fashion Martuza 1977 a
interrater score results

lating decentralization with the number of months an investigator ).If low

a nurse has worked on a nursing unit would not with reviewers If

likely recommend a yes should first

no
review the concept, , (
scoring , experts do not agrecdefinitionaconcept definition it
format for contin
uesshouldtobe lowrevisedthe Ifinvestigatorinterrateragreementshouldthen. ,-

Figure This dic oto the use of , check

mous scoring approach would limit the choice of
statistics for testing a corr lational hypothesis -

n 1.
the rating scale by reviewers to ensure
the two shown " /".
items they are applying the scale in a comparable man

ner For example one expert may be using the -

DECISION RULES FOR USING EXPERT REVIEWS . full
. range of relevance scores scores of 1 to 4 )to rate

Although differences in the educational and ex second expert may be using only scores
periential backgrounds of panel members can - the , (
en ;
hance the nature and scope of instrument reviews or 4 In this case reviewers need to be trained in
- items3.

the use of the ,

such alternative perspectives increase the likeli rating scale
hood of disagreements among experts Waltz - Once there is an acceptable level of reviewer
, .

et al
agreement on the use of the rating scale content
1991 recommended that investigators compare . validity of the items can be estimated ,
Investiga

. tors will find decision rules helpful for combining

( )
responses of expert revieweFors . -
Relevance Scale sim

ple decision rule for retaining individuexampleitems

-
1 • not relevant 3 - quito relevant would be to use only .
those items which ,

somowhat relevart d = highly relevant are rated as

2-

(
3 quite relevant or 4 highly relevant by both

COLUMNA COlUMN B Rater Scores

Hem Relevance Scal 1 or 2 30r4 Totals
1 .How consing nurses your
mira otten do someone 0507 3 .

Totals 4 25

)
212330

Instrument item relevance check sheet

Figure 1 .
)(
Reviewer directions:Theinstrument items shown in
Column A have been developed to measure thecon. Agemene • 2221 - 00
cept defined beiow Please read each item and score - it In Figure 2
Column Interrater agreement onitema30 instru ment
., -
Bfor.its relevance in representing the con
cops The Decentralization the amountofnursing - for point content-. two reviewers using a four
. relevance scale
CLINICAL METHODS 197

content reviewers To calculate an overall Index of collection instruments Panel reviews provide op
Content Validity CVI portunities for investigators to secure valuable -
divide the total number of ex

items ranked 3 or 4 by both reviewers by the total pert consultation from colleagues in nursing and
),

. ( -

other disciplines .
number of items Possible CVI scores can range Careful use of this expertise is

For the 30 ratings shown in Figure 2 , one of the most important components of the in -
from 0 to 1 . . .

agreement strumentdevelopment process Finally an inves

the CVI of 21 of7 .30 is .is%70 tigator also should keep in mind that the experts

There

among these two reviewers on the content validity who participate in instrument review have a pro
of the 30 items For new instruments investigators

should seek 80 or better agreement among re - . , -

viewers. % fessional investment in the study As such it is- an
sue at count of the inspent they have
. SUMMARY . ,
reviewed as well as the outcomes of the study
Authors on methods frequently recom , .
ofresearch

mend the use p nel of exp rts to maximize - the

valid well ion
likelihood of content construct d data Martuza V . ). and
1977 me t ducationfrenormBoston MA Allyn

measur Applying

panel review process as series of st ps b ginning eferenced

ced crise

collection ins ruments F gure 3 summariz s the & , - , ( : -

the- Bacon of -
Nunnally 1 )
with first r 1978 Psych metric . theory New York NY
instrum nt and nding
with pilot . resting of the instrument When ,
items are revised as a result of exp rt r vi ws the
revised ins rument also s ould be subj ct d to this
same series of st ps .,
This ar icle has . been an attempt to provide cri
teria for selec ing expert review rs and maxi
From the Scho l Nursing University of Al b ma s Bir
their participation in assessment of - .

,
- data -.(.,

,
mizing

room x .
Arsessiod of wine indesanor
collection instrument Birmingham UAB Station . 1210

Develop first draft of data .

Birmingham AL 35291
Copyright © 1992 by IV.B Saunders Company

,.
000855.00 , - .

on thelr knowiedge of , 189719210 01

Select expert reviewers based

/ Orient
0897
Instrument
The research development
opic

problem chniques
theory ^ experts Focus Groups A Useful
purpose theoretical - -
definitions and op rational to study /
, Technique Research
evaluaiesSubsetof instrumentreviewpanel Practicein :

for and
, Nursing

definitions Svatuet ot resitum panel

for item structure
with teement of quesion

clarimen propensive format level

for item relevance Helen Reiskin

! develop tud soly, ,

combining expert FOCUSbusinessGROUPSandindustryHAVElongGoldmanbeenused1962in );

responses
Use results of expert review to refine however it is not until recently that nurses have
and improve instrument item content begun to find them helpful( ,
and format as needed been used for example to evaluateFocusgroupsnewlyhaveem
ployed, , registered . &

Pilot lest instrument nurses Howard Hubelbank -

Moore(1989 to , develop, a),nurseretention,paro
),
Figure 3 . Stops in instrument review , gram DesRosier & Zellers ( 1989 to enhance -
-
.