See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/228453786

Modern Psychometric Methodology Applications of Item Response Theory

Article  in  Rehabilitation Counseling Bulletin · April 2007

DOI: 10.1177/00343552070500030501

CITATIONS READS

19 1,001

4 authors:

Christine A. Reid Stephanie A Kolakowsky-Hayner

Virginia Commonwealth University Santa Clara Valley Medical Center
24 PUBLICATIONS   201 CITATIONS    82 PUBLICATIONS   3,214 CITATIONS   

SEE PROFILE SEE PROFILE

Allen Lewis Amy Armstrong

State University of New York Downstate Medical Center Virginia Commonwealth University
48 PUBLICATIONS   585 CITATIONS    34 PUBLICATIONS   167 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Role and Function Study of Life Care Planners View project

Remote ADOS Administration View project

All content following this page was uploaded by Christine A. Reid on 29 August 2014.

The user has requested enhancement of the downloaded file.

RCB 50:3 pp. 177–188 (2007) 177

Modern Psychometric Methodology:

Applications of Item Response Theory

Christine A. Reid Item response theory (IRT) methodology is introduced as a tool for improving assess-
Stephanie A. ment instruments used with people who have disabilities. Need for this approach in
rehabilitation is emphasized; differences between IRT and classical test theory are clar-
Kolakowsky-Hayner ified. Concepts essential to understanding IRT are defined, necessary data assumptions
Allen N. Lewis are identified, and specific data analysis techniques, using software such as TESTFACT
Amy J. Armstrong and BILOG, are described. An example of IRT analysis applied to a subtest of the Gen-
Virginia Commonwealth eral Aptitude Test Battery (GATB) administered to people who have diverse disabilities
University is provided. Future potential uses of this approach in rehabilitation are outlined.

T
he purpose of this article is to introduce item re-
sponse theory (IRT) and the associated latent trait
and item response modeling techniques in an ef-
fort to improve psychometric methodology focused on the
assessment of people who have disabilities. The need for
such methodology in rehabilitation applications is em-
phasized, and examples of applications in related areas
are provided. Differences between IRT and “classical” test
theory are clarified, and implications of those differences
are discussed. Key constructs necessary to understand IRT
methodology are defined. Statistical software packages
necessary for IRT analyses are described, and resources for
developing expertise in using them effectively are identi-
fied. A practical example of IRT analysis of one of the
subtests of the General Aptitude Test Battery (GATB; U.S.
Department of Labor, 1970) administered to a rehabilita-
tion client population is given, accompanied by explana-
tions about how to select the appropriate analysis options
and how to interpret the resulting statistical output for
each phase of the analysis. Implications of the findings
from this example application are then discussed.
I MPORTANCE OF IRT
IRT techniques have tremendous potential for solving
measurement problems in rehabilitation. A test developer
using IRT generates a mathematical function to describe
the relationship between test performance and ability or
trait level. This function can be used to estimate a wide
range of ability levels, such as those common among sam-
ples of rehabilitation clients, regardless of the range of
ability within the original normative sample (Hambleton
& Swaminathan, 1985). Because this methodology fo-
cuses on the relationship between each individual item
and the underlying (latent) trait or ability assessed by the
instrument, each of those items can provide meaningful
information about an examinee. In contrast, traditional
testing requires administration of the entire test to each
examinee to meaningfully interpret the test results. If a
test has been developed using IRT, the level of the ability
or trait measured can be estimated for an individual com-
pleting any subset of the test items; administration of the
entire test is no longer necessary. Different subsets of
items can be given to different examinees or to the same
examinees at different points in time; interpretation of
the results does not depend on consistent administration
of one standardized set of items. Estimates of an exami-
nee’s individual ability or trait levels are generated
through formulas computing the levels along that latent
continuum with the greatest probability, given that indi-
vidual’s pattern of responses to the particular subset of
items administered. In contrast, for assessment instru-
ments developed without IRT techniques, meaningful in-
178
terpretation of test results requires administration of the
entire test and comparison of the obtained total score
with the results obtained from an appropriate normative
group. Applying item response models to tests used with
rehabilitation clients can provide more information about
the relationship between performance on test items and
the underlying ability or trait of interest than can tradi-
tional item analysis techniques.
IRT techniques are potent tools for studying the psy-
chometric characteristics of tests administered to rehabil-
itation populations. The long list of potentially useful
applications includes (a) study of potential bias in assess-
ment instruments, (b) generation of models to extrapolate
estimation of ability for individuals outside the range stud-
ied in the standardization sample, (c) enhanced equating
of the results of newly developed instruments with those
of old standards, (d) assessment of how accurately an in-
strument measures across the continuum of ability, (e) de-
velopment of instruments to measure accurately at a selected
target level of ability, and even (f) development of com-
puterized adaptive tests. In computerized adaptive testing,
the instrument “tailors itself” precisely to the individual
ability or trait level of each examinee, precisely measuring
ability with a fraction of the number of items required for
conventional testing.
Traditional scale development statistics such as item
difficulties (p values) and overall item–test score correla-
tions are no longer the only criteria available to test de-
velopers for selecting the best items for a test. Using IRT,
researchers can specify how well individual items distin-
guish between individuals who do or do not possess a par-
ticular target level of the trait or ability of interest. The
degree to which guessing negatively affects measurement
accuracy can also be assessed on an item-by-item basis.
Such detailed specification makes it possible to examine
the expected degree of measurement error at particular
levels of the ability or trait on an item-by-item basis,
rather than relying on one global standard error of mea-
surement (SEM) conventionally considered the counter-
part to overall reliability of the test as a whole. Test items
can be selected to minimize measurement error and max-
imize information gained at a particular targeted level of
ability.
ITEM R ESPONSE THEORY P RINCIPLES
IRT is a “modern” test theory utilizing a set of proposi-
tions or mathematical models related to individuals’ re-
sponses to items, providing a probabilistic way of linking
observable data to theoretical constructs, with the ability
to statistically adjust scores for properties of test items
such as difficulty, discriminating power, and liability to
guessing (Embretson & Reise, 2000; Lord, 1980; Van der
Linden & Hambleton, 1997). In its simplest form for di-
chotomized (right or wrong) answers, IRT specifies that
Rehabilitation Counseling Bulletin
the probability of answering an item correctly should in-
crease in a predictable manner as the level of the exami-
nee’s ability increases. When one is measuring traits (such
as level of depression), the dichotomy is between en-
dorsed and not endorsed, rather than right and wrong,
and the probability of endorsing an item should increase
in a predictable manner as the level of the examinee’s trait
in question increases. A plot of the probability of answer-
ing an item correctly (or endorsing the item) at every
level of examinee ability (or level of the underlying trait)
results in a curve that increases in slope until it reaches a
point of inflection, at which point the rate of increase
begins to decrease, so that the curve has a stretched-out
S shape. Simple linear regression does not capture the
complexity of this nonlinear relationship; a more sophis-
ticated model is needed. An item characteristic curve can
be fitted to the data, with characteristics of that curve de-
termined by its corresponding item response model.
The simplest model (the one-parameter logistic, or
Rasch model) requires an assumption that items differ
from each other only in their degree of difficulty. Item
characteristic curves for a test developed using this model
would look remarkably similar to each other, differing
only in their relative location on the ability or trait con-
tinuum, parallel to each other in every other way.
In contrast, item characteristic curves fitted to the
data using a two-parameter model would differ not only in
relative placement along the ability continuum but also in
the degree to which probability of answering an item cor-
rectly (or endorsing it) increases steeply with an increase
in level of underlying ability (or level of trait). Some
curves will have rather steep slopes before they hit the
point of inflection; others will be relatively flat. Curves
with a steep slope show evidence of more power to dis-
criminate between people who are above or below that
particular level of ability or trait.
Item characteristic curves fitted using the three-
parameter model differ not only in their level of difficulty
and their power to discriminate between people who are
or are not at a particular level of ability (or trait); they also
differ in the degree to which examinees with a low level
of ability (or trait) will answer that item correctly (or en-
dorse it). The lower asymptote of the curve (bottom left
corner of the S curve) will be higher for an item that is
easy to guess correctly and lower for one that is not so easy
to guess.
For items with multiple response options (such as a
rating scale) instead of one dichotomous answer, more
complex IRT models can be used. In essence, these mod-
els generate separate item characteristic curves for each
response option of each item, and they base ability or trait
estimates on the pattern of those responses. Santor, Ram-
say, and Zuroff (1994) used this technique to examine the
Beck Depression Inventory and to evaluate it for potential
gender bias.
Volume 50, No. 3 (Spring 2007)
An outstanding text for developing an intuitive (vs.
a mathematically based) understanding of IRT is Baker’s
(1985) classic, The Basics of Item Response Theory, which
includes a companion tutorial computer program. Al-
though the text in its original format is out of print, it has
been revised (Baker, 2001) and is now available online at
no charge from the ERIC Clearinghouse on Assessment
and Evaluation at http://edres.org/irt/baker/. The down-
loadable companion interactive tutorial (also provided
free of charge) includes specific exercises to parallel each
of Baker’s chapters, which help users build a conceptual
understanding of how IRT techniques work and why the
results have such useful characteristics.
“MODERN” VERSUS “C LASSICAL”
TEST THEORY
Latent trait modeling, item response modeling, and IRT
methodology are roughly equivalent terms used to de-
scribe techniques applying “modern psychometric theory,”
as opposed to “classical test theory.” Key differences be-
tween these two have been discussed by Baker (1992),
Cohen and Swerdlik (2005), Hays, Morales, and Reise
(2000), Hulin, Drasgow, and Parsons (1983), and Reid
(1995). IRT was developed to remedy at least three prob-
lems with psychometric assessment based on traditional
(classical) test development. The first problem with tradi-
tional methods lies in the fact that norms used for inter-
pretation of the test scores are sample specific. Both the
average level of ability and the range of abilities among
individuals whose performance was used to norm the test
influence subsequent test results. In contrast, IRT tech-
niques can be used to generate a mathematical formula to
extrapolate beyond the range of ability of the standardiza-
tion sample. This is particularly important for some reha-
bilitation populations, whose level of performance on a
given assessment instrument may be outside the range of
the original sample used to develop the test.
Another problem with traditional test development
methods is the necessity of administering the entire in-
strument to a given individual to obtain any usable in-
formation about his or her performance. A correct or
incorrect response to any individual test item is relatively
meaningless without information about performance on
the rest of the items. Any modification to shorten the in-
strument would require renorming, as if it were a com-
pletely new instrument. Adaptive testing, in which the
response to one item determines which item should be ad-
ministered next, is not possible under these conditions. In
contrast, IRT techniques model the relationship between
ability (or trait) and performance for each individual
item.
One more problem with traditional test development
methods is the instability of scores at extreme (either high
179
or low) levels of an ability or trait, even within the nor-
mative sample. Although the traditionally reported stan-
dard error of measurement is assumed to be constant
across all levels of an ability or trait, in practice it is higher
at the extreme levels (Hambleton & Swaminathan, 1985).
Generally, a test most accurately and efficiently measures
ability when the average level of item difficulty is equal
to the tested individual’s own ability level, the point at
which the examinee has a 50% probability of answering
each item correctly (Bolton, 2001). Through the use of
IRT, tests can be tailored to a desired ability level to ob-
tain the most accurate estimate of ability possible within
the shortest possible period of time. Such an increase in
measurement efficiency is important for rehabilitation ap-
plications, in which comprehensive assessment of knowl-
edge, skills, abilities, interests, values, and so forth may be
needed within a relatively short period of time.
KEY TERMS DEFINED
Latent trait: the underlying ability or trait presumed to be
measured by the test items. Although models of a latent
trait cannot be tested directly (because the latent trait can-
not be directly observed), they can be tested indirectly
through their implications for the joint probability distri-
bution of the item responses for a set of items (Borsboom,
Mellenbergh, & van Heerden, 2003).
Item characteristic curve (ICC): the mathematical
curve (logistic function) describing the relationship be-
tween the ability (or level of trait) of examinees and the
probability of answering a particular item correctly (or en-
dorsing it). The equation for an ICC under the three-
parameter logistic model can be expressed as
P(θ) = c + (1 − c) / {1 + exp[−1.7a (θ − b)]},
in which θ is the latent trait (or ability) and a, b, and c are
the item parameters. Equivalent yet alternative versions
of this formula can be found in Baker (1985) and Lord
(1980).
Difficulty parameter (b): the examinee ability level at
which approximately half of the examinees are likely to
answer a particular item correctly or endorse it. (Note
that the probability of a correct answer is greater than .5
at b for the three-parameter model.) The b parameter is
the inflection point on the item characteristic curve. It
corresponds to the ability or trait level best measured by
that item.
Discrimination parameter (a): the degree to which the
item has power to discriminate between individuals who
have or do not have the corresponding b level of a partic-
ular ability or trait. (Note that a is proportional to the
slope of the ICC at point b.)
Pseudoguessing parameter (c): the probability that a
low-ability examinee (or one with a low level of the trait
180
in question) would answer the item correctly (or endorse
the item), possibly through guessing. The c value serves as
the ICC’s intercept on the graph axis representing proba-
bility of obtaining a correct (or endorsed) score.
One-parameter model (Rasch model): an IRT logistic
model in which only the difficulty parameter (b) is con-
sidered relevant. Discrimination (a) and pseudoguessing
(c) parameters are considered constant; items that do not
satisfy such assumptions are excluded from tests developed
using this model.
Two-parameter model: an IRT logistic model in which
the item difficulty (b) and item discrimination (a) are
considered, but the effect of pseudoguessing (c, the lower
asymptote of the ICC) is assumed to be negligible.
Three-parameter model: an IRT logistic model in
which the item difficulty (b), item discrimination (a), and
pseudoguessing (c) parameters are all considered relevant
in estimating the ability of examinees.
Item information function: a description of how much
information about examinees’ underlying abilities or traits
is available at various levels along a particular ability or
trait continuum. The greater the item information is for a
given ability level, the less error is involved in estimating
examinee ability or trait level through use of that item.
Item information is inversely related to measurement
error associated with that item.
Test information function: the summed combination of
all the item information functions for a given test. This
function describes the degree of information provided by
the test at each level of an ability or underlying trait.
APPLICATIONS OF IRT
The value of applying IRT methodology to solve psycho-
metric problems has been well documented. Advantages
of applying IRT to a variety of assessment challenges have
been identified by Aiken (2003), Cook, Monahan, and
McHorney (2003), Hambleton (1989), Reid (1995), San-
tor and Ramsay (1998), and others. Examples of studies in
which IRT techniques were used to shorten instruments
used in rehabilitation or health care settings include
Bullis, Reiman, Davis, and Reid (1997), Bullis, Reiman,
Reid, and Davis (1995), Chang and Gehlert (2003), Jenkin-
son, Fitzpatrick, Garratt, Peto, and Stewart-Brown (2001),
Jenkinson, Norquist, and Fitzpatrick (2003), Kopec et al.
(1996), Mungas and Reed (2000), and Saliba, Orlando,
Wenger, Hays, and Rubenstein (2000). The usefulness of
IRT to develop computerized adaptive testing (CAT) ap-
plications for rehabilitation or health care settings was ad-
dressed by Cella and Chang (2000), Stoloff and Couch
(1992), and Ware, Bjorner, and Kosinski (2000). Exam-
ples of a focus on IRT equating or linking items between
tests related to rehabilitation or health care include arti-
cles by Hays et al. (2000), McHorney (2002), and Mc-
Rehabilitation Counseling Bulletin
Horney and Cohen (2000). A study by Haley, Andres,
and colleagues (2004) addressed both test shortening and
item equating applications for post–acute care settings
using IRT methodology. Use of IRT techniques to exam-
ine scale properties for common item equating and CAT
applications, again for post–acute care settings, was ad-
dressed by Haley, Coster, and colleagues (2004). The use
of IRT to identify differential item functioning (DIF) or to
distinguish between bias and real differences in an ability
or trait among groups was addressed by Fayers and Machin
(2000), Gumpel and Wilson (1996), and Hahn and Cella
(2003). Further detail about methodology for detecting
DIF was provided by Raju (1990).
To illustrate the potential for IRT to solve practical
testing problems, two applications were described more
fully by Reid (1995). The first is the promise of comput-
erized adaptive testing. Although computerized adaptive
tests are not yet commonly available for use with popula-
tions of people who have disabilities, development of such
tools would make it possible for tests to interactively “tai-
lor themselves” to the ability level of an individual in a
minimum amount of time. A computerized adaptive test
begins with administration of an item of medium diffi-
culty, and the examinee’s response to that item deter-
mines which level of item difficulty is administered next;
this process is repeated until the test items are primarily
those with a level of difficulty that the examinee is likely
to answer correctly 50% of the time (when measurement
error is minimized). Decreased testing time could result in
more accurate assessments by decreasing the effect of ex-
aminee fatigue. Given that computerized adaptive tests
can efficiently measure abilities without the artificial con-
straint of time limits for administrative convenience, the
influence of speed of response could be removed as a
threat to the validity of traditional timed tests.
An example of IRT that has already been applied to
a practical problem in rehabilitation was described by
Reid (1995), Bullis et al. (1995), and Bullis et al. (1997).
The problem was a barrier to the use of the Transition
Competence Battery for Adolescents and Young Adults Who
Are Deaf (TCB). The amount of time needed to adminis-
ter the instrument was prohibitive, according to feedback
from initial field testing. The TCB was designed to mea-
sure the competence of transition-age youth who are deaf
in the following areas: Job-Seeking Skills, Work Adjust-
ment Skills, Job-Related Social Skills, Money Manage-
ment, Health and Home, and Community Awareness.
Psychometric characteristics of the TCB were strong, but
administration of the entire instrument required more
than 6 to 8 hr. IRT techniques were used to develop a
“mini” screening version of the TCB. Items were selected
that best measured ability at a targeted level of proficiency.
Examinees who “pass” the screening version should not
have to spend time taking the entire TCB. In contrast,
examinees who do not score at sufficient levels on some
Volume 50, No. 3 (Spring 2007)
sections of the screening version should then take the cor-
responding subtests of the larger TCB exam to provide
detailed information about exactly which of their com-
petence deficits could benefit from further services or
education.
Statistical Software for IRT
The “gold standard” for assessing the dimensionality (num-
ber of factors accounting for variance) of a data set before
conducting IRT analyses is the TESTFACT program (Bock
et al., 2003), which provides full-information factor anal-
ysis. This kind of analysis examines the factor structure
based not only on correlations between items within the
test but on patterns (vectors) of individual responses as
they relate to the underlying latent trait or ability. Simi-
larly, the BILOG-MG 3 program (Zimowski, Muraki, Mis-
levy, & Bock, 2003) is the most versatile for IRT analysis
of dichotomous items. For multiple-response-option items,
programs such as MULTILOG or PARSCALE (Du Toit,
2003) are appropriate. These programs and other software
related to IRT are available from Assessment Systems
Corporation (http://www.assess.com). A useful tutorial for
IRT data analysis, including sample data sets and instruc-
tions to format them for later analysis, is provided by
Stark, Chernyshenko, Chuah, Lee, and Wadlington
(2001) at their IRT Modeling Laboratory Web site (http://
work.psych.uiuc.edu/irt/tutorial.asp).
Testing Assumptions
The use of IRT to evaluate an assessment tool requires sat-
isfaction of several assumptions. The primary assumption
is that the test is measuring only one trait: It is “suffi-
ciently unidimensional” (Stout, 1987). Factor analysis
(preferably using tetrachoric correlations when the latent
trait, a continuous variable, has been dichotomized into
right/wrong scoring) should reveal one factor that ac-
counts for much of the total variance, with any successive
factors accounting for much less variance in the data set
(Reckase, 1979). Related to the assumption of unidimen-
sionality is the assumption of local independence: The re-
sponse to one item is not dependent in any way on the
response to any other item. (For example, one test item
cannot provide information useful in deducing the answer
to another item.)
A second key assumption is that the relationship be-
tween ability and performance on an item can be de-
scribed by a monotonically increasing function (as ability
increases, the probability of answering an item correctly
increases). Violation of this assumption would result in a
significant misfit between the generated item characteris-
tic curve and the observed data; it could also be detected
through examination of item scatter plots.
181
IRT techniques should be used to analyze “power”
tests, which are designed to assess ability without the need
for a rapid response to each item. In contrast, the scores of
“speeded” tests are significantly affected by the speed of
examinees’ responses. Most individuals should be able to
complete all of the items for a power test; very few indi-
viduals will ever complete all of the items for a “speeded”
test. Hambleton and Swaminathan (1985) provide crite-
ria to assess the degree to which a test can be considered
an appropriate power test. Details about testing this as-
sumption (and the other assumptions described here) are
provided in the case example section of this article.
Some IRT models are designed to capture the com-
plexity of the relationship between an ability (or a trait)
and performance for each item of a test; others are less so-
phisticated, and they require the data to satisfy additional
assumptions. In particular, the one-parameter (Rasch)
model assumes that all of the items are equally discrimi-
nating at their respective levels of ability. The degree to
which data fit a given model can be assessed using fit sta-
tistics following item calibration, but, if one is focused on
Rasch analysis, there is another way to screen the data for
consistency in discrimination power before calibrating the
items. Hambleton and Swaminathan (1985) recommend
calculating the biserial correlation between each item and
the total test score and examining how many of these cor-
relations fall outside of an acceptable range. An applica-
tion of such screening, as well as details of model–data fit
statistics, is provided as part of the following case example.
P RACTICAL EXAMPLE:
G ENERAL APTITUDE TEST BATTERY
This example is taken from Reid’s (1993) study of IRT
model–data fit for selected subtests of the General Aptitude
Test Battery (GATB; U.S. Department of Labor, 1970) ad-
ministered to a sample of people who have disabilities. For
clarity in demonstrating the IRT analytical techniques
used, only one subtest is examined in this article: GATB
Subtest 3, Three-Dimensional Space.
Results of the analyses are used to answer the follow-
ing questions about the psychometric properties of this
GATB subtest:
1. Does the subtest measure primarily one
construct, or is it multidimensional when
administered to people who have disabili-
ties?
2. Does speed of response have a considerable
influence on test scores?
3. What is the nature of the relationship
between underlying ability and performance
for individual items in this subtest?
182
4. Do the items in this subtest vary in level of
difficulty?
5. Do the items in this subtest vary in the
degree to which they can discriminate
between examinees who do and those who
do not have a particular level of ability?
6. Do the items in this subtest vary in the
degree to which low-ability examinees
correctly answer, possibly through guessing?
Additional questions, relevant to future research and
practice using the GATB with a population of people who
have disabilities, are also addressed:
1. What are the estimated IRT parameters for
each calibrated item?
2. Given the model–data fit found in this
study, what sample size requirements should
be considered for future study related to
item bias, development of new items to
replace outdated ones, and related topics?
The actual parameter estimates can be used for a
variety of rehabilitation applications, including the de-
velopment of computerized adaptive tests, creation of a
shortened untimed version of the test, or efficient mea-
surement at the ability cutoff levels of specific aptitude
requirements for particular occupations or training pro-
grams. The selection of which parameter model suffi-
ciently fits the data has implications for the sample sizes
necessary for further research in this area; the more com-
plex the parameter model used, the greater the sample size
needed to accurately calibrate the item parameters.
Sample Characteristics
Examinees who completed this GATB subtest were adult
clients of a vocational evaluation center in a large south-
western state during the late 1980s (identifying informa-
tion was removed from the data prior to analysis).
Response data were available for 406 examinees with a
variety of disabilities. The sample included primarily in-
dividuals with physical impairments; intervertebral disk
disorders (23.5%) were the most commonly reported dis-
ability. Other disabilities commonly represented in this
sample were orthopedic impairments (16%); hearing im-
pairments (14.5%); mental health disorders such as de-
pression, anxiety disorders, personality disorders, or
schizophrenia (14.4%); nervous system disorders such as
epilepsy, multiple sclerosis, or neural trauma (7.5%); alco-
holism or other substance abuse disorders (6.3%); cardio-
vascular diseases (3.0%); and diabetes (2.0%). Disabilities
reported for fewer than 2% of the examinees in this sam-
ple included arthritis, visual impairments, asthma, mental
retardation, speech impairments, burns, cancer, digestive
disorders, and bone disorders.
Rehabilitation Counseling Bulletin
Example Instrument
The Three-Dimensional Space subtest is the third subtest
in the GATB. This subtest was designed to be a power test
(vs. a speeded test), but with time limits. The GATB has
been described as “the best researched of the multiple ap-
titude batteries. Because of its large amount of validity
data, it should be useful to vocational counselors” (Weiss,
1972, p. 1061). However, its speededness is a barrier to ef-
fective assessment of many individuals who have disabili-
ties that affect their speed of response. Weiss (1972)
explained, “Time limits for the ability tests [of the GATB]
are purely an administrative convenience (except, of course,
when only speed of response is being measured); however,
when an administrative convenience serves to complicate
the measurement of a variable for some individuals, it is
time to reexamine its desirability” (p. 1060). Hartigan and
Wigdor (1989) criticized the GATB’s “speededness, its
consequent susceptibility to practice effects and coach-
ing” (p. 116), but recognized its potential value as a re-
habilitation tool: “To ensure that handicapped applicants
who can compete with tested applicants are given that
opportunity, the GATB should be used when feasible to
assess the abilities of handicapped applicants . . . the test
should be used to supplement decision making” (p. 14).
Identification (through IRT analysis) of the items that
function best and equating that shortened subtest of items
to the standard GATB subtest could result in a shortened
form of the GATB that does not require time limits and
would more accurately measure the abilities of people who
have disabilities.
Speed Versus Power
IRT analyses are appropriate for power tests, not for tests
where speed of response is a significant factor in perfor-
mance. Hambleton and Swaminathan (1985) recommend
examining and reporting the percentage of examinees
completing the test, the percentage of examinees com-
pleting 75% of the items, and the number of items com-
pleted by 80% of the examinees. For this example, the
extent to which speed of response affected the number of
items completed by each examinee was examined using
the traditional item analysis feature of TESTFACT (Bock
et al., 2003). The third subtest of the GATB consists of 40
items. Only 3.4% of the sample completed all 40 items;
14.5% completed 75% of the items. Only 18 items were
completed by 80% of the examinees. Because time avail-
able was insufficient for most examinees to answer all of
the items, further steps of the IRT analyses did not include
all 40 items; only those items completed by 75% of the
examinees (n = 20 total items) were retained. That set of
20 items satisfied Hambleton and Swaminathan’s (1985)
guidelines for a power test.
Volume 50, No. 3 (Spring 2007)
Analysis of Dimensionality
The 20 subtest items completed by 75% of the examinees
were then factor analyzed using TESTFACT, which is
preferred over other factor analysis programs because it
uses not only the information inherent in each individual
item response but also the additional (“full”) information
that results from analysis of the unique pattern (vector) of
an individual’s collective set of item responses. In essence,
the factor analysis takes into account the relationships
among items as they relate to the underlying latent trait,
not just as they relate to other individual item responses.
Such analysis requires computation of tetrachoric correla-
tions, which are not yet provided as options in most stan-
dard statistical software packages. Using TESTFACT, a
stepwise principal components analysis of the matrix of
tetrachoric interitem correlations was performed for this
example, with the specification to extract the maximum
possible number of factors. Next, the principal factor so-
lution served as the basis for full-information factor analy-
sis, using information from vectors of item responses. The
resulting full-information solution could then be rotated,
first orthogonally to the varimax criterion and then
obliquely by the promax method, to aid interpretation of
the resulting factors.
One method of evaluating the degree of dimension-
ality of this factor solution is to generate a scree plot of
percentage of total variance accounted for by each factor
extracted. According to Reckase (1979), sufficient unidi-
mensionality for IRT analysis is demonstrated when a sin-
gle dominant factor accounts for at least 20% of the total
subtest variance, with the second and succeeding factors
accounting for considerably less variance. Figure 1 shows
the percentage of variance accounted for by each ex-
tracted factor for the GATB subtest evaluated in this ex-
ample. This plot shows a clear “elbow” between the first
factor and the second; the first factor accounts for 32.13%
of the variance, the second for 4.40% of the variance, the
third for 3.64% of the variance, the fourth for 3.20% of
the variance, and the last for 2.90% of the variance.
Another method of assessing the “essential unidi-
mensionality” (Stout, 1987) of the subtest is to examine
differences in chi-square approximation to the likelihood-
ratio statistics for successive stepwise factor models. These
statistics are available as part of the TESTFACT output
for full-information factor analysis. The differences in
chi-square values should approximate a chi-square dis-
tribution, with degrees of freedom corresponding to the
difference in degrees of freedom associated with the re-
spective factor models (Haberman, 1977). For the exam-
ple GATB subtest, the difference between chi-square
approximation to the likelihood-ratio statistics for the
one-factor model and the two-factor model was 13.62,
with 19 degrees of freedom; this value is not significant,
χ2(19, N = 406) = 13.62, p > .05. Given that there is no
significant difference between the fit of the one-factor
183
model and that of the two-factor model, essential unidi-
mensionality is demonstrated for this example.
Equal Discrimination of Items
Different IRT models carry different assumptions about
the nature of the relationship between item responses and
the latent trait or ability measured by them. The Rasch
model (one-parameter logistic model), for example, re-
quires that items differ from each other only in level of
difficulty; the degree to which items efficiently discrimi-
nate between those who have and those who do not have
that ability (or trait) does not vary. When one uses Rasch
scaling, it is important to discard items that do not fit the
requirements of the model. In contrast, a two-parameter
or a three-parameter model better characterizes the actual
relationship between the underlying trait and item perfor-
mance for many tests. However, there are some practical
advantages to using a one-parameter model, if one can
develop items that satisfy assumptions such as equal dis-
crimination power (the a parameter) among items. For ex-
ample, scoring a test developed using the Rasch model is
easy; one just adds the number of correct (or endorsed) re-
sponses to obtain a total score. In contrast, scoring a test
using a two- or three-parameter model requires a formula
that differentially weights item values according to their
discrimination power at given levels of that underlying
trait.
Hambleton and Swaminathan (1985) recommend
screening for equal discrimination of items through calcu-
lating the biserial correlation between each item and the
total subtest score and examining how many of those cor-
relations fall outside of what they consider an acceptable
FIGURE 1. GATB Subtest 3, Three-Dimensional
Space: Percentage of total subtest variance accounted
for by each extracted factor (maximum number of fac-
tors extracted by TESTFACT).
184
range, which consists of the mean of those correlations
plus or minus a value of 0.15. (Biserial correlations can be
produced using TESTFACT, but they are also available
from other widely used statistical software packages.) For
the GATB subtest used in this example, the mean biserial
correlation was 0.665. Using the range of the mean biser-
ial correlation plus or minus 0.15, an acceptable range is
between 0.515 and 0.815. Two items (Items 12 and 13),
constituting 10% of the total examined, fell outside of this
range. Although not extreme, this degree of diversity calls
into question the assumption of equally discriminating
items required by the one-parameter model.
Generation of IRT Parameter
Estimates
Many software programs are available for calibration of
items to generate IRT parameter estimates. For dicho-
tomized (yes/no, right/wrong) responses, the BILOG pro-
gram is the most versatile. The most recent version,
BILOG-MG 3 (Zimowski et al., 2003), is designed to han-
dle multiple groups (MG), which aids in tasks such as as-
sessing potential item bias (differential item functioning).
For this GATB example, BILOG was used to generate pa-
rameter estimates for each of the 20 items using three lo-
gistic models: a one-parameter, a two-parameter, and a
three-parameter model. For the three-parameter logistic
model, item difficulty (b), discrimination (a), and pseudo-
guessing (c) parameters were all estimated. Under the
two-parameter model, only item difficulty (b) and dis-
crimination (a) parameters were estimated. Under the
one-parameter model, only item difficulty (b) was esti-
mated for each item.
Item parameter estimation used the marginal maxi-
mum a posteriori (MMAP) method, which incorporates
moderately strong Bayesian priors on the item slope and
asymptote estimates. The relatively small sample size (for
IRT analysis) was considered insufficient to permit joint
estimation of both the prior distributions and the item
parameters, so default normal prior distributions were
selected.
Model–Data Fit: Item Level
One BILOG option for assessing the degree to which the
selected one-, two-, or three-parameter model fits the ac-
tual data uses generation of likelihood-ratio chi-square
values to identify items for which the pattern of observed
responses differs significantly from the pattern of re-
sponses predicted using the given model and associated
estimated parameters. However, these values are not con-
sidered dependable for subtests with 20 or fewer items (Du
Toit, 2003; Mislevy & Bock, 1990). Although the entire
third subtest of the GATB contains 40 items, few of the
examinees in the sample for this example responded to all
40 of the items. Instead, the items to which at least 75%
Rehabilitation Counseling Bulletin
of the examinees responded, a total of 20 items, were used
in item calibrations.
Two other indexes of goodness of parameter model fit
to the data are available through BILOG: standardized
posterior residuals and the population root mean square of
the posterior deviates (Du Toit, 2003; Mislevy & Bock,
1990). To generate these estimates, BILOG divides the
latent trait (or ability) continuum into multiple discrete
chunks. Within each chunk, a comparison is made be-
tween the expected values given that particular model
and the observed values falling within that chunk. Stan-
dardized posterior residuals are standardized differences
between the actual posterior probability of a correct (or
endorsing) response at various ability (or trait) levels and
the probability of those levels hypothesized according
to the fitted item characteristic curve. Mislevy and Bock
(1990) indicated that a standardized residual greater than
2.0 may indicate failure of an item response model to fit
the data at that point. However, Mislevy and Bock advise
taking into consideration the posterior weight, or propor-
tion of the examinee population at that level of ability.
Model misfit in the tail ends of the ability (or trait) con-
tinuum, where the probability of examinee performance
at that level is extremely low, is expected to have little ef-
fect. Therefore, only standardized posterior residuals with
posterior weights of 0.05 or greater are considered in this
example analysis. Using this criterion, the GATB exam-
ple subtest revealed 12 items (60%) evidencing some de-
gree of misfit under the one-parameter logistic model, 3
items (15%) evidencing some degree of misfit under the
two-parameter logistic model, and 1 item (5%) evidenc-
ing some degree of misfit under the three-parameter logis-
tic model.
The population root mean square of the posterior de-
viates is an overall index of fit for each item, given the fit-
ted item characteristic curve. This index is the square root
of a weighted average of the squares of standardized pos-
terior residuals at selected points along the ability contin-
uum. Again, a value greater than 2.0 suggests misfit of the
model for the particular item. GATB example results are
presented in Table 1, which identifies six items (Items 1,
4, 7, 11, 12, and 13) as misfitting under the one-parameter
model and no items misfitting under either the two-
parameter or the three-parameter model.
Overall (Subtest-Wide) Comparison
of IRT Model–Data Fit
Within BILOG, overall comparisons of the fit of different
response models can be made through examination of the
differences between the −2 log likelihood values associ-
ated with each model at the end of the last calibration
cycle for each subtest. The difference between values should
be distributed approximately as chi-square with degrees of
freedom equal to the difference in the number of parame-
Volume 50, No. 3 (Spring 2007) 185

ters estimated (across items) in the test or subtest of in- TABLE 1. GATB Subtest 3, Three-Dimensional
terest. Degrees of freedom for the three-parameter model, Space: Root Mean Squares of the Posterior Deviates
for example, would be three times the number of items 1-parameter 2-parameter 3-parameter
(because a, b, and c parameters are estimated for each); logistic logistic logistic
degrees of freedom for the two-parameter model would be Item no. model model model
two times the number of items; and degrees of freedom for
the one-parameter model would equal the number of 1 2.7801a 0.4092 1.0061
items in the test or subtest itself. Therefore, a difference 2 0.6966 0.5314 1.2313
between the chi-square values associated with the one- 3 1.9290 1.7944 0.6879
parameter model and those associated with the two- 4 2.6807a 0.3926 0.8977
parameter model would have degrees of freedom equal to 5 1.3964 0.7827 1.4937
the number of items in the subtest; the same rule would 6 1.6054 1.3759 1.2417
apply for the difference between the two-parameter 7 3.5519a 0.8459 1.5410
model and the three-parameter model. A significant dif- 8 1.3395 1.2185 1.6657
ference in the −2 log likelihood values, given the appro- 9 1.5238 0.6304 0.5820
priate degrees of freedom, would suggest that the model 10 0.7983 0.6174 0.6182
with the lesser −2 log likelihood value better fits the data. 11 2.6233a 0.6495 0.5410
For the GATB example subtest, there was a significant 12 4.3005a 0.9943 0.8216
difference in overall fit between the one-parameter 13 2.7847a 0.9587 1.2211
model and the two-parameter model, χ2(17, N = 406) = 14 1.6560 1.8820 1.5337
72.31, p < .001. However, there was not a significant dif- 15 1.4977 1.0919 0.9577
ference in overall fit between the two-parameter model 16 0.9389 0.7085 0.3831
and the three-parameter model, χ2(17, N = 406) = 20.05, 17 1.1285 1.2217 0.2167
p > .05. 18 1.5090 0.5457 0.3773
For this example, results of the three methods for 19 1.6392 0.5778 0.7996
assessing model–data fit are all consistent: There is a sig- 20 1.3896 1.2340 1.3679
nificant difference in fit between the one-parameter a
Root mean square of posterior deviates is greater than 2.0.
and two-parameter models, but not between the two-
parameter and three-parameter models. The data do not
fit the Rasch model; if Rasch techniques are to be used
with this subtest, the misfitting items must be discarded. TABLE 2. GATB Subtest 3, Item Parameter Estimates
for the Two-Parameter Logistic Model
However, a two-parameter model, estimating both the
item difficulty (b) and item discrimination (a) parameters, Difficulty Discrimination
fits the data nicely. It is not necessary to add the third Item no. parameter (b) parameter (c)
(c, pseudoguessing) parameter to fit the data. This means 1 −1.852 1.316
there are no significant differences among items in this 2 −1.218 0.823
subtest in the degree to which low-level examinees cor- 3a −0.855 0.895
rectly answer the items, possibly through guessing. 4 −1.564 1.184
The actual item parameters (under the two-parameter 5 −1.042 0.939
logistic model) for calibrated items in the third subtest 6 −1.436 0.663
(Three-Dimensional Space) of the GATB are presented 7 −1.047 1.258
in Table 2. Inspection of these parameters shows that Item 8a −0.401 0.808
1 is the easiest, with a b parameter of −1.852 correspond- 9 −0.079 0.638
ing to a lower point on the ability continuum. The most 10 −0.989 0.731
difficult item in this set is Item 9, with a b parameter of 11 −1.113 1.027
−0.079. This item measures best near the average level of 12 −0.136 1.334
the ability continuum (near the zero point). Interestingly, 13 −0.835 0.521
none of the items had difficulty or trait levels above zero, 14a −0.513 0.771
suggesting that this set is composed of relatively easy 15 −0.558 0.648
items. Perhaps the second half of this GATB test, the part 16 −0.285 0.811
that was not reached by at least 75% of the examinees, in- 17 −0.249 0.722
cludes more difficult items. One way to assess that would 18 −0.437 0.630
be to administer the second half of the items to a similar 19 −0.821 0.599
group of examinees and include several items from the 20 −0.328 0.657
previously calibrated set to serve as “linking” items to cal-
ibrate the next set on the same scale. a
Model–data misfit residuals are less than 2.0. or greater than +2.0.
186
The discrimination (a) parameters for the items in
Table 2 reflect the power of the item to discriminate be-
tween people who are or are not at the corresponding
b level of the trait. So, Item 12 discriminates relatively
well (a = 1.334) at the level of b = −0.136, but it would
not discriminate as well at a much lower ability; Item 1
(a = −1.852, b = 1.316) would do a better job of discrimi-
nating between people who are or are not at that lower
end of ability. The shape of the item characteristic curve
for Item 12 would be a stretched-out S shape, with the
lower asymptote representing the probability of people
with a low level of that ability correctly answering the
item (perhaps through guessing); the curve would rise at
an increasing rate until it reached the −0.136 level on the
ability or trait continuum; at that inflection point, it
would continue rising but at a lower rate, until it reached
the upper asymptote representing the probability of peo-
ple with a high ability level correctly answering or en-
dorsing that response. In contrast, an item with a lower
discrimination parameter, such as Item 13 (a = 0.521),
would have a flatter item characteristic curve, with a less
dramatic change in the probability of answering an item
correctly between an examinee whose ability is below the
b parameter (−0.835) and one whose level of ability is
above it.
The relatively good model–data fit for all but the
one-parameter model suggests that the items on this
GATB example subtest are measuring the latent trait, the
underlying ability measured by this subtest, in a pre-
dictable, understandable manner, even among a popula-
tion of examinees as diverse as the one sampled in this
study. As examinee ability increases, the probability of
answering a given test item increases in a predictable fash-
ion. For each item that fits its model, the relationship be-
tween ability and performance on that item is specified. It
is no longer necessary to look at the results of this entire
subtest to gain information about examinee ability levels,
because an estimate of ability can be generated from any
subset of these calibrated items.
Implications of This Example
Measurement accuracy for people who have disabilities
could be improved by removing the time limits on this
third subtest of the GATB, which would allow examinees
to complete each item. This modification would remove
the unfair penalty that time limits impose on individuals
whose disabilities limit their speed of response to the
GATB in its current format. In addition, it could improve
measurement accuracy through reducing the benefits of
employing a strategy of guessing or just filling in random
answers to all the items before time runs out.
The item parameter estimates generated can lead to
a variety of applications for the development of reha-
bilitation assessment tools. Development of a shortened,
Rehabilitation Counseling Bulletin
nontimed version of the GATB is an obvious application.
Other possible applications include developing tests for
specific levels of aptitude (computerized or paper-and-
pencil tests), as well as equating other test instruments to
the GATB through use of linking items selected from this
calibrated set.
Computerized adaptive testing could be developed to
tailor this GATB subtest to the particular level of ability
of each examinee. To do this, one would need to calibrate
a bank of all 40 GATB items for this subtest, expanding
the list of items resulting from this study, to ensure that
enough items would be available to accurately measure at
each level of ability. If not enough items appropriate for a
particular level of ability are currently part of this GATB
subtest, additional items should be created or imported
from another, similar test. Computerized adaptive testing
would make possible the most efficient measurement of
ability for each examinee. Measurement accuracy would
be maximized and time required to take the test would be
minimized for each examinee.
Creating adaptive tests is not the only way to tailor
tests for particular uses. Using generated item parameters,
one could select items that best measure at the cutoff level
of each aptitude required for a specific occupation or
training program, for example. Specific aptitude batteries
based on empirically derived ability levels required for
specific occupations could be developed as efficient
screening tools for rehabilitation clients considering those
occupations.
Equating other aptitude tests to this subtest of the
GATB is possible using item parameter estimates gener-
ated in this study. Given that the GATB is often held as a
standard with which to compare other aptitude assess-
ment instruments used in rehabilitation settings, equating
beyond simply correlating total test scores is quite desir-
able. Probably the easiest way to equate the items on this
GATB subtest to items in another test measuring the
same construct is to administer some linking GATB sub-
test items along with the other test. The whole set of
items, including the linking GATB items, can then be
calibrated together. To ensure that the new parameters are
on the same scale as the previously calibrated GATB
items, the item parameter estimates of the linking items
can be constrained to their originally calibrated values.
This method can also be used to test out new items for the
GATB subtest. Additional experimental items can be
added to the subtest and examined to see how their item
characteristic curves function. If an item is to be deleted
from the GATB subtest because it is outdated or biased
against people who have disabilities, a new item with a
similar item characteristic curve could be substituted.
The two-parameter logistic model seems to be neces-
sary and sufficient to describe the relationship between
underlying ability and performance on the third subtest of
the GATB. IRT modeling of this subtest with the two-
Volume 50, No. 3 (Spring 2007)
parameter model instead of a more complex model has
two advantageous implications for rehabilitation research.
First, the requirement for extremely large sample sizes to
ensure stability of item parameter estimates is relaxed.
Stable estimates of both the difficulty (b) and discrimina-
tion (a) parameters are feasible with relatively small sam-
ple sizes (hundreds, rather than thousands, of examinees).
A second benefit of less complex parameter modeling is
the feasibility of examining potential item bias or differ-
ential item functioning through comparison of the item
characteristic curves for people with disabilities versus
their nondisabled counterparts. Raju’s (1990) formulas for
calculating the exact area between item characteristic
curves, and criteria for determining the significance of the
differences between those curves, can be applied.
In conclusion, appropriate application of IRT tech-
niques could greatly improve psychometric methodology
in rehabilitation. This article has provided an overview of
such techniques, as well as guidelines for their use. Appli-
cations such as improving the efficiency and accuracy of
assessment tools, developing computerized adaptive in-
struments, identifying and rectifying potential bias in
tests, and equating new and improved instruments with
the old tools they are designed to replace could revolu-
tionize assessment of people who have disabilities.
ABOUT THE AUTHORS
Christine A. Reid, PhD, is a professor in the Rehabilitation
Counseling Department at Virginia Commonwealth University.
Her current research interests include assessment, ethics, deaf-
blindness, and life-care planning. Stephanie A. Kolakowsky-
Hayner is currently completing her dissertation in health-related
sciences with a concentration in rehabilitation leadership at Vir-
ginia Commonwealth University. Allen N. Lewis, Jr., PhD,
is an assistant professor and vice-chair of the Department of
Rehabilitation Counseling at Virginia Commonwealth Univer-
sity. He is also a senior research associate of the Community
Health Research Initiative of the L. Douglas Wilder School of
Government and Community Affairs at VCU. His current re-
search interests include exploring the disability experience in cul-
turally diverse populations, understanding health and disability
disparities, leadership in the public rehabilitation system, and the
application of program evaluation research technology to rehabil-
itation services. Amy J. Armstrong, PhD, is an assistant
professor in the Department of Rehabilitation Counseling at Vir-
ginia Commonwealth University. Her interest areas include the
employment of individuals with significant disabilities, advocacy,
welfare and poverty issues, disability policy and systems issues,
and distance education. Address: Christine Reid, Dept. of Reha-
bilitation Counseling, Virginia Commonwealth University, PO
Box 980330, 1112 E. East Clay St., Richmond, VA 23298-
0330; e-mail: creid@vcu.edu
187
REFERENCES
Aiken, L. R. (2003). Psychological testing and assessment (11th ed.).
Boston: Allen & Bacon.
Baker, F. B. (1985). The basics of item response theory. Portsmouth, NH:
Heinemann.
Baker, F. B. (1992). Item response theory: Parameter estimation techniques.
New York: Marcel Dekker.
Baker, F. B. (2001). The basics of item response theory (ERIC Clearing-
house on Assessment and Evaluation, University of Maryland, Col-
lege Park, MD). Retrieved October 30, 2004, from http://edres.org/
irt/baker/
Bock, R. D., Gibbons, R., Schilling, S. G., Muraki, R., Wilson, D. T., &
Wood, R. (2003). Testfact 4. Lincolnwood, IL: Scientific Software
International.
Bolton, B. (2001). Handbook of measurement and evaluation in rehabilita-
tion (3rd ed.). Gaithersburg, MD: Aspen.
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003). The the-
oretical status of latent variables. Psychological Review, 110(2), 203–
219.
Bullis, M., Reiman, J. W., Davis, C., & Reid, C. (1997). National field-
testing of the “mini” version of the Transition Competence Battery
for Adolescents and Young Adults Who Are Deaf. The Journal of
Special Education, 31(3), 347–361.
Bullis, M., Reiman, J., Reid, C., & Davis, C. (1995). Preliminary inves-
tigation of the “mini” version of the Transition Competence Battery
for Deaf Adolescents and Young Adults. Assessment in Rehabilitation
and Exceptionality, 2(3), 179–196.
Cella, D., & Chang, C. H. (2000). A discussion of item response the-
ory and its applications in health status assessment. Medical Care,
38(9, Suppl. 2), II66–II72.
Chang, C. H., & Gehlert, S. (2003). The Washington Psychosocial
Seizure Inventory (WPSI): Psychometric evaluation and future ap-
plications. Seizure, 12(5), 261–267.
Cohen, R. J., & Swerdlik, M. E. (2005). Psychological testing and assess-
ment: An introduction to tests and measurement. Boston: McGraw-Hill.
Cook, K. F., Monahan, P. O., & McHorney, C. A. (2003). Delicate bal-
ance between theory and practice: Health status assessment and
item response theory. Medical Care, 41(5), 571–574.
Du Toit, M. (Ed.). (2003). IRT from SSI: BILOG MG, MULTILOG,
PARSCALE, TESTFACT. Lincolnwood, IL: Scientific Software In-
ternational.
Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychol-
ogists. Mahwah, NJ: Erlbaum.
Fayers, P. M., & Machin, D. (2000). Quality of life: Assessment, analysis,
and interpretation. West Sussex, England: Wiley.
Gumpel, T., & Wilson, M. (1996). Application of a Rasch analysis to
the examination of the perception of facial affect among persons
with mental retardation. Research in Developmental Disabilities,
17(2), 161–171.
Haberman, S. J. (1977). Log-linear models and frequency tables with
small expected cell counts. Annals of Statistics, 5, 1148–1169.
Hahn, E. A., & Cella, D. (2003). Health outcomes assessment in vul-
nerable populations: Measurement challenges and recommenda-
tions. Archives of Physical Medicine and Rehabilitation, 84(Suppl. 2),
S35–S42.
Haley, S. M., Andres, P. L., Coster, W. J., Kosinski, M., Ni, P., & Jette,
A. M. (2004). Short-form activity measure for post-acute care.
Archives of Physical Medicine and Rehabilitation, 85(4), 649–660.
Haley, S. M., Coster, W. J., Andres, P. L., Ludlow, L. H., Ni, P., Bond,
T. L., et al. (2004). Activity outcome measurement for postacute
care. Medical Care, 42(1, Suppl. 1), I49–I61.
Hambleton, R. K. (1989). Principles and selected applications of item
response theory. In R. L. Linn (Ed.), Educational measurement (3rd
ed., pp. 147–200). New York: American Council on Education and
Macmillan.
188
Hambleton, R. K., & Swaminathan, H. (1985). Item response theory:
Principles and applications. Boston: Kluwer Nijhoff.
Hartigan, J. A., & Wigdor, A. K. (Eds.). (1989). Fairness in employment
testing: Validity generalization, minority issues, and the General Aptitude
Test Battery (Final report of the Committee on the General Apti-
tude Test Battery, Commission on Behavioral and Social Sciences
and Education, National Research Council). Washington, DC: Na-
tional Academy Press.
Hays, R. D., Morales, L. S., & Reise, S. P. (2000). Item response theory
and health outcomes measurement in the 21st century. Medical
Care, 38(9, Suppl. 2), II28–II42.
Hulin, C. L., Drasgow, F., & Parsons, C. K. (1983). Item response theory:
Application to psychological measurement. Homewood, IL: Dow Jones-
Irwin.
Jenkinson, C., Fitzpatrick, R., Garratt, A., Peto, V., & Stewart-Brown,
S. (2001). Can item response theory reduce patient burden when
measuring health status in neurological disorders? Results from
Rasch analysis of the SF-36 physical functioning scale (PF-10). Jour-
nal of Neurology, Neurosurgery, and Psychiatry, 71(2), 220–224.
Jenkinson, C., Norquist, J. M., & Fitzpatrick, R. (2003). Deriving sum-
mary indices of health status from the Amyotrophic Lateral Sclero-
sis Assessment Questionnaires (ALSAQ-40 and ALSAQ-5). Journal
of Neurology, Neurosurgery, and Psychiatry, 74(2), 242–245.
Kopec, J. A., Esdaile, J. M., Abrahamowicz, M., Abenhaim, L., Wood-
Dauphinee, S., Lamping, D. L., et al. (1996). The Quebec Back Pain
Disability Scale: Conceptualization and development. Journal of
Clinical Epidemiology, 49(2), 151–161.
Lord, F. M. (1980). Applications of item response theory to practical testing
problems. Hillsdale, NJ: Erlbaum.
McHorney, C. A. (2002). Use of item response theory to link 3 mod-
ules of functional status items from the Asset and Health Dynamics
Among the Oldest Old study. Archives of Physical Medicine and Re-
habilitation, 83(3), 383–394.
McHorney, C. A., & Cohen, A. S. (2000). Equating health status mea-
sures with item response theory: Illustrations with functional status
items. Medical Care, 38(9, Suppl. 2), II43–II59.
Mislevy, R. J., & Bock, R. D. (1990). BILOG 3: Item analysis and test
scoring with binary logistic models (2nd ed.). Chicago: Scientific Soft-
ware.
Mungas, D., & Reed, B. R. (2000). Application of item response theory
for development of a global functioning measure of dementia with
linear measurement properties. Statistics in Medicine, 19(11–12),
1631–1644.
Raju, N. S. (1990). Determining the significance of estimated signed
and unsigned areas between two item response functions. Applied
Psychological Measurement, 14, 197–207.
Rehabilitation Counseling Bulletin
Reckase, M. D. (1979). Unifactor latent trait models applied to multi-
factor tests: Results and implications. Journal of Educational Statistics,
4, 207–230.
Reid, C. (1993). Latent trait modeling of the General Aptitude Test Battery
used with a rehabilitation client population: An investigation of model–
data fit. Unpublished doctoral dissertation, Illinois Institute of Tech-
nology, Chicago.
Reid, C. (1995). Application of item response theory to practical prob-
lems in assessment with people who have disabilities. Assessment in
Rehabilitation and Exceptionality, 2(2), 89–96.
Saliba, D., Orlando, M., Wenger, N. S., Hays, R. D., & Rubenstein,
L. Z. (2000). Identifying a short functional disability screen for older
persons. The Journals of Gerontology. Series A, Biological Sciences and
Medical Sciences, 55(12), M750–M756.
Santor, D. A., & Ransay, J. O. (1998). Progress in the technology of
measurement: Applications of item response models. Psychological
Assessment, 10(4), 345–359.
Santor, D. A., Ramsay, J. O., & Zuroff, F. C. (1994). Nonparametric
item analyses of the Beck Depression Inventory: Evaluating gender
item bias and response option weights. Psychological Assessment, 6,
255–270.
Stark, S., Chernyshenko, S., Chuah, D., Lee, W., & Wadlington, P.
(2001). IRT Modeling Lab tutorial on item response theory. Retrieved
October 30, 2004, from http://work.psych.uiuc.edu/irt/tutorial.asp
Stoloff, M. L., & Couch, J. V. (Eds.). (1992). Computer use in psychol-
ogy: A directory of software (3rd ed.). Washington, DC: American
Psychological Association.
Stout, W. (1987). A nonparametric approach for assessing latent trait
unidimensionality. Psychometrika, 52, 589–617.
U.S. Department of Labor. (1970). Manual for the General Aptitude Test
Battery. Washington, DC: U.S. Government Printing Office.
Van der Linden, W. J., & Hambleton, R. K. (1997). Handbook of mod-
ern item response theory. New York: Springer-Verlag.
Ware, J. E., Jr., Bjorner, J. B., & Kosinski, M. (2000). Practical implica-
tions of item response theory and computerized adaptive testing: A
brief summary of ongoing studies of widely used headache impact
scales. Medical Care, 38(9, Suppl. 2), II73–II82.
Weiss, D. J. (1972). Review of the General Aptitude Test Battery. In
O. K. Buros (Ed.), Seventh mental measurements yearbook (Vol. 2,
pp. 1058–1061). Highland Park, NJ: Gryphon Press.
Zimowski, M. F., Muraki, E., Mislevy, R. J., & Bock, R. D. (2003).
BILOG-MG 3. Lincolnwood, IL: Scientific Software International.
View publication stats