variance.
The greater the proportion of the
CHAPTER 5: Reliability
Reliability
- Synonym for dependability or
consistency.
- In Psychometric sense, it only refers
to something that is consistent—not
necessarily consistently good or
bad, but simply consistent.
- Not an all-or-none matter; A test
may be reliable in one context and
unreliable in another. There are
different types and degrees of
reliability.
Reliability Coefficient
- An index of reliability, a proportion
that indicates the ratio between the
true score variance on a test and the
total variance.
The Concept of Reliability
- A score on an ability test is reflected
not only the testtaker’s true score on
the ability being measured but also
error.
- Error refers to the component of the
observed test score that does not
have to do with the testtaker’s ability.
- X=T+E
o X = observed score
o T = true score
o E = error
- A statistic useful in describing
sources of test score variability is the
variance (σ2).
- Variance from true differences is
true variance, and variance from
irrelevant, random sources is error
variance.
- σ2 = σ2th + σ2e
o In this equation, the total
variance in an observed
distribution of test scores (σ 2)
equals the sum of the true
variance (σ2th) plus the error
variance (σ2e).
- The term reliability refers to the proportion
of the total variance attributed to true
total variance attributed to true variance, the
more reliable the test.
o True differences are assumed to be
stable = they will yield consistent
scores on repeated administrations
of the same test as well as on
equivalent forms of tests.
o Because error variance may
increase or decrease a test score by
varying amounts, consistency of the
test score—and thus the reliability
—can be affected.
- The term measurement error refers to all of
the factors associated with the process of
measuring some variable, other than the
variable being measured. (Ex: administering
English-language test on the subject of 12th-
grade algebra to students from China)
o Can be categorized as being either
random or systematic.
o Random error – a source of error
in measuring a targeted variable
caused by unpredictable
fluctuations and inconsistencies of
other variables in the measurement
process.
 Sometimes referred to as
“noise,” this source of
error fluctuates from one
testing situation to another
with no discernible pattern
that would systematically
raise or lower scores. (Ex:
unanticipated events in the
vicinity of test
environment like rally or
lightning strike;
unanticipated physical
events happening within
the test taker like surge in
their blood pressure.)
o Systematic error – a source of error
in measuring a variable that is
typically constant or proportionate
to what is presumed to be the true
value of the variable being
measured.
 Once a systematic error
becomes known, it
becomes predictable—as
well as fixable. Note also
that a systematic source of
error does not affect score
consistency. (Ex: a
 systematic error source
would not change the
variability of the
distribution or affect the
measured reliability of the
instrument. In the end, the
individual crowned “the
biggest loser” would
indeed be the contestant
who lost the most weight
—it’s just that he or she
would actually weigh 5
pounds more than the
weight measured by the
show’s official scale.)
Sources of Error Variance
- Include test construction, administration,
scoring, and/or interpretation.
- Test construction – One source of variance
during test construction is item sampling or
content sampling, terms that refer to
variation among items within a test as well
as to variation among items between tests.
o The extent to which a test taker’s
score is affected by the content
sampled on a test and by the way
the content is sampled (that is, the
way in which the item is
constructed) is a source of error
variance.
o From the perspective of a test
creator, a challenge in test
development is to maximize the
proportion of the total variance that
is true variance and to minimize the
proportion of the total variance that
is error variance.
- Test administration – Sources of error
variance that occur during test
administration may influence the test taker’s
attention or motivation. The test taker’s
reactions to those influences are the source
of one kind of error variance.
o Examples of untoward influences
during administration of a test
include factors related to the test
environment: room temperature,
level of lighting, and amount of
ventilation and noise, for instance.
A relentless fly may develop a
tenacious attraction to an
examinee’s face.
o External to the test environment in
a global sense, the events of the day
may also serve as a source of error.
So, for example, test results may
vary depending upon whether the
test taker’s country is at war or at
peace (Gil et al., 2016). A variable
of interest when evaluating a
patient’s general level of
suspiciousness or fear is the
patient’s home neighborhood and
lifestyle.
o Other potential sources of error
variance during test administration
are test taker variables. Pressing
emotional problems, physical
discomfort, lack of sleep, and the
effects of drugs or medication can
all be sources of error variance.
Formal learning experiences, casual
life experiences, therapy, illness,
and changes in mood or mental
state are other potential sources of
test taker-related error variance.
o Examiner-related variables are
potential sources of error variance.
The examiner’s physical
appearance and demeanor—even
the presence or absence of an
examiner—are some factors for
consideration here. Some
examiners in some testing
situations might knowingly or
unwittingly depart from the
procedure prescribed for a
particular test. On an oral
examination, some examiners may
unwittingly provide clues by
emphasizing key words as they
pose questions. Clearly, the level of
professionalism exhibited by
examiners is a source of error
variance.
- Test scoring and interpretation – In many
tests, the advent of computer scoring and a
growing reliance on objective, computer-
scorable items have virtually eliminated
error variance caused by scorer differences.
However, not all tests can be scored from
grids blackened by no. 2 pencils.
Individually administered intelligence tests,
some tests of personality, tests of creativity,
various behavioral measures, essay tests,
portfolio assessment, situational behavior
tests, and countless other tools of assessment
still require scoring by trained personnel.
 o Scorers and scoring systems are
potential sources of error variance.
A test may employ objective-type
items amenable to computer
scoring of well-documented
reliability. Yet even then, a
technical glitch might contaminate
the data.
o If subjectivity is involved in
scoring, then the scorer (or rater)
can be a source of error variance.
Indeed, despite rigorous scoring
criteria set forth in many of the
better-known tests of intelligence,
examiner/scorers occasionally still
are confronted by situations where
an examinee’s response lies in a
gray area.
- Other sources of error – Surveys and polls
are two tools of assessment commonly used
by researchers who study public opinion. In
the political arena, for example, researchers
trying to predict who will win an election
may sample opinions from representative
voters and then draw conclusions based on
their data.
o The error in such research may be a
result of sampling error—the extent
to which the population of voters in
the study actually was
representative of voters in the
election.
o Alternatively, the researchers may
have gotten such factors right but
simply did not include enough
people in their sample to draw the
conclusions that they did. This
brings us to another type of error,
called methodological error.
o Certain types of assessment
situations lend themselves to
particular varieties of systematic
and nonsystematic error. For
example, consider assessing the
extent of agreement between
partners regarding the quality and
quantity of physical and
psychological abuse in their
relationship.
 A number of studies
(O’Leary & Arias, 1988;
Riggs et al., 1989; Straus,
1979) have suggested that
underreporting or over-
reporting of perpetration
of abuse also may
contribute to systematic
error. Females, for
example, may underreport
abuse because of fear,
shame, or social
desirability factors and
over-report abuse if they
are seeking help.
Reliability Estimates
Test-Retest Reliability Estimates
- One way of estimating the reliability of a
measuring instrument is by using the same
instrument to measure the same thing at two
points in time. In psychometric parlance,
this approach to reliability evaluation is
called the test-retest method, and the result
of such an evaluation is an estimate of test-
retest reliability.
- Test-retest reliability is an estimate of
reliability obtained by correlating pairs of
scores from the same people on two
different administrations of the same test.
The test-retest measure is appropriate when
evaluating the reliability of a test that
purports to measure something that is
relatively stable over time, such as a
personality trait. If the characteristic being
measured is assumed to fluctuate over time,
then there would be little sense in assessing
the reliability of the test using the test-retest
method.
- It is generally the case (although there are
exceptions) that, as the time interval
between administrations of the same test
increases, the correlation between the scores
obtained on each testing decreases. The
passage of time can be a source of error
variance. The longer the time that passes, the
greater the likelihood that the reliability
coefficient will be lower.
- When the interval between testing is greater
than six months, the estimate of test-retest
reliability is often referred to as the
coefficient of stability.
o An estimate of test-retest reliability
from a math test might be low if the
test takers took a math tutorial
before the second test was
administered. An estimate of test-
retest reliability from a personality
profile might be low if the test taker
suffered some emotional trauma or
received counseling during the
intervening period.
 - A low estimate of test-retest reliability might
be found even when the interval between
testings is relatively brief. This may well be
the case when the testings occur during a
time of great developmental change with
respect to the variables they are designed to
assess.
- An estimate of test-retest reliability may be
most appropriate in gauging the reliability of
tests that employ outcome measures such as
reaction time or perceptual judgments
(including discriminations of brightness,
loudness, or taste).
o However, even in measuring
variables such as these, and even
when the time period between the
two administrations of the test is
relatively small, various factors
(such as experience, practice,
memory, fatigue, and motivation)
may intervene and confound an
obtained measure of reliability.
*Psychology’s Replicability Crisis
- Low replication rate helped confirm that
science indeed had a problem with
replicability, the seriousness of which is
reflected in the term replicability crisis.
- (1) A general lack of published replication
attempts in the professional literature, (2)
editorial preferences for positive over
negative findings, and ((3) questionable
research practices on the part of authors of
published studies.
o Replication by independent parties
provides for confidence in a
finding, reducing the likelihood of
experimenter bias and statistical
anomaly. Indeed, had scientists
been as focused on replication as
they were on hunting down novel
results, the field would likely not be
in crisis now.
o Positive findings typically entail a
rejection of the null hypothesis. In
essence, from the perspective of
most journals, rejecting the null
hypothesis as a result of a research
study is a newsworthy event. By
contrast, accepting the null
hypothesis might just amount to
“old news.”
- Moreover, replication efforts—beyond even
that of the Open Science Collaboration—are
becoming more common (Klein et al, 2013).
Overall, it appears that most scientists now
recognize replicability as a concern that
needs to be addressed with meaningful
changes to what has constituted “business-
as-usual” for so many years.
Parallel-Forms and Alternate-Forms Reliability
Estimates
- The degree of the relationship between
various forms of a test can be evaluated by
means of an alternate-forms or parallel-
forms coefficient of reliability, which is
often termed the coefficient of equivalence.
- Parallel forms of a test exist when, for each
form of the test, the means and the variances
of observed test scores are equal. In theory,
the means of scores obtained on parallel
forms correlate equally with the true score.
More practically, scores obtained on parallel
tests correlate equally with other measures.
o The term parallel forms reliability
refers to an estimate of the extent to
which item sampling and other
errors have affected test scores on
versions of the same test when, for
each form of the test, the means
and variances of observed test
scores are equal.
- Alternate forms are simply different
versions of a test that have been constructed
so as to be parallel. Although they do not
meet the requirements for the legitimate
designation “parallel,” alternate forms of a
test are typically designed to be equivalent
with respect to variables such as content and
level of difficulty.
o The term alternate forms
reliability refers to an estimate of
the extent to which these different
forms of the same test have been
affected by item sampling error, or
other error.
- Obtaining estimates of alternate-forms
reliability and parallel-forms reliability is
similar in two ways to obtaining an estimate
of test-retest reliability:
(1) Two test administrations with
the same group are required, and
(2) test scores may be affected by
factors such as motivation, fatigue,
or intervening events such as
practice, learning, or therapy
(although not as much as when the
same test is administered twice).
 - An additional source of error variance, item
sampling, is inherent in the computation of
an alternate- or parallel-forms reliability
coefficient.
- Developing alternate forms of tests can be
time-consuming and expensive. Imagine
what might be involved in trying to create
sets of equivalent items and then getting the
same people to sit for repeated
administrations of an experimental test! On
the other hand, once an alternate or parallel
form of a test has been developed, it is
advantageous to the test user in several
ways.
- An estimate of the reliability of a test can be
obtained without developing an alternate
form of the test and without having to
administer the test twice to the same people.
Deriving this type of estimate entails an
evaluation of the internal consistency of the
test items. Logically enough, it is referred to
as an internal consistency estimate of
reliability or as an estimate of inter-item
consistency.
Split-Half Reliability Estimates
- An estimate of split-half reliability is
obtained by correlating two pairs of scores
obtained from equivalent halves of a single
test administered once.
- It is a useful measure of reliability when it is
impractical or undesirable to assess
reliability with two tests or to administer a
test twice (because of factors such as time or
expense).
o Step 1. Divide the test into
equivalent halves.
o Step 2. Calculate a Pearson r
between scores on the two halves
of the test.
o Step 3. Adjust the half-test
reliability using the Spearman–
Brown formula (discussed shortly).
- Simply dividing the test in the middle is not
recommended because it’s likely that this
procedure would spuriously raise or lower
the reliability coefficient.
- One acceptable way to split a test is to
randomly assign items to one or the other
half of the test.
- Another acceptable way to split a test is to
assign odd-numbered items to one half of
the test and even-numbered items to the
other half. This method yields an estimate of
split-half reliability that is also referred to as
odd-even reliability.
- Yet another way to split a test is to divide
the test by content so that each half contains
items equivalent with respect to content and
difficulty.
- In general, a primary objective in splitting a
test in half for the purpose of obtaining a
split-half reliability estimate is to create
what might be called “mini-parallel-forms,”
with each half equal to the other—or as
nearly equal as humanly possible—in
format, stylistic, statistical, and related
aspects.
- The third step requires the computation of
the Spearman-Brown formula, which
allows a test developer or user to estimate
internal consistency reliability from a
correlation of two halves of a test. It is a
specific application of a more general
formula to estimate the reliability of a test
that is lengthened or shortened by any
number of items.
- A formula is necessary for estimating the
reliability of a test that has been shortened or
lengthened. The general Spearman–Brown
(rSB) formula is:
o Where rSB is equal to the reliability
adjusted by the Spearman–Brown
formula, rxy is equal to the Pearson r
in the original-length test, and n is
equal to the number of items in the
revised version divided by the
number of items in the original
version.
- By determining the reliability of one half of
a test, a test developer can use the
Spearman–Brown formula to estimate the
reliability of a whole test. Because a whole
test is two times longer than half a test, n
becomes 2 in the Spearman–Brown formula
for the adjustment of split-half reliability.
The symbol rhh stands for the Pearson r of
scores in the two half tests:

- Usually, but not always, reliability increases
as test length increases. Ideally, the
additional test items are equivalent with
respect to the content and the range of
difficulty of the original items. Estimates of
reliability based on consideration of the
entire test therefore tend to be higher than
those based on half of a test.
- If test developers or users wish to shorten a
test, the Spearman–Brown formula may be
used to estimate the effect of the shortening
on the test’s reliability. Reduction in test
size for the purpose of reducing test
administration time is a common practice in
certain situations.
- A Spearman–Brown formula could also be
used to determine the number of items
needed to attain a desired level of reliability.
In adding items to increase test reliability to
a desired level, the rule is that the new items
must be equivalent in content and difficulty
so that the longer test still measures what the
original test measured.
Other Methods of Estimating Internal
Consistency
- Other formulas developed by Kuder and
Richardson (1937) and Cronbach (1951).
Inter-item consistency
- Refers to the degree of correlation among all
the items on a scale. A measure of inter-item
consistency is calculated from a single
administration of a single form of a test. An
index of inter-item consistency, in turn, is
useful in assessing the homogeneity of the
test.
o Tests are said to be homogeneous if
they contain items that measure a
single trait. As an adjective used to
describe test items, homogeneity
(derived from the Greek words
homos, meaning “same,” and
genos, meaning “kind”) is the
degree to which a test measures a
single factor. In other words,
homogeneity is the extent to which
items in a scale are unifactorial.
o In contrast to test homogeneity,
heterogeneity describes the degree
to which a test measures different
factors. A heterogeneous (or
nonhomogeneous) test is composed
of items that measure more than
one trait.
- The more homogeneous a test is, the more
inter-item consistency it can be expected to
have. Because a homogeneous test samples a
relatively narrow content area, it is to be
expected to contain more inter-item
consistency than a heterogeneous test. Test
homogeneity is desirable because it allows
relatively straightforward test-score
interpretation.
o It is often an insufficient tool for
measuring multifaceted
psychological variables such as
intelligence or personality. One
way to circumvent this potential
source of difficulty has been to
administer a series of homogeneous
tests, each designed to measure
some component of a
heterogeneous variable.
The Kuder-Richardson Formulas
- Dissatisfaction with existing split-half
methods of estimating reliability compelled
G. Frederic Kuder and M. W. Richardson to
develop their own measures for estimating
reliability.
- The most widely known of the many
formulas they collaborated on is their
Kuder–Richardson formula 20, or KR-20,
so named because it was the 20th formula
developed in a series.
o KR-20 is the statistic of choice for
determining the inter-item
consistency of dichotomous items,
primarily those items that can be
scored right or wrong (such as
multiple-choice items).
o Will get a lower reliability estimate
than split-half method if test items
are more heterogeneous.
o KR-20 is computed:
rKR20 = Kuder–Richardson formula 20 reliability
coefficient
k = number of test items
σ2 = variance of total test scores
p = proportion of test takers who pass the item
q = proportion of people who fail the item
Σpq = sum of the pq products over all items.

- The KR-21 formula may be used if there is
reason to assume that all the test items have
approximately the same degree of difficulty.
o Formula KR-21 has become
outdated in an era of calculators
and computers. Way back when,
KR-21 was sometimes used to
estimate KR-20 only because it
required many fewer calculations.
- The one variant of the KR-20 formula that
has received the most acceptance and is in
widest use today is a statistic called
coefficient alpha. You may even hear it
referred to as coefficient α−20.
Coefficient Alpha
- Developed by Cronbach (1951) and
subsequently elaborated on by others it may
be thought of as the mean of all possible
split-half correlations, corrected by the
Spearman–Brown formula.
- Appropriate for use on tests containing
nondichotomous items.
ra = coefficient alpha
k = number of items, is the variance of one
item
Σ = sum of variances of each item
σ2 = variance of the total test scores
- Preferred statistic for obtaining an estimate
of internal consistency reliability. A
variation of the formula has been developed
for use in obtaining an estimate of test-retest
reliability (Green, 2003).
- It is widely used as a measure of reliability,
in part because it requires only one
administration of the test.
- Unlike a Pearson r, which may range in
value from −1 to +1, coefficient alpha
typically ranges in value from 0 to 1. The
reason for this is that, conceptually,
coefficient alpha (much like other
coefficients of reliability) is calculated to
help answer questions about how similar
sets of data are.
- Still, because negative values of alpha are
theoretically impossible, it is recommended
under such rare circumstances that the alpha
coefficient be reported as zero (Henson,
2001).
- Also, a myth about alpha is that “bigger is
always better.” As Streiner (2003b) pointed
out, a value of alpha above .90 may be “too
high” and indicate redundancy in the items.
Average Proportional Distance (APD)
- Rather than focusing on similarity between
scores on items of a test (as do split-half
methods and Cronbach’s alpha), the APD is
a measure that focuses on the degree of
difference that exists between item scores.
- Accordingly, we define the average
proportional distance method as a measure
used to evaluate the internal consistency of a
test that focuses on the degree of difference
that exists between item scores.
- The general “rule of thumb” for interpreting
an APD is that an obtained value of .2 or
lower is indicative of excellent internal
consistency, and that a value of .25 to .2 is
in the acceptable range. A calculated APD
of .25 is suggestive of problems with the
internal consistency of the test.
- One potential advantage of the APD method
over using Cronbach’s alpha is that the APD
index is not connected to the number of
items on a measure. Cronbach’s alpha will
be higher when a measure has more than 25
items (Cortina, 1993).
Measures of Inter-Scorer Reliability
- Unfortunately, in some types of tests under
some conditions, the score may be more a
function of the scorer than of anything else.
- Variously referred to as scorer reliability,
judge reliability, observer reliability, and
inter-rater reliability, inter-scorer
reliability is the degree of agreement or
consistency between two or more scorers (or
judges or raters) with regard to a particular
measure.
- Inter-rater consistency may be promoted by
providing raters with the opportunity for
group discussion along with practice
exercises and information on rater accuracy
(Smith, 1986).
- Perhaps the simplest way of determining the
degree of consistency among scorers in the
scoring of a test is to calculate a coefficient
of correlation. This correlation coefficient is
referred to as a coefficient of inter-scorer
reliability.
The Importance of the Method Used for
Estimating Reliability
 - A published study by Chmielewski et al.
(2015) highlighted the substantial influence
that differences in method can have on
estimates of inter-rater reliability.
- Diagnostic reliability must be acceptably
high in order to accurately identify risk
factors for a disorder that are common to
subjects in a research study.
- The utility and validity of a particular
diagnosis itself can be called into question if
expert diagnosticians cannot, for whatever
reason, consistently agree on who should
and should not be so diagnosed.
- Results suggest that the reliability of
diagnoses is far lower than commonly
believed. Moreover, the results demonstrate
the substantial influence that method has on
estimates of diagnostic reliability even when
other factors are held constant.
Using and Interpreting a Coefficient of Reliability
- Reliability is a mandatory attribute in all
tests we use.
The Purpose of the Reliability Coefficient
- Transient error - a source of error
attributable to variations in the test taker’s
feelings, moods, or mental state over time.
Then again, this 5% of the error may be due
to other factors that are yet to be identified.
The Nature of the Test
- Closely related to considerations concerning
the purpose and use of a reliability
coefficient are those concerning the nature
of the test itself. Included here are
considerations such as whether
(1) The test items are homogeneous
or heterogeneous in nature;
(2) The characteristic, ability, or
trait being measured is presumed to
be dynamic or static
(3) The range of test scores is or is
not restricted; (4) the test is a speed
or a power test; and (5) the test is or
is not criterion-referenced.
- The child tested just before and again just
after a developmental advance may perform
very differently on the two testings. In such
cases, a marked change in test score might
be attributed to error when in reality it
reflects a genuine change in the test taker’s
skills.
- Homogeneity versus heterogeneity of the
test items. Tests designed to measure one
factor, such as one ability or one trait, are
expected to be homogeneous in items. For
such tests, it is reasonable to expect a high
degree of internal consistency.
- Dynamic versus static characteristics.
Static characteristic is unchanging, such as
intelligence. Unlike in measuring dynamic
characteristics, a test-retest measure would
be of little help in gauging the reliability of
the measuring
instrument.
- Restriction or inflation of range. Referred
to as restriction of range or restriction of
variance (or, conversely, inflation of range
or inflation of variance.
o If the variance of either variable in
a correlational analysis is restricted
by the sampling procedure used,
then the resulting correlation
coefficient tends to be lower.
o If the variance of either variable in
a correlational analysis is inflated
by the sampling procedure, then the
resulting correlation coefficient
tends to be higher.
- Speed tests versus power tests. A reliability
estimate of a speed test should be based on
performance from two independent testing
periods using one of the following: (1) test-
retest reliability, (2) alternate-forms
reliability, or (3) split-half reliability from
two separately timed half tests. If a split-half
procedure is used, then the obtained
reliability coefficient is for a half test and
should be adjusted using the Spearman–
Brown formula.
- Criterion-referenced tests. Designed to
provide an indication of where a test taker
stands with respect to some variable or
criterion, such as an educational or a
vocational objective.
o Some traditional procedures of
estimating reliability are usually
not appropriate for use with
criterion-referenced tests.
- In criterion-referenced testing, and
particularly in mastery testing, how different
the scores are from one another is seldom a
focus of interest. In fact, individual
differences between examinees on total test
scores may be minimal. The critical issue for
 the user of a mastery test is whether or not a
certain criterion score has been achieved.
The True Score Model of Measurement and
Alternatives to It
- Classical Test Theory (CTT) or true score
(or classical) model of measurement.
o One of the reasons it has remained
the most widely used model has to
do with its simplicity, especially
when one considers the complexity
of other proposed models of
measurement.
- True score as a value that according to
classical test theory genuinely reflects an
individual’s ability (or trait) level as
measured by a particular test.
- One’s true score on one test of extraversion,
for example, may not bear much
resemblance to one’s true score on another
test of extraversion.
o Comparing a test taker’s scores on
two different tests purporting to
measure the same thing requires a
sophisticated knowledge of the
properties of each of the two tests,
as well as some rather complicated
statistical procedures designed to
equate the scores.
- For starters, one problem with CTT has to
do with its assumption concerning the
equivalence of all items on a test;
o All items are presumed to be
contributing equally to the score
total.
- Another problem has to do with the length
of tests that are developed using a CTT
model. Whereas test developers favor
shorter rather than longer tests (as do most
test takers), the assumptions inherent in CTT
favor the development of longer rather than
shorter tests.
- For these reasons, as well as others,
alternative measurement models have been
developed.
Domain sampling theory
- The 1950s saw the development of a viable
alternative to CTT. It was originally referred
to as domain sampling theory and is better
known today in one of its many modified
forms as generalizability theory.
- Rebels against the concept of a true score
existing with respect to the measurement of
psychological constructs.
- Proponents of domain sampling theory seek
to estimate the extent to which specific
sources of variation under defined
conditions are contributing to the test score.
(CTT = portion of test score attributable to
error)
- Of the three types of estimates of reliability,
measures of internal consistency are perhaps
the most compatible with domain sampling
theory.
Generalizability theory
- In one modification of domain sampling
theory called generalizability theory, a
“universe score” replaces that of a “true
score” (Shavelson et al., 1989).
- Based on the idea that a person’s test scores
vary from testing to testing because of
variables in the testing situation. Instead of
conceiving of all variability in a person’s
scores as error, Cronbach encouraged test
developers and researchers to describe the
details of the particular test situation or
universe leading to a specific test score.
o This universe is described in terms
of its facets, which include things
like the number of items in the test,
the amount of training the test
scorers have had, and the purpose
of the test administration.
- According to generalizability theory, given
the exact same conditions of all the facets in
the universe, the exact same test score
should be obtained. This test score is the
universe score, and it is, as Cronbach noted,
analogous to a true score in the true score
model.
- Cronbach (1970): The person will ordinarily
have a different universe score for each
universe. Mary’s universe score covering
tests on May 5 will not agree perfectly with
her universe score for the whole month of
May. . . . Some testers call the average over
a large number of comparable observations
a “true score”; e.g., “Mary’s true typing
rate on 3-minute tests.” Instead, we speak of
a “universe score” to emphasize that what
score is desired depends on the universe
being considered. For any measure there
are many “true scores,” each
corresponding to a different universe.
- When we use a single observation as if it
represented the universe, we are
generalizing. We generalize over scorers,
over selections typed, perhaps over days. If
 the observed scores from a procedure agree
closely with the universe score, we can say
that the observation is “accurate,” or
“reliable,” or “generalizable.”
- There is a different degree of
generalizability for each universe.

- A generalizability study examines how

generalizable scores from a particular test
are if the test is administered in different
situations.
o Examines how much of an impact
different facets of the universe have
on the test score.
o Is the test score affected by the time
of day in which the test is
administered? The influence of
particular facets on the test score is
represented by coefficients of
generalizability.

- After this, it is recommended that the test

developers do a decision study.
o In the decision study, developers
examine the usefulness of test
scores in helping the test user make
decisions.
o Designed to tell the test user how
test scores should be used and how
dependable those scores are as a
basis for decisions, depending on
the context of their use.
o Taking a better measure improves
the sensitivity of an experiment in
the same way that increasing the
number of subjects does.
- From the perspective of generalizability
theory, a test’s reliability is very much a
function of the circumstances under which
the test is developed, administered, and
interpreted.

Item Response Theory

- Provide a way to model the probability that
a person with X ability will be able to
perform at a level of Y.
- Latent-trait theory – physically
unobservable
- Examples of two characteristics of items
within an IRT framework are the difficulty
level of an item and the item’s level of
discrimination