Properties of a Standardized Test
Reliability
® Consistency of test scores obtained by the
same persons under which they are re-
examined (Anastasi & Urbina, 1996).
® Extent to which a score is free from
measurement error (Kaplan & Saccuzzo,
2011)
o True score variance : Observed score
variance
® Consistency in measurement (Cohen &
Swerdlik, 2009).
o Measurement error = extent to which
measurements differ from occasion to
occasion.
Types of Reliability
1) Test Re-test Reliability
® Repeat identical test on second occasion.
® Reliability coefficient
o Correlation between scores obtained by
same persons upon two administrations
of the same test.
® Valuable most only if we are measuring
characteristics that do not change over time
(e.g., IQ).
o If an IQ test produces different scores,
conclude that it is due to random
measurement error not because the
person got smarter or less.
® Shows the extent to which scores can be
generalized over different occasions
(unchanging).
® Interval should be specified in test manual.
® Retest correlations decrease as interval
lengthens.
Two Possible Negative Effects of Test Re-test
• Carryover Effect
® First testing session influences
scores on second session.
® Test takers may remember their
answers from the first time they took
the test.
® Systematic carryover does not
harm the reliability.
o Everyone’s score improves
exactly 5 points (changes
aren’t random).
® Random carryover effects occur
when changes are not predictable
from earlier scores or when
something affects some test takers.
• Practice Effect
® Test takers tend to score better on a
test that’s been given the second
time.
® They have sharpened their skills
since the first time.
® Thus, time interval must be
evaluated carefully.
® The shorter time interval, the
greater risk for both carryover and
practice effects.
2) Alternate Forms Reliability
® Equivalent forms or Parallel forms reliability
® An alternative to test re-test reliability.
® It makes use of alternate forms of the test.
® Same persons can be tested with one form on
first occasion, and an equivalent form of
test on the second.
® Reliability coefficient
o Correlation between scores obtained by
same persons on the two forms of test.
® Ensure that test is truly parallel.
o Both tests must have the same number of
items, type and content, and equal range
or level of difficulty.
Limitations
® Can only reduce but not eliminate practice
effects.
® Random error and the difference between
the forms can elicit variation when both
forms of the test are given on the same day.
® Can be burdensome (having to develop two
forms of the same test).
3) Split Half Reliability
® Test is divided into half. Each half scored
separately.
® The results of ½ of test are compared to the
results of the other half.
® How is the test divided into two halves?
o Divide randomly. Calculate score for
first half and another score for second
half. Convenient but can cause
problems if difficulty is divided
between halves.
o Use odd-even system
® Correlation is an underestimate. Less reliable
because it has fewer items.
 o Apply Spearman-Brown formula. The
formula estimates the correlation
between the two halves if it had the
length of a full test.
o Use Spearman-Brown only when two
halves have equal variances.
o Otherwise (unequal variances),
Cronbach’s coefficient alpha is used,
which provides the lowest estimate of
reliability.
4) KR20 Formula
® Kuder-Richardson 20
® Calculates reliability of test where items are
dichotomous (right or wrong answers, 0 and
1)
® Formula: KR20 = N/N-1 {(s2 – Σpq)/s2}
o KR20 = reliability estimate
o N = number of items on the test
o s2 = variance of the total test score
o p = proportion of people getting each
item correct
o q = proportion of people getting each
item incorrect
§ For each item, q = 1 – p
5) Coefficient Alpha or Cronbach’s coefficient
® Developed by Cronbach to estimate internal
consistency of tests that are not
dichotomous.
® Applicable for personality and attitude
scales.
® SPSS software provides convenient way to
find coefficient alpha.
® Used in essays, etc.
How reliable is reliable?
® .70 to .80 reliability are good enough scores
in basic research.
® In clinical settings, high reliability is vital
(.90 to .95).
What to do about low reliability?
® Increase number of items
® Factor and item analysis
® Correction for attenuation – a formula used
to determine exact correlation of two
variables deemed to be affected by error.
Factors Affecting Test Reliability (For DO A SEA)
1. Test Format (structure, etc.)
2. Test Difficulty
3. Test Objectivity
4. Test Administration (who, when, where, and
how the test was administered)
5. Test Scoring
6. Test Economy (cost and quality of test)
7. Test Adequacy
Validity
® Extent to which it measures what it claims
to measure. Attributes a meaning to test
scores (Gregory, 2011).
® Measures what the test measures and how
well it does so (Anastasi, 1996).
® The agreement between a test score and the
quality it is believed to measure. “Does the
test measure what it is supposed to measure?”
(Kaplan & Saccuzzo, 2011).
® It is characterized in a continuum ranging
from weak to acceptable to strong.
Types of Validity
1) Face Validity
® Least stringent. Does the test appear to be
valid?
o If a test measures what it intends to
measure on a surface level.
® Doesn’t involve statistics.
® Done by face validators (registered
psychometricians, psychologists, and
guidance counselors).
2) Content Validity
® Built through choice of appropriate content
(questions, tasks, and items).
® Extent to which test represents all aspects
of a given construct or variable.
® Inspection of items. Panel of experts will
rate test items based on how they match the
domain specification.
® Considers the adequacy of representation.
3) Criterion Validity
® Criterion
o Standard followed in which a test score
is evaluated.
o Can be a test score, psychiatric
diagnosis, training cost, index of
absenteeism, amount of time.
o Characteristics of a criterion:
§ Relevant
 § Valid and reliable 3. Reading comprehension level
§ Uncontaminated (should not be a 4. Item difficulty
criterion of the supposed criterion) 5. Test construction factors
® Test effectiveness in estimating behavior in 6. Length of test
particular situation. 7. Arrangement of items
® Tells how well test corresponds to a 8. Patterns of answers
particular criterion.
Types of Criterion-Related Validity
• Concurrent Validity
® Extent to which test scores may be used
to estimate individual’s present
standing on a criterion.
• Predictive Validity
® Extent to which test scores can predict
future behavior or scores on another
test taken in future.
• Incremental Validity
® Related to predictive validity.
® Extent to which an additional
predictor explains something about
criterion measure that is not explained
by predictors in use.

4) Construct Validity
® Construct
o An informed scientific idea
developed to describe a behavior.
o Built by mental synthesis.
o Unobservable traits; thought to have
correlation with other variables.
® Extent to which test measures a theoretical
construct.
® Estimates the existence of an inferred,
underlying characteristic based on a limited
sample of behavior.
® Established through activities in which
researcher defines some construct and
develops instrumentation to measure it.
® Required when no criterion is accepted as
adequate to define quality being measured
(being woke).
® Statistical analysis.
® Good construct validity if there is an
existing psychological theory to support test
items.
® Involves logical analysis and empirical
data.

Factors Influencing Test Validity (ADRITLAP)

1. Appropriateness of the test
2. Directions/Instructions