CHAPTER 1: PSYCHOLOGICAL TESTING AND ASSESSMENT CHAPTER 1: PSYCHOLOGICAL TESTING AND ASSESSMENT

(2) some test administers don’t even have to

TESTING AND ASSESSMENT
be present
Roots can be found in early twentieth century in France 1905
(a) usually administered to larger
Alfred Binet published a test designed to help place Paris school children groups
WW1, military used the test to screen large numbers of recruits quickly (b) test takers complete tasks
for intellectual and emotional problems independently
WW2, military depend more on tests to screen recruits for service b) Scoring and interpretation procedures
(1) score: a code or summary statement,
PSYCHOLOGICAL usually (but not necessarily) numerical in
PSYCHOLOGICALTESTING
ASSESSMENT nature, that reflects an evaluation of
Process of measuring performance on a test, task, interview, or
Gathering & integration of
psychology-related some other sample of behavior
psychology-related data for the
DEFINITION variables by means of
purpose of making a (2) scoring: process of assigning such
devices/procedures
psychological evaluation with evaluative codes/ statements to
designed to obtain a
accompany of tools.
sample of behavior performance on tests, tasks, interviews, or
To answer a referral question, other behavior samples.
To obtain some gauge,
OBJECTIVE solve problem or arrive at a (3) different types of score:
usually numerical in
decision thru the use of tools of
nature (a) cut score: reference point,
evaluation
Testing may be usually numerical, derived by
PROCESS Typically individualized judgement and used to divide
individualized or group
Key in the process of selecting Tester is not key into the a set of data into two or more
ROLE OF
tests as well as in drawing process; may be classifications.
EVALUATOR
conclusions substituted (i) sometimes reached
SKILL OF Typically requires an educated Requires technician-like without any formal
EVALUATIOR selection, skill in evaluation skills
method: in order to
Entail logical problem-solving
Typically yields a test “eyeball”, teachers
OUTCOME approach to answer the referral
score who decide what is
ques.
passing and what is
3 FORMS OF ASSESSMENT: failing.
1. COLLABORATIVE PSYCHOLOGICAL ASSESSMENT – assessor and (4) who scores it
assesse work as partners from initial contact through final feedback (a) self-scored by testtaker
2. THERAPEUTIC PSYCHOLOGICAL ASSESSMENT – self-discovery and new
(b) computer
understandings are encouraged throughout the assessment process
3. DYNAMIC PSYCHOLOGICAL ASSESSMENT – follows a model (a) (c) trained examiner
evaluation (b) intervention (a) evaluation. Provide a means for evaluating c) psychometric soundness/ technical quality
how the assesse processes or benefits from some type of intervention during (1) Psychometrics: the science of
the course of evaluation.
psychological measurement.
Tools of Psychological Assessment (a) referring to how consistently and

A. The Test (a measuring device or procedure) how accurately a psychological

test measures what it purports
1. psychological test: a device or procedure designed to measure
to measure.
variables related to psychology (intelligence, personality, aptitude,
interests, attitudes, or values) (2) Utility: refers to the usefulness or practical
value that a test or other tool of assessment
2. format: refers to the form, plan, structure, arrangement, and layout of
has for a particular purpose.
test items as well as to related considerations such as time limits.
B. The Interview: method of gathering information through direct
a) also referred to as the form in which a test is
communication involving reciprocal exchange
administered (pen and paper, computer, etc)
Computers can generate scenarios.
1. interviewer in face-to-face is taking note of

b) term is also used to denote the form or structure of other a) verbal language
evaluative tools, and processes, such as the guidelines b) nonverbal language
for creating a portfolio work sample (1) body language movements
3. Ways That tests differ from one another: (2) facial expressions in response to
a) administrative procedures interviewer
(1) some test administers have an active (3) the extent of eye contact
knowledge (4) apparent willingness to cooperate
(a) some test administration c) how they are dressed
involves demonstration of (1) neat vs sloppy vs inappropriate
tasks 2. interviewer over the phone taking note of
(b) usually one-on-one a) changes in the interviewee’s voice pitch
(c) trained observation of b) long pauses
assessee’s performance c) signs of emotion in response
3. ways that interviews differ:
a) length, purpose, and nature
CHAPTER 1: PSYCHOLOGICAL TESTING AND ASSESSMENT CHAPTER 1: PSYCHOLOGICAL TESTING AND ASSESSMENT
b) in order to help make diagnostic, treatment, 6. interpretive report: a formal or official computer-generated account
selection, etc of test performance presented in both numeric and narrative form
4. panel interview and including an explanation of the findings;
a) an interview conducted with one interviewee with more a) the three varieties of interpretive report are
than one interviewer (1) descriptive
C. The Portfolio (2) screening
1. files of work products: paper, canvas, film, video, audio, etc (3) consultive
2. samples of ones abilities and accomplishments b) some contain relatively little interpretation and simply call
D. Case History Data: records, transcripts, and other accounts in written, pictorial attention to certain high, low, or unusual scores that
or other form that preserve archival information, official and informal accounts, needed to be focused on.
and other data and items relevant to assessee c) consultative report: A type of interpretive report
1. Sheds light on an individual's past and current adjustment as well as on designed to provide expert and detailed analysis of test
events and circumstances that may have contributed to any changes in data that mimics the work of an expert consultant.
adjustment. d) integrative report: a form of interpretive report of
2. Provides information about neuropsychological functioning prior to the psychological assessment, usually computer-
occurrence of a trauma or other event that results in a deficit. generated, in which data from behavioral, medical,
3. insight into current academic and behavioral standing administrative, and/or other sources are integrated
4. useful in making judgments for future class placements 7. CAPA: computer assisted psychological assessment. (assistance to the

5. Case history Study: a report or illustrative account concerning test user not the test taker)

person or an event that was compiled on the basis of case history a) enables test developers to create psychometrically sound
data tests using complex mathematical procedures and
calculations.
a) Might shed light on how one individual’s personality and
particular set of environmental conditions combined to b) enables test users the construction of tailor-made test
produce a successful world leader. with built-in scoring and interpretive capabilities.

b) Groupthink: work on a social psychological c) Pros:

phenomenon: contains rich case history material on (1) test administrators have greater access to
collective decision making that did not always result in the potential test users because of the global
best decisions. reach of the internet.
E. Behavioral Observation: monitoring the actions of others or oneself by visual or (2) scoring and interpretation of test data tend
electronic means while recording quantitative and/or qualitative information to be quicker than for paper-and- pencil
regarding those actions. tests
1. often used as a diagnostic aid in various settings: inpatient (3) costs associated with internet testing tend to be
facilities, behavioral research laboratories, classrooms. lower than costs associated with paper-and-
2. naturalistic observation: behavioral observation that takes place in pencil tests
a naturally occurring setting (as opposed to a research laboratory) for (4) the internet facilitates the testing of otherwise
the purpose of evaluation and information- gathering. isolated populations, as well as people with
3. in practice tends to be used most frequently by researchers in settings disabilities for whom getting to a test center
such as classrooms, clinics, prisons, etc. might prove as a hardship.
F. Role- PlayTests (5) greener: conserves paper, shipping
1. role play: acting an improvised or partially improvised part in a materials etc.
simulated situation. d) Cons:

2. role-play test: tool of assessment wherein assessees are directed to (1) test client integrity
act as if they were in a particular situation. Assessees are then evaluated (a) refers to the verification of the
with regard to their expressed thoughts, behaviors, abilities, etc identity of the test taker when a
G. Computers as tools test is administered online
1. local processing: on site computerized scoring, interpretation, or (b) also refers to the sometimes
other conversion of raw test data; contrast w/ CP and varying interests of the test
teleprocessing taker vs that of the test
2. central processing: computerized scoring, interpretation, or other administrator. The test taker
conversion of raw data that is physically transported from the same or might have access to notes,
other test sites; contrast w/ LP and teleprocessing. aids, internet resources etc.

3. teleprocessing: computerized scoring, interpretation, or other (c) internet testing is only testing, not
conversion of raw test data sent over telephone lines by modem from a assessment
test site to a central location for computer processing. contrast with CP 8. CAT: computerized adaptive testing: an interactive, computer-
and LP administered test taking process wherein items presented to the test
4. simple score report: a type of scoring report that provides only a listing taker are based in part on the test taker's performance on previous
of scores items

5. extended scoring report: a type of scoring report that provides a listing a) EX: on a computerized test of academic abilities, the
of scores AND statistical data. computer might be programmed to switch from testing
math skills to English skills after three consecutive failures
on math items.
CHAPTER 1: PSYCHOLOGICAL TESTING AND ASSESSMENT
H. Other Tools
1. DVD- how would you respond to the events that take place in the
video
a) sexual harassment in the workplace
b) respond to various types of emergencies
c) diagnosis/treatment plan for clients on videotape
2. thermometers, biofeedback, etc
TEST DEVELOPER
They are the one who create tests.
They conceive, prepare, and develop tests. They also find a way to
disseminate their tests, by publishing them either commercially or through
professional publications such as books or periodicals.
TEST USER
They select or decide to take a specific test off the shelf and use it for some
purpose. They may also participate in other roles, e.g., as examiners or
scorers.
TEST TAKER
Anyone who is the subject of an assessment
Test taker may vary on a continuum with respect to numerous
variables including:
o The amount of anxiety they experience & the degree to which
the test anxiety might affect the results
o The extent to which they understand & agree with the
rationale of the assessment
o Their capacity & willingness to cooperate
o Amount of physical pain/emotional distress they are
experiencing
o Amount of physical discomfort
o Extent to which they are alert & wide awake
o Extent to which they are predisposed to agreeing or
disagreeing when presented with stimulus
o The extent to which they have received prior coaching
o May attribute to portraying themselves in a good light
Psychological autopsy – reconstruction of a deceased individual’s
psychological profile on the basis of archival records, artifacts, &
interviews previously conducted with the deceased assesee
TYPES OF SETTINGS
EDUCATIONAL SETTING
o Achievement test: evaluation of accomplishments or the
degree of learning that has taken place, usually with regard to
an academic area.
o Diagnosis: a description or conclusion reached on the basis of
evidence and opinion though a process of distinguishing the nature
of something and ruling out alternative conclusions.
o Diagnostic test: a tool used to make a diagnosis, usually to
identify areas of deficit to be targeted for intervention
o informal evaluation: A typically nonsystematic, relatively brief,
and “off the record” assessment leading to the formation of an
opinion or attitude, conducted by any person in any way for any
reason, in an unofficial context and not subject to the same
ethics or standards as evaluation by a professional
CLINICAL SETTING
o these tools are used to help screen for or diagnose
behavior problems
o group testing is used primarily for screening: identifying those
individuals who require further diagnostic evaluation.
COUNSELING SETTING
o schools, prisons, and governmental or privately owned
institutions
o Ultimate objective: the improvement of the assessee in terms
of adjustment, productivity, or some related variable.
GERIATRIC SETTING
o quality of life: in psychological assesment, an evaluation of
variables such as perceived stress, loneliness, sources of
CHAPTER 1: PSYCHOLOGICAL TESTING AND ASSESSMENT
satisfaction, personal values, quality of living conditions, and
quality of friendships and other social support.
BUSINESS AND MILITARY SETTINGS
GOVERNMENTAL AND ORGANIZATIONAL CREDENTIALING
How are Assessments Conducted?
protocol: the form or sheet or booklet on which a testtaker’s
responses are entered.
o term might also be used to refer to a description of a set of test- or
assessment- related procedures, as in the sentence , “the
examiner dutifully followed the complete protocol for the stress
interview”
rapport: working relationship between the examiner and the
examinee
ASSESSEMENT OF PEOPLE WITH DISABILITITES
Define who requires alternate assessment, how such assessment are to be
conducted and how meaningful inferences are to be drawn from the data
derived from such assessment
Accommodation – adaptation of a test, procedure or situation or the
substitution of one test for another to make the assessment more suitable
for an assesee with exceptional needs.
Translate it into Braillee and administered in that form.
Alternate Assessment – evaluative or diagnostic procedure or process that
varies from the usual, customary, or standardized way a measurement is
derived either by virtue of some special accommodation made to the assesee
by means of alternative methods
Consider these four variables on which of many different types of
accommodation should be employed:
o The capabilities of the assesse
o The purpose of the assessment
o The meaning attached to test scores
o The capabilities of the assessor
REFERENCE SOURCES
TEST CATALOUGES – contains brief description of the test
TEST MANUALS – detailed information
REFERENCE VOLUMES – one stop shopping, provides detailed
information for each test listed, including test publisher, author,
purpose, intended test population and test administration time
JOURNAL ARTICLES – contain reviews of the test
ONLINE DATABASES – most widely used bibliographic databases
TYPES OF TESTS
INDIVIDUAL TEST – those given to only one person at a time
GROUP TEST – administered to more than one person at a time by single
examiner
ABILITY TESTS:
o ACHIEVEMENT TESTS – refers to previous learning (ex.
Spelling)
o APTITUDE/PROGNOSTIC – refers to the potential for
learning or acquiring a specific skill
o INTELLIGENCE TESTS – refers to a person’s general
potential to solve problems
PERSONALITY TESTS: refers to overt and covert dispositions
o OBJECTIVE/STRUCTURED TESTS – usually self-report,
require the subject to choose between two or more
alternative responses
o PROJECTIVE/UNSTRUCTURED TESTS – refers to all possible
uses, applications and underlying concepts of psychological
and educational tests
o INTEREST TESTS –
CHAPTER 2: HISTORICAL, CULTURAL AND LEGAL/ETHICAL CONSIDERATIONS CHAPTER 2: HISTORICAL, CULTURAL AND LEGAL/ETHICAL CONSIDERATIONS
A HISTORICAL PERSPECTIVE testakers from young children through senior
19TH CENTURY adulthood.
Tests and testing programs first came into being in China B. THE MEASUREMENT OF PERSONALITY
Testing was instituted as a means of selecting who, of many o Field of psychology was being too test oriented
applicants would obtain government jobs (Civil service) o Clinical psychology was synonymous to mental testing
o ROBERT WOODWORTH – develop a measure of adjustment
The job applicants are tested on proficiency in endeavors such as music,
and emotional stability that could be administered quickly and
archery, knowledge and skill etc.
efficiently to groups of recruits
GRECO-ROMAN WRITINGS (Middle Ages)
To disguise the true purpose of the test,
World of evilness questionnaire was labeled as Personal Data
Deficiency in some bodily fluid as a factor believed to influence Sheet
personality He called it Woodworth Psychoneurotic
Hippocrates and Galen Inventory – first widely used self-report test of
RENAISSANCE personality
Christian von Wolff – anticipated psychology as a science and o Self-report test:
psychological measurement as a specialty within that science Advantages:
CHARLES DARWIN AND INDIVIDUAL DIFFERENCES Respondents best qualified
Disadvantages:
Tests designed to measure these individual differences in ability and
Poor insight into self
personality among people
One might honestly believe
“Origin of Species” chance variation in species would be selected or
something about self that isn’t true
rejected by nature according to adaptivity and survival value. “survival of the
Unwillingness to report seemingly
fittest”
negative qualities
FRANCIS GALTON o Projective test: individual is assumed to project onto some
Explore and quantify individual differences between people. ambiguous stimulus (inkblot, photo, etc.) his or her own unique
Classify people “according to their natural gifts” needs, fears, hopes, and motivations
Displayed the first anthropometric laboratory Ex.) Rorschack inkblot
KARL PEARSON o
Developed the product moment correlation technique. C. ACADEMIC AND APPLIED TRADITIONS Culture
His work can be traced directly from Galton
WILHEM MAX WUNDT and Assessment
First experimental psychology laboratory in University of Leipzig
Focuses more on relating to how people were similar, not different from Culture: ‘the socially transmitted behavior patterns, beliefs, and products of work f a
particular population, community, or group of people’
each other.
JAMES MCKEEN CATELL
Evolving Interest in Culture-Related Issues
Individual differences in reaction time  Goddard tested immigrants and found most to be feebleminded
Coined the term mental test o -invalid; overestimated mental deficiency, even in native English-
CHARLES SPEARMAN speakers
Originating the concept of test reliability as well as building the  Lead to nature-nurture debate about what intelligence tests actually measure
mathematical framework for the statistical technique of factor  Needed to “isolate” the cultural variable
analysis  Culture-specific tests: tests designed for use with ppl from one culture, but not
VICTOR HENRI from another
Frenchman who collaborated with Binet on papers suggesting how mental o -minorities still scored abnormally low
tests could be used to measure higher mental processes ex.) loaf of bread vs. tortillas
 today tests undergo many steps to ensure its suitable for said nation
EMIL KRAEPELIN
o -take testtakers reactions into account
Early experimenter of word association technique as a formal test
LIGHTNER WITMER Some Issues Regarding Culture and Assessment
“Little known founder of clinical psychology” Verbal Communication
Founded the first psychological clinic in the U.S. o Examiner and examinee must speak the same language
PSYCHE CATELL o Especially tricky with infrequently used vocabulary or unusual
Daughter of James Cattell idioms employed
o Translator may lose nuances of translation or give unintentional
Cattel Infant Intelligence Scale (CIIS) & Measurement of Intelligence in Infants
hints toward more desirable answer
and Young Children
o Also requires understanding of culture
RAYMOND CATTELL
Nonverbal Communication and Behavior
Believed in lexical approach to defining personality which examines human
o Different between cultures
languages for descriptors of personality dimensions o Ex.) meaning of not making eye contact
20t h CENTURY o Body movement could even have physical cause
- Birth of the first formal tests of intelligence o Psychoanalysis: Freud’s theory of personality and psychological
- Testing shifted to be of more understandable relevance/meaning treatment which stated that symbolic significance is assigned to
A. THE MEASUREMENT OF INTELLIGENCE many nonverbal acts.
o Binet created first intelligence to test to identify mentally o Timing tests in cultures not obsessed with speed
retarded school children in Paris (individual) o Lack of speaking could be reverence for elders
o Binet-Simon Test has been revised over again Standards of Evaluation
o Group intelligence tests emerged with need to screen o Acceptable roles for women differ throughout culture
intellect of WWI recruits o “judgments as to who might be the best employee, manager, or
o David Wechsler – designed a test to measure adult leader may differ as a function of culture, as might judgments
intelligence test regarding intelligence, wisdom, courage, and other psychological
for him Intelligence is a global capacity of the variables”
individual to act purposefully, to think rationally and
to deal effectively with his environment.
Wechsler-Bellevue Intelligence Scale
Wechsler Adult Intelligence Test – was revised
several times and extended the age range of
CHAPTER 2: HISTORICAL, CULTURAL AND LEGAL/ETHICAL CONSIDERATIONS
o must ask ‘how appropriate are the norms or other
standards that will be used to make this evaluation’
Tests and Group Membership
ex.) must be 5’4” to be police officer - excludes cultures with short stature
ex.) Jewish lifestyle not well suited for corporate America
affirmative action: voluntary and mandatory efforts to combat
discrimination and promote equal opportunity in education and employment
for all
Psychology, tests, and public policy
Legal and Ethical Consideration
Code of professional ethics: defines the standard of care expected of members of a given
profession.
The Concerns of the Public
Beginning in world war I, fear that tests were only testing the ability to take
tests
Legislation
o Minimum competency testing programs: formal testing programs
designed to be used in decisions regarding various aspects of
students’ educations
o Truth-in-testing legislation: state laws to provide testtakers with a
means of learning the criteria by which they are being
judged
Litigation
o Daubert ruling made federal judges the gatekeepers to
determining what expert testimony is admitted
o This overrode the Frye policy which only admitted scientific
testimony that had won general acceptance in the scientific
community.
CHAPTER 2: HISTORICAL, CULTURAL AND LEGAL/ETHICAL CONSIDERATIONS
The right to be informed of test findings
o Formerly test administrators told to give participants only
positive information
o No realistic information is required
o Tell test takers as little as possible about the nature of their
performance on a particular test. So that the examinee would
leave the test session feeling pleased and statisfied.
o Test takers have the right also to know what recommendations are
being made as a consequence of the test data
The right to privacy and confidentiality
o Private right: “recognizes the freedom of the individual to pick
and choose for himself the time, circumstances, and particularly
the extent to which he wishes to share or withhold from others
his attitudes, beliefs, behaviors, and opinions”
o Privileged information: information protected by law from being
disclosed in legal proceeding. Protects clients from disclosure in
judicial proceedings. Privilege belongs to the client not the
psychologist.
o Confidentiality: concerns matters of communication outside
the courtroom
Safekeeping of test data: It is not a good policy to
maintain all records in perpetuity
The right to the least stigmatizing label
o The standards advise that the least stigmatizing labels should
always be assigned when reporting test results.

The Concerns of the Profession

Test-user qualifications
o Who should be allowed to use psych tests
o Level A: tests or aids that can adequately be administered,
scored, and interpreted with the aid of the manual and a general
orientation to the kind of institution or organization in which one is
working
o Level B: tests or aids that require some technical knowledge of test
construction and use and of supporting psychological and
educational fields
o Level C: tests and aids requiring substantial understanding of
testing and supporting psych fields with experience
Testing people with disabilities
o Difficulty in transforming the test into a form that can be taken
by testtaker
o Transferring responses to be scorable
o Meaningfully interpreting the test data
Computerized test administration, scoring, and interpretation
o simple, convenient
o easily copied, duplicated
o insufficient research to compare it to pencil-and-paper versions
o value of computer interpretation is questionable
o unprofessional, unregulated “psychological testing” online

The Rights of Testtakers

the right of informed consent
o right to know why they are being evaluated, how test data will be
used and what information will be released to whom
o may be obtained by parent or legal representative
o must be in written form:
general purpose of the testing
the specific reason it is being undertaken
general type of instruments to be administered
o revealing this information before the test can contaminate
the results
o deception only used if absolutely necessary
o don’t use deception if it will cause emotional distress
o fully debrief participants
CHAPTER 3: A STATISTICS REFRESHER
Why We Need Statistics
- Statistics are important for purposes of education
o Numbers provide convenient summaries and allow us to
evaluate some observations relative to others
- We use statistics to make inferences, which are logical deductions
about events that cannot be observed directly
o Detective work of gathering and displaying clues –
exploratory data analysis
o Then confirmatory data analysis
- Descriptive statistics are methods used to provide a concise
description of a collection of quantitative information
- Inferential statistics are methods used to make inferences from observations
of a small group of people known as a sample to a larger group of individuals
known as a population
SCALES OF MEASUREMENT
MEASUREMENT – act of assigning numbers or symbols to
characteristics of things according to rules. The rules serves as a
guideline for representing the magnitude. It always involves error.
SCALE – set of numbers whose properties model empirical properties of the
objects to which the numbers are assigned.
CONTINUOUS SCALE – interval/ratio. A scale used to measure
continuous variable. Always involves error
DISCRETE SCALE – nominal/ordinal used to measure a discrete
variable (ex. Female or male)
ERROR – collective influence of all of the factors on a test score.
PROPERTIES OF SCALES
- Magnitude, equal intervals, and an absolute 0
Magnitude
- The property of “moreness”
- A scale has the property of magnitude if we can say that a particular instance
of the attribute represents more, less, or equal amounts of the given
quantity than does another instance
Equal Intervals
- A scale has the property of equal intervals if the difference between two
points at any place on the scale has the same meaning as the difference
between two other points that differ by the same number of scale units
- A psychological test rarely has the property of equal intervals
- When a scale has the property of equal intervals, the relationship between
the measured units and some outcome can be described by a straight line or a
linear equation in the form Y=a+bX
o Shows that an increase in equal units on a given scale reflects
equal increases in the meaningful correlates of units
Absolute 0
- An Absolute 0 is obtained when nothing of the property being
measured exists
- This is extremely difficult/impossible for many psychological qualities
NOMINAL SCALE
Simplest form of measurement
Classification or categorization
Arithmetic operations can be performed with nominal data
Ex.) Male or female
Also includes test items
o Ex.) yes/no responses
ORDINAL SCALE
Classifies in some kind of ranking order
Individuals compared to others and assigned a rank
Imply nothing about how much greater one ranking is than another
Numbers/ranks do not indicate units of measure
No absolute zero point
Binet: believed that data derived from intelligence test are ordinal in nature
INTERVAL SCALE
In addition to the features of nominal and ordinal scales, contain equal
intervals between numbers
CHAPTER 3: A STATISTICS REFRESHER
No absolute zero point
Can take average
RATIO SCALE
In addition to all the properties of nominal, ordinal, and interval
measurement, ratio scale has true zero point
Equal intervals between numbers
Ex.) measuring amount of pressure hand can exert
True zero doesn’t mean someone will receive a scor e of 0, but means that 0
has meaning
NOTE:
Permissible Operations
- Level of measurement is important because it defines which
mathematical operations we can apply to numerical data
- For nominal data, each observation can be placed in only one
mutually exclusive category
- Ordinal measurements can be manipulated using arithmetic
- With interval data, one can apply any arithmetic operation to the
differences between scores
o Cannot be used to make statements about ratios
DESCRIBING DATA
Distribution: set of scores arrayed for recording or study
Raw Score: straightforward, unmodified accounting of performance, usually
numerical
Frequency Distributions
Frequency Distribution: All scores listed alongside the number of times
each score occurred
Grouped Frequency Distribution: test-score intervals (class intervals),
replace the actual test scores
o Highest and lowest class intervals= upper and lower limits of
distribution
Histogram: graph with vertical lines drawn at the true limits of each test
score (or class interval) forming TOUCHING rectangles- midpoint in center of
bar
Bar Graph: rectangles DON’T touch
Frequency Polygon: data illustrated with continuous line connecting the
points where test scores or class intervals meet frequencies
A single test score means more if one relates it to other test scores
A distribution of scores summarizes the scores for a group of
individuals
Frequency distribution: displays scores on a variable or a measure to reflect
how frequently each value was obtained
o One defines all the possible scores and determines how many
people obtained each of those scores
Income is an example of a variable that has a positive skew
Whenever you draw a frequency distribution or a frequency polygon, you
must decide on the width of the class interval
Class interval: for inches of rainfall is the unit on the horizontal axis
Measures of Central Tendency
Measure of central tendency: statistic that indicates the average or
midmost score between the extreme scores in a distribution.
The Arithmetic Mean
o “X bar”
o sum of observations divided by number of observations
o Sigma (X/n)
o Used for interval or ratio data when distributions are
relatively normal
The Median
o The middle score
o Used for ordinal, interval, and ratio data
o Especially useful when few scores fall at extremes
The Mode
o Most frequently-occurring score
o Bimodal distribution- 2 scores both have highest
frequency
o Only common with nominal data
CHAPTER 3: A STATISTICS REFRESHER CHAPTER 3: A STATISTICS REFRESHER
Measures of Variability Standard Scores
Standard Score: raw score that has been converted from one scale to another scale,
Variability: indication of how scores in a distribution are scattered or
where the latter has arbitrarily set mean and standard deviation
dispersed
-used for comparison
The Range Z-score
o Difference between the highest and lowest scores
conversion of a raw score into a number indicating how many standard
o Quick but gross description of the spread of scores
deviation units the raw score is below or above the mean of the distribution.
The interquartile and semi-interquartile range The difference between a particular raw score and the mean divided by the
o Distribution is split up by 3 quartiles, thus making 4 standard deviation
quarters each representing 25% of the scores
Used to compare test scores with difference scales
o Q2= median
o Interquartile range measure of variability equal to the T-score
difference between Q3 and Q1 Standard score system composed of a scale that ranges from 5 standard
o Semi-interquartile range interquartile range divided by 2 deviations below the mean to 5 standard deviations above the mean
Quartiles and Deciles No negatives
o Quartiles are points that divide the frequency distribution into
equal fourths Other Standard Scores
o First quartile is the 25th percentile; s econd quartile is the SAT
median, or 50th percentile; third quartile is the 75th GRE
percentile Linear transformation: when a standard score retains a direct
numerical relationship to the original raw score
o The interquartile range is bounded by the range of scores that
represents the middle 50% of the distribution Nonlinear transformation: required when data are not normally
o Deciles are similar but use points that mark 10% rather than distributed, yet comparisons with normal distributions need to be made
25% intervals o Normalized Standard Scores
o Stanine system: converts any set of scores into a When scores don’t fall on normal distribution
transformed scale, which ranges from 1 to 9 “normalizing a distribution involves ‘stretching’
The average deviation he skewed curve into the shape of a normal
o X-mean=x curve and creating a corresponding scale of
o Average deviation= (sum of all deviation scores)/ total standard scores, a scale called a normalized
number of scores standard score scale”
o Tells us on average how far scores are from the mean
The Standard Deviation
o Similar to average deviation
o But in order to overcome the (+/-) problem, each deviation is
squared
o Standard deviation: a measure of variability equal to the
square root of the average squared deviations about the mean
o Is square root of variance
o Variance: the mean of the squares of the difference b/w the
scores in a distribution and their mean
Found by squaring and summing all the deviation
scores and then dividing by the total
number of scores
o s = sample standard deviation
o sigma = population standard deviation
Skewness
skewness: nature and extent to which symmetry is absent
POSITIVE SKEW Ex.) test was too hard
NEGATIVELY SKEWED ex.) test was too easy
can be gauges by examining relative distances of quartiles from the median
Kurtosis
steepness of distribution
leptokurtic: relatively peaked
mesokurtic: somewhere in the middle
platykurtic: relatively flat

The Normal Curve

Normal curve: bell-shaped, smooth, mathematically defined curve, highest at center;
both sides taper as it approaches the x-axis asymptotically
-symmetrical, and thus have mean, median, mode, is same

Area under the Normal Curve

Tails and body

Standard Scores
Standard Score: raw score that has been converted from one scale to another scale,
where the latter has arbitrarily set mean and standard deviation
-used for comparison
CHAPTER 4: OF TESTS AND TESTING
Some Assumptions About Psychological Testing and Assessment
- Assumption 1: Psychological Traits and States Exist
o Trait: any distinguishable, relatively enduring way in which one
individual varies from another
o States: distinguish one person from another but are relatively less
enduring
Trait term that an observer applies, as well as strength or
magnitude of the trait presumed present
based on observing a sample of behavior
o Trait and state definitions also refer to individual variation
make comparisons with respect to the hypothetical average person
o Samples of behavior:
Direct observation
Analysis of self-report statements
Paper-and-pencil test answers
o Psychological trait covers wide range of possible
characteristics; ex:
Intelligence
Specific intellectual abilities
Cognitive style
Psychopathology
o Controversy regarding how psychological tests exist
Psychological tests exist only as constructs: an
informed, scientific concept developed or
constructed to describe or explain a behavior
Cant see, hear or touch infer existence
from overt behavior: refers to an
observable action or the product of an
observable action, including test- or
assessment-related responses
o Traits not expected to be manifested in behavior 100% of the time
Seems to be rank-order stability in personality
traits relatively high correlations between trait
scores at different time points
o Whether and to what degree a trait manifests itself is
dependent on the strength and nature of the situation
- Assumption 2: Psychological Traits and States Can Be Quantified and
Measured
o After acknowledged that psychological traits and states do exist, the
specific traits and states to be measured need to be defined
What types of behaviors are assumed to be
indicative of trait?
Test developer has to provide test users with a clear
operational definition of the construct under study
o After being defined, test developer considers types of item
content that would provide insight into it
Ex: behaviors that are indicative of a particular trait
o Should all questions be weighted the same?
Weighting the comparative value of a test’s items
comes about as the result of a complex interplay
among many factors:
Technical considerations
The way a construct has been defined (for
particular test)
Value society (and test developer) attach to
behaviors evaluated
o Need to find appropriate ways to score the test and interpret results
Cumulative scoring: test score is presumed to
represent the strength of the targeted ability or trait or
state
The more the testtaker responds in a
particular direction (as keyed by test
manual) the higher the testtaker is
presumed to possess the targeted trait or
ability
- Assumption 3: Test-Related Behavior Predicts Non-Test-Related Behavior
o Objective of test is to provide some indication of some aspects
of the examinee’s behavior
CHAPTER 4: OF TESTS AND TESTING
Tasks on some tests mimic the actual behaviors that the
test user is attempting to understand
o Obtained behavior is usually used to predict future behavior
o Could also be used to postdict behavior to aid in the
understanding of behavior that has already taken place
o Tools of assessment, such as a diary, or case history data, might be of
great value in such an evaluation
- Assumption 4: Tests and Other Measurement Techniques Have Strengths and
Weaknesses
o Competent test users understand a lot about the tests they use
How it was developed
Circumstances under which it is appropriate to
administer the test
How test should be administered and to whom
How results should be interpreted
o Understand and appreciation limitations for tests they use
- Assumption 5: Various Sources of Error Are Part of the Assessment Process
o Everyday error= misstates and miscalculations
o Assessment error= a long-standing assumption that factors other
than what a test attempts to measure will influence performance on a
test
o Error variance: component of a test score attributable to
sources other than the trait or ability measured
Assessees themselves are sources of error variance
o Classical test theory (CTT)/ True score theory: assumption is
made that each testtaker has a true score on a test that would be
obtained but for the action of measurement error
- Assumption 6: Testing and Assessment Can Be Conducted in a Fair and
Unbiased Manner
o Court challenged to various tests and testing programs have
sensitized test developers and users to the societal demand for fair
tests used in a fair manner
Publishers strive to develop instruments that are fair when
used in strict accordance with guidelines in the test manual
o Fairness related problems/questions:
Culture is different from people whom the test was
intended for
Politics
- Assumption 7: Testing and Assessment Benefit Society
o Many critical decisions are based on testing and assessment
procedures
WHAT’S A “GOOD TEST”?
- Criteria
o Clear instruction for administration, scoring, and interpretation
- Reliability
o A “good test”/measuring tool reliable
Involves consistency: the prevision with which the test
measures and the extent to which error is present in
measurements
Unreliable measurement needs to be avoided
- Validity
o Test is considered valid if it doesn’t indeed measure what it
purports to measure
o If there is controversy over the definition of a construct then the validity
is sure to be criticized as well
o Questions regarding validity focus on the items that collectively make
up the test
Adequately sample range of areas to measure
construct
Individual items contribute to or take away from
test’s validity
o Validity may also be questioned on grounds related to the
interpretation of test results
- Other Considerations
o “Good test” one that trained examiners can administer, score and
interpret with minimum difficulty
Useful
Yields actionable results that will ultimately benefit
individual testtakers or society at large
CHAPTER 4: OF TESTS AND TESTING
o Purpose of test compare performance of testtaker with
performance of other testtakers (contains adequate norms:
normative data)
Normative data provides standard with which results
measured can be compared
NORMS
- Norm-referenced testing and assessment: method of evaluation and a
way of deriving meaning from test scored by evaluating an individual
testtaker’s score and comparing it to scores of a group of testtakers
- Meaning of individual score is relative to other scores on the same test
- Norms (scholarly context): usual, average, normal, standard, expected or typical
- Norms (psychometric context): the test performance data of a
particular group of testtakers that are designed for use as a reference when
evaluating or interpreting individual test scores
- Normative sample: group of people whose performance on a particular test
is analyzed for reference in evaluation the performance of individual testtakers
o Yields a distribution of scores
- Norming: refers to the process of deriving norms; particular type of norm
derivation
o Race norming: controversial practice of norming on the basis
of race or ethnic background
- Norming a test can be very expensive user norms/program norms: consist
of descriptive statistics based on a group of testtakers in a given period of
time rather than norms obtained by form sampling methods
- Sampling to Develop Norms
- Standardization: process of administering a test to a representative
sample of testtakers for the purpose of establishing norms
o Standardized when has clear, specified procedures
- Sampling
o Developer targets defined group as population test
designed for
All have at least one common, observable
characteristic
o To obtain distribution of scores:
Test administered to everyone in targeted
population
Administer test to a sample of the population
Sample: portion of universe of people
deemed to be representative of whole
population
Sampling: process of selecting the
portion of universe deemed to be
representative of whole
o Subgroups within a defined population may differ with
respect to some characteristics and it is sometimes
essential to have these differences proportionately
represented in sample
Stratified sampling: sample reflects statistics of
whole population; helps prevent sampling bias and
ultimately aid in interpretation of findings
Purposive sampling: arbitrarily select sample we
believe to be representative of population
Incidental/convenience sampling: sample that is
convenient or available for use
Very exclusive (contain exclusionary
criteria)
- TYPES OF STANDARD ERROR:
o STANDARD ERROR OF MEASUREMENT – estimate the extent
to which an observed score deviates from a true score
o STANDARD ERROR OF ESTIMATE – In regression, an
estimate of the degree of error involved in predicting the value
of one variable from another
o STANDARD ERROR OF THE MEAN – a measure of sampling error
CHAPTER 4: OF TESTS AND TESTING
o STANDARD ERROR OF THE DIFFERENCE – estimate how large
a difference between two scores should be before the
difference is considered statistically significant
- Developing norms for a standardized test
o Establish a standard set of instructions and conditions under
which the test is given makes scores of normative
sample more comparable with scores of future testtakers
o All data collected and analyzed, test developer will summarize
data using descriptive statistics (measures of central tendency
and variability)
Test developer needs to provide precise
description of standardization sample itself
Descriptions of normative samples vary widely in
detail
Tracking
- Comparisons are usually with people of the same age
- Children at the same age level tend to go through different growth patterns
- Pediatricians must know the child’s percentile within a given age
group
- This tendency to stay at about the same level relative to one’s peers is
known as tracking (ie height and weight)
- Diets may alter this “track”
- Faults: some believe there is an analogy between the rates of physical growth
and the rates of intellectual growth
o Some say that children learn at different rates
o This system discriminates against some children
TYPES OF NORMS
o Classification of norms ex: age, grade, national, local,
percentile, etc.
o PERCENTILES
Median= 2nd quartile: the point at or below which 50% of
the scores fell and above which the remaining 50% fell
Might wish to divide distribution of scores into
Deciles (instead of quartiles): 10 equal parts
The X th percentile is equal to the score at or below which
X% of scores fall
Percentile: an expression of the percentage of people
whose score on a test or measure falls below a particular
raw score
Percentage correct: refers to the
distribution of raw scores (number of items
that were answered correctly) multiplied by
100 and divided by the total number of items
*not same as percentile
Percentile is a converted score that refers to a
percentage of testtakers
Percentiles are easily calculated popular way of
organizing test related data
Using percentiles with normal distribution real
differences between raw scores may be minimized near
the ends of the distribution and exaggerated in the middle
(worsens with highly skewed data)
o AGE NORMS
Age-equivalent scores/age norms: indicate the
average performance of different samples of testtakers
who were at various ages at the time the test was
administered
Age norm tables for physical
characteristics
“Mental” age vs. physical age (need to
identify mental age)
o GRADE NORMS
Grade norms: designed to indicate the average test
performance of testtakers in a given school grade
Developed by administering the test to
representative samples of children over a
range of consecutive grades
Mean or median score for children at
each grade level is calculated
CHAPTER 4: OF TESTS AND TESTING
Great intuitive appeal
Do not provide info as to the content or type
of items that a student could or could not
answer correctly
Developmental norms: (ex: grade norms and age
norms) term applied broadly to norms developed on
the basis of any trait, ability, skill, or other
characteristic that is presumed to develop,
deteriorate, or otherwise be affected by chronological
age, school grade, or stage of life
o NATIONAL NORMS
National norms: derived from a normative sample that
was nationally representative of the population
at the time the norming study was conducted
o NATIONAL ANCHOR NORMS
Many different tests purporting to measure the same
human characteristics or abilities
National anchor norms: equivalency tables for scores on
tests that purpose to measure the same thing
Could provide the tool for comparisons
Provides stability to test scores by
anchoring them to other test scores
Begins with the computation of percentile
norms for each test to be compared
Equipercentile method: equivalency of
scores on different tests is calculated with
reference to corresponding percentile scores
o SUBGROUP NORMS
Normative sample can be segmented by an criteria
initially used in selecting subjects for sample
Subgroup norms: result of segmentation; more
narrowly defined
o LOCAL NORMS
Local norms: provide normative info with respect to
the local population’s performance on some test
Typically developed by test users
themselves
- Fixed Reference Group Scoring Systems
o Norms provide context for interpreting meaning of a test score
o Fixed reference group scoring system: distribution of scored
obtained on the test from one group of testtakers (fixed reference
group) is used as the basis for the calculation of test scores for future
administrators on the test
Ex: SAT test (developed in 1962)
NORM-REFERENCED VERSUS CRITERION-REFERENCED EVALUATION
- Way to derive meaning from test score is to evaluate test score in relation
to other scores on same test ( Norm-referenced)
- Criterion-referenced: derive meaning from a test score by evaluating it on the
basis of whether or not some criterion has been met
o Criterion: a standard on which a judgment or decision may be
based
- Criterion-referenced testing and assessment: method of evaluation and
way of deriving meaning from test scores by evaluating an
individual’s score with reference to a set standard (ex: to drive must past
driving test)
o Derives from values and standards of an individual or
organization
o Also called Domain/content-referenced testing and
assessment
o Critique: if followed strictly, important info about
individual’s performance relative to others can be
potentially lost
Culture and Inference
- Culture is a factor in test administration, scoring and interpretation
- Test user should do research in advance on test’s available norms to
check how appropriate it is for targeted testtaker population
o Helpful to know about the culture of the testtaker
CHAPTER 4: OF TESTS AND TESTING
CORRELATION
Degree and direction of correspondence between two things.
Correlation coefficient (r) – expresses a linear relationship between two
continuous variables
o Numerical index that tells us the extent to which X and Y
are “co-related”
Positive correlation: high scores on Y are associated with high scores on X,
and low scores on Y correspond to low scores on X
Negative correlation: higher scores on Y are associated with lower scores
on X, and vise versa
No correlation: the variables are not related
-1 to 1
Correlation does not imply causation.
o Ie weight, height, intelligence
PEARSON r
Pearson Product Moment Correlation Coefficient
Devised by Karl Pearson
Relationship of two variables are linear and continuous
Coefficient of Determination (r 2) – indication of how much variance is
shared by the X and the Y variables
SPEARMAN RHO
Rank order correlation coefficient
Developed by Charles Spearman
Used when the sample size is small and when both sets of
measurements are in ordinal form (ranking form)
BISERIAL CORRELATION
expresses the relationship between a continuous variable and an artificial
dichotomous variable
o If the dichotomous variable had been true then we would use
the point biserial correlation
o When both variables are dichotomous and at least one of the
dichotomies is true, then the association between them can be
estimated using the phi coefficient
o If both dichotomous variables are artificial, we might use a special
correlation coefficient – tetrachoric correlation
REGRESSION
analysis of relationships among variables for the purpose of
understanding how one variable may predict another
SIMPLE REGRESSION: one IV (X) and one DV (Y)
- Regression line: defined as the best-fitting straight line through a set of
points in a scatter diagram
o Found by using the principle of least squares, which
minimizes the squared deviation around the regression line
Primary use: To predict one score or variable from another
Standard error of estimate: the higher the correlation between X and Y, the
greater the accuracy of the prediction and the smaller the SEE.
MULTIPLE REGRESSION: The use of more than one score to predict Y.
Regression coefficient: (b) slope of the regression line
o Sum of squares for the covariance to the sum of squares for X
o Sum of squares is defined as the sum of the squared
deviations around the mean
o Covariance is used to express how much two measures
covary, or vary together
Slope describes how much change is expected in Y each time X
increases by one unit
Intercept (a) is the value of Y when X is 0
o The point at which the regression line crosses the Y axis
THE BEST-FITTING LINE
The difference between the observed and predicted score (Y- Y’) is
called the residual
The best-fitting line is most appropriately found by squaring each residual
Best-fitting line is obtained by keeping these squared residuals as small as
possible
o Principle of least squares:
Correlation is a special case of regression in which the scores for both variables
are in standardized, or Z, units
CHAPTER 4: OF TESTS AND TESTING CHAPTER 4: OF TESTS AND TESTING
In correlation, the intercept is always 0
Third Variable Explanation
Pearson product moment correlation coefficient is a ratio used to - Third variable, ie poor social adjustment, causes TV viewing and
determine the degree of variation in one variable that can be estimated aggression
from knowledge about variation in the other variable - External influence is the third variable
Testing the Statistical Significance of a Correlation Coefficient Restricted Range
- Begin with the null hypothesis that there is no relationship between - Correlation and regression use variability on one variable to explain
variables variability on a second variable
- Null hypothesis rejected is there is evidence that the association - Restricted range problem: correlation requires variability; if the
between two variables is significantly different from 0 variability is restricted, then significant correlations are difficult to find
- t distribution is not a single distribution, but a family of distributions, Mulvariate Analysis
each with its own degrees of freedom - Multivariate analysis considers the relationship among combinations of three
- Degrees of freedom are defined as the sample size minus 2, or N-2 of more variables
- Two-tailed test General Approach
- Linear combination of variables is a weighted composite of the original
How to Interpret a Regression Plot
variables
- Regression plots are pictures that show the relationship between - Y’ = a+b1X1 + … bkXk
variables
- Common use of correlation is to determine the criterion validity
evidence for a test, or the relationship between a test score and some
well-defined criterion
- Middle level of enjoyableness because it is the one observed most
frequently – normative because it uses info gained from representative groups
- Using the test as a predictor is not as good as perfect prediction, but
it is still better than using the normative info
- A regression line such as in 3.9 shows that the test score tells us nothing
about the criterion beyond the normative info

TERMS AND ISSUES IN THE USE OF CORRELATION

Residual
- Difference between the predicted and the observed values is called the
residual
o Y-Y’
- Important property of residual is that the sum of the residuals always equals 0
- Sum of the squared residuals is the smallest value according to the principle
of least squares
Standard Error of Estimate
- Standard deviation of the residuals is the standard error of estimate
- A measure of the accuracy of prediction
- Prediction is most accurate when the standard error of estimate is
relatively small
Coefficient of Determination
- Correlation coefficient squared is known as the coefficient of
determination
- Tells us the proportion of the total variation in scores on Y that we know as
a function of information about X
Coefficient of Alienation
- Coefficient of alienation is a measure of nonassociation between two
variables
- Square root of 1-r2 –-- r is the coefficient of determination
- High value means there is a high degree of nonassociation between 2 variables
Shrinkage
- Tendency to overestimate the relationship, particularly if the sample of
subjects is small
- Shrinkage is the amount of decrease observed when a regression equation
is created for one population and then applied to another
Cross Validation
- Use regression equation to predict performance in a group of subjects other
than the ones to which the equation was applied
- Standard error of estimate obtained for relationship between the values
predicted by the equation and the values actually observed – called cross
validation
The Correlation-Causation Problem
- Experiments are required to determine whether manipulation of one variable
causes changes in another variable
- A correlation alone does not prove causality, although it might lead to other
research that is designed to establish the causal relationships between
variables
CHAPTER 5: RELIABILITY
RELIABILITY
- Dependability and consistent
- Error implies that there will always be some inaccuracy in our
measurements
- Tests that are relatively free of measurement error are deemed to be reliable
- Reliability estimates in the range of .70 and .80 are good enough for most
purposes in basic research
- Reliability coefficient: an index that indicates the ratio between the true
score variance on a test and the total variance
- HISTORY OF RELIABILITY:
o Charles Spearman (1904): The Proof and Measurement of
Association between Two Things
o Then Thorndike
o Item response theory has taken advantage of computer
technology to advance psychological measurement
significantly
o Based on Spearman’s ideas
- X=T+E CLASSICALTEST THEORY
o assumes that each person has a true score that would be
obtained if there were no errors in measurement
o Difference between the true score and the observed score results
from measurement error
o Assumption here is that errors of measurement are
random
o Basic sampling theory tells us that the distribution of
random errors is bell-shaped
The center of the distribution should represent the
true score, and the dispersion around the
mean of the distribution should display the
distribution of sampling errors
o Classical test theory assumes that the true score for an
individual will not change with repeated applications of the
same test
o Variance: standard deviation squared. It is useful because it can
be broken into components:
o True variance: variance from true differences are
assumed to be stable
o Error variance: random irrelevant sources
- Standard error of measurement: we assume that the distribution of
random errors will be the same for all people, classical test theory uses the
standard deviation of errors as the basic measure of error
o Standard error of measurement tells us, on the average, how
much a score varies from the true score
o Standard deviation of the observed score and the
reliability of the test are used to estimate the standard error
of measurement
- Reliability: proportion of the total variance attributed to true
variance.
o the greater portion of total variance attributed to true
variance, the more reliable the test
- Measurement error: refers to collectively, all of the factors associated
with the process of measuring some variable, other than the variable being
measured
o Random error: a source of error in measuring a targeted
variable caused by unpredictable fluctuations and
inconsistencies of other variables in the measurement process
This source of error fluctuates from one testing
situation to another with no discernible pattern
that would systematically raise or lower scores
o Systematic Error:
A source of error in measuring a variable that is
typically constant or proportionate to what is
presumed to be true value of the variable being
measured
Error is predictable and fixable
Does not affect score consistency
CHAPTER 5: RELIABILITY
SOURCES OF ERROR VARIANCE
TEST CONSTUCTION
o Item sampling or content sampling – refer to variation among
items within a test as well as to variation among items between
test\
The extent to which a test takers score is affected
by the content sampled on a test and
by the way the content is sampled (that is, the way in
which the item is constructed) is a source of error
variance
- TEST ADMINISTRATION
o may influence the test takers attention or motivation
o Environment variables, test taker’s variables, examiner
variables. Level of professionalism
- TEST SCORING AND INTERPRETATION
o Computer scoring and a growing reliance on objective,
computer-scorable items have virtually eliminated error
variance caused by scorer differences
o However, other tools of assessment still require scoring by trained
personnel
o If subjectivity is involved in scoring, then the scorer can be a
source of error variance
o Despite rigorous scoring criteria set forth in many of the better
known test of intelligence, examiner occasionally still are
confronted by situations where an examinees response lies in
a gray area
TEST-RETEST RELIABILITY
- Also known as time-sampling reliability
- Correlating pairs of scores from the same group on two different
administration of the same test
- Measure something that is relatively stable over time
- Sources of Error variance:
o Passage of time: the longer the time that passes, the greater
the likelihood that reliability coefficient will be lower.
o Coefficient of stability: when the interval between testing is
greater than 6 months,
- Consider possibility of carryover effect: occurs when first testing
session influences scores from the second session
- If something affects all the test takers equally, then the results are
uniformly affected and no net errors occurs
- Practice tests may make this effect happen
- Practice can also affect tests of manual dexterity
- Time interval between testing sessions must be selected and
evaluated carefully
- Poor test-retest correlations do not always mean that a attest is unreliable –
suggest that the characteristic under study has changed
PARALLEL-FORM OR ALTERNATE FORMS RELIABILITY
- compares two equivalent forms of a test that measure the same
attribute
- Two forms should be equally constructed, both format, etc.
- When two forms of the test are available, one can compare
performance on one form versus the other – equivalent forms
reliability or parallel forms
- Coefficient of equivalence: degree of relationship between various forms of
a test can be evaluated by means of an alternate-forms
- Parallel forms: each form of the test, the means and variances of
observed test scores are equal
- Alternateforms: different versions of a test that have been
constructed so as to be parallel
- (1) two test administrations with the same group are required
- (2) test scores may be affected by factors such as motivation etc.
- Problem: developing a new version of a test
INTERNAL CONSISTENCY
- How well does each item measure the content/construct under
consideration
- How consistent the items together
- Used when tests are administered once
- If all items on a test measure the same construct, then it has a good internal
consistency
- Split-half reliability, KR20, Cronbach Alpha
CHAPTER 5: RELIABILITY
SPLIT-HALF RELIABILITY
- Correlating two pairs of scores obtained from equivalent halves of a single
test administered once.
- This is useful when it is impractical to assess reliability with two tests or to
administer test twice
- Results of one half of the test are then compared with the results of the
other
- Rules in splitting forms into half:
o Do not divide test in the middle because it would lower the
reliability
o Different amounts of anxiety and differences in item
difficulty shall also be considered
o Randomly assign items to one or the other half of the test
o use the odd-even system: where one subscore is obtained for the
odd-numbered items in the test and another for the even-
numbered items
- To correct for half-length, apply the Spearman-Brown formula, which
allows you to estimate what the correlation between the two halves would
have been if each half had been the length of the whole test
Use this if test user wish to shorten a test
o
CHAPTER 5: RELIABILITY
o Test takers with the same score on a heterogeneous test may
have quite different abilities
o However, homogenous testing is often an insufficient tool for
measuring multifaceted psychological variable such as
intelligence or personality
Measures of Inter-Scorer Reliability
 In some types of tests under some conditions, the score may be more a function
of the scorer than of anything else
 Inter-scorerreliability: is the degree of agreement or consistency between two or
more scorers (or judges or rather) with regard to a particular measure
 Coefficient of inter-scorer reliability: coefficient of correlation to
determine the degree of consistency among scorers in the scoring of a test
 Kappa statistic is the best method for assessing the level of agreement among
several observers
o Indicates the actual agreement as a proportion of the potential
agreement following the correction for chance agreement
o Cohen’s Kappa – 2 raters
o Fleiss’ Kappa – 3 or more raters
HOMOGENEITY VS. HETEROGENEITY OF TEST ITEMS
 Homogeneous items has high degree of reliability

o Used to determine the number of items needed to attain a

desired level of reliability DYNAMIC VS. STATIC CHARACTERISTICS
- Reliability increases as the test length increases  Dynamic: trait, state, ability presumed to be ever-changing as a function of
situational and cognitive experiences
KUDER-RICHARDSON FORMULAS OR KR20/KR21  Static: trait, state, ability relatively unchanging
- Kuder-Richardson technique simultaneously considers all possible ways
of splitting the items RESTRICTION OR INFLATION OF RANGE
- The formula for calculating the reliability of a test in which the items are  If it is restricted, reliability tends to be lower.
dichotomous, scored 0 or 1, is the Kuder-Richardson 20 (see p.114)  If it is inflated, reliability tends to be higher.
- Introduced KR21 – uses an approximation of the sum of the pq
SPEED TESTS VS. POWER TESTS
products – the mean test score
 Speed test: test is homogenous, means that it is easy but short time
CRONBACH ALPHA  Power test: Few items, but more complex.
- Cronbach developed a formula that estimates the internal consistency of
CRITERION-REFERENCED TESTS
tests in which the items are not scored as 0 or 1 – a more general reliability
estimate, which he called coefficient alpha  Provide an indication of where a testtaker stands with respect to some variable
or criterion.
- Sum the individual item variances
o Most general method of finding estimates of reliability  Tends to contain material that has been mastered in hierarchical fashion.
through internal consistency  Scores here tend to be interpreted in pass-fail terms.
- Domain sampling: define a domain that represents a single trait or  Measure of reliability depends on the variability of the test scores: how different
the scores are from one another.
characteristic, and each item is an individual sample of this general
characteristic
The Domain Sampling Model
- Factor analysis deals with the situation in which a test apparently
- This model considers the problems created by using a limited number of items
measures several different characteristics
to represent a larger and more complicated construct
o Good for the process of test construction
- Our task in reliability analysis is to estimate how much error we would make by
- Most widely used as a measure of reliability because it requires only one
using the score from the shorter test as an estimate of your true ability
administration of the test
- Ranges from 0 to 1 “bigger is always better”
- Conceptualizes reliability as the ratio of the variance of the observed score on
the shorter test and the variance of the long-run true score
Other Methods of Estimating Internal Consistencies
- Inter-item consistency: refers to the degree of correlation among all the
- Reliability can be estimated from the correlation of the observed test score
with the true score
items on a scale
o A measure of inter-item consistency is calculated from a single Item Response Theory
administration of a single form of a test
- Classical test theory requires that exactly the same test items be
o An index of inter-item consistency, in turn, is useful in
administered to each person – BAD
assessing the homogeneity of the test
- Item response theory (IRT) is newer – computer is used to focus on the
o Tests are said to be homogenous if they contain items that
range of item difficulty that helps assess an individual’s ability level
measure a single trait
o More reliable estimate of ability is obtained using a
o Definition: the degree to which a test measures a single factor
shorter test with fewer items
o Heterogeneity: degree to which a test measures different
o Takes a lot of items and effort
factors
o Ex: homo=test that assesses knowledge only of #-D television
repair skills vs. a general electronics repair test (hetero)
o The more homogenous a test is, the more inter-item
consistency it can be expected to have
o Test homogeneity is desirable because it allows relatively
straightforward test-score interpretation
o Test takers with the same score on a homogenous test
probably have similar abilities in the area tested
CHAPTER 5: RELIABILITY CHAPTER 5: RELIABILITY
Generalizability theory
 based on the idea that a person’s test scores vary from testing to testing because
of variables in the testing situation
 Instead of conceiving of all variability in a person’s scores as error, Cronbach
encouraged test developers and researchers to describe the details of the
particular test situation or universe leading to a specific test score
 This universe is described in terms of its facets: which include things like the
number of items in the test, the amount of training the test scorers have had, and
the purpose of the test administration
 According to generalizability theory, given the exact same conditions of all
the facets in the universe, the exact same test score should be obtained
 Universe score: the test score obtained and its analogous to a true score in the true
score model
 Cronbach suggested that tests be developed with the aid of a
generalizability study followed by a decision study
 Generalizability study: examines how generalizable scores from a
particular test are if the test is administered in different situations
 How much of an impact different facets of the universe have on the test score
 Ex: is the test score affected by group as opposed to individual
administration
 Coefficients of generalizability: the influence of particular facts on the test score is
represented by this. These coefficients are similar to reliability coefficients in the
true score model
 Decision study: developers examine the usefulness of test scores in helping
the test user make decision
 The decision study is designed to tell the test user how test scores should be used
and how dependable those scores are as a basis for decisions, depending on the
context of their use

What to Do About Low Reliability

- Two common approaches are to increase the length of the test and to throw
out items that run down the reliability
- Another procedure is to estimate what the true correlation would have
been if the test did not have measurement error
Increase the Number of Items
- The larger the sample, the more likely that the test will represent the true
characteristic
o This could entail a long and costly process however
- Prophecy formula
Factor and Item Analysis
- Reliability of a test depends on the extent to which all of the items
measure one common characteristic
- Factor analysis
o Tests are most reliable if they are unidimensional : one factor
should account for considerably more of the variance than
any other factor
- Or examine the correlation between each item and the total score for
the test called
o Discriminability analysis: when the correlation between
the performance on a single item and the total test score is
low, the item is probably measuring something different
from the other items on the test

Correction for Attenuation

- Potential correlations are attenuated, or diminished, by measurement error
CHAPTER 6: VALIDITY CHAPTER 6: VALIDITY
The Concept of Validity
- Validity: as applied to a test, is a judgment or estimate of how well a test o CVR Content validity ratio
measures what it purports to measure in a particular context o ne Number of panelists
o Judgment based on evidence about the appropriateness of stating “essential”
inferences drawn from test scores o N Total number of panelists
o Validity of test must be shown from time to time to account for culture
CVR is calculated for each item
and advancement o Culture and the relativity of content validity
- Inference: a logical result or deduction Tests thought of as either valid or invalid
- “Acceptable” or “weak” validity of tests and test scores What constitutes historical fact depends to some
- Validation: process of gathering and evaluating evidence about validity extent on who is writing the history
o Test user and testtaker both have roles in validation of test Culture relativity
o Test users may conduct their own validation studies: may yield insights Politics (politically correct)
regarding a particular population of testtakers as compared to the Criterion-Related Validity
norming sample (in manual) - Criterion-relatedvalidity: judgment of how adequately a test score can be used to
o Local validation studies: absolutely necessary when test user plans to infer an individual’s most probable standing on some measure of interest (measure
alter in some way the format, instructions, language, or content of the of interest being the criterion)
test - 2 types:
- Types of Validity (Trinitarian view) *not mutually exclusive all contribute to a o Concurrent validity: index of the degree to which a test score is
unified picture of a test’s validity/ critiq ue approach is fragmented and related to some criterion measure obtained at the same time
incomplete (concurrently)
o Content validity: measure of validity based on an evaluation of the o Predictive validity: index of the degree to which a test score
subjects, topics, or content covered by the items in the test predicts some criterion measure
o Criterion-related validity: measure of validity obtained by - What Is a Criterion?
evaluating the relationship of scores obtained on the test to scores o Criterion: a standard on which a judgment or decision may be based;
on other tests or measures standard against which a test or a test score is evaluated (criterion-
o Construct validity: measure of validity that is arrived at by related validity)
executing a comprehensive analysis of: (umbrella validity every o Characteristics of criterion
other variety of validity falls under it) Relevancy pertinent or applicable to the matter at hand
How scores on test relate to other test scores and Validity (for the purpose which it is being used)
measures Uncontaminated Criterion contamination: term
How scores on test can be understood within some applied to a criterion measure that has been based, at
theoretical framework for understand the construct that least in part, on predictor measures
the test was designed to measure - Concurrent Validity
- Strategies: ways of approaching the process of test validity o Test scores are obtained at about the same time as the criterion
o Content validation strategies measures are obtained measures of the relationship between the test
o Criterion-related validation strategies scores and the criterion provide evidence of concurrent validity
o Construct validation strategies o Indicate the extent to which test scores may be used to
- Face Validity estimate an individuals present standing on a criterion
o Face validity: relates more to what a test appears to measure to the o Once validity of inference from test scores is established= faster, less
person being tested than to what the test actually measures expensive way to offer a diagnosis or a classification decision
o Judgment concerning how relevant the test items appear to o Concurrent validity of a test can be explored with respect to
be usually from testtaker, not test user another test
o Lack of face validity= lack of confidence in perceived Prior research must have satisfactorily demonstrated the 1 st
effectiveness of test which decreases testtaker’s test’s validity
motivation/cooperation *may still be useful 1st test= validating criterion
- Content validity - Predictive validity
o Content validity: a judgment of how adequately a test samples o Test scores may be obtained at one time and the criterion
behavior representative of the universe of behavior that the test was measures obtained at a future time, usually after some
designed to sample intervening event has taken place
Ideally, test developers have a clear vision of the Intervening event training, experience, therapy,
construct being measured clarity reflected in the medication, etc.
content validity of the test Measures of relationship between the test scores and a
o Test blueprint: structure of the evaluation; a plan regarding the types criterion measure obtained at a future time provide an
of information to be covered by the items, the number of items tapping indication of the predictive validity test (how accurately
each area of coverage, the organization of the items in the test, etc. scores on the test predict some criterion measure)
Behavior observation is a technique frequently used in test o Ex: SAT test score and freshman gpa
blueprinting o Judgments of criterion validity are based on 2 types of statistical
o The quantification of content validity evidence:
Important in employment settings tests used to The validity coefficient
hire and promote Validity coefficient: correlation coefficient
One method: method for gauging agreement among raters that provides a measure of the relationship
or judges regarding how essential a particular item is (C.H. between test scores and scores on the
Lawshe) criterion measure
“Is the skill or knowledge measured by this Ex: Pearson correlation coefficient used to
item… determine validity between 2 measures (r)
o Essential Affected by restriction or inflation of
o Useful but not essential range
o Not necessary
To the performance of the job?”
Content validity ratio (CVR):
CVR= ne – (N/2)
(N/2)
CHAPTER 6: VALIDITY
Is the range of scores employed
appropriate to the objective of the
correlational analysis
No rules regarding the validity coefficient (how
high or low it should/could be for test to be
valid)
Incremental validity
o More than one predictor
o Incremental validity: the
degree to which an additional
predictor explains something
about the criterion measure that
is not explained by predictors
already in use
Expectancy data
Expectancy data: provides info that can be
used in evaluating the criterion-related validity
of a test
Score obtained on expectancy
test/tables likelihood testtaker will score
within some interval of scores on a criterion
measure (“passing”, “acceptable”, etc.)
Expectancy table: shows the percentage of
people within specified test-score intervals
who subsequently were placed in various
categories of the criterion
o May be created from
scatterplot
o Shows relationships
Expectancy chart: graphic representation of
an expectancy table
o The higher the initial rating, the
greater the probability of
job/academic success
Taylor Russell Table – provide an estimate of the
extent to which inclusion pf a particular test in the
selection system will actually improve selection
Selection ratio – relationship between the
number of people to be hired and the number
of people available to be hired
Base rate – percentage of people under
existing system for a particular position
Relationship between predictor and
criterion must be linear
Naylor-shine Tables – difference between the means of
the selected and unselected groups to derive an index of
what the test is adding to already established procedures
o Decision theory and Test utility
Base rate – extent to which a particular trait,
behavior, characteristic or attribute exists in the
population
Hit rate – defined as the proportion of people a test
accurately identifies as possessing or exhibiting a
particular trait.
Miss rate – proportion of people the test fails to
identify as having or not having attributes
False positive (type I error) – possess
particular attribute but actually does not have.
Ex: score above cutoff score, hired but failed
the job.
False negative (type II error) – does not
possess particular attribute but actually does
have. Ex. Scored below cutoff score, not hired,
but could have been successful
in the job
- Construct Validity
o Construct validity: judgment about the appropriateness of inferences
drawn from test scores regarding individual standings on a variable
called a construct
CHAPTER 6: VALIDITY
Construct: an informed, scientific idea developed or
hypothesized to describe or explain behavior
Ex: intelligence, depression, motivation,
personality, etc.
Unobservable, presupposed (underlying) traits
that a test developer invokes to describe test
behavior/criterion performance
Viewed as unifying concept for all validity evidence
o Evidence of Construct Validity
Various techniques of construct validation that
provide evidence:
Test is homogeneous measures single
construct
Test scores increase/decrease as function of
age, passage of time, or experimental
manipulation (theoretically predicted)
Test scored obtained after some even or
passage of time differ from pretest scores
(theoretically predicted)
Test scores obtained by people from
distinct groups vary (theoretically
predicted)
Test scores correlate with scores on other tests
(theoretically predicted)
Evidence of homogeneity
Homogeneity: refers to how uniform a test
is in measuring a single concept
Evidence correlations between subtest
scores and total test scores
Item-analysis procedures have been used in
quest for test homogeneity
Desirable but not necessary
Contributes no info about how construct
being measured relates to other constructs
Evidence of changes with age
If test purports to measure a construct that
changes over time then the test scores,
too, should show progressive changes to
be considered valid measurement of
construct
Does not in itself provide info about how
construct relates to other constructs
Evidence of pretest-posttest changes
Can be evidence of construct validity
Some more typical intervening experiences
responsible for changes in test scores are:
o Formal education
o Therapy/medication
o Any life experience
Evidence from distinct groups/method of contrasted
groups
Method of contrasted groups: one way of
providing evidence for the validity of a test is
to demonstrate that scores on the test vary in a
predictable way as a
function of membership in some group
Rationale if a test is a valid measure of a
particular construct, test scores from groups of
people who would presumed with respect to
that construct should have
correspondingly different test scores
Convergent evidence
Evidence for the construct validity of a
particular test may converge from a
number of sources, such as tests or
measures designed to assess the
same/similar construct
CHAPTER 6: VALIDITY CHAPTER 6: VALIDITY
Criterion data may be influenced by
Convergent evidence: scores on a test
rater’s knowledge of ratee race,
undergo construct validity and correlate
highly in the predicted direction with scores gender, etc.
on older, more established and o Test fairness
already validated tests designed to Issues of fairness tend to be more difficult and
measure the same/similar construct involve values
Discriminant evidence Fairness: the extent to which a test is used in an
Discriminant evidence: validity coefficient impartial, just, and equitable way
showing little relationship between test Sources of misunderstanding
scores and /or other variables with which Discrimination
scores on the test being construct- validated Group not included in standardization
should not theoretically be correlated sample
Provides evidence of construct validity Performance differences between
Multitrait-multimethod matrix: “two or identified groups
more traits”, “two or more methods”
Relationship Between Reliability and Validity
matrix/table that results from correlating
variables (traits) within and between - A test should not correlate more highly with any other variable than it
methods correlates with itself
Factor analysis - A modest correlation between the true scores on two traits may be missed
Factor analysis: shorthand term for a class of if the test for each of the traits is not highly reliable
mathematical procedures designed to identify - We can have reliability without validity
factors or specific variables that are typically o It is impossible to demonstrate that an unreliable test is valid
attributes, characteristics, or
dimension on which people may differ
Frequently used as a data reduction
method in which several sets of scores and
correlations between them are analyzed
Exploratory factor analysis: researchers test
the degree to which a hypothetical model
fits the actual data
o Factor loading: conveys
information about the extent to
which the factor determines the
test score or scores
o Complex procedures
- Validity, Bias, and Fairness
o Test Bias
Bias: a factor inherent in a test that systematically
prevents accurate, impartial measurement
Technical means to identify and remedy bias
(mathematically)
Bias implies systematic variation
Rating error
Rating: a numerical or verbal judgment (or
both) that places a person or an attribute
along a continuum identified by a scale of
numerical or word descriptions, known as a
rating scale
Rating error: judgment resulting from
intentional or unintentional misuse of a
rating scale
Leniency error/generosity error: error in
rating that arises from the tendency on the
part of the rater to be lenient in scoring,
marking, and/or grading
Severity error: rater exhibits general and
systematic reluctance to giving ratings at
either the positive or negative extreme
Overcome restriction of range rating errors is to use
rankings: procedure that requires the rater to measure
individuals against one another instead of against an
absolute scale
Rater is forced to select 1st, 2nd, 3rd, etc.
Halo effect: fact that for some raters, some rates can do no
wrong
Tendency to give a particular ratee a higher
rating than he or she objectively deserves
CHAPTER 7: UTILITY CHAPTER 7: UTILITY
Utility: usefulness or practical value of testing to improve efficiency Based on norm-related considerations rather than
on the relationship of test scores to a criterion
Factors that Affect a Test’s Utility Also called norm-referenced cut score
Psychometric Soundness Ex.) top 10% of test scores get A’s
o Reliability and validity of a test
o Fixed cut score: set with reference to a judgment concerning
o Gives us the practical value of both the scores (reliability and
a minimum level of proficiency required to be included in a
validity)
particular classification.
o They tell us whether decisions are cost-effective
Also called absolute cut scores
o A valid test is not always a useful test
o Multiple cut scores: using two or more cut scores with
especially if testtakers do not follow test reference to one predictor for the purpose of categorizing
directions testtakers
Costs Ex.) having cut score that marks an A, B, C etc. all
o Economic and non economic
measuring same predictor
o Ex.) using a less expensive and therefore less stringent
o Multiple hurdles: for success, requires one individual to
application process for airline personnel. complete many tasks, with elimination at each level
Benefits
Ex.) written application group interview
o Profits, gains, advantages
personal interview etc.
o Ex.) more stringent hiring policy more productive
o Compensatory model of selection: assumption is made that
employees
high scores on one attribute can compensate for low scores on
o Ex.) maintaining successful and academic environment of
another attribute
university
Methods for Setting Cut Scores
Utility Analysis
The Angoff Method
What is Utility Analysis?
-a family of techniques that entail a cost-benefit analysis designed to yield information
 Judgments of experts are averaged
relevant to a division about the usefulness and/or practical value of a tool of assessment. The Known Groups Method

Utility analysis: An illustration

 Collection of data on the predictor of interest from group known to posses and not to
possess trait, attribute, or ability
What’s the companies goal?
Limit the cost of selection
 Cut score based on which test best discriminates the two groups performance
o Don’t use FERT
IRT-Based Method
Ensure that qualified candidates are not rejected
 Based on testtaker’s performance across all items on a test
o Set a cut score that yields the lowest false negative rate
 Some portion of test items must be correct
Ensure that all candidates selected will prove to be qualified
o Lowest dales positive rate  Item-mapping method: determining difficulty level reflected by cut score (?)
Book-Mark method: test items are listed, one per page, in ascending level of
Ensure, to the extent possible, that qualified candidates will be
selected and unqualified candidates will be rejected difficulty. An expert places a bookmark to mark the divide which separates
o False positives are no better or worse than false negatives testtakers who have acquired minimal knowledge, skills, or abilities and those
that have not.
o Highest hit rate and lowest miss rate
 Problems include training of experts, possible floor and ceiling effects, and the
How Is a Utility Analysis Conducted? optimal length of item booklets
-objective: dictate what sort of information will be required as well as the specific
methods to be used Other Methods
Expectancy Data  discriminant analysis: family of statistical techniques used to shed light on the
o Expectancy table provides indication of the likelihood that a relationship between certain variables and two or more naturally occurring
testtaker will score within some interval of scores on a criterion groups
measure ex.) the relationships between scores of tests and ppl judged to be
o Used to measure costs vs. benefits successful or unsuccessful at job
Brogden-Cronbach-Gleser formula
o Utility gain: estimate of the benefit of using a particular test
or selection method
o Most simply is benefits-cost
o Productivity gain: estimated increase in work output

Some Practical Considerations

The Pool of Job Applicants
o There is rarely a limitless supply of potential employees
o Dependent on many factors, including economic
environment
o We assume that top scoring individuals will accept the job, but
those individuals are more likely to be the ones being offered
higher positions
The complexity of the Job
o It is questionable whether the same utility analysis methods can
be used to measure the eligibility of varying complexities of jobs
The cut score in use
o Relative cut score: may be defines as reference point
CHAPTER 8: TEST DEVELOPMENT
TEST DEVELOPMENT
STEPS:
1.
TEST CONCEPTUALIZATION
2. TEST CONSTRUCTION
3. TEST TRYOUT
4. ITEM ANALYSIS
5. TEST REVISION
TEST CONCEPTUALIZATION
- Thoughts or stimulus that could be almost everything.
- An emerging social phenomenon or pattern of behavior might serve as the
stimulus for the development of a new test.
- Norm referenced: An item for which high scorers on the test respond
correctly. Low scorers respond to that same item incorrectly
- Criterion referenced: high scorers on the test get a particular item right
whereas low scorers on the test get that same item wrong.
- Pilot work: pilot study or pilot research. To know whether some items should
be included in the final form of the instrument.
o the test developer typically attempts to determine how best to
measure a targeted construct
TEST CONSTRUCTION
- Scaling: process of setting rules for assigning numbers in
measurement.
- L.L. Thurstone: credited for being the forefront of efforts to develop
methodologically sound scaling methods.
TYPES OF SCALES:
- Nominal, ordinal, interval or ratio
- Age-based scale
- Grade-based scale
- Stanine scale (raw score converted to 1-9)
- Unidimensional vs. multidimensional
o Unidimensional: measuring one construct
o Multidimensional: measuring more than one construct
- Comparative vs. categorical
o Comparative scaling: entails judgments of a stimulus in
comparison with every other stimulus on the scale
o Categorical scaling: stimuli are placed into one of two or more
alternative categories that differ quantitatively with respect to
some continuum
- Rating Scale: Which can be defined as a grouping of words,
statements, or symbols on which judgments of the strength of a
particular trait, attitude, or emotion are indicated by the testtaker
- Summative scale: when final score is obtained by summing the
ratings across all the items
- Likert scale: each item presents the testtaker with five alternative
responses usually on agree-disagree, or approve-disapprove continuum
- Method of paired comparisons: presented with two stimuli and
asked to compare
- Comparative scaling: judging of a stimulus in comparison with every other
stimulus on the scale
- Categorical scaling: testtaker places stimuli into a category; those
categories differ quantitatively on a spectrum.
- Guttman scale (Scalogram analysis): items range from sequentially
weaker to stronger expressions of attitude, belief, or feeling. A testtaker who
agrees with the stronger statement is assumed to also agree with the milder
statements
- Equal-appearing intervals (Thurstone): direct estimation because
don’t need to transform testtaker’s response to another scale
WRITING ITEMS
- 3 Questions of test developer
o What range of content should the items cover?
o Which of the many different types of item formats should be
employed?
o How many items should be written in total and for each content area
covered?
- Item pool: reservoir from which items will not be drawn for the final version of
the test (should be about double the number of questions as final will have)
CHAPTER 8: TEST DEVELOPMENT
- Item format
o Item format: variables such as the form, plan, structure,
o
arrangement and layout of individual test items
2 types
o
1.) selected-response format: testtaker selects a response from a set
of alternative responses
includes multiple choice, true-false, and matching
o
2.) constructed-response format: testtaker supplies or creates the
correct answer
includes completion item, short answer and essay
- Writing Items for computer administration
o Item bank: relatively large and easily accessible collection of test
questions
o Computerized Adaptive Testing (CAT): interactive, computer-
administered testtaking process wherein items presented to the
testtaker are based in part on testtaker’s performance on previous
items.
o Floor effect: the diminished utility of an assessment tool for
distinguishing testtakers at the low end of the ability, trait, or other
attribute being measured
o Ceiling effect: diminished utility of an assessment tool for
distinguishing testtakers at the high end of the ability, trait,
attribute being measured
o Item branching: ability of computer to tailor the content and order
of presentation of test items on the basis of responses to previous
items
SCORING ITEMS
- Cummulative scoring: testtakers earn cumulative credit with regard to a
particular construct
- Class/category scoring: testtaker responses earn credit toward
placement in a particular class or category with other testtakers whose
pattern of responses is presumably similar in some way
- Ipsative scoring: comparing a testtaker’s score on one within a test to
another scale within that same test
o ex.)“John’s need for achievement is higher than his need
for affiliation”
ITEM WRITING (KAPLAN BOOK)
Item Writing
- Personality and intelligence tests require different sorts of responses
- Guidelines for item writing
o Define clearly what you want to measure
o Generate an item pool
o Avoid exceptionally long items
o Keep the level of reading difficulty appropriate for those who will
complete the scale
o Avoid “double-barreled” items that convey two or more ideas at
the same time
o Consider mixing positively and negatively worded items
- Must be sensitive to ethnic and cultural differences
- Items that retain their reliability are more likely to focus on skills, while those
that lost reliability focused on more abstract concepts
Item Formats
- Simplest test uses dichotomous format
The Dichotomous Format
- Dichotomous format offers two alternatives for each item
o Ie. True-false examination
- Advantages:
o Simplicity
o True-false items require absolute judgment
- Disadvantages:
o True-false encourage students to memorize material
o “truth” often comes in shades of gray
o mere chance of getting any item correct is 50%
- Yes-no format on personality tests
- Multiple-choice = polytomous
The Polytomous Format
- Polytomous format resembles the dichotomous format except that each item
has more than two alternatives
o Multiple-choice exams
- Advantage:
o Little time for test takers to respond to a particular item
because they do not have to write
- Incorrect choices are called distractors
CHAPTER 8: TEST DEVELOPMENT CHAPTER 8: TEST DEVELOPMENT
- Disadvantages: o The midpoint representing the optimal difficulty is obtained by
o How many distractors should a test have? --> 3 or 4 summing up the chance of success proportion and 1.00 and then
o Distractors hurting reliability / validity of test dividing the sum by 2
o Three alternative multiple-choice items may be better than five Item Reliability Index
alternative items because they retain the psychometric value but take o Indication of the internal consistency of a test
less time to develop and administer o Equal to the product of the item-score standard deviation (s) and the
o Scoring of the MC exams? --> simply guessing should elicit correlation (r)
correctness o Factor analysis and inter-item consistency
o Correcting for this though, the expected score is 0 – as getting a o Factor analysis determines whether items on a test appear to be
question wrong loses you a point measuring the same thing
- Guessing can be good if you can narrow down a couple answers The Item-Validity Index
- Students are more likely to guess when they anticipate a lower grade on a test than o Statistic designed to provide an indication of the degree to which a test is
when they are more confident measuring what it purports to measure
- Guessing threshold describes the chances that a low-ability test taker will obtain o Requires: item-score standard deviation, the correlation between the item
each score score and criterion score
- True-false and MC tests are common to educational and achievement tests The Item-Discrimination Index
- Likert format, category scale, and the Q-sort used for personality-attitude tests o Measures how adequately an item separates or discriminates
between high scorers and low scorers
Likert Format o “d”
- Likert format: requires that a respondent indicate the degree of agreement with a o compares performance on a particular item with performance in the upper
particular attitudinal question and lower regions of a distribution of continuous test scores
o Strongly disagree ....... Strongly agree o higher d means greater number of high scorers answering the item
o For measurements of attitude correctly
- Used to create Likert Scales: scales require assessment of item o negative d means low-scoring examinees are more likely to answer the item
discriminability correctly than high-scoring examinees
- Familiar and easy ------ likely to remain popular in personality and attitude o Analysis of item alternatives
tests Item-Characteristic Curves?
Category Format o Graphic representation of item difficulty and discrimination
- Category format: uses more choices than Likert; 10-point rating scale
Other Considerations in Item Analysis
- Disadvantage: responses to items on 10-pt scales are affected by the
o Guessing
groupings of the people or things being rated
o Usually in some direction
- People change their ratings depending on context
o Depends on individuals ability to take risks
o This problem can be avoided if the endpoints of the scale are clearly
o Item fairness
defined and the subjects are frequently reminded of the definitions of
o Bias
the endpoints
o Speed tests
- Optimal number of points is 7? o Last items will appear to be more difficult because not
o Number depends on the fineness of the discrimination that
everyone got to them
subjects are willing to make
o When people are highly involved with some issue, they will tend to Qualitative Item Analysis
respond best to a greater number of categories Qualitative methods: techniques of data generation and analysis that rely
- Increasing the number of response categories may not increase reliability and primarily on verbal rather than mathematical or statistical procedures
validity Qualitative item analysis: various nonstatistical procedures designed to
- Visual analogue scale: respondent is given a 100-millimeter line and asked explore how individual test items work
to place a mark between two well-defined endpoints o Through means like interviews and group discussions
o Measures self-rate health “Think aloud” test administration
Checklists and Q-Sorts o approach to cognitive assessment that entails respondents
- Adjective Checklist: subject receives a long list of adjectives and indicates vocalizing thoughts as they occur
whether each one is characteristic of himself or herself o used to shed light on the testtker’s though processes
o Requires subjects either to endorse such adjectives or not, thus allowing during the administration of a test
only two choices for each item Expert panels
- Q-Sort: increases the number of categories o Sensitivity review: study of test items in which they are
o Used to describe oneself or to provide ratings of others examined for fairness to all prospective testtakers as well as for
Other Possibilities the presence of offensive language, stereotypes, or situations
- Forced-choice and Likert formats are clearly the most popular in ITEM ANALYSIS (KAPLAN BASED)
contemporary tests and measures The Extreme Group Method
- Checklists have fallen out of favor because they are more prone to error than are - Compares people who have done well with those who have done poorly
formats that require responses to every item on a test
- Frequent advice is to not use “all of the above” as a response option - Difference between these proportions is called the discrimination index
The Point Biserial Method
TEST TRYOUT - Find the correlation between performance on the item and
What is a good item? performance on the total test
o Reliable and valid - Correlation between a dichotomous variable and a continuous variable
o Helps to discriminate testtakers is called a point biserial correlation
- On tests with only a few items, using this is problematic because
ITEM ANALYSIS
performance on the item contributes to the total test score
o The Item-Difficulty Index
Pictures of Item Characteristics
o Obtained by calculating the proportion of the total number
- Valuable way to learn about items is to graph their characteristics, which
of testtakers who answered the item correctly “p”
you can do with the item characteristic curve
o Higher p= easier item
o Difficulty can be replaced with endorsement in non-
achievement tests
CHAPTER 8: TEST DEVELOPMENT
- Prepare a graph for each individual test item
o Total test score is used as an estimate of the amount of a
‘trait’ possessed by individuals
- Relationship between performance on the item and performance on the test
gives some info about how well the item is tapping the info we want
Drawing the Item Characteristic Curve
- To draw this, we need to define discrete categories of test
performance
- If the test has been given to many people, we might choose to make each
test score a single category
- Gradual positive slope of the line demonstrates that the proportion of people
who pass the item gradually increases as test scores increase
o This means that the item successfully discriminates at all levels
of test performance
- Ranges in which the curve changes suggest that the item is sensitive,
while flat ranges suggest areas of low sensitivity
- Item analysis breaks the general rule the increasing the number of items
makes a test more reliable
- When bad items are eliminated, the effects of chance responding can be
eliminated and the test can become more efficient, reliable, and valid
Item Response Theory
- According to classical test theory, a score is derived from the sum of an
individual’s responses to various items, which are sampled from a larger
domain that represents a specific trait or ability
- New approaches consider the chances of getting particular items right
or wrong – item response theory – make extensive use of item analysis
o With this, each item on a test has its own item characteristic curve
that describes the probability of getting each particular item right
or wrong given the ability level of each test taker
o Testers can make an ability judgment without subjecting the
test taker to all of the test items
- Technical adv: builds on traditional models of item analysis and can provide
info on item functioning, the value of specific items, and the reliability of a
scale
- Two dimensions used are difficulty and discriminability
- Most attractive adv. Is that one can easily adapt the IRT tests for
computer administration
o Computer can rapidly identify the specific items that are
required to assess a particular ability level
- “peaked conventional”
- “rectangular conventional” – requires that test items be selected to
create a wide range in level of difficulty
o problem: only a few items of the test are appropriate for
individuals at each ability level; many test takers spend much of
their time responding to items either considerably below their
ability level or too difficult to solve
- IRT addresses traditional problems in test construction well
- IRT can identify respondents with unusual response patterns and offer
insights into cognitive processes of the test taker
- May also reduce the biases against the people who are slow in
completing test problems
External Criteria
- Item analysis has been persistently plagued by researchers’ continued
dependence on internal criteria, or total test score, for evaluating items
Linking Uncommon Measures
- One challenge in test applications is how to determine linkages
between two different measures
Items for Criterion-Referenced Tests
- Traditional use of tests requires that we determine how well someone has
done on a test by comparing the person’s performance to that of others
- Criterion-referenced tests compares performance with some clearly
defined criterion for learning
o Popular approach in individualized instruction programs
o Regarded as diagnostic instruments
CHAPTER 8: TEST DEVELOPMENT
- First step in developing these tests involves clearly specifying the
objectives by writing clear and precise statements about what the
learning program is attempting to achieve
- To evaluate the items: one should give the test to two groups of
students – one that has been exposed to the learning unit and one that has
not
- Bottom of the V is the antimode – the least frequent score
- This point divides those who have been exposed to the unit from those who
have not been exposed and is usually taken as the cutting score or point , or
what marks the point of decision
- When people get scores higher than the antimode, we assume that
they have met the objective of the test
Limitations of Item Analysis
- Main Problem: though statistical methods for item analysis tell the test
constructor which items do a good job of separating students, they do not
help the students learn
- Although the data are available to give the child feedback on the
“bug” in their thinking, nothing in the testing procedure initiates this
guidance
TEST REVISION
Test Revision in the Life Cycle of an Existing Test
Tests get old and need revision
Questions arise over equivalence of two tests
Cross-validation and Co-validation
o Cross-validation: revalidation of a test on a sample of
testtakers other than those on whom test performance was
originally found to be a valid predictor of some criterion
o Validity shrinkage: decrease in item validities that
inevitably occurs after cross-validation of finding
o Co-validation: test validation process conducted on two or more
tests using the same sample of testtakers
o Co-norming: when co-validation is used in conjunction with
the creation of norms or the revision of existing norms
o Quality assurance during test revision
test givers must have some degree of
qualification, training, and testing
anchor protocol: test protocol scored by a highly
authoritative scorer that is designed as a model for
scoring and a mechanism for resolving scoring
discrepancies
scoring drift: a discrepancy between scoring in an
anchor protocol and the scoring of another protocol
The Use of IRT in Building and Revising Tests
(item response theory)
Evaluating the properties of existing tests and guiding test revision
Determining measurement equivalence across testtaker populations
o Differential item functioning (DIF): phenomenon, wherein an
item functions differently in one group of testtakers as compared
to another group of testtakers known to have the same level of
the underlying trait
Developing item banks
o Items from other instruments item pool scrutiny
preliminary item bank psychometric testing item bank
CHAPTER 9: INTELLIGENCE AND ITS MEASUREMENT
What is Intelligence?
Intelligence: a multifaceted capacity that manifests itself in different ways across the
lifespan. Usually includes abilities to:
Acquire and apply knowledge
Reason logically
Plan effectively
Infer perceptively
Make judgment and solve problems
Grasp and visualize concepts
Pay attention
Be intuitive
Find the right words and thoughts with facility
Cope with, adjust to, and make the most of new situations
Intelligence Defines: Views of the Lay Public
Both social and academic
Intelligence Defined: Views of Scholars and Test Professionals
Francis Galton
o First to publish on heritability of intelligence
o Most intelligent persons were those with the best sensory
abilities
Alfred Binet
o Made tests about intelligence, but didn’t define it
o Components of intelligence: reasoning, judgment,
memory, abstraction
o Added that definition is complex; requires interaction of
components
o He argued that when one solves a particular problem, the abilities
used cannot be separated because they interact to produce the
solution.
David Wechsler
o Best way to measure this global ability was by measuring
aspects of several “qualitatively differentiable” abilities
o Complexity of intelligence
o Conceptualization as an “aggregate” or “global” capacity
Jean Piaget
o Studied children
o Believed order of maturation to be unchangeable
o With age, increased schema: organized action or mental
structure that, when applied to the world, leads to knowing or
understanding.
o Learning occurred through assimilation (actively organizing
new information so that it fits in with what already is perceived
and thought) and accommodation (changing what is already
perceived or though so that it fits with new information)
o Sensorimotor (0-2)
o Preoperational (2-6)
o Concrete Operational (7-12)
o Formal Operational (12 and older)
All share interactionism: complex concept by which heredity and
environment are presumed to interact and influence the development of
one’s intelligence
Factor-analytic theories: focus is squarely on identifying the
ability(ies) deemed to constitute intelligence
Information-processing theories: focus is on identifying the specific
mental processes that constitute intelligence.
Factor-Analytic Theories of Intelligence:
Charles Spearman: pioneered new techniques to measure
intercorrelations between tests.
o Existence of a general intellectual ability factor (g) that
tapped by all other mental abilities.
g representing the portion of the variance that all intelligence tests have in
common and the remaining portions of the variance being accounted for
either by specific components (s) or by error components (e)
greater g = better test was thought to predict overall intelligence
CHAPTER 9: INTELLIGENCE AND ITS MEASUREMENT
group factors: neither as general as g nor as specific as s
o ex.) linguistic, mechanical, arithmetical abilities
Guilford: multiple-factor models of intelligence
o Explain mental activities by deemphasizing, any reference to g
Thurstone: conceived intelligence as being composed of 7 primary
abilities.
Gardner: developed theory of multiple intelligences
o Question over whether emotional intelligence exists.
o Logical-mathematical, bodily-kinesthetic, linguistic,
musical, spatial, interpersonal and intrapersonal
Raymond Cattell: fluid vs. crystallized intelligence
o Crystallized intelligence: acquired skills and knowledge and
their retrieval. Retrieval of information and application of
general knowledge
o Fluid intelligence: nonverbal, relatively culture-free, and
independent of specific instruction.
Horn: added more to 7 factors
o Vulnerable abilities: decline with age and tend to return
preinjury levels following brain damage
o Maintained abilities: tend not to decline with age and may return
to preinjury levels following brain damage.
Carrol:
o Three-stratum theory of cognitive abilities: like geology
o Hierarchical model: meaning that all of the abilities listed in a
stratum are subsumed by or incorporated in the strata above.
o Those in the first stratum are narrow abilities
CHC model (Cattell-Horn-Carroll)
o Some overlap some difference
o Doesn’t use g
o Has broader abilities than Carroll’s theory
McGrew: Integrated the Cattell-Horn and Carroll’s model
McGrew and Flanagan: integrated McGrew-Flanagan CHC Model
o Features 10 broad stratum abilities
o 70 narrow-stratum abilities
o Makes no provision for the general intellectual ability factor
(g)
o It was omitted because it has little practical relevance to cross-
battery assessment and interpretation
The Information-Processing View
Aleksandr Luria
o How (not what) information is processed
o Simultaneous/parallel processing: integrated all at once
o Successive/sequential processing: each bit individually
processed
PASS model: (Planning, attention, simultaneous, successive)-model of
assessing intelligence
Sternberg ‘The essence of intelligence is that it provides a means to govern
ourselves so that our thoughts and actions are organized, coherent, and
responsive to both out internally driven needs and to the needs of the
environment”
Measuring Intelligence
Types of Tasks Used in Intelligence Test
Infants: test sensorimotor, interviews with parents
Older child: verbal and performance abilities
Mental Age: index that refers to chronological age equivalent to
one’s test performance
Adults: retention of general information, quantitative reasoning,
expressive language and memory, and socialjudgment
Theory in Intelligence Test Development and Interpretation
Weschler made a dichotomous test (Performance and Verbal), but
advocated multifaceted definition
Thorndike: intelligence = social, concrete, abstract
Putting theories into test are extremely hard
Intelligence: Some Issues:
Nature vs. Nurture
Currently believed to be mix of two
CHAPTER 9: INTELLIGENCE AND ITS MEASUREMENT CHAPTER 9: INTELLIGENCE AND ITS MEASUREMENT
Performationism: all structures, including intelligence are had at birth
and can’t be improved upon
Led to predeterminism: one’s abilities are predetermined by genetic
inheritance and no learning or intervention can enhance it
Interactionist: ppl inherit certain intellectual potential
o Theres a limit to genetic abilities (i.e. can’t ever have x-ray vision)
The Stability of Intelligence
Stable pretty much throughout one’s adult life
Cognitive abilities seem to decline with age
The Construct Validity of Tests of Intelligence
Having construct validity requires having unified understanding of what
intelligence is
Very difficult. Spearman says its one thing, Guilford says its many
Thorndike approach is sort of compromise
o Look for one central factor with three additional factors
representing social, concrete, and abstract intelligences
Other Issues
Flynn effect: IQ scores seem to rise every year, but not coupled with
rise in “true intelligence”
Personality
o High IQ: Need for achievement, competition, curiosity,
confidence, emotional stability etc.
o Low IQ: passivity, dependence, maladjustment
o Temperament (used to describe infants)
Gender
o Men usually outscore in visual spatialization tasks and
intelligence scores
o Women tend to outscore in language-skill tasks
o But differences can be bridged
Family Environment
o Divorce can have negative effects
o Begins with “maternal effects” in womb
Culture
o Provides specific models for thinking, acting and feeling
o Assumed that if cultural factors can be controlled then
differences between cultural groups will be lessened
o Assumed that culture can be removed by the reliance on
exclusively nonverbal tasks
Tend not to be very good at predicting success in
various academic and business settings
o Culture loading: the extent to which a test incorporates the
vocabulary, concepts, traditions, knowledge and feelings
associated with a particular culture
o No test can be culture free
o Culture-fair intelligence test: test/assessment process
designed to minimize the influence of culture with regard to
various aspects of evaluation procedure
o Another approached called for cultural-specific intelligence tests
Ex.) BITCH measured streetwiseness
Lacked predictive validity and useful, practical
information
CHAPTER 10: TESTS OF INTELLIGENCE
The Stanford-Binet Intelligence Scales
First to have detailed administration and scoring instructions
First American test to test IQ
First to use alternate items (an item that can be used in place of
another)
Lacked minority group representation
Ratio IQ =(mental age/chronological age)x100
Deviation Ratio/test composite: performance of one individual
compared to the performance of others of the same age. Has mean of
100 and standard deviation of 16
Age scale: items grouped by age
Point scale: items organized by category The
Stanford-Binet Intelligence Scales: Fifth Edition
Measures fluid intelligence, crystallized knowledge, quantitative
knowledge, visual-processing, and short-term (working) memory
Utilizes adaptive testing: testing individually tailored to testtakers to
ensure that items are neither too difficult (frustrating) or too easy (false
hope)
Examiner establishes rapport with testtaker, then administers routing
test to direct, route examinee to test items most likely at optimal level of
difficulty
Teaching items: show testtaker what is expected, how to do it.
o Can be used for qualitative assessment, but not scoring
Subtests for verbal and nonverbal tests share same name, but involve
different tasks
Floor: lowest level of items on subtest
Ceiling: highest-level item of subtest
Basal level: base-level criterion that must be met for testing on the
subtest to continue
Ceiling level is met when testtaker fails certain number of items in a row.
Test discontinues here.
Scores: raw standard composite
Extra-test behavior: behavioral observation The
Wechsler Tests
-commonality between all versions: all yield deviation IQ’s with mean of 100
and standard deviation of 15
Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV)
Core subtest: administered to obtain a composite score
Supplemental/Optional Subtest: provides additional clinical
information or extending the number of abilities or processes
sampled.
Yields four index scores: Verbal Comprehension Index, a Working Memory
Index, a Perceptual Reasoning Index, and a Processing Speed Index
Wechsler Intelligence Scale for Children –Fourth Edition (WISC-IV)
Process score: index designed to help understand how testtakers process
various kinds of information
WISC-IV compared to the SB5
Wechsler Preschool and Primary Scale of Intelligence-Third Edition
(WPPSI-III)
New school for children under 6
First major intelligence test which adequately sampled total
population of the United States
Subtests labeled core, supplemental, or optional
Wechsler, Binet, and the Short Form
Short form: test that has been abbreviated in length to reduce time
needed to administer, score and interpret
used with caution, only for screening
provide only estimates
reducing the number of items usually reduces reliability and thus validity
Wechsler Abbreviated Scale of Intelligence
The Wechsler Test in Perspective
Factor Analysis
o Exploratory factor analysis: summarizing data when we
are not sure how many factors are present in our data
o Confirmatory factor analysis: used to test highly
specific factor analysis
CHAPTER 10: TESTS OF INTELLIGENCE
Other Measures of Intelligence
Tests Designed for Individual Administration
Kaufman Adolescent and Adult Intelligence Test
Kaufman Brief Intelligence Test
Kaufman Assessment Battery for Children
Away from information processing and towards a distinction
between sequential and simultaneous processing
Tests Designed for Group Administration
Group Testing in the Military
o WWI need for government to test intelligence as
means of differentiating “unfit” and “exceptionally
superior ability”
o Army Alpha Test: to army recruits who could read.
Included general information questions, analogies, and
scrambled sentences to reassemble
o Army Beta Test: to foreign or illiterate recruits,
included mazes, coding, and picture completion.
o After the war, the alpha and beta test were used
rampantly, and oftentimes misused
o Screening tools: instrument of procedure used to
identify a particular trait or constellation of traits
o ASVAB (Armed Services Vocational Aptitude Battery):
administered to prospective to recruits or high school
students looked for career guidance
5 career areas: clerical, electronics,
mechanical, skill-technical, and combat
operations
Group Testing in Schools
o Useful in developing child’s profile - but cannot be sole
indicator
o Groups of 10-15
o Starting in Kindergarten
o Also called traditional group testing, because more modern
forms can utilize computer. These more aptly called
individual testing
Measures of Specific Intellectual Abilities
Widely used intelligence tests only test a sampling of the many
attributable factors aiding in intelligence
Ex.) Creativity
o Commonly thought to be composed of originality,
fluency, flexibility, and elaboration
o If the focus is too heavily on whether an answer is
correct, doesn’t allow for creativity
o Achievement tests require convergent thinking:
deductive reasoning process that entails recall and
consideration of facts as well as a series of logical
judgments to narrow down solutions and eventually arrive
at one solution
o Divergent thinking: a reasoning process in which
thought is free in many different directions, making
several solutions possible
Associated words, uses of rubber band etc.
Test-retest reliability for some of these tests are
near unacceptable
CHAP.11: OTHER INDIVIDUAL TESTS OF ABILITY IN EDUCATION AND SPECIAL EDUCATION
Alternative Individual Ability Tests Compared with the Binet and Wechsler Scales
- None of these are clearly superior from a psychometric
standpoint
- Some less stable, most more limited in their documented validity
- Compare poorly to Binet and Wechsler on all accounts
- They don't rely on a verbal response as much as the B and W
- Just use pointing or Yes/No responses, thus do not depend on the complex
integration of visual and motor functioning
- Contain a performance scale or subscale
- Their specificity often limits the range of functions or abilities that they
can measure
- Because they are designed for special populations, some
alternatives can be administered totally without the verbal instructions
SPECIFIC INDIVIDUAL ABILITY TESTS
- Earliest individual tests typically designed for specific purposes or
populations
- One of the first – Seguin Form Board Test – in 1800s – produced only a
single score
o Used primarily to evaluate mentally retarded adults and
emphasized speed and performance
- After, the Healy-Fernald Test was developed as an exclusively
nonverbal test for adolescent delinquents
- Knox developed a battery of performance tests for non-English adult
immigrants to the US – administered without language; speed not
emphasized
- These early individual tests designed for specific populations, produced
a single score, and had nonverbal performance scales
- Could be administered without visual instructions and used with children
as well as adults
Infant Scales
- Where mental retardation or developmental delays are suspected,
these tests can supplement observation, genetic testing, and other
medical procedures
Brazelton Neonatal Assessment Scale (BNAS)
- Individual test for infants between 3days and 4weeks
- Purportedly provides an index of a newborn’s competence
- Favorable reviews
- Considerable research base
- Wide use as a research tool and as a diagnostic tool for special
purposes
- Commonly used scale for the assessment of neonates
- Drawbacks:
o No norms are available
o More research is needed concerning the meaning and
implication of scores
o Poorly documented predictive and construct validity
o Test-retest reliability leaves much to be desired
Gesell Developmental Schedules (GDS)
- Infant intelligence measures
- Used as a research tool by those interested in assessing infant
intellectual development after exposure to mercury, diagnoses of
abnormal brain formation in utero and assessing infants with autism
- Children of 2.3mth to 6.3yrs
- Obtains normative data concerning various stages in maturation
- Individual’s developmental quotient (DQ) is determined according
to a test score, which is evaluated by assessing the presence or
absence of behavior associated with maturation
- Provides an intelligence quotient like that of the Binet
o (development quotient / chronological age) x 100
- But, falls short of acceptable psychometric standards
- Standardization sample not representative of the population
- No reliability or validity
- Does appear to help uncover subtle deficits in infants
CHAP.11: OTHER INDIVIDUAL TESTS OF ABILITY IN EDUCATION AND SPECIAL EDUCATION
Bayley Scales of Infants and Toddler Development – Third Edition (BSID-III)
- Base assessments on normative maturational developmental data
- Designed for infants between 1 and 42mths
- Assesses development across 5 domains: cognitive, language, motor,
socioemotional, and adaptive
- Motor scale: assumes that later mental functions depend on motor
development
- Excellent standardization
- Generally positive reviews
- Strong internal consistency
- More validity studies needed
- Widely used in research – children with Down syndrome,
pervasive developmental disorders, cerebral palsy, language
impairment, etc
- Most psychometrically sound test of its kind
- Predictive though?
Cattell Infant Intelligence Scale (CIIS)
- Based on normative developmental data
- Downward extension of Stanford-Binet scale for 2-30mth olds
- Similar to Gesell scale
- Rarely used today
- Sample is primarily based on children of parents from lower and middle
classes and therefore does not represent the general population
- Unchanged for 60yrs
- Psychometrically unsatisfactory
MAJOR TESTS FOR YOUNG CHILDREN
McCarthy Scales of Children’s Abilities (MSCA)
- Measure ability in children between 2-8yrs
- Present a carefully constructed individual test of human ability
- Meager validity
- Produces a pattern of scores as well as a variety of composite scores
- General cognitive index (CGI): standard score with a mean of 100 and a
standard deviation of 16
o Index reflects how well the child has integrated prior
learning experiences and adapted them to the demands
of the scales
- Relatively good psychometric properties
- Reliability coefficients in the low .90s
- In research studies
- Good validity? Good assessment tool
Kaufman Assessment Battery for Children - Second Edition (KABC-II)
- Individual ability test for children between 3-18yrs
- 18 subtests in 5 global scales called sequential processing,
simultaneous processing, learning, planning, and knowledge
- Intended for psychological, clinical, minority-group, preschool, and
neuropsychological assessment as well as research
- Sequential-simultaneous distinction
o Sequential processing refers to a child’s ability to solve
problems by mentally arranging input in sequential or serial
order
o Simultaneous processing refers to a child’s ability to
synthesize info from mental wholes in order to solve a
problem
- Nonverbal measure of ability too
- Well constructed and psychometrically sound
- Not much evidence of (good) validity
- Poorer predictive validity for school achievement – smaller
differences between whites and minorities
- Test suffers from a noncorrespondence between its definition and
its measurement of intelligence
GENERAL INDIVIDUAL ABILITY TESTS FOR HANDICAPPED AND SPECIAL
POPULATIONS
Columbia Mental Maturity Scale – Third Edition (CMMS)
- Purports to evaluate ability in normal and variously handicapped children
from 3-12yrs
- Requires neither a verbal response nor fine motor skills
CHAP.11: OTHER INDIVIDUAL TESTS OF ABILITY IN EDUCATION AND SPECIAL EDUCATION
GENERAL INDIVIDUAL ABILITY TESTS FOR HANDICAPPED AND SPECIAL
POPULATIONS cont…
Columbia Mental Maturity Scale – Third Edition (CMMS) cont…
- Requires subject to discriminate similarities and differences by
indicating which drawing does not belong on a 6-by-9inch card
containing 3-5 drawings
- Multiple choice
- Standardization sample is impressive
- Vulnerable to random error
- Reliable instrument that is useful in assessing ability in many people
with sensory, physical, or language handicaps
- Good screening device
Peabody Picture Vocabulary Test – Fourth Edition (PPVT-IV)
- 2-90yrs
- multiple choice tests that require subject to indicate Yes/No in some
manner
- Instructions administered aloud (not for the deaf)
- Purports to measure hearing or receptive vocabulary, presumably
providing a nonverbal estimate of verbal intelligence
- Can be done in 15mins, requires no reading ability
- Good reliability and validity
- Should never be used as a substitute for a Wechsler or Binet IQ
- Important component in a test battery or used as a screening device
- Easy to administer and useful for variety of groups
- BUT: Tendency to underestimate IQ scores, and problems
inherent in the multiple-choice format are bad
Leiter International Performance Scale – Revised (LIPS-R)
- Strictly a performance scale
- Aims at providing a nonverbal alternative to the Stanford-Binet scale
for 2-18yr olds
- For research, and clinical settings, where it is still widely utilized to assess
the intellectual function of children with pervasive developmental disorders
- Purports to provide a nonverbal measure of general intelligence
by sampling a wide variety of functions from memory to
nonverbal reasoning
- Can be applied to the deaf and language-disabled
- Untimed
- Good validity
Porteus Maze Test (PMT)
- Popular but poorly standardized nonverbal performance measure of
intelligence
- Individual ability test
- Consists of maze problems (12)
- Administered without verbal instruction, thus used for a variety of special
populations
- Needs restandardization
TESTING LEARNING DISABILITIES
- Major concept is that a child average in intelligence may fail in school
because of a specific deficit or disability that prevents learning
- Federal law entitles every eligible child with a disability to a free
appropriate public education and emphasizes special education and
related services designed to meet his or her unique needs and prepare
them for further education, employment, and independent living
- To qualify, child must have a disability and educational
performance affected by it
- Educators today can find other ways to determine when a child needs
extra help
- Processed called Response to Intervention (RTI): premise is that early
intervening services can prevent academic failure for many students with
learning difficulties
- Signs of learning problem:
o Disorganization
o Careless effort
o Forgetfulness
o Refusal to do schoolwork or homework
o Slow performance
o Poor attention
o Moodiness
CHAP.11: OTHER INDIVIDUAL TESTS OF ABILITY IN EDUCATION AND SPECIAL EDUCATION
Illinois Test of Psycholinguistic Abilities (ITPA-3)
- Assumes that failure to respond correctly to a stimulus can result not only
from a defective output system but also from a defective input or
information-processing system
- Stage 1: info must first be received by the senses before it can be
analyzed
- Stage 2: info is analyzed or processed
- Stage 3: with processed info, individual must make a response
- Theorizes that the child may be impaired in one or more specific sensory
modalities
- 12 subtests that measure individual’s ability to receive visual, auditory,
or tactile input independently of processing and output factors
- purports to help isolate the specific site of a learning disability
- For children 2-10yrs
- Early versions hard to administer and no reliability or validity
- Now, with revisions, ITPA-3 psychometrically sound measure of
children’s psycholinguistic abilities
Woodcock-Johnson III
- Evaluates learning disabilities
- Designed as a broad-range individually administered test to be used in
educational settings
- Assesses general intellectual ability, specific cognitive abilities,
scholastic aptitude, oral language, and achievement
- Based on the CHC three-stratum theory of intelligence
- Compares child’s score on cognitive ability with sore on
achievement – can evaluate possible learning problems
- Relatively good psychometric properties
- For learning disability tests, three conclusions seem warranted:
o 1. Test constructors appear to be responding to the same
criticisms that led to changes in the Binet and Wechsler
scales and ultimately to the development of the KABC
o 2. Much more empirical and theoretical research is
needed
o 3. Users or learning disabilities tests should take great pains
to understand the weaknesses of these procedures and not
overinterpret results
VISIOGRAPHIC TESTS
- Require a subject to copy various designs
Benton Visual Retention Test – Fifth Edition (BVRT-V)
- Tests for brain damage are based on the concept of psychological deficit ,
in which a poor performance on a specific task is related to or caused by
some underlying deficit
- Assumes that brain damage easily impairs visual memory ability
- For individuals 8yrs+
- Consists of geometric designs briefly presented and then removed
- Computerized version developed
Bender Visual Motor Gestalt Test (BVMGT)
- Consists of 9 geometric figures that the subject is imply asked to copy
- By 9yrs, any child of normal intelligence can copy the figures with only
one or two errors
- Errors occur for people whose mental age is less than 9, brain
damage, nonverbal learning disabilities, emotional problems
- Questionable reliability
Memory-for-Designs (MFD) Test
- Drawing test that involves perceptual-motor coordination
- Used for people 8-60yrs
- Good split-half reliability
- Needs for validity documentation
- All these tests criticized because of their limitations in reliability and
validity documentation
- Good as screening devices though
Creativity: Torrance Tests of Creative Thinking (TTCT)
- Measurement of creativity underdeveloped in psychological testing
- Creativity: ability to be original, to combine known facts in new ways,
or to find new relationships between known facts
- Evaluating this a possible alternative to IQ
- Creativity tests in early stages of development
CHAP.11: OTHER INDIVIDUAL TESTS OF ABILITY IN EDUCATION AND SPECIAL EDUCATION

Creativity: Torrance Tests of Creative Thinking (TTCT) cont…

- Torrance tests separately measure aspects of creative thinking such as

fluency, originality, and flexibility
- Does not meet the Binet and Wechsler scales in terms of
standardization, reliability, or validity
- Unbiased indicator of giftedness
- Inconsistent tests, but available data reflect the tests’ merit and fine
potential
Individual Achievement Tests: Wide Range Achievement Test-3 (WRAT-4)
- Achievement tests measure what the person has actually acquired or done
with that potential
- Discrepancies between IQ and achievement have traditionally been
the main defining feature of a learning disability
- Most achievement tests are group tests
- WRAT-4 purportedly permits an estimate of grade-level functioning
in word reading, spelling, math computation, and sentence
comprehension
- Used for children 5yrs+
- Easy to administer
- Problems:
o Inaccuracy in evaluating grade-level reading ability
o Not proven as psychometrically sound
CHAP: 12: STANDARDIZED TESTS IN EDUCATION, CIVIL SERVICE, AND THE MILITARY CHAP: 12: STANDARDIZED TESTS IN EDUCATION, CIVIL SERVICE, AND THE MILITARY

Summary of K-12 Group Tests

- When justifying the use of group standardized tests, test users often have
problems defining what exactly they are trying to predict, or what the test - All are sound, viable instruments
criterion is
College Entrance Tests
Comparison of Group and Individual Ability Tests
- SAT Reasoning Test, Cooperative School and College Ability Tests, and American
- Individual tests require a single examiner for a single subject
College Test
o Examiner provides instructions
SAT Reasoning Test
o Subject responds, examiner records response
- Most widely used college entrance test
o Examiner evaluates response
o Examiner takes responsibility for eliciting a maximum
- Used for 1000+ private and public institutions
performance - Renorming of the SAT did not alter the standing of test takers relative to one
another in terms of percentile rank
o Scoring requires considerable skill
- Those who use the results of group tests must assume that the subject - New scoring (2400) is likely to reduce interpretation errors, as interpreters
was cooperative and motivated can no longer rely on comparisons with older versions
o Many subjects tested at a time - 45mins longer – 3hrs and 45mins to administer
o Subjects record own responses - may disadvantage students with disabilities such as ADD
o Subjects not praised for responding - Verbal section now called “critical reading” – focus on reading
o Low scores on group tests often difficult to interpret comprehension
o No safeguards - Math section eliminated much of the basic grammar school math questions
Advantages of Individual Tests - Weakness: poor predictive power regarding the grades of students who score in
- Provide info beyond the test score the middle ranges
- Allow the examiner to observe behavior in a standard setting - Little doubt that the SAT predicts first-year college GPA
- Allow individualized interpretation of test scores o But, AfricanAmericans and Latinos tend to obtain lower scores
Advantages of Group Tests on average
- Are cost-efficient o Women score lower on SAT but higher in GPA
- Minimize professional time for administration and scoring
Cooperative School and College Ability Tests
- Require less examiner skill and training
- Falling out of favor
- Have more objective and more reliable scoring procedures
- Developed in 1955, not been updated
- Have especially broad application
- Purports to measure school- learned abilities as well as an individual’s potential to
OVERVIEW OF GROUP TESTS undertake additional schooling
Characteristics of Group Tests
- Psychometric documentation not strong
- Characterized as paper-and-pencil or booklet-and-pencil tests because
- Little empirical data support its major assumption – that previous success in
only materials needed are a printed booklet of test items, a test manual, scoring
acquiring school-learned abilities can predict future success in acquiring such
key, answer sheet, and pencil
abilities
- Computerized group testing becoming more popular
- Most group tests are multiple choice – some free response American College Test
- Group tests outnumber individual tests - Updated in 2005, particularly useful for non-native speakers of English
o One major difference is whether the test is primarily verbal, - Produces specific content scores and a composite
nonverbal, or combination - Makes use of the Iowa Test of Educational Development Scale
- Group test scores can be converted to a variety of units - Compares with the SAT in terms of predicting college GPA alone or in
Selecting Group Tests conjunction with high-school GPA
- Test user need never settle for anything but well-documented and - Internal consistency coefficients are not as strong in the ACT
psychometrically sound tests
Using Group Tests
- Reliable and well standardized as the best individual tests
- Validity data for some group tests are weak/meager/contradictory
Use Results with Caution
- Never consider scores in isolation or as absolutes
- Be careful using tests for prediction
CHAP: 12: STANDARDIZED TESTS IN EDUCATION, CIVIL SERVICE, AND THE MILITARY
GRADUATE AND PROFESSIONAL SCHOOL ENTRANCE
TESTS GRADUATE
Record Examination Aptitude Test
- GRE purports to measure general scholastic ability
- Most frequently used in conjunction with GPA, letters of rec, and other
academic factors
- General section with verbal and quantitative scores
- Third section which evaluates analytical reasoning – now essay format
- Contains an advanced section that measures achievement in at least 20 majors
- New 130-170 scoring scale
- Standard mean score of 500, and SD of 100
- Normative sample is relatively small
- Psychometric adequacy is less than that of SAT – validity and reliability
- Predictive validity not great
- Overpredicts the achievement of younger students while
underpredicting performance of older students
- Many schools have developed their own norms and psychometric
documentation and can use the GRE to predict success in their programs
- By looking at a GRE score in conjunction with GPA, graduate success
can be predicted with greater accuracy than without the GRE
- Graduate schools also frequently complain that grades no longer predict
scholastic ability well because of grade inflation – the phenomenon of
rising average college grades despite declines in average SAT scores
o Led to corresponding restriction in the range of grades
- As the validity of grades and letters of rec becomes more questionable, reliance
on test scores increases
- Definite overall decline in verbal scores while quantitative and analytical scores
are gradually rising
Miller Analogies Test
- Designed to measures scholastic aptitudes for graduate studies
- Strictly verbal
- 60 minutes
- knowledge of specific content and a wide vocab are very useful
- most important factors appear to be the ability to see relationships and a
knowledge of the various ways analogies can be formed
- psychometric adequacy is reasonable
- does not predict research ability, creativity, and other factors important to grad
school
The Law School Admission Test
- LSAT problems require almost no specific knowledge
- Extreme time pressure
- Three types of problems: reading comprehension, logical reasoning
(~half), and analytical reasoning
- Weight given to the LSAT score is openly published for each school
approved by the American Bar Association
- Entrance into schools based on weighted sum of score and GPA
- Psychometrically sound, reliability coefficients in the .90s
- Predicts first-year GPA in law school
- Content validity is exceptional
- Bias for minority group members, as well as women
- -
CHAP: 12: STANDARDIZED TESTS IN EDUCATION, CIVIL SERVICE, AND THE MILITARY
NONVERBAL GROUP ABILITY TESTS
Raven Progressive Matrices
- RPM one of the best known and most popular nonverbal group tests
- Suitable anytime one needs an estimate of an individual’s general
intelligence
- Groups or individuals, 5yrs-adults
- Used throughout the modern world
- Uses matrices – nonverbal; with or without a time limit
- Research supports RPM as a measure of general intelligence, or
Spearman’s g
- Appears to minimize the effects of language and culture
- Tends to cut in half the selection bias that occurs with the Binet or
Wechsler
Goodenough-Harris Drawing Test (G-HDT)
- Nonverbal intelligence test, group or individual
- Quick, east, and inexpensive
- Subject instructed to draw a picture of a whole an and to do the best
job possible
- Details get points
- One can determine mental ages by comparing scores with those of the
normative sample
- Raw scores can be converted to standard scores with a mean of 100 and SD
of 15
- Used extensively in test batteries
The Culture Fair Intelligence Test
- Designed to provide an estimate of intelligence relatively free of cultural
and language influences
- Paper-and-pencil procedure that covers three age groups
- Two parallel forms are available
- Acceptable measure of fluid intelligence
Standardized Tests Used in the US Civil Service System
- General Aptitude Test Battery (GATB) – reading ability test that
purportedly measures aptitude for a variety of occupations
o Makes employment decisions in govt agencies
o Attempts to measure wide range of aptitudes from general
intelligence to manual dexterity
- Controversial because it used within-group norming prior to the
passage of the Civil Rights Act of 1991
- Today, any kind of score adjustments through within-group norming in
employment practices is strictly forbidden by law
Standardized Tests in the US Military: The Armed Services Vocational Aptitude
Battery
- ASVAB administers to more than 1.3 million people a year
- Designed for students in grades 11 and 12 and in postsecondary schools
- Yields can help identify students who potentially qualify for entry into the
military and can recommend assignment to various military occupational
training programs
- Great psychometric qualities
- Reliability coefficients are excellent
Through computerized format, subjects can be tested adaptively, meaning
that the questions given each person can be based on his or her unique
ability
- This cuts testing time in half