ARANDA, Juliana M.
BPSY 198
BS PSYCH 4C
BPSY 198
The Science of Psychological Measurement
Self-Assessment: Glossary
Test your understanding of elements of this chapter
by seeing if you can explain each of
the following terms, expressions, and
abbreviations:
A Statistics Refresher
arithmetic mean – Also known as the mean, this is
a measure of central tendency generated by taking
the average of all scores in a distribution.
average deviation- Variability is calculated by
adding the absolute values of all the scores in a
distribution and dividing by the total number of
scores.
bar graph-A graphic illustration in which
data representing frequency are set on the vertical
axis, categories are set on the horizontal axis, and
the rectangle bars describing the data are often
noncontiguous.
bimodal distribution- the central tendency consists
of two scores that are always occurring in the
distribution equal of times.
Distribution- A collection of test scores organized
for recording or analysis.
Dynamometer- A device used for measuring the
strength of a person's grasp.
Error- All factors other than those deemed to be
measured by a test contribute to test scores; errors
are variables in all tests and assessments,
frequency distribution- A table of the results and
the number of times each result has been obtained.
frequency polygon- A graphic
Data visualization in which
Numbers that represent frequency are oriented
vertically, test scores or categories are assigned to
the data, and the horizontal axis are characterized
by a continuous line connecting the two points exam
results or categories match frequencies,
Graph- A diagram or chart that consists of lines,
points, bars or other symbols describing and
illustrating data.
grouped frequency distribution- A tabular
overview of test scores in a class, often known as
class intervals where the test scores are
grouped by intervals,
histogram- A vertical line graph at the true limit of
each test result forms a series of adjacent rectangles.
interquartile range- A measure of variability in a
dataset, calculated as the gap between the third
Ma’am Mical
quartile point and the first quartile point in a
distribution that has been segmented into quartiles.
interval scale- Scale: A measurement system
where items can be ordered with equal intervals
between them, and each unit on the scale is of the
same magnitude as any other, with no absolute zero
point, which means mathematical operations cannot
be performed on the data.
Kurtosis- An indication of whether the center of a
distribution is steep: peaked or flat in its shape.
Leptokurtic- A description of the kurtosis of a
distribution that exhibits a pronounced peak in its
central portion.
linear transformation the extent to which the new
score differs from other scores on the same scale is
in line with the differences between the original score
and the other scores on the scales that were used to
derive it. This contrasts with a nonlinear
transformation.
Mean- Commonly referred to as the arithmetic mean,
it's a measure of central tendency obtained by
computing the average of all the scores within a
distribution.
Measurement- Assigning numerical values or
symbols to the attributes of individuals or objects
based on specific guidelines or rules.
measure of central tendency- Tendency refers to
one of three statistical measures that convey
information about the central or average score within
a distribution. The mean, or arithmetic mean, serves
as a central tendency measure and belongs to the
ratio level of measurement. The median is another
measure of central tendency, considering the order
of scores, and it falls under the ordinal level of
measurement. On the other hand, the mode, as a
measure of central tendency, is nominal in nature. In
the context of an 80-83 range, the arithmetic mean
is typically used.
measure of variability- a statistical measure that
reveals the extent to which scores within a
distribution are spread or dispersed. Common
measures of variability include the range, standard
deviation, and variance.
Median- middlemost score in a distribution
Mesokurtic- An explanation of the kurtosis of a
distribution that exhibits a central shape that is
neither excessively peaked nor entirely flat.
Mode- identifying the most
frequently occurring score in a
distribution,
negative skew- A distribution is negatively skewed
when there are relatively fewer scores at the lower
end of the distribution.
ARANDA, Juliana M.
BS PSYCH 4C
nominal scale- A measurement system in which all
measured items are categorized and classified
according to one or more distinguishing
characteristics, and they are assigned to distinct and
all-encompassing categories.
nonlinear transformation- In psychometrics, a
method of altering a score where the new score may
not have a direct numerical link to the original score,
and the differences between the new score and
other scale scores might not mirror the differences
between the original score and the original scale
scores. This differs from a linear transformation.
normal curve- A mathematically defined curve,
smoothly shaped like a bell, with its peak at the
center and gradually diminishing on both sides, yet
never quite touching the horizontal axis. This
concept is often associated with the area under the
curve.
normalized standard score scale- In essence, the
final result of reshaping a skewed distribution to
resemble a normal curve, often achieved by
employing nonlinear transformation techniques.
normalizing a distribution- A statistical correction
applied to distributions that meet specific criteria in
order to approximate a normal distribution,
enhancing the ease of understanding or working with
the data.
ordinal scale- Scale: A measurement system where
items can be ranked in order, but the ranking doesn't
provide information about the precise difference
between ranks, and there is no absolute zero point.
In the fields of psychology and education, most
scales are considered ordinal.
Platykurtic- An explanation of the kurtosis of a
distribution that has a relatively flat shape in its
central portion.
positive skew- When there are relatively few scores
that fall at the upper end of the distribution.
quartile One of the three demarcation points that
separate the four quarters of a distribution, with each
of them typically marked or labeled.
Range- A measure of variability obtained by
determining the disparity between the highest and
lowest scores in a distribution.
ratio scale- A ratio scale permits ranking with equal
intervals and meaningful math, thanks to an absolute
zero point. Yet, ratio scales are infrequently seen in
psychology and education.
raw score A direct, unaltered report of performance,
typically expressed in numerical form and often
utilized for assessment or diagnosis.
scale
BPSY 198
Ma’am Mical
semi-interquartile range A measure of variability
obtained by dividing the interquartile range by 2.
skewness an indicator of how symmetry is lacking
in a distribution, where positive skewness occurs
when there are relatively few scores at the positive
end, and negative skewness occurs when there are
relatively few scores at the negative end.
standard deviation: A measure of variability
calculated as the square root of the average of
squared deviations from the mean, essentially the
square root of the variance.
standard score A raw score transformed from one
scale to another, with the new scale having a mean
and standard deviation set arbitrarily and being more
commonly used and easily understandable.
Examples of standard scores include z-scores and
T-scores.
stanine A standard score obtained from a scale with
an average of 5 and a standard deviation of roughly
2.
T score known as the Thorndike score, computed
using a scale with a mean fixed at 50 and a standard
deviation set at 10. This scoring method is employed
by the creators of the MMPI.
tail Between 2 and 3 standard deviations above the
mean on the normal curve, and the area between -2
and -3 standard deviations below the mean. A
normal curve exhibits two tails.
variability A measure of how scores in a distribution
are spread or scattered.
variance A measure of variability that is calculated
by finding the average of the squared differences
between the scores in a distribution and their mean.
z score obtained by subtracting a specific raw score
from the mean and then dividing by the standard
deviation. It quantifies a score in units of standard
deviation, signifying how far the raw score is from the
mean of the distribution in terms of standard
deviation units.
Of Tests and Testing
age-equivalent scores measure how well someone
or something is doing (individual’s ability) by
comparing it to what's typically expected for a certain
age.
age norms are created to serve as a reference point
based on the age of the individual who obtained a
particular score. They differ from grade norms.
bivariate distribution Also called a scatterplot,
scatter diagram, or scattergram, this is a visual
representation of correlation achieved by plotting the
ARANDA, Juliana M.
BS PSYCH 4C
coordinate points for values of the X-variable and the
Y-variable on a graph.
classical theory assumes that every person has a
"true" score (T) they would get on a test if everything
was perfect and there were no mistakes.
coefficient of correlation typically symbolized by
"r," is a measure of the strength of the linear
relationship between two continuous variables. It's
represented as a number that can range from -1 to
1. While various methods can be employed to
calculate a correlation coefficient, the most
commonly used one is Pearson's "r."
coefficient of determination is a value that shows
the extent to which two variables share variance in
their calculations. It's determined by squaring the
correlation coefficient, multiplying the result by 100,
and expressing it as a percentage. This percentage
signifies the amount of variance explained by the
correlation coefficient.
construct A well-informed scientific concept created
to describe or explain behavior. Constructs can
include notions like "intelligence," "personality,"
"anxiety," and "job satisfaction."
content-referenced testing and
assessment objective; measure student mastery of
learning goals
convenience sample refers to what is available for
use.
correlation representation of the extent and
direction of agreement between two continuous
variables.
criterion benchmark used to assess a test or a test
score, which can manifest in various forms, including
specific behaviors or a defined set of behaviors.
criterion-referenced testing and
assessment also called domain-referenced or
content-referenced testing, involves evaluating test
scores by comparing them to a predefined standard,
in contrast to norm-referenced testing.
cumulative scoring A scoring method in which
points or scores from individual items or subtests are
added together, and a higher total score suggests a
higher level of the measured ability or trait. This
differs from class scoring and ipsative scoring.
curvilinearity Typically in the context of graphs or
correlation scatterplots, curvature refers to the
extent to which the plot or graph displays a curved
shape or pattern.
developmental norms established based on any
trait, ability, skill, or characteristic that is assumed to
change, decline, or be influenced by age, grade level,
or stage of life.
BPSY 198
Ma’am Mical
domain sampling In the context of construct
measurement, a subset of behaviors or test items
represents a particular construct. These selections
serve as indicative samples from a broader array of
potential behaviors or items designed to assess the
same construct.
domain-referenced testing and
assessment concerned with how accurately we can
predict a person's performance within a specific area
or subject.
equipercentile method A method used to compare
scores on multiple tests, such as in the development
of national anchor norms. It involves establishing
percentile norms for each test and determining the
specific score on each test that corresponds to a
particular percentile.
error variance in the true score model, accounts for
the part of score variation that arises from unrelated,
random factors like test design, administration, and
scoring, distinct from the trait or ability being
measured.
fixed reference group scoring system that uses a
fixed reference group's score distribution as a basis
for calculating scores in future test administrations.
This method is employed, for example, in the scoring
of exams like the SAT and GRE.
System
grade norms tailored as a reference based on the
grade or educational level of the test taker who
attains a specific score. This differs from age norms.
incidental sample Commonly known as
convenience sampling, this method involves
selecting individuals for a sample based on their
easy accessibility rather than their
representativeness of the population under study.
intercept In the equation for a
regression line, Y a bX,
the letter a, which stands for a
constant indicating where the line
crosses the vertical or Y -axis,
local norms Normative data or information related
to a particular population, often of specific interest to
the test user. This data is gathered for the purpose
of making meaningful comparisons or evaluations.
meta-analysis A set of statistical methods employed
to merge data from multiple studies and generate
unified estimates of the statistics under investigation.
multiple regression The examination of
connections between multiple independent variables
and a single dependent variable to comprehend how
each independent variable influences or predicts the
dependent variable.
national anchor norms An
ARANDA, Juliana M.
BS PSYCH 4C
equivalency table for scores on
two nationally standardized tests
designed to measure the same.
thing,
national norms derived from
a standardization sample that was.
nationally representative of the
population,
norm encompass behavior or performance that is
considered customary, average, standard,
anticipated, or typical within a given context. The
term "norm" is the singular form of "norms," referring
to an individual standard or benchmark that serves
as a point of reference for measuring and comparing
behavior or performance.
normative sample comprises a group of individuals
assumed to be a representative subset of the larger
population that could potentially take a specific test.
The performance data of this group on the test
serves as a reference or context for assessing
individual test scores.
Norming The process of deriving or creating norms,
norm-referenced testing and
assessment method of evaluating and interpreting
test scores by comparing an individual test taker's
score to scores of a group of test takers who took the
same test. This method is different from criterion-
referenced testing and assessment, where the focus
is on specific performance standards or criteria
rather than group comparisons.
Outlier An extremely atypical plot point in a
scatterplot; (2) any extremely a typical finding in
research
overt behavior observable action or the product of
an observable action refers to something that can be
seen or measured, such as a person's behavior, test
results, or assessment responses. These are
tangible outcomes or behaviors that can be
observed or quantified in a research or assessment
context.
Pearson r The statistic commonly referred to as the
Pearson correlation coefficient, or Pearson's
coefficient of product-moment correlation, is widely
utilized to calculate a measure of the connection
between two continuous variables when that
connection is linear in nature.
percentage correct On a test with responses that
are scored as either correct or incorrect, this is an
expression of the percentage of items answered
correctly. It's calculated by taking the number of
correct answers, multiplying it by 100, and then
dividing by the total number of items on the test. This
BPSY 198
Ma’am Mical
is distinct from a percentile, which ranks a person's
performance relative to others.
Percentile A percentile indicates where a person's
score ranks among others, while "percentage
correct" shows the proportion of correct answers on
a test.
program norms use stats from a specific group
during a certain time, without the usual formal data
collection process, to see how they did.
purposive sampling the random selection of
individuals for a sample based on the belief that they
represent the population under study is called
purposive sampling.
race norming The controversial practice of norming
on the basis of race or ethnic background
rank-order/rank-difference the process of
positioning individuals, groups, or businesses on an
ordinal scale in relation to others.
correlation coefficient - a number that shows how
two things are connected.
regression Studying relationships between
variables to predict one variable based on another is
known as "predictive analysis."
regression coefficient In the formula Y = a + bX,
both "a" (representing the ordinate intercept) and "b"
(the slope of the line) are regression coefficients. In
practical terms, the specific values of "a" and "b" are
calculated using algebraic methods.
regression line The result of simple regression
analysis; the graphic “line of best fi t” that comes
closest to the greatest number of points on the
scatterplot of the two variables,
sample refers to selecting a group of people who are
believed to be representative of the entire population
or universe under study.
Sampling A general reference to the process of
developing a sample
scatter diagram a picture or graph that shows the
relationship between two variables.
scattergram tells us how strong and in which way
two things are connected.
scatterplot This is also known as a scatter diagram,
which is a graphical representation of correlation. It's
created by plotting the coordinate points for two
variables.
simple regression Analyzing the relationship
between a single independent variable and a
dependent variable is known as "bivariate analysis."
Spearman’s rho also called the rank-order
correlation coefficient and the rank-difference
correlation coefficient. It is often preferred when
dealing with small sample sizes and when both sets
of measurements are ordinal in nature.
ARANDA, Juliana M.
BS PSYCH 4C
standard error of the estimate I n regression, the
standard error of the estimate measures the
prediction's precision. A weaker correlation results in
a higher standard error.
Standardization a representative group to create
reference points or norms for interpreting test scores
standardized test A test or measure that has
undergone standardization is subject to established
guidelines or norms, often defined by organizations
like the Standards for Educational and Psychological
Testing (SEPT).
state state in personality refers to a temporary
expression of a trait, showing a short-term inclination
for certain behavior, as opposed to a lasting trait. In
psychoanalytic theory, it conveys the dynamic
conflicts between the id, ego, and superego.
stratified-random sampling known as "stratified
sampling," where a sample is created by dividing a
population into specific subgroups, and every
member within those subgroups has an equal
chance of being included in the sample.
stratified sampling This is the process of "stratified
sampling," which involves creating a sample by
dividing a population into specific subgroups.
subgroup norms Norms for a specific subgroup
within a larger group are guidelines or standards
established for that particular subset.
test standardization description of "test
standardization," where a test is given to a
representative sample of test-takers in specific
conditions. The data is then scored and interpreted
to set a benchmark for future test administrations
with different test-takers.
Trait a distinctive and relatively lasting characteristic
that differentiates one individual from another, in
contrast to a "state," which is more temporary and
changeable.
true score theory This system, also called the true
score model or classical test theory, assumes that a
test score (or a response to an individual item)
comprises a stable component that reflects what the
test or item is intended to measure and a random
component representing measurement error.
user norms These are often called "program
norms," which are descriptive statistics based on a
group of test-takers during a specific period, rather
than formal norms obtained through systematic
sampling methods.
Y= a+bX equation of linear regression
Reliability
BPSY 198
Ma’am Mical
alternate forms These are “accommodations” or
“alternative methods” implemented to cater to the
needs of the assessee, often designed to measure
the same variable in a way that accommodates their
unique circumstances or requirements.
Alternate-forms reliability an estimate of
“equivalent-forms reliability,” which gauges how item
sampling and other errors impact scores on two
versions of the same test. It differs from “parallel-
forms reliability,” which assesses the consistency
between scores on entirely different forms of a test.
Assumption of local
independence
assumption of monotonicity
assumption of unidimensionality
coefficient alpha This statistic, also known as
Cronbach’s alpha or simply alpha, is commonly used
in test construction to estimate reliability. More
precisely, it equals the mean of all split-half
reliabilities.
Coefficient of equivalence This is an estimate of
“alternate-forms reliability” or “parallel-forms
reliability,” which assesses the consistency between
scores on two different versions or forms of a test.
Coefficient of generalizability : In generalizability
theory, an index of the influence that particular facets
have on a test score
coefficient of inter-scorer
reliability the degree to which measurements are
consistent and repeatable, and it also reflects the
extent to which measurements vary across different
occasions due to measurement error.
Coefficient of stability This is an estimate of “long-
term test-retest reliability,” which is determined by
assessing the consistency of test scores over time
intervals of six months or more.
Confidence interval a range or band of test scores
that is likely to include the “true score” of an
individual, indicating the degree of confidence in the
measurement’s accuracy.
Content sampling This refers to “content diversity”
or “variety of subject matter” found within test items.
It’s often discussed regarding the differences among
individual test items within a test or between items in
two or more tests. It can also be called “item
sampling.”
Criterion-referenced test testing assesses an
individual’s score in relation to a predetermined
standard, in contrast to norm-referenced testing,
which compares scores to a group.
Decision study This research, conducted after a
generalizability study, aims to assess the practicality
and usefulness of test scores in decision-making.
ARANDA, Juliana M.
BS PSYCH 4C
Dichotomous test item A test item or question with
a binary response option, like true-false or yes-no.
discrimination In IRT, the extent to which an item
distinguishes between individuals with varying levels
of the measured trait or ability.
Domain sampling theory A subset of behaviors or
test items representing a specific construct from the
complete set of possible options.
Dynamic characteristic A dynamic trait, state, or
ability that is subject to change based on situational
and cognitive experiences, as opposed to a static
characteristic.
Error variance In the true score model, error
variance is the part of variability in observed scores
not associated with the trait or ability being
measured, often stemming from factors like test
design, administration, and scoring.
Estimate of inter-item consistency
facet
generalizability study In generalizability theory,
these are the universe variables of interest, such as
the test’s item count, scorer training, and the test’s
administration purpose.
Generalizability theory Also known as domain
sampling theory, it involves assumptions about
measurement, where a test score (or even a
response to an individual item) comprises a stable
component measuring what the test or item is
designed for, along with less stable components that
can be considered as error.
Heterogeneity - how diverse the factors a test
measure is.
Homogeneity - a test measures a single factor.
inflation of range/variance Also called variance
inflation, it’s a term related to reliability estimates in
which the variance of one or both variables in a
correlation analysis is artificially increased due to the
sampling method, leading to higher correlation
coefficients. This is in contrast to “restriction of
range.”
Information function tell us which test questions
are most effective for people with a certain level of
ability compared to others. It helps identify the
questions that work best for different skill levels.
inter-item consistency or homogeneity of test
items, typically assessed using methods like the
split-half technique.
Internal consistency estimate of
reliability A reliability estimate for a test derived
from inter-item consistency measurement.
Inter-scorer reliability : Also referred to as inter-
rater reliability, observer reliability, judge reliability,
BPSY 198
Ma’am Mical
and scorer reliability, an estimate of the degree of
agreement or consistency between two or more
scorers
item characteristic curve (ICC) A visual depiction
of the probabilistic link between an individual’s trait
or ability level and the likelihood of responding to an
item as expected. This is also called a category
response curve, an item response curve, or an item
trace line.
Item response theory (IRT) Also known as latent-
trait theory or the latent-trait model, it’s a set of
assumptions about measurement, including the
assumption that a test measures a unidimensional
trait and quantifies the extent to which each test item
measures that trait.
Item sampling Also known as content sampling, it
pertains to the diversity of subject matter within test
items. This is often discussed in the context of the
differences between individual items in a test or
between items in multiple tests.
Kuder-Richardson formula 20 A set of equations
by G. F. Kuder and M. W. Richardson used to
calculate the inter-item consistency of tests.
Latent-trait theory Also known as the latent-trait
model, it’s a framework of measurement
assumptions, including the idea that a test measures
a unidimensional trait and quantifies how each test
item assesses that trait.
Odd-even reliability An estimate of split-half
reliability, achieved by dividing a test into two halves,
with odd-numbered items in one half and even-
numbered items in the other.
Parallel forms Two or more equivalent versions of
the same test in which the means and variances of
observed test scores are identical. This is in contrast
to alternate forms.
Parallel-forms reliability This estimate evaluates
the impact of errors on test scores for two versions
of the same test, assuming equal means and
variances for each form, in contrast to alternate-
forms reliability.
Polytomous test item A test item or question
offering three or more response choices, of which
only one is considered correct or aligned with the
intended trait or construct being measured.
Power test A test, typically for achievement or
ability, has no time limit or an extremely long one,
enabling everyone to attempt all items. Some items
are intentionally very difficult, preventing perfect
scores. Contrast with speed test.
Rasch model - It is employed to gauge an
individual's skill or trait level by considering their
answers to test questions.
ARANDA, Juliana M.
BS PSYCH 4C
Reliability The consistency or repeatability of
measurements, which also represents the degree of
variation between measurements due to
measurement error.
Reliability coefficient A general term for an index
of reliability or the ratio of true score variance on a
test to the total variance is known as a “reliability
coefficient.
Restriction of range/variance “Restriction of
variance” is a term associated with reliability
estimates, where the sampling procedure narrows
the variance of variables, causing lower correlation
coefficients. This is the opposite of “inflation of
range.”
Spearman-Brown formula An equation used to
estimate internal consistency reliability by
correlating two halves of a lengthened or shortened
test. This method is not suitable for heterogeneous
tests or speed tests.
Speed test A time-limited test, typically assessing
achievement or ability, in which speed tests feature
items of consistent difficulty.
Split-half reliability A timed test, often used to
evaluate achievement or ability, where speed tests
contain items of uniform difficulty.
Standard error of a score - A gauge of the extent
to which a test score could fluctuate because of
factors such as measurement inaccuracies.
standard error of measurement The degree to
which an observed score differs from a true score,
also known as the standard error of a score. This
concept is related to reliability and is relevant to the
Stanford-Binet Intelligence Scale.
Standard error of the difference A statistic used to
determine the level of difference between two scores
that should be regarded as statistically significant.
Static characteristic – The trait, state, or ability
remains quite constant over time.
Test battery A battery of tests and assessments
consists of different tests serving a common
objective. For instance, it can involve intelligence,
personality, and neuropsychological tests to create
a comprehensive psychological profile of an
individual.
Test-retest reliability An estimate of reliability is
obtained by correlating scores from the same
individuals on two different test administrations. This
is relevant to test-takers, error variance,
measurement equivalence across populations,
personality tests, and considering the rights of
individuals taking the test.
BPSY 198
Ma’am Mical
Theta In Henry Murray’s personality theory, it’s a
term for the interaction between a person’s need and
the environmental press.
Transient error A source of error in testing due to
fluctuations in the test-taker’s emotions, moods, or
mental state over time.
True score theory Classical test theory, also called
the true score model, is a framework of
measurement assumptions stating that test scores
consist of a stable component representing what is
intended to be measured and a random component
considered as error.
True variance In the true score model, it refers to
the variance component associated with genuine
differences in the ability or trait being measured that
is inherent in an observed score or distribution of
scores.
Universe In generalizability theory, the total context
of a particular test situation, including all the factors
that lead to an individual testtaker’s score
universe score In the context of generalizability
theory, it’s a test score that pertains to the specific
universe being evaluated or assessed.
Variance A measure of variability calculated as the
average of the squared differences between scores
in a distribution and their mean, known as the
variance.
Validity
base rate An index, often represented as a
proportion, that quantifies the prevalence of a
specific trait, behavior, characteristic, or attribute
within a population.
Bias refers to an inherent factor within the test that
consistently hinders accurate and unbiased
measurement.
central tendency error A rating error characterized
by a rater's tendency to avoid extreme ratings,
causing most ratings to cluster in the middle of the
rating scale. This is known as central tendency error.
concurrent validity
Concurrent validity is a type of criterion-related
validity that gauges the extent to which a test score
correlates with a simultaneously acquired criterion
measure.
confirmatory factor analysis
Factor analysis techniques are used when we test
how well an assumed factor structure aligns with the
observed relationships among variables.
construct
ARANDA, Juliana M.
BS PSYCH 4C
An educated and scientific concept created to
elucidate or define behavior; instances of such
constructs encompass "intelligence," "personality,"
"anxiety," and "job satisfaction.”
construct validity
An assessment of the validity of conclusions made
about an individual's position on a construct based
on test scores.
content validity
An evaluation of how effectively a test (or any other
measurement tool) captures behavior that
accurately represents the entire range of behavior it
was intended to measure, taking into account
considerations such as the design, context, cultural
factors, and measurement precision.
content validity ratio (CVR)
A formula, devised by C. H. Lawshe, utilized to
assess consensus among evaluators concerning the
significance of an individual test item for inclusion in
a test, particularly in the context of content-
referenced testing and assessment.
convergent evidence
In the context of construct validity, this refers to data
obtained from other measuring instruments created
to assess the same or a related construct as the one
being validated. These data consistently support the
same assessment or conclusion regarding a test or
another measuring tool, in contrast to discriminant
evidence.
convergent validity
Data that demonstrates that a test assesses the
identical construct as another test claiming to
evaluate that same construct.
criterion
The benchmark against which a test or a test score
is assessed; this benchmark can manifest in various
forms, such as a particular behavior or a defined set
of behaviors.
criterion contamination
A situation where a criterion measure is, either
wholly or partially, dependent on a predictor
measure.
criterion-related validity
An assessment of the degree to which a score or
index obtained from a test or another measuring
instrument can reliably predict an individual's most
likely position on a specific measure of interest,
which is the criterion.
decision theory
A set of techniques employed to assess, in
quantitative terms, the effectiveness of selection
processes, diagnostic classifications, therapeutic
interventions, or other assessment or intervention-
BPSY 198
Ma’am Mical
related procedures, often with a focus on their
optimality, typically considering cost and benefit
aspects.
discriminant evidence
In the context of construct validity, data from a test
or another measuring tool that indicates a weak or
negligible connection between the test scores or
other variables that, in theory, should not have a
substantial correlation with the test being assessed
for construct validity. This is in contrast to
convergent evidence.
expectancy chart
Visual depiction of a table that displays expected
outcomes.
expectancy data
Data, typically presented in the form of an
expectancy table, demonstrating the probability that
an individual taking a test will achieve a score within
a specific score range on a criterion measure,
particularly in the context of utility analysis.
expectancy table
Data displayed in a table, showing the probability
that a test-taker will achieve a score within a
particular score range on a criterion measure.
exploratory factor analysis
A category of mathematical methods used to
approximate factors, extract factors, or determine
the number of factors to keep.
face validity
An assessment of the extent to which a test or
another measuring instrument effectively assesses
what it claims to measure, solely based on visual or
superficial cues, like the content of the test items.
factor analysis
A category of mathematical techniques often used
for simplifying data, intended to pinpoint variables
that exhibit variability among individuals (or factors),
explore correlations, understand factor loadings,
their purposes, instances of two-factor situations,
and various types of these techniques.
factor loading
In the context of factor analysis, a metaphor that
implies a test (or a specific test item) embodies or is
associated with a certain level of one or more
abilities, which subsequently significantly influence
the test score (or the response to that particular test
item).
fairness
When discussing tests, it pertains to how a test is
employed in a fair, unbiased, and equitable manner,
considering factors like group membership,
individual test items, potential misconceptions, and
its implications for validity.
ARANDA, Juliana M.
BS PSYCH 4C
false negative
A particular form of misrepresentation where an
assessment tool falsely suggests that the test-taker
lacks a specific trait, skill, behavior, or quality, when
in reality, the test-taker does possess or exhibit that
trait, skill, behavior, or quality.
false positive
A measurement mistake involving an assessment
tool that incorrectly implies the test-taker has a
specific trait, skill, behavior, or attribute when, in
reality, the test-taker does not possess it. This can
be particularly seen in a family environment context.
generosity error
Commonly known as leniency error, it denotes a
somewhat inaccurate rating or evaluation by a rater
stemming from the rater's inclination to be overly
generous or insufficiently critical. This is in contrast
to severity error.
halo effect
A rating error characterized by the rater having an
overwhelmingly positive view of the subject being
rated and typically assigning ratings that are
excessively favorable in a positive direction. This
arises from a situation in which the rater is inclined
to be overly positive and insufficiently critical.
hit rate
The accuracy in identifying individuals who possess
or lack a specific trait based on test scores. This term
is related to several concepts and references,
including John Holland, the Holtzman Inkblot
Technique (HIT), and more.
homogeneity
incremental validity
Employed alongside predictive validity, it quantifies
the ability of new predictors to provide additional
explanatory value beyond the predictors already in
the model.
inference
A reasonable outcome or inference in a process of
logical reasoning.
intercept bias
It refers to the point where a regression line,
generated by a test or measurement process,
consistently underestimates or overestimates the
performance of individuals within a particular group.
This is different from 'slope bias' and is commonly
associated with interest inventories.
leniency error
Also known as a generosity error, it's a type of rating
mistake that arises from a rater's inclination to be
overly lenient and inadequately critical.
local validation study
BPSY 198
Ma’am Mical
The procedure of collecting evidence that pertains to
the accuracy of how well a test assesses its intended
construct, with the aim of assessing the validity of a
test or another measurement tool. This is typically
conducted using a different population from the one
for which the test was initially validated.
method of contrasted groups It's a way of
comparing two different groups to see if a test works
well for both.
miss rate
The percentage of individuals for whom a test or
another measurement method does not correctly
identify the presence or absence of a specific trait,
behavior, characteristic, or attribute. In this context,
a "miss" signifies an incorrect classification or
prediction and can be further categorized as false
positives and false negatives.
multitrait-multimethod matrix
An approach to assess construct validity that
involves looking at both convergent and divergent
evidence through a correlation table depicting the
relationships between traits and methods.
Naylor-Shine tables
Statistical tables that were previously commonly
utilized to aid in assessing the effectiveness of a
specific test.
predictive validity
A type of criterion-related validity that measures how
well a test score foretells a particular criterion
measure.
ranking
The arrangement of individuals, scores, or variables
into ranked positions or levels of importance, often
following an ordinal scale. This concept is
associated with Alan Raphael.
rating
A numerical or verbal assessment that positions an
individual or characteristic on a scale defined by a
set of numerical or descriptive terms known as a
rating scale.
rating error
An evaluation stemming from the deliberate or
inadvertent mishandling of a rating scale; this type of
error includes leniency error (or generosity error)
and severity error.
rating scale
A structured set of numerical or verbal labels used
by assessors or individuals to gauge the presence or
extent of a particular trait, attitude, or variable,
including in behavioral observation and preschool
assessment.
severity error
ARANDA, Juliana M.
BS PSYCH 4C
Inaccurate rating or assessment error resulting from
the rater's inclination to be excessively critical, as
opposed to a generosity error.
slope bias
This term, related to a different slope of a regression
line between groups, signifies that a test or
measurement process consistently provides distinct
validity coefficients for individuals in different groups.
This is in contrast to intercept bias.
Taylor-Russell tables
Statistical tables that were widely employed to offer
test users an estimate of how the inclusion of a
specific test in the selection process would enhance
decision-making.
test blueprint
An elaborate blueprint outlining the content,
structure, and number of items to be included in a
test.
test utility theory
In psychological testing, it refers to how well a test
serves a particular purpose, considering factors like
clinical use, costs, decision-making, diagnostic
capabilities, and reliability and validity.
utility
It's about how effective a test or assessment method
is for a particular purpose, considering factors like
benefits, costs, reliability, and validity.
validation
The procedure of collecting and appraising evidence
related to validity.
validation study
Research that involves collecting evidence to assess
how accurately a test measures its intended
construct, with the aim of evaluating the test's validity
or the validity of another measurement tool.
validity
A broad concept that encompasses an assessment
of how effectively a test or another measuring tool
assesses what it claims to measure. This evaluation
carries significant implications for the validity of
inferences and decisions based on these
measurements.
validity coefficient
A correlation coefficient that offers an indication of
the connection between test scores and scores on a
criterion measure.
Utility
absolute cut score
An anchor in a distribution of test scores used to
separate a dataset into multiple categories, often
established based on a decision about a minimum
BPSY 198
Ma’am Mical
level of proficiency needed to belong to a specific
category. This is distinct from a relative cut score.
Angoff method
- lets experts set a passing score for an exam. One
big advantage is that this score is based on what's
in the test, not on how a group of test-takers
performs.
base rate
An indicator, typically as a ratio, that reflects the
presence of a particular trait or behavior in a
population. Terms related to this concept include
"BASIC ID," "BCG formula," "BDI," "Beck
Depression Inventory (BDI)," "Beck Self-Concept
Test (BST)," "behavior raters," and "behavior rating
scale.
benefits (as related to test utility)
the positive outcomes or advantages that come from
using a test or assessment, such as making better
decisions or achieving specific goals.
bookmark method
An IRT-based method for setting cut scores that
arranges items by difficulty, with experts selecting
items of ideal difficulty. This is related to Edwin G.
Boring, Bosnia, the Boston Naming Test, and BPS
(Bricklin Perceptual Scales).
Brogden-Cronbach-Gleser
formula
The combined work of Hubert E. Brogden and
decision theorists led by Cronbach and Gleser has
given rise to a formula that calculates the benefits,
whether monetary or otherwise, of using a specific
selection instrument under specific conditions. In
essence, it measures the utility gain of using a
particular test or selection method.
compensatory model of selection
costs (as related to test utility)
A selection model for job applicants built on the idea
that strong performance in one area can
compensate for weaker performance in another area.
cut score
Often known as a cutoff score, it's a specific point
(typically numerical) determined by judgment and
used to categorize a dataset into two or more groups,
with subsequent actions or inferences based on
these classifications.
discriminant analysis
A set of statistical methods employed to explore the
connection between specific variables and multiple
naturally occurring groups.
fixed cut score
A reference point in a test score distribution that
categorizes data into groups, typically based on a
ARANDA, Juliana M.
BS PSYCH 4C
minimum proficiency judgment. These are also
called fixed or absolute cut scores.
item-mapping method
is like making a map to show how hard or relevant
test questions are in comparison to one another.
known groups method
To support construct validity, a test should be able
to differentiate between people who have a specific
trait and those who don't.
method of contrasting groups - a way to study two
groups, one with a specific trait and one without, to
see how that trait affects them.
method of predictive yield
A method for determining cut scores by considering
the number of available positions. This approach
factors in predictions about the probability of
candidates accepting offers, the quantity of job
openings, and the distribution of applicant scores.
multiple cut scores
The utilization of multiple cutoff scores in relation to
a single predictor to classify test-takers into more
than two groups, or the application of distinct cutoff
scores for each predictor when employing multiple
predictors for selection.
multiple hurdle (selection
process)
A multi-step decision process wherein reaching a
specific cutoff score on one test is a prerequisite for
progressing to the next evaluation stage within the
selection process.
norm-referenced cut score
A point of reference within a distribution of test
scores used to categorize data into two groups,
primarily based on norms or typical values, rather
than the association of test scores with a specific
criterion.
productivity gain"An actual increase in productivity
or work output, which can be approximated in utility
analyses using a specific test or evaluation method.
relative cut score
Commonly known as a norm-based cutoff score, this
is a designated point within a test score distribution,
dividing data into two groups primarily based on
norm-related factors rather than the correlation
between test scores and a specific criterion.
return on investment
The ratio of economic and/or non-economic gains
achieved from spending on the initiation or
enhancement of a specific testing program, training
program, or intervention, in comparison to the total
costs incurred for these enhancements.
selection ratio
BPSY 198
Ma’am Mical
A numerical figure that indicates the correlation
between the quantity of individuals to be recruited
and the number of individuals who are accessible for
recruitment.
top-down selection
Assigning open positions to candidates based on
their scores, with the highest scorer receiving the
first position, the second highest scorer obtaining the
next position, and so on until all positions are
occupied.
Utility (test utility)
In psychological testing and assessment, it pertains
to the utility of a test or assessment method for a
specific purpose. This considers factors such as
clinical use, cost-effectiveness, decision theory,
diagnostic capability, as well as questions related to
reliability, validity, and theory.
utility analysis
Methods to weigh the pros and cons of testing
versus not testing, examining likely outcomes. This
covers the BCG formula, cut scores, expectancy
data, practical factors, productivity gain, and the
intended purpose.
utility gain
An approximation of the advantage, whether in
monetary terms or other aspects, gained from the
application of a specific test or selection technique.
Test Development
anchor protocol
An examination response sheet created by a test
publisher for the purpose of verifying the correctness
of examiners' scoring.
biased test item
An item that exhibits bias towards a specific group of
test-takers compared to another group when
accounting for variations in group proficiency
(Camilli & Shepard, 1985).
binary-choice item
A multiple-choice item offering only two response
options is termed a binary-choice item, such as the
familiar true-false question. While these items
usually present a statement for the test-taker to
classify as fact or not, they can also use other pairs
of responses like agree-disagree, yes-no, or right-
wrong.
categorical scaling
A scaling system where stimuli are categorized into
one of several alternative groups, which vary
quantitatively along a certain continuum.
category scoring
ARANDA, Juliana M.
BS PSYCH 4C
An evaluation method where test responses
determine placement in a category. Sometimes, a
set number of qualifying responses is needed to join
a specific group, distinct from cumulative and
ipsative scoring.
ceiling effect
Reduced effectiveness of an assessment tool in
differentiating high-ability or high-trait individuals.
class scoring
A method of evaluation where test responses
determine placement in a specific category.
Sometimes, a fixed number of qualifying responses
is required to join a particular group, differing from
cumulative and ipsative scoring.
comparative scaling
In test development, a technique for creating ordinal
scales by employing a sorting task that involves
evaluating a stimulus in relation to every other
stimulus utilized in the test, also known as
comparative utility.
completion item
All the words composing the section of a sentence
completion item, excluding the empty space that the
test-taker is supposed to fill in, often used in
sentence completion tests.
computerized adaptive testing
(CAT)
An interactive computer-based test-taking process
in which the items presented to the test-taker are
influenced by their performance on previous items.
co-norming
The procedure of norming tests involves
administering two or more tests to a group of test-
takers from the same sample. When this method is
utilized to establish the validity of all the tests
undergoing norming, it can also be termed as co-
validation.
constructed-response format
A type of test item that requires a constructed
response, like essay questions or fill-in-the-blank, as
opposed to selecting from predefined options, which
are found in selected-response formats.
co-validation
Co-validation is the process of validating multiple
tests using the same test-taker sample. It can also
be called co-norming when combined with norm
establishment or revision.
cross-validation
Revalidation involves testing a different sample of
individuals from the group for which the test's
predictive validity was originally established.
DIF analysis
BPSY 198
Ma’am Mical
In the context of IRT (Item Response Theory), a
method of analyzing item response curves for
distinct groups of test-takers to assess the
equivalence of measurement instruments or items
across those groups.
differential item functioning
(DIF)
In IRT, differential item functioning (DIF) happens
when the same test item yields different results for
various groups, often due to non-construct-related
group differences.
DIF items
In IRT, DIF refers to test items where individuals
from different groups, assumed to have the same
underlying ability, show different probabilities of
endorsing based on their group membership.
essay item
An essay question is ideal for assessing a deep
understanding of a single topic and encourages
creative expression in the test-taker's own words.
expert panel
Expert witnesses contribute to improving a test's
content, fairness, and other aspects by providing
their subject matter and population-related insights.
floor effect
It's when a test can't tell well the difference between
people with very low skills or abilities.
giveaway item
A test item typically positioned at the start of an
ability or achievement test, intentionally made to be
straightforward, often with the goal of boosting the
test-taker's confidence or reducing test-related
anxiety.
guessing
Guessing on personality tests is not a major concern,
as it's assumed test-takers aim to make the best
choice, even when it's challenging.
Guttman scale
- It's a scale that goes from showing less of
something to showing more of it, like a thermometer
measuring temperature from cold to hot.
ipsative scoring
It's grading a test by comparing your performance to
how others did on the same test, rather than just
looking at your absolute score.
item analysis
It's a way of looking at how different questions in a
test perform, like how hard they are or how reliable
they are. This helps understand how the test works.