CHAPTER 6 • SCORER RELIABILITY
- The extent to which two people
EVALUATING SELECTION TECHNIQUES
AND DECISIONS
CHARACTERISTICS OF EFFECTIVE
SELECTION TECHNIQUES
• RELIABILITY
- The extent to which a score from a
test or from an evaluation
is consistent and free from error.
- If a score from a measure is not
stable or error-free, it is not useful.
- assessments are justified by the
4 WAYS TO DETERMINE TEST
RELIABILITY
• TEST-RETEST RELIABILITY
- The extent to which repeated
administration of the same test will
achieve similar results.
▪ TEMPORAL STABILITY
- The consistency of test scores
across time
• ALTERNATE-FORMS RELIABILITY
- The extent to which two forms of
the same test are similar
▪ COUNTERBALANCING
- A method of controlling for order
effects by giving half of a sample.
Test A first, followed by Test B, and
giving the other half of the sample
Test B first, followed by Test A
▪ FORM STABILITY
- The extent to which the scores on
two forms of a test are similar.
• INTERNAL RELIABILITY
- The extent to which similar items
are answered in similar ways is
referred to as internal consistency
and measures item stability.
▪ ITEM STABILITY
- The extent to which responses to
the same test items are consistent.
▪ ITEM HOMOGENEITY
- The extent to which test items
measure the same construct.
• THE PEOPLE WHO WILL BE TAKING
THE TEST
scoring a test agree on the test score,
or the extent to which a test is scored
correctly.
▪ INTERRATER
- Also known as inter-observer
reliability or inter-scorer reliability, is
a measure of the consistency and
agreement between two or more
independent raters or observers
when assessing or scoring the same
set of data, such as responses,
behaviors, or judgments.
METHODS USED TO DETERMINE
INTERNAL CONSISTENCY:
• KUDER-RICHARDSON FORMULA 20
(K-R 20)
- A statistic used to determine internal
reliability of tests that use items with
dichotomous answers (yes/no,
true/false).
• SPLIT-HALF METHOD
- A form of internal reliability in which
the consistency of item responses is
determined by comparing scores on
half of the items with scores on the
other half of the items.
• SPEARMAN-BROWN PROPHECY
FORMULA
- Used to correct reliability coefficients
resulting from the split-half method
• COEFFICIENT ALPHA
- A statistic used to determine internal
reliability of tests that use interval or
ratio scales.
EVALUATING THE RELIABILITY OF A
TEST
• THE MAGNITUDE OF THE
RELIABILITY COEFFICIENT
- To evaluate the coefficient, you can
compare it with reliability
coefficients typically obtained for
similar types of tests.
- For example, if you will be using the
test for managers, but the reliability
coefficient in the test manual was
established with high school
students, you would have less
confidence that the reliability
coefficient would generalize well to
your organization.
• VALIDITY
- Is the degree to which inferences
from scores on tests or
evidence. As with reliability, a test
must be valid to be useful. But just
because a test is reliable does not
mean it is valid.
5 COMMON STRATEGIES TO
INVESTIGATE THE VALIDITY OF SCORES
ON A TEST:
1. Content Validity
2. Criterion Validity
3. Construct Validity
4. Face Validity
5. Known-Group Validity
CHOOSING A WAY TO MEASURE
VALIDITY
• NEXT-DOOR NEIGBOUR RULE
• GANDY CRITICAL THINKING TEST
- A test itself can never be valid when
we speak of validity, we are speaking
about the validity of the test scores as
they relate to a particular job
• FACE VALIDITY
- extent to which a test appears to be
job related this perception is
important because if a test or its
items do not appear valid that
doesn't a curse and administrators
will not have confidence in the results
• BARNUM STATEMENTS
- such as those used in astrological
forecast that are so general that they
can be true of almost anyone.
 FINDING RELIABILITY AND VALIDITY
INFORMATION
• MENTAL MEASUREMENTS
YEARBOOK (MMY)
- Contains information about the
reliability
• TESTS IN PRINT VIII
- Another excellent source of
information.
COST-EFFICIENCY
- If two or more tests have similar
validities, then cost should be
considered.
- A particular test is usually designed
to be administered either to
individual applicants or to a group of
applicants.
- an increasing number of
organization or administering their
test over the internet or at remote
testing locations. In computer-
assisted testing an applicant takes a
test online computers course as a
result of the test and interpretation
or a media available
- this increase in efficiency does not
come at the cause of decreased
validity because as mentioned
previously test administered
electronically seem to yield results
similar to those administered
through the traditional paper and
pencil format
• COMPUTER-ADAPTIVE TESTING
(CAT)
- a type of test taken on a computer
in which the computer adapts the
difficulty level of questions asked to
the test taker's success in answering
previous questions.
ESTABLISHING THE USEFULNESS OF A
SELECTION DEVICE
• TAYLOR-RUSSELL TABLES
- a series of tables based on the
selection ratio, base rate, and believe
that yield information about the
percentage of future employees who
will be successful if a particular test is
used
- designed to estimate the percentage
of future employees who will be
successful on the job if an
organization uses a particular test.
• INFORMATION NEEDED FOR
TAYLOR-RUSSELL TABLES
- Criterion Validity Co-efficient
- Selection Ratio
- Base rate
BASE RATE IS USUALLY OBTAINED IN 1
OF 2 WAYS
• 1ST METHOD
- employees are split into two equal
groups based on their scores on
some criteria such as tenure or
performance.
• 2ND METHOD
- choose a criterion measure score
above which all employees are
considered successful
- after the validity selection, ratio, and
base rate figures have been obtained
the Taylor-Russell tables are
consulted.
PROPORTION OF CORRECT DECISIONS
- The proportion of correct decisions is
a fundamental concept in assessing
the effectiveness of selection
techniques in the context of
employee hiring or promotion.
- This metric measures the accuracy of
a selection method in correctly
identifying the individuals who are
the best fit for a job or role.
- In essence, it quantifies the
percentage of times a selection
procedure correctly identifies
qualified candidates, thereby
reducing the likelihood of hiring or
promoting individuals who are not
suited for the position.
LAWSHE TABLES
- Lawshe Tables are a statistical tool
used to evaluate the validity of
selection procedures, such as
interviews, tests, or assessments.
- These tables provide a systematic
approach to assessing the
relationship between a selection
technique and a specific job
performance criterion.
- By using Lawshe Tables, organizations
can determine the effectiveness of
their selection methods in predicting
job success. It's a structured way to
gauge the validity and reliability of
these techniques.
BROGDEN-CRONBACH-GLESER UTILITY
FORMULA
- a mathematical approach for
evaluating the utility or effectiveness
of a selection procedure.
- It considers the costs, benefits, and
probabilities associated with various
selection decisions, helping
organizations make informed choices
based on a cost-benefit analysis.
- This formula assists in optimizing the
selection process by balancing the
trade-off between hiring the right
candidates and managing the costs
and potential risks involved.
 DETERMINING THE FAIRNESS OF A • SINGLE-GROUP VALIDITY • MULTIPLE REGRESSION
TEST - The characteristic of a test that - A statistical procedure in which the
significantly predicts a criterion for scores from more than one criterion-
- Once a test has been determined to
one class of people but not for valid test are weighted according to
be reliable and valid and to have
another how well each test score predicts the
utility for an organization, the next
- If a test accurately predicts job criterion.
step is to ensure that the test is fair
performance for engineers but not - If we want to predict how well
and unbiased.
for salespeople, it shows single- someone will do in a job, we can use
group validity. their math and language test scores
• BIAS OR UNBIASED
together to get a more accurate
- the technical aspects of a test
• DIFFERENTIAL VALIDITY prediction.
- The characteristic of a test that
• FAIRNESS
significantly predicts a criterion for • UNADJUSTED TOP-DOWN
- can include bias, but also includes
two groups, such as both minorities SELECTION
political and social issues
and non-minorities, but predicts - Selecting applicants in straight rank
significantly better for one of the two order of their test scores.
• MEASUREMENT BIAS
groups - If we have test scores ranging from
- Group differences in test scores that
- If a test predicts job success for both 100 to 50, we would hire the person
are unrelated to the construct being
men and women, but it's much more with the score of 100 first, then the
measured.
accurate for men, it shows one with 99, and so on, in descending
- If a test has words favoring one race
differential validity. order.
over another, but those words
aren't relevant to the job, the test
• PERCEPTION OF FAIRNESS HELD BY
could be seen as biased and unfair
THE TEST TAKER • COMPENSATORY APPROACH
for that job.
- That is, a test may not have - A method of making selection
measurement or predictive bias, but decisions in which a high score on
• ADVERSE IMPACT
applicants might perceive the test one test can compensate for a low
- An employment practice that
itself or the way in which the test is score on another test.
results in members of a protected
administered as not being fair. - A high general weighted average
class being negatively affected at a
- If they believe the test is too long or might compensate for a low graduate
higher rate than members of the
that it's administered in a confusing record examination score.
majority class. Adverse impact is
way, their perception of fairness can
usually determined by the four-
be affected. • RULE OF THREE
fifths rule
- A variation on top-down selection in
- If 80% of men are hired for a job, but
which the names of the top three
only 40% of women are hired, it MAKING THE HIRING DECISION
applicants are given to a hiring
could indicate adverse impact
- After valid and fair selection tests authority who can then select any of
against women in the hiring
have been administered to a group of the three.
process.
applicants, a final decision must be - Giving the hiring manager the top
made as to which applicant or three candidates to choose from,
• PREDICTIVE BIAS
applicants to hire. At first, this may adding a bit of flexibility to the
- A situation in which the predicted
seem to be an easy decision—hire selection process.
level of job success falsely favors
the applicants with the highest test
one group over another.
scores. But the decision becomes • PASSING SCORE
- If a test wrongly predicts that men
more complicated as both the - The minimum test score that an
are more likely to succeed in a job
number and variety of tests increase. applicant must achieve to be
when that's not true, it's predictive
considered for hire.
bias.
- If a job requires a passing score of 70,
anyone scoring below that won't be
considered.
• MULTIPLE CUT-OFF APPROACH
- A selection strategy in which
applicants must meet or exceed the
passing score on more than one
selection test
- To get a job, one might need to do
well on both a math test and a
communication test, not just one of
them.

• MULTIPLE HURDLE APPROACH

- A selection practice of
administering one test at a time so
that applicants must pass that test
before being allowed to take the
next test.
- Passing a fitness test to get to the
interview stage in firefighter hiring

• BANDING
- A statistical technique based on the
standard error of measurement that
allows similar test scores to be
grouped.
- Test scores that are very close to
each other, like 85 and 86, can be
considered in the same group
because of the standard error of
measurement.

• STANDARD ERROR
- Used to determine how many points
should the applicants have to say
that their test scores are
significantly different.
- If two applicants have test scores
that differ by more than the
standard error, we can say their
scores are significantly different

• STANDARD ERROR OF
MEASUREMENT (SEM)
- The number of points that a test
score could be off due to test
unreliability.
- If a test has a SEM of 2 points, a
score of 80 could actually be
anywhere from 78 to 82 due to the
test's unreliability.
How does the length of a test generally
relate to its internal consistency?
- Longer tests tend to have higher
internal consistency.
What does single-group validity
assess?
- Whether the test will
significantly predict
performance for one group and
not others.
What is content validity concerned
with?
- The extent to which test items
sample the content they are
supposed to measure.
If a test lacks either form stability or
temporal stability, what method is
needed to determine the cause of the
unreliability?
- Test-retest reliability
What does criterion validity measure?
- The relationship between a test
score and some measure of job
performance.
How is adverse impact determined in
the context of testing?
- Both a and b.
Which type of anxiety, according to the
passage, is important to have temporal
stability for a test to be useful?
- State anxiety
What are the four characteristics of
effective selection techniques
mentioned in the book?
- Reliable, valid, cost-
efficient, legally defensible
What is adverse impact in testing?
- A statistically significant
difference in selection rates
between groups.
Which method involves splitting test
items into two groups, usually odd-
numbered and even-numbered items,
and correlating the scores on these
groups to determine internal
consistency?
- Split-half method
What is the purpose of the alternate-
forms reliability method?
- To eliminate effects of test-
taking order
How does differential validity differ
from single-group validity?
- Single-group validity is concerned
with predicting performance for
one group, while differential
validity involves the test being
more valid for one group than the
other.
What does the term "bias" in the
context of testing refer to?
- Group differences in test scores
unrelated to the construct being
measured.
In the context of test-retest reliability,
why is it important to have a time
interval between the two test
administrations?
- To reduce the potential advantage
to individuals who take the test a
second time
What is the main focus of construct
validity?
- Inferences about test construction.
In a predictive validity design, how is
criterion validity established?
- By correlating test scores with a
future measure of job
performance
What is the relationship between
reliability and validity?
- Reliability is necessary for
validity, but having reliability
does not guarantee validity.
Which characteristic is described as the
extent to which a score from a
selection measure is stable and free
from error?
- Reliability
What does internal consistency
measure?
- The consistency with which an
applicant responds to items
measuring a similar construct.
Which of the following terms refers to
a method that involves using a
computer program to calculate
internal reliability?
- Coefficient alpha
What is the impact of item
homogeneity on internal consistency?
- Higher item homogeneity leads
to higher internal consistency.
How is the test-retest reliability
determined?
- By counterbalancing test-taking
order