CHAPTER 7 – UTILITY

What Is Utility?
 in the context of testing and assessment it refers to the usefulness or practical value of testing to improve efficiency
 is also used to refer to the usefulness or practical value of a training program or intervention.

Factors That Affect a Test’s Utility

A number of considerations are involved in making a judgment about the utility of a test. Here we will review how a test’s
psychometric soundness, costs, and benefits can all affect a judgment concerning a test’s utility.

1. Psychometric soundness
 A test is said to be psychometrically sound for a particular purpose if reliability and validity coefficients are acceptably
high
 How can an index of utility be distinguished from an index of reliability or validity
Index of Reliability and Validity: The short answer to that question is as follows: An index of reliability can tell us
something about how consistently a test measures what it measures; and an index of validity can tell us something about
whether a test measures what it purports to measure.

Index of Utility: But an index of utility can tell us something about the practical value of the information derived from
scores on the test. Test scores are said to have utility if their use in a particular situation helps us to make better decisions
—better, that is, in the sense of being

Note:
 It was noted that reliability sets a ceiling on validity
 It is tempting to draw the conclusion that a comparable relationship exists between validity and utility and conclude that
“validity sets a ceiling on utility.”
 Generally speaking, the higher the criterion-related validity of test scores for making a particular decision, the higher
the utility of the test is likely to be. However, there are exceptions to this general rule. This is so because many
factors may enter into an estimate of a test’s utility, and there are great variations in the ways in which the utility of a
test is determined.
 Would it be accurate to conclude that “a valid test is a useful test”? the answer is no; it is not the case that “a valid
test is a useful test.” A test may be psychometrically sound, but it may have little utility—particularly if the
targeted test takers demonstrate a tendency to “bend, fold, spindle, mutilate, destroy, tamper with,” or otherwise fail to
scrupulously follow the test’s directions not

2. Costs
 In considerations of test utility, factors variously referred to as or economic, financial, or budget-related in nature
must certainly be taken into account budget-related.
 In fact, one of the most basic elements in any utility analysis is the financial cost of the selection device (or training
program or clinical intervention) under study. However, utility goes beyond money.
 Briefly, cost in the context of test utility refers to disadvantages, losses, or expenses in both economic and noneconomic
terms
As used with respect to test utility decisions:
 the term costs can be interpreted in the traditional, economic sense; that is, relating to expenditures associated with
testing or not testing.
 If testing is to be conducted, then it may be necessary to allocate funds to purchase (1) a particular test, (2) a supply of
blank test protocols, and (3) computerized test processing, scoring, and interpretation from the test publisher or
some independent service.
Associated costs of testing may come in the form of:
(1) payment to professional personnel and staff associated with test administration, scoring, and interpretation,
(2) facility rental, mortgage, and/or other charges related to the usage of the test facility, and
(3) insurance, legal, accounting, licensing, and other routine costs of doing business.
Note:
 The economic costs listed here are the easy ones to calculate. Not so easy to calculate are other economic costs,
particularly those associated with not testing or testing with an instrument that turns out to be ineffective.

3. Benefits
 Judgments regarding the utility of a test may take into account whether the benefits of testing justify the costs of
administering, scoring, and interpreting the test. So, when evaluating the utility of a particular test, an evaluation is
made of the costs incurred by testing as compared to the benefits accrued from testing. Here, benefit refers to profits,
gains, or advantages.
 There are also many potential noneconomic benefits to be derived from thoughtfully designed and well-run
testing programs. In industrial settings, a partial list of such noneconomic benefits—many carrying with them
economic benefits as well—would include:
 an increase in the quality of workers’ performance
 an increase in the quantity of workers’ performance;
 a decrease in the time needed to train workers;
 a reduction in the number of accidents;
 a reduction in worker turnover.
 EXAMPLE OF NONECONOMIC BENEFIT:
The cost of administering tests can be well worth it if the result is certain noneconomic benefits, such as a good
work environment. As an example, consider the admissions program in place at most universities.
 Appropriate selection of student applicants per program where their graduates succeed at their chosen careers.
 As a result, successful graduates enhance the university’s reputation and sends the message that the university
is doing something right.
 Other related benefits to a university:
- students who are successfully going through its programs may include high morale and a good
learning environment for students,
- high morale of and a good work environment for the faculty, and
- reduced load on counselors and on disciplinary personnel and boards.
 A good work environment and a good learning environment are not necessarily things that money can buy.

 EXAMPLE OF ECONOMIC BENEFIT:

Diagnostic test used to make decisions about involuntary hospitalization of psychiatric patients is a benefit to society
at large
 Tools of psychological assessment such as tests, case history data, and interviews may be used to make a
decision regarding involuntary psychiatric hospitalization.
 The more useful such tools of assessment are, the safer society will be from individuals intent on inflicting
harm or injury.

What Is a Utility Analysis?

 defined as a family of techniques that entail a cost–benefit analysis designed to yield information relevant to a
decision about the usefulness and/or practical value of a tool of assessment
 “family of techniques”
- because a utility analysis is not one specific technique used for one specific objective.
- is an umbrella term covering various possible methods, each requiring various kinds of data to be
inputted and yielding various kinds of output.

 In a most general sense, a utility analysis may be undertaken for the purpose of evaluating whether the benefits of
using a test (or training program or intervention) outweigh the costs.
 If undertaken to evaluate a test, the utility analysis will help make decisions regarding whether
 one test is preferable to another test for use for a specific purpose;
 one tool of assessment (such as a test) is preferable to another tool of assessment (such as behavioral
observation) for a specific purpose;
 the addition of one or more tests (or other tools of assessment) to one or more tests (or other tools of
assessment) that are already in use is preferable for a specific purpose;
 no testing or assessment is preferable to any testing or assessment
 If undertaken for the purpose of evaluating a training program or intervention , the utility analysis will help
make decisions regarding whether:
 one training program is preferable to another training program;
 one method of intervention is preferable to another method of intervention;
  the addition or subtraction of elements to an existing training program improves the overall training
program by making it more effective and efficient;
 the addition or subtraction of elements to an existing method of intervention improves the overall
intervention by making it more effective and efficient;
 no training program is preferable to a given training program;
 no intervention is preferable to a given intervention.

 The endpoint of a utility analysis is typically an educated decision about which of many possible courses of action
is optimal.

How Is a Utility Analysis Conducted?

Expectancy Data
 Expectancy Table: An expectancy table can provide an indication of the likelihood that a test taker will score
within some interval of scores on a criterion measure—an interval that may be categorized as “passing,”
“acceptable,” or “failing.”
 For example, with regard to the utility of a new and experimental personnel test in a corporate setting, an
expectancy table can provide vital information to decision-makers. An expectancy table might indicate, for
example, that the higher a worker’s score is on this new test, the greater the probability that the worker will be
judged successful.
 In other words, the test is working as it should and, by instituting this new test on a permanent basis, the company
could reasonably expect to improve its productivity.