Ian Koh (4230663) MSc Thesis, Methodology and Statistics

Intended journal: Frontiers in Psychology Utrecht University, The Netherlands

Test-Taking Heuristics and Biases as Potential Threats

to Test Score Validity
Ian Koh
Utrecht University–Cito

Gunter Maris Frederik Coomans

Cito–University of Amsterdam University of Amsterdam–KU Leuven

In their attempt to achieve maximal test performance, students behave in ways that they
perceive to be in line with their goal. However, their behaviour is guided by heuristics and
biases, an interaction of which leads to either a beneficial or detrimental outcome. We col-
lectively label heuristics and biases ‘test strategies’, explaining how these influence students’
test behaviour for better or worse. For one of these strategies, we also provide empirical
evidence from the 2009 edition of the Programme for International Student Assessment (?) to
support our argument. We then present a new approach to test scoring that aims to uncover
these strategies as much as possible to give us new insights about students. This also has the
consequence of preserving test score validity.

Keywords: educational testing, test score validity, psychometrics, behavioural economics,

scoring rules

Introduction tutions, but they do make it more challenging for students

Many people would have had the experience of taking a
standardised test at some point in their lives. Whether for
university admission, determining which academic track to
take in secondary school or even to qualify for a driving
licence, standardised tests are a staple feature of modern
living.
Pertinent to the idea of testing is validity, which the Cam-
bridge Dictionary of Statistics (?, p. 444) defines as ‘the
extent to which a measuring instrument is measuring what
was intended’. Tests can therefore be thought of as instru-
ments, where the object being measured is one’s ability in
some subject matter. In the area of K-12 education, this
means assessing students on subjects such as mathematics,
science, language and so on, with the aim of getting an idea
of how well students have mastered their academic material.
The outcome of such assessments tends to be summarised
as one piece of information: the test score. This information
has a potentially big influence on a student’s academic path.
This is particularly so in countries with educational sys-
tems that use tracking (as in the Netherlands), and countries
where the prevailing culture places a high premium on one’s
level of education (as in the East Asian countries).
Good test scores open options for students, allowing them
access to better schools, academic instruction and educa-
tional outcomes. Less satisfactory scores do not necessarily
result in students being placed in ‘inferior’ educational insti-
With many thanks to Maria Bolsinova and Maarten Marsman of
the University of Amsterdam; Timo Bechger, Daniël van der Palm
and Ivailo Partchev of Cito Institute for Educational Measurement,
Arnhem; and Matthieu Brinkhuis and Sarah Kuipers. I am grateful
for your insightful comments, discussions and assistance.
to obtain the same educational outcomes as their higher-
scoring peers.
Furthermore, in East Asian countries, where societies
have deep roots in Confucian thinking and place great em-
phasis on education, students’ test scores could result in
either prestige or denigration. While this might seem trivial
to a casual Western observer, test scores can be – and often
are – sources of great stress and personal dissatisfaction to
a typical East Asian student.
This being the case, then, it is not difficult to appreciate
the importance of a test score being valid. Such a score
must accurately reflect a student’s ability, because further
decisions concerning a student are made that rely on this
piece of information. These decisions have cascading effects
that extend far beyond one’s schooling years.
In order to ensure test score validity, one must first
identify and address threats to said validity. Two such ex-
amples are heuristics and biases, which feature in our every-
day decision making (see, for example, ?). These apply to
testing situations as well, and are very much involved in
influencing the actions that students take while answering
questions.
Heuristics help to simplify our surroundings so that de-
cision making becomes easier. Ideally, this helps us make
better choices, such as deciding on how best to answer a
test question. However, their effects are also influenced by
our biases, which reflect a certain way of thinking. The net
effect of this tug-of-war could be beneficial or detrimental.
To us, though, we are only trying to make the best of our
situation. This means that we will perceive this net effect as
something positive, even if it is not the case in reality.
And so it is with test behaviour. Students perceive their
actions on tests as beneficial to their scores, regardless of
whether their heuristics and biases that guide these actions

1
2 KOH, MARIS AND COOMANS
result in a helpful or hindering outcome. This outcome
can lead to very different implications for their final scores.
Because students do the best they can to maximise their
scores, and because their actions are guided by heuristics
and biases, we have decided to collapse these two terms into
one for convenience’s sake. From this point onwards, we
shall refer to them collectively as test strategies.
We now present two illustrative examples to sketch a pic-
ture of how strategies can lead to differential test outcomes.
Two Illustrations of Test Strategy Use
Consider two students with identical test scores. One
has employed a set of adaptive test strategies while going
through the questions, especially with regards to those ques-
tions to which the answer was not known. The other has
not used these strategies (or, at least, has used them to a
much lesser extent). This means that the latter’s test score
is (almost) fully reflective of ability in the tested subject. A
casual observer cannot therefore conclude that both students
are comparable in terms of their ability, because one of them
has achieved this test score with strategic aids, in addition
to ability.
On the flip side of the coin, consider two students with an
identical ability level on some academic subject. One uses
adaptive test strategies and the other does not, resulting in
them getting different test scores. An observer who looks
at both students’ scores would likely conclude that they are
not comparable in terms of academic ability, even though
this is, in reality, the case.
What these examples demonstrate is that test strategies
have the potential to distort how comparable students’ test
scores are when some students use them and others do not.
In other words, these strategies are potential threats to test
score validity.
Examples of Test Strategies
Before continuing, it is helpful to define the boundaries
of our discussion. We shall limit our discussion to multiple-
choice questions (MCQs). This is a test format that will be
familiar to most people: for each question, there are several
alternatives to choose from as the correct answer(s). We
further limit our discussion by only considering the case
where there is one correct answer to an MCQ, since there
are variations that involve selecting more than one option.
(For tests that are marked by machine, this is reflected in the
shading of multiple lozenges on the optical response sheet.)
Three test strategies come to mind: answer guessing,
answer switching and answer elimination. In a multiple-
choice context, guessing and switching are self-explanatory.
The former involves picking an answer from among the
alternatives (either at random or through some systematic
selection process, like elimination), while the latter involves
going back to a previously-answered question and altering
the response.
Answer elimination involves examining the given altern-
atives and crossing out those alternatives that are definitely
incorrect. This can be used in a situation where one does not
know what the correct answer is (in conjunction with the
guessing strategy), or in a situation where one knows that
a subset of the alternatives contains the correct answer but
does not know which option (within the subset) to select.
Characteristics Influencing Test Strategy Use
We now have some idea of what test strategies are, but
what influences students to use them differentially? We
believe there are three characteristics of students that influ-
ence whether and how they use test strategies, at least on a
multiple-choice test. These are (a) degree of confidence, (b)
representation bias and (c) risk aversion. Figure 1 is a schem-
atic diagram depicting how these student characteristics
relate to test strategies and each other.
We devote the rest of the thesis to examining the above
characteristics, providing empirical results as supporting
evidence for representation bias. As we better understand
how they influence students’ use of strategies, we will be
more able to see how these strategies threaten test score
validity.
Confidence Levels and Probability Distributions
We begin our discussion by introducing the idea of stu-
dents’ levels of confidence when they attempt MCQs. Every
test taker approaches a question with some degree of con-
fidence attached to each option. This refers to how sure one
is that a given option is correct compared to the others, and
is influenced by prior content knowledge.
Ideally, students would have prepared before taking a test,
but in reality not everyone does so. Even among those who
do, there may be gaps in knowledge. The act of evaluating
the response options to select the best one exposes these
gaps. For students who studied and therefore have much
content knowledge, discriminating among the options is
usually trivial. For students with lesser amounts of con-
tent knowledge, discriminating is much more challenging,
if at all possible. The degree of confidence attached to each
option will differ between the students with more content
knowledge and those with less.
At this point, we should stress that the confidence level we
mention in this thesis is not the same as the one used in null-
hypothesis significance testing. Our confidence levels relate
strictly to students’ subjective perceptions of which response
option most appropriately answers the test question.
It is useful to characterise a student’s level of confidence
meaningfully. Such a characterisation would reflect the posi-
tion of the student’s internal state between being completely
confident and completely unconfident. Thankfully, such a
metric already exists in the form of probability values. In
fact, one’s confidence level can be thought of as a subjective
probability – the more confident about a response option
one is, the more probable it is that the option is the correct
answer. (We shall use confidence levels and probabilities
interchangeably.)
Probabilities contain all available information about a
student’s choices and can be decomposed into two further
sources of information, known as ranks and order statistics.
In doing this, we have more insightful pieces of information
that give us a richer understanding of students’ thought
processes while answering MCQs.
 EXAM STRATEGIES AND TEST SCORE VALIDITY 3

Prior content
knowledge

Influences

How many response

options are eliminated
Ranks and Level of
order statistics confidence
Influences

Probability
distributions
Risk aversion

Fixed Varying
distributions distributions
Probability
values/weights
How might test
scores be affected
by strategy?
Learn about
Varying students’
prospects
Representation
bias Introduce elimin-
ation scoring (ES)

FS RF
How might test How do we learn what
scores be affected the ranks and order
Omit Guess by strategy? statistics for these
distributions are?

Explained
using

Differential
Prospect
switching
theory

Empirical
evidence

Figure 1. Schematic of thesis, started from the node labelled ‘Prior content knowledge’, showing student characteristics (level
of confidence, representation bias and risk aversion) related to the test strategies students use.

Introducing Ranks and Order Statistics can generically write that a student will have X 1 ,X 2 , . . . ,X c
as his or her confidence levels over all the options. The
Ranks and order statistics feature throughout the thesis, constant c refers to the total number of response options
and are themselves important concepts in the area of non- available.
parametric statistics. We explain what they are and how Ranks denote the order of preference that a particular
they work, taking reference from the first chapter of ?. After response option is associated with. Compactly, we write
defining ranks and order statistics, we put them into context
R(m) = k, (1)
with an example.
Let us first agree that, in an MCQ, a student will have vary- where
ing levels of confidence attached to each response option.
1 ≤ m ≤ c and 1 ≤ k ≤ c, and
(We discussed this in the previous subsection.) We do not
know what the values of these confidence levels are, so we m ∈ Z+ and k ∈ Z+ .
4 KOH, MARIS AND COOMANS
Table 1
An Example of a Student’s Ranks and Order Statistics
m π r o
1 0.15 2 0.10
2 0.45 4 0.15
3 0.30 3 0.30
4 0.10 1 0.45
We now unpack the above in words. The variables m
and k can take any positive integer value between 1 and c
inclusive, meaning that they are whole numbers. But what
do m and k mean? Perhaps a written explanation of R(m) = k
will be instructive. We read the expression as ‘the rank of
a response option, with index m, is the value k’. Hence,
every response option has a rank. We defer the process of
computing this rank to the example.
While ranks tell us about how preferred a response option
is in relation to the other options, order statistics are simply
the confidence levels themselves. However, there is a specific
way of displaying them (that is, in increasing order), so we
would express a student’s order statistics as
O 1,c ≤ O 2,c ≤ · · · ≤ O c,c .
The above is understood as a series of order statistics (or
variational series), while the individual elements of the series
are understood to be the order statistics.
At this point, we motivate our definitions with an ex-
ample. Consider a student attempting a four-option MCQ.
Having read through the question and the possible options,
the student generates the following information as seen in
Table 1. In the table, we write the ranks and order statistics as
lower-case r and o since these are realisations of the random
variables R and O. The symbol π denotes the confidence
levels, also known as (subjective) probabilities. The order
statistics in the column o are displayed per Equation 2.
The values of π were arbitrarily generated for this ex-
ample, but how did we compute the values for the r column?
The formula for calculating a response option’s rank is
c
X of information.
R(m) = 1 {m ≥k }
k=1
where, in our example, c = 4. The term 1 {m ≥k } is an indicator
function that outputs a value 1 when the probability π associ-
ated with m is indeed greater than or equal to that associated
with k, and 0 otherwise. Simply put, we are summing up
values of 1 and 0 to obtain the rank of our response option
of interest, which we index as m. Working out the ranks of
every response option, we have
R(1) = 1 + 0 + 0 + 1 = 2,
R(2) = 1 + 1 + 1 + 1 = 4,
R(3) = 1 + 0 + 1 + 1 = 3 and
R(4) = 0 + 0 + 0 + 1 = 1,
which turn out to be the same values as seen in Table 1.
The reader will notice that option 2, with the largest
confidence level of 0.45, has a rank of 4, while option 4, with
the smallest confidence level of 0.10, has a rank of 1. Going
back to our verbal definition of a rank, this is rather non-
intuitive as what we have here is a reverse order of prefer-
ence. Nonetheless, this convention is used for mathematical
convenience and will be better appreciated in subsequent
works focusing on an axiomatic development of our content.
Properties of Ranks and Order Statistics
An idea that follows is that ranks and order statistics
correspond to each other. We write this as
{R(m) = k} ↔ {O k,c = Xm }, (4)
and in our example we have
{2, 4, 3, 1} ↔ {0.15, 0.45, 0.30, 0.10}.
The curly braces in Equation 4 indicate the set of rank values
(left-hand side) and the set of order statistics that are also
the confidence levels (right-hand side).
One might think of this as looking at the r and o column
vectors in Table 1 and drawing a line from an element in r to
(2) a matching element in o. Every order statistic is associated
with a rank, depending on the former’s numerical value (that
is, X k ). Despite this association, it would be erroneous to
think that knowing the values in either r or o would allow
one to calculate the values in the other column.
Focusing only on the r column, it is easy to see why know-
ing the ranks tells one nothing about the values that the
order statistics will take. The o column, however, presents a
slightly trickier case. One might assume, since the o values
are known, that ranks can be constructed from these values.
Equation 1 shows why this is not possible. To calculate
the rank, one needs more than just the confidence levels –
the index (in m) of the response option in question and the
confidence level (in π ) associated with that index also need
to be known. As such, knowing the order statistics also tells
one nothing about what the ranks will be.
To summarise, we have it that ranks and order statistics
(a) correspond to each other and (b) are independent pieces
(3)
Inverse Ranks
In practice, we are more interested in a specific response
option than that option’s rank. Being able to write the re-
sponse option’s index, while still having access to its rank
information, is therefore useful. Fortunately, we can do this
by expressing ranks as inverse ranks (or anti-ranks). Where
a rank is defined as R(m) = k, the inverse rank is defined as
R −1 (k ) = m. (5)
Using Equation 1, we can also write Equation 5 as
R −1 (R(m)) = m. (6)
Equations 5 and 6 tell us nothing new over and above
Equation 1. We still know that a response option with index
 EXAM STRATEGIES AND TEST SCORE VALIDITY
m has a rank of value k – all we have done is reorganise
our existing information. We can now refer to any option
in terms of its inverse rank, without needing to know what
that option’s specific index is. In later examples, this shall
prove highly convenient.
The Different Phenomena Associated With Ranks
and Order Statistics
Over the course of the thesis, several testing phenomena
will be mentioned. These refer to test-taking strategies, scor-
ing rules and student characteristics. The reader is invited
to refer to Figure 1 for a brief overview of where these phe-
nomena lie in relation to each other and to the grand scheme
of the thesis. Terms that are presently unfamiliar will be
explained as we progress through the thesis.
We wish to end our brief introduction to ranks and order
statistics by stating that different phenomena relate to these
pieces of information in varying degrees. Hence, different
issues in the testing process are useful for giving us specific
information about either ranks or order statistics. We illus-
trate the associations between test phenomena and ranks
and order statistics in Figure 2.
From the figure, there are two ways to obtain informa-
tion about ranks and order statistics. The first is through
students’ levels of confidence, denoted by π . However, this
is purely hypothetical as one cannot realistically expect stu-
dents to precisely and accurately quantify the degree of con-
fidence they attach to every response option. By imposing
an elimination scoring rule (which will be explained in a
subsequent section), the second method presents a more
plausible method of learning about ranks and order statist-
ics. Elimination scoring also lets us learn about students’
levels of risk aversion, which then tell us something about
how willing they are to guess on a question (pertinent to a
formula scoring scenario).
Looking at ranks and order statistics separately, the
former are associated with the number-right (NR) and for-
mula (FS) scoring rules. The latter are associated with answer
guessing and switching (both test strategies), and overcon-
fidence (a student characteristic).
Fixed and Varying Probability Distributions
Having established the relevance of ranks and order stat-
istics to this thesis, we return to our earlier discussion of
probability distributions. A probability distribution can take
two forms: fixed and varying.
A fixed distribution is essentially a uniform distribution,
so that all options have the same probability value – or con-
fidence level – attached to them. Ranks and order statistics
are of little use to us in this instance, since they would all be
identical. They are therefore not featured when we discuss
fixed distributions.
A varying distribution is one where the probability val-
ues vary across the options, and also over students. We first
examine the fixed distribution before moving to the varying
distribution.
5
Fixed Probability Distributions
Consider three students who are answering a four-
alternative MCQ in some test. For argument’s sake, assume
that the test assesses them on a subject that they know noth-
ing about. As a result, each student is equally confident of
each presented alternative.
πAdam = (0.25, 0.25, 0.25, 0.25)
π Beth = (0.25, 0.25, 0.25, 0.25)
πCate = (0.25, 0.25, 0.25, 0.25)
The above represents a ‘probability distribution’ of confid-
ence levels over the available alternatives. Their respective
probability distributions are denoted by π .
Put differently, Adam, Beth and Cate all have the same
probability of getting the correct answer. Yet, after they
answer the question, the following results are obtained:
Adam: Correct response
Beth: Incorrect response
Cate: Omitted response (marked incorrect)
What could have resulted in the differences in their re-
sponses? Why did Cate choose to leave her answer blank?
We contend that these differences in responses can be traced
to differences in the degree of representation bias among
the students.
Representation Bias and Scoring Rules
Before diving into the matter at hand, we first take a
step back to explain what we mean by ‘representation’. The
term refers to a student’s personal schema of how much
credit is awarded to a question response. In answering a
multiple-choice question, one of three outcomes is possible:
the given response is correct, incorrect or omitted (that is,
skipped). The amount of credit awarded to each of these
three outcomes will depend on the instructions specified
by the test-maker. When a student’s personal schema is at
odds with the ‘test-specified schema’, we have an instance
of representation bias.
This test-specified schema is known as a scoring rule. It
determines how much credit a student receives for a correct,
incorrect or skipped response, which is particularly pertin-
ent to the latter two responses. Two examples of scoring
rules should help illustrate the issue.
Number-Right Scoring
In the commonest and simplest instance of a scoring rule,
students receive maximum credit for a correct response and
zero credit for an incorrect or a skipped response. This
scoring rule is known as number-right scoring (NR scoring).
When such a rule is used, the probability of receiving credit
for a skipped response is zero, whereas for a submitted re-
sponse, the probability is 1/c, where c indicates the number
of available choices (or alternatives). Hence, for a question
that one has difficulty with, it makes sense to guess, since
there is a higher probability of getting the question correct
compared to leaving a blank.
6 KOH, MARIS AND COOMANS
π
NR
R O
FS
Risk Elimination
aversion scoring
Figure 2. Different phenomena associated with ranks and order statistics. π refers to students’ levels of confidence, which
are expressed as probability values.
Consider, for instance, the case of a multiple-choice test
where there are four alternatives within each question. The
probabilities of receiving credit for either guessing or omit-
ting an answer are shown below.
Guess : Pr(correct) = 0.25, Pr(incorrect) = 0.75
Skip : Pr(correct) = 0, Pr(incorrect) = 1
In a situation where a student has absolutely no idea what
the correct answer is, the above shows that the most rational
course of action is to guess.
Of course, while guessing is good for students, it is a nuis-
ance for test markers. A student’s ability cannot be accur-
ately assessed if he or she makes use of guessing, particularly
so if the guesses result in correct answers. With guessing, a
student’s test score will reflect both ability and the positive
effects of having used the test strategy – basically, the test
score is higher than it should have been.
Having a scoring rule that helps to keep guessing be-
haviour in check might therefore be useful. This is where
formula scoring comes in.
Formula Scoring
Formula scoring (FS scoring) is a common alternative to
NR scoring. In this instance, one receives maximum credit
for a correct response, partial (or zero) credit for an omitted
response and zero (or negative) credit for an incorrect re-
sponse. As an example, a correct response receives 1 point; a
skipped response, 0 points; and an incorrect response, −0.33
points.
Formula scoring is set up to penalise guessing: It forces
students to think twice about making a hasty response to a
question to which they do not know the answer. As such, a
student answering a formula-scored test question will have
to carefully consider his or her degree of knowledge – or lack
thereof – before deciding what kind of response to make.
That incorrect responses are penalised constitutes a ‘stick’
variant of formula scoring. A ‘carrot’ variant also exists
where, in contrast to the earlier variant, the focus is on
rewarding students for omitted answers (see ?). Sticking
with the example, a correct response receives 1 point, but
now a skipped response might receive 0.33 points and an
incorrect response, 0 points.
Guessing
Switching
Overconfidence
Regardless of reward or penalty, the underlying logic
remains the same: In FS scoring, incorrect responses always
receive less credit than omitted ones. This thesis uses Traub
and colleagues’ version of the FS rule (that is, the ‘carrot’
variant).
Distinguishing Between Number-Right and Formula
Scoring
We can summarise the NR and FS scoring rules as follows.
In NR scoring, if one guesses, there is a 0.25 probability
of getting a correct answer and, hence, 1 point, and a 0.75
probability of receiving 0 points (an incorrect answer). If
one skips the question, 0 points are received with certainty.
In FS scoring, if one guesses, there is a 0.25 probability of
receiving 1 point, and a 0.75 probability of receiving −0.33
points. As with NR scoring, if one skips the question, 0
points are received with certainty.
The main difference between NR and FS scoring rules is in
how these rules distinguish incorrect and omitted responses.
In the former rule, there is no difference between incorrect
and omitted responses – both yield zero credit. In the latter,
an incorrect response receives less credit than an omitted
response.
We borrow notation from expected utility theory (?) to
express this more compactly as
NR scoring :
Uguess = u (1) · 0.25 +u (0) · 0.75
> u (0) · 1 = Uskip
FS scoring :
Uguess = u (1) · 0.25 +u (−0.33) · 0.75
Uskip = u (0) · 1.
Uguess and Uskip are understood as prospects (?). A prospect is
a course of action that one could take, out of several in con-
tention. When answering a multiple-choice question, this
would be to either answer or omit the question. Whichever
course of action one chooses, there is an outcome that oc-
curs with some probability. In our context, the outcomes are
either 1 or 0 (correct or incorrect).
The letter u represents the utility – framed in terms of
a gain or loss of points on a test – of some outcome. This
 EXAM STRATEGIES AND TEST SCORE VALIDITY
Table 2
Scoring Rules and the (Whole Number) Weights, or Awarded
Credit, Associated With Them
Incorrect Omitted Correct
Number correct 0 0 2 another reason that we use the RF rule.
Formula 0 1 2
Reverse formula 1 0 2 Misrepresenting the Scoring Rule
is the benefit, or satisfaction, of correctly or incorrectly an-
swering a test question. The utility of an outcome is then
multiplied by the probability of said value occurring (here,
the probability of a correct answer is 0.25, while that of an
incorrect answer is 0.75).
For the case of NR scoring, u (0) · 0.75 = u (0) · 1 = 0. These
are the portions of the prospects that represent incorrect and
omitted answering, respectively. In FS scoring, on the other
hand, u (−0.33) · 0.75 , u (0) · 1. Depending on a test-taker’s
willingness to take a risk and guess, Uguess could either be
greater or smaller than Uskip . This reduces instances of guess-
ing due to the possibility of a penalty to one’s total test score
if one makes a wrong guess.
Reverse Formula Scoring
Having outlined the differences between the NR and FS
rules, we present a third scoring rule which we label the
reverse formula (RF) scoring rule. This rule is not used in
any test format and we use it in our thesis for purposes of
argument.
In RF scoring, omissions are strongly discouraged. This
is achieved by providing an incentive to answer a question
in the form of credit, regardless of whether the answer is
correct or incorrect. RF scoring is essentially a mirror image
of FS scoring when it comes to how incorrect and omitted
items are treated. As such, a correct response receives 1
point; an incorrect response, 0.33 points; and an omitted
response, 0 points.
Please refer to Table 2 for an overview of the scoring rules
we have just presented. Do note that unlike our examples, we
use whole numbers to represent the points that are awarded
to each response type. We do this for ease of reading. The
earlier use of 0.33, however, is not arbitrary.
The value is arrived at by using a formula that was origin-
ally implemented to calculate the penalty to impose on incor-
rect answers in FS scoring (the ‘stick’ variant). This formula
is written as 1/(c − 1) (see also ?; authors use k instead of c).
Since we have been talking about scoring rules in the context
of four-option MCQs, we have that 1/(4 − 1) = 1/3 = 0.33.
The Conceptual Similarity Between the NR and RF
Scoring Rules
The NR and RF rules are more similar than they appear to
be at first glance. In both rules, it is to students’ advantage
to guess as no penalty is imposed for doing so (unlike in
FS scoring). They only differ in that RF scoring provides a
stronger incentive to guess than NR scoring. Otherwise, the
direction of the awarded credit remains the same (that is,
7
non-zero). If one does not answer, one receives zero points
with certainty. If one answers, one may or may not receive
credit (but zero credit is no longer a certainty).
We make use of this connection when presenting our
empirical example. In the example, we will also present
As mentioned earlier, the NR scoring rule is one of the
commonest rules to be found in a testing situation, and also
one that promotes guessing behaviour. Yet, there are almost
always occurrences of answer omissions on test scripts, sug-
gesting that not all students exploit this scoring rule to its
fullest. Cate would be an example of such a student.
One explanation for this is students who are unable to
finish the test in time, thereby leaving a string of omitted
answers near the end of the test. But what about those stu-
dents who finish the test, but still omit some answers? They
might have forgotten to fill in the answers, which, plausible
though it may be, is highly unlikely to happen. We thus
posit that these students may have internally misrepresented
the scoring rule prescribed by the test-maker.
That is, these students may have internalised the FS scor-
ing rule instead of the NR rule, resulting in them being less
willing to guess on test questions. The result is that their test
scores are qualitatively different from their peers who had
no compunctions guessing the right answers. In comparing
the scores of guessing and non-guessing students, it is the
latter group who are disadvantaged as their scores are lower
than should be.
However, in the case of high ability students taking a
high-stakes test, the slightest differentiation in scores could
lead to different educational outcomes. (For example, if two
students are applying to a very competitive university study
programme and are equivalent in every way except for a
slight difference in grades, the lower-scoring student could
very well be disadvantaged.) In such a scenario, differences
in test scores due to a representation bias are undesirable.
In our initial example, if Adam, Beth and Cate are all stu-
dents of similar ability and Cate omits her answer because
she misrepresents the scoring rule, she disadvantages herself
in comparison to her answer-guessing peers. It is therefore
important to be able to detect people like Cate so that they
can compete on a level playing field with other students.
We are now aware that a given scoring rule can be misrep-
resented as a different one. How does this misrepresentation
influence a test taker tendency towards omitting answers,
when his or her other peers do not do so?
A Prospect Theory Explanation of Guessing and Omit-
ting Behaviour
When we first presented the prospects available to a stu-
dent in NR and FS scoring situations, we expressed the form
of the prospects using notation from expected utility theory
(?). This behavioural economics theory tells us what a ra-
tional person does when faced with making a choice. In this,
it is a normative model of behaviour. However, people are
not always rational, which results in them selecting courses
8 KOH, MARIS AND COOMANS
of action that are contrary to what the theory predicts. Sev-
eral violations of expected utility theory are mentioned in
?.
An alternative theory is proposed by Kahneman and Tver-
sky (??) and referred to as prospect theory. It explains why
people violate expected utility theory’s predictions and is a
descriptive model of choice behaviour. Using prospect the-
ory, we can rewrite our previous presentation of prospects
as
NR scoring :
Vguess = v (1) · w (0.25) +v (0) · w (0.75)
> v (0) · w (1) = Vskip
FS scoring :
Vguess = v (1) · w (0.25) +v (−0.33) · w (0.75)
Vskip = v (0) · w (1).
Aside from Uguess/skip turning into Vguess/skip , and u (.)
turning into v (.), notice also that the probabilities are now
weighted by the function w (.). The functions v (.) and w (.)
constitute the crux of prospect theory. The former refers
to gains and losses – in a test, this would be the amount of
credit awarded or penalised. The latter function is a decision
weight, which ‘reflects the impact of [the probability of an
outcome occurring] on the over-all value of the prospect’ (?).
(Vskip could also have been written as Vomit , but we decided
to go with the former for reasons of taste.)
Can we actually observe students who differ qualitatively
on their guessing and omitting behaviour? How might we
go about gathering evidence of this in practice?
Empirical Evidence: PISA
We sought to answer these questions by analysing
publicly-available data from the 2009 edition of the Pro-
gramme for International Student Assessment, or PISA (?).
This is an international assessment of 15-year-old students
in member and partner countries of the Organisation for
Economic Co-operation and Development (OECD), admin-
istered once every three years.
Students are tested on three domains: mathematics, read-
ing and science. Several test forms, or booklets, are available.
For PISA 2009, 21 booklets were published. We only used 20
as the last one was meant for special education students and
therefore contained items that were much easier than the
others. The items in a test booklet were a mixture of both
dichotomous and polytomous questions.
In a dichotomous item, a response is either entirely correct
or entirely incorrect. Up until now, this has been the type
of response we have been considering. A polytomous item,
on the other hand, allows for partially correct responses.
An easy example is an open-ended mathematics question,
where it is possible to obtain partial credit for one’s working
even if the final answer is incorrect. This should not be
confused with FS scoring – a response scored using this rule
is still dichotomous. Whether partial credit is given, as in
the ‘carrot’ variant of FS, depends on whether a response is
omitted or incorrect and has nothing to do with a response’s
correctness.
We focused only on dichotomous mathematics items
(where 1 = correct and 0 = incorrect) and removed all poly-
tomous items from analysis. Maths items were chosen be-
cause these were considered to be least likely to be affected
by differences in translation (PISA tests are administered to
students in their local languages). We then fitted a latent
class model to the students’ responses.
Such a model is also known as a finite mixture model and
is used in the analysis of multivariate and categorical data. It
is appropriate when a researcher believes that item response
data belong to unobserved groups – termed ‘classes’ – which
are distinct from each other (??, p. 166). After accounting
for the dependence introduced by class membership, the
responses within each class are independent of each other.
The RF and FS Latent Classes
For our purposes, we believed that students’ PISA re-
sponses belonged to one of two scoring rule latent classes:
RF or FS. Given our earlier points, including the FS rule is an
expected choice, but the reader might be wondering why we
opted for the RF rule despite talking about misrepresenting
the number-right rule.
We used the RF rule for statistical convenience. The RF
and FS scoring rules are modelled using partial credit mod-
els – this made the latent class analysis less complicated
in contrast to using the NR and FS scoring rules. (The NR
rule is modelled using a Rasch model, which is statistically
different from a partial credit model.) As we pointed out
earlier, since the RF scoring rule is conceptually similar to
the NR rule, comparing the RF and FS rules is conceptually
similar to comparing the NR and FS rules.
We explain – in passing – what the Rasch and partial
credit models are before presenting our analysis results for
our PISA data.
What Are the Rasch and Partial Credit Models?
These are statistical models that are used to determine
the probability of an item response being correct. In our
analysis, we were also interested in the probability of an
item response being omitted or incorrect. The Rasch model
(see ??) is based on the logistic function, which is written as
ex
f (x ) = . (7)
1 + ex
The original argument x is replaced by the term θ − δ , so
that we have the probability function
eθ −δ
Pr(x = 1 | θ ) = , (8)
1 + eθ −δ
where x = 1 for a correct response and x = 0 for an incorrect
response. θ indicates an individual’s ability, while δ indicates
an item’s difficulty. These are measured on a common latent,
or unobserved, scale, which will be further explained in
the section on varying probability distributions and answer
switching.
The term θ − δ thus tells us about the disparity between
a student’s ability and an item’s difficulty. When the dif-
ference is positive (that is, when θ > δ ), the greater the
 EXAM STRATEGIES AND TEST SCORE VALIDITY
1.0
Probability
0.5
0.0
Ability, θ
Figure 3. An illustration of a Rasch model, used to represent
the NR scoring rule. The curves relate to the probabilities of
a correct response (solid line) and incorrect response (broken
line) as ability increases.
disparity, the greater the probability that an item will be
correctly answered. The reverse is true when the difference
between θ and δ is negative (θ < δ ) – the greater the dispar-
ity, the smaller the probability that an item will be correctly
answered.
A graphical representation of the Rasch model is found
in Figure 3. Just as we have a curve representing item cor-
rectness (solid line), we also have one representing item
incorrectness (broken line). We would write the probability
function for the ‘incorrectness curve’ as
e−(θ −δ )
Pr(x = 0 | θ ) = . (9)
1 + e−(θ −δ )
One issue that arises is that the above curve does not
distinguish between incorrect and omitted responses. All
it states is that the probability that an answer is incorrect
decreases as a student’s ability increases. However, it says
nothing about the probability of omitted responses. Al-
though these are marked as incorrect, their probability of
occurrence does not necessarily decrease with ability. This
is particularly true if a student internalises the FS scoring
rule, in which case the probability of occurrence is equal for
all levels of ability.
Past studies demonstrate that students who attempt tests
under (test-specified) NR and FS rules are qualitatively dif-
ferent from each other (?; ?; see also ?). This then implies
that we should not automatically assume that students with
different internally-represented scoring rules are equivalent.
PISA tests are scored in an NR fashion, which necessitates
using the Rasch model. But because of students who do not
comply with the test-specified scoring rule and differentiate
between incorrect and omitted responses, there is a need to
use a model that captures this difference.
Partial credit models, first introduced by ?, allow us to
do just that. Unlike a Rasch model, a partial credit model
consists of several probability functions that illustrate how
likely a given response will occur for some category. Our
categories, in this case, refer to the correct, incorrect and
omitted response types. For a question with j response types,
we write a partial credit model as
ejθ −δ j
Pr(x = j | θ ) = P2 jθ −δ ,
j=0 e
j
9
where j refers to one of the three response types and takes on
the values 0, 1 and 2 in accordance with Table 1. Note that the
above equation is one of several possible parametrisations
of the partial credit model. Thus, the mathematical form of
the model may appear differently in other literature.
The partial credit models used in our latent class analysis
reflect the polytomous nature of the response types. We
stress that this is not the same as the polytomous PISA items
we removed from analysis. Here, the (partial) credit associ-
ated with the response types can be thought of as personal
utility. A correct response yields the maximum utility, while
the type of internalised scoring rule determines the amount
of utility elicited by omitted and incorrect responses.
The probability functions of the partial credit models
characterising the RF and FS rules are shown in Figure 4.
Two curves replace the ‘incorrect curve’ earlier seen in the
Rasch model (broken and dotted lines). In RF scoring, the
dotted line represents the ‘omitted curve’ while the broken
line represents the incorrect curve. The reverse is true for
FS scoring – the dotted line represents the incorrect curve
and the broken line represents the omitted curve. In both
cases, the solid line represents the probability of a correct
response.
In layperson’s terms, then, what we specified in our par-
tial credit models was that under RF scoring, students of
lower ability omit more than students of higher ability. This
is because as ability, θ , increases, the probability of making
an omission decreases, while the probability of making incor-
rect and correct answers (that is, answering the question as
opposed to skipping it) increases. We specified the reverse in
FS scoring, in effect stating that students of higher ability are
split into those who omit and those who answer (correctly).
Results of the Latent Class Analysis
An algorithm based on the extended marginal Rasch
model (?) was written in R (?, version 3.2.3) for this analysis
and applied to all PISA booklets (excluding the one used for
special education students). Test booklets fell into one of two
broad categories: those with good latent class separation and
those with poor separation. For the booklets in the second
category, poor separation occurred because of insufficient
maths items (the number of maths items differed across
booklets), and so the algorithm was unable to decisively sort
students into one class or the other.
Naturally, we shall only consider the booklets with good
separation and, of these, will report results for only booklets
01 and 21, n = 40, 018. These booklets shared identical maths
items which were administered in the same order. (The other
well-separating booklets were 05, 11, 23 and 25.) We chose
these two booklets because the territories represented in
booklet 01 did not overlap with those of booklet 21, so we
ended up with a larger representation of students around
the world.
As seen in Figure 5, most students were sorted into the RF
class. This would suggest that the majority of students do
not misrepresent the test-specified scoring rule as an FS rule.
That said, a small but non-trivial proportion of students
(10) (circa 12 per cent) were classified as internalising the FS
scoring rule.
10 KOH, MARIS AND COOMANS

1.0
Probability

I/O
0.5

O/I
C
0.0

Ability, θ

Figure 4. An illustration of a partial credit model. The curves relate to incorrect, omitted and correct responses. In the legend,
these are abbreviated as I, O and C respectively.

For these students, omission was potentially problem-

atic because this was likelier to occur at higher levels of
ability (refer to Figure 4). At the country level, the propor-
tion of students sorted into the FS class ranged from zero
(Liechtenstein) to 34 per cent (Kyrgyzstan). We point out
that Liechtenstein’s results should be interpreted in context,
given that there were 23 Liechtenstein students as compared
to 397 Kyrgyz students. All country-level proportions of
students sorted into the FS latent class are visualised on a
world map in Figure 6, made using the rworldmap package in
R (?).
For the majority of students holding the RF rule, omission
was not so much of a problem since it was limited to the
low ability students. Even if these students did not omit any
questions, it would have been unlikely for these non-omitted
responses to yield enough credit to allow them to score fairly
well.
In a sense, it is good to know that most students did not
misrepresent PISA’s test-specified NR rule as an FS rule.
Nonetheless, the fact that not all students subscribed to the
test-specified scoring rule indicates that the non-compliant
students disadvantaged themselves by not using the guessing
strategy.
 EXAM STRATEGIES AND TEST SCORE VALIDITY 11

LCA Results, RF vs. FS (100 Iterations)

1.0
0.8
Proportion of Students

0.6
0.4
0.2
0.0

Class 0 (RF) Class 1 (FS)

Class Membership

Figure 5. Results of the latent class analysis, showing that the majority of students were sorted into class 0 (the RF scoring
rule). The step function shown in the figure is not a single line, but rather, is made up of thousands of data points representing
the students who completed PISA booklets 01 and 21. That the function progresses almost instantaneously from Class 0 to
Class 1 is indicative of good latent class separation.
KOH, MARIS AND COOMANS

0 0.35
Figure 6. Global heat map showing proportions of students per country who are sorted into the FS latent class. The darker the shade of blue, the larger the proportion (range: 0 to
35 per cent of the sample). Territories with no data (that is, non-participants of PISA 2009) are shaded black.
12
 EXAM STRATEGIES AND TEST SCORE VALIDITY
Varying Probability Distributions and the
Answer Guessing/Omitting Strategy
In the previous section, we discussed students’ individual
probability distributions in a simplistic setting. This allowed
us to explain what makes some students guess and others
omit in a clear and intuitive way. In the real world, students’
thought processes are not identical replications of each other
– each process is a function of a student’s unique background
and level of test preparation. Using fixed probability distribu-
tions to explain the differential use of test strategies among
students thus becomes insufficient.
We now expound on a more realistic case where every
student’s probability distribution is unique to that person.
Adam, Beth and Cate reprise their roles as our hypothetical
students. We are still examining their probability distribu-
tions with regards to one question of a test. However, we
now have it that the students possess varying degrees of
preparation for the question. As such, their ‘new’ probability
distributions can be written as follows.
πAdam = (0, 0.1, 0.4, 0.5)
πBeth = (0.01, 0.02, 0.07, 0.9)
π Cate = (0.5, 0.4, 0.1, 0)
For this example, option 4 is the correct answer. It is im-
mediately apparent that Adam and Beth will likely answer
the question correctly. Cate, again, will not.
Ranks and Order Statistics Revisited
As stated earlier, a student’s probability distribution gen-
erates information in the form of ranks and order statistics.
We want to know more about these pieces of information,
and test strategies and internalised scoring rules enable us
to do so.
In this section, we will see that internalised scoring rules
give us information about ranks and not order statistics,
while guessing/omitting behaviour gives us information
about order statistics but not ranks. The next section is also
in the context of varying probability distributions. There, we
will see how another test strategy, answer switching, gives
us information about order statistics but not ranks. These
sections serve as elaborations of Figure 2.
Guessing/Omitting, Internalised Scoring Rules and
Ranks
We see from the above that Adam and Beth’s probability
distributions have the same ranks but different order statist-
ics. Recall that order statistics are the probabilities reflecting
the magnitude of confidence across a question’s response
options, while ranks are the ordinal positions of these order
statistics. Hence, Adam and Beth have the same vector of
ranks,
rAdam = rBeth = (1, 2, 3, 4).
Considering Adam and Beth when they internalise the NR
scoring rule is not so interesting to us because of a lack of a
penalty were they to incorrectly answer the question. How-
ever, if they internalise the FS rule, then we notice differences
13
in their willingness to guess. (We use the credit values that
we mentioned earlier: 1 point for a correct response, −0.33
points for an incorrect response and 0 points for an omitted
response.) Under a formula scoring framework, the skipping
prospect produces the same utility for both students,
VAdam = VBeth = v (0) · w (1).
Guessing, on the other hand, results in different utilities,
VAdam = v (1) · w (0.5) +v (−0.33) · w (0.5) and
VBeth = v (1) · w (0.9) +v (−0.33) · w (0.1).
If, for simplicity’s sake, we assume that w (.) is an identity
function, then VBeth > VAdam and Beth will be more will-
ing than Adam to guess the answer to the question. (An
identity function is a function whose value is the same as
that of its argument. For example, in Adam’s case we have
w (0.5) = 0.5.)
Regardless of whether the internalised rule is NR or FS,
the presence of one coerces Adam and Beth to provide some
answer. In doing so, we learn about their most-preferred
response option, r 4−1 . However, knowing this tells us nothing
about the value of that rank’s order statistic.
If an answer is guessed, we know that the value of the
largest order statistic is large enough that it results in a
utility value more favourable towards guessing than omit-
ting. If an answer is omitted, then the converse occurs – the
largest order statistic is unable to tilt the balance towards
guessing and omitting yields a larger utility. From this test
strategy perspective, guessing or omitting tell us something
about how sufficiently large an order statistic is, but does not
associate it with a specific rank and, by extension, response
option.
Looking at Adam and Cate, we find that they have differ-
ent ranks but the same order statistics. As such,
rAdam = (1, 2, 3, 4) while
rCate = (4, 3, 2, 1).
The utility for skipping is the same for Adam and Cate (see
above) and, this time, the utility for guessing is also the same
for both students. However, the framing of the students’
guessing prospects is now different.
In Adam’s case, the guessing prospect reads: Select op-
tion 4 with a probability of 0.5 and earn 1 point; select one
of the others and receive a penalty of −0.33 points. Cate’s
guessing prospect reads: Select option 1 with a probability
of 0.5 and earn 1 point; select one of the others and receive a
penalty of −0.33 points. Adam and Cate’s guessing prospects
differ in the response option to which the greatest amount
of confidence is assigned (presented in bold for clarity).
Added Realism: Answer Elimination as a Strategy
In addition to what we have presented, another strategy
that will be familiar to many is that of answer elimination.
This is particularly helpful for questions where one is not
sure what the correct response option is, but can definitively
rule out at least one of the distractor options.
14 KOH, MARIS AND COOMANS
Through elimination, one ends up with a reduced set of
options, wherein the utility associated with making an incor-
rect choice becomes smaller and the utility associated with
making a correct choice larger. This in turn has implications
for the overall utility value, which will help with deciding
whether to guess or omit.
More Realistic Prospects, More Complexity
So far, we have added some realism to our example and
shown that varying probability distributions make it more
complex to determine values of utility, or V . This then makes
it less straightforward to identify students who are more
inclined to guess versus omit. For example, given a group
of students who adopt an FS rule instead of a test-specified
NR rule, we know that these students will have a greater
proclivity to omit their answers compared to their more
compliant peers.
However, these non-compliant students will also vary in
their degree of omitting. For some, omitting answers will
appreciably hinder their test scores, whereas for others, this
will not be much of an issue. It could be that, even though
the latter subgroup view an incorrect guess unfavourably,
they are sufficiently prepared and so are highly confident of
a specific response option being correct. (This is what we
saw with Beth.)
What interests us is the former subgroup – those who
omit to the point of hindering their scores. These are the
ones who would benefit the most from some kind of inter-
vention to correct their way of interpreting a test’s scoring
rule. Being able to determine the utility that students get
from the guessing prospect is of great use in helping us
identify this subgroup.
That said, determining the values of V is exactly the prob-
lem we face. While our example involving Adam, Beth and
Cate conveniently contained the values of their individual
probabilities (which are also the order statistics), this is not
the case in reality. For argument’s sake, we simplified the
weighting function, assuming that it was an identity func-
tion. Again, we do not normally expect this to be so for
actual students (see ?).
This therefore makes it important to develop a method
through which we can learn the values of the ranks and
order statistics. This was referred to in the introduction as
elimination scoring, and we will discuss this at a later point.
Varying Probability Distributions and the
Answer Switching Strategy
Having talked about guessing and omitting, we move
on to another test strategy often used by students: answer
switching. This refers to the act of answering a question,
moving on to the rest of the test and going back to the initial
question some time later and changing the answer.
Since the late 1920s, several studies have investigated
when and why students switch answers on multiple-choice
tests. Received wisdom has it that one’s initial response to
a question, or item, is likeliest to be correct. One should
therefore be very cautious about switching one’s answers.
Time and time again, however, empirical research uncovers
evidence to the contrary: For students who change their
answers on a test, doing so is generally beneficial to their
scores.
Convention establishes three categories of answer
switches on a multiple-choice test: wrong to right (WR),
wrong to wrong (WW) and right to wrong (RW). Previous
studies display these categories as percentages of the total
number of switches made. Unsurprisingly, the WR category
is of interest, and a ratio of WR to RW switches (WR/RW)
is calculated to conveniently express the degree to which
answer switching is beneficial.
A review by ? summarised the proportions of WR
switches from 28 studies conducted between 1929 and 1983.
These proportions ranged from 44.5 to 71.3 per cent, with
most studies having a proportion of WR switches greater
than 50 per cent (only four did not).
? and ? are more recent examples where the majority of
answer switches are WR. For all these studies, the WR/RW
ratio was greater than one, indicating that there were more
WR switches than RW ones. These results fly in the face
of received wisdom, constituting ‘empirical wisdom’ that
answer switching is not a test behaviour to be discouraged.
We now turn to our hypothetical three students to demon-
strate the relation between answer switching and order stat-
istics.
Order Statistics and Switching
Recall that, in the case of varying probability distributions,
the confidence levels of Adam, Beth and Cate with regards
to a four-option MCQ are as follows:
πAdam = (0, 0.1, 0.4, 0.5)
πBeth = (0.01, 0.02, 0.07, 0.9)
πCate = (0.5, 0.4, 0.1, 0)
Between Adam and Beth, both have the highest amount
of confidence that option 4 is the correct answer (which we
decided earlier that it was). We also mentioned that Beth is
much more confident about her choice than Adam is. What
are the implications of this?
Adam has assigned similar levels of confidence to options
3 and 4 (0.4 and 0.5 respectively). That the bulk of his confid-
ence has gone to these options (0.4 + 0.5 = 0.9, as compared
to 0 + 0.1 = 0.1) indicates that he is uncertain about which is
correct. For instance, Adam could decide initially to shade
option 3 as the correct answer, but then change his answer
to option 4 some time later. This would be an instance of a
WR switch. Alternatively, he could select option 4 at first
but switch it later to option 3, which would be a RW switch.
Cate’s order statistics are similar to Adam’s, therefore
she would encounter the same uncertainty that he does.
However, her indecision is between options 1 and 2, both of
which are incorrect. She may select option 1 initially and
switch to option 2 or vice versa, but regardless, her course
of action is a WW switch.
Through the above example, we contend that answer
switching is most likely to occur when the order statistics
are similar to each other. But there is a catch – the values of
these order statistics are unavailable to us, so how might we
go about testing our hypothesis?
 EXAM STRATEGIES AND TEST SCORE VALIDITY
δ1 δ2 θA δ3
Figure 7. An illustrative latent scale showing Adam’s ability
(θ A ) relative to three test items’ difficulties (δ 1 ,δ 2 and δ 3 ).
Ability and Difficulty as Substitutes for Order Statist-
ics
While we do not know the values of O, there is another
way of characterising a student’s uncertainty while answer-
ing a question. We do so by drawing from psychometrics
(?), using a framework called item response theory (IRT; see
?). This framework concerns the interplay between student
ability and item difficulty, which were earlier introduced in
Equation 8. Student ability and item difficulty are measured
on a common latent scale, and we refer the reader to Figure 7
for an illustration thereof.
We use IRT to estimate the probability that a student
answers a given item correctly. In Figure 7, we know that
Adam’s ability level is greater than the difficulty levels of
items 1 and 2 (a positive-valued discrepancy). Therefore, the
probability that he answers these items correctly is greater
than 0.5. His ability level is not greater than that of item 3
(a negative-valued discrepancy), and so the probability that
he answers that item correctly is less than 0.5.
Let us consider an alternative scenario where Adam’s
ability level is equal to the difficulty level of item 2, so that
θ − δ = 0. In such a case, the probability that he correctly
answers that item is
eθ −δ e0
Pr(x = 1 | θ,δ ) = = = 0.5.
1 + eθ −δ 1 + e0
This suggests that his answer could go either way – we
cannot know for certain if he will answer correctly or incor-
rectly.
From Adam’s perspective, this is seen as a situation where
he is highly uncertain of his answer. One possibility is that
he selects one answer but is unsure of its correctness, having
absolutely no idea if any the other three options is a viable
alternative. (Among the three options, then, he would select
from them at chance levels.)
A second possibility is that Adam shortlists two or more
response options, but has trouble deciding among them. This
is the possibility we are interested in, as his order statistics
are assumed to be similar over the shortlisted options. Since
we do not know what the values of his order statistics are,
we shall rely on his ability and item difficulty instead.
Ability and difficulty are linked to order statistics and
ranks in another, more statistical way. To explain this link,
we use the analogy of students as Bayesian statisticians
engaged in model selection. But instead of selecting a model,
students aim to select the best response option out of all the
available ones. We can think of students’ confidence levels
15
as their likelihoods, which we express as
L 1 = Pr(Data | H 1 ),
L 2 = Pr(Data | H 2 ),
L 3 = Pr(Data | H 3 ) and
L 4 = Pr(Data | H 4 ).
We define Hi ,i = 1, . . . , 4, as students’ hypothesis of which
option is correct. The data in question represents students’
prior content knowledge, background experiences – any-
thing that helps them to make a decision with regards to
choosing their best option.
We know that, in the absence of any information (or
data, as it were), selecting the best response option occurs at
chance levels. In other words, without any data, students se-
lect the best option with an a priori probability Pr(Hi ) = 1/4.
We also say that their data come from some distribution F .
Using the above, we get to what we truly are interested
in: the a posteriori probabilities of the hypotheses. Using
the data as a prior distribution and combining it with our
likelihoods, we have, using Bayes’ theorem, that a posteriori
Li
Pr(Hi | Data) = .
L1 + L2 + L3 + L4
Because the above process is entirely randomly generated,
every time a replication of the ‘experiment’ occurs, the most-
preferred hypothesis – and the likelihoods – changes.
We are now ready to show how the posterior probabilities
relate to the probabilities that figure in an IRT model. The
posterior probabilities depend on the data, which we con-
ceive of as being random. However, these data are missing.
Hence when we integrate over the distribution of the data
we obtain
Z
Pr(Hi | F ) = Pr(Hi | Data) dF (Data),
and these are the probabilities in the IRT model. The distri-
bution F depends on the person ability and option charac-
teristics.
We illustrate the relationship with a concrete example.
Suppose the the likelihoods are modelled as
−λi x i
L−1
i = x i ∝ λi e ,
such that a student prefers the option for which x i is smallest.
We obtain that
λi
Pr(Li > L j , j , i) = ,
λ1 + λ2 + λ3 + λ4
which becomes a Rasch model if λ4 = eθ (where the correct
option is marked with the subscript 4) and λ1 = λ2 = λ3 =
eδ /3 .
With this, students then know which likelihood, or re-
sponse option, best answers the question. They are con-
sequently able to decide between the available prospects.
Operationalising Student Ability and Item Difficulty
In the Rasch model, it is generally known that the total
test score, or sum score, and proportion of correct answers
16 KOH, MARIS AND COOMANS
for an item are sufficient statistics for student ability and
item difficulty respectively (?). For item difficulty, the smaller
the proportion of students correctly answering the item, the
more difficult that item is.
A sufficient statistic is one whose estimate contains all
available information about a parameter (?), thus providing
a means of summarising the given data. In our case, the para-
meter(s) refer to student ability and item difficulty, while the
data consist of students’ test responses. Since sum scores
and proportions of correct answers are sufficient statistics,
we can then use these as our operational definitions of our
afore-mentioned parameters.
A Possible Way of Testing Our Hypothesis
A shared feature of the studies in the above-mentioned
literature is that the student samples used were relatively
small in comparison to the samples we are presently able to
work with. For example, when we conducted the latent class
analysis earlier, we used a sample of 40, 018 students who
participated in PISA 2009. Given the size of the data sets
available to us today, we are able to count the proportions
of WR, WW and RW switches for entire cohorts of students.
For example, in the Netherlands, children end their
primary education (basisonderwijs) around the age of 12.
During this time, they sit for the Centrale Eindtoets (?), ab-
breviated the Cito test after its developer, the Central Insti-
tute for Educational Measurement (Centraal Instituut voor
Toetsontwikkeling). This test is conducted over three days
and assesses students’ mathematics and language abilities.
Their test results, together with personalised reports and re-
commendations from teachers, determine which educational
track they attend for secondary school.
When students complete the Cito test, their scripts are
marked by a machine that makes digital scans when in-
stances of answer switching are detected. These scans, com-
bined with knowledge about student ability and item dif-
ficulty, make it possible to see if answer switching indeed
occurs when one’s ability level is close to that of item diffi-
culty.
While the second author in this thesis is an employee of
Cito, privacy concerns preclude any analysis of Cito test data
at this time. We look forward to embarking on a large-scale
analysis with this data in the future, when this hurdle is
cleared. For now, we end our discussion on varying probab-
ility distributions and move on to a new scoring rule that lets
us learn more about certain test characteristics of students.
Elimination Scoring
We propose elimination scoring as a new scoring rule
that allows us to learn what we could not from regular scor-
ing rules: ranks and order statistics. In addition, this new
rule gives us information about which students are over- or
underconfident.
How Elimination Scoring Works
Elimination scoring bears some resemblance to the an-
swer elimination strategy that was earlier mentioned. When
this rule is in effect, students answer MCQs by eliminating
all options that they consider incorrect. Points are awarded
for making a correct choice (eliminating an incorrect option
or retaining the correct one). Similarly, points are lost when
an incorrect choice is made (eliminating the correct option
or retaining an incorrect one).
The ways in which students eliminate and retain response
options give us an opportunity to learn about their ranks
and order statistics. Since these ways are basically courses
of action, we refer to them as prospects as well. In a four-
option MCQ, there are four prospects to be considered: elim-
inating none of the options (prospect A), eliminating one
option (prospect B), eliminating two options (prospect C) and
eliminating three options (prospect D). We now go through
the prospects in sequence and explain the points system
associated with each.
Prospect A is rendered below.
1 2 3 4
Since none of the options has been eliminated, a penalty of
−2 points is imposed with certainty.
Prospect B involves the least-preferred option being elim-
inated, r 1−1 . (Recall that ranks – the subscript in r −1 – are
expressed in reverse order of preference.)
r 1−1
1 S2 3 4
S
If the eliminated option is the correct one, a penalty of −4
points is levied with probability o 1 (this is the smallest value
of all the order statistics). Otherwise, 0 points are awarded
with probability o 2 +o 3 +o 4 .
Prospect C involves the elimination of the two least-
preferred options, r 1−1 and r 2−1 .
r 1−1 r 2−1
1 S2 S3 4
S S
If either r 1−1 or r 2−1 happens to be the correct option, a penalty
of −2 points is imposed with probability o 1 +o 2 . Otherwise,
2 points are awarded with probability o 3 +o 4 .
Prospect D entails full compliance with the scoring rule’s
instructions, so that response options r 1−1 , r 2−1 and r 3−1 are
eliminated.
r 3−1 r 1−1 r 2−1
S1 S2 S3 4
S S S
If any of the eliminated options is the correct one, 0 points
are awarded with probability o 1 +o 2 +o 3 . Otherwise, the cor-
rect response has been identified and 4 points are awarded
with probability o 4 .
We can thus summarise the calculation of one’s utility
values over the four prospects as
VA = v (−2) · w (1),
VB = v (0) · w (o 2 +o 3 +o 4 ) +v (−4) · w (o 1 ),
(11)
VC = v (2) · w (o 3 +o 4 ) +v (−2) · w (o 1 +o 2 ) and
VD = v (4) · w (o 4 ) +v (0) · w (o 1 +o 2 +o 3 ).
 EXAM STRATEGIES AND TEST SCORE VALIDITY
An Amalgamation of the NR, FS and RF Scoring Rules
From the above, we see that elimination scoring combines
the features of the scoring rules we have mentioned. Pro-
spect A is similar to the RF scoring rule in that it discourages
guessing by imposing a penalty for it. Prospects B and C are
akin to formula scoring, where a penalty is imposed if the
correct answer is eliminated. In FS terms, this is the same as
selecting an option other than the correct one. Prospect D
corresponds to the NR rule being complied with, as points are
awarded for making the right decision and none otherwise.
What elimination scoring does, then, is discourage test
behaviours like guessing and non-compliance with scoring
instructions. It also provides an incentive to eliminate all in-
correct options, since the payoff for the correct combination
of eliminations is much higher than that for prospects B and
C (4 points as opposed to 0 or 2).
How Does Elimination Scoring Tell Us About Ranks,
Order Statistics and Student Over-/Underconfidence?
We can consider the information we learn about students
in the contexts of full and partial compliance with the scoring
rule. Full compliance refers to the adoption of prospect D,
while partial compliance refers to the adoption of prospects
B and C.
For ranks, when students adopt prospect D, we learn their
most preferred option, r 4−1 . When they adopt prospect B or
C, we learn their least-preferred option(s), r 1−1 (and possibly
r 2−1 ). These options are the eliminated ones. For those op-
tions that the students do not eliminate, we know that these
are perceived to have about the same rank (otherwise one
or more would have been eliminated).
For order statistics, when students adopt prospect D, we
learn the response option to which they assign the highest
value, o 4 . As with ranks, when they adopt prospect B or
C, we learn which prospect(s) has the smallest value(s) in
the case of the eliminated option(s). We learn that the non-
eliminated options are likely to be close to each other in
value.
Unlike any of the other scoring rules, elimination scoring
gives us information pertaining to how over- or underconfid-
ent students are. Before delving into the issue of confidence,
we want to point out a particular feature of this scoring rule
that one can observe in prospects C and D.
Consider a student who eliminates the response options
r 1−1 and r 2−1 . Assume that these eliminations are, indeed,
correct and that the student is now deciding which of the
remaining options to eliminate. Further assume that the
student is having trouble deciding between the options. Pro-
spects C and D can be framed as follows.
The student will receive 2 points with certainty if the
choice is made to not eliminate a third option. However,
should a third option be struck out, there is an o 4 probability
of receiving twice as many points for a correct answer. This
is accompanied by a larger 1 −o 4 = o 1 +o 2 +o 3 probability of
receiving 0 points for an incorrect elimination.
Whether or not students decide to eliminate the third
option tells us something about their appetite for risk. The
long-run outcomes of this decision, however, tell us some-
17
thing about how confident they are. For example, if we have
a student who always goes with prospect D, but frequently
receives 0 points for her efforts, we have reason to believe
that she may be overconfident. On the other hand, if another
student always goes with prospect C, but frequently receives
2 points because the correct answer is not eliminated, we
learn that he may be underconfident.
Interestingly, given that a student has made two initial
correct eliminations, the choice between prospects C and D
is reminiscent of a problem posed by the French economist
Maurice Allais (?). This problem is also detailed in ? as a
starting point for developing their prospect theory of beha-
viour. Allais presents his problem as choice between two
gambles:
Gamble 1A :
Pr(Receive $1 million) = 1.00
Gamble 1B :
Pr(Receive $1 million) = 0.89
Pr(Receive nothing) = 0.01
Pr(Receive $5 million) = 0.10.
The majority of his participants opted for Gamble 1A. This
gamble corresponds to prospect C in our case.
Gamble 1B is similar – but not identical – to prospect
D, as the possibility of receiving the same amount of credit
as the previous gamble is removed (in Allais’s example, $1
million; in elimination scoring, 2 points). We can merge this
possibility with that of receiving nothing, so that one now
faces a 0.90 probability of receiving nothing and a much
smaller 0.10 probability of receiving $5 million. This is, in
principle, what prospect D of elimination scoring entails.
Unlike Allais’s participants, there is no reason to expect
the majority of students to prefer prospect C to prospect D.
We can attribute this to the consequences of participating in
an economics experiment as opposed to participating in an
academic test. The former is a low-stakes situation (if there
is even anything to lose in the first place), while the latter
determines one’s exam grade and consequences such as the
educational track available to one given the obtained grade.
Nonetheless, in a situation where one is unsure of which
of the two non-eliminated options is correct, the decision to
go with prospect D instead of C is a function of overconfid-
ence. The more overconfident one is, the more likely one is
to choose prospect D (and vice versa), in the same way that
a more risk-loving and confident person is likelier to choose
Gamble 1B over 1A.
Next, we present some examples that indicate the best
prospect (A, B, C or D) to select under different combinations
of the value and weighting functions, w (.) and v (.). The best
prospect for a specific combination is one which yields the
highest utility, V .
Combinations of w (.) and v (.)
Table 3 displays a matrix of results. Within each cell
is the prospect with the highest utility (that is, the most
preferred prospect) for Adam, given a specific combination
18 KOH, MARIS AND COOMANS
of weighting and value functions. Recall that these functions
are crucial for determining the amount of utility one receives
for a given prospect and were first mentioned in ?. The fig-
ures representing the columns are the weighting functions,
while those representing the rows are value functions.
We present three hypothetical weighting functions that
show how students with varying amounts of confidence
transform their order statistics. In Table 3, the x-axis stands
for the actual order statistics o, and the y-axis stands for
the transformed – or weighted – order statistics w (o). An
overconfident student will overweight an order statistic if
its value is close to 1, making the response option associated
with that statistic very attractive. Conversely, that student
will underweight a statistic whose value is closer to 0, leading
to its associated response option being discounted.
The opposite happens with an underconfident student.
In this case, order statistics that are high are underweighted
and given less importance than should be the case, while
those that are low are overweighted. One could say that this
presents a rather pessimistic perspective of one’s chances of
correctly answering the question. An ideal student is one
who does not overweight or underweight his or her order
statistics. Such a student takes the statistics’ values as they
are and does not distort them with internal perceptions and
biases.
As with their weighting counterparts, we also have value
functions representing ideal, overconfident and underconfid-
ent students. Where the weighting functions deal with the
transformation of the order statistics, the value functions
indicate the degree of a person’s appetite for risk. One can
better understand this idea by examining the value function
plots in two parts: the side to the right of the y-axis and the
side to the left.
The x-axis of a value function stands for the pay-off, while
the y axis represents the value attained when one receives
a particular pay-off. In our context, this pay-off refers to
the amount of (awarded or penalised) points associated with
a student’s choice of prospect. Therefore, the right side of
the function represents the value associated with a correct
elimination decision, while the left side represents the value
associated with an incorrect elimination decision.
Given the same absolute value of a pay-off (for example,
−2 and 2 points), an overconfident student will attach greater
value to the positive one than to the negative one. This in-
dicates more risk-loving behaviour, and we expect that such
a student is likelier to make guesses and risk incorrectly an-
swering a question (prospect D). An underconfident student
behaves the opposite way and is expected to take a ‘safer’
approach to a question (prospect A, B or C, depending on
degree of preparedness). An ideal student is neither risk
loving nor risk averse, and will derive the same absolute
value for positive and negative pay-offs of the same absolute
magnitude.
There is a fourth value function in addition to the three
hypothetical ones. We have labelled this the original value
function because it is the one whose shape was proposed in
Kahneman and Tversky’s (?) initial presentation of prospect
theory. The equation for the function, as well as the para-
meter values used to determine its shape, can be found in
?.
Going back to Adam, we will consider these four value
functions together with the three weighting functions, and
evaluate the prospects that he is likeliest to adopt given his
order statistics. Recall that πAdam = (0.0, 0.1, 0.4, 0.5).
Adam’s Likeliest Prospects
Here, we go through some examples that lead to the like-
liest prospects in the cells of Table 3. Importantly, the pro-
spects in the table are the result of comparisons between the
C and D prospects for every combination of w (.) and v (.).
We will shortly explain why the A and B prospects do not
feature in our comparison.
First, we establish some arbitrarily-chosen values that
we will use in our examples. For each prospect (C and
D), we have values for w overconfident (.), w underconfident (.),
v overconfident (.), v underconfident (.) and v original (.). The functions
w ideal (.) and v ideal (.) are identity functions. Thus, we have
the following:
Prospect C:
w over (0.9) = 0.99,w over (0.1) = 0.05
v over (2) = 4,v over (−2) = −1.5
w under (0.9) = 0.75,w under (0.1) = 0.05
v under (2) = 1.5,v under (−2) = −2.25
v orig (2) = 1.84,v orig (−2) = −4.14
Prospect D:
w over (0.5) = 0.55,w under (0.5) = 0.45
v over (4) = 6,v under (4) = 3,v orig (4) = 3.39.
We will also use the > and < operators to denote which of the
two prospects is greater, and consequently which prospect
is likelier.
The values for v original (.) are not arbitrary and come from
the value function equation found in ?, p. 309. For quick
reference, the equation is defined as
x α
 if x ≥ 0
v (x ) = 
 −λ(−x ) β (12)
if x < 0,

where α = β = 0.88 and λ = 2.25 (values can be found in the
same reference).
Case 1. We begin by considering the case where w (.) and
v (.) are both identity functions (that is, both ideal functions).
With reference to Equation 11, we have that
v (2) · w (0.4 + 0.5) +v (−2) · w (0.1) < v (4) · w (0.5)
| {z
Prospect C
} | {z }
Prospect D
2 · 0.9 − 2 · 0.1 < 4 · 0.5.
This corresponds to the w ideal (.), v ideal (.) cell of Table 3.
Case 2. We consider the case where v (.) is an identity
function, but not w (.). For w over (.),
2 · 0.99 − 2 · 0.05 < 4 · 0.55.
 EXAM STRATEGIES AND TEST SCORE VALIDITY 19

Table 3
Example of Adam’s Most Optimal Prospects, Given His Order Statistics and Combinations of Weighting and Value Functions
(Horizontally- and Vertically-Arranged Figures, Respectively)

0.0 0.5 1.0 0.0 0.5 1.0 0.0 0.5 1.0

Overconfident Ideal Underconfident

D D D

Original

D D D

Ideal

C C C

Overconfident

D D D

Underconfident
20 KOH, MARIS AND COOMANS
For w under (.),
2 · 0.75 − 2 · 0.05 < 4 · 0.45.
These correspond respectively to the left and right cells of
the row represented by v ideal .
Case 3. In this case, we have the opposite – now w (.) is
an identity function, but not v (.). For v orig (.), we have
1.84 · 0.9 − 4.14 · 0.1 < 3.39 · 0.5, (13)
which corresponds to the v orig (.), w ideal (.) cell. For v over (.),
4 · 0.9 − 1.5 · 0.1 > 6 · 0.5,
corresponding to the v over (.), w ideal (.) cell. Lastly, for
v under (.),
1.9 · 0.9 − 2.25 · 0.1 < 3 · 0.5,
corresponding to the v under (.), w ideal (.) cell.
With these three cases we have obtained the Adam’s
likeliest prospects for six out of 12 combinations in Table 3.
Using the values we outlined earlier, the rest of the cells can
be filled out. To save on space, we will not outline the steps
to obtain the utility of the likeliest prospect in each of them.
Why We Left Out Prospects A and B
We considered only prospects C and D in our above ex-
ample because prospects A and B are not possible in practice.
To demonstrate why, let us continue using Adam’s order
statistics to calculate the utility associated with prospects A
and B. We find that we have
VA = v (−2) · w (1)
= −2 and
VB = v (0) · w (0.1 + 0.4 + 0.5) −v (−4) · w (0.0)
= 0.
Seeing as the above prospects yield utility values of −2 and
0, as long as C and D yield positive utility values – which
they do – there is no reason to consider A and B at all.
For VB , note that in Adam’s case, the utility is 0. This
is only because his smallest order statistic, o 1 , is equal to 0.
If it were some positive value, then VB would be negative
just like VA , providing even less of a case for its being the
likeliest prospect.
Regardless, we believe there are situations where pro-
spects A and B have their use, and would like an elimination
scoring rule that is organised in such a way that these pro-
spects have a realistic chance of being the most-preferred
ones. To achieve this aim, redesigning our original rule is
needed.
Elimination Scoring, Revised
In this new form of elimination scoring (which we now
abbreviate as ES), we have a gain-framed rule that also func-
tions as an extension of FS scoring. By imposing a gain
frame, we put students in a situation where no points are
ever deducted from errors. Instead, their maximum and
minimum points are non-negative values. We summarise
our system of awarding credit in Table 4.
In ES scoring, a penalty of 0 points is imposed when a
mistake is made. In the previous version of ES, a mistake – or
incorrect choice, as we termed it earlier – could be one of two
types: eliminating the correct answer or failing to eliminate
an incorrect answer. Here, we define only the former as an
incorrect choice. If a mistake is not made, a test candidate is
guaranteed at least 1 point for that question. Similar to FS
scoring, this measure helps discourage impulsive answering,
but because no points are deducted, students do not have to
associate a mistake with a ‘punishment’.
If a student is completely ignorant of what the correct
and incorrect response options are, omitting is encouraged
by the guarantee of 1 point. Conversely, random guessing is
discouraged by the penalty of 0 points for eliminating the
correct answer.
In a situation where a student definitively knows that
one of the options is incorrect, eliminating that option is en-
couraged by prospect B, which earns the student 1 14 points.
Related to this is prospect C, which presents a situation
where a student definitively knows that two options are in-
correct, but cannot distinguish between the two remaining
ones. This prospect earns the student 1 24 points.
Finally, when a student knows the answer for sure, pro-
spect D is the best choice. This results in the maximum 1 34
points being awarded.
Calculating the utility for each prospect is no longer the
same as in Kahneman and Tversky’s (?) prospect theory.
The basic formula for computing utility is
V = (1 +p) · w (.), (14)
where p is the proportion of correct eliminations. For ex-
ample, in prospect B, since one response option is correctly
eliminated, p = 1/4.
Using Adam’s order statistics, and assuming that his
weighting function is an identity function for simplicity’s
sake, we can thus calculate the utility values of his prospects
as
VA = (1 + 0)
= 1,
1
VB = (1 + ) · w (0.1 + 0.4 + 0.5)
4
= 1.3,
2
VC = (1 + ) · w (0.4 + 0.5)
4
= 1.4 and
3
VD = (1 + ) · w (0.5)
4
= 0.9.
We see immediately that prospect C – eliminating the op-
tions associated with the order statistics 0.0 and 0.1 – is
likeliest.
 EXAM STRATEGIES AND TEST SCORE VALIDITY 21

Table 4
Credit Awarding System in the Revised Version of Elimination Scoring
Prospect Points Action on test question Illustrative order statistics, o

– 0 Eliminate correct option (1, 0, 0, 0)

A 1 Omit (complete ignorance) (0.25, 0.25, 0.25, 0.25)
B 1 41 Eliminate one incorrect option (0.1, 0.3, 0.3, 0.3)
C 1 42 Eliminate two incorrect options (0.1, 0.1, 0.4, 0.4)
D 1 34 Eliminate three incorrect options (certainty) (0, 0, 0, 1)

With Beth’s order statistics, we have

VA = (1 + 0) · w (1)

2.0
= 1,

Utility

1.0
1
VB = (1 + ) · w (0.02 + 0.07 + 0.90)
4
= 1.2,

2
VC = (1 + ) · w (0.07 + 0.90)
4
= 1.5 and
3
VD = (1 + ) · w (0.90)
4
= 1.6.
Given her certainty about one of the response options, it
makes sense to choose prospect D, which is what we see
from the above.
We shall consider one final example where a test can-
didate is completely ignorant. That is, the student is un-
able to (a) eliminate any of the incorrect response options
and (b) has no inkling of what the correct option might
be. Such a student has a probability distribution of π =
(0.25, 0.25, 0.25, 0.25), and the prospect utility values are
VA = (1 + 0) · w (1)
= 1,
1
VB = (1 + ) · w (0.25 + 0.25 + 0.25)
4 latter occurs when more options are eliminated than should
= 0.9,
2 zero at the point where all options are eliminated. This
VC = (1 + ) · w (0.25 + 0.25)
4
= 0.8 and
3
VD = (1 + ) · w (0.25 + 0.25)
4 The Interdisciplinarity of Elimination Scoring
= 0.4.
For such a student, the best course of action is clearly pro-
spect A – skip the question altogether.
It is worth nothing that ES scoring is geared to penalise
students for choosing a suboptimal prospect in the form of
reduced utility. This helps to dissuade students from guess-
ing on questions to which they do not know the answer,
consequently preserving test score validity.
0.0
0.00 0.25 0.50 0.75 1.00
p
Figure 8. The utility curve for when prospect C – eliminat-
ing two response options – is most preferred. The broken
vertical lines illustrate the reduction in utility when the
suboptimal prospects B and D are chosen. This curve holds
for any number of response options, not just four.
Figure 8 graphically depicts our numerical examples. It
shows how a student’s utility peaks only when the pro-
portion of correctly-eliminated response options is at its
optimum point, p = 0.50 (marked with a solid vertical line),
given that student’s set of order statistics.
The points p = 0.25 and p = 0.75, marked with broken
vertical lines, represent suboptimal proportions of elimin-
ated options. The former occurs when a student attempts
to eliminate fewer options than he or she should, and the
be. Beyond q 2 , utility continues decreasing until it reaches
makes sense as eliminating all possible options is akin to
eliminating the correct option in terms of undesirability, and
should be treated the same way.
The ES rule represents an intersection of psychometrics
and behavioural economics. While behavioural economics
yields information about what an individual student might
do on a test, psychometrics links students’ individual per-
formances together and makes them comparable. This also
provides information about test items and the students’ abil-
ities relative to the items.
We can use ES scoring to gain a better understanding of
22 KOH, MARIS AND COOMANS
test-taking students. Just as we used a latent class model
to identify students who internalised an FS or RF scoring
rule, we can do the same thing with ES scoring. For example,
despite the emphasis given to not guessing, there may still be
some students who decided to do so. These non-compliant
test-takers might internalise a scoring rule such as NR. A
latent class model will then be useful in helping us detect
them.
Of course, such a model will be more complex than the
one we used for the FS and RF rules. While the NR rule
is modelled using a Rasch model, the ES rule is modelled
with a partial credit model (with the four prospects as the
categories). Nevertheless, the logic behind the latent class
analysis is the same and we can expect to find students in
two separate classes if they exist.
Latent class analysis could also be applied to students’
weighting and value functions. With reference to Table 3,
we might specify different classes of weighting functions,
different classes of value functions and even classes repres-
enting combinations of weighting and value functions. In
the third scenario, this would be a 12-class model (3 w (.)
classes × 4 v (.) classes).
By combining information about the scoring rule, weight-
ing and value function classes, it is possible to create an
individualised testing profile for each student. This allows
educators to understand how every student is likely to be-
have on a test and identify areas in which intervention or
assistance is indicated. We believe that the ES rule will be
a useful tool to both students and teachers, and lead to test
scores that are more reflective of students’ abilities.
Summary
We now summarise what we have covered in this thesis.
First, we introduced test strategies and described why they
are able to influence test score validity. We limited our
discussion to multiple-choice questions (MCQs), proposing
that students who attempt such questions form their own
distribution of probability values, or levels of confidence. We
then decomposed the information provided by these prob-
ability values into ranks and order statistics, two sources of
information which are independent of each other.
With regards to students’ probability distributions, we
first introduced a simplified scenario where the levels of con-
fidence were identical across all response options. Ranks and
order statistics were not featured in this section. However,
making use of a uniform distribution allowed us to demon-
strate how prospect theory (?) explains the differential use
of the answer guessing/omitting strategy among students.
Deciding between guessing and omitting was seen to be
driven by a student’s internalised scoring rule, of which we
introduced three – number right (NR), formula scoring (FS)
and reverse formula scoring (RF).
We provided empirical evidence of students who sub-
scribed to either the RF or FS scoring rule, using a latent
class – or finite mixture – model to classify PISA students
into one class. Whether or not a student was sorted into the
FS latent class depended on whether answers were omitted
on the test. We visualised our results as a global heat map
that displayed the proportions of students in the FS class
across the participating countries. For the most part, stu-
dents did not misrepresent the test-specified NR rule as an
FS one. For those who did misrepresent the scoring rule, the
country with the highest proportion of such students was
found to be Kyrgyzstan.
Next, we considered the more realistic case of varying
probability distributions, and how these affected two test
strategies: answer guessing/omitting and answer switching.
With the former, we stated that varying distributions added
to the complexity of determining which choice of strategy
students would use. For the latter, we contended that answer
switching is likeliest when a student’s order statistics are
very close in value to each other.
Finally, we introduced elimination scoring as an extension
of the FS rule and a means of learning about students’ ranks
and order statistics. We explained how it works and gave
an example using a hypothetical student’s order statistics.
We explained why only prospects C and D appeared, and
gave justification for why prospects A and B would never
be selected. Recognising that the latter two prospects have
their place in a testing situation, we revised our ES rule so
that all prospects would be likely, depending on students’
order statistics. We added examples to demonstrate how
our revised scoring rule works and proposed that ES scoring
could be a useful addition to the educator’s toolbox.
Discussion and Conclusion
Today’s educational systems are designed for the masses,
resulting in a one-size-fits-all approach when it comes to test-
ing students. The same test scripts and assessment criteria
are applied to everyone, either within the whole system or
within a certain educational track. Based on a student’s test
results, potentially substantial and long-lasting decisions are
then made with regards to his or her academic future.
Given that students are unique products of an infinite
combination of backgrounds, characteristics and experi-
ences, we cannot expect them all to respond equally to the
same assessment criteria. We would like students to un-
dergo a bespoke testing process that caters to their relative
strengths and weaknesses, but this arrangement is expensive,
resource-intensive and ultimately unrealistic.
However, in combining ideas from psychometrics and be-
havioural economics, we have presented a testing approach
that takes us some way towards this ideal. Through our
(revised) elimination scoring rule, students are discouraged
from using score-maximising strategies – like guessing –
that are not based on their existing ability. This rule also
benefits students who would normally be disadvantaged
in NR- or FS-scored tests, for instance because they refuse
to guess or switch their answers. These students always
have a most optimal prospect for their personal set of order
statistics, and they always receive credit as long as they
choose one of the four prospects.
By implementing the ES rule, we can learn about how stu-
dents approach tests and, importantly, can more accurately
measure their level of content knowledge. This is particu-
larly helpful for students who are in educational systems
where high-stakes testing is the norm.
 EXAM STRATEGIES AND TEST SCORE VALIDITY 23

This thesis represents a theoretical introduction to a line

of research aiming to understand and help students using
tests. Much work remains to be done to see if our proposed
scoring rule works in practice, and if it results in the desired
changes in test behaviour when applied to a large student
population. Regardless, it is an important first step towards
changing the way tests are viewed and used, so that they
open doors for students and not close them.