SETTING & MAINTAINING

EXAM STANDARDS
Raja C. Bandaranayake

DEFINITIONS
Standard setting is a process of determining how
much is good enough.
The standard or criterion level of performance is a
point on the scale of measurement at which
separation of competence and incompetence
occurs.
Cut-score, cut-off score or passing score represents
this standard on a given test for making decisions
pertaining to the purpose for which the test was
conducted, e.g., to certify competence.

ERROR IN MEASUREMENT
True score is a conceptual measure indicating true
extent of competence in a given subject, e.g.,
Anatomy.
Observed score is the score assigned as a result of
taking a test, say in Anatomy.
The difference between true and observed scores is
indicative of the amount of error in the
measurement.
The reliability of a test and the associated standard
error of measurement are estimates of the
amount of error in the measurement.

DECISION ERRORS
False positive:
passing an incompetent examinee
False negative:
failing a competent examinee

NORM- & CRITERION-REFERENCED STANDARDS

NORM-REFERENCED
Relative
Based on peerperformance
Varies with each group
Cut-off point not
related to competence

CRITERIONREFERENCED
Absolute
Not related to peer
performance
Standard set prior to
exam
Referenced to a defined
level of performance

METHODS OF STANDARD SETTING

1.

Test-centred methods

Standards derived from hypothetical

decisions based on test content before the
test is answered.

2. Examinee-centred methods

Standards derived from reviewing examinees

performance before deciding cut-off score.

3. Compromise methods

Provide flexibility for adjusting the

standard based on the examinees
performance on the test.

NEDELSKY (1954) METHOD:

Example

Consider N judges and n MCQ items of 1 in 5 type

Judge A identifies 2 options in item 1 as those which a minimally
competent examinee should eliminate as incorrect.
MPL for that item for Judge A [MPLA1] = 1/(5-2) = 1/3
Similarly, in item 2 he identifies 3 options, giving an MPL A2 = 1/(5-3)
= 1/2
He repeats this process for each item.
The exam MPL for Judge A [MPLA] = MPLA1 +MPLA2 + MPLA3 +
.MPLAn
Similarly, Judge Bs MPL [MPLB] is determined
The MPL for the exam (= cut-off score) is: (MPL A + MPLB +
MPLC +.MPLN) / N

ANGOFF (1971) METHOD

Example

N judges consider 100 minimally competent examinees taking an

MCQ exam of n items.
Judge A estimates that, of these examinees, 50 should answer item
1 correctly, 20 item 2 correctly, 70 item 3 correctly, and so on to
item n.
The MPL for Judge A [MPLA] = (0.5 + 0.2 + 0.7 + . xn) / n X 100 =
(say) A%.
Similarly, for Judges B, C, D, E, ..N, the MPLs would be B%, C%, D
%, E% N%, respectively.
The MPL (cut-off score) for the exam is: (A% + B% + C% + D% + E%
+....N%) / N

EBEL (1972) METHOD

Example

Assume that Judge A assigns items in a 200-item MCQ test to the cells of a
relevance-by-difficulty matrix, as follows.
He then estimates the percentage of items in each cell of the matrix that a
minimally competent examinee should be able to answer correctly (as indicated
within the cell).
Each cell also includes the products of these two values.

ESSENTIAL

EASY
15 x 100% = 1500

MEDIUM
25 x 80% =2000

HARD
10 x 60% = 600

IMPORTANT

20 x 80% = 1600

40 x 60% =2400

20 x 50% =1000

ACCEPTABLE

10 x 50% = 500

25 x 40% = 1000

5 x 10% = 50

QUESTIONABLE 10 x 30% = 300

15 x 20% = 300

5 x 0%

EBEL (1972) METHOD - contd.

Example

The MPL for Judge A [MPLA] is then:

(1500 + 1600 + 500 + 300 + 2000 + 1000 + 300 + 600 + 1000
+ 50 + 0) / 200 = 56.25 %
Similarly, the MPL for Judges B [MPLB], C [MPLc], D [MPLD]
..N [MPLN] are determined.
The MPL for the exam (cut-off score) is:
(MPLA+ MPLB+ MPLc+ MPLD + ..MPLN) / N

PROPOSED EBEL MODIFICATION

EASY

MEDIUM

ESSENT.

6x 100% = 600

12 x 80% = 960

7 x 50% = 350

IMPORT.

12 x 80% = 960

24 x 60% = 1440

19 x 40% = 760

ACCEPT.

5 x 60% = 300

12 x 50% = 600

3 x 10% = 30

MPL:

HARD

=600 + 960 + 350 + 960 + 1440 + 760 + 300 + 600 + 30

=6000/100
= 60

HOFSTEE METHOD

fmax
Failure
Rate%

fmin

20

15

10

35

40

45

cmin

cmax

50

Cut-off
score(%)

A plot of cut-off scores for a given exam against

resulting failure rates is given

cmin = 40%
cmax = 45%
fmin = 10%
fmax = 20%

A = point representing cmin,fmax

B = point representing cmax,fmin
Line AB intersects the curve at a cut-off point of

CUT-OFF SCORE FOR 1 IN 5 MCQ

[FRACS PART 1]
Probability of guessing (=1 in 5)

= 20%

Total ignorance score

= 20%

Maximum possible score

=100%

Effective range of scores

100%

= 20% to

Mid-point of this range

= 60%

Additional factor (as PG exam)

= 5%

Nominal cut-off score (60%+5%) = 65%

CUT-OFF SCORES:
MARKER QUESTIONS

1. Comparison of exam scores

Mean score in this exam:

56.7%

Average exam mean score over last 4 years: 59.4%

Thus mean score in this exam is:
Assuming this candidate group is of same
standard as in last 4 yrs, this exam is:

2.7% lower

2.7% harder

CUT-OFF SCORES:
MARKER QUESTIONS - contd.

2. Comparison of marker scores

Mean score in this exam on previously used
questions (N=162):

62.5%

Mean score on same questions when they

were each last used:

60.5%

Thus, when compared with previous

candidates, this group of candidates, on
these items, scored (62.5-60.5)% =
Thus this group of candidates is:
than previous groups

2.0% higher
2.0% better

CUT-OFF SCORES:
MARKER QUESTIONS contd.

3. Estimating examination difficulty

Thus it is expected that their mean score
in this exam would be:
2.0% higher
But their mean score in this exam is: 2.7% lower
Thus this exam is really:

4.7% harder

CUT-OFF SCORES:
MARKER QUESTIONS contd.
4. Determining cut-off score
The cut-off level for an average exam is:

65.0%

Thus the cut-off level for this exam

should be (65 4.7)% =

60.3%

Cut-off score = 60.3%

HOFSTEE CURVE

20

Failure
Rate%

15

10

55

60

65

70

Cut-off
score(%)