• Embed Doc
  • Copy Link
  • Readcast
  • Collections
  • 1
    CommentGo Back
Download
 
SOLSTICE 2007 Conference, Edge Hill University 1
Computer Adaptive Testing in Higher Education: A casestudy
Mariana Lilley
1
, Trevor Barker 
2
, Carol Britton
3
 
1
University of Hertfordshire,m.lilley@herts.ac.uk 
2 
University of Hertfordshire,t.1.barker@herts.ac.uk 
3 
University of Hertfordshire,c.britton@herts.ac.uk ABSTRACT
At the University of Hertfordshire we have developed a computer-adaptive test (CAT)prototype. The prototype was designed to select the questions presented to individuallearners based upon their ability. Earlier work by the authors during the last five yearshas shown benefits of the CAT approach, such as increased learner motivation. It wastherefore important to investigate the fairness of this assessment method. In the studyreported here, statistical analysis of test scores from 320 participants show that in allcases scores were highly correlated between CATs and other assessment methods(p<0.05). This was taken to indicate that learners of all abilities were not disadvantagedby our CAT approach.
KEYWORDS
Student assessment, Computer-adaptive test, Assessment fairness
INTRODUCTION
The past ten to fifteen years have witnessed a significant increase in the use ofcomputer-assisted assessment in Higher Education. Hardware developments andsubsequent proliferation of computer technology in conjunction with the ever-increasingstudent numbers are amongst the main reasons for this trend (Freeman and Lewis,1998; O’Reilly and Morgan, 1999; Wainer, 2000; Joy et al., 2002). Computer-assistedassessments are applications that support student testing, from actual testadministration to scoring and student performance reporting. The benefits of thesecomputerised tools over traditional paper-and-pencil tests are well reported in relevantliterature and range from accuracy of marking to the potential to quickly assess largegroups of students (Pritchett, 1999; Harvey and Mogey, 1999; De Angelis, 2000; Masonet al., 2001).A significant number of computer-assisted assessments currently being used in HigherEducation are the so-called computer-based tests (CBTs). CBTs are traditionally not
 
SOLSTICE 2007 Conference, Edge Hill University 2
tailored towards individual students, as the same fixed set of questions is administeredto all students regardless of their ability within the subject domain.Conventional CBTs differ from computer-adaptive tests (CATs) primarily in the way thatthe questions administered during a given assessment session are selected. In a CAT,one question is administered at a time and the selection of the next question isdependent on the response to the previous one. In summary, whilst CBTs mimicaspects of a paper-and-pencil test, CATs mimic aspects of an oral interview (Freedle &Duran, 1987; Syang & Dale, 1993). To this end, the first administered question within aCAT is typically one of average difficulty. A correct response will make a more difficultquestion follow. Conversely, an incorrect response will cause an easier question beadministered next. By dynamically selecting the sequence and level of difficulty of thequestions administered to each individual student's proficiency level, the CAT approachhas the potential to offer higher levels of interaction and individualisation than thoseoffered by its CBT counterpart. This can, in itself, lead to increased student motivation(Lilley et al., 2004). Because of individual differences in ability levels within the subjectdomain being tested, the CBT static approach often poses problems for some students.For example, a given question might be too easy and thus uninteresting for one studentand too difficult and therefore bewildering to another student. More importantly,questions that are too difficult or too easy provide tutors with little information regardingstudent ability. We argue that it is only by asking questions at the boundary of what astudent understands that we can obtain useful information about what he or she haslearned. By adapting the level of difficulty of the questions to match the ability of thetest-taker, questions that provide little information about a given student can be avoided.Despite the predicted benefits of computerised adaptive testing, the approach hasreceived relatively little attention from British Higher Education institutions (Joy et al.,2002).The use of computer-adaptive tests (CATs) in Higher Education, as a means ofenhancing student assessment as well as fully exploiting the computer technologyalready available, is the focus of ongoing research at the University of Hertfordshire. Tothis end, a CAT application based on the Three-Parameter Logistic Model from ItemResponse Theory (IRT) was designed, developed and evaluated. Earlier work by theauthors showed the efficacy of the approach in the domain of English as a secondlanguage (Lilley & Barker, 2002; Lilley et al., 2004). Our current focus of research is theuse of computerised adaptive testing within the Computer Science domain.In the next section of this paper, the reader is provided with an overview on IRT andcomputerised-adaptive testing. We then report on the findings of an empirical study, inwhich over 300 students participated in summative assessment sessions using our CATapplication. Potential benefits and limitations of the CAT approach in addition to ourviews on how the work described here can be developed further are presented in thefinal section of this paper.
 
SOLSTICE 2007 Conference, Edge Hill University 3
COMPUTER-ADAPTIVE TESTS
Computer-adaptive tests (CATs) are typically based on Item Response Theory (IRT)(Wainer, 2000). IRT is a family of mathematical functions that attempts to predict theprobability of a test-taker answering a given question correctly. Since we aimed todevelop a computer-adaptive test based on the use of objective questions such asmultiple-choice and multiple-response, only IRT models capable of evaluating questionsthat are dichotomously scored were considered to be appropriate. The Three-Parameter Logistic (3-PL) Model was chosen over its counterparts One-ParameterLogistic Model and Two-Parameter Logistic Model as it takes into consideration thequestion's discrimination and the probability of a student answering a question correctlyby guessing.Equation 1 shows the mathematical function from the 3-PL model used to evaluate theprobability P of a student with an unknown ability
θ 
correctly answering a question ofdifficulty b, discrimination a and pseudo-chance c. In order to evaluate the probability Qof a student with an unknown ability
θ 
incorrectly answering a question, the function
)(1)(
θ θ 
PQ
−=
is used (Lord, 1980). Within a CAT, the question to be administered nextas well as the final score obtained by any given student is computed based on the set ofprevious responses. This score is obtained using the mathematical function shown inEquation 2 (Lord, 1980).
Equation 1: Three-Parameter Logistic Model
)(7.1
11)(
ba
eccP
−−
+−+=
θ 
θ 
 Equation 2: Response Likelihood Function
 j j
u jn ju jn
QPuuu L
−=
∏
=
1121
)|,...,,(
θ 
 
By applying the formula shown in Equation 1, it is possible to plot an Item CharacteristicCurve (ICC) for any given question.
Figure 1: 3-PL ICC for a correct response where
a 
=1.5,
b 
=0 and
c 
=0.1
00.10.20.30.40.50.60.70.80.91-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
Ability
θ
   P  r  o   b  a   b   i   l   i   t  y  o   f  a  c  o  r  r  e  c   t  r  e  s  p  o  n  s  e
 
of 00

Leave a Comment

You must be to leave a comment.
Submit
Characters: ...
02 / 11 / 2011This doucment made it onto the Rising List!
You must be to leave a comment.
Submit
Characters: ...