COMPUTERBASEDTESTING

INTRODUCTION

ModerntesttraditionbeganwiththeFrenchpsychologistAlfredBinetwhocreated thefirstintelligencetestin1905.Thistestwascreated toidentifychildren whowere likelytoperformbadlyatschool(Schultz&Schultz,1992).Thetestwasasocalled individualtest,becauseoftheindividual contactbetweenthepsychologistandthe child:Thepsychologistaskedaquestionandthechildgaveananswer,justlikeinan intervieworinaconversationbetweentwopeople.Nowadaysthistestisstill of importancebecauseofitsbasicdesign.ThisdesignwasthatBinetchangedtheorder of thequestionsdependingonthechildsanswers.Ifachildgaveincorrectanswershe askedeasierquestionsuntilthechildgaveaminimumnumberof correctanswersina row.ThenBinetmovedontoamoredifficultlevelofquestionsuntilthechildgavea certainnumberofincorrectanswersinarow.Inotherwords,Binetadaptedtheorder ofquestionstothechildscompetencylevel. Individualteststakealongtimetoadminister.DuringtheFirstandtheSecondWorld Wartherewasaneedforprocedurestoselectandassign alotof armyworkersina veryfastbutaccurateway.Anewmethodofgroupadministered,orcollective testing,cameintouse.Now,oneproctoralone,thatisapersonwhoisinchargeofthe testsession,couldgiveinstructionstomanytesttakersatthesametime. WellknowngroupadministeredintelligencetestsusedduringWorldWarIarethe Army a (readalpha),averbaltest,and theArmy b(readbeta),anonverbaltestfor peoplewhocouldnotreadorwrite.DuringWorldWarIImorethan9millionpeople weretestedwiththegroupadministeredArmyGeneralClassificationTest(AGTC)in theUnitedStatesofAmerica.Technologyin thosedayswasrathersimple.Materials usedtoadministerthesetestsweresomethingtowriteon paper andsomethingto writewith apencil,hencethenamepaperandpenciltestsorp&ptests. Inthe1960sthefieldinpsychologycalledcognitivepsychology hasintroducedthe computerintopsychologybystatingthatcomputerprogramsoperatesimilarlytothe humanmind.Accordingtocognitivepsychologistsacomputerhadamind,useda languageandwasabletoprocessinformation(Schultz&Schultz,1992).Sincethen, computertechnologyhasbecomeapartofeverydaylifeandhashadanenormous impactontestuseandtestconstruction(Hardinge,1997Jansen,1997).

COMPUTERBASEDTESTING

Inthisreadingwewillanalysethep&ptestsinordertodiscoverwhattheirrelationis withtechnologyandcomputer.Then,currentandfutureuseofcomputersintesting willbelookedatindetail. PAPER&PENCILTESTS Definition Thegroupadministeredtestsdescribedabovewerethefirstpaperandpencil(p&p) testsor astheysometimesarecalledprintedtests. Ap&ptestisasetofquestions(oritems)tobeansweredbythetesttakerwithatool towriteonadocument.Thefollowingmaterialsareusedinap&ptest: abookletcontainingprintedquestions ananswersheetonwhichthetesttakermarkstheanswers apencilorapeninordertomarktheanswers. P&ptestsaremainlybasedonclassicaltesttheory.Thistheoryassumesthateach observedtestscoreiscomposedofatruescoreandanerror.So,inmostcasesthe observedscoredoesnotequalthetruescore.Consequentlytheobservedscoremaybe higherorlowerthanthetruescore(seethesectiononclassicaltesttheoryformore informationonthissubject).

Examplesof p&ptestsbasedonclassicaltesttheory aretheDAT(Differential AptitudeTest)andtheBennet,amechanicalcomprehensiontest. Afteracandidate completedatest,theresponsesgiventotheseparatetestitemsneedtobetransformed intoatestscore thatmeanssomething.Thiscanbedonemanuallyorbymeansofa computer.

Wenowwilldealwithhowthecomputercanbehelpfulinscoringtheanswerson p&ptests togettoameaningfultestscore.

Evolutionoftechnologyforthetransformationofp&presponsesintotestscores Fourimportantevolutionsinsupportingthescoring ofp&ptestanswerscanbe identified.Thesearetheimplementationof 1. Punchcards 2. OMR(OpticalMarkReader) 3. OCR(OpticalCharacterRecognition)

COMPUTERBASEDTESTING

4. Emarking

Punchcardswerecardsinwhichapatternofholeswascuttorepresentinformation, whichwasthenreadbyacomputer.Thismethodisnotusedanymore.Usingthe OpticalMarkReader(OMR)testtakersanswertheitemsfillinginasocalledlozenge (asmallarea)onaform.Themarkreaderisamachinethatrecogniseswhetherthe lozengehasbeenfilledornot.Then,thisinformationcanbetransformedintotest scores.OCR(OpticalCharacterRecognition)wasanimprovementoftheOMR systemandproducedmorereliabletestscores.OMRwascreated torecogniseif certainblocksontheformwherefilledin ornot,whilstOCRwasdevelopedtobe abletorecognisewrittencharactersandsigns.Currently,socalledemarkingisonthe marketandmakesitpossibletorecognizeandrecordonscreenhandwrittenanswers bycandidates. Disadvantagesofp&ptests Firstofall,usingap&ptestistimeconsumingasitrequiresalotofadministration timeduringpreparation,administrationandprocessingofthescores.Second,itis slowandinaccuratewhenresponsesaretransformedintotestscores.Third,itis lackingflexibilityfromtheperspectiveoftheorganizationthatusesthetest.For instance,itisnotpossibletopresentdifferenttests oneaftertheotherwithout intervention,or tochangethenumberoftests,nortocomposedifferentsetsoftests. Thismeansthatyoucannotadjustyourtestoritsitemstothecandidatescompetency ifthetestyouoriginallypresentedproofstobeeithertoodifficultortooeasy. Furthermorep&ptestshaveanoldfashionedimageandstyle.Andfinally,the restriction ofthiswayoftestingisthatithasafixedtestingtimeandafixedorderof items. Thesedisadvantagesimplythatcandidateshavetogothroughalong,nonattractive testsession,withmanyquestionsthatcouldbetoodifficultortooeasyinrelationto theirowncompetencylevels.Allcandidatesmustfilloutthesametestitemswithout thepossibilityofindividualisedtestpartsadjustedtofitthecandidate.Unfortunately, therealsowillbelearningeffectswhenpeopletakemorethanonetestsession.This meansthatpeopleprobablywillobtainbetterscoreswhentheytaketestsmoreoften thanonceandthisespeciallyaffectsp&pteststhathavefixedcontent.

COMPUTERBASEDTESTING

Thequestionthenariseswhetherthereareanyalternativestop&ptests.Answering thisquestionistheobjectiveofthenextsection.

COMPUTERBASEDTESTING Intheprevioussectionwehavebeenwitnessesofhowcomputerusecansupportp&p testing.Nowwewill checkoutwhetherthecomputercanbeusefulalsointackling thedisadvantagesofthep&ptests. Startingthisexplorationitwouldbegoodtoknow whatishiddenbehindthetermcomputerbasedtesting(CBT).

Definition Avarietyoftermsanddefinitionsareusedtodescribetestingwithcomputers.Sooner orlaterthereaderwhoisstudyingtherelationbetweencomputersandtestingwillbe confrontedwiththefollowingterminologyandabbreviations:

CBT(ComputerBasedTesting) CAT(ComputerAssistedTestingComputerAidedTestingComputer AdministeredTestingComputerAdaptiveTesting) CBA(ComputerBasedAssessment) ComputerisedAssessment ComputerisedTesting CAA(ComputerAssistedAssessment).

Letusnowturntothemeaningsoftheseterms. Ananalysisofdefinitionsanddescriptionsofthesetermsrevealsthatsome descriptionsarerathervague,forinstance:Administeringtestselectronicallyusinga computerorusinganelectronictestingsystem.Thesekindofdescriptionsdonot makeacleardistinctionbetweenp&ptestsandcomputerbasedtesting(CBT), becausecomputersmayalsobeusedasasupporttofacilitatecertainpartsofp&ptest procedures,aswesawintheprevioussectiononp&ptests.Furthermoreitcaneasily beseenthatitisconfusingtouseCATasanabbreviation,becausethisabbreviation canpointtodifferentmeanings.WhenCATisusedtheauthororspeakercertainly mustexplaininwhatsenseheusesthisabbreviation.

COMPUTERBASEDTESTING

Themaindifferencebetweencomputerbasedtesting(CBT)andp&ptestingseemsto betheuseofonscreendevicesbyCBT.Itthereforeseemsjustifiedtogivetwo descriptionsofCBT:anarrowoneandabroadone. WhenwespeakaboutCBTastheassessmentofapersonscapacitiesthrough softwareandhardwareusingonscreendevices,weusethenarrowdefinitionofCBT. ThetermCBTshouldperhapsbestbeusedforthisnarrowdefinitiononly.Whenwe talkaboutadministeringtestswith supportofhardwareandsoftware,orwithsupport ofacomputer,weusethedefinitioninabroadsense.Thisseconddefinitioncouldbe usedforinstanceforap&ptestwhichisscoredbyacomputer.Herewesuggestto betterusethetermCAA(ComputerAssistedAssessment). BecausethewidelyusedtermCATismostlylinkedtosocalledadaptivetesting (duringadaptivetestingthedifficultyoftheitemsisadaptedtothecompetenciesof thetesttaker)itisadvisabletopreservethisabbreviationsolelyforComputer AdaptiveTesting.

StagesintheevolutionofCBT CBTfindsitsoriginsintheearly1960s(Jansen,1997).Fromthenoncomputer elementswereregularlyusedasadministrativesupportfortesting,mainlytoconvert testanswersintotestscoresasdiscussedinthep&ptestsectionbefore,andto introducenamesandscoresofthecandidates.TheBelgianDepartmentofDefence introducedautomatedprocessesattheRecruitmentandSelectionCentreduringthe 1970s.Punchcardswereusedforthecorrectionofthetests.Duringtheeighties, graduallyseveralcomputerisedtestsystemscameintouse.Meanwhile,OMR(optical markreader)systemswerereplacingpunchcards.Inthesecondhalfofthenineties theCBTsystemwasfurtherelaborated.Recently intheBelgianDepartmentof Defence,from2001onward,nomorep&ptestswereusedandafullyautomated systemwasintroduced. WithrespecttogeneralCTB,McBride(1998)observedthreestagesintheevolution ofCBTandcalledthemthethreegenerationsofproblems. Duringthefirstgenerationcostwasthecentralproblem.Atfirst,thecomputer equipment,development,softwareandmaintenancewereextremelyexpensive.This isthecasewitheachnewlyimplementedtechnology.McBridementionsthecaseof theU.S.DepartmentofDefencesASVABsystem(ArmedServicesVocational AptitudeBattery).ThisbatteryisaseriesoftestsdevelopedbytheDepartmentof
5

COMPUTERBASEDTESTING

Defenceinthe1960sforenlistmentpurposes(Hardinge,1997).In1988,costsfora CBTversionwereestimatedbetween25millionand50millionUSdollars,which wasmoreexpensivethanthep&pversionoftheASVAB.Eightyearslater,thecost wasestimatedlessthanthreemillionUSdollars,whichmadeitbeneficialtousethe CBTversion.In2002thesituation withrespecttocosthaschangedevenmore.For instance,primaryandsecondaryschoolsinSouthBend,Indiana,U.S.A.,now administertheirtestsonthecomputer.ThenewtestprograminSouthBendcostsonly 3000USDollarmoreayearthantheoldp&ptestingprogram.Nowadays,computer equipmentismuchcheaperthanbeforeandthisofcoursedecreasescosts. McBridecalledthesecondgenerationproblemsProblemsin convertingPrinted TeststoComputerAdministration.Thissecondgenerationwasaboutproblemsthat arisewhenp&ptestsaretransformedintoCBTtests.Afirstproblemisthat transformationscouldcausedifferencesintestresultsbetweenthep&pversionand theCBTversion(Neuman&Baydoun,1998).Thisisespeciallythecasewhentest takersareconfrontedwithatimelimit,thesocalledspeedorspeededtests.Oneof thereasonsfordifferencesinresultsmaybethattheactofmarkingtheanswersheet isabsentintheCBTversiontheabsenceofthisactcouldincreasethespeedof answering.Asecondproblemwithinthisgenerationisofadifferentkindandhasto dowithdifferencesincomputertypes.Whathappenswhenthetypesofcomputers thatwereusedintheCBTsystemarenolongeravailable?Forexample,morerecent computersaremuchfasterthantheonesofjustafewyearsago.Alsoatpresent, computersmayhaveadifferentkeyboard,whichcouldcauseadifferenceinresponse reactiontime. Inthethirdgeneration,problemswerecentredon developinginnovativeteststo measureabilities.ThemostinterestingthingaboutCBTisthepossibilitytodevelop newmeasuresofabilitythatcouldnotberealisedthrough p&p.Thesenewkindsof testingwillbediscussednow.

ADVANTAGESOFCBT Psychotechnicalinnovations Thefeaturesof CBTcan gobeyondthelimitationsofthep&ptests.P&Ptestsare only basedonclassicaltesttheory(CTT).CBTtestscan eitheruseCTTasabasisas well,butitcanalsouseanewtesttheoreticalapproach(Crocker&Algina,1986).

COMPUTERBASEDTESTING

ThisnewapproachiswhatMcBrideisreferringtowhenhespeaksaboutthird generationproblems. InFigure1threeCBTproceduresareshown.TCTs(TraditionalComputerised Testing)mainworriesarewithtransferringp&ptestsintocomputerversions.IGT (ItemGenerativeTesting)deliversanengineon acomputerthatgeneratesitems.TCT andIGTarebasedon classicaltesttheory.Incontrast,CAT(ComputerAdaptive Testing)isbasedonanewtheory,IRTorItemResponseTheory.Thistheory estimatestheabilitylevelofacandidateonthebasisofhisorherresponsesto previouslyadministereditems.SometimesIRTisreferredtoasModernTestTheory asopposedtoClassicalTestTheory. CBT

CTT

IRT

TCT Limitedparallel versions

IGT Infiniteparallel versions

CAT

Figure1.ComputerBasedTestingconcepts Anoverviewof ItemGenerativeTesting(IGT)iselaboratedinIrvineandKyllonen (2002).ThereadermaygetintroductionstoCATinWainer,Dorans,Eignor, Flaugher,Green,Mislevy,SteinbergandThissen(2000),andinWeiss(1983). Furthermore,SteegeandFritscher(1991)havetreatedsomeoftheadvantagesof CAT.CTTandIRTarecomparedwitheachotherintheworkofCrockerandAlgina (1986). TraditionalComputerisedTesting(TCT).Withtheriseofnewcomputertechnology it becomespossibletoconstructdifferenttestversionsmeasuringthesame competencieswithdifferentitems.Inotherwords,nowitispossibletodevelop equivalentformsof thesametest.Theseequivalentformsarecalledparalleltest versionsorparallelversions.Theitemsarestoredinthememoryofacomputer.The actofcreatingsuchadatabaseisoftenreferredtoasitembanking.However,onlya limitednumberofparallelversionspertestareavailableusingthistechniqueoftest construction:inmostcasesthereareonly twoparallelversions.Twoexamplesof

COMPUTERBASEDTESTING

companiesthatdeliverthiskindoftestbatteriesareCPM(ConsultantsinPersonnel Management)andSHL (SavilleandHoldsworthLtd).Theypresenttestsindifferent languagesandareabletodeliverordevelopparallelversionsoftests. ItemGenerativeTesting.Thisformoftestingmakesitfeasibletoproduceaninfinite numberofparallelversions.Thereisnoneedtokeepitemsinadatabank,because itemsarecreatedautomaticallybyacomputeralgorithm(asetofrulestosolvea problem)atthetimeoftesting.Inotherwords,itemsaregeneratedautomaticallyjust beforetestingstarts,hencethenameItemGenerativeTesting.Hardinge(1997) mentionedthatthisnewmodelservesasabasisforatestbatteryforselection purposesintheUnitedKingdomArmy.TheBARB(BritishArmyRecruitBattery) wasintroducedin1992.Instructionsaregivenonscreen andeachcandidateanswers auniquesetofitems,becauseoftheuniquefeatureof endlessparallelversions. Nevertheless,allsetsofitemsareofequaldifficulty.Touchscreensareused:by touchingthecomputerscreenacandidatemarkstheanswers.Testresultsare correctedinrealtime.AnotheradvantageisthatIGTprinciplesprovidethebasisfor developingadaptivetestforms. ComputerAdaptiveTesting(CAT). ComputerisedAdaptiveTestingisavery clever wayoftesting.Thecomputerprogramsearchesforanappropriate testitembasedon thecandidatesresponsetothepreviousquestion.Onthebasisofanalgorithm,items areselectedfromabankofitems.Infact,eachtimethecandidatehasresponded,the computerprogramadaptstothecandidatesanswer.Atthestartofthetestingsession, anitemofmediumdifficultyisshown.Thereafter,eachgoodanswerisfollowedby theselectionofaharderitemandeachincorrectanswerisfollowedbyaneasieritem (Wainer,etal.,2000Weiss,1983).Thissoundsfamiliartoourears.Indeed,this procedureremindsusofthetechniqueAlfredBinetusedtoassessyoungchildren withhisintelligencetest. Technologicalprogress Alongsidepsychotechnicalinnovations,CBTjoinsvarioustechnologicalnovelties together,solvingmanydisadvantagesofp&ptests.Scoresareaccuratelycalculatedin realtimeandcanbecommunicatedtothecandidateimmediatelyafterthetestsession. Reportsandscoresaregeneratedautomaticallyinordertoallowinterviewersor teacherstointerprettestresultsassoonaspossible.Testtimecanbereducedby flexibledirectingcandidatesthrougha testaccordingtothecandidatescapacities.

COMPUTERBASEDTESTING

Flexibilityisfurtherenhancedbyintegratingtestsinanetwork,whichpermitsto regulatetestadministration.ThenameforsuchanetworkisTestManagerorTest AdministrationSystem.Automatedcalculationsandstorageofdataandtestscoresof candidatesmakeresearchaloteasier.Ideally,theCBTsystemmustallowtheuseof differentkindsoftestsandassessmentinstruments,suchasmultiplechoicetests, questionnaires,testsusingimagesorgraphics,andopenendedanswers. Usingsuchpsychotechnicalandtechnicaladvantagesshouldresultinasystemthatis moreuptodate,attractiveandclientbasedthancomparablep&ptestsystems. Studiesindeedshowthatcandidatesattitudestowardstakingselectiontestsvia computersgenerallyarepositive(Steege&Fritscher,1991).

TESTSTAGESANDAUTOMATION Figure2givesanoverviewofthepsychotechnicalandtechnologicalprogresses duringthelastcentury.Thefirstcolumngivesatestexampleofacertaintimeperiod, whichwasdiscussedearlierinthisreading.Inthesecondcolumnthetestconceptor teststageisdescribedandinthethirdcolumntheappropriatedegreeofautomationis shown.Forinstance,theBinetSimontestisdescribedasanindividualadaptive noncomputerised(ormanuallyadministered)test. Test BinetSimon/ Terman Merill Army a,Army b BDSB70 BDSB80 BDSB90 ASVAB,IGT Group p&p Group p&p Coststage Conversionstage Innovativestage CBT Testconceptorstage Individualadaptive Degreeofautomation Manually Manually CAA Computerised

Non Computerised

Figure2.Teststagesinrelationtodegreeofautomation

Therearetwotypesofgroupadministeredp&ptests:ontheonehandthemanually filledoutformwithitsmanualconversionintotestscores,ontheotherhandtheCAA (computerassistedassessment)form,includingcomputerisedsupportfor administrativepurposesandconversionsofanswersintotestscores.InFigure2,the

COMPUTERBASEDTESTING

threeCBTgenerationsusedbyMcBridearegroupedasonscreendevicesorCBT. BothCAAandCBTcanbeseen ascomputerisedtestprocedures.Becausesometests existindifferentformsorhavebeendevelopedindifferenttimeperiodsitis possibleforthesametesttobeclassifiedunderdifferenttestconcepts.Anexampleof CAAandCBTsystemsistheselectionbatteryusedbytheBelgianDefenceForces, namelytheBelgianDefenceSelectionBattery(BDSB).Initsoriginitwasdeveloped asap&ptestbattery,BDSB70wasthefirststepintheCAAandstartedinthe1970s. Lateron,theBelgianDefenceStaffwasveryreluctanttodevelopaCBTbattery becauseofthecosts.Theychangedtheiropinionwhenautomationofotherpersonal managementactivitieswasintroducedduringthe1980s:BDSB80 wasthesecondstep towardsCBT.Then,inthelate1990s,thedecisionwastakentoduplicatethep&ptest onthecomputerresultinginfiftypercentp&ptestdeliveryandfiftypercentCBTtest delivery.Finally,in2000,thechoicewasmadetocreateafullycomputerisedtest battery.

InFigure2thesmallupwardarrowsindicatethatcompanieswhotrytointroducea newsophisticatedtechnologymaybeconfrontedwithfailuresandthencanbeforced tofallbackontechnologyofalesssophisticatednature.Anexampleofthisagainis theBDSB.DuringthecostperiodCBTtechnologywastooexpensive,resultinginthe continuationofthep&psystemwithaCAAnature.Meanwhile,CAAchangedfrom punchcardstoOCRandatacertainpointitbecamelogicaltointroduceCBT.The costperiod thereforecanbeinterpretedasatransitionperiodbetweenCAAandCBT. Insum,itispossiblethatcompaniesaresituatedindifferentteststageswithregardto differentpartsoftheirtestbatteries.

DEVELOPINGACBTSYSTEM WaystodevelopaCBTsystem WhendevelopingaCBTsystemtheultimategoalmustbeanintegrated networkthat allowstheusertoadaptthekindandnumberofassessmentinstrumentsdependingon hisobjectivesandthecontextoftheassessment:itallowstoflexiblyhandleor managetestuse.ForthisreasonaCBTsystemisoftencalledaTestManager.A TestManagerisnotaperson,butasystemthatgivestheusertheopportunityto choosetestsoutofadatabaseholdingadiversityoftests,andtocomposedifferent testbatteries.ThefollowingfourwaystodevelopaCBTsystemwillbediscussed:
10

COMPUTERBASEDTESTING

buyingasystem developingasystemindependently anincompanysystem hiringasystem.

Forauserthreewaysexisttobuyanetwork:theusermaybuyacommercialofthe shelfproduct(COTSproduct,thatisaproductthatispurchasedlikeitoriginally was constructedbythedeliveringcompany)atailormadenetworkorasemitailormade network. TheTestManagerboughtasaCOTSproductmustbeacceptedandusedasitwas designedbythedeliveringcompany.Adjustmentsdonotbelongtothecontract. The usersometimesdependstotallyonthecompanythatdelivers,whichmightresultina lessflexiblebutlessexpensivesystem.OneofthecompaniesdeliveringaCOTStest managementsystem theCareerHarmonyAssessmentManagementSystemor CHAMisConsultantsinPersonnelManagement. Inthecaseof developingatailormadenetworkindependently,thesystemisfully constructedonthebasisoftheneedsexpressedbytheuser.Adjustmentsarepartof thecontract. Tailormadeworkisveryexpensiveandtakesalotofplanningandtime. Whenusingthesemitailormadeapproach,theexistingincompanynetworkis adapted,oraCOTSproductisadapted,incooperationbetweentheuserandthe deliveringcompany.Onepossiblesolutionistocooperatewithuniversities.Inthis wayaflexiblenetworkmightbeputdown,meetingtheuserscriteriaascloselyas possible. Besidesbuyinganetwork,theusermaywishtoeitherdevelopabrandnewCBT networkincompanyortohiretheservicesofacompanythatorganisesthewhole CBTassessmentprocedure.Ofcourse,developingaCBTsystemwithoutanyhelp fromanexternalconsultancycompanyrequiresalotoftechnicalandpsychotechnical expertiseandresourcesinsidethecompany.Ifthereisnotenoughexpertiseinthe ownorganization,analternativemaybetohireaCBTassessmentprocedure.There aremanycompaniesspecialisedinorganisingCBTassessmentsandtheusermight havecandidatestestedincompanyoratlocation.Thenextcasesillustratewhich servicesareavailable.

11

COMPUTERBASEDTESTING

ETS(EducationalTestingServices)annuallydeliversmorethan1,3millionCBTtests worldwide.CAT*ASI(ComputerAdaptiveTechnologies,Inc.*Assessmentsystems, Inc.)managesanationwidenetworkoftestcentresandaccommodatesCBTsessions forhighstaketests(highstakestestsareusedfordecisionsofgreatimportance,such asadmissionsessionsforuniversitiesorforajob).Theyofferservicessuchastesting technology(graphics,differentquestiontypes),thedevelopmentofnewitems, managementoftestsessions,deploymentanddeliveryofthematerials,and processingofdataandreports.

ThedevelopmentofaCBTsystem Inmostofthecases,userswantingtodevelopaCBTsystemwillhavetoperforma marketanalysisandwriteRequestsforProposals(RFP)inordertoreceivetenders fromcompanies.Thebasicunderlyingprincipleisthesameasfortraditionaltest development.Thetestdevelopershouldbearinmindthatthecomputerisonlya meanstosupportadministrationandconstruction,andnotanendinitself(foran introductionintestingpsychology,seeMcIntire&Miller(2000)fordesigninga tests,RusselandPeterson(1997)formsagoodstartingpoint). Ofcourse,CBTrequiresspecificneeds.Forinstance,itmustbeclearwhetherthetest objectivesneedspecialhardware,suchasjoysticksforpsychomotorassessmentinthe caseofpilotordriverselection.InParshall,Spray,KalohnandDavey(2002)advice canbefoundconcerningthepracticeofimplementingCBT.Specialmattersof concernarefurthermoresecurityandmaintenance(InternationalTestCommission, 2005).Inordertokeepitemsanddatasafe,theCBTsystemmustofferthepossibility ofrestrictedaccessviapasswords.Inthatwaytheusercandecidewhohasaccessat whatmomentandatwhatlevel.Differentlevelscouldincludeproctors(test administrators),developers,psychologistsandmanagers.Second,thedatabase containingpersonneldataofthecandidatesmustbesecured.Third,thepossibility mustexisttocreatebackupsofthedata.Andfinally,itisimportantthatthesystemis securedagainstcurrentorelectricityinterruptions. ThemainpointswithregardstomaintenanceofaCBTsystemcanbesplitintwo ways.Itcanbeassuredfirstlyby thedeliveringcompany whocanofferatraditional helpdesk,whichmeansthatproblemsaresolvedviaemailorvialocalinterventions. Amoresophisticatedformofhelpdeskcanbeestablishedviaaremotecontrolor distancehelpdesk,on thebasisofadirectlinkbetweentwocomputersoverthe
12

COMPUTERBASEDTESTING

Internet.ThenameforthisremotecontrolhelpdeskisVirtualPrivateNetwork (VPN).

IMPLEMENTATIONOFCBT Dependingonthesetting,thequestionariseswhetheritispracticalandcosteffective toapplyCBT.First,CBTisespeciallyattractiveandeffectiveinthefieldoflarge scaleselection,largescaleeducationalprogramsandwherethereisaneedfor classificationandallocationoflargenumbersofcandidates(Bennet,2001Steege& Fritscher,1991Zakrewski&Bull,1998).Ontheotherhand,CBTcanalsobevery usefulinsituationswithonlyafewcandidateswherecomplexskillsneedtobe measured.CBTcanmeasuresuch abilitiesmuchmoreeffectively,they arenot measuredaswellbyp&ptests.

InthenextsectionexamplesofCBTinthreedomainswillbediscussed.Thesethree domainsareselection,educationalandschoolprogramsandtrainingandappraisal.

Selection InthedomainofthemilitaryselectiontheASVABandBARBprojectshavealready beenmentioned.OtherCBTbatteriesareMicropat(atestbatteryforhelicoptercrew) andpilotselectionbatteries.InthedomainofStateDepartmentselectionor GovernmentselectionCAT*ASIisoneofthecompanieswhichorganisesCBT sessions.Bell,IBMandAT&Tarelargeprivateorganisationswhichhavedeveloped theirownCBTselectiontools(McBride1988).

EducationalandSchoolPrograms InthiscontextCBTandCATarefrequentlyusedforadmissionissuesinrelationtoa schooloreducationalprogram.AwellknownadmissiontestistheGMAT(Graduate ManagementAdmissionTest).OtherCBTtools,likeTOEFL(TestofEnglishasa ForeignLanguage),formapartofevaluationandassessmentprocedurestodetermine themasteryleveloftheEnglishlanguage.In TheNetherlandstheCentralInstitutefor TestDevelopment(CITO)istheStateDepartmentproducerofnationwideCBT tests,andisspecialisedinCATtechniques.AfinalexampleisthatofLievensand Coetsier(2002)whohavereportedonSituationalJudgmentTestsinstudentselection.

13

COMPUTERBASEDTESTING

Trainingandappraisal ExamplesofCBTwithregardtotheassessmentofcomplexskillsorcompetencies asforinstanceneededforaircrew,airtrafficcontrollersorpilotsaremanagement testsandsimulationtests.TheCanadianAirForceimplementssuchasimulator,the CanadianAutomatedPilotSelectionSystem(CAPSS).CBTmayalsobeappealing whenitisdifficulttotrainorevaluateskillsinrealsituations.Thismayoccurwhena trainerneedstoevaluatewhethertraineesmasterobservationtechniquestobeused duringAssessmentCentre(AC)procedures.ACproceduresarea seriesofindividual andgrouptestsduringwhichcandidatesareobservedandevaluated.Imaginethatthe traineeneedstopracticeobservationtechniquesduringareal selectionprocedure. Thiswouldbeveryinefficient.Inordertoavoidthesepracticalproblemsthetrainees couldtakeaCBTACexercise.SuchanexercisewasdevelopedforuseintheDutch FlemishOpenUniversitystudyprogramforsocialsciences.Traineesinteractwiththe computerprogramusingdifferentmediatechniques.Traineesgetinstructions, observefourcandidatesonthecomputerscreenandevaluatethefourobserved candidates.Thescoresgivenbythetraineesthenarecomparedtoexpertscores, whichappearonscreen.Inaddition,traineesperformaninterviewwithacandidate viathecomputerscreen.Finally,traineesdecideonafinalscore,whichiscompared toascoregivenbyanexpert.Onlinehelpandcommentsareproducedbythe computer.Inthiswaypersonnelcostsfortraineeguidanceisminimised.Another advantageisthattraineesgenerallyseemsatisfiedwiththisprocedure.

Every advancementhasitsdownsides,andCBTmakesnoexceptiontothissaying. Therefore,inthenextsectionsomeCBTproblemswillbehighlighted.

PROBLEMSRELATEDTOCBT IngeneralCBTisusedmoreandmore.Nevertheless,manypotentialusersare reluctanttoimplementCBTsystems.Sixproblemareasformpossiblegroundsfor thisreluctance. Firstofall,thereisageneralruleconcerningtheacceptanceofnewtechnologies:at thestartofthenewtechnology,acceptanceisverylow,becausepeopleareusedtothe oldtechnology(Jansen,1997).Somepeoplestillthinkintermsofp&ptests.In addition,itshouldbenotedthatcomputerbusinessiscurrentlyrecoveringfromthe recentcrisisinthetechnologicalworld.

14

COMPUTERBASEDTESTING

Second,alotoforganisationsstillarestrugglingwithproblemsrelatedtothep&p timeperiod,suchashowtointegratetheadministrativepartofthetesttakingintothe computer(Jansen,1997).TheseproblemsaresituatedintheCAAgeneration. Thirdly,anothersetofproblemsrelatestothefirstgenerationofCBTproblems. PurchasingordevelopingaCBTsystemcouldappeartobeveryexpensive,certainly ifacomputersystemisnotalreadyinstalled.Furthermore,asJansen(1997)specifies, abudgetforCBTisnopriority.Instead,organisationsprefertoinvestinthe computerisationoftheiradministrationandoffices. Fourth,equivalenceofCBTtestversionsandp&ptestversionsshouldbechecked (Neuman&Baydoun,1998),especiallyinthecontextofspeededtests.Thiskindof problemscanbereferredtoasCBTproblemsofthesecondgeneration. Inthefifthplace,thecreationandfacilitationof theinnovativetestgenerationofCBT havetheirownspecificdifficulties.Thedangerofcompletedependenceon technologyisrealistic.Thereisaneedforlogisticsupport,e.g.enoughcomputers, electricity,roomsifnot,CBTsimplycannottakeplace.Incontrast,p&ptestsare moremobile,becausetheycanbetakentoeverylocationonecanimagine.Also,there aresomespecificproblemsrelatedtoCAT.First,eventhemostdifficultitemscanbe memorisedbycandidates.Consequently,theitembankmustbelargeenoughsothat thesmarterpeoplewillnotbeabletomemoriseitemsandrecognizethemlateron. FortheconstructionofaCATdatabankingeneral,manyitemsandcandidatesare needed.Furthermore,theconstructionofopenendeditemsisadifficultmatterin CAT.AsfortheIGT,critiquesmentionthatthesetestsarealittlebiasedtowards measuringcandidatesworkingmemoriesandspeededaspectsoftheirperformance (Hardinge,1997). Finally toclosethissectionfocusingonCBTrelatedproblemsthedevelopment andimplementationofCBTrequiresateamofexpertscomposedoftestdevelopers withexperienceinthefieldofCBTtechnology,computerscientistsforanalysisand programming,andstatisticalexperts.

FUTUREDevelopments AccordingtoRusselandPeterson(1997)technologywillprobablycontinuetorapidly changetheworldforthenextdecades.Asmoreandmorecompaniesbecome accustomedtocomputersandCBT,newwaysofCBTwill spreadmoreandmore. Thefollowingchangesareverylikelytoimpactuponthefuture.

15

COMPUTERBASEDTESTING

Achangeseemsplausiblefromkeyboardandmouse,totouchscreenandpenbased computers.Atthemomentresearchisfocusingonhowtoconvertwritten material ontothecomputerscreen.Touchscreenshavealreadybeenimplemented,for instanceduringtheBARBproject. AsecondtendencymightbetheincreaseoftestsusingvisualstimuliinCDROM technology.Thesekindsoftestshavealreadybeenconceived,mainlyasVideoBased SituationalTests. Third,anotherfuturedevelopmentcouldbetheimplementationofnewtestfeatures. Inthepast,testswerelimitedtowrittenandoralassessmentprocedures.Currently, featureslike(stereo) sounds,animation andfullmotionvideo,andinteractivetest simulations,seemtobepromisingandtypicalforanexpandingmultimedia technology(Drasgow&OlsonBuchanan,1999McBride,1998Russel&Peterson, 1997). Inthefourthplace,CBTmaycreatepossibilitiestodiscoverandanalysenew psychologicalconstructs,aswasdonebyKyllonenandhisteamsettinguptheLAMP (LearningAbilitiesMeasurementProgram)(Steege&Fritscher,1991).CBTalso makesitpossibletoimprovethemeasurementoftraditionalpsychologicalconstructs. Inthefifthplacethereisthegrowingimportanceof theInternet(WebBased Assessment,onlineassessment,eassessment,oreapplications)inalldomainsof testing.DifferentcompanieshavealreadyputcompleteInternetbasedtestsystemson themarket.TheSwedishcompanyEnlighthasdevelopedEnlightTestStationthat enablestoproduceeassessments.AnotherexampleisthatoftheDutchcompany VanderMaesen.They alsodevelopedaMultimediaSituationalJudgementTestfor social competencies.Oneofthelatestdevelopmentsinthisdomainissocallede marking,whichopenstheopportunity,incontrasttomultiplechoicequestions,forthe analysisofnaturallanguageprocessingtechniques(Sturman&Kispal,2003). Sixthandlastly,oneofthelatestinnovationsisthepalmtop,asmallcomputerthat fitsinyourhandorpalm.Thissmallinstrumentmaybeusableforadministering psychotechnicaltests.Forthemoment,developmentsfiveandsixarenotappliedona largescale,mostlybecauseof problemswiththesecurityofthedata.

CONCLUSIONS Psychologicaltestinghasevolvedfromanindividualorientedtestingviaagroup orientedp&padministrationtoadiversifiedapproachbasedoncomputertechnology,

16

COMPUTERBASEDTESTING

whichallowstestingof peopleindividuallyoringroupsonsimplebehavioursormore complexbehaviours.Thesechangestookplaceduringtheperiod1905 2005.The changingagentwastechnology,mainlythecomputer,movingthroughthreestages eachcharacterizedbyacentralproblemnamely1)cost,2)conversionofp&pteststo CBT,and3)thecreationofinnovativetests. Thecomputerhasbecomeapartofeverydaylife,butCBTstillisnotfullyaccepted. Thereremainproblems.Organisationsstrugglewithproblemsrangingfromp&pand CAAproblems,toCBTproblemsoftheinnovativegeneration.Certainly,CBT continuestobecostly(Bennet,2001). CBTsystemsmaybedevelopedindifferentwaysandareespeciallyefficientin meetinglargescaleassessmentneeds.Undercertainconditionsitmaybeusefulto developCBTproceduresforotherthanlargescalesettings.Thewaystodevelopa CBTsystemandthecosteffectivenessoftheefforttoconstructaCBTsystemin relationtotheobjectivesmustalwaysbeconsidered.Asageneralrule,CBTsystems oughttoberealisedaccordingtotheprinciplesoftraditionaltestdevelopment,apart fromadditionalnewissues,suchassecurityandmaintenance.Itisalsoimportantnot tolettechnologydeterminethecontentofassessment. AlthoughCBThassomedisadvantages,theadvantagesoutnumberthese disadvantages.InthelastdecadessomeexcitinginnovativeCBTsystems,suchas CATandIGT,havebeenutilizedinalotofcontextsandholdpromisingprospectsfor futureresearchandpractice. Itisexpectedthatcomputerandaudiovisualtechnologywillcontinuetochangethe attitudestowardsassessmentsinlearningandworkcontexts(Bennett,2001 Hardinge,1997Jansen,1997).Adaptivelearningandflexibilityinwaysofthinking andbehavingwillbecomemoreimportant.OneoftheanswerstothisisCBT.CBT hasthepotentialofbecomingthetoolwithwhichitwillbepossibletoassessthe impactofthesechangesontheabilitiesoffuturecandidates.

17