Comprehension questions Probably the most common teks wsed to ates Ustenlng ae compe henson questions. The ie i simple: a text is presented to tes. how wel they have understoad the content, Tis common pro fd abilities Furthermore they appear easy to devise, although a we Shall se, hero canbe rable Some genera pins Before looking at partiular question types, Is useful to consider some gonetal points. The fst issue concerns the te of infomation syathesise information or dtew conclusions, local questions, whieh required testers to locate deta ot understand nda wore, tne val questions, which required tet‘akers to understand prese but inelvant detallsnot elated to the mal topic. They found tha he global questions wore harder than the local questions Tae 32, Ta for tting ncersanding of eral meaningsTable 53 Tasks forgoing beyond lal eral waa ‘Undenuag t + Zot nn Sepon a perm ple a a hy Gener passags ompeemion sect whicharecane gen theta nes mT of Heenan symone wet ma “deed on pice ih ‘Sion ne nd hen boning pode ae, ‘psd esa ee ee ms omer aa unedng shan pote oad ha ea stig tren etn st wh ne put od rong uoks 137 and served mining! purpose in an evluation to (Shohamy | fd Inor, LI?) The eaetreh suggests that questions ed 10 focus on thekey infront he text: not eeevant deta. ‘nother important Heuer wheter the questions should be pro sided efor listening er afer In virtually all eaword Ustening ‘tuations teers know why they are Tseng and what jaforma tion they ned get rom the teat, If we ek test-takers often 10 | teat wth no speed purpose, and ater listening ive them ques tions hut nlte understanding anything and everything. we are ‘sing them fo do someting very unnatural. Hence, mos scholis| fecommend giving tes takers the questions Before lstening (Berne. {W2s;Brndl, 1988, Rick, 1990, However researc onthe effects of ‘question preview is rather mid. Buck (1990) made two versions ofa {est ane with tema preview and one witha preview, and gave thm to too groupe of test-takers. Although test-takers clearly felt that ‘question priview sided comprehension, results showed tha It id ot make 4 sgatieandference oes aie. However, question preview did make « considerable difference o tem discrimination, ggsting the preewe daesalet item processing, busin ways not jeu anderstod. Berne (1995) found that question preview Ted 10 Fianifeant improvements im tet performance compared tthe tflects of chor pre isting ates, whereas Sheeran (1987) found shat evew did not slgnifeanty improve petnmance. Wu {0 found that preview of mulplechae quesuons factated Procrssng or the more advanced Tisteners. but not forthe bess vanced ii very diet ro make fim recommendations about question preview, bt dacs neem to ave a posive psychological value or | festtaters, an conversely the uncertain of noe knowing wb hey re inten at negatiepeveholopiel lle: (Buck 1990; Sherman, 1997), Another problem ie that alist of questions often makes ile Sense out ofconen, and ts ot tl lstenes hear the pose thet the questions st to seem relevant (Sherman, 197), Quesion| prevne may not motivate litening as mich as es-dvelopers hope. ‘Seeman (1897) fund tthe mos power improvement in peor ‘mance came rom showing the questions fe licens had head he Passage ner there they heard a second ime "There le some evidence to sggst that Ustenlng passage is Inincaly nteretng, there fe no need! to motivate the Hse tin quests ~Hstenar wil simply Uten out of Sateeet (Buck188 aserssine utsreyine 1990), the questions then focus oa the main top or dhe cena ‘area of the discourse, good listeners will have the Snformaion| tscessry to respond, In which cate question preview may not be necessary. however, the test-developer dos wheter the passage 4s ininsialy interesting, or doubts whether the questions faces om the main points of interest, then providing question preview seems ‘he mos prudent course of eto. There are some conventions about the use of comnpetension ques: ons. te seems to be widely accepted that questions wil ak for Information in the same ord: in which it eccursIn the pase Thompson, 195), and aso that once an item ef information hes ‘been used in one question, Kt wll ot be vse agin in anates ack, 1990). Testdevelopers who break these conventions may cantse test-takers, which could lato unelable performance, in many testing situations, eapeily testing academic Istening, ‘esther want rake notes whl scening, Whether this allowed ‘oF not vill depend on the denon ofthe listening consract and tarpeclanguage se station. Hale ard Gourtney (84) found th allowing stadents vo take notes daring Kstening dnt signicaniy Improve thelr performance on the TOBE: seni txt. and urging them to take notes signifi impaired their pesormance aly, there i the issue of question spacing. Ils ipontant to leave enough time for test-takers to Fespou ta each question, before ‘ing on to the nex. This expecially problematic when test-takers fave ask to complete task tthe same time they a Hering = expecially when Bling in information gids, but this also happens ‘with. shorcanswer questions ‘or multiple-choice questions. At Brindley 1888) notes, 1 very important tat gustlons are spaced fut so chat esters wil not be responding fo one question tt the same time a tey need tobe listening 1 the information necessary Short-answer questions ‘Comprehension questions ae rately simple to we if we aloe Tes-takers to construct dhe responses themselves H we desig he ‘questions so wat the answer wl be sors, socing shold be reason Daniculsy suite for tenting the understanding of elearywented reatng sks 139 Informatien, We can strate he process by ataching questions 10 ‘passage fom Chapter 2 where the young man talks about Leon sample st th oan oa on Sc pr gn re ute murnanot andthe he prope one “tunes ant mer paige taraen an inept opr trae en oon rae cnt win msn ta on ond en fo ao tare ad rou pt prs nen you sh erp Sts ns epg Et of tpg tee gy wes ‘ron pang sisson repo a ate he Sore A ayo ad pl dan en Sarre ey fh ge ef @ wen Sk mah tr png ie a (thom ray pope arnt wp toplayt {G2 Dh spetereant top? (Ob Wyatt very wore oto ong DeLHow tee tbeyin staring teh? Sometimes short-anser quetions ca be presented In a sentence Completion format. Example 5.2 ghes sample Heme from the De ‘comber 188 Tseng paper ofthe Cambridge Fist Certicate re ewe in Shaper, ample 2 4. Fn’ tot aed Nakgound were ot ey 12 She ten produced ‘These sen ence-completion tems could just as easly be presented a5 shorvansver comprehension questions. However, the fc that hese Items areeaty to progce, dass ol tmean tha seas to produce ou! ones Lik al test items thee ate pall, ana ems need 1 Be Drewested and bad anes need tobe Identied ae ler deleted oeMo assessine Listentve. hors the trem 0 constast thet own cesponses ‘means that we need ro make deslns about which responses il be ‘marked correct and wich wil ot. This the most cfcult aspect of lung open-ended questions. the cat fms tha tk fora simple fect scoring Is at nly so dieu. For example i the pase says ‘the tain lft atten ol, and the question asks. What me {hd he ta ee, here iteehance of embgut. "he way to make marking easy so wre Rem that have Unabig uous responses. This offen quite posible, ad many ets wl alow plenty of such ems. However, the result maybe questions that fos ‘on superficial understanding of early stated information, and as lesiedevelopes, we ae ollen tying 10 assess comprehension ‘wo potential probiems: fs, determining what constiates ares: sonable invrptetation of the Text, and secondly, what constutes salen response othe questin. valuating responses Imagine tes about a buryar who ‘war stating something very small, and then a question asks, What wa saken? The evans lnswer would scem oe meting small. So Row should we score the ansier Something trivia given by a etson who assumed that Something small moda! something of no val? is this a comet Fesponse or not! What about a response ‘rthing of tue? When ‘we know that some sal things could aesully be guite valuable, i 1s nor easy fo determine which should be accepted ay coect and ‘There ae the problem of deciding what i lft information tconsiute a fal espons. Imagine that the sre text continues by saying thatthe woman wid many things to stop the burglaries = puting bars over the windows and hr & guard If the question ‘sks, Whar id she do about her problem, what the ctect re ‘sponse is enoug tos se red to stop the bugles oF do we {ait ha he teat ake sy ow shee stop shen? Alera we could uss that she woud ryt sop sem, without string the text And how will the tstaakers know their respnse Is good ‘enough Tnese are dieu decision hat should ny be ade i he A spategy or determining cores responses “The fact i that comprehension ss nota simple dichotomy, and die ‘hotomous scoring (Le. scoring each item as being either correct of | Incorrect of re-response queonscuas& complex connie it ‘simple icotomy. The scorer has 10 moke a whole series of dec ‘ons shott which rxponses are aceepabl, and which 0 not How ean we decide? With niin resouees the test can be ad wininters tos group of empetent stoner and thelr responses used 21 sir judging the acceptability of responses: ul any of us| ‘ll no ve sient resciees to 1a out ems nts Wa neha ‘ase, whet searing the items, heap a record of the rerponsck note hic ane defiiely gh, which ate deity wrong, and which ee anderline Keep a note ofthe papers which have the borderline “eis As you conte eong you wl come 1 ndesand beter how the tears are nteractng with the task and wat he question lemesiurng, and asa rest yon wil peobably develop a beter idea of actly wih ot the borderline cates sbould be accepted a cores find which should not. Then go back and re-mark the borderline expanses Slowly a reasonably consistent marking scheme should merge, even though there ms sl be some inconsistencies. 1h ‘ot am ida ao scoring ot I the al wor of limited esources, ‘wecannotavays pre-test everthing we wou ike Shortage of te In any to, listener need to know how mich time they have (0 respond, snd this priulaly important with ehor-ansier ques tions beause ther fs esualy only fie fr short anewers H tes: lakers gr ansiers which ae 0 long. they may spend foo mh time on one gestion and cold mis the neat question, oF even the tent cenlng passage, Asef statpy 10 reste he espouse C0 tho more ha thee words ox exatp,eough ts snot always practicable,Sometimes test-akers tke a long time over question because they rethinking abou de response bt general estate have tunerstood the pasmge thinking time wil be shor compared the time taken to wite the respon. So hen text takers ate spending tra time thinking ofa response. itis usally ease they have not ‘understood the ext well eoug to respond If divi tes-akers| want extra time to think about thee responses, sssume their compte henson is lacking an find some way 0 force them to nove on. Longer responses Sometimes we may want 10 ask comprehension questions that require longer responses. For example, if we asked why something happened. we might get cwo oF thee sentences in response, When fesponses are longer and more complex, we will have to develop tore sot of tng scale wo evaluat thee sual. Developing a ating sale involves deciding what responses tet takers at itely to produce, how thse shouldbe evaluated rave tach ote and how many mats shold be abd foreach, Having developed sae, the Job bas only jut equ, Hie then mecessiy to ty out on a sample of tasks, preerbly with a numberof eters ‘Then laters must practise wing the seal un hey canal apply it ‘consistently. Developing a reusonabe rating sales not usualy Aifcult, ta t does reqlte time and effort to develop i propety IMeWamara, 1990) Muitipl-cholce questions Selected responses can be of many ype, bur dhe most common i the Imukiple-choice fem with Chee, four oF even five options, Com Struting these tems isa igh evel professional ski, which takes onsderable tne and tani to do well All ters gh ob tested before being usd in ay bigh-staes assessment, But tht patculaly the case with multiple-choice tems. They ate complex land unpredictable, and afer pre-tesing expect to uow sins. of ‘seriously modify. a lure proportion of your muliple chair items, esha ee all of tem Mn pee ae vey paid wh orgs 189 Muluiph-choie items ae dificlt wo wte because of thei com: plexity. any scholars beieve that they hae 2 stong med eet (Brinley, 1996) and tht they make considerable processing destands| ‘om the twctaker Vlanson and Fnsen, 1954), They ea force te takers to readjust det inerpretadon ft does wot ape with the ‘puions Wissan, DeVincezl and Tang, 1996) WU (1998) found that they favored more advanced lseners and that misinterpretions of the opis et test takers selecting the rang optians, and conve sey texte akers electing he corec options forthe wrong reasons Ibe tsteakers ae all rom the same Lt backround, giving the questions in the ft language works very wel nd is probably the best way of ensuring that seni test sone are not contaminated ‘with ear sil: However, in many cases test takers will be fom diferent ieslanguage backgrounds and wil be necessary ge ‘questions in the 12 The most conuon way i to provide waen “qvestions and so muliplecboice itening teste are tess of reading smut is stoning ~ whic often may not be a Bad ng ‘There ze tw basic formats for presenting the questions. ta Example 33 below the ste sn he form of question, foowed by leave responses amples ‘Who spt bac home om he se hoe? 1) ut bore tank 1 aime terion (0 our ne (© rene half pas ine ‘An atertive is the sentence-completion format, where the stm 5 an incomplete sentence an the options are alternative completions. ‘xample ies the same tem in sera Bramplesa She wotbuck bone eo her ster’ ase (a Ju bei o {0 inh ofternon {a rent hal pst nine ample 55 Is a ypicel multiple-choice comprehension question taken fron & TOBIC sample tat (IOMIG MISS, 145), Tere takerhear a vor dialogue, and then respond io compreetsion question based on the dstoge ample 58 Teaatan ba ema: Cae rely ting? Mer jes he's re of th start Bie he at ame hing et ieee Fema the ha ster on fo nn ial ‘Thay ed: iy i Carls changing hist (tava “This tr requires understanding a numberof shor wrterances oh & tera eran ewe, The es-taker must equ te mening Of he Isls nee with he's red ofthe restaurant busines Hens t0 ‘ny comerhing ently aierent. The to expressions mean the sasie thing but they ace not simple synonyms. Example 6 fom the Test of English as a Foreign Language website hap! /ewwelL ory te-pahem. Examples. Maley mind ute lon of Fesale ain heme of uti pore Nator ha dh woman np? 1s) Th nso eh rn on, 16) Depa wil beer soo ‘This example is diferent for rwo teasons, Fri, the question ie ‘reseed auraly (rote that i hes if the nattar's voice notice= bly diferent fom the speakers). Secon, the question Is gien fer the teat has been hear, so tst-tahers will ot know what Uist for before they hea he wat hen the questions ae writen ‘he tet book test taker can often tae a uch puck before Hern ‘another interesting point shot Fxample 56 hat the woman fence in oer to understand that the woman is actly rf Dermlssonto sith of he television. Although thi im es infer fee to mae, he fst ther hat to understand the Inferred meaning by taking ro account the communicative station. In he net example, lo rom the TOEIC sample test CHOEIC MT 95,198), stats eat a shor talk, and then answer comprehen ston questions. Brame 57 km 9 aur anal sat ming eon. Bnch and Daf hss mang sna from out ron ond deeopnt de orton il jh isn dante ar nee prs of {ohare ot angler frm our ew ie aa wk tions of ‘hee me ple el joss a he dls bat hat Nas Deh ‘et up meaning dlatg oom 10 iho ending theming (0) oad str ath (@ Fesory worker: (2 What he peopled fea! (a) Myowtsome ew ruc (0) va mea, (9) Vit ch rer parent ‘These eo tems are cleat testing comprehension. The questions are Printed in the est oak, ahd so test-takers ay have tine sean the tems fst, whieh means they should be able 10 listen fr specie Information, rather than just generally tying to understand every ‘Bing. QI Who is atoning the meeting” eequtes understanding 3 ery short plece of explicit stated alormation, vleome to 04 Semiannue ses meting oF pera its enough just 1 eat the fexpesson sles meting: Q2 Whar will he people do fst reqs festtthes to understand that the expresion ‘now plas hep our Selves to ie delicious buf tha has een se up inthe adjoins ‘ning toon” is an istration go and eat. Again not 9 eal Inference, it a pragmatic inference neverthalos, ‘carenWis sie the seope ofthis book nino det of how to wate 00d multiple-choice ites (see Popham, 1885), but a few words of| ‘vice might hep. Make sure the questa Is about someting om hic there could be plausible siematives, soften the case that ‘here ts one obvious asteactor which definitely wrong, but whieh seems lel to atract students who do not utdertan It mach hater to find the seen and third disteatrs. option tha no One ‘hooters is worthless. When loking for aeratve options, ink of ‘the information sim to tat inthe const pion oy words ‘hat sound similar. Something that would spear plausible to a Person whe did no hear the test il ten work “Make surethat theormt he question does ot giveaway the cortet ‘option, Checkthatal he optons thester, anthar he correct ones not Tonge, nor difeentinany way Make surethat neither the ero not ‘he optionsareumblguous and chek that none os inortectoptions ‘ould be considered comtect Take care tha the sewer cannot be ride from background knowledge nr basi resoning ‘Although they ate complex and dificult 0 make, multipe-choice ines can be use est ane of tenn su sl: fom unde ‘taning a the most exit itera lvl, though combing infor ‘on from dierent pat ofthe text, making pragmatic inferences understanding Implet meanings 0 summarising a synthesising “Trtfalbe questions “Theres another typeof seestad option quesion that very popula, the tefl forsat. Thi very simple afer postings text one ‘or more statements ate give, ind est takers have wo decide whether each statement is tec orale. Exarple Sa ilstites he format wih series of trueiase questions from the December 1998 CCS Example 8 Aer stein the sateen, ester atk whee hey ae tie 0 fe Lary ners aretha hous. . oo 2 He ites heal sep long, 5 a {ethnks etre may nd a sep nog ‘isatrewnaatalctedby htepiagpatem EE ewing sts 147 ‘There is some dhagrement about the uly of this question ype \Weindey, 1998). Burger ard Dery (1982) laos that they are et suitable for sting stening because senes normaly ews ua whet ssa, not sn wht isnot sad and ax there is no tet for them to refer ack i Istners have no tears of checking fale sateen, ‘The ather major problem with true/false questions i that est ‘akots Can ge hal the questions erect by eelom gassing This also a problem with three of four-opuon multple-chae questions. “There area number of ways of dealing ith guessing ust, i postbie once the effets of random moesing By inelding more items on the test ~ the move items there are the less elect each Individual gacss has onthe total core, A second problem arises when Some est-tkors are guessing and others are moti important 10 tel est takers thar they should ues, so that they are all pursing the ‘There is another advantage fo Raving textcakers qucss. Many soesses are 30 purely candom auesses at al Bven when they donot now the correct answer testtakers often have some comprebansion| ‘ofthe poin being tested, and s0 they are inclined to favour one response mor another, pethaps without even knowing wh. a suc ‘ates, the gues is based on some degree of peril comprehension ‘whl s eleraninformnation abut the consinact being mensoed Inference questions During the course of the previous chapters, 1 have continually lamed that inference i tthe core of lng processing. Ths, Js important o assess inereaing ai, and many scholars recom amend questons that go beyond Ural meaning (Boyle aid Su, 19 rger ad Dates, 1982; Thompton, 19; Welt, 1989), lle fence questions can, wee, he leat t9 make eeatss the answers are ot expel stated inthe ex. 1s important to distinguish between awo types of ifeences in test ass. The fst inferences about what the speaker mes "hese are wally construct relevans inferences, The secon ler cenoes about what de esteveloper expects an what the Bes fst {aking stotegy i There ate sialy constuct neva flees ft we show ry 0 Bee then nti ests. "The main problem wilh inference items i that I¢ (soften veryica oy wh cern at ny ec ge wang Amst he pose to gt the Hen wrong ems wh ak fr ‘esonal rato or predletons aoa! fre eens Ce cannot Se marked wrong for omampe At, quests mint be page 6 ‘eden the see ta they require something which can oly Be Ine by understanding the pasage. 1 fen possible W te Connon sense or general Know to intr teaonate ep Furthemore infrnce are base on around iow and Important make sine ht the hedge she yl x takers Many ofthese poems can be toed by using nip eee stems Even the coe rsporne isnot te best posable nen long athe aber opine at lene prefer shoal be Ais whieh the eo response Mae seo atest ahr to ‘now he Bs option, ae then aking them to choose he caret The sor of nfrmaton that can usual be «+ sing forthe min eof the ext of wsection ofthe ext + ashing about anything whic sno lealy state, bu thts cle and deliberately inicated by the speaker, ung cholc of words ot tone of wice ~ the connotations of words i a parcuaty ech source of inferences + aking about any prgmateiplistion, ological entallment tha follows om from what the speaker sai «asking she meaning of aditee speech acts. plicily stated Information. Once we attempt to test pragmatic or Socios tnowieige which lavolver interpreting meaning fn tems of «wider communicative conte, then ifrencing becomes teen more important. Good stein test wil general Ttlade I ference item. takes ine and uuble to make ther, but they ace