Fall 2012 EIL 460 Test Specification Packet

Intermediate Oral Grammar Virgiawan (Virgi) Kristianto, Scoot Marta, Daniel Johnson, and Julie Kierski Version 1 ----------------------------------- page 1 Version 2 ----------------------------------- page 59

Understanding Katie Honken, Version Version Version

Instructions Sophie Qin, Becky Menendez, and Mica Swyers 1 ----------------------------------- page 23 2 ----------------------------------- page 82 2.1 --------------------------------- page 99

The Barry Spec -- Testing Pragmatics Alberto Domínguez, Widya Ratna Kusumaningrum, and Yelena Forrester Knezevic Version 1 ----------------------------------- page 37 Version 2 ----------------------------------- page 116

Amending the SAT for English Language Learners (ELLs) Wahru (Esti) Sumarno, Jian Tao, and Erin Lamboi Version 1 ----------------------------------- page 46 Version 2 ----------------------------------- page 135

Intermediate Oral Grammar, Version 1 Virgiawan (Virgi) Kristianto, Scoot Marta, Daniel Johnson, and Julie Kierski

Intermediate Oral Grammar Achievement Test V1.0
Daniel Johnson Julie Kierski Virgiawan Kristianto Scoot Marta

Spec Packet, EIL 460, Fall 2012, page 1

Background: This test is designed for students in an intensive English program. It is intended for students at an intermediate level. In our program, students are placed into one of six level (100-600), and this test is an achievement test for those at the 300 level. Thus, if both instruction and learning were successful over the semester, the students should be able to fulfill all requirements of the test. The test requirements were built from the 300-level (See Appendix Figure 1) as written for the Intensive English Institute at the University of Illinois at Urbana-Champaign. Our intent in designing this test was to measure students' ability to produce grammar and grammatical structures orally as well as in writing. It is very common for oral grammar production to lag behind written grammar recognition and production. Therefore, after administering this test, we should have a better understanding of students' abilities in written grammar vis-à-vis oral grammar. This can then inform instruction if there is an unacceptably large gap between the two. In addition, administering this test (and the studying that students will do for it) will be an additional learning opportunity to attempt to encourage students to incorporate new grammar into their everyday usage.

Description of Assessment: This oral grammar test is one part of a final achievement-based assessment of how well students meet the proficiency goals for the 300-level course set, which consists of four separate courses (Listening, Speaking, and Pronunciation; Grammar; Reading; and Writing) each taught by separate instructors. The final assessment consists of a series of separate tests assessing each course/skill in an attempt to gather sufficient evidence to determine the students’ mastery of the overall course goals. The final assessment will be given once at the end of the semester. To pass on to the next level of foreign language instructions, students must demonstrate achievement of “very well” or “satisfactorily” on all four tests. A student’s end-of-term achievement scores in each of the areas will be combined together and used along with the teachers’ collective assessments to determine the student’s placement level for the next term. This oral grammar test will assess students’ ability to produce specific grammatical forms in the context of communicative tasks. The format of this oral test does not allow for the best assessment of certain discrete grammar forms and all of the benchmarks at the 300 level. Therefore, a separate written grammar test assessing those benchmarks will be given to the students; this test will also contribute to the final assessment. Since the oral grammar test items are communicative tasks, the data gathered from this test will also be used to assess the students’ communicative ability, specifically listening, speaking, and pronunciation. This spec document will only cover the oral grammar test section.

Spec Packet, EIL 460, Fall 2012, page 2

or scenarios. where appropriate General Instructions: The test will be administered by one of the student’s teachers (from one of the courses in the 300-level course set) so the student will be comfortable conversing with and being in a room with the administrator. articles. Throughout the oral test. The interviewer is the person who will give students instruction and prompts as necessary. The audio video system will be used to record students’ responses and to enable the evaluators to assess students’ responses outside the room. including both simple and progressive aspects Show an awareness of countability and use appropriate nouns forms. and possibility in the present and future time frames. The student may need some time to examine a picture or think Spec Packet. EIL 460. The interviewer should make the student feel as comfortable as possible. Then the student will complete a series of seven tasks that focus on different grammatical aspects. necessity. and quantifiers at least some of the time Use modals accurately to express ability. text or picture prompts.Description of Test Takers and Test Environment: Test takers: The students who will take this achievement task will have completed the 300-level course set at the IEI. the interviewer should give the student sufficient time. Students generally have an intermediate level of grammar knowledge and fair communicative competence. Fall 2012. to think through the task. one interviewer and one student. if the student needs it. Environment: This test will be administered in a language laboratory equipped with an audio video system. page 3 . The test will start with a series of warm-up questions to ease into the test. These tasks will focus on one or two major grammatical forms and consist of an interview format. Objectives: Through the oral grammar component of the achievement assessment evaluation the student will demonstrate how proficient they are orally in the following grammar proficiency benchmarks for the course. Ability to ask Wh.questions accurately Relate events with clarity using a variety of verb tenses in present and past. The students are presumably familiar with the types of tasks in the test as they have encountered similar tasks in classroom activities. There will be two people inside the room. These warm-up questions are part of the assessment.

b) The oral test will also be videotaped. the content of student responses is not the assessment target and therefore will not be assessed. If the student is struggling with the task. This includes confirmation checks. the LS teacher. The conditions and Spec Packet. speaking. This will provide good data on the student’s natural production of English especially for the assessment of listening. any responses or utterances in the student’s first language will of course not be assessed. clarification requests. 2) satisfactorily. Some assessment tasks require the use of a pencil and worksheet. Since this is a grammar test. EIL 460. or 4) unsatisfactorily. so mistakes made in these sorts of situations should have a relatively small impact on assessment. Then. the interviewer may try to elicit an answer by asking guiding questions. page 4 . they will only look at the spoken grammar produced. and use of conversation and listening strategies.about the scenario. Even if a student misses the prompt or context. Some tasks require the interviewer to explain the instructions to the student. Scoring: All of the students’ utterances produced within the test room setting will be open to assessment. All of these teachers will have training and be familiar with proficiency goals to which they are grading. and pronunciation. the focus of the assessment is on the language produced through the tasks. These materials are intended only for specific tasks and not for note-taking or practice. whereas the other. One of the teachers will solely focus on accurate use of the grammar orally and assign a rating to the student. Student Response Attributes: Student answers should be only in English and in complete sentences. so they will be familiar with the tasks’ formats. and any asides. However. accuracy. together the two teachers can discuss an appropriate score c) For each grammar component being assessed in the tasks. This is a fact that the grammar evaluator should bear in mind. he or she can access the recording. If there is disagreement among the two teachers or if one teacher is wavering in their judgment of proficiency for a student. In practice. In the sample tasks below. these instructions are bold-faced. 3) inconsistently. this may be somewhat more difficult to enact because grammar and content are inextricably intertwined. a) Two classroom foreign language teachers who will do the scoring. As this is a test of English. students will receive a score of 1) very well. however. will focus more on content. which the student will be given. This should not be a problem though as the students will have encountered all the tasks before. fluency. Fall 2012.

Fall 2012. page 5 . (See Appendix Figure 2) Spec Packet.scoring scale are listed below in the chart at the end of this document. EIL 460.

Fall 2012. EIL 460. page 6 .SAMPLE TASKS Spec Packet.

Below are sample questions that the interviewer may ask. If a student is more reticent. These questions will elicit student’s answers that use the present simple tense.Oral Warm-Up Interview Interviewer General Instructions: This warm-up interview should feel more like a friendly conversation than a formal interview. Easier questions should be asked at first. However. then progressively more difficult questions can be asked. Spec Packet. This warm-up interview should be a natural conversation between the interviewer and the student. Fall 2012. if the interviewer determines that the student is more advanced. the interviewer may have to ask more questions to elicit responses. The interviewer may respond to indicate that he or she is listening to the student and may ask for clarification if an utterance or meaning is unclear. Questions should elicit answers using the present simple tense. page 7 . Questions should address topics that the student is familiar with and has gained lexical knowledge of through lessons taught. EIL 460. The interviewer will ask the student a few questions to engage the student in conversation. What do you do in your free time? What’s your favorite restaurant and why? What is your home country like? What do you like about living in America? Task Instructions to Give Students: There are no instructions for the students. the interviewer should not dominate the conversation. How to Formulate New Tasks: New questions to ask the student can be formulated based on the content covered throughout the courses.

4) answer the WH questions of the student using complete sentences instead of chunk of words. then take that role as the student’s chosen character. 5) prompt the student for what he or she should ask. 7) end the task if the student stops producing the intended responses or the time is over. Fall 2012.Task 1: WH questions Interviewer General Instructions: This part of the task will focus on how well the student can demonstrate the use of WH – questions. Ask WH questions about that character to the interviewer. How to Formulate New Tasks: When choosing another situation. Task Instructions to Give Students: In this part of the task. 6) record the response of the student. This practice question will not be scored. Avoidance needs to be made upon situations which require too much technical vocabulary. imagine a character that you are familiar with and curious about such as your favorite celebrity. 2) direct the student to ask WH questions with regards to his or her chosen character. using one word only such as hobby or address if the student consistently struggles to create questions. make sure the situation is widely known for triggering the student’s participation in asking questions using simple present tense. In this task. EIL 460. This test should last no longer than five minutes. The WH questions must use simple present tense. 3) ask the student to create one question for practice. the evaluator will 1) pose a question to the student about a certain character that he or she is very familiar with. Spec Packet. The chosen situation needs to be really familiar for both students and the evaluator and to focus on the use of the grammar rather than other aspects of language. page 8 .

They show what John. Fall 2012. EIL 460. the schedules show what John DID on Monday and Tuesday. Remember. This task should last no longer than five minutes.Task 2: Simple Past Interviewer General Instructions: This portion of the test will focus on how well the student can correctly ask questions in the simple past. the student will employ a variety of verbs and question words. did on Monday and Tuesday. The student should be allowed 15-20 seconds to look at the schedule and think about what they want to say and then respond orally. This is an information gap activity. Spec Packet. page 9 . The interviewer will respond to the questions using the Interviewer’s Sheet. Ask questions so that you can fill in all the blanks. There are several blanks in the schedule. Ideally. a university student. Examples: “What time did John’s chemistry class start on Monday?” “What did John do from 11:00 AM to 1:00 PM on Tuesday?” “When did John’s soccer practice end on Monday?” “What class did John have at 10:00 AM on Tuesday?” Task Instructions to Give Students: Look at the schedules. then read the directions to the student. Be as clear as possible that this what John DID on Monday and Tuesday. the interviewer will hand the mock schedule (Student’s Sheet) to the student. In this task. wherein the student’s job is to complete the blank portions of the schedule by asking questions.

EIL 460.STUDENT’S SHEET John’s Schedule 9:00-10:00 AM 12:00-1:00 PM 1:00-2:00 PM 5:30-6:30 PM 3:00-_:__ PM Monday Spanish Class Tuesday 9:00-10:00 AM 1:00-2:00 PM 2:00-_:__ PM _:__-8:30 PM Spanish Class Lunch Study Geography Class Chemistry Class Soccer Practice Study 11:00 AM – 1:00 PM 10:00-11:00 AM __________ Class Math Class Dinner 11:30 PM-12:30 AM _:__-11:30 PM 8:30 PM – 12:00 AM 12:00-1:30 AM Spec Packet. Fall 2012. page 10 .

If the day of the test is known in advance. page 11 . If an activity other than a schedule is chosen. Spec Packet. EIL 460.INTERVIEWER’S SHEET Monday 9:00-10:00 AM 12:00-1:00 PM 1:00-2:00 PM 5:30-6:30 PM 3:00-5:00 PM _:__-11:30 PM 11:00 AM – 12:00 PM Spanish Class Lunch Tuesday 9:00-10:00 AM Spanish Class English Class Lunch Nap Geography Class Chemistry Class Soccer Practice Dinner Study 11:00 AM – 1:00 PM 1:00-2:00 PM 2:00-3:30 PM 8:30 PM – 12:00 AM 12:00-1:30 AM 7:30-8:30 PM 10:00-11:00 AM Math Class Dinner Study 11:30 PM-12:30 AM Watch TV Watch Movie How to Formulate New Tasks: If creating a new schedule or activity.questions and verbs. it might be beneficial to make the “schedules” for the two days preceding the test. Fall 2012. use one which the student will be familiar with and will elicit various types of Wh. Choice of classes and activities in the schedule should be tailored such that they will be recognizable to students in order to retain the focus on grammar and not on vocabulary. ensure that the fictitious person’s name will be easy to pronounce and clearly identifiable as either female or male to avoid difficulties in pronunciation or subject pronoun choice.

2) have the student create 5 guesses to which you must not respond. you will 1) imagine your birthday party 2) guess orally 5 gifts that you get using modals of degree of certainty in simple present tense How to Formulate New Tasks: Other scenario might be applicable for this task. it is necessary to set a scenario in which people usually make guesses such as guessing gifts. Task Instructions to Give Students: In this part of the task. etc. This test should last no longer than five minutes. However. construct 5 sentences using modals of degree of certainty in present tense. Fall 2012. Have the student choose one of the scenarios that he or she really understands. the evaluator will 1) ask the student to imagine a certain situation in which he or she can make guesses such as guessing gifts. or weather prediction. In this part of the task. 4) prompt the student if he or she pauses for certain amount of time. 5) end the task if he or she struggles to construct the guess or the time is over. the possibility of a certain team winning a game. EIL 460. Spec Packet. In this task. a football team’s possibility to win a game.Task 3: Modals (Possibility) Interviewer General Instructions: This part of the task will focus on how well the student can demonstrate the use of modal of degree of certainty in present tense. a weather report. page 12 . 3) forbid the students to jot down his or her guesses before he or she produces them orally.

In this task. You are looking out your window and this is what you see. Task Instructions to Give Students: Look at the picture below. the interviewer can end the test.Task 4: Present Progressive Interviewer General Instructions: This portion of the test will focus on how well the student can correctly use the present progressive and control it in conversation. EIL 460.question. “What is this lady doing?” “What is the shoe store owner doing? Once the student has reached the point where he/she is having a difficult time producing any more meaningful thoughts or has described every person in the picture. You have 15 seconds to review the picture before you speak. This is an adaptive test. The student will have 15 seconds to look at the picture and think about what they want to say and then respond orally. then read the description. the evaluator will hand a picture to the student. 1985) Spec Packet. This task should last no longer than five minutes. Describe all the things that you see that are happening. The test will begin with the student speaking freely. (Communicative Grammar Practice. For example. While listening closely to the description. the student should look at the picture. but if a student consistently struggles to answer. the interviewer can help direct the students focus by pointing to a picture and ask a WH. Leo Jones. Imagine that you are in your apartment. page 13 . Fall 2012.

Spec Packet. Whatever image is used. the teacher may choose another picture that focuses more on the vocabulary or on a topic covered during the course. When choosing another picture. EIL 460. it’s important to make sure that the picture has enough images to produce a wide variety of sentences in the progressive form. Fall 2012.How to Formulate New Tasks: A video shown on silent may be subsisted for this picture scenario card. it’s important that the students have general knowledge of the vocabulary in order to retain the focus on grammar. Additionally. page 14 . This would create a more challenging task for students though since the students won’t have the 15 seconds to compose their students as in the picture task. otherwise it could become a test of vocabulary knowledge.

Fall 2012. and Mrs. While listening closely to the description. the student should look at the picture. A thief stole Mrs. the interviewer can end the task. Task Instructions to Give Students: Look at the picture below. You have 15 seconds to review the picture before you speak. Describe all the things that the guests were doing when the thief stole the jewelry at midnight. This task should last no longer than five minutes. the interviewer will hand the picture over to the student then read the description. The test will begin with the student speaking freely. In this task. the interviewer can help direct the students focus by pointing to a picture and ask a W-H question. but if a student consistently struggles to answer. “What was this man doing?” Once the student has reached the point where he/she is having a difficult time producing any more meaningful thoughts or has described every person in the picture. Spec Packet.Task 5: Present Progressive Interviewer General Instructions: This portion of the test will focus on how well the student can correctly use past progressive and control it in conversation. page 15 . Gold’s jewelry at midnight. The student will have 15 seconds to look at the picture and think about what they want to say and then respond orally. Gold invited some friends to their house for the weekend. For example. Mr. EIL 460.

page 16 . Spec Packet. Additionally. If names are assigned to people in the picture. make sure the names of the fictitious characters can be pronounced easily like this task does with colors. it’s important to make sure that the picture has enough images to produce a wide variety of sentences in the progressive form. This would create a more challenging task for students though since the students won’t have the 15 seconds to compose their students as in the picture task. otherwise it could become a test of vocabulary knowledge. the teacher may choose another picture that focuses more on the vocabulary or on a topic covered during the course. When choosing another picture. Fall 2012. Whatever image is used.(Azar Beginner Grammar Textbook Series) How to Formulate New Tasks: A video shown on silent may be subsisted for this picture scenario card. EIL 460. as in this sample task. it’s important that the students have general knowledge of the vocabulary in order to retain the focus on grammar.

Fall 2012. For example. page 17 . the teacher could imagine that they are a policeman and interview the suspect (the student) by asking a variety of questions that should product the progressive aspect such as “What were you doing at 8 AM yesterday?” or “We found evidence of food in your car. EIL 460. What where you eating in your car?” Spec Packet.This task can also be made into a 1-on-1 conversation by asking the student questions about what they did yesterday during certain times.

Think of some rules that you will tell your class on the first day. the evaluator will read the prompt to the student. some prompting may be appropriate. imaginable for this language feature. EIL 460. Whatever might be chosen. generating content). such as “What about homework?” or “What about talking in class?” Care should be taken not to suggest too many forms to the student. they may try to avoid them altogether by using ability or advice modals. Fall 2012. Both activity and directions can be useful tools to attempt to prevent this sort of avoidance. page 18 . If the student seems to be having serious difficulties with inventing rules (i. bearing in mind that avoidance.Task 6: Modals (Necessity) Interviewer General Instructions: This portion of the test will focus on the student’s knowledge and control of necessity modals. may intrude. however. The student will have 15-20 seconds to think before beginning to respond orally. How to Formulate New Tasks: Other scenarios are.e. This test should last no longer than five minutes. Try to structure the task in such a way that it is not possible (or not easy) to use ability modals like ‘can’ and ‘can’t. Task Instructions to Give Student: Imagine you are a teacher in a high school. In this task. ensure that it is possible to elicit various necessity modals. Spec Packet.’ If students are not comfortable with necessity modals. of course. It is the first day of class. whether intentional or unintentional.

"a". and vice versa. the interviewer may elicit responses by using guiding statements or questions such as the ones below: Interviewer (pointing to an item): Can you describe this? Or if the student needs more guidance. then the interviewer should prompt the student to describe a single item (to elicit the use of the articles "a" or "an"). If the student only describes plural nouns. EIL 460. "many". Task Instructions to Give Students: Describe what you see in this picture. Fall 2012. Interviewer Follow-up: How many apples are there? Student: There are a lot of apples in the fridge. You may use words such as "a lot of". the student will be given a photo of an open refrigerator/freezer with various kinds of food inside of it.Task 7: Articles / Quantifiers Interviewer General Instructions: In this task. but only the articles. This task should take no longer than five minutes. the interviewer should prompt the student to describe a non-count item. The student will be assessed on the correct use of articles and quantifiers with the accompanying count and non-count noun(s). The student should describe at least 6 items in the photo. and "an". quantifiers. Interviewer: Are there any apples in the fridge? Student: Yes. The student may be creative in their descriptions and produce longer sentences. "lots of". "some". Spec Packet. If the student is having trouble producing the sentences. The student will describe the items in the photo. page 19 . This task will also produce data for the assessment of verb agreement in the student's utterances. and verb agreement will be assessed for the oral grammar assessment. there are. If the student only produces statements using count nouns. and vice versa.

color drawings. Pictures can be black and white line drawings. page 20 . may be used for this task. these nouns should be ones that the students should have in their lexical knowledge through lessons that were taught in the 300 level or in the previous 100 or 200 level courses. EIL 460. such as a shelf in a store or a room in a house. Fall 2012. Whatever picture that is used should show a variety of count and non-count nouns. Spec Packet.How to Formulate New Tasks: Other picture scenarios. or color photos as long as the picture is clear and large enough to see the individual items in the picture.

EIL 460. Fall 2012.Appendix Figure 1 Spec Packet. page 21 .

Listeners’ attention is diverted to the errors from the message. overall communication of ideas is still clear Inconsistently Significant and consistent errors that sometimes confuse meaning Warm-up Simple Present WH. Fall 2012.Figure 2 SCORE Grammar A few errors. EIL 460.Questions Many errors that making meaning unclear. Unsatisfactorily Task 1: Task 2: Simple Past Possibility Modals Task 3: Present Progressive Task 4: Task 5: Past Progressive Task 6: Necessity Modals Task 7: Articles Quantifiers Spec Packet. page 22 . overall communication of ideas is clear Very Well Satisfactory Several errors.

Students who do not demonstrate sufficient ability will be placed in a basic “classroom survival skills” class in conjunction with their ESL courses at the University of Illinois. 4. Version 1 Katie Honken. EIL 460. General Objectives The objective of this test is to measure students’ ability to understand and implement different forms of feedback in an academic setting. Assessing students’ ability to understand written directions of assignment. While the EPT tests writing skills and pronunciation. 3. Spec Packet. 2. Sophie Qin. Additionally. Specific Objectives 1. this supplement will assess students’ ability to succeed in a university course and classroom setting by following instructions in a variety of formats. this test targets intermediate-level English-language learners who would be expected to succeed in American university-level courses.e.Understanding Instructions. it will identify which topics need to be covered in a “classroom survival skills” course. This EPT supplementary section is necessary to reduce the influence of students’ ability to follow instructions on their performance in a university-level course (i. Assessing students’ ability to understand and apply a rubric.. Fall 2012. losing points due to inability to follow instructions properly). and Mica Swyers Introduction This test is designed to serve as a supplement to the English Proficiency Test (EPT) given at the University of Illinois Urbana-Champaign for incoming non-native speakers of English. Assessing student’s ability to understand written feedback. page 23 . Becky Menendez. Assuming incoming students have a sufficient score to enter the university. Assessing students’ ability to understand and follow oral directions for in-class work.

margin. C: The title is not bold and the text size is 10pt.) Spacing: Double-spaced Size: 12pt (This is what 12pt. font looks like. among them only one format is correct and the other three are with different problems including spacing. Response Attributes (RA) To select the right format from the four options. Then they will the information from instructions to the formats displayed in screenshots. The computer testing software will record the result and categorize it as correct or incorrect. Fall 2012. D: The font is not Times New Roman. centered Which of the following formats meets the teacher’s requirement? ________ Spec Packet. Correct answer earns 2 points while incorrect answers earn 0 points. and select the format that meets all requirements. The three distractors will depict various formats that arise from misunderstanding of the instructions. Answer key: B: Correct. A: The title is not centered and the passage is single spaced. the examinees need to read through the instructions. EIL 460. The key screenshot correctly follows the instructions provided in the item stem. and those instructions are required to be understood by students in university-level courses. bold. page 24 . Sample Question: In a writing class students are required to edit the provided passage based on the following format: Font: Times New Roman (This is what Times New Roman looks like. The examinees will face four types of formats in screenshots.) Title: Times New Roman. The item stem will be given in a written format: detailed instructions of editing a paragraph will be provided to examinees.Objective #1: Assessing students’ ability to understand written directions of assignment Item 1: Formatting Prompt Attributes (PA) This is a four-option multiple choice test item. size or editing mistakes in the title.

EIL 460.Choice A Choice B Choice C Spec Packet. Fall 2012. page 25 .

EIL 460.Choice D Spec Packet. Fall 2012. page 26 .

5 point. Correct answer of a whole file name earns 5 points. This may require examinees to read through the instructions and understand the sample presented. capitalization and symbols. and Robert Pattinson’s paper is named “Environment Issues.Item 2: Writing correct file names Prompt Attributes (PA) This is a writing test item. The right answer should be correct in all aspects including the sequence of various information. and a sample file name is also presented to help them understand the instructions.” the file name should be: 04_15_Pattinson_Robert_Environment Issues Please use the provided information and write the correct file names on the blank. The instructions of writing correct file names are required to be comprehended by students in university-level courses. 1) Student’s Name: Barack Obama Date: August 5th Title of Paper: How to be a Good President ___________________________________________________________________ 2) Student’s Name: Mitt Romney Date: November 24th Title of Paper: If I were the President of the United States ___________________________________________________________________ Answer Key: 1) 08_05_Obama_Barack_How to be a Good President 2) 11_24_Rommey_Mitt_If I were the President of the United States Spec Packet. EIL 460. the teacher requires students to upload their homework on the course management system with correct file names. Response Attributes (RA) Examinees will come up with correct file names based on the information provided. The file name should follow this format: mm_dd_Last Name_First Name_Title_of_Paper Example: If it is April 15th. page 27 . Fall 2012. and each incorrect portion deducts 0. The item stem will be given in a written format: detailed instructions of writing a correct file name will be provided to examinees. Sample Question In a writing class.

get into groups of three or four and come up with a paragraph responding to the question on the board. Response Attributes (RA) To select the correct choice of the four options. The examinee faces a computer monitor divided into quadrants (one for each choice). The three distractors will depict different scenarios that could arise from following the instructions incorrectly or incompletely.” Spec Packet. for tasks such as groupwork.Objective #2: Assessing students’ ability to understand and follow oral directions for in-class work. You should only turn in one paragraph per group at the end of the class. The item stem will be given in oral format: the examinee will listen to a sound file of a teacher giving instructions for an in-class assignment. EIL 460. Prompt Attributes (PA) This is a four-option multiple choice test item. peer review. each with a different illustration. Sample Item The examinee(s) should hear: “Now. Then s/he will have to process the information and select the picture (by clicking) that best illustrates the instructions that were given. The key picture correctly illustrates the directions given in the oral item stem. Fall 2012. or an in-class assignment. These will be instructions that a student would hear in a university-level course. the examinee will have to listen to the sound file. page 28 . The computer testing software will record the item that the student selected and categorize it as correct or incorrect.

EIL 460. page 29 . Picture D is incorrect because while the students are engaged in a lively discussion. and the first student’s paper is multiple pages long. where the figures are discussing and producing one paper. Picture B is incorrect because each student is turning in a separate paper.(Answer Key to Sample Item: The correct choice is Picture A. Fall 2012. Picture C is incorrect because each student is working separately (and without discussion) on a paragraph.) Spec Packet. there is no evidence that they are producing a paragraph to submit at the end of class.

The stem briefly introduces a fictional student who must complete an assignment. Next. EIL 460. so that reading comprehension abilities do not interfere with the student’s ability to identify the correct answer. The second distracter (B) is incorrect because it is recommending a change for the wrong category. and they will compare the assignment against the rubric requirements to acquire the information they need to correctly answer the question. 2. The grammar and complexity of the scenario description and item directions should be very basic. Fall 2012. One choice will always be “nothing” because the fictional student has already earned the maximum score in that category. The second and third distracters (C & D) are incorrect because they recommend changes to components that are already correct. The student should not recognize the key immediately. The first distracter (A) is incorrect because the fictional student did not earn the maximum number of points. The second question asks students to identify one change the fictional student could make to increase their score in the specified category. In cases where the fictional student has earned the maximum number of points in the specified rubric category. based on the requirements laid out by the rubric. They must examine both the rubric and the assignment. The choices are always 3. the key will be the choice (D) that correctly identifies one change that would raise their score in the specified category. The other choices will each reflect one of the more specific requirements of the assignment. Students should not feel challenged by the language of the prompt itself. rubrics should Spec Packet. they should also be fairly basic. page 30 . Each item consists of a stem and two questions. The only correct choice is whichever number corresponds to the score the fictional student should receive in that specific category. 1. and 0. The first distracter (B) is incorrect because it recommends a change for the wrong category. Of the two questions. although for the right category. In general. multiple-choice test item.Objective #3: Assessing students’ ability to understand and apply a rubric. Prompt Attributes (PA) This is a four-item. The third distracter (C) recommends the wrong change. the first question focuses on one of the rubric categories and asks how many points the fictional student earned with their attempt at the assignment. It also introduces a rubric (in the form of a table) for that assignment. it introduces the fictional student’s attempt at the assignment (in the form of a screenshot). However. Rubric categories should not include technical writing terms or concepts (as students may not have received instruction concerning these yet). the key will be the choice (A) that states that the example student should change nothing. The grammar and complexity of the assignment should not exceed that of the materials they might be expected to encounter in ESL114 class materials. In cases where the fictional student has NOT earned the maximum number of points in the specified rubric category.

include language very comparable to what is used in ESL114 rubrics, even if this language is more challenging than the language contained in the prompt or assignment. Response Attributes (RA) Students will select the correct answer for each of the two questions in the item. This may require re-reading of the rubric and of the assignment screenshot. Students will be assessed on whether each of their selections is correct or incorrect. Correct answers earn 1 point each. Incorrect answers earn 0 points. Thus, each of these types of items may earn a maximum of 2 points (1 for each of the two questions). Sample Item: Jemima has been given a paragraph writing assignment in her ESL class. The instructor provided the following rubric, by which the paragraph will be scored: Paragraph Rubric 3 points Topic sentence 2 points 1 points
The main idea is unclear.

0 points
The paragraph does not have a main idea.

The main idea of the The main idea is paragraph is clear and clear but it is incorrectly placed. appears at the beginning of the paragraph. Paragraph has at least three supporting detail sentences that relate back to the main idea. Grammar errors and incorrect word choices do not distract the reader from the important points Paragraph has two supporting detail sentences that relate back to the main idea . Grammar errors and incorrect word choices are distracting.

Support

Paragraph has only one supporting detail sentence that relates back to the main idea. Grammar errors and incorrect word choices make it difficult to understand the paragraph. The paragraph meets 1 out of 3 of the formatting requirements.

Paragraph has no supporting detail sentences that relate back to the main idea. Grammar errors and incorrect word choices prevent the reader from understanding the paragraph. The paragraph does not meet any of the 4 formatting requirements. The paragraph is less than 65 words long.

Grammar

Formatting The paragraph meets The paragraph
all 3 formatting meets 2 out of 3 of requirements: double- the formatting spaced, Times New requirements. Roman, Size 12.

Length

The paragraph is at least 85 words long.

The paragraph is 75 The paragraph is 65 to 84 words long. to 74 words long.

When Jemima finishes writing her paragraph, this is what it looks like on her computer screen:

Spec Packet, EIL 460, Fall 2012, page 31

Question #1: Based on the instructor’s rubric, how many points will Jemima get in the category of formatting? A. 3 B. 2 C. 1 D. 0 Question #2: What is one change Jemima could make to get more points in the category of formatting? A. Nothing. She already has the highest number of points possible in this category. B. She should make her paragraph longer. C. She should change her paragraph spacing. D. She should change her font size.

Key: 1. B 2. D Adapted from: Writing Academic English. Alice Oshima and Ann Hogue. Fourth Edition. – The Longman Academic Writing Series

Spec Packet, EIL 460, Fall 2012, page 32

Objective #4: Assessing student’s ability to understand written feedback
Prompt Attributes (PA): This is a four-option multiple-choice test question. The task tests test-takers’ ability to understand and apply written feedback. Test-takers are shown a screen-shot of a paragraph written by a student with teacher comments. Four options will be shown as screen-shots of the same paragraph with changes. Three of the options will show the paragraph with incorrect or incomplete application of teacher feedback. One option will show one way that the teacher feedback could be applied well. More than one option might seem to apply all of the teacher feedback, but test-takers should choose the best option. Response Attributes (RA): The test-taker should first examine the paragraph with teacher comments. Next, the test-taker will examine the four options and choose the one that best applies teacher feedback. Students will give either a correct or incorrect response. Sample Item The following paragraph was submitted by a student. The teacher has given the student feedback so that the student may revise the paragraph. Teacher Feedback:

Spec Packet, EIL 460, Fall 2012, page 33

page 34 . EIL 460.Which of the following best applies the teacher feedback for the paragraph shown above? Option A: Option B: Spec Packet. Fall 2012.

page 35 . Fall 2012. EIL 460.Option C: Option D: Spec Packet.

Fall 2012. EIL 460.Answer Key to Sample Item: Option A: (Does not have at least five sentences) Option B: (Correct) Option C: (Attempted to apply all feedback but still incorrect) Option D: (Does not have a transition and at least five sentences. page 36 . sentence split incorrectly) Spec Packet.

Fall 2012 Alberto Domínguez Widya Ratna Kusumaningrum Yelena Forrester Knezevic Item types: . The test-taker will have signed off on something like the following: General specs: .Multiple choice These specs are applicable in a classroom setting where students are aware of the pragmatic content and have agreed to be tested on potentially sensitive subjects. EIL 460.The Barry Spec -.Requesting information . Version 1 Alberto Domínguez.Short answer .Excusing yourself .Oral interview .Making small talk . Fall 2012. Widya Ratna Kusumaningrum.Expressing concern Methods: .Conversation Spec Packet. and Yelena Forrester Knezevic Pragmatic Test Specs: The “Barry Item” EIL 460.Testing Pragmatics. page 37 .

request and give information. When you start driving your car. page 38 . Their answers will be divided into three levels (lower. However. EIL 460. I understand that I will not be tested on whether my answers follow strict patterns or formulas. excuse myself from a situation.e. Response Attribute Test takers will respond to the test item orally. Fall 2012. neighborhood). 1. appropriateness level). and express concern to friends about their potentially harmful behavior. you don’t have enough time since you’re in a hurry to catch your morning flight or you’re late. You usually chit-chat and gossip with them about anything. Sample Imagine that you are living in a suburb and have a good relationship with your neighbors. Their utterances will be recorded using recording or other devices. Prompt Attribute Test item consists of an oral and recorded instruction which asks test takers to respond using their sociolinguistics and strategic competences on the assigned topics. nor on whether I agree with the implications of certain test items. These level classification are based on the use of ‘magic formula’ (greeting. discourse. your friends come to stop your car and small talk with you. friendship. This item involves student’s understanding about interpersonal relationships (i. intermediate.” General Objectives: Test takers will use their rationale to demonstrate integrated competences (strategic. Last night you heard a police siren in the early morning. 1. or high). Excusing Expression 1. accept and decline invitations. I understand that some of these topics may be sensitive. and sociolinguistic). Their responses will be assessed based on adequacy and appropriateness of the language context.“This class will include tasks that test my ability to conduct small talk. How will you excuse yourself from conversation politely? Spec Packet. Rather. I will be evaluated based on the politeness and appropriateness of my responses.

Choice (a) and (b) will be both be incorrect choices. 3. 2. He steps closer to you to talk about the tailgate this weekend. responses will be assessed according to a less formal rubric because the interlocutors are friends. Choice (d) is a rude choice. This item involves the concept of greeting and excusing phrases. we find that the sentences have more greetings than the excusing expression itself. Choice (c) is the correct answer since this sentence consists of greeting phrase and excusing expression in which the tendency of the answer is still in accordance to the objective of the item. and more formal). Fall 2012. Sample It is a lunch break and your professor wants to meet with you. Prompt Attribute The test item is an oral instruction which have been recorded before. but it becomes a good distractor as opposed to choice (a) and (b) in which students who do not understand the item so much. average formal.2. While you are walking to his office. The item assesses student’s understanding of excusing expression which involves interpersonal relationship and academic settings (friendship between two students in schools). What will you say to your friend? 3. Prompt Attribute The item is presented as a multiple choice item which assesses students’ mastery of sociolinguistic politeness in excusing themselves. Since meeting with the professor is really important for your grade. EIL 460. you meet your friend from another class. 2. will choose this option. formal. you don’t want to come late. Students will be challenged to use proper vocabulary and strategies. These sentences tend to open a conversation rather than excuse the test-taker from it. sociolinguistic and vocabulary). If we break these sentences apart. In this sample question. Response Attribute Their oral responses will be recorded and assessed based on the level of appropriateness to the context (level of formality). There are four scoring rubrics used (less formal. The item requires multiple competences (strategic. page 39 . Response Attribute Spec Packet.

(d) Yes Professor. The paragraph will indicate the relationship. I don’t want to miss it. The response will include a greeting. etc. After having class. theoretically test spec is not easy to understand especially how to make the proper prompt attribute and response attribute. page 40 . it is a priority. I am sorry. Since it is not easy to have an appointment with her. Fall 2012. EIL 460. I need to go now to meet doctor after the class. Then there will be another sentence which will establish the need the test-taker has and how the interlocutor is involved in the requesting of the information. The response should avoid rudeness Spec Packet. Requesting Information 1. I will study hard about test specs. your professor is calling you and wants to discuss about your study. between test-taker and interlocutor. As a teacher candidate in non native speaking country. 1. That’s why I need to study hard about it. After this. (a) Yes Professor. but I need to meet doctor right now. The relationship will inform how the information should be requested: level of politeness. there will be a question worded appropriately based on the interlocutor being addressed. proper greeting. Students need to master the concept of ‘less greeting and more excusing’. it is important for students to avoid too many greetings because that would lead to further conversation. or lack thereof.Students will select the correct answer based on the sociolinguistic situation in target language. 3. you have an appointment with your doctor. What will you say to your professor to excuse yourself? (b) Yes Professor. The testtaker will then be prompted to orally request the information needed for the situation described in the prompt. but I am so sorry. simulating how the test-taker would request information. When you are ready to leave the class.Response Attribute: The item will be responded to orally.Prompt Attribute: This item assesses ability to request information. theoretically test spec is not easy to understand especially how to make the proper prompt attribute and response attribute. Sample (c) Yes Professor. You only have 1 hour to get there by bus. When excusing themselves. The prompt will consist of a paragraph. I realize that it is important for me to expose my students to the typical items.

A topic of interest will be brought up in the prompt. EIL 460. Spec Packet. 1. The request will come in the form of a question. A full answer will consist of a response which provides the interlocutor with information they can use to address their concern. He asks you whether the tickets are already for sale and if not when they will be on sale.Response Attribute: The item will be responded to orally. If this is not stated in the prompt the test-taker will provide the information requested of them politely and concisely. appropriate and effective in conveying meaning. and the variation will be stated in the prompt. The test-taker may respond by stating a lack of knowledge of the information in an appropriate manner if indicated in the prompt. As you enter a gas station you see a middle-aged woman at the counter and approach her. Fall 2012.Prompt Attribute: This item will test the test-taker’s ability to give information to others. The interlocutor in the item will vary in age and familiarity with the test-taker. The interlocutor will then request information from the test-taker relating to this interest. 2. Alternately. The setting of the interaction will influence the formality of the prompt conversation. the prompt may tell the test-taker that she or he does not know the answer and that they must respond to the question by stating this in a manner appropriate for continuing the conversation. Ask the woman for directions to your friend’s address in an appropriate and polite way. Convey this to him and propose some other way he may find the information. You do not know if the tickets are being sold or when they will begin to be sold.Sample: You are drinking coffee at your friend’s apartment. page 41 . The knowledge of the test-taker will vary. The response must logically address the interlocutor and fully answer the question. the prompt may be presented as a dialogue leading up to the target interaction. while being concise. Her friend overhears you mention an upcoming concert at a local bar. The prompt will consist of a paragraph describing an interaction between the test-taker and another person.2. You have never been in the neighborhood where you are currently lost.Sample: You have gotten off at the wrong bus stop and are late to a dinner party at your friend’s home. 2.

and then you typically produced poorquality work due to lack of time. There will be a recent example of the test-taker’s behavior that moves the student to worry about the friend’s conduct. The test-taker will be asked to express concern to this friend. philosophical conversations. On weekends you go out with a group of friends to movies. Fall 2012. EIL 460.) 1. Response Attribute: The test-taker will be expected to respond to the task given in the prompt. Then it will be established that the test-taker gave up this behavior while the other did not. The examiner should supply the friend’s answers based on the test-taker’s own utterances. Sample: You are midway through your junior year of college. it will be established that they were close in earlier years and still maintain friendly social contact. In the past. Overly sensitive topics should be avoided unless previously agreed to by the test-taker. The stress of urgent deadlines weighed heavily on your Spec Packet. Barry is a close friend. The test item will be a short written passage that establishes the social background of the two friends. You often avoided starting assignments until the last possible date.1. A response should proceed from the established social relationship that was mentioned in the prompt. the type should be one that people might plausibly engage in repeatedly despite its detrimental effects. and social stigma. with whom you have been living since freshman year. They might imply that the two of them have had similar discussions in the past. Finally. rarity. Next. You enjoy talking together and frequently have long. grading should not focus on grammatical accuracy. Prompt Attribute (based on the Barry item from LTA C3. both you and Barry have had problems with procrastinating. and might express dismay that their friend has fallen back into those harmful old habits. page 42 . 1. However. First. Barry. As such. the test-taker will be asked to raise these concerns with the friend. You have a roommate. the potentially harmful behavior will be introduced as one in which both friends used to partake.3): This conversational task will propose a situation in which a test-taker’s friend is engaging in potentially harmful behavior. The response should make sense. Expressing concern (The type of behavior can be modified according to severity. concerts. and parties. but on whether or not the test-taker states their concern effectively. They should express concern about their friend’s potentially damaging behavior. Various types of addiction could fit here. As long as the point is made. the test-taker might refer to previous experiences that they and their friend have shared.

Barry also seemed to improve for a while. and which would be an appropriate setting for strangers to converse. The stem will provide an initial opening of small talk by a stranger. and will ask the test-taker to choose the most appropriate response from among four choices. at the beginning of the spring term. Your grades have improved. without too many of this overall item type appearing on the same test. Last semester he just barely achieved a C average. 1.minds. b: Ask for clarification even though no clarification should be necessary. 1. he shows no sign of wanting to change. Prompt Attribute: This multiple-choice item will test sociolinguistic competence by assessing the test-taker’s response to a situation involving small talk. Initiate a conversation with Barry in which you discuss his procrastination problem. the two of you would stay up late cramming for each test because you hadn't prepared well enough earlier. 1. He spends most of his time playing video games and hardly ever opens a textbook. Fall 2012. They will be of the following types: a + d: Imply that the stranger’s comment is true. During final exams. EIL 460. Now. and you feel less anxious. You have now managed to get your procrastination under control. You are worried. page 43 . The stem will introduce a situation which the test-taker can be expected to be familiar with. Sample: Spec Packet. you normally finish tasks a day or two ahead of time. The order of these answer types should be varied. These days. c: Offer noncommittal agreement without minimizing or escalating the situation. Response Attribute: There will be four answers to choose from. Small talk: The test-taker should choose (c) in order to receive credit. but offer a response that is minimizing and impolite. but now he appears to be backsliding.

” d. either assenting and asking questions or admitting lack of familiarity. looks like it. weather. The testtaker should not ask questions about the examiner’s appearance or personal life.” . The person next to you turns to you and says. it’s not a big deal. The test-taker will respond politely and make a few generic comments of their own. *c. page 44 .” a.). “Yeah. and the line is moving very slowly.“Can you believe this? The line hasn’t moved in ten minutes. Fall 2012.“Man. “What do you mean?” 2. Response Attribute: Each participant’s responses should proceed logically from the previous utterance. EIL 460. etc.” b. sports). looks like we picked the wrong line. Then the examiner will initiate the conversation. Sample starting lines: .You are waiting in line at the grocery store. 2. If a test-taker is not well-versed in a topic (e. the test-taker will do so gracefully.” Choose the most appropriate response. “Come on.) 2. It gets earlier and earlier every year. When it comes time to bow out. they should be able to go along anyway.” .“Wow. Prompt Attribute: The examiner and test-taker will roleplay a situation in which small talk with strangers could be likely to occur. “Wow. I can’t wait to get out of here” (or: “…get home”). . some mutual obstacle. The conversation should not stray outside the boundaries of typical small talk (sports.” Spec Packet. (This task could come at the beginning of an oral interview with the examiner.“How ’bout them Steelers?” . The examiner will begin by outlining the parameters of the situation and instructing the test-taker to engage in small talk for a short while before bowing out.“That house already has Valentine’s Day decorations up. with politeness but without overfamiliarity.g. “I’m not worried. this rain is really something.

Prompt Attribute: The examiner and test-taker will roleplay a situation in which the test-taker is responding to small talk initiated by a friend. page 45 .“Hi! I didn’t see your brothers at the game last week. and it’s been a week and a half. Fall 2012. Sample starting lines: . either assenting and asking questions or admitting lack of familiarity. they should be able to go along anyway.” . The conversation should not stray outside the boundaries of typical friendly small talk (sports. weather.“Hey. sports). inquiring as to the friend’s own opinions. friends or family. Then the examiner will initiate the conversation. EIL 460. etc. 3. using a “Good seeing you” or “See you around” type of closing statement. man. When it comes time to bow out.). How are they doing?” . I haven’t seen you around much lately.3. The test-taker will respond politely and familiarly. The examiner will begin by outlining the parameters of the situation and instructing the test-taker to engage in small talk for a short while before bowing out. (This task could come at the beginning of an oral interview with the examiner. Response Attribute: Each participant’s responses should proceed logically from the previous utterance. the test-taker will do so familiarly.g.“Can you believe this? She said she was going to give back our grades in a week. some mutual obstacle.) 3. how’s it going?” . How have you been?” Spec Packet. If a test-taker is not well-versed in a topic (e.“Hey.

surprisingly enough. with their wide ranging ethnicities. Both groups together made-up one fifth of all SAT test-takers in 2001. 2002). This issue becomes even more pressing when validity and reliability are taken into account. Not only in respects to test design but more so in the demographics of the test takers. cultures and resulting in more diversified racial/ethnic backgrounds that the test makers have to take into As illustrated in the chart below. However. The steepest increases in test volume since 1973 have been among students of Asian or Hispanic/Latino descent. With a growing number of English language for the oarsman–regatta analogy question. which should be the goal of not only SAT test makers but all test makers alike. has to tackle. Jian Tao. becomes increasingly apparent. However. the proportion of African American test-takers has also increased. cultural backgrounds. these changes in the ethnic backgrounds of test takers basic knowledge of the world. modifications in the SAT have not reflected these particular changes. and native languages. Over the years. the proportion of white test-takers decreased significantly from 87 percent in 1973 to 66 percent in have had serious implications for test makers who share different first languages. In upholding the views of the validity theory. Background: The Changing Population of SAT Takers The SAT is a standardized test used in determining college admission into United States SAT Amendment for ELLs learners (ELLs)coming from different parts of the world to taking take the SAT. dramatic changes have occurred to the makeup of the test takers 2001 (Lawrence et al. the need to address account. EIL 460.Amending the SAT for English Language Learners (ELLs). Version 1 Wahru (Esti) Sumarno. and Erin Lamboi universities and colleges. page 46 . Over the past 75 years. the SAT has undergone many changes. Fall 2012. which demonstrates why the test is has been accused of the various issues involved with such a heterogeneous group. there should be a focus on maintaining a Spec Packet. This test is Known cultural bias towards the white and the wealthy.

more valid exam for the future. In 1928 the test was reduced to seven subtests administered in 115 minutes. the SAT has been modified even further to five rather than six questions. many of the early modifications to the test aimed at providing more liberal time limits. and in 1929. However. One of the major changes in the SAT has been that of time. students were told that they more familiar five-choice question (College Board)." These modifications were meant to make the being tested.certain degree of flexibility. Even so. Despite the changes that have been made over the years. how it has evolved. the time allotted to each particular section of the test has gradually increased over the should not expect to finish. and ensure that what the test was created to evaluate was actually what was years. to six subtests" (College Board). Then. EIL 460. the antonym format was changed to the test more reliable. why it has evolved. there is still much more work Spec Packet. which will allow for modifications to be made in response to the changes in our society. this is not the case. Studying the history of the SAT. and to what extent it has A Brief Review of Major Changes in the SAT evolved or been modified constitutes a great resource for future SAT test makers to use in order to create a more reliable. Fall 2012. "Early versions of the SAT were quite speeded—as late as 1943. more recently. page 47 .

The writing section includes a short essay and multiple-choice questions on identifying errors and improving grammar and usage. General Description: The SAT is a standardized test used by colleges and universities in the United States to assist in the The purpose of the test is to assess “the critical thinking skills you’ll need for academic success in college…and how well you apply what you’ve learned in school to analyze and solve problems. mathematics and writing sections. unbiased.” It includes critical reading. geometry. the way you’ll need to in college (SAT official guide. pp. Fall 2012. and reliable test. algebra. with varying types of questions. including sentence completions and questions about short and long reading passages. The mathematics section includes questions on arithmetic operations. Spec of Sentence Completion in the Current SAT 1. statistics and probability. page 48 . Sentence completions generally test the Spec Packet. the needs of ELLs who are taking the test in the medium of their second or foreign languages need to be addressed and taken into account when creating the SAT in order to validly administer a fair. More specifically. Critical Reading sections normally begin with 5 to 8 sentence completion questions.that needs to be done. The Critical Reading (formerly Verbal) section of the SAT is made up of three scored sections: two 25-minute sections and one 20-minute section. To narrow down our scope. the remainder of the questions is focused on the reading passages. The critical reading section includes reading passages and sentence completion. this project will focus on sentence completion in the critical reading section. EIL 460.3). the most challenging part of the test.

2. The questions are weighted equally. the test is designed to evaluate students’ critical thinking and problem solving skills. response Attributes The test takers will be required to read and compare the five alternative choices presented in the test item. 3. Prompt Attributes Sentence completion is administered by means of multiple-choice questions. both in meaning and in form. that they must complete using the choices provided. the daunting vocabulary redirects the test making it more of a language test than an assessment of critical Spec Packet. Language Issue: The recondite vocabulary presented in the current test items prevent the test from assessing what the SAT is supposed to test. they have to do close reading and grasp a clear understanding of the sentence structure. page 49 . in reality.. The objective is To select the best choice among the five possible answers. For each incorrect answer onefourth of a point is deducted. EIL 460. Fall 2012. and select best completes the sentence. Test takers are given an incomplete sentence. one raw point is added.student's vocabulary and understanding of sentence structure and organization by requiring the student to select one or two words that best complete a given sentence. Questions on the SAT reading-sentence completion are all multiple-choice questions which have five answer choices. content and basic background knowledge associated etc. For each correct answer. only one of which is correct. with one to two missing words. Yet. Modifications for ESL test takers) 1). As mentioned above.

Spec Packet. Fall 2012. 3). it is obvious that certain modifications need to be made in order to meet the needs of these "new" test takers. Cultural Issue: The prevalence of cultural bias has been(don’t need this) a consistent problem or consistently problematic in regards to the content of the test questions.Along with the fact that Asian students are among the largest group of international SAT test takers. in regards to the SAT. its The SAT's almost almost exclusive emphasis on western culture. the best example of this is the famous oarsman–regatta analogy question (Wikipedia). Time Issue: It is commonly acknowledged that we need more time to process information in the L2 than than the L1. it seems that the next appropriate step. Reforms to the test are imperative if validity is to be upheld. Caucasian Americans. As the number of ESL learners taking the SAT increases so does the responsibility of to test makers to create a test that actually assesses critical thinking and problem solving skills rather than a language test consisting of "vocabulary-loaded" questions which render huge obstacles for ESL test takers and do not fairly nor appropriately demonstrate how well they have truly mastered the target skills. Over the past decades. Considering the fact that nowadays the test is going moving beyond the "united states border" and targeting students all around the world. politics and history tends to places certain test takers at an unfair advantage over others. in order to ensure that what the SAT aims to test or the objectives of the test are being adequately and appropriately met we can take out this last part of the quote (College Board).thinking and problem solving skills. the major accusations against the test makers and the test itself have been related to cultural bias. would be to adjust to these changes. page 50 . EIL 460.towards wealthy. especially when both the two languages vary so greatly as does English and non-alphabet languages (Asian languages). 2). specifically.

writing. in specific. The majority of international SAT test takers must travel to the major cities of their countries to take the test. and way of life are used as examples in the SAT so will other countries be used. Korea. is through sentence completion. page 51 . The purpose and or objectives will remain the same as well which are to "assess critical thinking skills" of high school students entering into college through testing math. religions. which influences the level of familiarity a student may have with the material presented. One mode of assessing critical reading skills. ethnicities. certain test takers will not have an advantage over others due to their cultural background. EIL 460. and critical reading skills. that native Spec Packet. etc. cultures. which is the focus of this spec. the revised SAT examples will represent various parts of the world. history. Just as England and the United States and their government. Along with important changes such as including questions that represent a greater range of countries and cultures. and this is if the SAT is offered in that particular area. etc. In this way.Amendment to the Current SAT Spec: General Description The revised version of the SAT for ESL learners. This process of long-distance traveling not only costs money but takes time and energy. will continue to be a standardized test used by colleges and universities in the United States to assist in the admissions process. Indonesia. such as China. Fall 2012. is the necessity to make the SAT more accessible. In regards to cultural bias. creates added stress and uneasiness.

the sentence completion questions of the revised version will be similar to the original. location of test taking center. is uncontrollable.speakers or students residing in the United States do not have to endure to the same extent. then it only makes sense that ESL learners will have an even more difficult time understanding. Fall 2012. Therefore. page 52 . making the Revised SAT available in more countries and more accessible within each of these countries by offering the test outside of the cities as well as within. If the location of where the SAT is offered. Just as only offering computer-based tests. ESL students have not been Spec Packet. With respect to the actual test format. etc. rules of the test. their time and money spent traveling as well as the difficulties faced finding room and board. however with the inclusion of synonyms the fairness of the test becomes more balanced. In this way. understanding requirements. The word choices will not change and the level of difficulty will remain the same. shuts off certain groups of students. so does only offering the SAT in cities difficult and costly to reach. the ESL student is still being exposed to the same material as a native speaker of English. However. more students will have access. the revised version will include synonyms to each answer choice. coming prepared with the correct testing materials. let alone at the high school level. All of these factors influence the performance of the ESL student therefore affecting the results of the test as a whole. is still required to critically analyze the sentence and word choices in order to decide on the best answer. with no regard to those who may not be able to afford or obtain this means of communication. in which country or city. If the vocabulary used in the SAT is difficult for native speakers of English at the University level. Hopefully creating an equal opportunity for students to take the test regardless of where they live or their economic status. then the student's specific situation should be taken into account. EIL 460.

EIL 460. offering more breaks during testing. more time should be allotted to ESL SAT takers to compensate for the extra time spent manipulating a language other than their own as well as factors such as jet lag and culture shock. In this way speed does not become an issue like it had become in the past before the test was modified. Offering the test over two days instead of one. this is the important question. without having to actually systematically analyze try each choice. and allowing more time overall to take the test. they just understand at a different level. or is the objective to be able to understand and actually use the words in a comprehendible way. those test takers who depended on the Holistic approach were at advantage because they simply had to look at the question. offering more time and including less questions. which was already familiar and easily recognized what choices were wrong. Is it the ability to recognize and define difficult words that ESL learners may never use and that most native speakers do not even use. Furthermore. in which each answer choice is critically analyzed by the student were at a disadvantage due to time restraints. which is not to say they do not understand. Research has shown that test-takers relying solely on the "systematic" method of test taking. On the other hand. eliminating them immediately. Spec Packet. Fall 2012. What is really being tested. page 53 .exposed to as much input in English vocabulary as native speakers have.

revised version. Fall 2012. China. Use tests/tasks that are more global and pertain examples from all parts of the world. racial and ethnic diversity among the test takers. page 54 . cultural and historical issues related to theWestern world. but for instance. In this way the student is them as ESL students but to many native English speakers as well. The objective is To select the best choice among the five possible answers. should also include those of other parts of the world. such as Korea. Prompt Attributes given an incomplete sentence. still required to critically think about the answer. Specifically. and Indonesia. the synonyms for each choice should also be provided. EIL 460.2. the test should not only cover the social. Test takers are reflecting the great cultural. Japan. but not expected to understand and be able to use words not only uncommon to Spec Packet. that they must complete using the choices provided. Provide words equally difficult to those provided on the original SAT exam. with one to two missing words. however. Sentence completion is administered by means of multiple-choice questions. is still exposed to the same words that are given . in the in the original SAT.

instead of one and offer more breaks during testing. starting with the most obvious to the least obvious. He may eliminate any choices given. predict the most appropriate word to complete the sentence item by through an analysis of sentence structure. Fall 2012. The test answers and eliminate incorrect answers. The test taker will read the prompt and review all five multiple choice answers. test taker will study (review) all five choices. analyze sentence structure. reread the prompts in order to ensure Spec Packet. complete understanding of what is being asked. Then the test taker will return to the five multiple choice until the correct choice stands alone. page 55 . Then. and analyzing its he will select the correct answer from among the choices given sentence structure. to acknowledge the affects of jet lag and the overall stress caused by traveling. try to predict the meaning or main idea of the sentence. Allot more time to ESL learners to complete the test so that. context. context. Offer the test over two days Response Attribute taker will then return to the prompt. which the majority of ELLs must endure in order to arrive at testing sites.. context and cultural background. EIL 460.

4. warfare…avoidable reconciliation…assured Revised Sample Item 1: ruination…impossible conflict…inevitable Spec Packet. _____was_____. page 56 . warfare…avoidable reconciliation…assured ruination…impossible diplomacy…simple conflict…inevitable Because the Empire of Japan aimed to dominate East Asia and Indochina during the World War II ran counter to the autonomy of then Republic of China initially led by Sun Yat-sen. EIL 460. _____was_____. Fall 2012. Sample Item Original Sample Item 1: Because King Philip’s desire to make Spain to dominant power in sixteenth-century Europe ran counter to Queen Elizabeth’s insistence on autonomy for English.

EIL 460.diplomacy…simple There is no doubt that Larry is a genuine_____: he excels at telling stories that fascinate his listeners. Fall 2012. page 57 . pilferer braggart dilettante Original Sample Item 2: prevaricator There is no doubt that Larry is a genuine_____: he excels at telling stories that fascinate his listeners. braggart dilettante (amateur) pilferer (thief) raconteur Revised Sample Item2: prevaricator (liar) raconteur (noted story-teller) •Revised Sample Item 1: Spec Packet.

Fall 2012.Because the Government of Dutch aimed to colonize Indonesia after its Independence Day ran counter to the autonomy of the Republic of Indonesia initially led by Soekarno. EIL 460. page 58 . _____was_____. (A) reconciliation…assured (B) warfare…avoidable (C) ruination…impossible (D) conflict…inevitable (E) diplomacy…simple I like this but do not quite understand the prompt and am afraid to correct it without knowing what you meant :) Spec Packet.

Version 2 Virgiawan (Virgi) Kristianto. and Julie Kierski Intermediate Oral Grammar Achievement Test V2. EIL 460.0 Fall 2012 Daniel Johnson Julie Kierski Virgiawan Kristianto Scoot Marta Spec Packet. Scoot Marta. page 59 . Daniel Johnson. Fall 2012.Intermediate Oral Grammar.

The decision is a collective agreement by the teachers. a separate written grammar test assessing those benchmarks will be given to the students. It is intended for students at an intermediate level. the students should be able to fulfill all requirements of the test. This is a separate assessment that will also contribute to the final overall assessment. The test requirements were built from the 300-level (See Appendix Figure 1) as written for the Intensive English Institute at the University of Illinois at Urbana-Champaign. This can then inform instruction if there is an unacceptably large gap between the two. In our program. Since the oral grammar test items are communicative tasks. This oral grammar test will assess students’ ability to produce specific grammatical forms in the context of communicative tasks. the data gathered from this test will also be used to assess the students’ communicative ability. and this test is an achievement test for those at the 300 level. Therefore. this test will also contribute to the final overall assessment. It is very common for oral grammar production to lag behind written grammar recognition and production. students should demonstrate achievement of “very well” or “satisfactorily” on each test. Therefore. after administering this test. In addition. given once at the end of the semester. The format of this oral test does not allow for the best assessment of certain discrete grammar forms and all of the benchmarks at the 300 level. The final assessment. A student’s end-of-term achievement scores in each of the areas will be combined together and used along with the teachers’ classroom assessments to determine the student’s placement level for the next term. EIL 460. Spec Packet. administering this test (and the studying that students will do for it) will be an additional learning opportunity to attempt to encourage students to incorporate new grammar into their everyday usage. page 60 . students are placed into one of six level (100-600). To pass on to the next level of foreign language instructions. Reading. and Writing) each taught by separate instructors. Therefore. which consists of four separate courses (Listening and Speaking. Fall 2012. Our intent in designing this test was to measure students' ability to produce grammar and grammatical structures orally as well as in writing. we should have a better understanding of students' abilities in written grammar vis-à-vis oral grammar. Description of Assessment: This oral grammar test is one part of a final achievement-based assessment of how well students meet the proficiency goals for the 300-level course set. Thus. if both instruction and learning were successful over the semester. consists of a series of separate tests assessing each course/skill in an attempt to gather sufficient evidence to determine the students’ mastery of the overall 300-level course goals. Grammar. it is possible for a student to receive an assessment of “inconsistently” or “unsatisfactorily” in one test but still pass on to the next level. specifically listening and speaking.Background: This test is designed for students in an intensive English program.

or scenarios. Students generally have an intermediate level of grammar knowledge and fair communicative competence. articles. The audio video system will be used to record students’ responses and to enable the evaluators to assess students’ responses outside the room. and possibility in the present and future time frames. where appropriate General Instructions: The test will be administered by one of the student’s teachers (from one of the courses in the 300-level course set) so the student will be comfortable conversing with and being in a room with the administrator. Objectives: Through the oral grammar component of the achievement assessment evaluation the student will demonstrate how proficient they are orally in the following grammar proficiency benchmarks for the course. The interviewer should make the student feel as comfortable as possible. one interviewer and one student. The interviewer is the person who will give students instruction and prompts as necessary.This spec document will only cover the oral grammar test section. necessity. These tasks will focus on one or two major grammatical forms and consist of an interview format. including both simple and progressive aspects Show an awareness of countability and use appropriate nouns forms. and quantifiers at least some of the time Use modals accurately to express ability. The students are presumably familiar with the types of tasks in the test as they have encountered similar tasks in classroom activities. EIL 460. Description of Test Takers and Test Environment: Test takers: The students who will take this achievement task will have completed the 300-level course set at the IEI. text or picture prompts. The student will complete a series of eight tasks that focus on different grammatical aspects.questions accurately Relate events with clarity using a variety of verb tenses in present and past. Spec Packet. There will be two people inside the room. Ability to ask Wh. Environment: This test will be administered in a language laboratory equipped with an audio video system. This spec does not provide a rubric or discuss scoring for the listening and speaking assessment. Fall 2012. page 61 .

whereas the other. If the student is struggling with the task. the interviewer may try to elicit an answer by asking guiding questions. clarification requests. This includes confirmation checks. so they will be familiar with the tasks’ formats. If there is disagreement among the two teachers or if one teacher is wavering in their judgment of proficiency for a student. However. Since this is a grammar test. if the student needs it. One of the teachers will solely focus on accurate use of the grammar orally and assign a rating to the student. any responses or utterances in the student’s first language will not be assessed. Some tasks require the interviewer to explain the instructions to the student. the interviewer should give the student sufficient time. the focus of the assessment is on the language produced through the tasks. they will only look at the spoken grammar produced. The student may need some time to examine a picture or think about the scenario. This should not be a problem though as the students will have encountered all the tasks before. and any asides. Even if a student misses the prompt or context. these instructions are bold-faced. The teachers will be familiar with the proficiency goals to which they are grading as well as the student they are assessing. Fall 2012. Some assessment tasks require the use of a pencil and worksheet. Student Response Attributes: Student answers should be only in English and in complete sentences. As this is a test of English. the content and meaning of student responses is not the assessment target and therefore will not be assessed. e) The oral test will also be videotaped. Then. which the student will be given. This will provide good data on the student’s natural production of English. to think through the task. the LS teacher. together the two teachers can discuss an appropriate score Spec Packet. These materials are intended only for specific tasks and not for note-taking or practice. Scoring will be conducted within the conditions below: d) Two classroom foreign language teachers who will do the scoring. page 62 . accuracy. The grammar evaluator should bear in mind that content and meaning will be evaluated in the listening and speaking assessment. will focus more on content. and use of conversation and listening strategies. Scoring: All of the students’ utterances produced within the test room setting will be open to assessment. In the sample tasks below.Throughout the oral test. EIL 460. he or she can access the recording. fluency.

the teachers may decide that scoring “satisfactorily” on seven of the eight tasks defines achievement on the test. (See Appendix Figure 2) Since this oral grammar test is one part of a student’s overall final assessment. or 4) unsatisfactorily. However. 2) satisfactorily.f) For each grammar component being assessed in the tasks. evaluators may consult the recording. or they may decide that scoring “satisfactorily” on four of the eight tasks and “inconsistently” on two other tasks is sufficient. the student will receive a score of 1) very well. Fall 2012. page 63 . 3) inconsistently. We suggest that scores of “satisfactorily” on at least six of the eight tasks defines achievement on the test. If for some reason it is necessary to find out the exact number of errors or the specific kinds of errors a student made on the test. this test is a lowstakes test and does not require fine-grained distinctions and exact numerical scoring. Spec Packet. The conditions and scoring scale are listed below in the chart at the end of this document. as flexibility is allowed. The scoring rubric also allows for flexibility in how loosely or tightly an evaluator may define the four scores from semester to semester. Scoring is done through the expert judgments of experienced teachers who have been teaching the student throughout the semester. EIL 460. The student will receive a score for each of the eight grammar tasks.

Fall 2012. EIL 460.SAMPLE TASKS Spec Packet. page 64 .

Saudi Arabia is very hot. However. I eat pizza everyday.Task 1: Simple Present Interviewer General Instructions: This warm-up interview should feel more like a friendly conversation than a formal interview and will be used to assess the student’s ability to use simple present tense. Easier questions should be asked at first. if the interviewer determines that the student is more advanced. The interviewer may respond to indicate that he or she is listening to the student and may ask for clarification if an utterance or meaning is unclear. This warm-up interview should be a natural conversation between the interviewer and the student. Spec Packet. Below are sample questions that the interviewer may ask. the interviewer should not dominate the conversation. What do you do in your free time? What’s your favorite restaurant and why? What is your home country like? What do you like about living in America? Task Instructions to Give Students: There are no instructions for the students. These questions will elicit student’s answers that use the simple present tense. page 65 . How to Formulate New Tasks: New questions to ask the student can be formulated based on the content covered throughout the courses. Fall 2012. the interviewer may have to ask more questions to elicit responses. Sample Student Responses: I like to watch movies. then progressively more difficult questions can be asked. Questions should elicit answers using the present simple tense. Questions should address topics that the student is familiar with and has gained lexical knowledge of through lessons taught. If a student is more reticent. The interviewer will ask the student a few questions to engage the student in conversation. EIL 460.

14) end the task if the student stops producing the intended responses or the time is over. This test should last no longer than five minutes. 13) record the response of the student. In this task. Spec Packet. Ask WH questions about that character to the interviewer. Sample Student Responses: Where do you live? What are your hobbies? How can I become famous like you? How to Formulate New Tasks: When choosing another situation. make sure the situation is widely known for triggering the student’s participation in asking questions using simple present tense. EIL 460. page 66 . Task Instructions to Give Students: In this part of the task. 11) answer the WH questions of the student using complete sentences instead of chunk of words. Avoidance needs to be made upon situations which require too much technical vocabulary. then take that role as the student’s chosen character. The WH questions must use simple present tense. 10) ask the student to create one question for practice. Fall 2012.Task 2: WH questions Interviewer General Instructions: This part of the task will focus on how well the student can demonstrate the use of WH – questions. The chosen situation needs to be really familiar for both students and the evaluator and to focus on the use of the grammar rather than other aspects of language. 9) direct the student to ask WH questions with regards to his or her chosen character. This practice question will not be scored. imagine a character that you are familiar with and curious about such as your favorite celebrity. 12) prompt the student for what he or she should ask. using one word only such as hobby or address if the student consistently struggles to create questions. the evaluator will 8) pose a question to the student about a certain character that he or she is very familiar with.

Task Instructions to Give Students: Look at the schedules. then read the directions to the student. In this task. This task should last no longer than five minutes. did on Monday and Tuesday. There are several blanks in the schedule. The interviewer will respond to the questions using the Interviewer’s Sheet. page 67 . the interviewer will hand the mock schedule (Student’s Sheet) to the student. Fall 2012. a university student. They show what John. Sample Student Responses: What time did John’s chemistry class start on Monday? What did John do from 11:00 AM to 1:00 PM on Tuesday? When did John’s soccer practice end on Monday? What class did John have at 10:00 AM on Tuesday? Spec Packet. Ideally. Remember. EIL 460. Ask questions so that you can fill in all the blanks. the student will employ a variety of verbs and question words. Be as clear as possible that this what John DID on Monday and Tuesday. This is an information gap activity. wherein the student’s job is to complete the blank portions of the schedule by asking questions. The student should be allowed 15-20 seconds to look at the schedule and think about what they want to say and then respond orally.Task 3: Simple Past Interviewer General Instructions: This portion of the test will focus on how well the student can correctly ask questions in the simple past. the schedules show what John DID on Monday and Tuesday.

Fall 2012. EIL 460. page 68 .STUDENT’S SHEET John’s Schedule 9:00-10:00 AM 12:00-1:00 PM 1:00-2:00 PM 5:30-6:30 PM 3:00-_:__ PM Monday Spanish Class Tuesday 9:00-10:00 AM 1:00-2:00 PM 2:00-_:__ PM _:__-8:30 PM Spanish Class Lunch Study Geography Class Chemistry Class Soccer Practice Study 11:00 AM – 1:00 PM 10:00-11:00 AM __________ Class Math Class Dinner 11:30 PM-12:30 AM _:__-11:30 PM 8:30 PM – 12:00 AM 12:00-1:30 AM Spec Packet.

ensure that the fictitious person’s name will be easy to pronounce and clearly identifiable as either female or male to avoid difficulties in pronunciation or subject pronoun choice. it might be beneficial to make the “schedules” for the two days preceding the test. If an activity other than a schedule is chosen. If the day of the test is known in advance. page 69 .INTERVIEWER’S SHEET Monday 9:00-10:00 AM 12:00-1:00 PM 1:00-2:00 PM 5:30-6:30 PM 3:00-5:00 PM _:__-11:30 PM 11:00 AM – 12:00 PM Spanish Class Lunch Tuesday 9:00-10:00 AM Spanish Class English Class Lunch Nap Geography Class Chemistry Class Soccer Practice Dinner Study 11:00 AM – 1:00 PM 1:00-2:00 PM 2:00-3:30 PM 8:30 PM – 12:00 AM 12:00-1:30 AM 7:30-8:30 PM 10:00-11:00 AM Math Class Dinner Study 11:30 PM-12:30 AM Watch TV Watch Movie How to Formulate New Tasks: If creating a new schedule or activity.questions and verbs. use one which the student will be familiar with and will elicit various types of Wh. Choice of classes and activities in the schedule should be tailored such that they will be recognizable to students in order to retain the focus on grammar and not on vocabulary. Spec Packet. EIL 460. Fall 2012.

Fall 2012. My parents might buy me an iPod. the possibility of a certain team winning a game. etc. the evaluator will 6) ask the student to imagine a certain situation in which he or she can make guesses such as guessing gifts. Task Instructions to Give Students: In this part of the task. However. How to Formulate New Tasks: Other scenario might be applicable for this task. construct 5 sentences using modals of degree of certainty in present tense. page 70 . In this task. 9) prompt the student if he or she pauses for certain amount of time. you will 3) imagine your birthday party 4) guess orally 5 gifts that you get using modals of degree of certainty in simple present tense Sample Student Responses: I may get a book. This test should last no longer than five minutes. a weather report. My boyfriend could give me a rose. In this part of the task. a football team’s possibility to win a game. it is necessary to set a scenario in which people usually make guesses such as guessing gifts. 8) forbid the students to jot down his or her guesses before he or she produces them orally. 10) end the task if he or she struggles to construct the guess or the time is over. Spec Packet. Have the student choose one of the scenarios that he or she really understands. 7) have the student create 5 guesses to which you must not respond. EIL 460. or weather prediction.Task 4: Modals (Possibility) Interviewer General Instructions: This part of the task will focus on how well the student can demonstrate the use of modal of degree of certainty in present tense.

The test will begin with the student speaking freely. 1985) Spec Packet. the interviewer can end the test. then read the description. This is an adaptive test. “What is this lady doing?” “What is the shoe store owner doing? Once the student has reached the point where he/she is having a difficult time producing any more meaningful thoughts or has described every person in the picture. the student should look at the picture. In this task.question. This task should last no longer than five minutes. While listening closely to the description. Describe all the things that you see that are happening. the evaluator will hand a picture to the student. You are looking out your window and this is what you see. Fall 2012. Task Instructions to Give Students: Look at the picture below. but if a student consistently struggles to answer. page 71 . (Communicative Grammar Practice. For example. The student will have 15 seconds to look at the picture and think about what they want to say and then respond orally. Leo Jones. the interviewer can help direct the students focus by pointing to a picture and ask a WH. You have 15 seconds to review the picture before you speak.Task 5: Present Progressive Interviewer General Instructions: This portion of the test will focus on how well the student can correctly use the present progressive and control it in conversation. EIL 460. Imagine that you are in your apartment.

Spec Packet. Fall 2012. the teacher may choose another picture that focuses more on the vocabulary or on a topic covered during the course. How to Formulate New Tasks: A video shown on silent may be subsisted for this picture scenario card. page 72 . it’s important that the students have general knowledge of the vocabulary in order to retain the focus on grammar. This would create a more challenging task for students though since the students won’t have the 15 seconds to compose their students as in the picture task. EIL 460. Whatever image is used. When choosing another picture. otherwise it could become a test of vocabulary knowledge. it’s important to make sure that the picture has enough images to produce a wide variety of sentences in the progressive form.Sample Student Responses: A boy is selling newspapers. A lady is walking her dog. The man with a moustache is sweeping. Additionally.

the interviewer can help direct the students focus by pointing to a picture and ask a W-H question. the interviewer will hand the picture over to the student then read the description. and Mrs. the interviewer can end the task. Mr. While listening closely to the description. This task should last no longer than five minutes. Task Instructions to Give Students: Look at the picture below. the student should look at the picture. Describe all the things that the guests were doing when the thief stole the jewelry at midnight. For example. Spec Packet. EIL 460. “What was this man doing?” Once the student has reached the point where he/she is having a difficult time producing any more meaningful thoughts or has described every person in the picture. You have 15 seconds to review the picture before you speak. In this task. Fall 2012. page 73 .Task 6: Past Progressive Interviewer General Instructions: This portion of the test will focus on how well the student can correctly use past progressive and control it in conversation. Gold’s jewelry at midnight. A thief stole Mrs. Gold invited some friends to their house for the weekend. The student will have 15 seconds to look at the picture and think about what they want to say and then respond orally. The test will begin with the student speaking freely. but if a student consistently struggles to answer.

page 74 . Mr. (When the thief stole the jewelry). the teacher may choose another picture that focuses more on the vocabulary or on a topic covered during the course. a mouse was eating cheese. Gold were sleeping. Mr. it’s important to make sure that Spec Packet. Blue was watching TV. EIL 460. and Mrs. When choosing another picture. Orange was reading.(Azar Beginner Grammar Textbook Series) Sample Student Responses: When Ms. Fall 2012. Additionally. This would create a more challenging task for students though since the students won’t have the 15 seconds to compose their students as in the picture task. How to Formulate New Tasks: A video shown on silent may be subsisted for this picture scenario card.

What were you eating in your car?” Spec Packet. EIL 460. otherwise it could become a test of vocabulary knowledge. This task can also be made into a 1-on-1 conversation by asking the student questions about what they did yesterday during certain times. page 75 . it’s important that the students have general knowledge of the vocabulary in order to retain the focus on grammar. Whatever image is used. the teacher could imagine that they are a policeman and interview the suspect (the student) by asking a variety of questions that should product the progressive aspect such as “What were you doing at 8 AM yesterday?” or “We found evidence of food in your car.the picture has enough images to produce a wide variety of sentences in the progressive form. If names are assigned to people in the picture. make sure the names of the fictitious characters can be pronounced easily like this task does with colors. Fall 2012. For example. as in this sample task.

such as “What about homework?” or “What about talking in class?” Care should be taken not to suggest too many forms to the student. generating content). In this task. ensure that it is possible to elicit various necessity modals. Whatever might be chosen. some prompting may be appropriate. You have to bring a textbook and a pen to class. however. Fall 2012. of course. they may try to avoid them altogether by using ability or advice modals. Try to structure the task in such a way that it is not possible (or not easy) to use ability modals like ‘can’ and ‘can’t. Sample Student Responses: You must not smoke in class. bearing in mind that avoidance. the evaluator will read the prompt to the student.Task 7: Modals (Necessity) Interviewer General Instructions: This portion of the test will focus on the student’s knowledge and control of necessity modals. imaginable for this language feature. How to Formulate New Tasks: Other scenarios are. may intrude. Think of some rules that you will tell your class on the first day. It is the first day of class. Both activity and directions can be useful tools to attempt to prevent this sort of avoidance. Task Instructions to Give Student: Imagine you are a teacher in a high school. EIL 460. Spec Packet. This test should last no longer than five minutes. If the student seems to be having serious difficulties with inventing rules (i.’ If students are not comfortable with necessity modals. page 76 . You must turn your cell phone off.e. The student will have 15-20 seconds to think before beginning to respond orally. whether intentional or unintentional.

The student will be assessed on the correct use of articles and quantifiers with the accompanying count and non-count noun(s). EIL 460. If the student only describes plural nouns. the student will be given a photo of an open refrigerator/freezer with various kinds of food inside of it. then the interviewer should prompt the student to describe a single item (to elicit the use of the articles "a" or "an"). "many". and vice versa. and vice versa. The student should describe at least 6 items in the photo. the interviewer should prompt the student to describe a non-count item. and verb agreement will be assessed for the oral grammar assessment. This task should take no longer than five minutes. If the student only produces statements using count nouns. Interviewer Follow-up: How many apples are there? Student: There are a lot of apples in the fridge. "some". Task Instructions to Give Students: Describe what you see in this picture. The student may be creative in their descriptions and produce longer sentences. the interviewer may elicit responses by using guiding statements or questions such as the ones below: Interviewer (pointing to an item): Can you describe this? Or if the student needs more guidance. This task will also produce data for the assessment of verb agreement in the student's utterances. Interviewer: Are there any apples in the fridge? Student: Yes. quantifiers. but only the articles.Task 8: Articles / Quantifiers Interviewer General Instructions: In this task. The student will describe the items in the photo. there are. page 77 . "lots of". You may use words such as "a lot of". and "an". "a". Fall 2012. If the student is having trouble producing the sentences. Spec Packet.

may be used for this task. How to Formulate New Tasks: Other picture scenarios. There are three peppers in the fridge.Sample Student Responses: There is fruit in the bottom drawer. Whatever picture that is used should show a variety of count and non-count nouns. these nouns Spec Packet. Fall 2012. page 78 . such as a shelf in a store or a room in a house. EIL 460. The cake is on the top shelves.

EIL 460. Fall 2012.should be ones that the students should have in their lexical knowledge through lessons that were taught in the 300 level or in the previous 100 or 200 level courses. page 79 . Spec Packet. or color photos as long as the picture is clear and large enough to see the individual items in the picture. color drawings. Pictures can be black and white line drawings.

page 80 .Appendix Figure 1 Spec Packet. Fall 2012. EIL 460.

EIL 460. overall communication of ideas is still clear Inconsistently Significant and consistent errors that sometimes confuse meaning Simple Present Task 1: Many errors that making meaning unclear.Questions Task 2: Task 3: Simple Past Possibility Modals Task 4: Present Progressive Task 5: Task 6: Past Progressive Task 7: Necessity Modals Task 8: Articles Quantifiers Spec Packet.Figure 2 SCORE Grammar A few errors. Unsatisfactorily WH. Fall 2012. Listeners’ attention is diverted to the errors from the message. page 81 . overall communication of ideas is clear Very Well Satisfactory Several errors.

These test items are representative of a variety of items that would appear of the test to reflect different teacher’s preferences for formatting and instructions. Spec Packet. etc. Fall 2012. this test targets intermediate-level English-language learners who would be expected to succeed in American university-level courses. oral. Sophie Qin. Specific Objectives 5.e. Assuming incoming students have a sufficient score to enter the university. Additionally. Assessing students’ ability to understand written directions of assignment. losing points due to inability to follow instructions properly). 7. 6. While the EPT tests writing skills and pronunciation. EIL 460. 8. this supplement will assess students’ ability to succeed in a university course and classroom setting by following instructions in a variety of formats. including accent. page 82 . Assessing students’ ability to understand and follow oral directions for in-class work. it will identify which topics need to be covered in a “classroom survival skills” course. Assessing students’ ability to understand and apply a rubric. Becky Menendez.. and Mica Swyers Introduction This test is designed to serve as a supplement to the English Proficiency Test (EPT) given at the University of Illinois Urbana-Champaign for incoming non-native speakers of English. General Objectives The objective of this test is to measure students’ ability to understand and implement different forms of feedback in an academic setting. Version 2 Katie Honken. Assessing student’s ability to understand written feedback. written vs. speed of delivery. This EPT supplementary section is necessary to reduce the influence of students’ ability to follow instructions on their performance in a university-level course (i.Understanding Instructions. Students who do not demonstrate sufficient ability will be placed in a basic “classroom survival skills” class in conjunction with their ESL courses at the University of Illinois.

among them only one format is correct and the other three are with different problems including spacing. The examinees will face four types of formats in screenshots. EIL 460. Response Attributes (RA) To select the right format from the four options. A: The title is not centered and the passage is single spaced. The computer testing software will record the result and categorize it as correct or incorrect. and those instructions are required to be understood by students in university-level courses.) Title: Times New Roman. font looks like. and select the format that meets all requirements. Answer key: B: Correct. The key screenshot correctly follows the instructions provided in the item stem. bold. the examinees need to read through the instructions. page 83 . C: The title is not bold and the text size is 10pt. margin. The item stem will be given in a written format: detailed instructions of editing a paragraph will be provided to examinees. size or editing mistakes in the title. centered Which of the following formats meets the teacher’s requirement? ________ Spec Packet.) Spacing: Double-spaced Size: 12pt (This is what 12pt. Sample Question 1 (Written): In a writing class students are required to edit the provided passage based on the following format: Font: Times New Roman (This is what Times New Roman looks like. Fall 2012. D: The font is not Times New Roman.Objective #1: Assessing students’ ability to understand written directions of assignment Item 1: Formatting Prompt Attributes (PA) This is a four-option multiple choice test item. Correct answer earns 2 points while incorrect answers earn 0 points. Then they will the information from instructions to the formats displayed in screenshots. The three distractors will depict various formats that arise from misunderstanding of the instructions.

page 84 .Choice A Choice B Choice C Spec Packet. EIL 460. Fall 2012.

EIL 460. Fall 2012. page 85 .Choice D Prompt Attributes (PA) Spec Packet.

the examinees need to read through the instructions. among them only one format is correct and the other three are with different problems including spacing. size or editing mistakes in the title.” Spec Packet. (This simulates the situation where students can ask the instructor to repeat instructions but within reasonable limits. Then they will the information from instructions to the formats displayed in screenshots. The computer testing software will record the result and categorize it as correct or incorrect. and select the format that meets all requirements. such as those given in class. D: The font is not Times New Roman. The title should be Times New Roman as well. 12 point font. Sample Question 2 (Oral): The examinee(s) should hear: “Your paper should be double-spaced in Times New Roman. The key screenshot correctly follows the instructions provided in the item stem. Answer key: B: Correct. The three distractors will depict various formats that arise from misunderstanding of the instructions. EIL 460. bolded and centered at the top of the page. Fall 2012. Response Attributes (RA) To select the right format from the four options. C: The title is not bold and the text size is 10pt.) The examinees will face four types of formats in screenshots. and those instructions are required to be understood by students in university-level courses. page 86 . margin. The item stem will be given in an oral format: detailed instructions of editing a paragraph will be provided to examinees. Correct answer earns 2 points while incorrect answers earn 0 points.This is a four-option multiple choice test item which tests students’ ability to hear and understand oral formatting instructions. Students will be allowed to listen to the clip twice. A: The title is not centered and the passage is single spaced.

Fall 2012. EIL 460.Choice A Choice B Spec Packet. page 87 .

EIL 460. page 88 .Choice C Choice D Item 2: Writing correct file names Spec Packet. Fall 2012.

” the file name should be: 04_15_Pattinson_Robert_Environment Issues Please use the provided information and write the correct file names on the blank. The right answer should be correct in all aspects including the sequence of various information. and a sample file name is also presented to help them understand the instructions.Prompt Attributes (PA) This is a writing test item. Spec Packet. The instructions of writing correct file names are required to be comprehended by students in university-level courses. EIL 460.5 point. 1) Student’s Name: Barack Obama Date: August 5th Title of Paper: How to be a Good President ___________________________________________________________________ 2) Student’s Name: Mitt Romney Date: November 24th Title of Paper: If I were the President of the United States ___________________________________________________________________ Answer Key: 1) 08_05_Obama_Barack_How to be a Good President 2) 11_24_Rommey_Mitt_If I were the President of the United States Objective #2: Assessing students’ ability to understand and follow oral directions for in-class work. The file name should follow this format: mm_dd_Last Name_First Name_Title_of_Paper Example: If it is April 15th. capitalization and symbols. page 89 . the teacher requires students to upload their homework on the course management system with correct file names. This may require examinees to read through the instructions and understand the sample presented. Fall 2012. and each incorrect portion deducts 0. Response Attributes (RA) Examinees will come up with correct file names based on the information provided. The item stem will be given in a written format: detailed instructions of writing a correct file name will be provided to examinees. and Robert Pattinson’s paper is named “Environment Issues. Correct answer of a whole file name earns 5 points. Sample Question In a writing class.

peer review. Test-takers will be allowed to replay the clip once. Then s/he will have to process the information and select the picture (by clicking) that best illustrates the instructions that were given. the examinee will have to listen to the sound file.) The examinee faces a computer monitor divided into quadrants (one for each choice). for tasks such as groupwork. get into groups of three or four and come up with a paragraph responding to the question on the board. The drawings below are “mock-up” pictures that would not actually be included in the actual test.Prompt Attributes (PA) This is a four-option multiple choice test item. The item stem will be given in oral format: the examinee will listen to a sound file of a teacher giving instructions for an in-class assignment. You should only turn in one paragraph per group at the end of the class. Sound clips should feature different voices (male vs. female. The three distractors will depict different scenarios that could arise from following the instructions incorrectly or incompletely. (This simulates the situation where students can ask the instructor to repeat instructions but within reasonable limits. This will represent students receiving instructions from different teachers in a university-setting. These will be instructions that a student would hear in a university-level course.) and different speeds of delivery. test-takers still have to determine which choice best illustrates the correct following of instructions. Instead. They could also be given short video clips of groupwork scenarios. different accents. Fall 2012. etc. EIL 460. In either case. Response Attributes (RA) To select the correct choice of the four options. students might see pictures of real humans depicting different scenarios resulting from following the instructions correctly/incorrectly. The key picture correctly illustrates the directions given in the oral item stem.” Spec Packet. or an in-class assignment. each with a different illustration. The computer testing software will record the item that the student selected and categorize it as correct or incorrect. Sample Item The examinee(s) should hear: “Now. page 90 .

Picture B is incorrect because each student is turning in a separate paper. where the figures are discussing and producing one paper. Fall 2012. Picture C is incorrect because each student is working separately (and without discussion) on a paragraph.) Spec Packet. EIL 460. page 91 . Picture D is incorrect because while the students are engaged in a lively discussion. and the first student’s paper is multiple pages long.(Answer Key to Sample Item: The correct choice is Picture A. there is no evidence that they are producing a paragraph to submit at the end of class.

The second and third distracters (C & D) are incorrect because they recommend changes to components that are already correct. The only correct choice is whichever number corresponds to the score the fictional student should receive in that specific category. Prompt Attributes (PA) This is a four-item. the key will be the choice (A) that states that the example student should change nothing. The other choices will each reflect one of the more specific requirements of the assignment. The student should not recognize the key immediately. In general. The second question asks students to identify one change the fictional student could make to increase their score in the specified category. 2. The second distracter (B) is incorrect because it is recommending a change for the wrong category. Next.Objective #3: Assessing students’ ability to understand and apply a rubric. the first question focuses on one of the rubric categories and asks how many points the fictional student earned with their attempt at the assignment. The stem briefly introduces a fictional student who must complete an assignment. the key will be the choice (D) that correctly identifies one change that would raise their score in the specified category. and 0. The first distracter (A) is incorrect because the fictional student did not earn the maximum number of points. Rubric categories should not include technical writing terms or concepts (as students may not have received instruction concerning these yet). rubrics should Spec Packet. The third distracter (C) recommends the wrong change. However. The first distracter (B) is incorrect because it recommends a change for the wrong category. Fall 2012. it introduces the fictional student’s attempt at the assignment (in the form of a screenshot). although for the right category. In cases where the fictional student has earned the maximum number of points in the specified rubric category. It also introduces a rubric (in the form of a table) for that assignment. EIL 460. Students should not feel challenged by the language of the prompt itself. 1. based on the requirements laid out by the rubric. page 92 . They must examine both the rubric and the assignment. The grammar and complexity of the scenario description and item directions should be very basic. and they will compare the assignment against the rubric requirements to acquire the information they need to correctly answer the question. so that reading comprehension abilities do not interfere with the student’s ability to identify the correct answer. Of the two questions. In cases where the fictional student has NOT earned the maximum number of points in the specified rubric category. One choice will always be “nothing” because the fictional student has already earned the maximum score in that category. The grammar and complexity of the assignment should not exceed that of the materials they might be expected to encounter in ESL114 class materials. The choices are always 3. Each item consists of a stem and two questions. multiple-choice test item. they should also be fairly basic.

Size 12. Paragraph has at least three supporting detail sentences that relate back to the main idea. This may require re-reading of the rubric and of the assignment screenshot. Incorrect answers earn 0 points. The main idea of the The main idea is paragraph is clear and clear but it is incorrectly placed.include language very comparable to what is used in ESL114 rubrics. Fall 2012. by which the paragraph will be scored: Paragraph Rubric 3 points Topic sentence 2 points 1 points The main idea is unclear. Support Paragraph has only one supporting detail sentence that relates back to the main idea. each of these types of items may earn a maximum of 2 points (1 for each of the two questions). Grammar errors and incorrect word choices are distracting.the formatting spaced. The paragraph is less than 65 words long. Roman. Paragraph has no supporting detail sentences that relate back to the main idea. Sample Item: Jemima has been given a paragraph writing assignment in her ESL class. 0 points The paragraph does not have a main idea. Correct answers earn 1 point each. Grammar errors and incorrect word choices make it difficult to understand the paragraph. EIL 460. Grammar errors and incorrect word choices do not distract the reader from the important points Paragraph has two supporting detail sentences that relate back to the main idea . The paragraph does not meet any of the 4 formatting requirements. appears at the beginning of the paragraph. When Jemima finishes writing her paragraph. page 93 . Length The paragraph is at least 85 words long. The paragraph is 75 The paragraph is 65 to 84 words long. Students will be assessed on whether each of their selections is correct or incorrect. Grammar errors and incorrect word choices prevent the reader from understanding the paragraph. even if this language is more challenging than the language contained in the prompt or assignment. this is what it looks like on her computer screen: Spec Packet. The paragraph meets 1 out of 3 of the formatting requirements. Times New requirements. to 74 words long. Thus. Response Attributes (RA) Students will select the correct answer for each of the two questions in the item. Grammar Formatting The paragraph meets The paragraph all 3 formatting meets 2 out of 3 of requirements: double. The instructor provided the following rubric.

D Adapted from: Writing Academic English. 0 Question #2: What is one change Jemima could make to get more points in the category of formatting? A. B. – The Longman Academic Writing Series Spec Packet. 2 C. She should make her paragraph longer. 3 B.Question #1: Based on the instructor’s rubric. EIL 460. She should change her paragraph spacing. C. Fourth Edition. 1 D. Fall 2012. Key: 1. She already has the highest number of points possible in this category. how many points will Jemima get in the category of formatting? A. Nothing. D. Alice Oshima and Ann Hogue. page 94 . B 2. She should change her font size.

Test-takers are shown a screen-shot of a paragraph written by a student with teacher comments. the test-taker will examine the four options and choose the one that best applies teacher feedback. page 95 . Sample Item The following paragraph was submitted by a student. Next. but test-takers should choose the best option. One option will show one way that the teacher feedback could be applied well. Teacher Feedback: Which of the following best applies the teacher feedback for the paragraph shown above? Option A: Spec Packet. Response Attributes (RA): The test-taker should first examine the paragraph with teacher comments. EIL 460. More than one option might seem to apply all of the teacher feedback. Students will give either a correct or incorrect response. Fall 2012. Three of the options will show the paragraph with incorrect or incomplete application of teacher feedback. Four options will be shown as screen-shots of the same paragraph with changes. The task tests test-takers’ ability to understand and apply written feedback. The teacher has given the student feedback so that the student may revise the paragraph.Objective #4: Assessing student’s ability to understand written feedback Prompt Attributes (PA): This is a four-option multiple-choice test question.

page 96 . Fall 2012.Option B: Option C: Spec Packet. EIL 460.

page 97 .Option D: Spec Packet. EIL 460. Fall 2012.

sentence split incorrectly) Spec Packet.Answer Key to Sample Item: Option A: (Does not have at least five sentences) Option B: (Correct) Option C: (Attempted to apply all feedback but still incorrect) Option D: (Does not have a transition and at least five sentences. Fall 2012. EIL 460. page 98 .

Understanding Instructions, Version 2.1 Katie Honken, Sophie Qin, Becky Menendez, and Mica Swyers

Introduction
This test is designed to serve as a supplement to the English Proficiency Test (EPT) given at the University of Illinois Urbana-Champaign for incoming non-native speakers of English. Assuming incoming students have a sufficient score to enter the university, this test targets intermediate-level English-language learners who would be expected to succeed in American university-level courses. While the EPT tests writing skills and pronunciation, this supplement will assess students’ ability to succeed in a university course and classroom setting by following instructions in a variety of formats. These test items are representative of a variety of items that would appear of the test to reflect different teacher’s preferences for formatting and instructions, including accent, speed of delivery, written vs. oral, etc. Students who do not demonstrate sufficient ability will be placed in a basic “classroom survival skills” class in conjunction with their ESL courses at the University of Illinois. This EPT supplementary section is necessary to reduce the influence of students’ ability to follow instructions on their performance in a university-level course (i.e., losing points due to inability to follow instructions properly).

General Objectives
The objective of this test is to measure students’ ability to understand and implement different forms of feedback in an academic setting. Additionally, it will identify which topics need to be covered in a “classroom survival skills” course.

Specific Objectives
9. Assessing students’ ability to understand written directions of assignment. 10. Assessing students’ ability to understand and follow oral directions for in-class work. 11. Assessing students’ ability to understand and apply a rubric. 12. Assessing student’s ability to understand written feedback.

Spec Packet, EIL 460, Fall 2012, page 99

Objective #1: Assessing students’ ability to understand written directions of assignment
Item 1: Formatting
Prompt Attributes (PA) This is a four-option multiple choice test item. The item stem will be given in a written format: detailed instructions of editing a paragraph will be provided to examinees, and those instructions are required to be understood by students in university-level courses. The examinees will face four types of formats in screenshots, among them only one format is correct and the other three are with different problems including spacing, margin, size or editing mistakes in the title. The key screenshot correctly follows the instructions provided in the item stem. The three distractors will depict various formats that arise from misunderstanding of the instructions. Response Attributes (RA) To select the right format from the four options, the examinees need to read through the instructions. Then they will the information from instructions to the formats displayed in screenshots, and select the format that meets all requirements. The computer testing software will record the result and categorize it as correct or incorrect. Correct answer earns 2 points while incorrect answers earn 0 points. Answer key: B: Correct. A: The title is not centered and the passage is single spaced. C: The title is not bold and the text size is 10pt. D: The font is not Times New Roman. Sample Question 1 (Written): In a writing class students are required to edit the provided passage based on the following format: Font: Times New Roman (This is what Times New Roman looks like.) Spacing: Double-spaced Size: 12pt (This is what 12pt. font looks like.) Title: Times New Roman, bold, centered Which of the following formats meets the teacher’s requirement? ________

Spec Packet, EIL 460, Fall 2012, page 100

Choice A

Choice B

Choice C

Spec Packet, EIL 460, Fall 2012, page 101

Choice D

Prompt Attributes (PA)

Spec Packet, EIL 460, Fall 2012, page 102

This is a four-option multiple choice test item which tests students’ ability to hear and understand oral formatting instructions, such as those given in class. The item stem will be given in an oral format: detailed instructions of editing a paragraph will be provided to examinees, and those instructions are required to be understood by students in university-level courses. Students will be allowed to listen to the clip twice. (This simulates the situation where students can ask the instructor to repeat instructions but within reasonable limits.) The examinees will face four types of formats in screenshots, among them only one format is correct and the other three are with different problems including spacing, margin, size or editing mistakes in the title. The key screenshot correctly follows the instructions provided in the item stem. The three distractors will depict various formats that arise from misunderstanding of the instructions.

Response Attributes (RA) To select the right format from the four options, the examinees need to read through the instructions. Then they will the information from instructions to the formats displayed in screenshots, and select the format that meets all requirements. The computer testing software will record the result and categorize it as correct or incorrect. Correct answer earns 2 points while incorrect answers earn 0 points. Answer key: B: Correct. A: The title is not centered and the passage is single spaced. C: The title is not bold and the text size is 10pt. D: The font is not Times New Roman.

Sample Question 2 (Oral):
The examinee(s) should hear: “Your paper should be double-spaced in Times New Roman, 12 point font. The title should be Times New Roman as well, bolded and centered at the top of the page.”

Spec Packet, EIL 460, Fall 2012, page 103

EIL 460.Choice A Choice B Spec Packet. Fall 2012. page 104 .

EIL 460. Fall 2012.Choice C Choice D Item 2: Writing correct file names Spec Packet. page 105 .

Prompt Attributes (PA) This is a writing test item. The item stem will be given in a written format: detailed instructions of writing a correct file name will be provided to examinees.” the file name should be: 04_15_Pattinson_Robert_Environment Issues Please use the provided information and write the correct file names on the blank. Correct answer of a whole file name earns 5 points.5 point. and a sample file name is also presented to help them understand the instructions. Response Attributes (RA) Examinees will come up with correct file names based on the information provided. Spec Packet. EIL 460. 1) Student’s Name: Barack Obama Date: August 5th Title of Paper: How to be a Good President ___________________________________________________________________ 2) Student’s Name: Mitt Romney Date: November 24th Title of Paper: If I were the President of the United States ___________________________________________________________________ Answer Key: 1) 08_05_Obama_Barack_How to be a Good President 2) 11_24_Rommey_Mitt_If I were the President of the United States Objective #2: Assessing students’ ability to understand and follow oral directions for in-class work. The file name should follow this format: mm_dd_Last Name_First Name_Title_of_Paper Example: If it is April 15th. page 106 . the teacher requires students to upload their homework on the course management system with correct file names. This may require examinees to read through the instructions and understand the sample presented. The right answer should be correct in all aspects including the sequence of various information. and Robert Pattinson’s paper is named “Environment Issues. Fall 2012. Sample Question In a writing class. capitalization and symbols. and each incorrect portion deducts 0. The instructions of writing correct file names are required to be comprehended by students in university-level courses.

depending on the prompt). if the question item specifically mentions. different accents. page 107 . for example. about 15-20 seconds each.Prompt Attributes (PA) This is a four-option multiple choice test item. The drawings below are “mock-up” pictures that would not actually be included in the actual test.) and different speeds of delivery. Test-takers will be allowed to replay the clip once.) The examinee faces a computer monitor divided into quadrants (one for each choice). Instead. The background setting of the pictures/videos will be simple and uncluttered. The computer testing software will record the item that the student selected and categorize it as correct or incorrect. You should only turn in one paragraph per group at the end of the class. Sound and video clips should feature different voices (male vs. The item stem will be given in oral format: the examinee will listen to a sound file of a teacher giving instructions for an in-class assignment.” Spec Packet. Fall 2012. In either case. peer review. The key picture correctly illustrates the directions given in the oral item stem. or an in-class assignment. They could also be given short video clips of groupwork scenarios. for tasks such as groupwork. These will be instructions that a student would hear in a university-level course. Group dialogues will be based on actual in-class activities from a variety of university classes. a question written on the board. This will represent students receiving instructions from different teachers in a university setting. Response Attributes (RA) To select the correct choice of the four options. get into groups of three or four and come up with a paragraph responding to the question on the board. EIL 460. this will appear in the pictures/video clips. etc. Sample Item The examinee(s) should hear: “Now. students might see pictures of real humans depicting different scenarios resulting from following the instructions correctly/incorrectly. Each picture/video will draw from the same group of student actors (although the total number of actors may vary from scene to scene. female. All background elements not essential to the question item will be identical in all four choices. Then s/he will have to process the information and select the picture (by clicking) that best illustrates the instructions that were given. All photos and video clips will be in color and will be captured in the same classroom. the examinee will have to listen to the sound file. However. (This simulates the situation where students can ask the instructor to repeat instructions but within reasonable limits. Sound and video clips will be long enough to give test takers a sense of how the directions are being followed. each with a different illustration. The three distractors will depict different scenarios that could arise from following the instructions incorrectly or incompletely. test-takers still have to determine which choice best illustrates the correct following of instructions.

) Spec Packet. there is no evidence that they are producing a paragraph to submit at the end of class. and the first student’s paper is multiple pages long. Fall 2012. Picture B is incorrect because each student is turning in a separate paper. Picture D is incorrect because while the students are engaged in a lively discussion. page 108 .(Answer Key to Sample Item: The correct choice is Picture A. EIL 460. where the figures are discussing and producing one paper. Picture C is incorrect because each student is working separately (and without discussion) on a paragraph.

The grammar and complexity of the scenario description and item directions should be very basic. The second and third distracters (C & D) are incorrect because they recommend changes to components that are already correct. the key will be the choice (D) that correctly identifies one change that would raise their score in the specified category. Prompt Attributes (PA) This is a four-item. Each item consists of a stem and two questions. The only correct choice is whichever number corresponds to the score the fictional student should receive in that specific category. Fall 2012. 1. The stem briefly introduces a fictional student who must complete an assignment. they should also be fairly basic. The second question asks students to identify one change the fictional student could make to increase their score in the specified category. The other choices will each reflect one of the more specific requirements of the assignment. and they will compare the assignment against the rubric requirements to acquire the information they need to correctly answer the question. It also introduces a rubric (in the form of a table) for that assignment. Rubric categories should not include technical writing terms or concepts (as students may not have received instruction concerning these yet). 2. In cases where the fictional student has earned the maximum number of points in the specified rubric category. although for the right category. rubrics should Spec Packet. Students should not feel challenged by the language of the prompt itself. The second distracter (B) is incorrect because it is recommending a change for the wrong category. However. multiple-choice test item. it introduces the fictional student’s attempt at the assignment (in the form of a screenshot). The third distracter (C) recommends the wrong change. so that reading comprehension abilities do not interfere with the student’s ability to identify the correct answer. One choice will always be “nothing” because the fictional student has already earned the maximum score in that category. the key will be the choice (A) that states that the example student should change nothing. based on the requirements laid out by the rubric. and 0. page 109 . The grammar and complexity of the assignment should not exceed that of the materials they might be expected to encounter in ESL114 class materials. Next. In cases where the fictional student has NOT earned the maximum number of points in the specified rubric category. They must examine both the rubric and the assignment. EIL 460. the first question focuses on one of the rubric categories and asks how many points the fictional student earned with their attempt at the assignment. The student should not recognize the key immediately.Objective #3: Assessing students’ ability to understand and apply a rubric. The choices are always 3. The first distracter (A) is incorrect because the fictional student did not earn the maximum number of points. In general. Of the two questions. The first distracter (B) is incorrect because it recommends a change for the wrong category.

0 points The paragraph does not have a main idea. Length The paragraph is at least 85 words long. to 74 words long. Incorrect answers earn 0 points. Correct answers earn 1 point each. Paragraph has at least three supporting detail sentences that relate back to the main idea. Times New requirements. Grammar errors and incorrect word choices prevent the reader from understanding the paragraph. by which the paragraph will be scored: Paragraph Rubric 3 points Topic sentence 2 points 1 points The main idea is unclear. The paragraph is less than 65 words long. Response Attributes (RA) Students will select the correct answer for each of the two questions in the item. The instructor provided the following rubric. page 110 . each of these types of items may earn a maximum of 2 points (1 for each of the two questions). Size 12. Roman. Support Paragraph has only one supporting detail sentence that relates back to the main idea. Sample Item: Jemima has been given a paragraph writing assignment in her ESL class. The paragraph does not meet any of the 4 formatting requirements. Grammar errors and incorrect word choices are distracting. Grammar Formatting The paragraph meets The paragraph all 3 formatting meets 2 out of 3 of requirements: double. even if this language is more challenging than the language contained in the prompt or assignment. Fall 2012. Grammar errors and incorrect word choices make it difficult to understand the paragraph.include language very comparable to what is used in ESL114 rubrics. Students will be assessed on whether each of their selections is correct or incorrect. this is what it looks like on her computer screen: Spec Packet. When Jemima finishes writing her paragraph. EIL 460.the formatting spaced. Thus. The paragraph meets 1 out of 3 of the formatting requirements. The paragraph is 75 The paragraph is 65 to 84 words long. Grammar errors and incorrect word choices do not distract the reader from the important points Paragraph has two supporting detail sentences that relate back to the main idea . appears at the beginning of the paragraph. Paragraph has no supporting detail sentences that relate back to the main idea. This may require re-reading of the rubric and of the assignment screenshot. The main idea of the The main idea is paragraph is clear and clear but it is incorrectly placed.

Question #1: Based on the instructor’s rubric. D Adapted from: Writing Academic English. She should change her paragraph spacing. D. Alice Oshima and Ann Hogue. Key: 1. B 2. 1 D. how many points will Jemima get in the category of formatting? A. She should make her paragraph longer. 2 C. B. She should change her font size. 3 B. EIL 460. Fourth Edition. – The Longman Academic Writing Series Spec Packet. Fall 2012. C. Nothing. She already has the highest number of points possible in this category. page 111 . 0 Question #2: What is one change Jemima could make to get more points in the category of formatting? A.

Four options will be shown as screen-shots of the same paragraph with changes. Teacher Feedback: Which of the following best applies the teacher feedback for the paragraph shown above? Option A: Spec Packet. page 112 . Sample Item The following paragraph was submitted by a student. Students will give either a correct or incorrect response. Fall 2012. Next. but test-takers should choose the best option. the test-taker will examine the four options and choose the one that best applies teacher feedback. One option will show one way that the teacher feedback could be applied well. The task tests test-takers’ ability to understand and apply written feedback. The teacher has given the student feedback so that the student may revise the paragraph. Response Attributes (RA): The test-taker should first examine the paragraph with teacher comments. Test-takers are shown a screen-shot of a paragraph written by a student with teacher comments. Three of the options will show the paragraph with incorrect or incomplete application of teacher feedback. EIL 460. More than one option might seem to apply all of the teacher feedback.Objective #4: Assessing student’s ability to understand written feedback Prompt Attributes (PA): This is a four-option multiple-choice test question.

Fall 2012.Option B: Option C: Spec Packet. EIL 460. page 113 .

Fall 2012. page 114 .Option D: Spec Packet. EIL 460.

sentence split incorrectly) Spec Packet. EIL 460. page 115 .Answer Key to Sample Item: Option A: (Does not have at least five sentences) Option B: (Correct) Option C: (Attempted to apply all feedback but still incorrect) Option D: (Does not have a transition and at least five sentences. Fall 2012.

Fall 2012.Oral interview .Expressing concern Methods: . page 116 .Making small talk . EIL 460. Fall 2012 Alberto Domínguez Yelena Forrester Knezevic Widya Ratna Kusumaningrum Language Skills: Speaking Pragmatics Item types: . and Yelena Forrester Knezevic Pragmatic Test Specs: The “Barry Item” EIL 460. Version 2 Alberto Domínguez.Excusing oneself .The Barry Spec -.Requesting information .Testing Pragmatics. Widya Ratna Kusumaningrum.Conversation .Short answer Spec Packet.

The test will be administered in a one-on-one setting. Testing will occur after the instructor feels that students have reached the appropriate level. page 117 . Participants and Setting: This test is designed for advanced-level ESL students. no previous knowledge of these topics is expected before students begin the class. The objectives for the course will be to increase students’ sociolinguistic. EIL 460. However. and express concern to friends about their potentially harmful behavior. The examiner will not be the student’s current instructor. General specs: These specs are applicable in a classroom setting where students are aware of the pragmatic content and have agreed to be tested on potentially sensitive subjects.” Spec Packet. but a teacher of another section of the course. Fall 2012. Students should be familiar with the types of tasks that will be tested before taking this test. with one examiner and one test-taker at a time. this test is designed to measure students’ proficiency in these areas. Rather. request and give information. I understand that I will not be tested on whether my answers follow strict patterns or formulas. because this is not a diagnostic or placement test.Background: This test will be administered in a pragmatics course for advanced-level ESL students. The testtaker will have signed off on something like the following: “This class will include tasks that test my ability to conduct small talk. I understand that some of these topics may be sensitive. Instructors will introduce the concepts during the course. excuse myself from a situation. nor on whether I agree with the implications of certain test items. strategic and discourse competence. I will be evaluated based on the politeness and appropriateness of my responses. They will be tested on topics that have been introduced by their teacher during the course of the semester.

Each examiner will evaluate test-takers’ responses based on a scoring rubric. Competence Type Strategic Fulfills all of the Fulfills most of objectives of the objectives the task. and logical organization and response. of the task. but does not fulfill any of them. Few examples of clear. discourse and sociolinguistic competence. or addresses most of the objectives. strategic. examples of clear. and uses most of the Fulfills all or most of the following criteria: Judges the formality of a situation with major errors. General Scoring Rubric: The speaking tasks will be assessed by the examiner using the rubric provided in this section. Minimal. 4 3 Fulfills some of the objectives of the task. and uses only a few of the Spec Packet. EIL 460. page 118 . The test items will have a maximum possible score of 12 points. 2 Does not fulfill any of the objectives of the task. effective. with a minimum of 0 (if the test-taker does not attempt any answer). Fulfills all or most of the following criteria: Correctly judges the formality of a situation. and sociolinguistic). effective and logical organization of response. and uses the Somewhat clear. The test item score will consist of the summation of the subscores for the three categories in the rubric. or addresses a few objectives. and uses the 1 Discourse Sociolinguistic Fulfills all or most of the following criteria: Judges the formality of a situation with minor errors. discourse. effective and logical organization. but fulfills only a few of them.General Objectives: Test takers will use their rationale to demonstrate integrated competences (strategic. or addresses all of the objectives. Fall 2012. but does not fulfill all of them. Clear. Their responses will be assessed based on adequacy and appropriateness of the language context. Fulfills all or most of the following criteria: Incorrectly judges the formality of a situation. effective and logical organization.

B is 89.9-70 D is 69. ** Percentages will be calculated from the aggregate score of the student on all test items and score will be reported using the customary letter grade for classroom assessments. EIL 460. A is 100-90 appropriate forms.9-80 C is 79. Uses greetings and farewells inappropriately . Fall 2012. appropriate forms. May also be rude.* A score of 0 can be given if the test-taker does not attempt any answer. Demonstrates little understanding of the relationship (as defined in the test item) between testtaker and interlocutor. but it is important to remember that the focus of this test is Spec Packet.9 or below Time Allotment: 30 minutes Interviewer General Instructions: This section should be read by the examiner to test-takers before they participate in this test: “You have been studying pragmatics in this course. Uses greetings and farewells as appropriate. The subjects of the test items may make you uncomfortable. Demonstrates minimal understanding of the relationship (as defined in the test item) between testtaker and interlocutor.9-60 F is 59. Demonstrates advanced understanding of the relationship (as defined in the test item) between testtaker and interlocutor. Demonstrates adequate understanding of the relationship (as defined in the test item) between testtaker and interlocutor. inappropriate forms. appropriate forms. Exhibits major errors in greetings and farewells. Mainly uses greetings and farewells as appropriate. page 119 .

the examiner should prompt the test-taker whenever necessary. weather. etc. the test-taker could say in response to a question about a sports team.” Small talk Creating new items: writers can create new sample starting lines. (This task could come at the beginning of an oral interview with the examiner. with politeness but without overfamiliarity.g. page 120 . How was their last game?”) One-word utterances (“Yeah.” “Really?”. Fall 2012. The test-taker will respond politely and make a few generic comments of their own. The test items should be answered only in English.) 1.pragmatics. You will be tested on your ability to appropriately interact with other people in English. Prompt Attribute: The examiner and test-taker will roleplay a situation in which small talk with strangers could be likely to occur. Remember that the goal of the test is to assess how well you follow the sociolinguistic. Response Attribute: Each speaker’s responses should proceed logically from the previous utterance. “I haven’t really been following them. either assenting and asking questions or admitting lack of familiarity. and discourse norms that we have discussed in class. The examiner will begin by outlining the parameters of the situation and instructing the test-taker to engage in small talk for a short while before ending the conversation. Sample starting lines: Spec Packet. strategic. some mutual obstacle. and should avoid introducing material such as “My son has been sick” (since such material is likely to distract the reader emotionally). To reduce the item difficulty. and anything in another language will not be graded. If a test-taker is not well-versed in a topic (e. The conversation should not stray outside the boundaries of typical small talk (sports. “Uh-huh?”. 1. You will not be graded on whether you agree with the items or not. To increase the difficulty of the item. When it comes time to end the conversation. sports). EIL 460. 1.) are appropriate as long as they offer a conversational opening for the interlocutor. the examiner should be directed not to prompt the test-taker when the test-taker does not issue a satisfactory response. (For example. Then the examiner will initiate the conversation. The test-taker should not ask questions about the examiner’s appearance or personal life. the test-taker will do so gracefully. etc. they should be able to go along anyway.).

” (You will have 2 minutes to conduct this short conversation.” . EIL 460. Examiner: That’s too bad. Respond using small talk strategies.“Man. Fall 2012. Well. I don’t like it. this rain is really something. Test-taker: Yeah. Examiner: Wow.“Wow. no. I can’t wait to get out of here” (or: “…get home”). Glad I remembered my umbrella.“That house already has Valentine’s Day decorations up. sorry. starts up a conversation. page 121 . . this rain is really something.You are standing on a street corner when the person next to you.) 1. a stranger. Test-taker: Um. it really is. stay dry! Examiner: You too. Examiner: Do you know if it’s supposed to keep raining all day? Examiner: I hope it stops soon. Test-taker: Yeah. you got it.” . I have to pick up my daughter from school. I didn’t check the weather. . I have to pick up my daughter from school.“How ’bout them Steelers?” (substitute a local sports team) . Score of 9: Examiner: Do you know if it’s supposed to keep raining all day? Test-taker: No. It gets earlier and earlier every year. this rain is really something. But it looks like it’ll be like this for a while. Test-taker: I hope so too. Spec Packet. Sample Response Score of 12: Examiner: Wow.“Can you believe this? The line hasn’t moved in ten minutes.

Examiner: Do you know if it’s supposed to keep raining all day? Test-taker: Excuse me. Test-taker: Yeah. I have to pick up my daughter from school.) Spec Packet. (This task could come at the beginning of an oral interview with the examiner. Test-taker: Maybe. this rain is really something. EIL 460. it’s not bad at all. Score of 3: Test-taker: Good luck. Examiner: I hope it stops soon. you too. 2. I have to pick up my daughter from school. Then the examiner will initiate the conversation. I guess. Prompt Attribute: The examiner and test-taker will roleplay a situation in which the test-taker is responding to small talk initiated by a friend. Examiner: Wow. Test-taker: No. Fall 2012. this rain is really something. The examiner will begin by outlining the parameters of the situation and instructing the test-taker to engage in small talk for a short while before ending the conversation. Score of 6: Examiner: Wow. Examiner: Do you know if it’s supposed to keep raining all day? Test-taker: Okay. Test-taker: Probably. bye. Examiner: Okay.Examiner: Thanks. Examiner: Bye. Examiner: That’s too bad. page 122 .

(For example. in response to a question about sports. EIL 460. either assenting and asking questions or admitting lack of familiarity. How’s your family? Test-taker: They’re doing all right.“Hi! I didn’t see your brothers at the game last week. the test-taker could say. The conversation should not stray outside the boundaries of typical friendly small talk (sports. Sample Response Score of 12: Examiner: Hey. How are they doing?” (You will have 2 minutes to conduct this short conversation. If a testtaker is not well-versed in a topic (e.) 3. The test-taker will respond politely and familiarly. you know.” . using a “Good seeing you” or “See you around” type of closing statement.g.” etc. weather. keeping busy. How about yours? Spec Packet. they should be able to go along anyway.“Hey.2. you know I’m not much of a sports person.). sports). page 123 . “Well. how’s it going?” . How have you been? Test-taker: Oh. and it’s been a week and a half. Examiner: That’s too bad. How have you been?” . inquiring as to the friend’s own opinions. . Respond using small-talk strategies. Sample starting lines: Your friend has just started up a conversation with you. some mutual obstacle.“Hey. I haven’t seen you around much lately. I haven’t seen you around much lately. etc. friends or family. How was the last game?”) One-word utterances (“Uh-huh. man. Things have been crazy at work. Fall 2012.) are appropriate as long as they offer a conversational opening for the interlocutor. 2.” “Yeah. When it comes time to end the conversation. the test-taker will do so familiarly.“Can you believe this? She said she was going to give back our grades in a week. Response Attribute: Each speaker’s responses should proceed logically from the previous utterance.

Examiner: Yeah. page 124 . I haven’t seen you around much lately. Examiner: See you. Fall 2012. I’m sorry to hear that. How about yours? Test-taker: Oh. Examiner: Yeah. but all right.. Test-taker: That’s good. Give him my best. How’s school going? Test-taker: It’s busy. Examiner: That’s good. how about you? Test-taker: They’re doing okay. Score of 9: Test-taker: That’s good. Examiner: They’re okay. Examiner: Will do. sorry. How have you been? Spec Packet. EIL 460. Examiner: Hey. How have you been? Test-taker: Fine. Test-taker: Oh. How’s your family? Examiner: Hey. I haven’t seen you around much lately. he’s feeling better now..Examiner: They’re okay. Examiner: Yeah.. Score of 6: Test-taker: See you around. but my son has been sick. but my son has been sick. he’s feeling better now. Test-taker: See you around. Examiner: Fine. Examiner: See you.

Examiner: Lately my son has been sick. Spec Packet. Examiner: Yeah.Examiner: How’s school going? Test-taker: They’re fine. Fall 2012. Test-taker: Oh. Test-taker: I’ve been fine.. good. he’s feeling better now.. Test-taker: Oh? Test-taker: It’s fine. How’s your family? Examiner: Yeah. Lately my son has been sick. I’ll see you later. Examiner: Yeah. Examiner: Glad to hear it. EIL 460. Examiner: See you. Examiner: That’s good. Examiner: How’s your family? Examiner: How’s school going? Test-taker: Um.. page 125 . Test-taker: . We were pretty worried. Score of 3: Test-taker: I bet. Test-taker: How is your family? Examiner: We were pretty worried. Examiner: Hey. Test-taker: Okay. I haven’t seen you around much lately. I don’t know. he’s feeling better now. How have you been? Test-taker: Okay..

between testtaker will inform how the information should be requested: level of politeness. Test items built from this description may consist of diverse topics in order to accurately access the student’s ability to request information in many situations. page 126 . The response should avoid rudeness while being concise. Spec Packet. Examiner: Bye. Fall 2012. Ask the woman for directions to your friend’s address in an appropriate and polite way. 1. The prompt will consist and interlocutor. The test-taker will then be prompted to orally request the information needed for the situation described in the prompt.Test-taker: Bye. The relationship of a paragraph. Requesting Information 1. Prompt Attribute: This item assesses ability to request information. Sample Item (You have 3 minutes to read and think over the item and 1 minute to respond): You have gotten off at the wrong bus stop and are late to a dinner party at your friend’s home. simulating how the test-taker would request information. You have never been in the neighborhood where you are currently lost. appropriate and effective in conveying meaning. The paragraph will indicate the relationship. etc. there will be a question worded appropriately based on the interlocutor being addressed. or lack thereof. After this. Response Attribute: The item will be responded to orally. As you enter a gas station you see a middle-aged woman at the counter and approach her. proper greeting. Then there will be another sentence which will establish the need the test-taker has and how the interlocutor is involved in the requesting of the information. EIL 460. 1. The response will include a greeting.

The request will come in the form of a question. EIL 460. A topic of interest will be brought up in the prompt. could I also take the bus? Score of 6: I need your help. The knowledge of the test-taker will vary. and the variation will be stated in the prompt. Response Attribute: The item will be responded to orally. Score of 3: How do I get to 123 Fake Street? 2. 2. I would appreciate it if you could help me. The prompt will consist of an interaction between the test-taker and another person. can you please help me find 123 Fake Street? I’m not sure I can walk from here. The test-taker may respond by stating a lack of knowledge of the information in an appropriate manner if indicated in the prompt. The interlocutor will then request information from the test-taker relating to this interest.1. the prompt may tell the test-taker that she or he does not know the answer and that they must respond to the question by stating this in a manner appropriate for continuing the conversation. can you please tell me how to get there? Score of 9: Hello. Fall 2012. Sample Responses: Score of 12: Excuse me. Alternately. A full answer will consist of a response which provides the interlocutor with information they can use to address their concern. Prompt Attribute: This item will test the test-taker’s ability to give information to others. Where is 123 Fake Street? I’m a little late. page 127 . The response must logically address the interlocutor and fully answer the question. Spec Packet. the prompt may be presented as a dialogue leading up to the target interaction. If this is not stated in the prompt the test-taker will provide the information requested of him or her politely and concisely. The interlocutor in the item will vary in age and familiarity with the test-taker. The setting of the interaction will influence the formality of the prompt conversation. sorry to bother you but I was on the bus and I seem to be lost. I am trying to find 123 Fake Street.

Score of 3: Oh yeah. The prompt begins with the examiner reading a sample paragraph about assigned topics. I’m sure you could contact the bar and find out. He asks you whether the tickets are already for sale and if not when they will be on sale. Fall 2012. Prompt Attribute The test item measures the student’s ability to excuse themselves. The test item asks them to respond using a proper greeting & closing as well as proper usage of politeness and familiarity levels. the test item will assess test taker’s sociolinguistics. Sample Item (You have 3 minutes to read and think over the question and 2 minutes to respond): You are drinking coffee at your friend’s apartment. page 128 . I hope you can get tickets in time. I don’t know about anything about that. Sample Responses: Score of 12: You know I’m not sure when the tickets will go on sale. Sorry I couldn’t be more helpful. I wasn’t planning on attending the concert. overhears you mention an upcoming concert at a local bar. Spec Packet.2. The prompt involves an interpersonal relationship (schoolmate. I don’t know when they go on sale. Convey this to him and propose some other way he may find the information. whom you just met. etc). Her friend. EIL 460. 2. You do not know if the tickets are being sold or when they will begin to be sold. neighbourhood. I do know that the bar has flyers around campus and maybe you could call them and ask them about ticket sales. Score of 9: I wish I knew. Score of 6: Sorry. Excusing Expression 1. With diverse topics. strategic and discourse competences.

How will you excuse yourself from conversation politely? Sample Response 1 Score of 12: Hey. However. okay? see you! Score of 6: Hi. You usually gossip with them about everything. um. all right? Bye! Score of 9: Hi all. He steps closer to you to talk about the Spec Packet.I am in a rush and will be totally late if I don’t go right now. you don’t have enough time since you’re in a hurry to catch your morning flight or you will be late. I will be late. But.1. can we talk about it later? In the afternoon. I’d love to talk with you about last night. I also heard about it. Response Attribute The test taker will respond to the item orally. can we discuss it later. girls. Sample Item 2 (You have 2 minutes to read and think and 1 minute to respond) It is a lunch break and your professor wants to meet with you. Sample Item 1 (You have 2 minutes to read and think and 1 minute to respond) Imagine that you are living in a suburb and have a good relationship with your neighbors. Fall 2012.. you see your friend from another class.. The Test-taker also needs to incorporate the interpersonal relationship factor into the response which will differ from one response to another. you heard a police siren in the early morning. girls. in the afternoon probably. Let’s do it this evening. EIL 460. your friends come to ask you about the siren. When you go out to get in your car for work this morning. I don’t want to delay my flight. The response should be in the form of a short answer. but.umm. While you are walking to his office. Their response will be graded based on the use of a proper greeting-closing and the appropriateness of the politeness and familiarity level. page 129 . I am in a hurry and I need to go now. Last night. Score of 3: My friends. listen.

3): This conversational task will propose a situation in which a test-taker’s friend is engaging in potentially harmful behavior.tailgate this weekend. EIL 460. but I need to meet Professor Brown right now. Since meeting with the professor is really important for your grade. I wanna see Professor Brown first and then come talk to you. The test-taker will be asked to express concern to this friend. it will be established that they were close in earlier years and still maintain friendly social contact. if you wanna talk about the tailgate. I would love to. The test item will be a short written passage that establishes the social background of the two friends. How about talking about it later? Bye! Score of 9: Oh hi Sandy. excuse me first. the tailgate right. Prompt Attribute (based on the Barry item from LTA C3. I have to meet with Professor Brown. Creating new items: The type of behavior can be modified according to severity. you don’t want to show up late. wanna talk about the tailgate later? Here’s the deal. and social stigma. Finally. the potentially harmful behavior will be introduced as one in which both friends used to partake. Increasing any of these factors will also raise the item difficulty. I know what you want to talk about. I can’t talk now. Next. the type should be one that people might plausibly engage in repeatedly despite Spec Packet. rarity. What will you say to your friend? Sample Response 2 Score of 12: Hi Sandy. Then it will be established that the test-taker gave up this behavior while the other did not. However. Fall 2012. Alright? Bye! Score of 6: Sandy. First. page 130 . There will be a recent example of the testtaker’s behavior that moves the test-taker to worry about the friend’s conduct. the test-taker will be asked to raise these concerns with the friend. Expressing concern 1. Score of 3: Sandy. I need to meet and talk to Professor Brown.

concerts. and you feel less anxious. the test-taker might refer to previous experiences that they and their friend have shared. and parties. As long as the point is made. decide what you want to say. You have now managed to get your procrastination under control. Your grades have improved. On weekends you go out with a group of friends to movies.its detrimental effects. The test-taker should express concern about their friend’s potentially damaging behavior. Fall 2012. with whom you have been living since freshman year. the two of you would stay up late cramming for each test because you hadn't prepared well enough earlier. and carry out the conversation): You are midway through your junior year of college. Barry. both you and Barry have had problems with procrastinating. page 131 . The stress of urgent deadlines weighed heavily on your minds. so that test-takers would not have to build on the past as much in order to get full credit. EIL 460. Barry also seemed to improve for a while. 1. You often avoided starting assignments until the last possible date. Barry is a close friend. He spends most of his time playing video games and hardly ever Spec Packet. The examiner should supply the friend’s answers based on the test-taker’s own utterances. 1. During final exams. grading should not focus on grammatical accuracy. As such. In order to reduce the item difficulty. You have a roommate. A response should proceed from the established social relationship that was mentioned in the prompt. The response should make sense. They might imply that the two of them have had similar discussions in the past. but now he appears to be backsliding. Various types of addiction could fit here. philosophical conversations. but on whether or not the test-taker states their concern effectively. Sample Item (you have 10 minutes to read. These days. you normally finish tasks a day or two ahead of time. and might express dismay that their friend has fallen back into those harmful old habits. In the past. You enjoy talking together and frequently have long. a briefer background prompt could be developed. and then you typically produced poor-quality work due to lack of time. Response Attribute: The test-taker will read the written prompt and respond to the task in a conversation with the examiner.

You’re always playing games and never doing any homework. So you think I’m online too much? Examiner: Yeah. Spec Packet. I guess. But you’re also my friend. You might have to take calculus twice just because you weren’t studying enough. Test-taker: That’s true. he shows no sign of wanting to change. that’s the problem. I’m in the middle of a game. But it’s my life. 1. page 132 Test-taker: Well. why not? Score of 9: Test-taker: Hey. thanks. Sample Response Score of 12: Test-taker: Hey. Test-taker: Exactly. Examiner: Fine. Test-taker: Yeah. Barry. and it went really well for a while? Examiner: Yeah. and I wanted to let you know I was worried. You are worried. Now. Initiate a conversation with Barry in which you discuss his procrastination problem. . I’ll pause the game. it seems like you’re backsliding now. Test-taker: Sure. I’ll be going to the library tonight if you want to come.opens a textbook. Fall 2012. You know what I mean? Examiner: Well. Examiner: Huh? Hang on. I think we need to talk. EIL 460. Examiner: All right. Barry. at the beginning of the spring term. Remember how we both tried to stop procrastinating. I’m worried you’ll fail your classes. Last semester he just barely achieved a C average.

I guess. really? Test-taker: Yeah. Test-taker: Yeah. Examiner: Okay. Fall 2012. Examiner: Really? Examiner: All right. What’s up? Examiner: Huh? Hang on. Examiner: Yeah. I’ll wait. maybe. EIL 460. Test-taker: I think it will be hard for you to do well in school if you don’t focus more.Examiner: Yeah. Examiner: Oh. Test-taker: No. Test-taker: Um. I’m in the middle of a game. Spec Packet. But it’s my life. Examiner: Yeah. we’ll see. Barry? Examiner: Huh? Hang on. But it’s my life. Test-taker: I think you’re spending too much time on your computer. Examiner: Yeah. Test-taker: Um. I remember. Test-taker: Excuse me. page 133 . I’ll pause the game. I’m in the middle of a game. I think you’re playing too many computer games. Score of 6: Test-taker: I used to have trouble with this too. but now it’s easier for me because I spend more time studying. Test-taker: So I want you to study more. Examiner: Fine. I have to talk to you. It’s procrastination. It’s a problem. Test-taker: Okay.

Test-taker: Okay. Examiner: What? Sure I am. Examiner: Is there something you wanted to tell me? Test-taker: Um. and F. Test-taker: Okay. I don’t know. I’m in the middle of a game.scribd.fullerton. Davidson. Examiner: All right. (2007). Language Testing and Assessment: An Advanced Resource Book. Examiner: Huh? Hang on. EIL 460. Score of 3: Test-taker: Excuse me. Fall 2012. London.com/doc/95899066/Article-3 Spec Packet. page 134 . http://ed. G. UK: Routledge Press. Examiner: Fine.Examiner: So? Test-taker: I just think you’re spending too much time on it.. References: Fulcher.edu/sped/documents/SPEDApp/Oral_Interview_Rubric.pdf http://www. Test-taker: You’re not paying attention. I’ll pause the game.

With a growing number of English language learners (ELLs) coming from different parts of the world to take the SAT. the proportion of white test-takers decreased significantly from 87 percent in 1973 to 66 percent in 2001 (Lawrence et al. cultural backgrounds. which demonstrates why the test has been accused of cultural bias towards the white and the wealthy. This issue becomes even more pressing when validity and reliability are taken into account. the proportion of African American test-takers has also increased. the need to address the various issues involved with such a heterogeneous group. Over the years. However. Jian Tao. these Spec Packet. not only in respects to test design but more so in the demographics of the test takers. modifications in the SAT have not reflected these particular changes. with their wide ranging ethnicities. the SAT has undergone many changes. The steepest increases in test volume since 1973 have been among students of Asian or Hispanic/Latino descent. and native languages. dramatic changes have occurred to the makeup of the test takers resulting in more diversified racial/ethnic backgrounds that the test makers have to take into account.Amending the SAT for English Language Learners (ELLs). Background: The Changing Population of SAT Takers As illustrated in the chart below. However. 2002). Over the past 75 years. becomes increasingly apparent. Fall 2012. EIL 460. Both groups together made-up one fifth of all SAT testtakers in 2001. Version 2 Wahru (Esti) Sumarno. page 135 . surprisingly enough. and Erin Lamboi SAT Amendment for ELLs The SAT is a standardized test used in determining college admission into United States universities and colleges. This test is known for the oarsman–regatta analogy question.

Fall 2012. why it has evolved.changes in the ethnic backgrounds of test takers have had serious implications for test makers who share different first languages. this is not the case. EIL 460. page 136 . In upholding the views of the validity theory. more valid exam for the future. there should be a focus on maintaining a certain degree of flexibility. and to what extent it has evolved or been modified constitutes a great resource for future SAT test makers to use in order to create a more reliable. "Early versions of the SAT were quite speeded—as late as 1943. cultures and basic knowledge of the world. the time allotted to each particular section of the test has gradually increased over the years. how it has evolved. which will allow for modifications to be made in response to the changes in our society. Even so. In 1928 the test was reduced to seven subtests administered in 115 Spec Packet. However. students were told that they should not expect to finish. many of the earlier modifications to the test aimed at providing more liberal time limits. One of the major changes in the SAT has been that of time. A Brief Review of Major Changes in the SAT Studying the history of the SAT. which should be the goal of not only SAT test makers but all test makers alike.

and reliable test. Spec of Sentence Completion in the Current SAT 1. more recently. and in 1929. To narrow down our scope. EIL 460. this project will focus on sentence completion in the critical reading section. Then. and ensure that what the test was created to evaluate was actually what was being tested. to six subtests" (College Board)." These modifications were meant to make the test more reliable. the way you’ll need to in college" (SAT official guide. The purpose of the test is to assess “the critical thinking skills you’ll need for academic success in college…and how well you apply what you’ve learned in school to analyze and solve problems. Fall 2012. page 137 .3). there is still much more work that needs to be done. the antonym format was changed to the more familiar five-choice question (College Board). the SAT has been modified even further to five rather than six questions. pp. Despite the changes that have been made over the years.minutes. the most challenging part of the test. The writing section includes a short essay and multiple-choice questions on identifying errors and Spec Packet. unbiased. More specifically. mathematics and writing sections. It includes critical reading. the needs of ELLs who are taking the test in the medium of their second or foreign languages need to be addressed when creating the SAT in order to administer a fair. General Description: The SAT is a standardized test used by colleges and universities in the United States to assist in the admission process. The critical reading section includes reading passages and sentence completion.

EIL 460. with one to two missing words.improving grammar and usage. The mathematics section includes questions on arithmetic operations. Test takers are given an incomplete sentence. For each incorrect answer onefourth of a point is deducted.. Response Attributes Spec Packet. Critical Reading sections normally begin with 5 to 8 sentence completion questions. algebra. the remainder of the questions are focused on the reading passages. Questions on the SAT reading-sentence completion are all multiple-choice questions which have five answer choices. only one of which is correct. one raw point is added. 2. The objective is to select the best choice among the five possible answers. Students have to do close reading and grasp a clear understanding of the sentence structure. Sentence completions generally test the student's vocabulary and understanding of sentence structure and organization by requiring the student to select one or two words that best complete a given sentence. that they must complete using the choices provided. For each correct answer. page 138 . 3. statistics and probability. including sentence completions and questions about short and long reading passages. with varying types of questions. content and basic background knowledge associated with the question. Fall 2012. The questions are weighted equally. Prompt Attributes Sentence completion is administered by means of multiple-choice questions. geometry. The Critical Reading (formerly Verbal) section of the SAT is made up of three scored sections: two 25-minute sections and one 20-minute section.

and select best that completes the sentence. Being a language test.The test takers will be required to read and compare the five choices presented in the test item. Yet. As the number of ESL learners taking the SAT increases so does the responsibility of test makers to create a test that actually assesses critical thinking and problem solving skills. Cultural Issue: The prevalence of cultural bias has been a consistent problem in regards to the content of the test questions. Fall 2012. in reality. the daunting vocabulary redirects the test to be more of a language test than an assessment of critical thinking and problem solving skills. EIL 460. Considering the fact that currently the test is moving beyond the United States and targeting students all around the world it only seems appropriate that the test should Spec Packet. page 139 . Caucasian Americans. both in meaning and in form. the test is designed to evaluate students’ critical thinking and problem solving skills. render huge obstacles for ESL test takers and does not fairly nor appropriately demonstrate how well they have truly mastered the target skills. the major accusations against the test makers and the test itself have been related to cultural bias. 2). Over the past decades. towards wealthy. IV. Critique of Current SAT 1). consisting of "vocabulary-loaded" questions. As mentioned above. specifically. Language Issue: The recondite vocabulary presented in the current test items prevents the test from assessing what the SAT is supposed to test. The best example of this is the famous oarsman–regatta analogy question previously mentioned.

EIL 460. politics and history places certain test takers at an unfair advantage over others.also adjust to these changes. Furthermore. Time Issue: It is well known that we need more time to process information in our L2 than than the L1. Every year. page 140 . China can be regarded as the most extreme example of this. 4). especially when the two languages vary so greatly as does English and non-alphabet languages (Asian languages). Test Fairness: Despite the growing number of international students taking the SAT. 3). choosing computer-based over paper-based test is also Spec Packet. The SAT's almost almost exclusive emphasis on western culture. eliminating them immediately. those test takers who depended on the Holistic approach were at an advantage because they simply had to look at the question. Reforms to the test are imperative if validity is to be upheld. which was already familiar and easily recognized what choices were wrong. the test centers in the given region remain extremely limited in a way that exclude a large number of students who wish to go to an American college or university. The fact that Asian students are among the largest group of international SAT test takers. it is obvious that certain modifications need to be made in order to meet the needs of these "new" test takers and ensure that what the SAT aims to test is being adequately and appropriately met. stay in a hotel and then try to find their way to the test center in a large. On the other hand. unfamiliar city. research has shown that test-takers relying solely on the "systematic" method of test taking. Apart from the limited access to the test. thousands of Chinese high-school students have to travel all the way to Hong Kong. Fall 2012. in which each answer choice is critically analyzed by the student were at a disadvantage due to time restraints.

Spec Packet. will continue to be a standardized test used by colleges and universities in the United States to assist in the admissions process.placing many students. the revised version will allow the test takers to use an English dictionary. Amendment to the Current SAT Spec: General Description The revised version of the SAT for ESL learners. Rather. Fall 2012. in specific. The questions of the revised version will be similar to the original with a few modifications made in respects to the criticism mentioned above. which is the focus of this spec. EIL 460. questions will represent fictional countries. is through sentence completion. all of which cast doubt on the fairness of the test. and critical reading skills. The word choices will not change and the level of difficulty will remain the same. page 141 . writing. The dictionary will be provided during testing and will be available to ESL test takers as well as native English test takers. to ensure fairness to all test takers. The objectives will remain the same which are to "assess critical thinking skills" of high school students entering into college through testing math. Both the scarcity of test centers and being computer-based only. and ethnicities. V. In regards to cultural bias. excludes students from poor and middle class families from taking the SAT. However. cultures. unfamiliar with the computer." One mode of assessing critical reading skills. at a disadvantage. the revised SAT examples will exclude all cultural references. The test questions will not longer be dominated by references to Western Society.

The objective is to select the best choice among the five possible answers. that they must complete using the choices provided. Offering the test over two days instead of one. a handy dictionary will be offered to every test taker. and allowing more time overall to take the test. In this way speed does not become an issue like it had become in the past before the test was modified. The test will also be offered in both paper-based and computer-baed forms depending on each areas preference. fictional scenarios will be created when needed. offering more time and including less questions. offering more breaks during testing. The test should allot more time to ESL learners to complete the test and offer longer breaks during testing. with one to two missing words. In addition. Spec Packet.With respect to time. the test items should avoid social. the vocabulary remains the same provided in the original SAT exam. page 142 . Prompt Attributes Sentence completion is administered by means of multiple-choice questions. cultural and historical issues that might facilitate problem solving. Instead. in the revised version. In order to put test takers of different cultures. Test takers are given an incomplete sentence. however. EIL 460. The Revised SAT will also be made available in more countries and more accessible within each of these countries by offering the test outside of the cities as well as within. races and ethnicities at a fair platform. more time should be allotted to ESL SAT takers to compensate for the extra time spent manipulating a language other than their own as well as factors such as jet lag and culture shock. Fall 2012. 2.

A. ruination…impossible D. Fall 2012. Through the process of elimination the test taker will eliminate answers they deem incorrect until the correct choice is decided upon. reconciliation…assured B. through an analysis of sentence structure. page 143 . _____was_____. Sample Items Original Sample Item 1: Because King Philip’s desire to make Spain to dominant power in sixteenth-century Europe ran counter to Queen Elizabeth’s insistence on autonomy for English. conflict…inevitable E. Then reread the prompts and answer choices critically.3. EIL 460. in order to ensure complete understanding of what is being asked. warfare…avoidable C. Response Attributes The test taker will first read the prompt and review the five answer choices. diplomacy…simple Revised Sample Item 1: Spec Packet. then predict the most appropriate word to complete the sentence item. context and meaning. 4.

braggart (71) dilettante (118) Spec Packet. cultures. and ethnicities of the world. EIL 460. _____was_____.Because King Erin’s desire to make Lamboiville to dominant power in sixteenth-century Europe ran counter to Queen Tracy’s insistence on autonomy for Taoville. it would be impossible to take into account all the different countries. reconciliation…assured warfare…avoidable ruination…impossible conflict…inevitable diplomacy…simple diplomacy…simple Reason of Revision: Since the test is targeting students all around the world. it should not only involve questions related to Western history. Fall 2012. Therefore. Original Sample Item 2: There is no doubt that Larry is a genuine_____: he excels at telling stories that fascinate his listeners. However. the revised version of the SAT will use fictional examples. page 144 .

For example. Fall 2012. ESL test takers as well as native test takers.pilferer (19) prevaricator (11) raconteur (97) Reason of Revision: As mentioned above. choice (c) and (d) are extremely infrequent used. test contains numerous uncommon words making it more of a language test to native speakers. page 145 . Obviously. EIL 460. Spec Packet. not even mention EFL/ESL learners. we searched the five choices in the Corpus of Contemporary American English (COCA) (450 million words in total) and came up with the number of tokens listed on the right side of the corresponding word. Therefore we have provided a dictionary for both native speakers and ESL learners to use during testing. Then there seems little justification to test a word that students will rarely use.

Sign up to vote on this title
UsefulNot useful