Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument During Spring 2010, Audrey Gutierrez and I designed and

implemented an original diagnostic test for the Peace Resource Center (PRC), a not-for-profit organization in Seaside, California. Since January 2011, graduate students in the MATESOL program at the Monterey Institute of International Studies have been offering free ESL classes to students in the local community. The students who attend the ESL program represent a range of proficiency levels, and currently, all of them are Spanish speakers. Many of the learners

have limited or interrupted formal education, and they are likely unaccustomed to meeting the expectations of typical academic environments where doing homework and taking placement tests and achievement tests is common. In a needs analysis designed for Curriculum Design during the Winter of 2011, we found that 1/3 of the learners received an education of less than 6th grade, 1/3 of the learners completed middle school, and the remaining 1/3 completed high school. With this fact in mind, we realized that many of the students may not be accustomed to formal education, especially within the United States. Furthermore, their skills may be oriented more towards speaking and listening rather than reading and writing an assumption that we wanted to assess. Also during the needs analysis, students in the program with school-aged children indicated that they wanted to be able to communicate with their childrens teachers, especially during parent-teacher conferences. Also, many learners wanted to be able to understand materials that come home with their children. As a result of these needs, we constructed a diagnostic test that is based on parent-teacher communication and attempts to measure the constructs of reading, listening, writing, and speaking skills that function in tandem in some instances in order to complete a task. For example, a parent will need to understand materials, like a report card, in order to speak about it at a conference. We

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument decided that a diagnostic test was best suited to encompass the many particular variables

that characterize the PRC student population. Further, a diagnostic test will help to guide us toward creating individualized learning experiencesexperiences which will better serve the needs of our learners, teachers, and invested stakeholders. During a unit on parentteacher communication, it may be helpful to know what students can already accomplish, and where they need more guidance. After selecting our constructs, we chose to situate them in authentic and relevant communicative scenarios involved in typical parent-teacher communication. In the realm of such communication, it is not unusual to call upon all four skills of language proficiency, sometimes independently, but more often integratively, which is the rationale behind our integrated skills test design. The reading subtest measures the construct of reading comprehension operationalized in this context as the ability to demonstrate understanding of a childs report card. Test-takers are directed to answer a series of multiple-choice questions about the content of the report card, addressing both integrative and discrete point items. According to Grabe (2009), Fluent reading comprehension requires a number of processing subskills and linguistic knowledge bases (p. 441), including vocabulary knowledge, morphology, syntax, and discourse knowledge. Most of the items on the reading subtest require the learner to call upon these skills in interpreting the information contained within the tables of the report card. The listening subtest requires both listening in the stimulus material and the task posed to the learner and reading in the learners response (Wesche, 1983). According to Buck (2001), listening is an interaction between a number of information sources, which

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument include the acoustic input, different types of linguistic knowledge, details of the context,

and general world knowledge, and listeners use whatever information they have available (p. 3). Because the listening sub-test develops the theme of performance assessment (Brown, 2005) and role-play from the preceding reading sub-test, it addresses Bucks (2001) definition with an emphasis on details of the context and general world knowledge. The construct is measured through multiple-choice questions. Boyle and Suen (1994) argue that using multiple-choice formats to measure listening is possible when it is meaningful use of the language in context. As discussed by Weigle (2002), adults, in particular minority group members or immigrant status learners, may need writing skills for survival in the workplace, advanced subject degrees or for job enhancement and other interests (p. 6). In our scenario of parent-teacher communication, a teacher may depend on parents written responses to notes the teacher sends home or perhaps parents may be requested to provide responses to surveys that a school disseminates. A particularly important aspect of writing is that the addressee is not present, thus generating content in the absence of a partner is a formidable obstacle in learning to write (Weigle, 2002, p. 3). Using Weigles (2002) term of knowledge telling, The writer uses a mental representation of the writing assignment to call up both content knowledge and a schema for the type of discourse required by the assignment (p. 32). The speaking subtest attempts to assess the learners speaking proficiency through an oral proficiency interview. Van Lier (1989) qualifies his endorsement of the OPI with the caveat that further research in speaking assessment may likely reveal an advantage to cultivating a conversational structure over an interview structure. In keeping with our

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument goals of establishing a low-pressure, informal assessment tool that meets the needs of our target population, we attempted to cultivate a conversational tone in our oral proficiency interview. We aimed to achieve a conversational structure by embedding the speech construct in a role-play. Supported by Brown (2005), role-plays are tests used in the communicative movement. The role-play in our original test simulates an authentic communicative task, which learners might typically encounter: the parent-teacher conference. After establishing descriptive analyses for the first portion of the project, we then conducted statistical analyses of the pilot test results. Specifically, we analyzed the relationship between subtests in terms of Pearsons r and overlapping variance, and concluded the work by discussing our experiences, practicality, and washback. While our program currently does not receive funding from the state, it could expand, and this diagnostic test may be one way to document assessment, either through portfolio assessment or through reports from teachers.

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument References Boyle, J. & Suen, D. (1994). Communicative considerations in a large-scale listening test. In J. Boyle and P. Falvey (Eds.), English language testing in Hong Kong (pp. 32-55). Hong Kong: The Chinese University Press. Brown, J.D. (2005). Testing in language programs: A comprehensive guide to English language assessment. New York: McGraw-Hill. Buck, G. (2001). Assessing listening. Cambridge: Cambridge University Press. Grabe, W. (2009). Teaching and testing reading. In M.H. Long & C.J. Doughty (Eds.), The handbook of language teaching (pp. 441-462). West Sussex, UK: Wiley Blackwell. van Lier, L. (1989). Reeling, writhing, drawling, stretching, and fainting in coils: Oral proficiency interviews as conversation. TESOL Quarterly, 23, 489-508. Weigle, S. C. (2002). Assessing writing. Cambridge: Cambridge University Press. Wesche, M. (1983). Communicative testing in a second language. The Modern Language Journal, 67, 41-55.

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument

Name: ____________________________ TEST # ______________ PART 1: READING ACTIVITY

Instrucciones - Usted tiene 15 minutos para completar la actividad de lectura. Lea el prompt. Utilice la boleta de calificaciones adjunta para contestar las siguientes preguntas.

Directions You have 15 minutes to complete the Reading Activity. Read the prompt. Use the enclosed report card to answer the following questions.

Prompt: Pretend that you have a child named Sophie. You have just received Sophies first kindergarten report card. Pregunta: Finga que usted tiene una hija que se llama Sophie. Usted acaba de recibir la primera boleta de calificaciones de ella. Ella es en el jardn infantil. 1. In which areas did Sophie perform very well? a. Physical education and respect for authority b. Effort in academic work and good use of time c. Attentive listening and following directions d. Organizational skills and respect for others 2. In which area did Sophie perform poorly? a. Plays well with others b. Shows good self-control c. Makes good use of time d. Shows good study habits 3. In the second trimester, the teacher says Sophie is continuing to improve what skills? a. Math b. Reading c. Science d. Spelling

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument

4. In the second trimester, what do the teachers frequently remind Sophie to do? a. Socialize with her peers b. Turn in her homework c. Hold her pencil properly d. Pay attention during class 5. Which program does the student participate in? a. Special Day Class b. Resource Specialist c. English Language Development d. Bilingual Program 6. How many days was Sophie absent in the first trimester? a. 0 b. 1 c. 2 d. 3 7. How many days was Sophie late in the second trimester? a. 0 b. 1 c. 2 d. 3 8. How many days total has Sophie been absent this academic year? a. 3 b. 4 c. 5 d. 6 9. In which subject did Sophie improve from first to second trimester? a. Physical Education b. Health c. Works neatly d. Shows self-control

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument

10. How many more trimesters does Sophie have to complete? a. 0 b. 1 c. 2 d. 3

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument

+ Outstanding Performance Performance as Expected - Performance Less Than Expected

9
Teacher Comments First Trimester Sophie is still learning when to visit with peers and when to focus on her work. She is working on learning how to hold a pencil correctly for writing her letters and numbers minding proper form. She enthusiastically participates chorally in phonemic awareness activities, which will augment her skills and help her to develop her reading skills.

1st Other Subjects Physical Education Visual and Performing Arts Health Organizational Skills/Work Habits Listens attentively and follows directions Makes good use of time Returns completed homework on time Works neatly Demonstrates organizational skills Participates in class discussions and activities Citizenship/Conduct Accepts responsibility for own behavior Cooperates, works, and plays well with others Shows self-control Respects the rights, feelings, and property of others Respects authority

2n
d

3rd

Second Trimester Sophie continues to struggle with knowing when it is appropriate to socialize with peers and when to listen to teacher. She also needs frequent reminders to hold the pencil correctly. She continues to improve her reading skills.

+ Third Trimester Promoted to next grade.

Programs in Which Student is Participating

Special Day Class English Language Resource Development Specialist Bilingual Program Speech Therapy Dual Language Programs Supplemental Intervention: Reading Math
Attendance Days Absent Days Late Days Tardy Absences/Tardies Affect Student Progress (Y/N) N N 1st 3 1 2nd 2 2 3rd

Teacher Signature Principal Signature

Date Date

Current Progress Permits Promotion Yes No

1st X

2nd X

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument

10

Name: ____________________________ TEST # ______________ PART 2: LISTENING ACTIVITY Instrucciones - Tiene 20 minutos para completar la actividad de comprensin oral. 1. Por favor, tome 2 a 3 minutos para leer el prompt y las preguntas. 2. Escuche el mensaje de telfono dos veces. Por favor, no tome notas. 3. Luego usted tendr 10 minutos para contestar las preguntas.

Directions You have 20 minutes to complete the Listening Activity. 1. Please take 2-3 minutes to read the prompt and questions. 2. Listen to the phone message two times. Please do not take notes. 3. Then you will have 10 minutes to answer the questions.

Prompt: Pretend that your name is Mrs. Romero and that you have a child named Sophie. Sophies teacher called you today, however you were not at home. Pregunta: Finga que se llama usted Mrs. Romero y que tiene una hija que se llama Sophie. Hoy la maestra de Sophe le llam, pero no estaba en su casa. [Test taker will listen to an audio recording; the transcript is provided here] Hi there, Mrs. Romero, this is Mrs. Cadman, Sophies teacher, calling from the Seaside Elementary School. Its about 3:30pm on Monday. I know we havent met yet, but I just wanted to call you and say hi. I like to call home sometimes just to see how everything is going and to touch base on your childs progress in school. Theres nothing to worry about, everything is fine! Also, we are having a family potluck after school in my classroom on Friday and I would like to invite you and your family. It starts at 4:00pm. I hope you can come! Please bring a beverage, snack, side dish, main dish, or dessert. Sophie tells me you make great enchiladas. Maybe you can bring those! Please give me a call back when you can to chat and RSVP. The schools number is (831) 512-5555 and my extension number is 17. Have a wonderful day Mrs. Romero. 1. What is the teachers name? a. Mrs. Cadman b. Mrs. Lopez c. Mrs. Jacobs d. Mrs. Roberts

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument

11

2. When is the potluck? a. Thursday at 2:00pm b. Thursday at 4:00pm c. Friday at 2:00pm d. Friday at 4:00pm 3. What does the teacher suggest that you bring? a. Hamburgers b. Enchiladas c. Sandwiches d. Quesadillas 4. Where will the potluck be held? a. Cafeteria b. Playground c. Classroom d. Front office 5. Who is invited? a. Students only b. Students parents c. Students siblings d. All of the above 6. What is a potluck? a. A type of birthday party for children. b. A type of party for fundraising for adults. c. A type of festival where people perform and sing. d. A type of party where everyone brings food to share. 7. What is an extension number? a. a number to leave a message for someone. b. a number to reach a specific person. c. a number to call a personal cell phone. d. a number for an elementary school.

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument 8. What does RSVP mean? a. Respond to an invitation. b. Ask for directions to a party. c. Send suggestions to the host. d. Invite your friends to a party. 9. Which of the following is appropriate for a school event? a. Beer b. Soda c. Wine d. Coffee 10. Which of the following is a main dish? a. Orange juice b. Ice Cream c. Beef tacos d. French fries

12

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument

13

PART 3: WRITING ACTIVITY

Instrucciones - Por favor, lea la carta que la Sra. Cadman le ha enviado a casa porque an no le ha respondido a su mensaje de telfono.

Directions Please read the letter that Mrs. Cadman has sent home to you because you have not responded to her phone message yet.

Dear Mrs. Romero, How are you? I bet that you are very busy! I hope that you got my message I left you on Monday. I called to say hi and to see how everything is going. I also wanted to invite you to our family potluck this Friday at 4:00pm. I have included an invitation with this note. Please write me back and let me know if you will be coming, with whom, and what you will bring. Sincerely, Mrs. Cadman

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument

14

Ahora escrbale su respuesta a la Sra. Cadman. Now write your response to Mrs. Cadman. _________________________________________________________________________________________________________ _________________________________________________________________________________________________________ _________________________________________________________________________________________________________ _________________________________________________________________________________________________________ _________________________________________________________________________________________________________ _________________________________________________________________________________________________________ _________________________________________________________________________________________________________ _________________________________________________________________________________________________________

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument PART 4: SPEAKING ACTIVITY

15

Directions You have 5 minutes to complete the Speaking Activity. 1. Please read the prompt. 2. Speak with the teacher about the report card.

Instrucciones - Tiene 5 minutos para completar la actividad de hablar. 1. Por favor, lea el scenario. 2. Hable con el maestro sobre la boleta de calificaciones.

Prompt: Pretend that you have a child named Sophie. You have scheduled a meeting with Sophies teacher to talk about Sophies first kindergarten report card. The teacher will ask you specific questions about the report card during this portion of the test. Please refer to the report card to answer the questions. Pregunta: Finga que tiene una hija que se llama Sophie. Ya ha hecho una cita con la maestra de Sophie para hablar de la primera boleta de calificaciones de Sophie. La maestra le va a preguntar cosas especficas sobre la boleta durante este seccin. Por favor, refiera a la boleta para contestar las preguntas.

FOR TEST ADMINISTRATOR During this portion of the assessment, you will be the teacher and the test taker is to be Sophies Mother. Please direct the learner to use the report card to answer the questions during the test. For the purposes of completing this section of the test, please use the following questions within 5 minutes. Questions: 1. What worries you? 2. What things can we do to help Sophie talk less during class? 3. Why do you think attendance is important?

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument

16

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument Scoring Objective Portion: Name: ___________________________ Test # _______ Part 1 - Reading. 10 Points 1. A: _____ ( 2. C: _____ ( 3. B: _____ ( 4. C: _____ ( 5. D: _____ ( 6. D: _____ ( 7. C: _____ ( 8. C: _____ ( 9. A: _____ ( 10. B: _____ ( Reading Score: Part 2 - Listening. 10 Points 1. A: _____ ( 2. D: _____ ( 3. B: _____ ( 4. C: _____ ( 5. D: _____ ( 6. D: _____ ( 7. B: _____ ( 8. A: _____ ( 9. B: _____ ( 10. C: _____ ( Listening Score:

17

) ) ) ) ) ) ) ) ) )

) ) ) ) ) ) ) ) ) )

Part 3 - Writing. 10 points Score 1: ______ Score 2: ______ Writing Score (Average x 2): Part 4 - Speaking. 10 Points Score 1: _______ Score 2: ______ Speaking Score (Average x 2):

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument Writing Scores

18

Adapted from BEST Literacy Test Manual (2008) and the Test of Written English (TWE) Task Box Please read the letter that Mrs. Cadman has sent home to you because you have not responded to her phone message yet. To fulfill the task, the writing must do all of these things: - Identify the writer and the addressee (Mrs. Cadman / Mrs./Mr. Romero) - Recognize/Apologize for not calling back yet. - Specify whether or not the parent will be attending, with whom, and what they will bring. - Express thanks and appropriately conclude the message.
5 Demonstrates competence in writing on both the rhetorical and syntactic levels, though it will have occasional errors Well organized and well developed (addresses the teacher, concludes with a salutation) Effectively addresses the writing task (please see goals listed in task box) Uses appropriate details to support letter response Demonstrates syntactic variety and appropriate word choice with few syntactic errors Is adequately organized (may omit one formal salutation) Adequately addresses the writing topic but may miss some parts of the task (see the tasks described in the task box) Uses some details to support the letter response Demonstrates adequate but undistinguished or inconsistent facility with syntax and usage Inadequate organization or development (omits formal salutation, ideas not completely developed) Parts of the task have been addressed (see the tasks described in the task box) Few details added to support the letter response Noticeably inappropriate choice of words or word forms as well as flawed syntactic structure

4 Demonstrates minimal competence in writing on both the rhetorical and syntactic levels

3 Demonstrates some developing competence in writing, but it remains flawed on either the rhetorical or syntactic level, or both

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument

2 Suggest incompetence in writing

19
Failure to organize or develop Limited parts of the task have been addressed (please see the taxk box to see the tasks) Little or no detail, or irrelevant specifics Serious and frequent errors in usage or sentence structure as well as focus

1 Demonstrates incompetence in writing

A paper in this category will contain serious and persistent writing errors, may be illogical or incoherent, or may reveal the writers inability to comprehend the question. The written response is severely underdeveloped.

5-Score Writing Sample:

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument 4-Score Writing Sample

20

3-Score Writing Sample

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument 2-Score Writing Sample

21

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument Integrated Speaking Scores Adapted from TOEFL iBT 5-point scale (R1 + R2 scores for a total of 10 points)

22

The answers to the questions are supposed to highlight the most salient features of the report card: The teachers comment: Sophie continues to struggle with knowing when it is appropriate to socialize with peers and when to listen to teacher. She also needs frequent reminders to hold the pencil correctly. She continues to improve her reading skills. Attendance: Sophie does not have perfect attendance and also has some tardies, so it would be important for the parent to note this and respond to these flaws accordingly. Sophie has shown improvement in physical education and respecting authority

Score 5

General Description The response fulfills the demands of the task, with at most minor lapses in completeness. It is highly intelligible and exhibits sustained, coherent discourse about the report card by addressing the three salient features. A response at this level is characterized by all of the following:

Delivery Speech is generally clear, fluid, and sustained. It may include minor lapses or minor difficulties with pronunciation or intonation. Pace may vary at times as the speaker attempts to recall information. Overall intelligibility remains high.

The response addresses the task appropriately, but may fall short of being fully developed. For example, the learner may not recognize that Sophie has improved in two areas. It is generally intelligible

Speech is generally clear, with some fluidity of expression, but it exhibits minor difficulties with pronunciation,

Language Use The response demonstrates good control of basic and complex grammatical structures that allow for coherent, efficient (automatic) expression of relevant ideas. Contains generally effective word choice, in reference to the report card. Though some minor (or systematic) errors or imprecise use may be noticeable, they do not require listener effort (or obscure meaning). The response demonstrates fairly automatic and effective use of grammar and vocabulary, and fairly coherent

Topic Development The response presents a clear progression of ideas and conveys the relevant information required by the task. It includes appropriate detail, by addressing the three salient points, though it may have minor errors or minor omissions.

The response is sustained and conveys relevant information required by the task. However, it exhibits some

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument

and coherent, with some fluidity of expression, though it exhibits some noticeable lapses in the expression of ideas. A response at this level is characterized by at least two of the following: intonation, or pacing and may require some listener effort at times. Overall intelligibility remains good, however. expression of relevant ideas. The response may exhibit some imprecise or inaccurate use of vocabulary or grammatical structures or be somewhat limited in the range of structures used. Such limitations do not seriously interfere with the communication of the message. The response is limited in the range and control of vocabulary and grammar demonstrated (some complex structures may be used, but typically contain errors). This results in limited or vague expression of relevant ideas and imprecise or inaccurate connections. Automaticity of expression may only be evident at the phrasal level.

23
incompleteness, inaccuracy, lack of specificity with respect to content, or choppiness in the progression of ideas. Overall, the learner may identify the salient features of the report card, but does not fully explain their concerns for the teachers comments or that Sophie has improved in some areas. The response conveys some relevant information but is clearly incomplete or inaccurate. It is incomplete if response omits key ideas, like understanding the teachers message or the importance of attendance, the learner still makes vague reference to key ideas, or demonstrates limited development of important information. An inaccurate response demonstrates misunderstanding of key ideas from the stimulus. Typically, ideas expressed may not be well connected or cohesive so that familiarity with the stimulus is necessary to follow what is being discussed. The response provides minimal relevant content to

The response is connected to the task, though it may be missing some relevant information or contains inaccuracies. Perhaps the learner does not completely comprehend the message from the teacher or does not completely discuss the importance of attendance from their perspective. It contains some intelligible speech, but at times problems with intelligibility and/or overall coherence may obscure meaning. A response at this level is characterized by at least two of the following:

Speech is clear at times, though it exhibits problems with pronunciation, intonation, or pacing and so may require significant listen effort. Speech may not be sustained at a consistent level throughout. Problems with intelligibility may obscure meaning in places (but not throughout).

The response is limited in content or coherence or is only minimally connected to the

Consistent pronunciation and intonation

Range and control of grammar and vocabulary limit

Heidi Marie Laidemitt MATESOL Component 8 Assessment Instrument

task. A response at this level is characterized by at least two of the following: problems cause considerable listener effort and frequently obscure meaning. Delivery is choppy, fragmented, or telegraphic. Speech contains frequent pauses and hesitations. Problems with intelligibility obscure meaning throughout. expression of ideas and connections among ideas.

24
the teachers comment, attendance issues, or Sophies improvement. Ideas that are expressed are often inaccurate or incomplete.

The response is unrelated to topic or speech may be largely unintelligible.

Range and control of grammar and vocabulary severely limit (or prevent) expression of ideas and connections among ideas. May rely on isolated words or short utterances to communicate ideas.

The response fails to provide relevant content. Response is limited to vague utterances, or repetitions (including repetition of prompt).