In the previous chapter, you saw that a standardized test is an assessment instrument for which there are uniform procedures for administration, design, scoring, and reporting. It is also a procedure that, through repeated administrations and ongoing research, demonstrates criterion and construct validity, But a third, and perhaps the most important, element of standardized testing is the presupposition of an accepted set of standards on which to base the procedure. This feature of an edu- cational and business world caught up in a frenzy of standardized measurement is perhaps the most complex, and is the subject of this chapter. A history of standardized testing in the United States reveals that during most of the decades in the middle of the twentieth century, standardized tests enjoyed a popularity and growth that was almost unchallenged. Standardized instruments brought with them convenience, efficiency, and an air of empirical science, In schools, for example, millions of children could be led into a room, seated, armed with a lead pencil and a score sheet, and almost instantly assessed on their achieve- ment in subject-matter areas in their curricula. Standardized test advocates’ utopian dream of quickly and cheaply assessing students across the country soon became a political issue, and would-be office holders to this day promise to “reform” education with tests, tests, and more tests. ‘Toward the end of the twentieth century, such claims began to be challenged on all fronts (see Medina & Neill, 1990; Kohn, 2000), and at the vanguard of those challenges were the teachers of those millions of children. Teachers saw not only possible inequity in such tests but a disparity between the content and tasks of the tests and what they were teaching in their classes. Were those tests accurate mea- sures of achievement and success in the specified domains? Were those efficient, well-researched instruments based on carefully framed, comprehensive, validated standards of achievement? For the most part, they were not. As educators became aware of this weakness. ‘we saw the advent of a movement to establish standards on which students of all ages and subject-matter areas might be assessed. Appropriately, the last 20 years have seen a mushrooming of efforts on the part of educational leaders to base the plethora of schooladministered standardized tests on clearly specified criteriaCharTER 5 Standards-Based Assessment 105, within each content area being measured. For example, most departments of education at the state level in the United States have now specified (or are in the process of spec- ifying) the appropriate standards (that is, criteria or objectives) for each grade level (kindergarten to grade 12) and each content area (math, language, sciences, arts). The construction of such standards makes possible a concordance between standardized test specifications and the goals and objectives of educational pro- grams. And so, in the broad domain of language arts, teachers and educational administrators began the painstaking process of carefully examining existing cur- ricular goals, conducting needs assessments among students, and designing appro- priate assessments of those standards. A subfield of language arts that is of increasing importance in the United States, with its millions of non-native users of English, is English as a Second Language (ESL), also known as English for Speakers of Other Languages (ESOL), English Language Learners (ELLs), and English Language Development (ELD). (Note: The once-popular term Limited English Proficient [LEP] has now been discarded because of the negative connotation of the word limited.) ELD STANDARDS ‘The process of designing and conducting appropriate periodic reviews of ELD stan- dards involves dozens of curriculum and assessment specialists, teachers, and researchers (Fields, 2000; Kuhlman, 2001). In creating such “benchmarks for accountability” (O'Malley & Valdez Pierce, 1996), there is a tremendous responsi- bility to carry out a comprehensive study of a number of domains: + literally thousands of categories of language ranging from phonology at one end of a continuum to discourse, pragmatics, functional, and sociolinguistic elements at the other end; + specification of what ELD students’ needs are, at thirteen different grade levels, for succeeding in their academic and social development; + a consideration of what is a realistic number and scope of standards to be included within a given curriculum; + a separate set of standards (qualifications, expertise, training) for teachers to. teach ELD students successfully in their classrooms; and a thorough analysis of the means available to assess student attainment of those standards. Standards-setting is a global challenge. In many non-English-speaking coun- tries, English is now required subject starting as early as the first grade in some countries and by the seventh grade in virtually every country worldwide. In Japan and Korea, for example, “communicative” curriculum in English is required from third grade onward. Such mandates from ministries of education require the spec- ification of standards on which to base curricular objectives, the teachability of106 craerex 5 Standards-Based Assessment which has been met with only limited success in some areas (Chinen, 2000. ‘Yoshida, 2001; Sakamoto, 2002). California, with one of the largest populations of second language learners in the United States, was one of the first states to generate standards. Other states follow sin- ilar sets of standards.To view the standards developed by the California Department of Education, visit the website at http://www.cde.ca.gov/standards/ The preamble to ‘about 70 pages of ‘strategies and applications” of the California standards sets the tone The Listening and Speaking standards for English-language learners (ELLs) identify a student’s competency to understand the English language and to produce the language orally, Students must be prepared to use English effectively in social and academic settings. Listening and speaking skills provide one of the most important building blocks for the foundation of second language acquisition. These skills are essential for developing reading and writing skills in English; however, to ensure that ELLs acquire proficiency in English listening, speaking, reading, and writing, it is important that students receive reading and writing instruction in English while they are developing fluency in onal English. ‘To ensure that ELLs develop the skills and concepts needed to demonstrate proficiency on the English-Language Arts (ELA) Listening and Speaking standards, teachers must concurrently use both the ELD and the ELA standards, ELLs achieving at the Advanced ELD proficiency level should demonstrate proficiency on the ELA standards for their own and all prior grade levels. This means that all prerequisite skills needed to achieve the ELA standards must be learned by the Early Advanced ELD proficiency level, ELLs must develop both fluency in English and proficiency on the ELA standands.Teachers must ensure that ELLS receive instruction in listening and speaking that will enable them to demonstrate proficiency on the ELA Speaking Applications standards. An example of standards for listening and speaking, beginning level, is reproduc in Table 5.1. ELD ASSESSMENT ‘The development of standards obviously implies the responsibility for corre: assessing their attainment. As standards-based education became more accepted in 1990s, many school systems across the United States found that the standardized t= of past decades were not in line with newly developed standards Thus began the is active process not only of developing standards but also of creating standards4 assessments. The comprehensive process of developing such assessment in Califo still continues as curriculum and assessment specialists design, revise, and numerous tests (Morgan & Kuhlman, 2001; Stack et al., 2002; see also the hitp://www.cde.ca.gov/statetests/celdt/celdt.html), Between 1999 and 2yo sad Aunuapt Aj7e40 “sixo1 Unum aydusis noge ‘suonsanb 0} saseyjd 10 spiom ajduuis ijt puodsoy, Sujsn “sanuanu dye gearqens oviee 71-6 Sopessy ajduns pue sBuna0i8 peroos wouluoD asn Apuepuadopul -soseayd yd 10 spiom ajfuis sto) jeoneuiuress 843 Areiuouuipry pue 29} © yilanyeads 0) ulog 8-9 sper 7 Aiviopenele pune Hin ta ssasuodlsay pom-om 0} -9UO YpLAL suonsonb ajduuis somsury “(aseayl so spiom ajfiuss 2) suo} paewnuei® fy Avejuounipnu pue may e yiim >yveds 0} UlBog S-€ sepei Supjeads pue Suruaysr] aseyd ant yduuis pue sBurjoai8 pejoos wound asn Ajuapuadapuy (soamoid Suymeup ‘iamsue sasuodsos low-ongy 0} -2U0 Ulan suonsanb ayduis sawsury “gape 10 spioM aj8us suo} rend ‘uysn ’Saoua}ues 10 spuow ‘na & yun yeads of uilag ZH Sapep uorsuayosdo-) ‘suoneniunwitwo E1pIW 7B EA JO vonenjena ¥ sisiqeuy uoyeaqununuos uuoisuayoiduo sauosayeD 1a108 cHapreR 5 Standards-Based Assessnent the California English Language Development Test (CELDT) was developed. The CELDT is a battery of instruments designed to assess the attainment of ELD stan- dards across grade levels. (For reasons of test security, specifications for this test are not available to the public.) The process of administering a comprehensive, valid, and fair assessment of ELD students continues to be perfected. Stringent budgets within departments of education worldwide predispose many in decision-making positions to rely on tra- ditional standardized tests for ELD assessment, but rays of hope lic in the exploration of more student-centered approaches to learner assessment, Stack, Stack, and Fern (2002), for example, reported on a portfolio assessment system in the San Francisca Unified School District called the Language and Literacy Assessment Rubric GALAR), in which multiple forms of evidence of students’ work are collected Teachers observe students year-round and record their observations on scannable forms. The use of the LALAR system provides useful data on students’ performance at all grade levels for oral production, and for reading and writing performance in clementary and middle school grades (1-8). Further research is ongoing for high school levels (grades 9-12) CASAS AND SCANS At the higher levels of education (colleges, community colleges, adult schools language schools, and workplace settings), standards-based assessment systems have also had an enormous impact. The Comprehensive Adult Student Assessment System (CASAS), for example, is a program designed to provide broadly bases assessments of ESL curricula across the United States. The system includes mor= than 80 standardized assessment instruments used to place learners in programs diagnose learners’ needs, monitor progress, and certify mastery of functions basic skills. CASAS assessment instruments are used to measure functions reading, writing, listening, and speaking skills, and higher-order thinking skills CASAS scaled scores report learners’ language ability levels in employment ans adult life skills contexts, Further information about CASAS may be found on the: website http://www.ed.gov/pubs/EPTW/eptw14/eptw14a.html. A similar set of standards compiled by the U. $. Department of Labor, now known as the Secretary's Commission in Achieving Necessary Skills (SCANS), ou lines competencies necessary for language in the workplace. The competencs cover language functions in terms of + resources (allocating time, materials, staff, etc.), interpersonal skills, teamwork, customer service, etc., * information processing, evaluating data, organizing files, etc., systems (¢.g., understanding social and organizational systems), and technology use and application.CHAPTER 5 Standards-Based Assessmem 109 ‘These five competencies are acquired and maintained through training in the basic skills (reading, writing, listening, speaking); thinking skills such as reasoning and cre- ative problem solving; and personal qualities, such as selfesteem and sociability. For more information on SCANS, consult http://wdr.doleta.gov/SCANS/teaching/. TEACHER STANDARDS In addition to the movement to create standards for learning, an equally strong move- ment has emerged to design standards for teaching. Cloud (2001,p.3) noted that a stu- dent's “performance [on an assessment] depends on the quality of the instructional program provided, . .. which depends on the quality of professional development.” Kuhiman (2001) emphasized the importance of teacher standards in three domains: 1. linguistics and language development 2. culture and the interrelationship between language and culture 3. planning and managing instruction Professional teaching standards have also been the focus of several committees in the international association of Teachers of English to Speakers of Other Languages (TESOL). For more information, consult http://www.tesoLorg/assoc/alstandards/ index.html. How to assess whether teachers have met standards remains a complex issue.Can pedagogical expertise be assessed through a traditional standardized test? In the first of Kuhiman’s domains—linguistics and language development—knowledge can perhaps be so evaluated, but the cultural and interactive characteristics of effective teaching are less able to be appropriately assessed in such a test. TESOL's standards committee advo- cates performance-based assessment of teachers for the following reasons: + Teachers can demonstrate the standards in their teaching. + Teaching can be assessed through what teachers do with their learners in their classrooms or virtual classrooms (their performance). + This performance can be detailed in what are called “indicators*: examples of evidence that the teacher can meet a part of a standard. * The processes used to assess teachers need to draw on complex evidence of performance. In other words, indicators are more than simple “how to” statements. + Performance-based assessment of the standards is an integrated system. It is neither a checklist nor a series of discrete assessments. + Each assessment within the system has performance criteria against which the performance can be measured. + Performance criteria identify to what extent the teacher meets the standard. » Student learning is at the heart of the teacher's performance,110 courte 5 Standards-Based Assessment ‘The standards-based approach to teaching and assessment presents the profes sion with many challenges. However thorny those issues are, the social consequence of this movement cannot be ignored, especially in terms of student assessment. THE CONSEQUENCES OF STANDARDS-BASED AND STANDARDIZED TESTING A couple of decades ago I had the pleasure and challenge of serving on the TOEFI Research Committee. Among other things, it was a good opportunity to hear son of the “inside” stories about the TOEFL. One of those stories, as told by Russ Webster (personal communication), illustrates the high-stakes mature of this global marketed standardized test. A ring of enterprising “business” persons organized a group of pretend te takers to take the TOEFL in an early time zone on a given day. (In those days the te: were administered everywhere on the same day across a number of time zones. ‘TOEFL. administrations ended in some East Asian countries as much as 8 to 14 ho. before they began in the United States.) ‘The task of each test-taking “spy” was not to pass the TOEFL, but to memoriz: subset of items, including the stimulus and all of the multiple-choice options, a immediately upon leaving the exam to telephone those items to the central or nizers.As the memorized subsections were called in, a complete form of the TOI ‘was quickly reconstructed. The organizers had employed expert consultants to g erate the correct response for cach item, thereby re-creating the test items and th correct answers! For an outrageous price of many thousands of dollars, prearran; buyers of the results were given copies of the test items and correct responses ¥ a few hours to spare before entering a test administration in the Western Hemisphy The story of how this underhanded group of entrepreneurs were caught brought to justice is long tale of blockbuster spy-novel proportions involving the and, eventually, international investigators. But the story shows the huge gate-keep role of tests like the TOEFL and the high price that some were willing to pay to 1 access to « university in the United States and the visa that accompanied it. ‘The widespread global acceptance of standardized tests as valid procedure: assessing individuals in many walks of life brings with it a set of consequences fall under the category of consequential validity discussed in Chapter 2. Som those consequences are positive. Standardized tests offer high levels of practic and reliability and are often supported by impressive construct validation stu ‘They are therefore capable of accurately placing tens and hundreds of thousan testtakers onto a norm-referenced scale with high reliability ratios (most ran between 80 and 90 percent). For decades, university admissions offices arounc world have relied on the results of tests such as the Scholastic Aptitude Test (S/ the Graduate Record Exam (GRE), and the TOEFL to screen applicants. respectably moderate correlations between these tests and academic performCHAPTERS Standards-Based Assessment 111 are used to justify determining the future of students’ lives on the basis of one rel- atively inexpensive sitdown multiple-choice test. Thus has emerged the term high-stakes testing, based on the gate-keeping function that standardized tests perform. Are the institutions that produce and utilize high-stakes standardized tests jus- tified in their decisions? An impressive array of research would seem to say yes. Consider the fact that correlations between TOEFL scores and academic perfor- mance in the first year of college are impressively high (Henning & Cascallar, 1992). Are tests that lack a high level of content validity appropriate assessments of ability? ‘A good deal of research says yes to this question as well.A study of the correlation of TOEFL results with oral and written production, for example, showed that years before TOEFL's current use of an essay and oral production section, significant pos- itive correlations were obtained between all subsections of the TOEFL and inde- pendent direct measures of oral and written production (Henning & Cascallar, 1992). Test promoters commonly use such findings to support their claims for the efficacy of their tests. But several nagging, persistent issues emerge from the arguments about the con- sequences of standardized testing. Consider the following interrelated questions: 1. Should the educational and business world be satisfied with high but not per- fect probabilities of accurately assessing test-takers on standardized instru- meats? In other words, what about the small minority who are not fairly assessed? 2. Regardless of construct validation studies and correlation statistics, should fur- ther types of performance be elicited in order to get a more comprehensive picture of the test-taker? 3. Does the proliferation of standardized tests throughout a young person's life give tise to test-driven curricula, diverting the attention of students from cre- ative or personal interests and in-depth pursuits? 4. Is the standardized test industry in effect promoting a cultural, social,and political agenda that maintains existing power structures by assuring opportu- nity to an clite (wealthy) class of people? Test Bias Itis no secret that standardized tests involve a number of types of test bias.That bias ‘comes in many forms: language, culture, race, gender, and learning styles (Medina & Neill, 1990). The National Center for Fair and Open Testing, in its bimonthly newsletter Fair Test, every year offers dozens of instances of claims of test bias from teachers, parents, students, and legal consultants (sce their website: www fairtest. org). For example, reading selections in standardized tests may use a passage from a literary piece that reflects a middle-class, white, Anglo-Saxon norm. Lectures used112 chupree 5 Standards-Based Assessment for listening stimuli can easily promote a biased sociopolitical view. Consider the fol lowing prompt for an essay in “general writing ability’on the IELTS: You rent a house through an agency. The heating system has stopped working, You phoned the agency a week ago, but it has still not been mended. Write a letter to the agency. Explain the situation and tell them what you want them to do about it. While this task favorably illustrates the principle of authenticity, a number of cut tural and economic presuppositions are evident in such a prompt, calling into ques- tion its potential cultural bias. In an era when we seek to recognize the multiple intelligences present within every student (Gardner, 1983, 1999), is it not likely that standardized tests promote logical- mathematical and verbal-linguistic intelligences to the virtual exclusion of the other contextualized, integrative intelligences? Only very recently have traditionally receptive tests begun to include written and oral production in their test battery—a positive sign, But is it enough? It is also clear that many otherwise “smart” people do not perform well on standardized tests.'They may excel in cognitive styles that are not amenable to a standardized format. Perhaps they need to be assessed by such performance-based evaluation as interviews, portfolios, samples of work, demonstra- tions, and observation reports? Pethaps.as Weir (2001, p. 122) suggested, learners and teachers need to be given the freedom to choose more formative assessment rather than the summative assessment inherent in standardized tests, Expanding test batteries to include such measures would help to solve the problem of test bias (which is extremely difficult to control for in standardized items) and to account for the small but significant number of test-takers who are not acc rately assessed by standardized tests. Those who are using the tests for gate-keepinz purposes, with few if any other assessments, would do well to consider multiple mez sures before attributing infallible predictive power to standardized tests. Test-Driven Learning and Teaching Yet another consequence of standardized testing is the danger of testdrive= learning and teaching, When students and other test-takers know that one singi: measure of performance will determine their lives, they are less likely to take a pos itive attitude toward learning. The motives in such a context are almost exclusive’: extrinsic, with little likelihood of stirring intrinsic interests. Test-driven learning is = worldwide issue. In Japan, Korea, and Taiwan, to name just a few countries, students approaching their last year of secondary school focus obsessively on passing the year-end college entrance examination, a major section of which is English (Kubs 2002). Little attention is given to any topic or task that does not directly contribu to passing that one exam. In the United States, high school seniors are forced to give: almost as much attention to SAT scores. Teachers also get caught up in the wave of testdriven systems, In Florida, ce mentary school teachers were recently promised cash bonuses of $100 per studentre VRP RR Vey rr ees Ts CHAPTER 5 Standards-Based Assessment 113 a reward for their schools’ high performance on the state-mandated grade-level test, the Florida Comprehensive Achievement Exam (Fair Test, 2000). The effect of this policy was undue pressure on teachers to make sure their students excelled in the exam, possibly at the risk of ignoring other objectives in their cur- ricula, But a further, ultimately more serious effect was to punish schools in lower-socioeconomic neighborhoods. teacher in such a school might actually be a superb teacher, and that teacher's students might make excellent progress through the school year, but because of the test-driven policy, the teacher would receive no reward at all. ETHICAL ISSUES: CRITICAL LANGUAGE TESTING One of the by-products of a rapidly growing testing industry is the danger of an abuse of power. In a special report on “fallout from the testing explosion” Medina and Neill (1990, p. 36) noted: Unfortunately, too many policymakers and educators have ignored the complexities of testing issues and the obyious limitations they should place on standardized test use, Instead, they have been seduced by the promise of simplicity and objectivity. The price which has been paid by our schools and our children for their infatuation with tests is high. Shohamy (1997, p. 2) further defines the issue:“Tests represent a social technology deeply embedded in education, government, and business; as such they provide the mechanism for enforcing power and control. Tests are most powerful as they are often the single indicators for determining the future of individuals.” Test designers, and the corporate sociopolitical infrastructure that they represent, have an obligation to maintain certain standards as specified by their client educational institutions. ‘These standards bring with them certain ethical issues surrounding the gatekeeping nature of standardized tests. Shohamy (1997) and others (such as Spolsky, 1997; Hamp-Lyons, 2001) see the ethics of testing as an extension of what educators call critical pedagogy, or more precisely in this case, critical language testing (see 7BP Chapter 23, for some com- ments on critical language pedagogy in general). Proponents of a critical approach to language testing claim that large-scale standardized testing is not an unbiased process, but rather is the “agent of cultural, social, political, educational, and ideological agendas that shape the lives of individual participants, teachers, and learners” (Shohamy, 1997, p.3).The issues of critical language testing are numerous: + Psychometric traditions are challenged by interpretive, individualized proce- dures for predicting success and evaluating ability. + Test designers haye a responsibility to offer multiple modes of performance to account for varying styles and abilities among test-takers.114 CHAPTER 5 Standards-Based Assessment + Tests are deeply embedded in culture and ideology. + Testtakers are political subjects in a political context. ‘These issues are not new: More than a century ago, British educator F Y. Edgeworth (1888) challenged the potential inaccuracy of contemporary qualifying examinations for university cntrance. In recent years, the debate has heated up. In 1997, an entire issue of the journal Language Testing was devoted to questions about ethics in ae | guage testing, One of the problems highlighted by the push for critical language testing is the widespread conviction, already alluded to above, that carefully constructed standart ized tests designed by reputable test manuficturers are infallible in their predictive idity. One standardized test is deemed to be sufficient; follow-up measures are com sidered to be t00 costly. A further problem with our testoriented culture lies in the agendas of those wh design and those who utilize the tests. Tests are used in some countries to deny cis» zenship (Shohamy, 1997, p. 10)."Tests may by nature be culturebiased and therefor= may disenfranchise members of a nonmainstream value system. Test givers are alw:zy= in a position of power over testtakers and therefore can impose social and political ise ologies on test-takers through standards of acceptable and unacceptable items. Tess promote the notion that answers to real-world problems have unambiguous right as wrong answers with no shades of gray, A corollary to the latter is that tests presurm to reflect an appropriate core of common knowledge, such as the competenc= reflected in the standards discussed earlier in this chapter. Logic would therefore tate that the test-taker must buy in to such a system of beliefs in order to make the c= Language tests, some may argue, are less susceptible than general-knowledge t to such sociopolitical overtones, The research process that undergirds the TOEFL to great lengths to screen out Western cultural bias, monocultural belief systems, other potential agendas. Nevertheless, even the process of the selection of cor alone for the TOEFL involves certain standards that may not be universal, and the v fact that the TOEFL is used as an absolute standard of English proficiency by most « versities does not exonerate this particular standardized test. As a language teacher, you might be able to exercise some influence in ways lests are used and interpreted in your own milieu. If you are offeree variety of choices in standardized tests, you could choose a test that offers least degree of cultural bias. Better yet, you might encourage the use of mult measures of performance (varying item types, oral and written production other alternatives to traditional assessment) even though this might cost money. Further, you and your co-teachers might help establish an institutic system of evaluation that places less emphasis on standardized tests and emphasis on an ongoing process of formative evaluation. In so doing, you be offering educational opportunity to a few more people who would othe: be climinated from contention,CHAPTER 5 Standards-Based Assessment 115 EXERCISES [Note: @) Individual work; (G) Group or pair work; (C) Whole-class discussion.] es 1. © Evaluate the standards-based assessment movement. What are its advantages ae and disadvantages? How might one compensate for potential disadvantages? (fan 2. @ Consult the California English Language Development Test websites listed. From what you can glean from that information, how would you evaluate the oe CELDT in terms of content validity, face validity, and authenticity? dard 3. @ Consult the TESOL website on teacher standards (page 109). What would i you say are a few minimal standards that language teachers should measure com up to? In your own institutional context, or one that you are familiar with, how would you assess @ teacher's attainment of those standards? 4. (G/O) Look at the four questions posed on page 111 regarding the conse- quences of standardized testing, In groups, one question for each group, or as a whole class, respond to those questions, 5. @ Log on to the website for the National Center for Fair and Open Testing (see page 111). Report back to the class on the topics and issues sponsored by that organization. 6. (© Explain the claim that “test-takers are political subjects in a political con- text” (page 113) and Shohamy’s assertion that large-scale standardized testing is the “agent of cultural, social, political, educational, and ideological agendas.” Draw up a list of pos and Don'ts through which teachers might overcome the potential political agendas in the use of standardized tests, FOR YOUR FURTHER READING Kohn, Alfie. 2000. The case against standardized testing. Westport, CT: Heinemann. Kohn makes a strong case for the unfairness of the widespread exclusive hits use of standardized testing in elementary and secondary schools in the United States. His arguments may be appropriate for any other test-driven tred a Sacha ag educational institution in any country. bitiple Hamp Lyons, Liz. (2001). Ethics, faimess(es), and developments in language testing. fp. and In Catherine Elder (Ed.), Experimenting with uncertainty: Essays in honour of more Alan Davies (pp. 222-227). (Studies in Language Testing #11). Cambridge: tional Cambridge University Press. Ee The political and ethical issues in testing are capsulized in this brief article nee The author addresses the question of what “fairness” is and how one might discern when a test is unfair.