Feedback Follow ofTeacher Comment

Up:

the

Influence

on Student WritingAssignments
by Deborah W. Dunsford Introduction Writing improvement is becoming an increasingly important topic at most universities. Feedback from potential employers and research repeatedly shows that college students' writing abilities are below expectations (Lindner et al., 2004). Several universities have implemented writing-intensive course requirements for undergraduate students that will ultimately require faculty in all disciplines toprovide additional writing opportunities in their curriculums (Univ. of Florida, 2004. The Gordon Rule; Texas A&M Univ., 2004. Writing Intensive Courses at Texas A&M Univ.; Martin and Burnett, 2003). For agricultural education and communication programs, this focus frequently takes the form of service courses that teach writing skills (Kansas State Univ., 2004; The Ohio State Univ., 2004; Texas A&M Univ., 2004. Core Curriculum; Univ. of Florida, 2004. Agricultural Education). As the demand for seats in courses that teach writing skills continues to grow, instructors try to balance the need to provide students with feedback on their writing assignments with the amount of time it takes to provide that feedback. While writing instructors from all disciplines generally agree that revision is one of the best ways to encourage students to improve their papers, few know what comments or what type of comments are most likely to help their students revise successfully. Research into revision and how and why students revise their texts has long been part of composition literature. So has research into teacher comment

on student texts. However, there is little work that brings the research areas together. This study may provide a link between these two important areas of research. Composing a piece of written discourse has long been considered a non-linear, recursive process (Britton, 1975; Rohman and Wlecke, 1964). Later researchers built on this model describing composing as a continuous loop where any element may follow any other element (Bridwell, 1980; Faigley and Witte, 1981; Flower et al., 1986; Sommers, 1980). Although the recursive nature of the process is not in question, an actual definition for revision is less clear. Several definitions use only the etymological definition of "seeing again" (Boiarsky, 1980). Sommers (1980, p. 380) defines revision as "... a sequence of changes in a composition changes which are initiated by cues and occur continually throughout the writing of a work." Drawing on all these definitions, the operational definition for revision used in this study refers to the additions, deletions, substitutions, and rearrangements of units of meaning that students make in their texts in an effort to convey better their intended meaning to an audience. Teacher comment is another key area of composition research. Much research shows that teacher response can have a major impact on a student's attitude toward the text and toward writing in general. De Beaugrande (1979) claimed that if students see grammar, punctuation and spelling as priorities in teacher comment, then those are the errors they will repair. Miller (1982) suggested two separate sets of teacher comments one on content and the other on writing problems. Murray (1979) advocated doing away with comment completely and using one-on-one conferences toprovide feedback to students. Peterson et al. (2004) suggest that the type of paper plays a role in the type of comments teachers provide. Narrative papers receive a greater percentage of editing-related comments and persuasive papers tend to receive a greater percentage of revision-related comments (Peterson et al., 2004).

Besides types of comments, other research examines the quality of those comments. Lynch and Klemans (1978) and surveyed found that students students about responded their more responses to teacher comment

positively to comments that not only told them what was wrong with a paper, but why. Straub (1996) explored directive versus facultative comments on student texts and the potential control the comments represented. In general composition researchers agree that the goal of teacher comment on papers is to wean students away from criticism from the teacher and toward forming their own ability to review and revise their texts. Purpose The purpose of this study was to determine how the types and location of teacher feedback on a group of student texts influenced the revision choices that group of students made to their texts. The research objectives were to determine if the location of teacher feedback influenced students' revision choices and if the type of content or type of teacher comment influenced those choices. Methods The subjects in this study were 62 students enrolled in media writing classes at a land grant institution in the South. Each of the four classes studied had enrollment limits of 16 students and the students were primarily juniors and seniors majoring in journalism or agricultural journalism. A few students were either minoring in journalism or planned to obtain teaching certification in journalism. The majority of the students were female (69%), which reflects a nationwide trend in communication departments. The classes met four times per week during the 16week term. The Monday and Wednesday lecture sessions covered techniques in news writing and textbook material on different types of news stories, as well as some basic information on methods of writing. The other two weekly class meetings were 75-minute lab sessions. The students used Microsoft Word software for all of

their assignments. Besides the lab periods, students had access to the computerequipped classroom through the work day and had access to their own computers or other campus computer labs throughout the semester. Data was collected from the students' four major writing assignments. The four assignments were as follows: (1) Write a pair of short, one-paragraph leads from a choice of assigned fact sets; (2) write a news story from a short speech and questionand-answer session presented by a guest speaker; (3) write a news story about a coming event on campus or other item of the student's choice that quotes at least one source (i.e., they had to go interview someone and write the story); and (4) write a short feature story on a topic of their choice that quoted at least two sources and required additional background sources. Students wrote for both print and broadcast media. Theteacher comments and the revisions students made on these assignments provided the data for this study. Students had the option to revise one of the two versions of each of the major assignments. If the students opted to revise one of their papers, the grade they received on the original paper counted as two-thirds of the grade on the final paper. The grade on the revised paper counted as the remaining one third. This method encouraged students to make their best effort on the original paper. The students' grades on the revised papers would not be lower than the original grade they received, although the grade could remain unchanged. For purposes of this study, only the papers that the students opted to revise were analyzed. Each of the four classes received four different methods of instructor feedback with a different method used on each of their four major assignments. The comment methods were marginal and concluding written comments on their papers, marginal comments only, concluding comments only, and only oral comments to the class as a group. When revising their papers, the students were required to return the graded original paper along with the revised version.

To protect the students' identities and to eliminate any chance of bias associated with any particularstudent, each student was assigned a random number, and an uninterested third-party placed this number on the students' texts and then cut off the students' names and course section numbers. Topreserve the regular classroom environment during the study, the students were not told about the study until the end of the semester, after the last paper had been turned in. The students received a written explanation of the study and the use of their texts (anonymously). At this time they were offered the chance to have their papers removed from the study. None of the students selected this option. This study met all university requirements for human studies research and all necessary forms are on file with the university's research office. After the student texts had been collected they were sorted by assignment and teacher comment type (marginal and end, marginal only, end only and oral only comment). The texts were sorted numerically for ease in coding and an index card was established for each student number. These cards provided a method of tallying the number and types of revisions on each text. The data from these cards provided the basis for the statistical analysis in this study. Structural revisions made by the students in a second, revised paper were compared to their original, graded papers. Structural revisions in this study were additions, deletions, substitutions, and rearrangements (Sommers, 1980). These structural revisions were examined at the level of units of meaning that may or may not correspond to the physical division of paragraphs within the text. According to Rodgers (1967), paragraph divisions frequently do not correspond with units of meaning within a text, and he suggests that a "stadia of discourse" is a better unit than the somewhat arbitrary paragraph indention. The "stadia," according to Rodgers, is a sentence or group of sentences that contain a single topic which may or may not be contained in a single paragraph. This idea is particularly important when working with journalistic writing. Paragraphs in a newspaper or on an audio script are frequently shorter to accommodate the requirements of the

newspaper's narrow columns or the readability for a television or radio reporter. The variable of interest for this study was units of meaning, sentences or groups of sentences that share a common topic. An ANOVA was performed on the revision data that, in effect, combined the four classes into a single group for statistical purposes (Ott, 1988). This method is appropriate because the students were not assigned randomly to the classes used in the study. The analysis examined the four treatments (marginal and end comment, marginal comment only, end commend only and oral comment only) and the four revision types (additions, deletions, substitutions and rearrangements) to determine if there were any significant differences between the treatments and the outcomes. In the analysis, differences with p <.10 are considered significant. This significance level was used to help offset Type II error that could easily result from the relatively low number of subjects, the imprecise measurement methods, and the exploratory nature of this research (Lauer and Asher, 1988). Next, using percentages and graphs, the data were analyzed for similarities and differences among the combination of treatments and the resulting revisions. A naturalistic inquiry method was used to examine the relationship between specific instructor comments and the specific revisions that resulted from that comment (Lincoln and Guba, 1984). To analyze the data, teacher comment that was written on the student texts or given in the oral comments were recorded and written on individual index cards. The cards were sorted into groups of those with similar meanings using Lincoln and Cuba's ( 1984) method. The groups were then cross checked and again collated into groups by meaning or the problem they addressed. Seven groups were established, each of which addressed a different aspect in the texts (Table 1). At this stage in the study there was no differentiation made between oral and written comments, as those distinctions were covered in the quantitative phase of the study. Results and Discussion Analysis of the teacher comment types resulted in seven comment types: positive

comments, overall quality of all or a section of text, material that does not belong in the text, material that is out of place, wordiness or over length, wording or sentence needs work, and meaning is unclear. Studentresponses were examined based on each comment type. "Positive comments," as might be expected, did not result in a lot of revisions, although some students did revise some of these sections of their texts. All of the revisions resulting from these comments were improvements. Comments on the "overall quality of all or a section of the text" asked for big changes. Generally student response to these comments was deleting or substituting material in their texts, which is not surprising because the comments frequently related to coherence or focus. Again, the student revisions associated with this comment were generally improvements, but in some cases deleting material weakened the story. Student responses to the comment that to "material that did not belong in the text" also resulted in deletions. Students tended to act more frequently on some of these comments than others because a few of the comments offered more specific instructions ("Doesn't go in the story" vs. "How does this fit?"). Therefore, some of the comments were not acted on by the students, probably because of uncertainty of how to solve the problem (Flower et. al., 1986). Generally revisions made in responseto this comment group improved the student texts. "Material that is out of place" comments relatedto organization problems in student texts and usually suggested that the material belonged in the story, just not where the student had placed the information. As expected, students generally optedto rearrange their texts in response to this comment. Specific comments in this category resulted in more revisions that improved the texts than did less instructive comments ("Move Up" vs. "Out of place"). Students responded frequently to the less specific comments by removing the material. Comments on "wordiness" and "wording or sentence needs work" problems

frequently resulted in students deleting material from their texts. However, many students did a good job of combining sentences and paragraphs to tighten up the text and reduce the paper's overall length. Getting just the right word can be a particular problem for student writers and comments in this area included "awkward," "choppy," or "vary sentence structure." Responses to these comments frequently resulted in fine tuning rather than fixing structural or coherence problems. Many revisions in response to these comments included combining sentences and altering sentence structure. A few resulted in deletions, but there were more substitutions used in response to this comment than toother comments. Again, the more specific the instructions, the more often the students revised successfully. "Unclear meaning" comments usually refer to the need for more information including specific detail or other clarifications. Some of the comments went so far as to ask for specific numbers or other specific information, others were "vague" and "confusing." Revisions resulting from this comment group varied including deletions and additions. The quality of the students' revisions also varied. Results from the qualitative portion of this study indicate that the more directive the teachercomment on student texts, the more successful student revisions will be on the text. Students tended to respond to teacher comment if they knew how to make the requested change or improvement. If they did not know how to make the change or how to improve the text, they frequently deleted the material or ignored the comment. According to Spandel and Stiggins (1990), students frequently misread instructors comment and fail when they are trying to revise their texts. Occasionally students would substitute material, which ultimately resulted in a few additions. There were few rearrangements and those changes were usually in response to a specific comment toalter the order of ideas in paragraphs. In response to one of the main questions of this study, teacher comment "Does influence the choices students make in revising their texts," and a second question "Does the lack of teachercomment influence student revision?" Indications from the

qualitative portion of this study are that students are even more likely to make revisions in the absence of written comment when oral only comment is presented. As with the other student responses to teacher comment, students perceive a benefit from revising their texts based on the incentive of an improved grade. The F test included all 64 of the students in the study combined into one large group. This option was chosen to maintain the natural classroom environment as much as possible. The data were coded by treatment and by revision outcomes. Based on this analysis, the only significant outcome at p <.10 was deletions. A Scheffe S test showed that marginal comment and oral comment only treatments were similar for deletions, as were marginal and end comment and end comment only treatments. However, marginal comment and oral comment only treatments were significantly different than marginal and end comments and end comment only treatments. This means that the students' responses to each pair of treatments were similar, but that they responded differently to the treatments not contained in each pair. The significance of deletions and the relationship between the two pairs of treatments provides several options for interpretation. Flower et al. (1986) suggest that if students do not know how toaddress a problem, they will frequently delete the material. That is likely the case in this study. second, the similarities between responses to marginal and end comment and end comment only suggest that students may be reading and interpreting these comments in much the same way. The same should be said for the other pair of treatments, marginal only comment and oral comment. Examining the means of these two treatments, .38 for marginal only and .51 for oral only, indicates that students made fewer deletions on average in response to these two comments than for the other comment pair. Deletion means for marginal and end comment and end comment only were .74 and .81 respectively. The students made more deletions based on these two treatment types. One obvious similarity between thee two treatments is that they both include comments on the students' texts. This may indicate that students either read these

comments more often or that they somehow responded to these comments differently than they did to marginal only comment or oral only comment at least when it came to making deletions in their texts. Although the F test showed a significant difference for only deletions, the descriptive statistics associated with the various treatment totals are worth discussion. Of the 307 total revisions by type made by students in this study, 109, or 35.62%, were deletions; 85, or 27.28% were substitutions; 59, or 19.28%, were additions and 55, or 17.32%, were rearrangements. Total revisions broken down by teacher comment location (Table 2 and Figure 1) were marginal and end comment, 83; end only comment, 80; oral only, 83 and marginal only, 61. The primary difference is in the revision types, with deletions showing a much higher incidence than substitutions; additions and rearrangements are fairly even at the lower end of the range. The high number of deletions is not unexpected (Flower, et al., 1986) Comparing comment location by revision type also provides an interesting discussion. For marginal only comment, there was a relatively low overall number of revisions and an even distribution (16 additions, 16 deletions, 15 substitutions, and 14 rearrangements a range of 2). One possibility for this relatively low number of revisions with this comment location relates to the lack of space available for providing feedback. Another reason may be that students do not read these comments. Bolker ( 1978) suggested that students dissociate themselves from teacher comment because they fear disappointment. Also, while end comment often points out problems in a text, it is frequently tempered with positive comment and is sometimes less directed at a specific point or error in the text (Smith, 2004). Oral only comment elicited a relatively large number of revisions ( 18, 24, 24, and 17 a range of 7). This number of revisions is somewhat higher than anticipated at the outset of the study. One of the theories was that students receiving only oral comment on their texts would revise less because of the somewhat fleeting nature of

the feedback. However, information on the audio tapes of the oral comment sessions suggests one reason for the unexpected strength of this response. The written notes for these sessions look, for the most part, like a laundry list of what was wrong (and occasionally right) with the class' texts. However, the audio tapes of these sessions include not only the comments on the problems, but usually examples of all or most of the problems pulled fromstudent papers. The instructor did not return the students' texts until the end of the class period in an attempt to keep the students' attention on that day's material. Therefore, when the instructor went through the comment list, the texts were readily available for discussion. No student names were mentioned, but the students did ask questions and apparently, from the number of revisions on their texts, were able to make use of the information. Conclusions and Implications Based on the results of this study, teacher comment influences student revision choices and the more directive the teacher comment, the better chance the students will revise their texts successfully. This agrees with Flower et al. (1986), Newkirk (1981) and Shuman (1975), but this study builds on their work by providing specific illustrations of teacher comment that offers problem identification and revision strategies paired with actual student revisions. The placement of written teacher comment does have some influence

on student revisions. In this study, there were more total revisions associated with oral only comment than the other three types. The previously mentioned audio tapes indicate that the oral comment sessions frequently included multiple examples of a problem and multiple solutions. These additional examples may be part of the reason for the additional revisions. Another possibility may be Bolker's ( 1978) suggestion that students fear teacher comment and, because the oral comment is less direct, it is therefore less threatening and students are more apt to listen. Oral feedback may help build a sense of community rather than force

students to view problems in the texts as theirs alone. The combination of research methods used in this study added strength to the conclusions of both portions of the research. For example, deletions were the only statistically significant response in the experimental study. This outcome could be explained more clearly using results from the naturalistic inquiry portion of the study. Matching specific teacher comments with specific revisions revealed that many of the comments suggested or hinted at deletion as a revision option. Also, the results of both portions of the study pointed to the importance of more detailed teacher comment either in the form of more revisions associated with concluding comments on the texts or the more frequent and more successful revisions from specific comments on the texts. Several alterations would enhance future studies using this method. First, the use of Rodger's (1967) stadia of discourse for identifying changes in the texts was a little too coarse for the revisions involved. Reviewing changes at the word or phrase level would be potentially more accurate. Identifying a way to limit the variety of teacher comment statements by using a check sheet or other method would better focus the naturalistic inquiry portion of the study and help further identify which comments elicited successful revisions from the students. Implications for future research include further examination of oral comment as a feedback method on student papers. The potential time savings for writing instructors, as well as the possibilities of greater improvement in student writing make this an important area for future study. Continued evaluation of written comment would also be extremely valuable. Careful evaluation of written comment location and content could lead to better writing and better use of instructor's time. Finally, one of the major goals of providing comment on student texts is to help the students learn tointernalize the ability to evaluate their own texts. Identifying feedback methods that can help students learn to evaluate

their own writing more successfully will enable them to become better writers. Agricultural Communications programs frequently offer writing courses as either part of their curriculum or as a service to their colleges. Providing efficient and timely feedback on these papers is an increasing challenge as faculty work toward tenure or promotion with ever-growing studentdemand. Refining our methods of providing students with feedback on their papers will ultimately improve our students' writing ability while still making the most efficient use of faculty member time and resources the best of both worlds for all concerned. -1Questia, a part of Gale, Cengage Learning. www.questia.com

Publication Information: Article Title: Feedback Follow Up: the Influence of Teacher Comment on Student Writing Assignments. Contributors: Deborah W. Dunsford - author. Journal Title: NACTA Journal. Volume: 50. Issue: 2. Publication Year: 2006. Page Number: 12+. 2006 North American Colleges and Teachers of Agriculture. Provided by ProQuest LLC. All Rights Reserved.

Examining the Effect of Feedback inBeginning L2 Composition

by Carolyn Gascoigne Introduction While reviewing research on English-as-a-second-language (ESL) writing, Goldstein (2001) articulated the following simple, yet unresolved question: "What role does teacher commentary playin helping students become more efficient writers?" (p. 78). Regardless of the context for writing (Ll, ESL, or L2), "teachers and students alike intuit that written responses can have a great effect on student writing and

attitude" (Leki, 1990, p. 58). Some of us even prefer to believe that studentwriting improves "in direct proportion to the amount of time [we] spend on their papers" (Hariston, 1986, p. 117). The illusion that the time we invest in correcting and commenting on

student writing has a perfect and positive correlation to the quality of a student's final product is now under attack. Indeed, years of working as "composition slaves" (Hariston, 1986) has not produced the results that the countless hours of reading, correction, and commentary would demand. Instead of a simple equation wherein a given amount of feedback equals a predictable amount of improvement, Leki (1990) painted a more realistic picture in which she described writing teachers as those who "behave with the same combination of a sense of responsibility and a sense of helplessness as a coach of a football team, booing and cheering while pacing the margins of the student's paper ... or like a coroner diagnosingthe cause of death" (p. 57). Given that providing written feedback and writing evaluative commentary is one of the "great tasks" (Gonners & Lunsford, 1993, p. 200) both quantitatively in terms of sheer number of hours, and qualitatively in terms of personal investment, one might think it would also be a central area of examination. Although there has been a growing body of literature devoted to the impact of peer response on ESL student revision, studies of teacher response and its effects on revision have been few (Ferris, 1997). According to Leki (1990), there may be a fairly large amount of information examining the type of teacher response in Ll writing, yet "examples of feedback and subsequent student action are rare" (p. 64) and studies of teachers' responses in the L2 setting are "practically nonexistent" (Zamel, 1985, p. 83). For example, while reviewing all published investigations of teachers' written commentary on rhetoric and content in ESL and L2 settings, an area of research that did "not really begin until the 1990s" (p. 75), Goldstein (2001) uncovered a paltry 15 studies. Of these 15, only 4 looked at the relationship between teacher-written commentary and either subsequent student revision or essay scores. The 11 others examined student

perceptions of commentary or the type of teacher feedback on final drafts. In an attempt to address this lacuna, the present study replicated one of the few ESL investigations that examined the effect of teacher feedback on subsequent composition revisions. The major distinction, however, was that the present study focused its attention on a beginning L2 writingpopulation. Results of this investigation will help determine whether or not the ESL findings are unique to that particular population, or are more universal in nature. Type of Feedback In the undeniably few studies examining teacher feedback, there are several trends that emerged. First, there has been considerable debate over where teacher/commentator attention should be placed: on form or content (Hall, 1990). To help inform this debate, Fatham and Whalley (1990) compared revision scores on both form and content among four groups of ESL writers: Group 1 received no feedback whatsoever; Group 2 received grammar or formfocused feedback only; Group 3 received content feedback only, and; Group 4 received feedback on both grammar and content. Theauthors expected that students would focus on different aspects of their compositions depending upon the type of feedback that the teacher provided (p. 182). After scoring original compositions and revisions, the authors found that students made statistically significant improvements in grammatical accuracy only when they were given explicit feedback on grammar. More surprising, however, wasthe finding that all groups significantly improved the content of their compositions irrespective of thetype of feedback received. In other words, students improved the content of their revisions even when teachers provided no feedback concerning the content of the original essay. Specific feedbackon grammar errors appeared to have a greater effect on grammar revisions than general content comments had on revisions of content. Nevertheless, the authors concluded that feedback on grammar and content, whether

given alone or simultaneously, both had a positive effect on revision (p. 185). Despite these encouraging findings, many researchers have lamented the focus that L2 teachers tend to place on form, or surface-level features, rather than content. For many of us, the practice of calling attention to error is still the most common procedure for responding to ESL and L2 writing(Cohen & Cavalcanti, 1990; Ferris, 1995; Leki, 1991; Zamel, 1985). For Gumming (1983), Error identification appears to be ingrained in the habitual practices of L2 teachers who perhaps by reason of perceiving their role solely as instructors of the formal aspects of language, therefore restrict their activities to operations exclusively within the domain of formal training, rather than that of cognitive development (p. 6). Zamel (1985) believed that this trend is due to the fact that ESL (and L2) teachers overwhelmingly view themselves as language teachers, rather than writing teachers. A second major criticism found in the literature

concerned the manner in which feedback (form or content focused) is supplied. Even among Ll studies, the research has criticized teachers for being "too general (rubberstamping students' papers with remarks like 'be specific'), for being too specific (giving students advice that is so text specific that they cannot use it in subsequent writing), and for focusing too heavily on surface features" (Goldstein, 2001, p. 60). Others, such as Burkland & Grimm (1986), lamented the futility of providing feedback because discard the paper" (p. 62). Summarizing the situation, typical feedback problems: (1) teachers respond to most writing as if it were a final draft, thus reinforcing an extremely constricted notion of composing; Zamel (1985) provided an extensive list of many Ll students "read the grade and simply

(2) teachers' marks and comments usually take the form of abstract and vague prescriptions and directives that students find difficult to interpret; (3) teachers (especially ESL and L2 teachers) are often more concerned with language-specific errors and problems and rarely expect students to revise beyond the surface level; (4) marks and comments are often confusing, arbitrary, and inaccessible; (5) teachers appropriate students' texts and impose their own purpose and ideas on students; (6) teachers send mixed messages to students by addressing minor surface features and larger issues of rhetoric and context in the same version of a text. For example, mechanical errors might be pinpointed at the same time that students are asked to elaborate upon an idea; and (7) teachers often fail to provide explicit, text-specific directions, guidelines, and revising strategies, (p. 79-82) Hearing the resounding criticism of both the type and the shape of feedback given to student writers, one must begin to question the value of composition correction and feedback altogether. (Burkland & Grimm, 1986; Hillocks, 1982; Knoblauch & Brannon, 1981; Leki, 1990). On a more optimistic note, surveys of L2 writers (Cohen, 1987; Cohen & Cavalcanti, 1990; Ferris, 1995; McCurdy 1992) have revealed that L2 writers are generally "happy with the feedback they receive, and claim that they pay attention to it and find it helpful" (Ferris, 1995, p. 36). ESL writers inFerris's study, for example, reported wanting and paying the most attention to comments on grammar (67%), with attention to content-oriented feedback close behind (63%) (p. 40). This enthusiastic feedback by ESL writers is much more positive than that revealed by Ll

surveys.

Indeed,

Leki

(1991)

found

that

her

ESL

students

equate

good writing in English with error-free writing and both "want and expect their teachers to correct errors" (p. 203), whereas Ll students reported not paying much attention to teacher commentary, not understanding it, or feeling some hostility. Inspired by these encouraging findings concerning ESL and L2 students' selfperceived acceptance of and positive attitudes toward teacher feedback, Ferris (1997) sought to examine more closely theinfluence that teacher commentary actually had on ESL student revisions. To this end, Ferris examined 1,600 marginal and end comments written on 110 first drafts of papers by 47 advanced ESL students. She then examined the 110 revised drafts in an attempt to measure the influence, if any, that the commentary had on subsequent revisions, and to see whether the changes actually led to improvements. Her specific research questions were: (1) What characteristics of teacher commentary appear to influence student revision? and (2) Do revisions influenced by teacher feedback lead to substantive and effective changes instudents' papers? (p. 317) Two sets of analyses were completed. The first examined the feedback provided by the teacher andthe second measured the effect the feedback had on the revision process. The comment analysis specifically targeted feedback length, feedback type (i.e., making a request or giving information),the use of hedges (e.g., "maybe," "please," "might"), and text-based versus general commentary.The effect of the comments on revision was assessed according to an original 0-6 rating scale (O = no discernible change, 6 = substantial positive change). Ferris found that marginal requests for information, general requests (regardless of syntactic form), and summary comments on grammar led to the most substantive revisions. The use of hedges did not inhibit effective revisions as anticipated. In fact, they seemed to encourage positive change. Less influential were questions or

statements that provided information to the students and positive comments. Length of feedback appeared to be an important variable. In general "longer comments and those which were text specific were associated with major changes more than were shorter, general comments" (p. 330). Moreover, when either minor or substantive changes occurred, they overwhelmingly resulted in an improvement of the students' papers. Very few of the changes (less than 5%) were found to have a negative impact. Finally, text-specific comments resulted in more positive changes than general comments. Ferris concluded that not all feedback is equal and certain forms may be particularly difficult for students to interpret. She also found that although students generally addressed comments givenin the form of a question, the changes that resulted had mixed effects on the revisions. This suggested "that although the students appeared to understand from the comment that something was required of them, they were less clear about how to incorporate the requested changes successfully" (p. 331). A sec ond problematic type of comment was the "give-information comment" (e.g., "Iowa law favors parental rights. Michigan and California consider the best interests of thechild." p. 321). Often, this type of comment did not lead to change or produced mixed results in therevision when it did. Ferris indicates that the give-information comment is less effective because it does not explicitly instruct the writer to incorporate the information that is supplied. Given that the manner in which teachers comment and, more important, the effect of comments on subsequent revisions has gone largely unexplored (Conners & Lunsford, 1993; Goldstein, 2001; Leki, 1990), and that this lacuna is especially acute in the nonnative writing environment (Ferris, 1997; Zamel, 1985), Ferris's study was an essential one. However, Ferris examined an advanced ESL writing population only. Would her findings still hold in an L2 context, in particular an introductory L2 environment? Or are the effects of feedback context specific? If Ferris's findings were not applicable to an L2 context, then a host of studies targeting a range of abilities and languages would need to be run before firm generalizations could be made. The

purpose of the present study, therefore, was to examine the effect of feedback on subsequent revisions in the beginning L2 environment. The results of this effort should be of practical interest to the L2 teacher, as well as help delineate or reinforce the applicability of Ferris's findings. Method Instead of looking at the influence of teacher commentary in advanced ESL students' revisions, the present study attempts to measure the influence that commentary has on subsequent composition revisions in a beginning L2 French class at the postsecondary level. Thus, the primary research questions are: (1) What characteristics of teacher commentary appear to have the greatest effect on beginning L2 composition revisions, and (2) Do revisions prompted by teacher feedback lead to substantive and effective changes inbeginning L2 students' papers? Participants Twenty-two of the 25 subjects were freshmen; 19 were female and 6 were male. all participants were native speakers of English who had either no formal exposure to French prior to this course, or were placed into the beginning course as the result of their score on a standardized placement exam. The Course The introductory French course was part of a four-semester language requirement. The course focused on speaking, writing, reading, listening comprehension, culture, and grammar. The Vis-a-Vis (Amon, Muyskens & Omaggio Hadley, 2000) first-year textbook program was used. This course met for 80 minutes, three days a week for 16 weeks. Other than the formal writing activities discussed below, writing was primarily used as a support skill (i.e., filling in blanks for grammar activities, creating

vocabulary lists). The Writing Tasks Writing activities (one per chapter, eight chapters total) came directly from the textbook so that students were exposed to the vocabulary, structures, and themes essential to the writing task. Each in-class writing assignment began with a prewriting activity consisting of 3-5 guiding questions. Next, students were to draft a short composition (average composition length was 92 words) based upon the ideas generated in the prewriting activity. Students were mainly asked to compose descriptive texts (e.g., describe your best friend; describe the house of your dreams; talk about your family; describe your eating habits). Students had fifty minutes to complete the prewriting andwriting tasks. Procedure Only compositions from chapters 4-8 were selected for examination. The first three writing activities, which averaged only 37 words, were found to be too short for meaningful analysis. Ultimately, 516 marginal and end comments written on 114 first drafts of papers by 25 beginning French languagestudents at the University of Nebraska at Omaha were examined and catalogued by the investigator and a second independent reader according to the following features identified by Ferris (1997): (1) Comment length in number of words; (2) Comment type (pragmatic intent and syntactic form) ; (3) The use of hedges (e.g., please, maybe, perhaps), and; (4) Whether the comment was text based or general, (p. 320)

Comment type was further broken down into the following categories: (a) Ask for information/question (e.g., Did you consult any sources?) (b) Make a request/question (e.g., Can you provide an example here?) (c) Make a request/statement (e.g., This would be better earlier in the essay.) (d) Make a request/imperative (e.g., Add a citation.) (e) Give information/question (e.g., The first case was in 1899. Does this change your view?) (f) Give information/statement (e.g., The first case was in 1899, not 1919.) (g) Make a positive comment, statement, exclamation (e.g., This is a great start.) (h) Make a grammar/mechanics comment, question, statement, or imperative, (p. 322) Papers with comments were returned to students for revision. Students had two days out of class to revise their compositions. The 114 revised drafts were then examined' to measure the influence of the commentary. Revisions were assessed by the investigator according to Ferris's (1997) scale: O No discernible change made by student in response to comment; 1 Minimal attempt to address comment, effect generally negative or negligible; 2 Substantive change in response to comment, effect generally negative or negligible; 3 Minimal attempt in response to comment, effect mixed; 4 Minimal attempt to address comment, effect generally positive;

5 Substantive change in response to comment, effect mixed; 6 Substantive change in response to comment, effect generally positive, (p. 322) A 10% sample was verified by a second independent reader. Interrater reliability was 96%. Findings Feedback Type Although no feedback or correction code was used, there was relatively little variation in the type and shape of the teacher's comments. First, all feedback was given in the Ll (English). second, nearly all comments were direct and succinct with an average comment length of only four words. Third, 398 of the 516 comments were text based rather than general, and there were only three cases where a hedge was used. As for comment type, there were no "asking for information" questions, no requests in the shape of a question, and no requests for information in the shape of a statement. Instead, there were 64 examples of a request in the shape of an imperative. For example, a request for detail was directly stated, "Add more detail here," rather than politely requested, "Can you add more detail here?" or "This would be better with more detail." There were no cases where factual information was provided to the writer either in the form of a question or a statement. There were 118 examples of a positive comment, statement or exclamation (e.g., "This is great!"), and 334 "comments, questions, statements, or imperatives" (Ferris, p. 322) focusing on grammar or mechanics, nearly all of which were supplied as an imperative. Examples of formfocused feedback include "Pay attention to verb endings," or "Don't forget agreement." There were no codes, symbols, or systems used in conjunction with the commentary. However, an arrow was often drawn to link feedback to the phrase or

sentence in question. Comment Effect Ferris's (1997) scale was used to measure the influence of the 64 "requests/imperative" comments. Using the 0-6 scale, the average rating was 4.4, or "minimal attempt to address comment, effect generally positive." There were two cases where "no discernable change was made by the studentin response to the comment," and two cases where there was a "minimal attempt to address the comment, with an effect that was generally negative or negligible." There were 12 cases where the revisions showed evidence of "minimal attempt in response to commentary with a mixed effect" and 14 examples of a "minimal attempt to address comments with a generally positive effect." There were 16 cases where there was "substantive change in response to comment with mixed effect," and 18 cases where there was "substantive change with a generally positive effect." The 118 positive comments elicited no change whatsoever in any of the revisions. In contrast, the 334 comments devoted to grammar and mechanics had a profound effect. Eighty-eight percent of all such comments led to a successful correction, 8% led to an incorrect change, and a mere 3% were ignored by the students. Discussion Several differences in the outcome of the present study and that of Ferris (1997) are worth noting. First, Ferris examined 110 sets of first and second drafts producing 1,600 marginal and end comments. The present study targeted 114 sets of first and second drafts that yielded a mere one third (516) of the marginal and end comments produced in the Ferris study. The much smaller number of comments in the L2 context is likely due to the brevity of the L2 compositions. However, this is only speculation given that the average length of the ESL compositions is not known. second, whereas Ferris found that the use of hedges led to positive

change in the students' revisions, no conclusions or comparisons stemming from the present study can be drawn due to the rare use of hedges (three total). Similarly, Ferris found that length of feedback correlated positively with the success of the revision. In the present study, all comments were remarkably terse (the average length was four words), yet feedback still led to a large number of successful revisions. Finally, no comparisons of comment type can be made among the present data or to those of the Ferris study because nearly all comments took the shape of a grammarfocused imperative or a positive comment, statement, or exclamation. Similarities among the outcomes of the two studies exist as well. For example, just as Ferris found that textspecific comments produced more positive changes, so did the present investigation. Also, Ferris found that less than 5% of the student revisions led to a negative change. Similar results were found in the present study where 8% of the changes were incorrect. Conclusion The copious occurrence of brief form-focused comments in this study unwittingly served to reinforce the stereotype of the L2 teacher as one concerned with language rather than composition, and form over meaning. In the words of Zamel (1985) "teachers (especially ESL and L2 teachers) are often more concerned with language-specific errors and problems and rarely expect students to revise beyond the surface level" (p. 79). Certainly, the level of L2 ability examined here (first semester) is one at which one expects to find a great deal of mechanical errors. It is also a context in which students typically "want and expect teachers to correct their errors" (Leki, 1991, p. 203). The clear focus on grammar and mechanics and the simple and direct feedback provided is not entirely surprising, yet it does limit the extent to which comparisons with the Ferris study can be drawn. Unfortunately, this investigation did not yield meaningful data concerning the effect of contentfeedbackother than general positive comments, statements, and exclamations-or the effect of different comment types (statement, question, request, or provision of information). It

did,

however,

reveal

that

beginning

L2

writers

incorporate teacher feedback concerning grammar and mechanics and that they tend to do so successfully. Even at the earliest stages of E2 instruction, teachers need not feel that they must supply surface-level corrections to students during the revision process. Indicating that a mechanical error has taken place through the use of a short and direct imperative or statement appears to suffice. Although further study is still desperately needed concerning less formfocused feedback, the present study demonstrates encouraging findings concerning the effectiveness of what-right or wrong-is still perhaps the most ubiquitous type of L2 composition feedback. For those who view themselves as language teachers rather than writing teachers, these findings imply that L2 students do successfully incorporate short and direct form-focused feedback. Moreover, knowing that students are capable of such revisions, and that even first-semester L2 students can successfully revise at the surface level, should encourage teachers to expand their feedback repertoire even further. -1Questia, a part of Gale, Cengage Learning. www.questia.com

Publication Information: Article Title: Examining the Effect of Feedback in Beginning L2 Composition. Contributors: Carolyn Gascoigne - author. Journal Title: Foreign Language Annals. Volume: 37. Issue: 1. Publication Year: 2004. Page Number: 71+. 2004 American Council on the Teaching of Foreign Languages. Provided by ProQuest LLC. All Rights Reserved.

Effects of Response Mode and Time Allotment on College Students' Writing.

by Benjamin J. Lovett , Lawrence J. Lewandowski , Cassie Berger , Rebecca A. Gathje Written essays area common feature of classroom and high stakes standardized tests at

many age levels. However, little is known about how small alterations in the writing task affect students' writing, an issue made more important by the increasing use of task alterations for students with disabilities. In the present study, 140 college students completed a standardized assessment of writing ability under one of two timing conditions (10 minutes, 15 minutes) and with one of two response formats (handwritten, word-processed). Students who used a word processor wrote significantly more than did students who handwrote their essays. However, the extended time allotment only increased students' amount ofwriting in the word processor condition. Only small differences between the groups' essays were found in spelling, grammar, and related characteristics. Implications of these results for future research and writingassessment policies are discussed. Traditionally, two types of test items have been used in educational assessment: items in which thestudent selects the correct response from a set (as seen in multiple-choice, truefalse, and matching items) and items in which the student constructs a response on his of her own (as in short answer or essay items). The latter type of test item has become more common in large-scale standardized testing for several reasons. First, writing samples are thought by many to be the best method of assessing writingability (Conlan, 1986; Linn & Miller, 2005). Second, computer programs have been developed to scorewriting samples (Dikli, 2006), reducing the financial and logistical challenges associated with this type of assessment. Third, multiple-choice tests have continued to raise concerns about the ability to measure complex reasoning and problem solving (for a review, see Phelps, 2003). Although essay tests have certain acknowledged advantages over selected-response tests (e.g., utility in assessing certain higher-order learning objectives), essay tests also have potential limitations. One such limitation is that the response mode of the test may significantly affect examinees' scores; composing an essay using a computerized word processor program may lead to a different score than composing an essay by hand (Russell, 1999). A second limitation is that time limits, which determine the amount of text that students can compose, may significantly affect students' scores, since the amount

of text written is a robust correlate of holistic measures of essay quality (Hopkins, 1998; Powers, 2005). In the present study, we explore these issues empirically, asking how time limits and response modes interact to affect students' essay composition. Before reviewing relevant literature on writing assessment, this study briefly discuss one of the motivations behind the study. An increasing number of students in higher education have diagnoses of common disabilities (e.g., learning disabilities, attention problems, psychiatric disorders, etc.) that may adversely impact their scores on standardized tests (Konur, 2002). In many countries, alterations are made to the administration of the tests (for example, testing accommodations) in the hopes ofgiving these students a fair chance at showing their skills (Bolt & Thurlow, 2004; Hampton & Gosden, 2004). Extending test time limits and allowing examinees to use computers to write are among the most common accommodations offered, but little is known about how these accommodations affect essay examinations. An appropriate test accommodation should mitigate performance obstacles of students with disabilities (e.g., large print for a student with visual limitations), while having less of an effect on the performance of non-accommodated students (Fuchs & Fuchs, 2001). However, many testing accommodations provide at least some benefit to students both with and without disabilities (Sireci, Scarpati, & Li, 2005). Indeed, some work shows that students who have poor academic skills but no disability diagnoses benefit more from accommodations than those with official diagnoses do (e.g., Elliott & Marquart, 2004). The present study, then, was conducted in part for its potential implications for the use of extended time and word processor accommodations, both for students with and without disability diagnoses. Response Mode and Writing Performance A number of studies have compared writing produced with and without the aid of a word processor. In a recent meta-analysis, Goldberg, Russell, and Cook (2003) examined 26 of these studies published between 1992 and 2002 using students in K-12 educational settings. These investigators concluded that word processors lead reliably to writing in

greater quantities (with a weighted effect size of d = .50) andto writing of better quality (judged using a variety of measures, depending on the study; weighted effect size of d = . 41). These positive effects are consistent with results from earlier research (summatized in Bangert-Drowns, 1993), as well as the few studies that have been published since Goldberg and colleagues' meta-analysis (e.g., Burke & Cizek, 2006). Variability in word processing's effects has led researchers to search for potential moderators and mediators. Student age appears to be one moderator: Goldberg et al. (2003) found, in regard to both quantity and quality of writing, that middle and high school students benefited more from the use of word processors than elementary school students. Computer skills, a related variable, may also determine students' degree of benefit; Russell (1999) found that middle school students with above-average keyboarding skills benefited substantially from word processing, whereas students with below-average skills actually showed a decrease in essay performance. Similarly, a recent study by Burke and Cizek (2006) found a set of complex interactions between response mode and self-perceived computer skills on the essays written by sixth graders in which higher computer skills were generally associated with more benefit from word processing. One issue in measuring the effects of response mode on writing performance has been rater bias in the assessment of writing composed using different modes; that is, it is reasonable to suspect that judges who score essays may be more generous to essays that are either handwritten of word-processed. Research has generally shown that judges are more generous to essays that are handwritten. In a landmark study on the topic, Powers, Fowles, Farnum, and Ramsey (1994) asked college students to write two essays on different topics, one in handwritten form and one in word-processed form. The handwritten essays were then word-processed exactly as originally written, without correcting for misspellings and grammar, and the word-processed essays were written down on paper. These investigators found that when original handwritten essays were word-processed and scored again, the scores decreased significantly, and when the computer-produced essays were handwritten and scored again, the mean increased. In

qualitative assessments, raters reported that word-processed essays appeared to be shorter and claimed to give handwritten essays the "benefit of the doubt," especially when the handwriting was poor. Other studies on the topic have found similar results (e.g., Arnold et al., 1990; MacCann, Eastment, & Pickering, 2002), suggesting that all essays should be presented in the same way (either typed or handwritten) to scorers to avoid rater bias effects. Time Allotment and Writing Performance The substantial relationship between essay length and quality is well known (e.g., Hopkins, 1998; Powers, 2005). Time limits clearly have the potential to serve as ceilings in determining essay length. However, few studies have directly examined the effects of time allotment on writing. Many studies have focused on the speededness of tests--that is, the degree to which scores on tests are the product of working quickly (Lu & Sireci, 2007)--but the tests in these studies are rarely measures of writing skill. In their review of the small extant literature on time allotment and writing quality, Powers and Fowles (1996) noted that time allotment has generally, but not always, been found to increase writing performance. In their own comprehensive study, these investigators assigned college students randomly to 40-minute or 60-minute conditions, finding that scores of the examinees in the 60-minute conditions were approximately one half of a standard deviation higher than scores of examinees in the 40-minute conditions; extended time allotments, then, had a meaningful effect on scores. In a more recent study, Crawford, Helwig, and Tindal (2004) examined several features of fifth- and eighth-grade students' compositions when the students were given either 30 minutes or three days tocomplete a composition assignment. Each student completed one assignment under each of the two time conditions. Crawford and colleagues found that although fifth-graders' three-day compositions were superior to their 30-minute compositions, no such effect was present for eighth-graders, suggesting that age may be a moderator of timing effects. In addition, time had a greater effect on the scores of students who were receiving special education services, even though the nondisabled

fifth-grade students' essays also showed a (smaller) benefit. Of course, giving examinees three days to complete an assignment raises other concerns, since examinees will clearly not be spending the entire interval working on the assignment. In our study, we felt it was important to keep all time intervals brief enough that examinees could be expected to be working steadily throughout the intervals. The Present Study If response mode and time allotment have been found to affect the writing performance of students of varying ages and education levels, it is reasonable to ask how these factors might interact. That is, does the effect of time depend on the response mode, and vice versa? This question is important because many students with disabilities are given alterations in test administration that include simultaneous changes to both response mode and time allotment. In addition, more generally, high stakes tests (e.g., college and graduate admissions tests) and classroom assessments both vary substantially in time allotment and response mode. The present study investigated this issue in a sample of college students whose writing was evaluated under varying conditions. Method Participants The participants were 140 college students (73 females, 67 males) taking ah introductory psychology course at a large, private university in the northeastern United States. They ranged in age from 18 to 22 years old (M = 19.1), and most (75%) were either first- or second-year students. The participants' college grade point averages (GPAs) ranged from 2.0 to 4.0 (M = 3.2). They completed testing in small group sessions. Each session was randomly assigned, using one of four conditions: hand writing for 10 minutes (HW-10), hand writing for 15 minutes (HW-15), word-processed writing for 10 minutes (WP-10), and word-processed writing for 15 minutes (WP-15). This assignment resulted in four groups of participants (ns ranging from 29 to 42) that did not differ significantly with

respect to gender composition ([chi square] = 1.04, p = .80), age (F = 1.38, p = .25), year in school (F = .30, p = .83), or college GPA (F = .84, p = .48; for more details on these data, see Table 1). Measures The study included three primary measurement tools: one assessed participants' motor speed in either typing or hand writing, a second tool assessed general essay writing skill, and a third tool assessed participants' ability to compose brief sentences quickly. To measure each of these three qualities, we sought tasks that could be scored objectively, and, where relevant, tasks that had evidence of validity showing that the tasks measured the desired qualities. Writing/typing speed task. Students were asked to type or hand write (depending on the experimental condition) the word forms of numbers (i.e., "one, two, three ...") for one minute as quickly as possible. Each character correctly typed/written was counted to form a total score used to measure motor speed. Assessment of spontaneous writing skills. The spontaneous writing materials from Form B of the Test of Written Language, Third Edition (TOWL-3; Hammill & Larsen, 1996) were used to assess participants'writing quality. Students were shown a picture and given a set length of time to write a story about the picture. The TOWL-3 essay rating procedures yield three separate scores: one for "contextual conventions" (spelling and mechanics), one for "contextual language" (vocabulary, grammar, and syntax), and one for "story construction" (prose, plot, and organization); these three scores are typically summedto generate a "spontaneous writing composite" score. Additionally, the number of words that participants wrote were counted to obtain a writing quantity measure. The TOWL-3 is one of the most commonly used standardized measures of writing skill (Salvia & Ysseldyke, 2007), and although it is typically administered to students at the high school level and below, most of the dimensions of writing that it assesses are widely recognized as being important at the college level and beyond (e.g., Jones, 1994). In any

case, the present study did not use the TOWL-3 norm-referenced scores to make absolute judgments of students' writing skill levels, but rather used the scores to compare groups that completed the test under different conditions. In addition, although the essay task is brief (10-15 minutes), this is a common length of time allotted for individual essays on both high-stakes and classroom assessments. Writing fluency measure. The writing fluency subtest from the Woodcock-Johnson Tests of Achievement, Third Edition, Form A (WJ-III; Woodcock, McGrew, & Mather, 2001) was used to assess students' ability towrite simple expository text quickly. For each item on this test, students were asked to write a sentence that contained all three of the words shown for that item (e.g., dog, big, is); students were given seven minutes to complete as many items as they could. This test has shown adequate reliability ([rho] = .88) and there is substantial evidence of construct validity, based on correlations with other WJ-III subtests as well as with other measures of writing. Procedures We conducted all sessions in the same small, quiet classroom, with no more than 10 students being tested at a time. After completing informed consent forms, we gave participants one minute to complete the motor speed task, and when the minute was over, we passed out the TOWL-3 stimulus picture. We gave participants in the hand written (HW) sessions pencils and lined paper to write with, whereas participants in the word processor (WP) sessions used a word processing program (Microsoft WordPad Version 5.1) to type their compositions on laptop computers (Dell Inspiron 1405 models with standard keyboards and a 14-inch monitor) that we provided. This program does not have spell-checking or similar aids. After completing the TOWL-3 essay assignment, all students completed the WJ-III Writing Fluency subtest, using a hand written response format. We then asked students to complete a brief questionnaire, including demographic information as well as questions about computer usage. For scoring, we double-spaced and printed all WP essays, and we typed, double-spaced, and printed all HW essays, so that judges would not allow the typed format to influence

their ratings of writing quality. We trained two judges to score TOWL-3 essays, and each of the essays was scored by one of the two judges, with 20% of the essays doublescored to ensure reliability across judges. The inter-rater reliability (using a Pearson correlation) for number of words written was 1.0 (100%); the coefficients for the contextual conventions, contextual language, and story construction subtests from the TOWL-3 were .90, .88, and .76, respectively, and the coefficient for the TOWL total score was .88. Results We present mean scores in Table 1 for demographic and performance variables of all four groups (WP-10 minutes, WP-15 minutes, HW-10 minutes, and HW-15 minutes). There were no significant differences on demographic variables. The differences on performance variables were generally unsurprising; students in the WP conditions wrote more words in their essays and wrote more words during the speed task than students in the HW conditions did, although no significant differences in quality (i.e., TOWL scores) were present in these analyses. A second group of analyses examined correlations among writing measures for the HW groups separately from the WP groups. Because time allotment had very little effect on correlations, data was collapsed across time conditions. Table 2 displays the correlation matrices for both response modes, and in some ways the correlations are quite different. For example, the writing/typing speed score correlated significantly with all writing measures for the WP conditions, but did not correlate significantly with any quality measures for the HW groups. Similarly, the number of words written by the WP groups correlated significantly with all writing measures, yet much lower correlations were found between these measures for the HW groups. Writing speed and fluency, then, were better predictors of quality (and rice versa) when essays were completed using a word processor. Table 3 presents analysis of variance (ANOVA) models that examined the interaction between responsemode and time allotment. Of most interest were the Total TOWL score

(quality) and essay length (quantity) measures. With regard to quality, there were no differences found for writing format/responsemode or time condition and no interaction (p-values ranging from .08 to .95). However, when considering quantity, we found an interaction of time and response mode conditions, F(1,139) = 10.73, p < .01. The main effects for both time and response mode were also significant (p < .01). Further analysis revealed that the WP groups wrote significantly longer essays than the HW groups, and additional time increased essay length only in the WP conditions. The WP-15 group (M = 365 words) significantly outperformed the WP-10 group (M = 261 words), yet there was no difference in essay length between HW-10 (M = 215 words) and HW-15 (M = 216 words) groups. The three remaining TOWL scores were of less interest in that they are constituents of the Total TOWL score already analyzed for writing quality. However, analyses of these variables yielded a small effect for time on the Contextual Conventions subtest, with slightly better scores for the groups receiving 10 minutes to write. We also found a small effect of response mode on the Contextual Language subtest score, with those in the WP condition outperforming the HW groups. These analyses are also presented in Table 3. Discussion We examined the effects of two common test administration variables, time allotment and response mode, on the length and quality of written essays in college students. The results in this study were mixed. No effect of test conditions was found for writing quality, while the word processor proved to be better than hand written response mode for essay length (quantity). The effect was larger when students had more time to write. Increasing the time allotment had no effect when students were hand writing essays, but the same increase in time allotment had a greater effect when essays were completed using a word processor. Moreover, there was no relationship between essay length and quality (r = .03) for hand written essays, yet this relationship was quite strong (r = .61) for word-processed essays. These and other correlations among measures for both response modes suggest that the relationship betweenwriting speed

and writing quality is sensitive to whether compositions are typed or hand written. The findings in this study are consistent with many other studies examining the effects of administration conditions on writing performance. For instance, Russell and Haney (1997) found, as did we, that longer essays were composed on a computer rather than by hand. Our finding that essay length correlated significantly with quality is also congruent with past work (e.g., Espin, Scierka, Skare, & Halverson, 1999; Mattern, Camara, & Kobrin, 2007), although only for the word processor response mode. Finally, our finding that word-processed essays were longer but not of higher quality recalls the results of Truell, Alexander, and Davis (2004), who found that marketing students who used word processors completed a classroom test more quickly but did not obtain higher scores. In sum, our study coincides with others in the literature, showing that word processing affords an opportunity to be more time efficient and/or more generative in a fixed time period, and that it may or may not improve writing quality, depending on the task, scoring procedures, sample under study, and other features. However, two discrepancies between our findings and previous work raise interesting questions. First, why did increased time help only students who word-processed their essays? Composition is generally understood to be a high-effort academic task (for a review, see Torrance & Galbraith, 2006), and if students experienced hand writing essays as requiring more effort than word processing them (given that our students were used to writing using computers), students may have been more motivated to expand and revise their essays with the extra time when using a word processor, whereas students in the hand written conditions may have been satisfied with a minimally adequate essay product. A related question is raised by our correlations between essay length and quality: why was a positive correlation between these variables only found in the word processor conditions? Part of the answer may lie in the fact that the variables exhibited higher variability in the word-processor conditions than in the hand written conditions; the standard deviation for essay length was smaller for all hand written essays than for word-

processed essays, and lack of variability restricts the possible values of the correlation coefficient (Nunnally & Bernstein, 1994). Another possible explanation is that word processing makes it easier to revise text by deleting text that is repetitive, irrelevant, or that otherwise detracts from the essay (MacArthur, 2006), and so word-processed essays may be less likely to contain additional text that does not contribute to the essays' purposes. If hand written essays had more irrelevant text (owing in part to the higher effort required to revise), any relationship between length and quality would be attenuated significantly. Implications for Testing Accommodations Two of our findings appear to have implications for the provision of extended time and word processor testing accommodations. First, when combined, the longer time allotment and word processor use led tostudents writing substantially longer essays, and so if length of composition is important on a writing test, test users and administrators should know that students with disabilities who are given extended time and word processor accommodations are given a very substantial benefit over their classmates, at least in terms of essay length. Since the present study showed that the effects of extended time and word processor are found in students without disabilities, there may be little rationale for restricting these accommodations to students with disabilities (see Zuriff, 2000, for more on the logic of restricting accommodations). A second implication of our findings for testing accommodations comes from the interaction between time allotment and response mode. Our finding that extended time benefited only the fluency of students who word-processed their essays suggests that providing extended time alone as ah accommodation may not be beneficial to students on writing tests, unless a word processor is also provided. Since word processor accommodations may lower the requirements for examinee effort and self-motivation, more students may take advantage of any extra time that is provided. A new trend in the testing accommodations arena is the design of tests that MI examinees can access, so that accommodations are unnecessary, a concept called universal test

design (Thompson, Johnstone, Anderson, & Miller, 2005). A writing test with universal design might be taken using a word processor and under conditions of generous time limits, equivalent to providing the accommodations to all examinees. With regard to word processors, computer-based writing is becoming so ubiquitous that some test administrators (e.g., those of the GRE) have moved to completely computer-based testing. Alternatively, word processors may be given as ah option to any examinees that request them. However, the "optional" approach should be used cautiously, as the present study has shown that correlations between different essay scores are quite different for essays that are word-processed. Limitations and Directions for Future Research The present study has several limitations, each of which suggests future research that would be beneficial. First, although we were interested in the effects of common testing accommodations, our lack of a group of participants with disabilities precluded a complete test of appropriateness. Although some scholars (e.g., Phillips, 1994; Zuriff, 2000) would consider our finding that nondisabled students benefited from accommodations to be sufficient evidence against their being restricted to students with disabilities, other scholars (e.g., Fuchs & Fuchs, 2001; Sireci, Scarpati, & Li, 2005) argue that as long as students with disabilities benefit more from an accommodation than nondisabled students, the accommodation is appropriate, and as such, future work should include students with disabilities. A second extension would involve manipulating the test administration variables as within-subject factors,to determine whether individuals' scores actually improved as a function of time allotment and response mode. Examinees could complete one essay under each of several conditions, and for time allotment, they could even be stopped after a certain length of time, and then given an extension to see how the same essay changes. When extended time has been studied in the context of reading and mathematics performance (e.g., Lewandowski, Lovett, Parolin, Gordon, & Codding, 2007), this procedure has been useful, although writing is a somewhat different

task, and changes to ah essay may be difficult to record. A third extension relates to our writing stimulus (the TOWL-3 stimulus), which was not normed on college samples of designed specifically for college students, and was only a single, brief essay prompt. Given the importance of interest in motivating students to write, various writing prompts with a range of time demands should be tested to determine which stimuli lead to the greatest motivation, including allowing students choice in selecting a topic to write about. Of course, it should be noted that our own task was of a similar length to those found on many standardized writing tests, suggesting a degree of generalizability to our findings. The present study, then, should open the door to future research that examines the effects of task characteristics on the writing performance of students with and without disabilities, and with a range of academic skills. On a more practical note, test administrators and users might take away two messages from the present study's results. First, use of a word processor may be necessary for extended time accommodations to exert their full effects. Second, many college students appear to use word processing as their default mode of composition, suggesting that writing tests that require hand written responses may put these students in an unfamiliar situation, compromising the validity of the assessment. Therefore, in high-stakes testing situations, word processors should be considered as ah element of universal test design (Thompson et al., 2005), furthering the goal of appropriate assessment for all students. References Arnold, V., Legas, J., Obler, S., Pacheco, M. A., Russell, C., & Umbdenstock, L. (1990). Do students get higher scores on their word-processed papers? A study of bias in scoring hand-written vs. word-processed papers. Whittier, CA: Rio Hondo College. Retrieved from ERIC database (ED 345818). Bangert-Drowns, R. L. (1993). The word processor as an instructional tool: A metaanalysis of word processing in writing instruction. Review of Educational Research, 63,

69-93. Bolt, S. E., & Thurlow, M. L. (2004). Five of the most frequently allowed testing accommodations in state policy. Remedial and Special Education, 25, 141-152. Burke, J. N., & Cizek, G. J. (2006). Effects of composition mode and self-perceived computer skills on essay scores of sixth graders. Assessing Writing, 11, 148-166. Conlan, G. (1986). "Objective" measures of writing ability. In K. L. Greenberg, H. S. Wiener, & R. A. Donavan (Eds.), Writing assessment: Issues and strategies (pp. 109125). New York: Longman. Crawford, L., Helwig, R., & Tindal, G. (2004). Writing performance assessments: How important is extended time? Journal of Learning Disabilities, 37, 132-142. Dikli, S. (2006). An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment, 5(1), 1-36. Elliott, S. N., & Marquart, A. M. (2004). Extended time as a testing accommodation: Its effects and perceived consequences. Exceptional Children, 70, 349-367. Espin, C. A., Scierka, B. J., Skare, S., & Halverson, N. (1999). Criterion-related validity of curriculum-based measures in writing for secondary school students. Reading & Writing Quarterly, 14, 5-27. Fuchs, L. S., & Fuchs, D. (2001). Helping teachers formulate sound testing accommodation decisions for students with learning disabilities. Learning Disabilities Research and Practice, 16, 174-181. Goldberg, A., Russell, M., & Cook, A. (2003). The effect of computers on student writing: A meta-analysis of studies from 1992 to 2002. Journal of Technology, Learning, and Assessment, 2(1), 1-51. Hammill, D. D., & Larsen, S. C. (1996). Test of Written Language. (3rd ed.). Austin, TX:

Pro-Ed. Hampton, G., & Gosden, R. (2004). Fair play for students with disability. Journal of Higher Education Policy and Management, 26, 225-238. Hopkins, K. D. (1998). Educational and psychological measurement and evaluation. (8th ed.). Boston, MA: Allyn & Bacon. Jones, E. A. (1994). Defining essential writing skills for college graduates. Innovative Higher Education, 19, 67-78. Konur, O. (2002). Assessment of disabled students in higher education: Current public policy issues. Assessment & Evaluation in Higher Education, 27, 131-152. Lewandowski, L., Lovett, B. J., Parolin, R., Gordon, M., & Codding, R. S. (2007). Extended time accommodations and the mathematics performance of students with and without ADHD. Journal of Psychoeducational Assessment, 25, 17-28. Linn, R. L., & Miller, M. D. (2005). Measurement and assessment in teaching. (9th ed.). Upper Saddle River, NJ: Merrill. Lu, Y., & Sireci, S. G. (2007). Validity issues in test speededness. Educational Measurement: Issues and Practice, 26(4), 29-37. MacArthur, C. A. (2006). The effects of new technologies

on writing and writing processes. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 248-262). New York: Guilford. MacCann, R., Eastment, B., & Pickering, S. (2002). Responding to free examination questions: computer versus pen and paper. British Journal of Educational Technology, 33, 173-188. Mattern, K., Camara, W., & Kobrin, J. L. (2007). SAT Writing: An overview of research

and psychometrics to date (Research Note RN-32). New York: College Board. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. (3rd ed.). New York: McGraw-Hill. Phelps, R. P. (2003). Kill the messenger: The war on standardized testing. Edison, NJ: Transaction. Phillips, S. E. (1994). High-stakes testing accommodations: Validity versus disabled rights. Applied Measurement in Education, 7, 93-120. Powers, D. E. (2005). "Wordiness": A selective review of its influence, and suggestions for investigating its relevance in tests requiring extended written responses (ETS Research Memorandum RM-04-08). Princeton, NJ: Educational Testing Service. Powers, D. E., & Fowles, M. E. (1996). Effects of applying different time limits to a proposed GRE writingtest. Journal of Educational Measurement, 33, 433-452. Powers, D. E., Fowles, M. E., Farnum, M., & Ramsey, P. (1994). Will they think less of my handwritten essay if others word process theirs? Effects on essay scores of intermingling handwritten and word-processed essays. Journal of Educational Measurement, 31, 220-233. Russell, M. (1999). Testing on computers: A follow-up study comparing performance on computer and on paper. Education Policy Analysis Archives, 7(20). Retrieved January 28, 2008 from http://epaa.asu.edu/epaa/v7n20. Russell, M., & Haney, W. (1997). Testing writing on computers: An experiment comparing studentperformance on tests conducted via computer and via paper-andpencil. Education Policy Analysis Archives, 5(3). Retrieved January 28, 2008 from http://epaa.asu.edu/epaa/v5n3.html Salvia, J., & Ysseldyke, J. E. (2007). Assessment in special and inclusive education.

(10th ed.). Boston, MA: Houghton Mifflin. Sireci, S. G., Scarpati, S., & Li, S. (2005). Test accommodations for students with disabilities: An analysis of the interaction hypothesis. Review of Educational Research, 75, 457-490. Thompson, S. J., Johnstone, C. J., Anderson, M. E., & Miller, N. A. (2005). Considerations for the development and review of universally designed assessments (NCEO Technical Report No. 42). Minneapolis, MN: National Center on Educational Outcomes. Torrance, M., & Galbraith, D. (2006). The processing demands of writing. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 67-80). New York: Guilford. Truell, A. D., Alexander, M. W., & Davis, R. E. (2004). Comparing postsecondary marketing studentperformance on computer-based and handwritten essay tests. Journal of Career and Technical Education, 20, 69-78. Woodcock, R. W., McGrew, K. S., & Mather, N. (2001). Woodcock-Johnson Tests of Achievement. (3rd ed.). Itasca, IL: Riverside Publishing. Zuriff, G. E. (2000). Extra examination time for students with learning disabilities: An examination of the maximum potential thesis. Applied Measurement in Education, 13 (1), 99-117. Benjamin J. Lovett, Ph.D., is an assistant professor of psychology at Elmira College, where his research focuses on conceptual and psychometric issues in psychoeducational assessment and psychiatric diagnosis. Lawrence J. Lewandowski, Ph.D., is Meredith Professor of Teaching Excellence in the psychology department at Syracuse University. Cassie Berger, M.S., is a Ph.D. candidate in school psychology at Syracuse University. Rebecca A. Gathje, Ph.D., is a postdoctoral fellow at the Munroe-Meyer Institute and a clinician working out of the Grand Island Behavioral Health Clinic in Grand Island,

Nebraska.
Table Group Means on Demographic Handwritten Handwritten Measure (n = Age College WJ-111 Writing Words Context. Context. Story TOWL Speed Written ConvenLanguage Construction Total Measure Age College WJ-111 Writing Words Context. Context. Story TOWL * ** *** Table Performance Tusk p p Speed Written Conventions Language Construction Total p < < 2 Intercorrelations 2.23 1.49 1.17 <.05 .01 .001 .05 .03 .03 (years) GPA Fluency Task 22.04 1.96 3.61 28.58 *** 41.2 F 1.39 .838 * *** .04 10 29) min. (n = 15 30) 19.5 3.3 29.5 Task 136.6 215 11.1 tions 20.2 9.9 19.5 10.3 40.3 21.0 11.1 43.5 [[eta].sup.2] .03 .02 .07 .39 .33 21.3 11.7 41.8 min. (n 1 and Performance Rst Variables Word-processed Word-processed 10 = 19.0 3.1 27.4 137.7 216 10.5 min. 39) 19.0 3.2 29.1 199.7 261 11.5 (n 15 = 19.0 3.3 25.7 181.2 365 9.8 min. 42)

(years) GPA Fluency

Variable 1. 2. 3. 4. 5. 6. 7. Speed WJ-III Context. Context. Story TOWL Words Total Written Task

1 -.25 .52 .46 .46 ** .34 5 Task Fluency Conventions Language Construction Total Written above the the (n ** **

2 .31 * ** ** .24 * .18 6 .09 -.24 .16 -.05 -.88 .60 diagonal = 59); ** were those ** .13 .28 .15 * *

3 .05 .18 ** .52 .59 .83 .40 ** 7 .28 .00 .69 .45 .72 .61 observed below in (n <.05 < 3 .01 -** ** ** ** ** **

4 .11 .19 -** .63 .85 .54 .19 -** ** **

Fluency Conventions Language .57

-.33

Construction

Variable 1. 2. 3. 4. 5. 6. 7. Note. the Speed WJ-III Context. Context. Story TOWL Words Correlations written in * ** Table Two-Way Analyses Numbers Variable and of p

* .25 -.04 .29 -.09 .03 -in were

students diagonal =

hand

conditions

the

observed

word-processed p

condition

81).

Variance of Source

for

TOWL-3 Words

Essay

Scores

and

Written [[eta].sup.2.sub.p]

MS

Contextual Response Time Response Response Time Mode Contextual Mode .78 52.0 Mode 45.69 x .44 4.25 Time

Conventions .04 * 9.30 Language 5.57 .09 * .00 .04 .87 .00 .03 .01

Response Response Time Response TOWL Response Time Response Number Response Time Response * ** *** p < .001

Mode Story Mode

Time 55.08

9.23 Construction 3.77 .63

.99

.01 .03 .00

9.27 Mode Mode 21.16 Mode of Mode 93458.25 Mode x p p Time 325338.44 11.11 x Time x Time Total 195.12

.24 3.17 .34 .23 Words 38.67 ** 10.73

.02 Score

.00 .02 .00

.01 Written *** ** .05 <.01

.00 .22 .08 .07

90251.87 <

Questia, a part of Gale, Cengage Learning. www.questia.com

Reading and Responding to Student Writing: a Heuristic for Reflective Practice

by Richard E. Straub In "Composition Theory in the Eighties: Axiological Consensus and Paradigmatic Diversity," Richard Fulkerson laments the wide spread fragmentation and confusion in contemporary composition theory. He is particularly troubled by the ways we have come to tangle our talk about the goals of teaching writing-what he calls our different "philosophies" or "axiologies"-and the various means we have devised to achieve them. By 1990, he claims, compositionists had come to some consensus about their commitment to a rhetorical emphasis in writing instruction. At the same time, however, he sees "a growing complexity and conflict over means of reaching it" (410). The field, Fulkerson suggests, is in a knot, the means and ends of composition

caught in a giant snarl. I believe, with Fulkerson, that compositionists have come to some general agreement about the importance of audience and, more broadly, to appreciate the perspectives afforded by a social view of writing. I agree that there is a deep confusion about means and ends in the teaching of writing, and I suspect the problem has gotten worse, not better, in the past ten years, as we have expanded the range of discourses we study, learned more about the ways that context and community bear on reading and writing, gotten more interested in looking at students individually, at work in particular settings, and (regrettably) given less and less attention in our scholarship to questions of classroom practice. But I'm not so sure that we have come to a consensus about the goals of writing instruction or about the nature of good writing. Having spent some time in the 1980s sorting through the evaluative criteria espoused by the teachers in William Coles and James Vopat's What Makes Writing Good? for a dissertation and then studying the responding practices of recognized teachers for Twelve Readers Reading in the 1990s, I have come to see a great diversity both in our goals for teaching and in our views toward what makes writing good. In fact, I think a good bit of confusion has been caused by a general failure to define our values for writing and make them consistent with our goals, theories, and classroom practice. In this essay I would like to help teachers clarify their values for student writing, get their assumptions and classroom methods in dialogue with their goals, and then suggest a way to deal with this disciplinary confusion. To do so, I will: 1 construct a protocol of a teacher reading and responding to a student essay, to show the complexity we run into as teacher-- readers and all the things we have to sort through, focus on, and consider when we read and respond to student writing; 2 provide a map of the various concerns we can take up in our reading, and use it to examine reading and response in relation to the larger contexts of the writing class; 3 create and model a heuristic for us to reflect on our own reading and

responding practices in light of our philosophy, assumptions, and practice. Whereas Fulkerson takes up the problem of ends and means at the disciplinary level, as a problem for composition scholars, I am concerned first of all with the problem such theoretical conflicts have practically for individual teachers and actual classroom practice. Instead of looking to theory to address the problem from top down, I will approach the problem from bottom up by calling onteachers to work through the problem in terms of their own commitments and problems, with the help of theory. A PROTOCOL OF TEACHER RESPONSE: THE SCENARIO Imagine the following situation. It's halfway through a first-year writing course at a large state university, in a writing program staffed mainly with graduate teaching assistants who teach two courses a semester. Writing program administrators favor a process-based expressivist approach to teaching writing, employing no standard textbook and requiring students to write six essays, most of which are based on personal experience, most of them to be taken through a series of drafts and revisions. Teachers are encouraged to privilege the writer's individual voice, be liberal about the use of narrative, value details and examples from the student's personal experience, and encourage experimentation with organization and style. They are urged to emphasize the development of the writer over the quality of the written product and to respect the needs of the individual student. The teacher is an experienced teaching assistant who has taken the required training course for new teachers and, to satisfy an emerging interest in teaching, one other course in composition theory. She has found herself invigorated by the expressivist emphasis on personal voice and writing that is close to the self, and she has gone into her teaching with genuine excitement. At the same time, she has developed a real commitment to the importance of writing for readers. She sees herself as a handsoff teacher intent on making the classroom student-centered. A fairly successfulstudent with a history of doing well in her own writing, she does not

underestimate the importance of clarity and focus, an authoritative stance, and tight organization in academic writing. In fact, she wonders if more attention shouldn't be given to such concerns, as well as to grammar and mechanics, in first-year composition. Over the semester, she has tried, with less success than she had hoped for, to spend no more than 15-20 minutes reading and commenting on each studentpaper. Nevertheless, she plans on giving this batch of papers slightly fuller comments than previouspapers because she sees it as a crucial point in the class, a culmination of their work to this point. This student essay is the final draft of the fourth paper. The teacher has talked informally with students about their plans for the paper, and the students themselves, guided by a set of questions, have responded to earlier drafts of the paper. In class, students have been doing a lot of smallgroup workshops to help one another invent, draft, revise, and edit each of their papers, and once a week the teacher has selected a sample or two from the course reading anthology to analyze and discuss specific strategies of writing, for instance, how to recognize different voices, how to start a story or set up an introduction for an essay, and how to format and punctuate dialogue. Over the past two weeks, in anticipation of this latest paper, the class has been examining the ways writers like Toni Morrison and Annie Dillard use detail not only to describe something but also attach some value or emotional sense to the object. Here is the assignment students have been given: Most of you are off at school and in a new place, away from the people and settings you have become accustomed to and attached to. Similarly, those of you who are from Tallahassee have likely not had the time or the opportunity to visit some of the old places that are special to you. Choose some place, atmosphere, or situation that you miss from home-or, if you are at home, that you have not had the chance to experience for some time, and miss. Depict this scene, mood, or setting in a way that will allow your reader-someone who does not know about it-to see the significance it has for you. Remember that since your aim is to give readers a sense of place, you will do

well to use specific details. Across the course students have been urged to look at the assignments as a rough guide for what they might do in their own writing; if they prefer to pursue a somewhat different focus than is presented in the assignment, they are encouraged to go their individual route so long as they find a way to deal with the same general subject. Although this is the final draft and students are not required to take this paper through another rewrite, they have the option of rewriting and submitting, for a change in grade, one or two of their course papers by the end of the term. The writer is a confident, perhaps even a cocky, student who comes across (in the view of the teacher) as someone who thinks he is a better writer than he is. However, his confidence is by no means groundless. The teacher has been both disappointed by and taken with aspects of his previous writings. A PROTOCOL OF TEACHER RESPONSE: THE READING The teacher picks up the first paper. David. "The Four Seasons." All right, she thinks, might as well plunge right in. Let's see if he walks the walk this time. Gets more than a few good lines in here and there. Lives up to his own billing. Okay, 20 minutes per paper. Don't comment on everything. Focus. She starts to read.1 The Four Seasons I like Tallahassee very much. The heat and sunshine almost everyday makes each day very pleasant. He actually, she thinks, likes this relentless heat? He obviously hasn't been here in the summer. I intend to spend my next four and one half years here, but I miss my other home, Syracuse, New York. One thing that I truly miss about Syracuse is the four seasons. Each season is distinct and clear in its own way. I will do my best to describe each

season to you, but remember that my description cannot compare to experiencing each season for itself. An okay start, she thinks. A good thesis statement. But he's slow getting into what he has to say. Try to grab the reader's attention, she jots in the margin. And this escape hatch won't do. It's a cop out. Why the disclaimer? she writes, and reads on. In the Spring the ground is soft from the melting snow. You can feel the moist ground wanting to seep into your shoes. Good, she thinks, a really nice image. Evocative. She puts a check next to the line in the margin and writes, Good detail. I can feel the squishing. She's about to underline the "you" and write, Shift in point of view, but she doesn't want to take away from the praise she's just given-he needs the applause-and decides for the moment to let it go. She wonders whether to mark the capital S onSpring or tick-mark it and let him find it. She puts a slash through the letter, and continues reading. As the ground begins to dry, the trees begin to blossom and the faint smell of pollen lingers in the air. The flowers work their way out of the ground and bloom for another year. Good, some more vivid description. She thinks of the cherry trees that used to blossom each spring in her backyard. But it's pretty generic: a composite of a mythical spring day up north. He could get a lot more specific. Which trees? What kind of flowers? she writes, then picks up with her reading again. The familiar sound of geese is heard overhead as you look into the sky and see a "V"formation travelling north for the summer. A long winter's nap has ended for the bears, squirrels, rabbits and other hibernating animals. After they awake, their chattering conversations ramble through the forest. She's not sure she likes "'V' formation," and she's still not sure about this use of "you."

Why doesn't he use "I," she wonders, or stick with the third person? But she's clearly troubled by the string of general language, the stereotyped ideas: "The familiar sound of geese"? "A long winter's nap"? "Chattering conversations"? He's just blowing through these descriptions, not thinking about them. Coasting. She sharpens her pencil, drops her hand to the page, and next to the last two sentences writes, Really ? Squirrels and rabbits hibernating? Bears and rabbits chattering? He's getting careless here, she thinks, relying far too much on cliches about spring. But she catches herself. She wants to keep it constructive. She reads back over the sentences. Maybe it's just the way it sounds, a matter of voice or tone. Maybe he's thinking this is the way a description of spring is supposed to be: nice and light and homey. Hold off, she tells herself. Let's see what he does next. Not only do the animals come out of their shelter in the springtime, but also people. Many people have a tendency to "hole up" in the wintertime. All your neighbors, that you thought had died, open up their houses to allow the spring breeze to come along and carry away that musty air that built up during winter. You can hear voices and lawnmowers everywhere as people are outside doing their springtime yard work. Wives are planting new flowers while husbands are raking and mowing the lawn. Spring is the season of awakening where everything becomes refreshing. Following Spring is the season that most people look forward to, that is Summer. Summer is the time of the year when kids are everywhere, because school has been let out. You can hear their voices and giggles fill the atmosphere. People are always outside in the summertime because the sun beats down onto the earth and warms everything up. There are enormous amounts of families going to the beach for the weekend or going on vacation for a week. As you look down the road, you can see heat waves resting on the pavement. The foliage is green and spirits are high. There is a feeling of warmth amongst neighbors, friends, and family. The wheels, she thinks, have started to spin. Where to start? What to take up? There are some good moments. Here at the end: The heat resting on the pavement. Good

concrete detail, she writes next to it. It's an image that will lead your readers to imagine how hot it gets in Syracuse. This is the kind of detail I've been looking for. And up here at the start of the paragraph: People holing up, hibernating. That's different, she thinks. Something worth saying. Though the sentences are awkward. Good idea, she writes, but you can clean up the sentence structure. And the last two lines. The warmth outside connected with a warmth among neighbors. In contrast to the hibernating that he says people tend to do when it's cold. Nice. But she's concerned about all these muddy generalizations. People air out their houses and do yard work. All of them are wives or husbands, and they plant or they rake and mow. Don't, she finds herself writing in the margins, the women in your neighborhood ever cut the grass? Does Mr. Sworin ever plant the flowers? Spring, we find, is the season of awakening. Spring is followed by summer. Things warm up and people get back outside, go to the beach, and go on vacation. Not much here, she thinks. He's not telling us anything we don't already know. He's got to get more focused, get more specific detail. What happened to all that work we did last week? So much for Annie Dillard. Where is he in all this description? What significance do the seasons in Syracuse have for him? Did he read the assignment? She goes to the paragraph and underlines, in successive sentences, "kids," "voices," "people," "earth," "everything," "families," and "foliage." Don't attack, she thinks. Keep it upbeat. She puts a line down the length of the paragraph and writes: These are all general terms. Let's hear some details. He's done some sharp writing for short stretches in his earlier paperstoo, but he's not been able to sustain it. He can do better than this. She takes a breath, about to move on, but her eye is pulled back to the page. She writes Mechanical transition next to the first sentence of the paragraph about summer, squiggles a line under "enormous amounts of families," and puts her pencil down. He's stuck and time's flying. A quarter after. She's got to move on. She goes through the rest of the paper, making sure to acknowledge other sharp descriptions, pointing out places where generalizations steer him away from making the descriptions his, quickly noting places where a sentence might be smoothed out,

and snagging him at the end just as he is about to slip into the same escape hatch he devised in the introduction. This, she thinks, might be the one he goes back to for the last revision. She leans back and tries to settle on her overall sense of the paper. She gets something positive in mind for a start, and writes her end note to David: Pretty good, David. You have some fine descriptions of the four seasons in here. But I'm not sure how much you've experienced them or let us experience them as readers. You're relying too muchon cliches: geese flying north in the spring, summers at the beach, raking leaves in the fall, and sitting by the fire with that special someone in the winter. They're descriptions that we're all familiar with and they don't give any insight into how the four seasons are unique to you. You seem more intent on capturing some homey images of spring than in sharing with us your own views of the seasons in Syracuse. (I wonder: Are you trying to take on too much here with all four seasons?) Try focusing on particular scenes and specific events you have seen or, better yet, that have happened to you. Maybe it'd help to sketch out lists of things that you associate personally with these seasons, select the ones that are most vivid and representative, and work those into your essay. This might be one you choose to revise. She debates for a moment and writes "C" at the bottom of the page. Then she thinks of his sharp descriptions and, next to it, slashes a B. Nothing to write home about, but better than he's been doing. One down. MAPPING THE CRITERIA FOR READING STUDENT WRITING This is a full and fairly complex reading and response.' The teacher seems intent not simply onlooking at the writing summatively, to assess it, but using her response as an occasion for leading the writer back into revision and teaching him about writing. It may not be adequately focused. It may not be as coherent as it could be. It may be too evaluative in its posture, putting too great an emphasis on the teacher's role as critic and judge. And it may or may not be just what the studentneeds at this time. But it has a depth and ambitiousness that are worthy of praise. It is a rich pastiche of a reading,

marked by a criss-cross of impulses and purposes that we routinely experience as we read student writing, and the comments do a fairly good job of expressing theteacher's concerns to the student. In constructing this case, I want to suggest the rich and often bewildering complexity that is involved in reading, evaluating, and responding to student writing,3 and dramatize the need for finding some way to give order to the great variety of concerns we may take up as teachers of writing. I also want to capture something of the dynamic relationship between reading and response-that is, between reading and evaluating student texts, on the one hand, and communicating with the student about his writing, on the other. In doing so, I hope to explain why reading and response are so demanding and show how they might be made more manageable and used more effectively. I have selected this sample student writing, originally presented in Twelve Readers Reading, to tie this analysis to the ways that the wellrecognizedteacher-scholars featured in that study respond to student writing, extend the analysis Ronald Lunsford and I do there of the relationship between reading and response, and examine response more fully in relation to the larger classroom context and the teacher's approach to composition.4 Although I might have highlighted a reading from any of a number of pedagogical perspectives-- a rhetorical pedagogy, a social constructionist pedagogy, a post-process pedagogy, or some critical pedagogy-I have chosen an expressivist pedagogy because of its popularity in the classroom and its ability to accommodate a broad spectrum of features. In the section ahead, I'll chart the various qualities we might look for when we read student texts. Instead of an exhaustive list or a survey, I'd like to provide a map and a compass for reading, evaluating, and responding to student writing: some instruments that might help us as teachers figure out where we are and make a plan for where we want to go. TEXTUAL FEATURES What, then, can we learn by looking back on this response to "The Four Seasons" as a case study-one instance of a teacher (in this case, a hypothetical teacher) caught in the act, as it were, of reading student writing? This teacher takes up a variety of issues at a variety of levels. She looks at the text formally, in terms of what the writer has to say

and how he presents it on the page. She gives a fair amount of attention to local matters such as correctness, usage, sentence structure, and grammatical point of view. But she does not emphasize these superficial textual matters over more substantial matters of content and form. She considers the focus, scope, and overall shape of the essay; she considers David's use (or lack) of specific detail; and she considers the quality and substance of his descriptions: * An okay start, she thinks. A good thesis statement, but he's slow getting into what he has to say. Are you trying to take on too much here with all/our seasons? * People holing up, hibernating. That's different, she thinks. Something worth saying. * But she's clearly troubled by the string of general language, the stereotyped ideas: * "The familiar sound of geese?" "A long winter's nap?" "Chattering conversations?" These are the kinds of concerns-matters of correctness, style, organization, and ideaswe usually talk about when we talk about qualities of student writing. They deal with the text and the immediate construction of that text from the words on the page. Although they do not "reside" in the text, we talk about them, for practical purposes, as formal concerns, as features based in, and recoverable from, the words on the page.5 They may be plotted out as features of the written text, the meeting ground where, as Wolfgang Iser notes, the implied author and implied reader come together, in a carefully negotiated process, to make meaning.6 When monitoring our own criteria, it's important not only to determine the extent to which we privilege local matters versus global matters but to get at exactly how we understand these terms. What constitutes "substantive content"? What do we mean when we say a paper is "well organized"? What makes for our sense of an "informal voice" or an "effective style"? The map lays out general categories for reading and evaluating writing; we have to give them local habitation, not just a name. For the protocol teacher, it is clear, for instance, that writing is not merely a matter of

managing sentences and paragraphs or tending to the superficial properties of discourse; it's a matter of getting and shaping something to say. Her reading and response suggest that it is the content of writing-and the forming of that content-that makes writing good or bad, makes it worth or not worth reading. Further, she clearly privileges (at least for this type of writing) a particular kind of content: original writing that grows out of and reflects the student's own experience and perceptions. CONTEXTUAL CONCERNS The close attention this teacher gives to the text, especially the emphasis she places on content, tells us something important about her values as a writing teacher. But it only begins to account for the depth of her reading and the range of concerns that we might address when we read studentwriting. In fact, for every time this teacher focuses on some isolated quality closely tied to the text, she also brings to her reading some concern that goes beyond the text, beyond the words on the page, and invests it with the meanings, values, and perspectives of some broader context. Consider the following instances: * Try to grab the reader's attention. * Maybe he thinks this is the way a description of spring is supposed to be: nice and light and homey. * What happened to all that work we did last week? So much for Annie Dillard. * Really? Squirrels and rabbits hibernating? Bears and rabbits chattering? * Where is he in all this description? What significance do the seasons in Syracuse have for him? * He's just blowing through these descriptions, not thinking about them. Coasting. In each of these comments the teacher expands her orientation from the text to some

larger context of the writing.' The first response looks at the organization of the writing in terms of the rhetorical situation. The second considers the assumptions about genre that the student may be bringing to this kind of writing. The third looks at the text in relation to the work in class. The fourth looks at the writer's statements explicitly in terms of their accuracy against a larger social reality. The fifth and sixth view the writing in terms of the student behind the text, his experience and involvement. All of these contexts are brought into play in her reading. These "texts" that go with the text, these contexts, inform and influence how she views the paper, but they also mark the various areas of writing that she attends to in her reading and response. "Context" in this sense is not only a necessary condition for language and meaning, as Deborah Brandt argues; it is also a mental construct, a set of expectations that we use to interpret writing, analyze its strengths and weaknesses, and talk about how texts work.8 We "bring" certain sets of concerns-what Kenneth Burke would call certain "terministic screens"-to our reading even as the text elicits other sets of concerns, other contexts, as we read. By sorting out the various qualities and contexts brought into play in the protocol teacher's reading of "The Four Seasons" and charting them on a map, we might get a clearer feel for the geography of reading and the concerns that figure into our own ways of reading, evaluating, and responding to student writing. The Rhetorical Context The "text" beyond the text that has come to draw most of our attention is the rhetorical context-the circle of concerns that unites writer, text, and reader in some common purpose, in response to some situation or need. A student writes a paper, it is presumed, with some rhetorical purpose: to inform, explain, entertain, persuade, or generally achieve some effect on readers. The teacher is then called to read the paper in terms of how the text may affect readers and how well it achieves the writer's intentions. Teachers who attend to the rhetorical context most obviously focus on the way the text meets the demands of an audience. How well does David describe the seasons of Syracuse for those who are unfamiliar with them? What does he do to

engage readers in his discussion? They also focus on the purpose of the writing and assess how well the text is working to realize the writer's intentions. Does David want to show his audience how the seasons in Tallahassee are no match for the seasons in Syracuse? Will his descriptions enable readers to see the significance the seasons have for him?' Some teachers, looking to get a firmer grasp of what the student is trying to accomplish in a piece of writing, have come to call on the writer to identify his own intentions. In statements that accompany the paper, they have students define their aims, assess their progress, and raise questions for readers to consider. Such "process memos" (J. Sommers) or "reflective cover letters" (Yancey) provide an additional resource for reading student writing-and an additional set of concerns to address in our reading and response. Teachers read in terms of the rhetorical context, in addition, when they address the tone of the writing and the writer's voice, persona, and ethos-all of which deal somehow with the ways authors construct themselves as speakers and establish certain relationships with prospective readers. The protocol teacher actively invokes the rhetorical context in her reading of "The Four Seasons," viewing the writing as an instance of someone saying something to someone else, a writer addressing a reader for some purpose: * You have some fine descriptions of the four seasons in here. But I'm not sure how much you've experienced them or let us experience them as readers. * It's an image that will lead your readers to imagine how hot it gets in Syracuse. * Not much here, she thinks. He's not telling us anything we don't already know. * You seem more intent on capturing some homey images of spring than in sharing with us your own views of the seasons in Syracuse. * And this escape hatch won't do. It's a cop out. In the first three instances, the teacher considers the writing in terms of the audience,

in the last two, in terms of the writer's intentions and persona. The rhetorical context, then, may be plotted as a dynamic set of concerns encircling and interacting with the text. When we view student writing as a rhetorical activity, a meaningful act of communication, we read in a way that, according to Fulkerson, has come to dominate contemporary composition studies. We look to help the writer see where the writing achieves and does not yet achieve what he (presumably) set out to accomplish and thereby dramatize the need for revision. As Nancy Sommers notes, "As writers we need and want thoughtful commentary to show us when we have communicated our ideas and when not, raising questions from a reader's point of view that may not have occurred to us as writers. We want to know if our writing has communicated our intended meaning and, if not, what questions or discrepancies our reader sees that we, as writers, are blind to" (148). The Classroom Context When we read student writing, we are not, to be sure, just readers reading a piece of writing. We are teachers who read. We've decided on the kinds of writing that students are to do. To a large extent, we've determined the qualities that are to be valued in their writing. We may not warm up to the ubiquitous "What do you want?" but no question that students in a writing course ask makes more sense. In any class, even when we allow them to choose their own topics and genres, students are somehow left to give us what we call for. They are not writers so much as they are writers in apprenticeship: students. And as students they are imbricated in a set of power relations that always gives someone else a great deal of control over their writing. So when it comes time to read student writing, we invariably approach it as more than just readers. In addition to processing the text we also have to diagnose the strengths and weaknesses of the writing, evaluate thestudent's performance in terms of the work of the class, and use the writing to teach the studenthow to write better. As

Patricia Murray writes, "We need to recognize that we bring with us to a reading of student texts an inevitable evaluation that stems from a discourse community that more often than not is different from that of the students. We need to look at how our responses as teachers of composition affect what we advise our students..." (84). If we are readers, we are always, to one extent or another, teacher-readers, reading student writing in terms of the assignment, the genre of writing, the work of the course, and our teacherly expectations. The Assignment and Genre of the Writing Much of what we value in student writing is determined by what we ask students to do in the assignment and, by extension, by the kind of writing we've assigned. Assignments, in effect, perform the function of designating, among other things, the genre of writing to be pursued and the features of writing to be highlighted. They link the writing to a set of established forms and practices. Different assignments and different genres, of course, impose different demands on writers and elicit different expectations from readers. A personal narrative will lead us to look for different things than an informative report or an argumentative essay. A memoir will lead us to expect something different than a parable or a historical narrative. Assigning writing that is meant simply to display knowledge will lead us to look for something different than if we assigned writing to learn (Larson). Assigning freewriting exercises or other kinds of "low stakes assignments" will likely lead us to engagestudent writing differently than "high stakes assignments" (Elbow "High Stakes"). Holding students firmly to the demands of the assignment or, alternately, allowing them the freedom to depart from the assignment will establish different reading postures and lead us to look for different features when we read (Jolliffe). In the sample case, the writing assignment and the genre constraints of the personal descriptive essay clearly affect the protocol teacher's way of reading "The Four Seasons." She expects the writing to have an abundance of concrete detail that will enable readers to picture what the seasons are like in Syracuse and, further, that reflect the special meaning these seasons have for the writer. She also expects the writing not just to rehash general descriptions but to

tell readers something they probably do not know about the seasons in this northeastern city: * Spring, we find, is the season of awakening. Spring is followed by summer. Things warm up and people get back outside, go to the beach, and go on vacation. Not much here, she thinks. He's not telling us anything we don't already know. * What significance do the seasons in Syracuse have for him? Did he read the assignment? * They're descriptions that we're all familiar with and they don't give any insight into how the four seasons are unique to you .... Try focusing on particular scenes and specific events you have seen or, better yet, that have happened to you. Given the way contexts spill over into one another and collide, we should expect certain conflicts to come about. And they do. How much are we to view the writing in terms of the assignment, the purposes or "intentions" we designate for the writer to take up? How much are we to let students themselves decide the course of their writing and the genres they employ?'O In reading "The Four Seasons," we have to decide how much we are going to allow, or even encourage, David to shape the writing in his own ways. Are we going to approve his commonplace descriptions of the seasons? Are we going to call him on the fact that he doesn't adequately treat the significance that the Syracuse seasons have for him? Are we going to contest his mechanical, self-conscious opening paragraph as inappropriate for the kind of writing we have assigned? In the protocol, the teacherseems inclined to allow the student his intentions-up to a point. She is willing to have him go with his broad treatment of the four seasons. But she is not so interested in having him pursue his own intentions that she is willing to forego her interest in seeing more concrete descriptions, seeing him adopt a new way of getting in and out of the essay, and seeing more of his experiences and views of the seasons in Syracuse. Her way of reading the writing is clearly influenced by the assignment and what she expects this kind of writing to do.

The Work in Class Beyond reading within the rhetorical context, beyond reading in light of the assignment or a particular genre, we must somehow read student texts with an eye on the larger pedagogical setting. Invariably, the work we do in class and the expectations we bring to student texts conspire to make us strong readers, determined to privilege certain features of discourse. If we have been emphasizing introductions and conclusions, we tend to look for how students begin and end their own writing. If we have spent several class periods introducing conventions of reasoned evidence in argumentative writing, we usually look for such features in student papers. The goals we set out to achieve in the course also influence our reading and response. If we are determined to lead students to come up with polished written products, we will be more attentive to matters of unity, coherence, clarity, and correctness. If we are intent on giving students practice in their writing processes, we will emphasize certain strategies of writing and probably place less emphasis ontextual completeness. If we are committed to preparing students for their future courses, we will be more attuned to the conventions that are practiced in academic discourse. The protocol response to "The Four Seasons" illustrates how elements from the classroom context routinely become the focus of our reading. The teacher, struck by David's generic descriptions, thinks that all the work they did with detail over the past couple of weeks has been lost on him. The lessonon Annie Dillard's meticulous observation is called to mind and prompts an emotional response that leads her to look all the more expectantly for signs that David has or hasn't taken it to heart. A moment later, she is taken aback enough by the writing that she wonders if he even read the assignment. Soon after, she is taken by his image of the heat rising off the pavement. "This, " she tells him, is the kind of detail I've been looking for. " Optimally, our reading is an extension of the larger classroom conversation, our emphases as teachers reflecting the instruction and goals of the course (Knoblauch and Brannon; O'Neill and Mathison Fife; Straub, "Classroom Instruction").

These contexts-the assignment, the work in class, and the teacher's emphases and expectations-form some of the most powerful influences in our reading of student writing. They may be plotted as part of the larger classroom contexts that inform the ways we read. These contexts are not neat or mutually exclusive. One context spills over into other contexts in the way that bays, inland waterways, and oceans spill into one another: although they are at some point, from some perspective, clearly distinguishable as entities on their own, with their own distinct properties, they flow into each other and merge. A single comment, in fact, may invoke several contexts simultaneously, for example, the rhetorical context and the classroom context, as in the following response: "Good sharp detail [the text]. It's the kind of `concrete naming' [classroom context] that will help readers picture the scene [rhetorical situation]." It's not important to know just when one gives way to another. What's important is that, when we read, we immerse ourselves in these waters. The Individual Student The revived interest in the teaching of writing in the 1960s and 1970s, the New Rhetoric that would eventuate in contemporary composition studies, was powered by two major concerns: a concern for viewing all writing, even student writing, as rhetorical; and a concern for invention and, with it, an interest in composing processes. Both concerns would lead compositionists to an inevitable interest in the student writer as an individual: her experiences, her ideas, her composing methods, and struggles as a learning writer. If writing is, more than clearly and correctly laying out an argument, a way of saying something to someone else, student writers must be guided to find something worth saying. If they are to find something to say, they must learn to investigate their experiences. If they are to form these ideas more aptly in texts, they must be given practice in drafting and revision. Since students are at different stages of development as writers, coming from sometimes vastly different backgrounds and having very different discursive

experiences, educators have looked more and more to gear instruction to the individual student. The field has devoted so much attention to students as individuals that "the student" has become a context of its own in the teaching of writing-not unlike the study of the author's life and work developed into a context of its own in traditional literary scholarship. The next three contexts of student writing bring this "individualstudent" into the picture and prompt us to view the writing in light of her identity and experience. The Student's Writing Process As teachers, we are in the rather special position of seeing not just finished texts, but writing on the way to realization. We get to see the writing as it unfolds, help students as they draft and revise, and teach the act of composing. Our ways of reading student texts are dramatically influenced by the extent to which we view writing as a process and allow the drafting to determine what we look for and consequently what we see. When we read with such an orientation, we look at a piece of writing as part of a larger process, viewing early drafts with an eye toward revision and later drafts in relation to what the writer has come up with before. We generally read first for overall content and organization, and deal with matters of paragraph structure, sentence style, and word choice only when the larger conceptual matters have been worked through (though not necessarily worked out) or as the final draft approaches. We sometimes even look at final drafts, not summatively as finished products, but in terms of how they may be revised. Viewing writing as a process, then, colors the lenses that we bring to student writing. Once we decide whether to encourage David to continue work on "The Four Seasons" or call the writing to a halt, we make one giant step in determining what we look for and how we read. As soon as the protocol teacher, early in her response, asks the writer "which trees? What kind of flowers?" she is already orienting her responses toward revision. She's not looking back at writing that is complete; she's implicitly looking forward to writing that is to be done, considering the writing in terms of how

it might be improved. She also anticipates revision when, on noting David's overreliance ongeneralizations in the fourth paragraph, she remarks: "Let's hear some details. " Her commentarybecomes explicitly concerned with guiding the student toward revision half way through her end comment: * Try focusing on particular scenes and specific events you have seen or; better yet, that have happened to you. Maybe it'd help to sketch out a few lists of things that you associate personally with these seasons, select the ones that are most vivid and representative, and work those into your essay. This might be one you choose to revise. By taking a process approach to our reading and viewing student writing as work in progress, we don't have to deal with everything at once. We can slow down, focus on certain issues at certain times, and deal more fully with the concerns we do take up (Krest; Sommers; Straub, "TeacherResponse"). We can guide students more purposefully through their work as writers and use our responses to teach, not just to critique or grade. The Student's Work in the Course The ways we read student writing are also influenced by how we envision the "text" of the course. How much are we going to look at the student's writings discretely, as separate projects? How much are we going to look at the writing in terms of the student's ongoing work in the course in light of his other writings, the strategies he has been working on, and his development as a writer? How does this paper stack up against the other papers the student has written? Is there some quality of writing that he's been working on that should be addressed? Is there any evidence of progress with some composing strategy? In the sample case, is David's voice more distinctive than in his earlierpapers? Are his sharp, imagistic descriptions in this paper notably better than what he has come up with before? These prior texts form yet another context, another "text beyond the text," that we may attend to in our reading and response. The

protocol teacher makes several evaluations that indicate that she takes an expansive view of "the text." She looks at "The Four Seasons" as a discrete paper and as a work in progress, yet she also views the writing in terms of the student's overall work in the course. She has evidently seen some potential in David's writing and has come to expect a certain level of work as a result: "He's done some sharp writing for short stretches in his earlier papers too, but he's not been able to sustain it. He can do better than this." Paying attention to the student's evolving text across the course is one of the surest ways of individualizing instruction and using our comments to meet the student where he is in his work as a learning writer (Krest; Onore; Phelps; Straub, "Classroom Instruction"). The Student Behind the Text Our evaluation of student writing is also shaped by our "reading" of the student-our sense of thestudent behind the text. Our reading of a paper like "The Four Seasons" may well be affected by our sense of David as a writer and as a student in the class: his attitudes, efforts, and capabilities. We may also view the words on the page in light of our ideas about David's experience and identity, whether as an independent agent or, increasingly now, as a socially constructed subject, defined (among other things) by race, gender, class, sexual preference, and religion. How fully has he captured the kind of winters he has experienced? What personal associations does he have of spring in Syracuse? How may his perceptions of Syracuse and the seasons be seen as a function of his race, gender, or class? When we read student writing, more often than not we read the text and the student or person behind the text. More than evaluate, rank, and judge, we diagnose, coach, guide, explain, and teach. Whom we are coaching, guiding, and teaching is as important as what we are teaching. If the student has admittedly had a struggle writing about a subject that is too close to him, we might steer him toward a different topic. If we know he lacks confidence or has a difficult history with writing, we might be more inclined to look for successes to build on. If he has struggled before and is now making progress, we might look at his writing more forgivingly and play up his accomplishments in our response. If he has

done solid work on the first three papers, we might well expect more from him on the fourth. "Students use and learn from our comments when we monitor their writing rather than simply evaluate their final papers," Margie Krest says. "At each point in our monitoring we must strive to be sensitive to our students as writers so that our comments foster positive attitudes about writing" (38-39). The protocol teacher is sensitive to the writer and the person behind this text. Even before she begins her reading, she has clicked in to certain ways of seeing the writing in terms of her take onDavid as a student writer. She picks up the first paper, pauses when she sees that it is David's, decides nevertheless to forge ahead, and quips to herself: "Let's see if he walks the walk this time." Not much later, she reads the text against the background of his attitudes toward writing, stroking his confidence even as she uses this confidence to push him to work harder: "she doesn't want to take away from the praise she's just given-he needs the applause-and decides for the moment to let it go." She also makes some inferences from the text, for better or worse, about David's motives as a writer: * He's getting careless here, she thinks, relying far too much on cliches about spring. * He's just blowing through these descriptions, not thinking about them. Coasting. But she's careful not to confront the student with such presumptions about his motives and processes, and reminds herself to keep the commentary constructive. Not surprisingly, when she looks at the writing in terms of David's personal experience she sees this experience in psychological terms, according to his own individual perspectives, not in terms of any social identities. In addition to looking at the text in terms of the larger rhetorical and pedagogical context, then, we may view it in terms of some image of the student behind the text, his particular experiences and views, building this image on information we get directly from our interactions with the student or indirectly from the student's text. In doing so, we read-not just the writer's texts-but the writer. These writerly concerns

may be plotted as a general context that, like the classroom context, provides a broad interpretive frame for reading the text and its immediate rhetorical context. Classroom writing never comes as a text on its own. It always implies a rhetorical context, a pedagogical context, and a particular student writer. These contexts routinely come into play in our reading, as a matter of course overlapping, complementing, and competing with one another. Such interanimation of contexts is one of the things that makes reading so resourceful. It is also one of the things that makes reading student writing so complex and demanding. Problems occur when we do not acknowledge these various contexts as focuses for our reading. In pursuing other treasures, we read right over them. Other problems occur when the contexts come into play (as they will, spontaneously) and we do not know how to attend to them. We take up one, we take up another, as we will, but with no clear focus or purpose. Still other problems occur when the contexts we do address do not match, in kind or degree, the emphases of our classroom instruction. We spend two weeks considering how writers should adapt writing to the audience, and then when we read we focus almost exclusively on organization and detail-or we read from our own specialized position as writing teachers and do not reach beyond this role to give the writer a sense of how the writing might be read by prospective members of its target audience. As teacher-readers, we have to develop a repertoire of concerns and a knack for reading student writing in ways that will complement and reinforce the work we are doing in class. To do so, we have to reflect on our day-to-day teaching, critically examine our instruction in terms of our goals, articulate the contexts and concerns that are most important to the course, and focus on them when we read our students' writing. As we ask students to keep certain key features of discourse in mind as they write, we should keep certain features of writing-and certain contexts-in mind as we read and respond. OTHER CONTEXTS THAT INFLUENCE OUR READING The text, the rhetorical context, the classroom context, and the student context are the

most obvious frames that inform our reading and establish the features we look for when we readstudent writing. But there are other contexts that are less obvious but often no less powerful in the effects they have on our reading: our approach to teaching writing, our interpretive theories, the academic setting, the writing program, the larger institutional interests, our own prior experiences with teacher response, the immediate circumstances of our reading. In fact, these concerns may be so ingrained that it may be difficult to see them as sets of expectations, as contexts and constraints, that influence (and make possible) our reading.11 We don't focus on features associated with these contexts in student texts. We rarely acknowledge them as we read, and we almost never make reference to them in our comments. Nevertheless, they are so imbricated in what we do that, even though they usually remain tacit, they shape what we look for and hence what we see. These concerns, which are brought to the text in the form of teacher expectations, may be plotted as a backdrop for the more immediate contexts that inform our reading, illustrating the full complexity of the interests and influences that define us as teacher-readers. Approach to Teaching Writing In "Composition Theory in the Eighties," Fulkerson looks to sort out the various components that make up a theory of composition, or what I am calling here an approach to teaching writing.12 A full theory of composition, he posits, must have at least four components, each of which, to avoid confusion, must be kept distinct: * a "philosophy" made up of one's goals for instruction and a set of criteria for good writing; * a conception of the writing processes students might employ in order to reach these goals; * a set of classroom practices that are designed to bring about these goals;

* an epistemology, a conception about what counts for knowledge. Each of these theoretical components has some real effect on how we read and what we read for. There is no theoretical commitment, no matter how explicit or tacit it is, that more profoundly influences our ways of reading than our goals for instruction. Is the writing class a service course? Does it exist primarily to clean up student writing and perform a gatekeeping function for the university? Is it to give students practice in their composing processes and help them become more confident writers? Is it to help students learn to write better, to use writing to learn to think more independently and more critically, and see what value writing might have for them in their lives? Is it designed to prepare students for the writing they will be expected to do in other courses-or somehow prepare them for the writing they may have to do when they take on a job? Is it to help students develop powers of critical thinking, introduce them to the latest electronic media, engage them in the examination of their culture, or expose them to other cultures? These questions about the aims of composition turn on the broader issue of how we envision the larger goals of college education. Is an undergraduate degree meant to secure students a good job? To broaden their ideas about the world and understand their place in it? To lead them to become good citizens? To mobilize them to resist oppressive social structures? These matters are often decided for us, at least ostensibly, by the writing program, by the department, by the college, or by the teaching approach we adopt. But they are emphasized or deemphasized according to the decisions and actions of the individual teacher. How we envision these larger goals of instruction will influence how we read and what we look for when we read and evaluate student writing. Our reading will also be influenced by how we view knowledge and envision the nature of the writer's content and meanings (Anson, "Response Styles"). Are they to come primarily from some fixed common knowledge? Are they to be appropriated from the statements of some prior discourse? Are they to come from the writer's own

circle of experience-- his own views and personal knowledge? The content of student writing depends to a large extent on the assignment and the genre of writing, but it also depends on the writer's specific subject matter. The more we lean toward an expressivist view of writing, the more we'll value the writer's discovery of ideas based on, and giving shape to, his own experience. The more we lean toward a social view of writing, especially some recent social constructionist views, the more we'll privilege subject matter outside the self and look at invention as a matter of discerning what has already been said and negotiating various voices and perspectives in prior texts (Bartholomae; Bizzell; Lu). Our ways of reading are also affected by our pedagogical theories. If we see writing instruction as a matter of simply giving students plenty of time to write in different genres and along the way get feedback from others in the class, we will be more inclined than we otherwise might to give students a lot of room to make their own decisions as writers (Elbow, Writing Without Teachers; Moffett; Murray). Instead of specifying changes we'd like to see made, we will provide reader responses, make suggestions or frame our comments as questions for the student to take up as he will, in his own way. If we believe that class time is best used to provide students direct instruction in writing or engage them in hands-on practice with certain composing activities like observing, planning, defining, and arranging, we will be more inclined to look for instances where students make, or might make, use of these strategies (Hillocks, Berthoff). If, in an effort to emphasize the social dimension of writing, we look to turn our classroom into a microcosm of a discourse community and establish students themselves as the sole audience of all course writing-- and perhaps even as arbiters of what is to count as good writing-we might find ourselves in the curious position of having to resist many of the comments that come naturally to us as teachers (Cooper). We might not have recourse, for instance, to question a text that lacks unity, relies too much on cliches, or is utterly informal. The protocol teacher appears to adopt an approach that is largely expressivist. She assumes an expressivist axiology, a subjective epistemology, an expressivist view of

the composing process, and a mixed pedagogy. She posits expressivist goals inasmuch as she looks to foster the development of the student's own thoughts, the development of his own voice, and his development both as a writer and as an individual. But she seems driven not simply by an interest in the student's self-expression but also by a keen interest in writing for readers, the defining trait of a rhetorical axiology. She clearly assumes that knowledge and truth are located in personal experience and perception. She enacts a vision of the composing process as a fairly individualistic activity involving a series of recursive activities of planning, drafting, and revising. She seems to shuffle between ateacher-centered and a student-centered pedagogy. She relies on the presentational mode of analyzing model texts and makes frequent use of writing workshops, including collaborative peer-response workshops. At times, she seems intent on directing David to make specific changes. At other times, she seems content to raise an issue for him to take up as he chooses. These theoretical commitments find their way into this teacher's reading of "The Four Seasons." It is not enough that David describe the seasons, he must describe them through his own experience. He must do this, as the assignment reminds him, in ways that reveal the significance the Syracuse seasons have for him and, what's more, that show how these descriptions are unique to him. She focuses primarily on the content of the writing, but she also deals a good bit with correctness, sentence structure, and the formal shaping of the writing, including the use of such formal devices as thesis statements and transitions. For all her attention to matters beyond the text, she still sees the writing primarily in terms of a cognitive view toward text making.13 If her work with David's essay here is typical, this teacher, it might be said, is fairly formalistic and rhetorical in her ways of reading student texts. But ultimately, with all the evaluative remarks she makes, both negative and positive, she seems to assume a fairly authoritative stance in her encounter with David's writing. Although a theory of composition more often than not remains tacit, the teacher typically acting intuitively according to quiet promptings, it exerts a

powerful influence over our ways of reading and responding to studentwriting. Interpretive Theory Our ways of reading are often colored by our literary theory (or theories). The strategies that we bring to literature spill over into our reading of student writing, leading us to look at texts in certain ways, with certain emphases, for particular kinds of meanings. A lot of us, steeped in Modernism and raised on the legacy of New Criticism, look to perform close readings of the text, appreciating statements that "show" rather than "tell," seeking out patterns of imagery and theme, and demanding coherence. Many of us, informed by reader-response criticism and attuned to the process of interpretation, look to dramatize our ways of negotiating meaning and point to productive or troublesome gaps in student writing. Others of us don the lenses of structuralism and attend to various conventions of discourse or look through a Bakhtinian lens for the voices and intertextual struggles at work in a text. More recently, under the pervasive influence of poststructural literary theory, we may find ourselves looking, among other things, to reveal the conflicting voices, hidden ideologies, and unresolved tensions that are inscribed in the words of our students' texts. Although evidence of such theoretical influence is not always readily noticeable, the pull they may exert over our reading is no less forceful. An even more forceful, if only slightly more obvious, influence in our ways of reading student writing is exerted by the roles we bring to the text as readers, among them: 1 Teacher-reader as evaluator (focused on assessment) 2 Teacher-reader as educator (focused on instruction) 3 Teacher-reader as universal audience (representing readers in general) 4 Teacher-reader as target audience (representing a specific group of readers)

5 Teacher-reader as a common reader (presenting herself as an individual reader) 6 Teacher-reader as implied reader (assuming the role suggested by the text) The traditional default mode for teachers has been to read the text new critically as an evaluator, or in what Alan Purves calls the role of judge, with the teacher-reader looking primarily to assess the strengths and weaknesses from a privileged position above the rhetorical context, typically without any reference to the audience. A second customary role for teachers to assume is that of instructor, akin to what Purves calls education's version of the diagnostician/therapist. This teacher-reader also establishes a certain distance from the text and positions herself outside the immediate rhetorical context. But instead of looking to critique the writing or fix it, she looks to read the writing from the perspective of someone primarily interested in helping or improving the writer. Increasingly, however, as Fulkerson notes in his survey, contemporary teachers insinuate themselves into the rhetorical situation and assume the role of some prospective audience. But here again there is more than a little complexity. The teacher, we have come to see, may take on several different roles as "a reader." She may assume the role of an abstract, universal audience, and speak for how the writing works for "the reader" or "readers," without any specification of who these readers might be. She may act as a representative of the paper's target audience, whether it is one specified by the assignment or somehow indicated by the student writer himself-that is, in the role of some actual audience whom the writer is trying to address, what Ede and Lunsford call an "audience addressed." She may read the writing as an individual, common reader, showing students how she experiences, understands, and reacts to their writing, whether she highlights her own subjectivity or masks it in favor of some less personal, less partial perspective (Elbow Writing Without Teachers; Gibson; Macrorie; Moffett). Or she may resist bringing a predetermined role to the text and look instead to pick up on the readerly role that the text implicitly invites readers to adopt what Ede and Lunsford call an "audience

invoked." A teacher looking to discern the reader implied in much of "The Four Seasons," for instance, might find the easy-going, accepting audience of popular magazines or local newspaper columns. These roles are not distinct or airtight. A teacher does not simply adopt one to the exclusion of the others. A given teacher may emphasize one role or another, but she will likely assume different roles in a single reading-and perhaps all of them over the course of a full set of student papers. The object, again, is to match our reading roles with our instruction. The protocol teacher adopts a number of these reader roles in her response. She is evidently at ease in the role of the teacher as evaluator. Most of her interactions with the text, in fact, are made from this perspective, among them: * An okay start, she thinks. A good thesis. But he's slow getting into what he has to say. * He's just blowing through these descriptions, not thinking about them. Coasting. She very frequently casts herself in the role of teacher as educator: Don't attack, she thinks. * Keep it upbeat. * He's done some sharp writing for short stretches in his earlier papers too, but he's not been able to sustain it. He can do better than this. But she also assumes various roles as a teacher-reader. Try to grab the reader's attention, she tells David at the start of the paper, invoking the needs of readers in general. Much more often, though, she assumes the role of the target audience: It's an image that will lead your readers to imagine how hot it gets in Syracuse. * These are all general terms. Let's hear some details.

From time to time the protocol teacher even takes on the role of a subjective reader. "Good," she thinks as she reads about the moist ground wanting to seep into your shoes, "a really nice image," and then tells David, " I can feel the squishing. Moments later, reading the description of the trees beginning to blossom and the faint smell of pollen lingering in the air, she thinks of the cherry trees that used to grow in her yard. Both reactions register impressions that the writing has elicited from her as a real reader. This is the kind of response Peter Elbow extols when he speaks about providing students with "movies of the reader's mind," that is, the ways the teacher processes the text moment-to-moment as an everyday reader ("Options"). Some of us may like to think that we leave our subjectivity behind when we teach a class or read astudent paper, but nothing could be further from the truth. Our work as teachers, like our students' work as students, is saturated with our own experiences, interests, and values. Response and judgment, Robert Schwegler reminds us, "are shaped by the reader's knowledge; ideology (personal, social, cultural); social situation; and interpretive strategies (often shaped themselves by social and cultural ideology and class) as well as by textual features and content" (212). We readily challenge sexist remarks about men mowing and raking while women are left to plant flowers. We overlook stylistic flaws when a student writes about the sudden death of her father. We are easily enamored by an essay about Yellowstone National Park if we have just made our first trip out there ourselves. No matter which roles we adopt as teacher-readers, no matter how much we are wont to admit or try to suspend our own views, two things are clear: (1) what we read for and how we read are invariably affected by our own assumptions, interests, and values, and (2) just how much we admit-or try to suspend-our personal values and beliefs in our reading greatly influences how we read our students' writing. Any monitoring of what we look for when we read student writing must involve an examination of our roles as teacher-readers (Cowan; P. Murray; Purves; Straub and Lunsford), the ways our values and experiences may come to bear on our ways of reading (Fish; Freund; Tompkins), and, as some suggest, even our personalities (Thompson).

The Academic Setting Regardless of the type of writing we pursue or the kind of community we look to create in class, we are always somehow influenced by the fact that we are dealing with writing in an academic setting. We might, for instance, place greater value than we otherwise would on thoughtfulness and reasoned evidence. We might privilege analysis over emotion or seriousness over casualness and humor. Lil Brannon and C. H. Knoblauch have shown how readers, when they invoke the perspective of the academy, foist on student texts standards and values that are peculiarly academic-and that wouldn't be stressed in other settings. They cleverly use the example of how our bias against allowing students to rely on emotional argument might well be different if we were acting in another discourse community, say, in a court room and not a classroom. Similarly, Joseph Williams has shown how teachers' harsh, bionic eyes for error in student writing are made all too human when they don the lenses of nonteachers and read other kinds of writing. As members of the academic community, we are primed to look for a certain formal tone in student writing, pay special attention to correctness, and look for other conventions and discursive practicesthat are valued in school. The protocol teacher's response shows the pressures of these constraints. In spite of her expressed priorities, even as she sees that the draft in front of her is not yet ready for word-by-word editing, she has trouble resisting taking note of David's problems with transitions and inappropriate usage. She assumes that writers should have something fresh and distinctive to say-or at least have something to add to the communal storehouse of knowledge. These concerns are fundamental to our expectations as writing teachers. They may be so familiar and obvious, in fact, that it may be difficult to see them as expectations at all. But when our expectations as teachers become so rote and generalized that we fail to recognize the particular context of the writing or become so strong that they override the student's choices and purposes, we impose our own idealized text on students, appropriate student writing, and ultimately discourage students from giving themselves to their work as writers (N. Sommers; Brannon and

Knoblauch). Writing Program Constraints, Institutional Constraints, and Grades There are still other contexts that quietly but routinely shape the ways we read student texts. Some writing programs exert an obvious influence over the teaching staff, requiring the use of certain textbooks, emphasizing process or product, supporting or not supporting collaborative learning, pushing one kind of writing or another, and, in doing so, placing a premium on some features and contexts of writing and downplaying others. The protocol teacher's decision to resist comments about wording and transitions or to hold back on critical commentary could each derive from the influence of the writing program. The larger institution itself may also exert a palpable influence. Some institutions create an atmosphere where students are seen as mere clients or consumers, and teaching is seen as just another task to check off a much larger schedule. Other institutions create a more positive environment, where students and teaching are the heart of the matter. These overarching views toward instruction surely have an impact on how teachers see themselves as writing teachers, how they see their classes (and how much they put into them), and how they read their students' writing. One of the most obvious and powerful institutional constraints that affect our reading of student writing is the requirement to give grades-an issue that is always bubbling at the surface of new teachers' concerns and simmering just below the surface when any teacher reads student writing. I'm not interested here in the question of how we decide on grades-or if we should put grades onindividual papers. I'm interested in noting how putting grades on student papers affects the ways we read and respond to student writing-sometimes significantly. Consider the case of the protocolteacher. She's reading what's intended as a final draft of "The Four Seasons." She sees, as she's reading, that it is not realized enough to warrant closing the project down and treating it as a finished essay. She responds to the paper as if it will be revised. When she decides to slash her original C into a cross between a C and a B, I would suggest, she

makes a choice that affects her final take on the writing and that affects her response. With this generous grade, she allows David-and herself-- more room to rest content with the paper. She can back away from it cleanly and move on, giving the student a wide berth in whether or not to go back to it and revise. But if instead, let's say, she had opted to give more play to the skeptical side of her reading and gave the paper a C or C-, her posture would change, her reading would change, and her response would almost certainly change along with them. Her response would likely get more evaluative, more defensive.14 Putting grades on student papers-or refraining from putting grades on papers, for that matter-invariably marks our ways of reading and responding to student texts (Elbow, "Ranking"). It is another powerful constraint to consider when we consider our work as readers of student writing. The Teacher's Past Experience with Teacher Response It is a basic principle of response, especially when we first start out as teachers: we do unto others what has been done unto us. We make the kinds of comments we remember getting on our own writing: they are what we are familiar with, what we have come to associate with response. So our first ways of reading, our default response style, is usually some version of traditional teacherresponse: rather abbreviated if not terse in tone and style, and usually evaluative and controlling. Whatever stylistic features we develop later are often echoes of comments that have come from teachers in our past who have made an impression. How often have I started out an end comment with the kind of high praise I remember once getting from Joan Bobbitt on a graduate paper I wrote on Thomas Hardy: This is a herculean effort! How often have I acknowledged a student's taking on a formidable task the way I remember Jim Phelan acknowledging my own attempt at the formidable task of coming to terms with his first book: Although I can't say that I'm finally persuaded by your argument, I admire and appreciate the careful reading you give this work. How often have I taken heart at feeling pressed to be upfront and critical in a comment, knowing that I could try to frame the criticism in a way that was both sharp and helpful, as Barnett Gutenberg did a long time ago for me when he wrote on an

undergraduate paper I'd written about Jay Gatsby and Thomas Sutpen: Writing is meant to express, not impress. Too often you go for the flashy word instead of the right word. Try to say more simply and clearly what you have to say, in language that you can control. How much has my own upbeat, encouraging response style evolved as a reaction to the burning humility and anger I have felt on receiving comments that were insensitive and (as Nancy Sommers so aptly puts it) mean-spirited? I am convinced that the voice and fullness of my own comments as ateacher are a direct result of the long line of teachers I've had, from high school through grad school, who respectfully and generously gave themselves to nurturing my work through their comments. And I believe all of us, as Janet Emig and Robert Parker suggest, have such influences from our pasts at work in the ways we read and respond to our own students' writing. The Immediate Circumstances of Reading and Responding The pull of all these contexts may take effect only as another context-our immediate circumstances as we read-allows them to take hold. The number of papers we have, the mood we're in, the order in which we read them, the number of papers we've already worked through, the number left to go, the other work we have to do before the next class, all the things that come to mind, fill our hearts, and would pull us away from our reading. And, of course, time. How much needs to be said about how time itself affects-or downright determines-how we read and respond to our students' writing? The way our readings get fuzzier, our comments shorter and grumpier, as we spend more and more time on a set of papers and get more tired, less patient, and more than a bit less forgiving? The way our comments get astonishingly more efficient as class time rapidly approaches-and remarkably more benign if we hand papers back a week late or a day or two after all the others? Will the protocol teacher have the time to respond this fully to a Glassful of twenty-four students? Will she be as involved and clear-headed on the twelfth or twenty-fourth paper as she is with David's, her first? Will the rote introduction in this paper trigger reactions to other such introductions in laterpapers? Will David's sharp images and easy cliches lead her to be less accepting

of another student's cliches? Just how much sway these immediate circumstances exert on our reading will depend, of course, on how clearly we are able to keep in mind our vision of what we want to accomplish. The more contexts that we bring or that enter into our reading, the fuller and more textured our response. But the more contexts we bring into play-or, worse, that come into play unbidden-the greater the potential for distractions or confusion. One of the main difficulties in learning to read and respond to student writing is just this: we have to decide, from a dazzling array of options, on the terms that we are to read by and the concerns we will most value in our reading. Further, we have to figure out how to match these criteria to our individual teaching styles and our particular classroom goals. And, further still, we have to discover what is most important on this paper, for this student, at this time. And, yes, always, always, working against time. It's no wonder, then, why we learn quickly to put our best battle faces on when we bring home a new batch ofpapers. We know it can easily turn into a struggle. ANALYSIS OF THE PROTOCOL TEACHER'S READING Fulkerson grants a major status to the values that teachers bring to their reading of writing-right up next to their goals for instruction. In fact, he says the most obvious inconsistency within a giventeacher's theory of composition "is subscribing to one axiology and then evaluating from another" (422). He does little to define the nature of this crucial component of composition theory or elucidate its importance-they are not central to his project. It is my purpose to do both because they are central to mine. A teacher's concept of what makes writing good is the most sensitive and vulnerable contact zone between her theory and her classroom practice-and the best place to begin to reflect on one's work as a teacher. In the following section, I will analyze the protocol teacher's reading of "The Four Seasons" and consider (as best as I can given the limitations of this scenario) how the values she enacts in her reading square with her commitments, theories, and practice. In doing so, I'd like to help teachers see how they might use this map of criteria to examine their own ways of reading, clarify their

own values and goals, and reconcile their own theories and practice. The protocol teacher does well to reflect her theories and values in her reading of David's essay. We would expect such a teacher to emphasize content over organization, style, and correctness, and she does. Fully half of the fifty interpretations she makes across this reading protocol focus on the writer's ideas and descriptions. Fittingly, almost all of them are concerned with how well David makes these descriptions of the seasons vivid and personal, putting into practice her subjectivist concern for the writer making his own meanings. Another ten interpretations focus on, or look at some textual quality in terms of, the individual student writer: his experience, his involvement in the writing, and his earlier work in class. Such routine evocation of the "student" context puts into practice her emphasis on looking to improve the student writer over simply improving the writing. Six additional interpretations view the writing in terms of the rhetorical situation. Only nine of the matters she takes up in her reading deal with local matters, and over half of all her interpretations invoke some larger context of writing. Identifying the concerns a teacher takes up is only a start. Other questions also have to be considered: How does she define her key criteria? What, in her view, constitutes good content? Has she been emphasizing such a view in her day-to-day instruction? What sort of voice does she most value? Is such a valuing consistent with this type of writing and her pedagogy? What makes organization "effective"? When she calls on a student to make his style "more direct" or "more informal," what exactly does she mean? Has she made these meanings clear to students? We never simply apply a prepackaged set of criteria wholesale to our students' writing. If our terms, as Bakhtin reminds us, always come shot through with prior meanings, they also get filtered through our own experiences and perceptions. We give them our own particular twist, our own individual take, emphasizing one connotation or another. As teachers, it's our job to unpack these meanings, talk with students about what we mean when we use them, and explain how they fit in with the methods and goals of the course. For the protocol teacher, for instance, good content depends to a large extent on the originality

and distinctiveness of the writer's descriptions. Such a definition fits in well with the type of writing the class is working on and her expressivist goals. A writer's voice, in order to be effective, must be informal and personal-- a voice that speaks casually with readers and captures something of the writer's individual way of seeing the world. Such an understanding also seems to go well with the descriptive, personal writing she calls for on this assignment, and it is in line with her student-centered, expressivist pedagogy. There are, however, some problems and inconsistencies. She seems to think that student writers should be fairly explicit in structuring their writing. They would do well to announce their focus in a thesis statement, forecast topics to come, and make explicit transitions. But are explicit thesis statements necessary or useful in the context of such personal writing? Can such thesis statements typically be found in the personal essays of published writers? Are they consistent with her expressivist pedagogy? Even if these values might be reconciled with her theories, there is another related issue: has she done enough to talk about the nature of such formal strategies of organization in class? Her concern with transitions also seems misguided. She reads the paragraph about summers in Syracuse and sees a problem with an overly mechanical transition. But it seems more apt-and more theoretically consistent-to see what happens here as a matter of the writer failing to get beyond the obvious statements one might make about the seasons: not a formal problem so much as a problem of invention or a rhetorical problem involving the writer's failure to make the writing responsive to readers. Her concern with David's use of the second person "you"-presumably because it is too informal-is also problematic. The trouble is that the writer's use of "you" seems well within the bounds of the genre and not unsuited to the assigned audience for the assignment: those (common readers) who are unfamiliar with this place. The writer's use of "you" is also not inconsistent with the conventions of personal writing, which are more relaxed than the constraints of academic discourse. At any rate, like the other instances above, it seems an unnecessary concession to formal academic discourse,

given her pedagogical allegiances. There are more significant problems. When David fails to provide more descriptions based on his own experience and views, she chides him for not following the assignment. Yet her student-centered pedagogy-- and her own classroom policyallows students great latitude in interpreting and following the demands of the assignment. (To her credit, she does not express this concern to David in her comments. She also doesn't insist that he focus his writing on only one or two seasons if he takes up a revision-or that he go back and revise this paper.) More troublingly, she criticizes David for not adopting the voice of an expert on the Syracuse seasons and establishing his authority as a writer. The fact .that he is writing a personal essay and the fact that he is writing for an audience of everyday readers both mitigate against her expecting David to adopt the voice of an expert. Such an authoritative voice might be appropriate for an academic audience. It might even be appropriate for some magazine audiences. It might be rightly expected in a course based on a social constructionist axiology or a social epistemology. But it is not necessary for the circumstances of this assignment or the framework of this class. There is no reason David cannot effectively meet the demands of the situation laid out for him by employing the casual personal voice he adopts here-or, for that matter, the second person "you" perspective. This is a method any writing teacher might use to examine her own theories and practices from the ground up, starting with what she actually looks for in her own students' writing and then trying literally to come to terms with them. Do you emphasize the content of student writing as much as you think you do? As much as your pedagogy calls for? Do you deal with organization in ways that are consistent with the genre of writing and your theories? Do the contexts you address reflect your values and classroom practice? If you are an expressivist, does your reading deal in some detail with the student's own experience and views? If you espouse a rhetorical axiology, do the majority of your responses focus on the writer's persona, purpose, and audience? If you are a social constructionist or if you employ a cultural pedagogy,

does your reading emphasize the social side of the map and deal, in particular, for instance, with the conventions of the discourse community or the cultural assumptions that are underneath the writer's claims? How do you understand the various terms you employ? Do you define "substantive content" in ways that are consistent with your expressivist epistemology? Do you talk about organization in ways that reflect your allegiance to a rhetorical axiology? Does your emphasis on a particular kind of voice go with our social constructionist agenda? Do you read student texts from the perspective of the audience that you ask students to address in their writing? Do you talk about writing as a recursive process in class but then expect students to always be at a certain stage of writing-finished by now with their discovery and planning, done with their drafting, focusing now only on local revisions? By sorting out our criteria and goals, we can crystallize our theoretical commitments, bring our theory to bear more fruitfully on our practice, enrich our instruction, and sharpen our response to student writing. Moreover, we can make our theory and teaching and response work together to achieve what we are most committed to achieving in the writing classroom. ANALYSIS OF THE PROTOCOL TEACHER'S RESPONSE TO THE WRITER Our sense of what makes writing good is the nerve center that connects and operates our theories and our classroom practice. Everything we assume about the teaching of writing and want to achieve as writing teachers comes down, I believe, to what we value and look for when we read our students' papers. This is where it all comes together: our goals, our commitments about good writing, our views toward the writing process, and our pedagogy. It is how we actually read our students' writing, page by page, paper by paper, student by student, that gives breath to our assumptions and that makes or breaks our instruction. But reading and evaluating our students' texts is only half the work. The other half is what we choose to call to the student's attention and how we present our readings to students in the form of comments. As important as our ways of reading student texts are, they are finally only as useful as they inform our work in class, are manifested in our comments, engage students in

reenvisioning their writing, and teach them something meaningful about their work as writers. I'll make use of a simple heuristic to analyze the protocol teacher's comments and evaluate how well she turns her reading of "The Four Seasons" into a response to David. The soundness of a teacher's commentary depends on how well she accomplishes four general tasks and how well they mesh with her theory and classroom practice.15 1 How many comments does the teacher make? How well does she engage the issues of the writing? Does she take on too much in her response? Does she make enough comments to engage the writer in revision? In this protocol the teacher writes 29 comments to Davidessentially 29 distinct statements, individually and in clusters: 16 in the margins, another 13 in her end note. It is a generous helping of comments, well above the average of 22 comments that informed writing teachers employ in their responses (Straub and Lunsford). The number of comments enables her to give a fairly close reading to the writing and get into a meaningful exchange with the writer. 2 How well does the teacher specify and communicate her comments? The comments are written out in statements that are, for the most part, text-specific and detailed, and the teacher does a good job in expressing her points to David. In several instances, she even goes on to explain or illustrate some initial response in one or more follow-up comments: * Good detail. I can feel the squishing. * Good concrete detail... It's an image that will lead your readers to imagine how hot it gets in Syracuse. This is the kind of detail I've been looking for. * You're relying too much on cliches: geese flying north in the spring, summers at the beach, raking leaves in the fall, and sitting by the fire with that special someone in the

winter. They're descriptions that we're all familiar with and they don't give any insight into how the four seasons are unique to you. This is not to say that her comments are always well-stated or that the response couldn't do with a couple more explanatory comments. The student may have some trouble making sense of her asking in the first paragraph, Why the disclaimer? He may not get the gist of her saying Shift in point of view in the second paragraph. The teacher would also do well to add some follow-- up explanation of what she's looking for when, just a short time later, she writes Which trees? What kind of flowers? or when she circles the general terms in the next paragraph and says Let's hear some details. What kind of details is she hoping to see? What effect would they have? Her terse Mechanical transition could also use some explanation. 3 How well does the teacher capture her key concerns and invoke what she sees as the most important contexts in her responses? Are her responses focused and coherent? The protocol teacher does well on this score. Eighteen of her twenty-nine comments are focused on the content of writing, in this case, on David's descriptions. Another dozen go beyond the formal text and invoke some larger context: the rhetorical situation, the writer's personal experience, and his work as a writer. Only three comments deal with local matters-all of them in marginal comments. The end note is reserved for dealing in greater depth with large conceptual and rhetorical matters. The response is particularly apt, given this teacher's expressivist assumptions and the assignment, in looking at the writing in terms of David's own experience with the seasons of Syracuse.16 It reflects what seems to be a strong rhetorical emphasis. And it does well to provide some direct process-based advice that the writer might follow to help him come up with the kind of substantive content that this teacher is looking for in the writing. The protocol teacher also does a fairly good job of keeping David's attention on the areas she considers most important at this time. She smartly avoids presenting a number of her initial reactions in the form of comments to David-her seeing his opening gambit as a cop out, her sense that he is getting careless and just

blowing through his descriptions, coasting. For all her good choices about staying focused on what's most important to her at this stage of the writing, however, she might do well to resist commenting on transitions, usage problems, and minor errors, as her emphasis on student development rather than on completed texts would advise. There is much more substantive revision for David to work on here before he attends in any detail to such local matters. The teacher's marginal comments are not overly intrusive on the student's text, and her end note is laid out well, in some detail. The two complement one another. Her end note, further, does not merely repeat the same concerns she raises in the margins, but focuses on certain key concerns and, for the most part, explains them more fully. 4 How does the teacher construct herself as a teacher-reader through the ways she frames her comments-and what kind of relationship does she establish with the student? The protocol teacher's comments are far more evaluative and directive than they might be, especially for an expressivist with a student-centered pedagogy and a commitment to giving students greater authority over their decisions and purposes as writers. Twelve of her twenty-nine comments are presented in the form of evaluations, seven of them negative. Much to her credit, she has a penchant for making note of what is praiseworthy and looking to build on the writer's strengths. She compliments David several times on his vivid descriptions. At times she even goes beyond general praise and explains what she likes. Still, her response reveals the limits of our grammar of praise. She likes David's way of looking at Syracuseans coming out of hibernation after a long winter, and tells him so. Good idea, she writes. But she says nothing in her comment about what in her eyes makes the idea good: how it is different from the usual descriptions of people getting outdoors in the spring, how it consequently might capture a reader's attention. She also likes the way he pursues this metaphor and sees the emerging warmth of the season as stirring a parallel warmth of feeling among neighbors. But again she offers only a quick note of praise-Nice-and moves on, failing to make the most of the opportunity. And while she starts her end note with two praise

comments, she leaves both of them unspecified and unelaborated, allowing them to be read merely as a gesture, a conventional pat on the back at the start of a teacher's end note. How different the comment would read if she specified her praise here in the way she specifies her criticism in the ensuing comments: * Pretty good, David. You have some fine descriptions of the four seasons in here. The moist ground seeping into your shoes in the spring. The heat resting on the pavement in the summer. The aroma of dried leaves in the fall. The soft scrunching of snow in the winter. Her comments are made more critical by several additional comments that she presents in the form of closed questions, all of which are more evaluative than interrogatory: * Why the disclaimer? * Really? Squirrels and rabbits hibernating? Bears and rabbits chattering? * Don't... the women in your neighborhood ever cut the grass? To soften some of this evaluative edge, the teacher might employ more readerresponsecommentary, the kind of commentary she provides to David's description of the moist ground in spring: Good detail. I can feel the squishing. She might explain, for example, how the reference to geese goes only so far to help her imagine how nature announces the return of spring in Syracuse and to identify the kinds of trees she associates with spring. Such specific moment-by-moment reactions and reader responses might also help establish more of the kind of reader-to-writer relationship she seems to value and help her put into practice her commitment to lead students to assume authority over their writing choices. Still, overall she does a solid job of transforming her reading into a response, using her comments to reinforce the work in class, and guiding David's ongoing work as a writer.

CONCLUSION Writing at the turn of the last decade, Richard Fulkerson seems hopeful that the confusion he saw in composition studies could be substantially reduced and the consensus he saw emerging could be developed into wide-spread agreement. Such development, he seems to suggest, would come about mainly at the level of theory, through the work of composition scholars. I have no doubt that theory can help us better understand the nature and goals of composition and help reduce the confusion between ends and means that seems rampant both in our scholarship and in our dayto-day teaching of writing. But I also believe that theory and disciplinary knowledge can go only so far in these efforts. The real work of clarifying our values and goals and connecting our theory and classroom practice is a practical problem that must be addressed first of all by teachers, individually. In this article I have looked to meet the problem head on by calling on teachers to cast themselves in the role of reflective practitioners and critically examine what is at the heart of our theory and practice: our concept of what makes writing good. To facilitate such a project, I have laid out and ordered the range of concerns that we take up as teachers when we read and respond to student writing. I have looked, in turn, to devise a heuristic we can use to clarify our goals, define our values and get them in line with our theories and practice, and test how well our criteria and goals are realized in our comments to students. More than our philosophy or theories, more than the kinds of writing we assign, more than the classroom activities we employ, more than the strategies we ask students to practice, it is our concept of writing and the evaluative criteria that we bring to our day-to-day reading of student texts that enable us to merge our theory and practice, define us as one sort of teacher or another, and determine the strength of our teaching. By sorting through our values, we can clarify our own commitments and develop a sharper vision for our work as writing teachers. We can become better readers of our students' texts, become more able to adapt our responses to specific students in

specific situations, and, quite simply, improve our instruction. Moreover, by starting with our work as teachers and interrogating our own theories and instructional practices, we will also be in a better position to contribute as teacherscholars to the disciplinary knowledge of composition and even perhaps, over time, to the kind of disciplinary continuity that Fulkerson and others hope to achieve. This is the key to working through the inconsistencies that frustrate our attempts to think clearly about the teaching of writing. This is where our energy is best spent in trying to work out our theory and practice and make our teaching more consistent, more purposeful, and ultimately more effective: with the teacher in the classroom looking at her own values, methods, and goals. Tallahassee, Florida -1-

The Effect of Audio and Written TeacherResponses on EFL Revision. Student

by Ana Maria Morra , Maria Ines Asis Providing feedback is one of the foreign language writing teacher's most important tasks (Ferris, 2007). However, there is less certainty about the techniques that should be used (K. Hyland & F. Hyland, 2006). This article reports on research that investigated the effects of two types of teacher feedback, on-tape and written, and of the absence of feedback on students' (n = 89) error correction. A comparison of the number of macro (content, organization) and micro (vocabulary, grammar, mechanics) errors before and after the experiment yielded a statistically significant reduction in the

number of mistakes in final drafts.Students perceived the type of response received (either taped or written) had been helpful in revising their papers and considered the most beneficial aspect of teacher feedback to have been a focus on micro errors. This study offers insights into an ecclectic approach to teaching writing in similar EFL contexts. Over the last twenty-five years, the focus of writing instruction has changed from product to process, from seeing students' writing less as a finished product than as a piece of work perfectible through multiple drafting with feedback between drafts (K. Hyland, 2003; K. Hyland & F. Hyland, 2006). Whether the process approach to writing instruction per se has brought about positive results in foreign language pedagogy has provoked much controversy, resulting in a thrust towards a more balanced view of process and form (Applebee, 2000; Bromley, 2003). No doubt the process approach has had a major impact onwriting research and instruction (K. Hyland, 2003). It continues to be applied in the teaching Of English as a foreign language (EFL) because by placing considerable emphasis on revising and responding to writing, it allows teachers and students more meaningful interaction. Process feedback, with its emphasis on the recursive nature of writing, has emerged as an essential component of the approach and has stayed in the forefront of instructional practice. Review of Literature Several studies have shown the positive results of teacher feedback. These studies have focused onfeedback on form (Ferris, 1997) and content (Fathman & Whalley 1990), others on different means ofdelivery, such as electronic (Greenfield, 2003; Warschauer, 2002; Williams, 2002) and conferencing (Goldstein & Conrad, 1990; Patthey-Chavez & Ferris, & F. 1997). Hyland, Additional 2006). A those research, few who studies have however, have has shown believed questioned no in clear the the effect of teacher feedback on language errors (Goldstein & Conrad, 1990; K. Hyland signs of improvement of students' writing accuracy after response, so the debate has continued between effectiveness ofcorrective feedback on form and those who have not (Guenette, 2007).

The most adamant argument against grammar correction came from Truscott (1999, 2007), whose position has provoked claims for additional research before the controversy over the effectiveness of error correction on learners' ability to write accurately is settled (Ferris, 2004). Still other researchers have not found significant differences onstudent revision and editing involving the explicitness of teacher response (direct, indirect) or the means used (written, verbal) (Ferris, 1995, 2006; K. Hyland, 2003). Both a paucity of research on teacher response to student writing and, in some cases, contradictory results, have left many key questions only partially answered (K. Hyland & F. Hyland, 2006). One aspect in particular has remained virtually unexplored: the effect of taped feedback on learners' revisions, hence the need for further research on teacher feedback in EFL contexts. Purpose of the Study The present study aimed at contributing to efforts to improve EFL writing by investigating the effects oftwo different modes of teacher response: written comments on the margins and recorded feedback. Reformulations made by students who did not receive any teacher feedback during the process were also included in the data for analysis. Context and Rationale of the Study This study was carried out at the School of Languages, National University of Cordoba, Argentina, where teachers were concerned about students' poor performance in the composition component of their exams. Many explanations could have reasonably accounted for these poor results: overcrowded classrooms, a deficit of student reading and writing strategies, lack of prior knowledge of content and rhetorical structures, or a flaw at some stage of the teaching practice, among others. These considerations led to an examination of the feedback students were receiving and their reactions to it. Teachers annotatedstudents' written assignments by writing observations on the margins or by making general comments at the bottom of students' papers. Remarks concentrated

heavily on language use rather than on content and global organization. Because learners complained about their not understanding teachers' marginal comments, especially those on macro aspects such as subject matter and relevance of ideas, teachers questioned their approach and began to explore alternatives. A verbal response such as face-to-face conferencing was unfeasible in this context due to the large number of students per class and the lack of school facilities for teachers and students to meet for the sessions. A good choice seemed to be the "taped commentary" suggested by K. Hyland (1990, 2003). On-tape feedback required the teacher to record his or her response on a tape cassette and to write a number on the student's paper to signal what the observation referred to (K. Hyland, 2003, p. 182). As previously said, this means of providing feedback has been virtually unexplored except for K. Hyland's study (1990). In spite of the scant support from empirical research, on-tape feedback appeared intuitively attractive to instructors in the EFL Department. A comparison of the advantages and disadvantages of this type offeedback and the one traditionally used in our classes yielded the following results (see Table 1). This study then was undertaken to investigate the traditional written response (with codes on margins) that the staff had been using in their courses for a decade, the innovative on-tape response tried out for two semesters, and the lack of teacher response throughout a multiple draft writing cycle. The researchers of this study proposed the following alternative hypotheses: 1. On-tape feedback may be more effective than written feedback in

helping students reformulate theirwritten work at the macro level (content, relevance, organization, coherence, logic). 2. Written feedback using codes on margins may be more effective than on-

tape feedback in helpingstudents correct errors at the micro level (surface sentence errors,

grammar, use of lexical items, spelling, punctuation). Method Setting and Subjects The study was carried out in a five-year teacher training and translation studies undergraduate program at a college in Argentina. The study involved six groups of students taking EFL courses at a post-intermediate level of proficiency according to the criteria established by the Association of Language Testers in Europe. Three of these groups had enrolled in the Teacher Training program and the other three in the Translation Studies stream. For each orientation there were two experimental groups (the innovative one with taped response and the traditional one with written codes on margins), and one control group (no teacher response). Eightynine students, producing a total of 267 pieces of writtenwork, participated in the study. Two teachers also participated, each in charge of the three groups in one program of studies (either Teacher Training or Translation). Materials and Procedures The two experimental groups and the control group in each program of studies received the same writinginstruction for 6 weeks. On week 7, the groups were given a topic and were asked to write a short opinion essay. They produced two drafts and a final version. Between drafts, the innovative group received taped feedback, and the traditional group received written feedback with codes. Previously, the researchers and the two teachers providing feedback had had several meetings for standardization ofcriteria. In these meetings, instructors received detailed instructions and training on how to use a taxonomy of errors designed by the researchers (see Table 2 for this taxonomy and the codes used forwritten feedback). Teachers were asked to concentrate on the macro aspects of writing (content and organization) in their feedback on the first draft and on macro and micro (vocabulary, grammar, mechanics) components in their comments on the second draft. No feedback was given to the control groups except for a

general instruction for students to revise their compositions and make any changes the writers considered appropriate to improve their written work. All the groups had deadlines for first and second drafts and for their final versions. When this writing cycle was over, students answered a semi-structured questionnaire (see Appendix) about the learners' reactions to the type of feedback they had received or the absence of feedback. Data Analysis and Results Working on the initial drafts, we identified, charted, and then attributed each teachertaped comment orwritten mark to one of the 19 categories shown in Table 2 plus an additional category (used only in the analysis stage for flaws and errors not classified by the taxonomy employed). When a comment or mark did not correspond to one of the categories on the list, we classified the observation ourselves according to the 20 types established. We proceeded in the same way on the final drafts. Last, we compared the number of macro and micro errors in the first draft and in the final version for each student and, eventually, for each group. The coding stage was long and complex. Teachers had not applied

the feedback categories consistently. For example, in written feedback we found marks with no codes attached or with codes not listed in the taxonomy agreed upon. For taped feedback, we judged some comments as "vague," "unnecessary," or "misleading." In both cases, we found instances of incorrect feedback, that is, teachers had suggested a change be made even though there was no error. We spent long hours analyzing and coding the data, going over ambiguous comments and, in cases of disagreement, appealing to a third party, a colleague thoroughly acquainted with our taxonomy. Instances of students' unsuccessful revisions (incorrect reformulations, no change made in spite of teacher comment) were set apart for future analysis. The data were analyzed using the non-parametric Wilcoxon's Paired Signed Rank Test to investigate, by means of a unilateral analysis, the statistical significance of the difference

between the number of macro and micro errors before and after the taped or the written feedback. Wilcoxon's test was also used to analyze control-group data to investigate whether the absence of feedback had any impact on the number of errors found in the final versions of the students' essays (see Table 3). The results of the statistical tests indicated that the null hypothesis could be rejected in all cases and that the differences found after treatment (taped, written feedback, no feedback) were statistically significant. The results also showed that these changes had been positive both at the macro and micro level in all but one group. The Translation Studies group, which had received taped feedback, showed negative changes at the micro level--that is, after treatment, this group registered an increase in the number of micro errors. In regards to the two control groups, the differences found were also statistically significant and positive in all cases. In sum, five of the six groups that participated in the study showed a statistically significant reduction in the number of errors in the final drafts of their essays. Results of the questionnaire revealed that 87% of the subjects in the experimental groups were able to understand their teacher's comments. Moreover, the students stated that the type of response received (either taped or written) had helped them revise and rewrite their papers. Also, 56% of the subjects in the control groups acknowledged that the opportunity to review and reformulate their papers even without any observation or comment on the part of the teacher had been beneficial. When asked about the features of their written work reformulated after feedback or after being given the opportunity to revise and rewrite, 88% of students said that they could correct errors in grammar, lexis, spelling, and punctuation (micro errors); 44% also stated that they had succeeded in overcoming weaknesses in content, coherence, relevance, and meaning (macro errors). As to the type of feedback preferred, 96% of those who received on-tape comments chose this type offeedback; 88% of all students who completed the questionnaire ticked written feedback as their preference, and only 14% of those who did not

get feedback said they had enjoyed reviewing and rewriting their compositions without any kind of teacher response. Discussion and Conclusions Responding to student writing appears to have a positive effect on learners' written work regardless ofthe means used (written or taped). In this study, the number of students' errors and weaknesses at both the micro and the macro level decreased significantly in the final version except for one group in the Translation Studies program. The implications of this result will be discussed below. Another important finding of the study is that the mere opportunity for students to revise their compositions on their own, without teacher feedback, resulted in a reduction of the number of flaws. This outcome emphasizes the importance of self-assessment, contributes to research that shows improvement of students' end-products after rereading and rewriting their own papers without any feedback (Ferris and Roberts 2001), and supports findings of limited teacher response to the progress students make in their writing (Graham, 1983, cited by Fathman & Whalley, 1990; Truscott, 2007). Thus, it would probably be beneficial, as Fathman and Whalley (1990) suggest, to encourage teachers to make a place for more independent writing in their classes (such as diaries, which demand no teacher intervention) and to promote more self-guided revision at home, thus paving the way for greater autonomy in assessment and revision. Furthermore, the perception of 88% of our subjects that the most beneficial aspect of teacher feedback or of the opportunity to revise their writing on their own was linked to formal aspects of language use suggests polemic issues for teachers of EFL writing at the university level, at least in the context of where this study was conducted: What should be the main objective of a writing class? How much attention must be paid to language accuracy in the composition class? Should teachers be satisfied with a spotless composition even if it is weakened by poor argumentation or insubstantial content? Or should they teach strategies for focusing on relevance and richness of ideas as well as on language forms? The students in this study seemed to adhere to a documented

belief among EFL writers that places high value on a composition free of surface errors (K. Hyland & F. Hyland, 2003; Leki, 1990; Morra, Badenes & Musso, 2003). This conviction may grow out of a major concern in EFL academic contexts about the learner's language proficiency being far from a near-native operational command. The students involved in this study are expected to improve their accuracy as they progress in their studies because they will be graduating as teachers of EFL or becoming translators. Probably as a result of this, learning to write is synonymous with acquiring and rehearsing the use of lexical items, syntactic patterns, cohesive devices, and other language forms, a practice which focuses on accuracy and clear exposition before content. EFLwriting teachers might choose, then, to help learners develop awareness that producing effective texts also involves substantive content and information-gathering strategies. Further research is needed in similar EFL contexts to find out whether the feedback modes studied here result in improved writingperformance. This study also supports research in EFL feedback that suggests that students do take teachers' observations seriously and do not feel upset by them (Ferris, 1995; Ferris & Roberts, 2001; Leki, 1990). Such an attitude contrasts with evidence provided by studies of feedback in English as the native language (Ferris, 2007). Our classroom experience indicates that EFL writers are aware of their language limitations and, consequently, expect and want their teachers to correct errors. A similar attitude in second-language learners is reported by F. Hyland (1998). In regard to the mode of teacher feedback preferred, the taped response was chosen by almost 100% of the students who experienced this type of feedback in the study and could thus compare it with the more familiar written type. These students stated that the recordings' "shortened distance" with the teachermade them feel as if they were actually "talking" with the teacher even if they could not participate in a face-to-face dialogue. They reported a sense of commitment that led them nearly always to hand in their revised papers back on time. But perhaps a more important point students made was that, with their papers in hand, they felt they could grasp the teacher's comments easily. They could clearly see what she or he was aiming at, where ambiguity impaired comprehension, and

where there were gaps in the line of thought. Similar student reactions have been reported by K. Hyland (1990). As mentioned earlier, one of the groups that received taped feedback showed an increase in the number of micro errors after treatment. K. Hyland (1990) suggested that both those who give and those who receive taped feedback tend to concentrate more on macro aspects of writing. In fact, the final version of the essays of the students belonging to this group revealed more clarity in the expression of ideas despite the number of language errors. To conclude, most of our subjects expressed preference for either written feedback or taped feedback, and a small percentage found self-reviewing effective. Thus, student response to feedback may be more or less effective depending on the mode of teacher response. A possible pedagogical implication for teachers could be to offer students a taste of different types of feedback for them to choose from, thus responding to students' individual needs. Future studies should attempt to investigate this further with a single group of students; this was not feasible in this study. Also, it would be interesting to complement the quantitative nature of the present piece of research with a qualitative one to capture teachers' perception of the effectiveness of the type of feedback they provide. Appendix Survey Thank you for participating in our research project. As a last step, we would like you to answer this questionnaire. Please, choose the answer(s) that best represent(s) your experience and complete the blank spaces provided. A. Which of the following types of feedback (comments made by the teacher) were you familiar with before the experiment? (Note: You can choose more than one answer here.) 1. Use of a minimum "abstract" code, e.g., underlining of words or phrases; use of signs

such as ? ! [] ^ [??]. 2. Words or letters written on the margins, such as ST, Agr, Sp, WO, etc. 3. Longer and more detailed comments and suggestions written on the margins 4. and at the end of the written piece. 5. Tape-recorded feedback. 6. Feedback sent by e-mail. 7. Face-to-face conferences with the teacher. 8. Oral or written comments made by peers. 9. Other (please specify briefly): -10. None of them. B. What type of feedback were you given during this experiment? 1. Comments written on the margins (with codes). 2. Tape-recorded feedback. 3. None. 1. If you received teacher feedback, how helpful was the feedback when re-writing your essay? a. Very helpful. b. Quite helpful.

c. Rather helpful. d. Not helpful at all. 2. If you had no teacher feedback, how helpful was the chance to revise on your own? a. Very helpful. b. Quite helpful. c. Rather helpful. d. Not helpful at a11. C. In which of the following aspects was the feedback you received or the chance to revise withoutfeedback most helpful? (Note: You can choose more than one answer here.) 1. To express your ideas more clearly, e.g. emphasizing what is important and discarding secondary ideas. 2. To organize your writing in a better way. 3. To improve your grammar. 4. To improve your vocabulary. 5. TO improve your spelling. 6. To improve your punctuation. 7. Other (please, specify) -D. How many times did you submit your essay during the experiment?

1. Only once. 2. Two times. 3. Three times. E. If you did not comply with the three scheduled submissions, why didn't you? 1. You didn't think rewriting your work was going to help you improve your writing skills. 2. You were not able to understand the comments made and, therefore, you were not able to correct them. 3. You could not meet the stipulated deadlines. 4. Other (please specify briefly): -F. Which of the types of feedback you are now familiar with have helped you the most to improve yourwriting skills? 1. Oral or written comments made by peers. 2. Comments written on the margins (with codes). 3. Longer and more detailed comments and suggestions written on the margins and at the end of the essay. 4. Minimum "abstract" code (underlining, use of signs such as? ! [] ^ [??]). 5. Tape-recorded feedback. 6. Face-to-face conferences with the teacher. 7. Feedback sent by e-mail.

8. Other (please specify briefly): 9. None of them. Thank you. References Applebee, A. N. (2000). Alternative models of writing development. In R. Indrisano & J. R. Squire (Eds.), Perspectives on writing: Research, theory and practice (pp.90-111). Newark, DE: International Reading Association. Bromley, K. (2003). Building a sound writing program. In L. Mandel Morrow, L. Grambell, & M. Pressley (Eds.), Best practices in literacy instruction (pp. 143-166). New York: The Guildford Press. Fathman, A. K. & E. Whalley. (1990). Teacher response to student writing: Focus on form versus content. In B. Kroll(Ed.),. Second language writing. Cambridge: Cambridge University Press. Ferris, D. (1995). Student reactions to teacher response in multiple-draft composition classroom. TESOL Quarterly, 29(1), 33-53. Ferris, D. (1997). The influence of teacher commentary on student revision. TESOL Quarterly, 31(2), 315-339. Ferris, D. (20041). The "grammar correction" debate in L2 writing: Where are we, and where do we go from here? (and what de we do in the meantime ...?). Journal of Second Language Writing, 13, 49-62. Ferris, D. (2006). Does error feedback help student writers? New evidence on the shortand long-term effects of written error correction. In K. Hyland & F. Hyland (Eds.), Feedback in second language writing(pp. 81-104). Cambridge: Cambridge

University Press. Ferris, D. (2007). Preparing teachers to respond to student writing. Journal of Second Language Writing, 16, 165-193. Ferris, D., & Roberts B. (2001). Error feedback in L2 classes: How explicit does it need to be? Journal of Second Language Writing, 10, 161-184. Goldstein, L., & Conrad, S. (1990). Student input and negotiation of meaning in ESL writing conferences. TESOL Quarterly, 24(3), 443-460. Greenfield, R. (2003). Collaborative e-mail exchange for teaching secondary ESL: A case study in Hong Kong. Language Learning and Technology, 7(1), 46-70. Guenette, D. (2007). Is feedback pedagogically correct? Journal of Second

Language Writing 16(1), 40-53. Hyland, F. (1998). The impact of teacher-written feedback on individual writers. Journal of Second Language Writing, 7(3), 255-286. Hyland, K. (1990). Providing productive feedback. ELTJournal, 44(4), 279-285. Hyland, K. (2003). Second language writing. Cambridge: Cambridge University Press. Hyland, K., & Hyland, F. (2006). Feedback in second language writing. Cambridge: Cambridge University Press. Leki, I. (1990). Coaching from the margins: Issues in written response. In B. Kroll, (Ed.), Second languagewriting (pp. 57-68). Cambridge: Cambridge University Press. Morra, A. M., Badenes, G., & Musso, E. (2003). Error correction by undergraduate and postgraduatestudents. Advances in ESP research in Latin America. Proceedings of the VIII Latin American ESP Colloquium (pp. 105-112). Salta, Argentina: Comunicarte

Editorial Oxford, R. (2001). Cognitive processes and learning strategies: A seminar to integrate theory and practice in Argentina. Unpublished manuscript, National University of Cordoba, Argentina. Patthey-Chavez, G., & Ferris, D. (1997). Writing conferences and the weaving of multivoiced texts in college composition. Research in the Teaching of English, 31, 51-90. Reid, J. (1995). Learning styles in the ESL/EFL classroom Boston, MA: Heinle & Heinle. Truscott, J. (1999). The case for "the case for grammar correction in L2 writing classes": A response to Ferris. Journal of Second Language Writing, 8, 111-122. Truscott, J. (2007, in press). The effect of error correction on learners' ability to write accurately. Journal of Second Language Writing. Warschauer, M. (2002). Networking into academic discourse. Journal of English for Academic Purposes, 1, 45-48. Williams, J. (2002). Undergraduate second language writers in the writing center. Journal of Basic Writing, 21(2), 73-91. Dr. Ana Maria Mona is Chair of English at the School of Languages, National University of Cordoba, Argentina, and Head of the Master Program in Spanish as a Foreign Language at the same university. Her e-mail address is anamorra@gmail.com. Maria Ines Asis (MA) is a lecturer in English for Speakers of other Languages (ESOL) and curriculum leader for the additional learning support division at Southgate College, London, UK. Email: ines.asis@southgate.ac.uk.
Table 1. Advantages and disadvantages of written and taped commentary.

Type

of

feedback Written margins * * on * on very appropriation * * Recorded on tape papers teacher individual paid * and use * ideas * * * * Table 2. Taxonomy codes of Helps to Helps concentrate meaning * Engages to for attention their More if

Advantages viable well of * given root of of work

Disadvantages Vague * Too problems and much surface teacher's student Impersonal confusing focus overlooked on errors,

Effective

Comments *

use helpful Risk

editing

language

Time students reward the * at on more of Comments very Contributes self-monitoring less writers Consumes Motivates errors for used by teachers less on papers teacher into * the *

consuming Cannot Students more than Students listening a than language students' helpful to skilled time more providing feedback and on with skills content on may be may on done anywhere

reformulating

their

concentrate listening comments poor be

disadvantage

written Codes

response.

Error

type 1.

and

description

(written Content

feedback)

Relevance, 2.

richness Organization,

of

information

RI Cohesion

Coherence,

Thesis/Topic Supporting Concluding Logical Use of connectors, Referencing "it," development transitional transitional (shifts "this," 3. Collocations Word Choice of word in vague

sentence(s) sentences sentence(s) of signals phrases, pronouns, or ideas (one-word and usage ambiguous Lexis Colloc form (specificity topic) in WF relation of

TS SS CS LD Trans Pr S Ref sentences) reference)

to

WC

4. Subordination Double Omission Word order subject-verb Verb Agreement excluded Prepositions 5. Punctuation period, (commas: semicolon Spelling Others Table 3. (used Results of the only for subordinate, between with (tense, (subject-verb, pronoun of (order of subject subject

Grammar Sub D O imbedded form) singular-plural, reference) Prep Mechanics main main SP analysis Wilcoxon's of Paired data) Signed Rank clauses, connectors) Punct Subj Subj questions, others) V Agr WO

inversion,

clauses,

punctuation

Non-Parametric Test.

Groups

in

the

Translation

Studies

Program

Macro Written [SIGMA] T Taped [SIGMA] T No [SIGMA] T Groups in Macro Written [SIGMA] T Taped [SIGMA] T No [SIGMA] T Micro Written [SIGMA] T Taped [SIGMA] T No [SIGMA] feedback + 71 + feedback 34.5 feedback + 124 feedback + 66 feedback + 72.5 feedback + 185 the Teacher feedback + 78 feedback + 134 feedback + 136

Errors SS 35 SS 35 SS 17 Training Errors SS 53 SS 21 SS 17 Errors SS 35 SS 35 SS 12 [SIGMA]-4.5 16 [SIGMA]-101.5 16 [SIGMA]-12 12 [SIGMA]-12 13 [SIGMA]-17.5 20 [SIGMA]-5 Program 12 [SIGMA]-0 16 [SIGMA]-3 16 [SIGMA]-0

T Groups in Micro Written [SIGMA] T Taped [SIGMA] T No [SIGMA] T 17 feedback + 90 SS feedback 60 feedback + 192 the Teacher

17 Training Errors SS 60 SS [SIGMA]-17 17 12 [SIGMA]-1 13 20 [SIGMA]-28 Program

Feedback Follow signments

by Deborah W. Dunsford Introduction Writing improvement is becoming an

Up:

the

Influence ofTeacher Comment on Student WritingAs

increasingly

important

topic

at

most

universities. Feedback from potential employers and research repeatedly shows that college students' writing abilities are below expectations (Lindner et al., 2004). Several universities have implemented writing-intensive course requirements for undergraduate students that will ultimately require faculty in all disciplines to provide additional writing opportunities in their curriculums (Univ. of Florida, 2004. The Gordon

Rule; Texas A&M Univ., 2004. Writing Intensive Courses at Texas A&M Univ.; Martin and Burnett, 2003). For agricultural education and communication programs, this focus frequently takes the form ofservice courses that teach writing skills (Kansas State Univ., 2004; The Ohio State Univ., 2004; Texas A&M Univ., 2004. Core Curriculum; Univ. of Florida, 2004. Agricultural Education). As the demand for seats in courses that teach writing skills continues to grow, instructors try to balance the need to provide students with feedback on their writing assignments with the amount of time it takes to provide that feedback. While writing instructors from all disciplines generally agree that revision is one of the best ways to encourage students to improve their papers, few know what comments or what type of comments are most likely to help their students revise successfully. Research into revision and how and why students revise their texts has long been part ofcomposition literature. So has research into teacher comment on student texts. However, there is little work that brings the research areas together. This study may provide a link between these two important areas of research. Composing a piece of written discourse has long been considered a non-linear, recursive process (Britton, 1975; Rohman and Wlecke, 1964). Later researchers built on this model describing composing as a continuous loop where any element may follow any other element (Bridwell, 1980; Faigley and Witte, 1981; Flower et al., 1986; Sommers, 1980). Although the recursive nature of the process is not in question, an actual definition for revision is less clear. Several definitions use only the etymological definition of "seeing again" (Boiarsky, 1980). Sommers (1980, p. 380) defines revision as "... a sequence of changes in a composition changes which are initiated by cues and occur continually throughout the writing of a work." Drawing on all these definitions, the operational definition for revision used in this study refers to the additions, deletions, substitutions, and rearrangements of units of meaning that students make in their texts in an effort to convey better their intended meaning to an audience.

Teacher comment is another key area of composition research. Much research shows that teacherresponse can have a major impact on a student's attitude toward the text and toward writing in general. De Beaugrande (1979) claimed that if students see grammar, punctuation and spelling as priorities in teacher comment, then those are the errors they will repair. Miller (1982) suggested two separate sets of teacher comments one on content and the other on writing problems. Murray (1979) advocated doing away with comment completely and using one-on-one conferences to provide feedback to students. Peterson et al. (2004) suggest that the type of paper plays a role in the type of comments teachers provide. Narrative papers receive a greater percentage of editing-related comments and persuasive papers tend to receive a greater percentage of revision-related comments (Peterson et al., 2004). Besides types of comments, other research examines the quality of those comments. Lynch and Klemans (1978) surveyed students about their responses to teacher comment and found thatstudents responded more positively to comments that not only told them what was wrong with a paper, but why. Straub (1996) explored directive versus facultative comments on student texts and the potential control the comments represented. In general composition researchers agree that the goal of teacher comment on papers is to wean students away from criticism from the teacher and toward forming their own ability to review and revise their texts. Purpose The purpose of this study was to determine how the types and

location of teacher feedback on a group of student texts influenced the revision choices that group of students made to their texts. The research objectives were to determine if the location of teacher feedback influenced students' revision choices and if the type of content or type of teacher comment influenced those choices. Methods The subjects in this study were 62 students enrolled in media writing classes at a land grant institution in the South. Each of the four classes studied had enrollment

limits of 16 students and thestudents were primarily juniors and seniors majoring in journalism or agricultural journalism. A fewstudents were either minoring in journalism or planned to obtain teaching certification in journalism. The majority of the students were female (69%), which reflects a nationwide trend in communication departments. The classes met four times per week during the 16-week term. The Monday and Wednesday lecture sessions covered techniques in news writing and textbook material on different types of news stories, as well as some basic information on methods of writing. The other two weekly class meetings were 75minute lab sessions. The students used Microsoft Word software for all of their assignments. Besides the lab periods, students had access to the computer-equipped classroom through the work day and had access to their own computers or other campus computer labs throughout the semester. Data was collected from the students' four major writing assignments. The four assignments were as follows: (1) Write a pair of short, one-paragraph leads from a choice of assigned fact sets; (2) write a news story from a short speech and question-andanswer session presented by a guest speaker; (3) write a news story about a coming event on campus or other item of the student's choice that quotes at least one source (i.e., they had to go interview someone and write the story); and (4) write a short feature story on a topic of their choice that quoted at least two sources and required additional background sources. Students wrote for both print and broadcast media. Theteacher comments and the revisions students made on these assignments provided the data for this study. Students had the option to revise one of the two versions of each of the major assignments. If thestudents opted to revise one of their papers, the grade they received on the original paper counted as two-thirds of the grade on the final paper. The grade on the revised paper counted as the remaining one third. This method encouraged students to make their best effort on the original paper. The students' grades on the revised papers would not be lower than the original grade they received, although the grade could remain unchanged. For purposes of this study, only the papers that the students opted to revise were analyzed.

Each of the four classes received four different methods of instructor feedback with a different method used on each of their four major assignments. The comment methods were marginal and concluding written comments on their papers, marginal comments only, concluding comments only, and only oral comments to the class as a group. When revising their papers, the students were required to return the graded original paper along with the revised version. To protect the students' identities and to eliminate any chance of bias associated with any particular student, each student was assigned a random number, and an uninterested thirdparty placed this number on the students' texts and then cut off the students' names and course section numbers. To preserve the regular classroom environment during the study, the students were not told about the study until the end of the semester, after the last paper had been turned in. The students received a written explanation of the study and the use of their texts (anonymously). At this time they were offered the chance to have their papers removed from the study. None of the students selected this option. This study met all university requirements for human studies research and all necessary forms are on file with the university's research office. After the student texts had been collected they were sorted by assignment and teacher comment type (marginal and end, marginal only, end only and oral only comment). The texts were sorted numerically for ease in coding and an index card was established for each student number. These cards provided a method of tallying the number and types of revisions on each text. The data from these cards provided the basis for the statistical analysis in this study. Structural revisions made by the students in a second, revised paper were compared to their original, graded papers. Structural revisions in this study were additions, deletions, substitutions, and rearrangements (Sommers, 1980). These structural revisions were examined at the level ofunits of meaning that may or may not correspond to the physical division of paragraphs within the text. According to Rodgers (1967), paragraph divisions frequently do not correspond with units ofmeaning within a text, and he suggests that a "stadia of discourse" is a better unit than the somewhat arbitrary paragraph indention. The

"stadia," according to Rodgers, is a sentence or groupof sentences that contain a single topic which may or may not be contained in a single paragraph. This idea is particularly important when working with journalistic writing. Paragraphs in a newspaper or on an audio script are frequently shorter to accommodate the requirements of the newspaper's narrow columns or the readability for a television or radio reporter. The variable of interest for this study was units of meaning, sentences or groups of sentences that share a common topic. An ANOVA was performed on the revision data that, in effect, combined the four classes into a single group for statistical purposes (Ott, 1988). This method is appropriate because the students were not assigned randomly to the classes used in the study. The analysis examined the four treatments (marginal and end comment, marginal comment only, end commend only and oral comment only) and the four revision types (additions, deletions, substitutions and rearrangements) to determine if there were any significant differences between the treatments and the outcomes. In the analysis, differences with p <.10 are considered significant. This significance level was used to help offset Type II error that could easily result from the relatively low number of subjects, the imprecise measurement methods, and the exploratory nature of this research (Lauer and Asher, 1988). Next, using percentages and graphs, the data were analyzed for similarities and differences among the combination of treatments and the resulting revisions. A naturalistic inquiry method was used to examine the relationship between specific instructor comments and the specific revisions that resulted from that comment (Lincoln and Guba, 1984). To analyze the data, teacher comment that was written on the student texts or given in the oral comments were recorded and written on individual index cards. The cards were sorted into groups of those with similar meanings using Lincoln and Cuba's ( 1984) method. The groups were then cross checked and again collated into groups by meaning or the problem they addressed. Seven groups were established, each of which addressed a different aspect in the texts (Table 1). At this stage in the study there was no differentiation made between oral and written comments, as those distinctions were covered in the quantitative phase of the study. Results and Discussion

Analysis of the teacher comment types resulted in seven comment types: positive comments, overall quality of all or a section of text, material that does not belong in the text, material that is out ofplace, wordiness or over length, wording or sentence needs work, and meaning is unclear. Student responses were examined based on each comment type. "Positive comments," as might be expected, did not result in a lot of revisions, although somestudents did revise some of these sections of their texts. All of the revisions resulting from these comments were improvements. Comments on the "overall quality of all or a section of the text" asked for big changes. Generally student response to these comments was deleting or substituting material in their texts, which is not surprising because the comments frequently related to coherence or focus. Again, the student revisions associated with this comment were generally improvements, but in some cases deleting material weakened the story. Student responses to the comment that to "material that did not belong in the text" also resulted in deletions. Students tended to act more frequently on some of these comments than others because a few of the comments offered more specific instructions ("Doesn't go in the story" vs. "How does this fit?"). Therefore, some of the comments were not acted on by the students, probably because ofuncertainty of how to solve the problem (Flower et. al., 1986). Generally revisions made in response to this comment group improved the student texts. "Material that is out of place" comments related to organization problems in student texts and usually suggested that the material belonged in the story, just not where the student had placed the information. As expected, students generally opted to rearrange their texts in response to this comment. Specific comments in this category resulted in more revisions that improved the texts than did less instructive comments ("Move Up" vs. "Out ofplace"). Students responded frequently to the less specific comments by removing the material. Comments on "wordiness" and "wording or sentence needs work" problems frequently resulted instudents deleting material from their texts. However, many students did a good job of combining sentences and paragraphs to tighten up the text and reduce the paper's

overall length. Getting just the right word can be a particular problem for student writers and comments in this area included "awkward," "choppy," or "vary sentence structure." Responses to these comments frequently resulted in fine tuning rather than fixing structural or coherence problems. Many revisions in response to these comments included combining sentences and altering sentence structure. A few resulted in deletions, but there were more substitutions used in response to this comment than to other comments. Again, the more specific the instructions, the more often the students revised successfully. "Unclear meaning" comments usually refer to the need for more information including specific detail or other clarifications. Some of the comments went so far as to ask for specific numbers or other specific information, others were "vague" and "confusing." Revisions resulting from this comment group varied including deletions and additions. The quality of the students' revisions also varied. Results from the qualitative portion of this study indicate that the more directive the teachercomment on student texts, the more successful student revisions will be on the text. Studentstended to respond to teacher comment if they knew how to make the requested change or improvement. If they did not know how to make the change or how to improve the text, they frequently deleted the material or ignored the comment. According to Spandel and Stiggins (1990),students frequently misread instructors comment and fail when they are trying to revise their texts. Occasionally students would substitute material, which ultimately resulted in a few additions. There were few rearrangements and those changes were usually in response to a specific comment to alter the order of ideas in paragraphs. In response to one of the main questions of this study, teacher comment "Does influence the choices students make in revising their texts," and a second question "Does the lack of teachercomment influence student revision?" Indications from the qualitative portion of this study are thatstudents are even more likely to make revisions in the absence of written comment when oral only comment is presented. As with the other student responses to teacher comment, students perceive a benefit from revising their texts based on the incentive of an improved grade.

The F test included all 64 of the students in the study combined into one large group. This option was chosen to maintain the natural classroom environment as much as possible. The data were coded by treatment and by revision outcomes. Based on this analysis, the only significant outcome at p <.10 was deletions. A Scheffe S test showed that marginal comment and oral comment only treatments were similar for deletions, as were marginal and end comment and end comment only treatments. However, marginal comment and oral comment only treatments were significantly different than marginal and end comments and end comment only treatments. This means that the students' responses to each pair of treatments were similar, but that they responded differently to the treatments not contained in each pair. The significance of deletions and the relationship between the two pairs of treatments provides several options for interpretation. Flower et al. (1986) suggest that if students do not know how to address a problem, they will frequently delete the material. That is likely the case in this study. second, the similarities between responses to marginal and end comment and end comment only suggest that students may be reading and interpreting these comments in much the same way. The same should be said for the other pair of treatments, marginal only comment and oral comment. Examining the means of these two treatments, .38 for marginal only and .51 for oral only, indicates that students made fewer deletions on average in response to these two comments than for the other comment pair. Deletion means for marginal and end comment and end comment only were .74 and .81 respectively. The students made more deletions based on these two treatment types. One obvious similarity between thee two treatments is that they both include comments on the students' texts. This may indicate that students either read these comments more often or that they somehow responded to these comments differently than they did to marginal only comment or oral only comment at least when it came to making deletions in their texts. Although the F test showed a significant difference for only deletions, the descriptive statistics associated with the various treatment totals are worth discussion. Of the 307 total revisions by type made by students in this study, 109, or 35.62%, were deletions; 85, or 27.28% were substitutions; 59, or 19.28%, were additions and 55, or 17.32%, were

rearrangements. Total revisions broken down by teacher comment location (Table 2 and Figure 1) were marginal and end comment, 83; end only comment, 80; oral only, 83 and marginal only, 61. The primary difference is in the revision types, with deletions showing a much higher incidence than substitutions; additions and rearrangements are fairly even at the lower end of the range. The high number of deletions is not unexpected (Flower, et al., 1986) Comparing comment location by revision type also provides an interesting discussion. For marginal only comment, there was a relatively low overall number of revisions and an even distribution (16 additions, 16 deletions, 15 substitutions, and 14 rearrangements a range of 2). One possibility for this relatively low number of revisions with this comment location relates to the lack of space available for providing feedback. Another reason may be that students do not read these comments. Bolker ( 1978) suggested they fear that students dissociate themselves from teacher comment because

disappointment. Also, while end comment often points out problems in a text, it is frequently tempered with positive comment and is sometimes less directed at a specific point or error in the text (Smith, 2004). Oral only comment elicited a relatively large number of revisions ( 18, 24, 24, and 17 a range of 7). This number of revisions is somewhat higher than anticipated at the outset of the study. One of the theories was that students receiving only oral comment on their texts would revise less because of the somewhat fleeting nature of the feedback. However, information on the audio tapes of the oral comment sessions suggests one reason for the unexpected strength of this response. The writtennotes for these sessions look, for the most part, like a laundry list of what was wrong (and occasionally right) with the class' texts. However, the audio tapes of these sessions include not only the comments on the problems, but usually examples of all or most of the problems pulled from student papers. The instructor did not return the students' texts until the end of the class period in an attempt to keep the students' attention on that day's material. Therefore, when the instructor went through the comment list, the texts were readily available for discussion. No student names were mentioned, but the students did ask questions and

apparently, from the number of revisionson their texts, were able to make use of the information. Conclusions and Implications Based on the results of this study, teacher comment influences student revision choices and the more directive the teacher comment, the better chance the students will revise their texts successfully. This agrees with Flower et al. (1986), Newkirk (1981) and Shuman (1975), but this study builds on their work by providing specific illustrations of teacher comment that offers problem identification and revision strategies paired with actual student revisions. The placement of written teacher comment does have some influence on student revisions. In this study, there were more total revisions associated with oral only comment than the other three types. The previously mentioned audio tapes indicate that the oral comment sessions frequently included multiple examples of a problem and multiple solutions. These additional examples may be part of the reason for the additional revisions. Another possibility may be Bolker's ( 1978) suggestion that students fear teacher comment and, because the oral comment is less direct, it is therefore less threatening and students are more apt to listen. Oral feedback may help build a sense of community rather than force students to view problems in the texts as theirs alone. The combination of research methods used in this study added strength to the conclusions of both portions of the research. For example, deletions were the only statistically significant response in the experimental study. This outcome could be explained more clearly using results from the naturalistic inquiry portion of the study. Matching specific teacher comments with specific revisions revealed that many of the comments suggested or hinted at deletion as a revision option. Also, the results of both portions of the study pointed to the importance of more detailed teacher comment either in the form of more revisions associated with concluding comments on the texts or the more frequent and more successful revisions from specific comments on the texts.

Several alterations would enhance future studies using this method. First, the use of Rodger's (1967) stadia of discourse for identifying changes in the texts was a little too coarse for the revisions involved. Reviewing changes at the word or phrase level would be potentially more accurate. Identifying a way to limit the variety of teacher comment statements by using a check sheet or other method would better focus the naturalistic inquiry portion of the study and help further identify which comments elicited successful revisions from the students. Implications for future research include further examination of oral comment as a feedback methodon student papers. The potential time savings for writing instructors, as well as the possibilities of greater improvement in student writing make this an important area for future study. Continued evaluation of written comment would also be extremely valuable. Careful evaluation of writtencomment location and content could lead to better writing and better use of instructor's time. Finally, one of the major goals of providing comment on student texts is to help the students learn to internalize the ability to evaluate their own texts. Identifying feedback methods that can helpstudents learn to evaluate their own writing more successfully will enable them to become better writers. Agricultural Communications programs frequently offer writing courses as either part of their curriculum or as a service to their colleges. Providing efficient and timely feedback on these papers is an increasing challenge as faculty work toward tenure or promotion with ever-growing student demand. Refining our methods of providing students with feedback on their papers will ultimately improve our students' writing ability while still making the most efficient use of faculty member time and resources the best of both worlds for all concerned.

The Importance of Feedback

by William Conerly

I'm a sucker for factory tours. Though this plant was noisy and humid, I learned a little about extruding plastic film and a lot about the value of feedback. The plastic film made by this firm let ions flow from one side to the other, with nothing else flowing either way. That's all the chemistry that I understood, despite the owner's best efforts to rectify the time I had spent snoozing in high school. The company's customers specified the quality of the film in terms of defects per roll. The rolls were large, with hundreds of feet of yard-wide film, and the defects were microscopic. The customers were demanding no more than 4,000 defects per roll, and producers had been having difficulty meeting that quality benchmark consistently. The company I visited had a defect counter. The film came out of the plastic extruder onto a drum. When the drum was full, it was carted over to the defect counter. The film came off the drum, through the defect counter, and onto another drum. The good rolls were shipped, the bad rolls were tossed out. It obviously pained the owner to toss out rolls because of poor quality. But that was theold way, he explained. Someone had the bright idea of saving a bit of work. The defect counter could be mounted on theoutflow end of the extruder, just as the film was going onto the drum. That would eliminate a step inthe production process, saving a few minutes of time for each roll. The defect counter was sophisticated in its detection of flaws, but lowtech in its counting of them: it had a mechanical counter, like the ones that are sometimes used to count customers entering a store. Each defect registered with a little "click." Like anyone would be, the extruder operators were bothered when the clicker started sounding off. Their job was to continuously monitor the conditions inside the machine, such as temperature, pressure, and humidity. Each important condition had its own meter, with a minimum and maximum noted on the dial,

and the operators had controls to change these factors. They soon learned that a long series of clicks meant that something was out of line, and they would quickly adjust their controls. Pretty soon the counter was generally quiet. The operators began reacting when they heard just a few clicks. The dials would show that everything was within tolerances, but perhaps not at the ideal levels. After a few months the operators learned that if they reacted as soon as they heard a click, they could reduce successive clicks. They also learned the "sweet spot" on each dial to prevent clicks. It required them to monitor their gauges more closely, but it also made the job more fun. Friendly competition developed to see who would have the fewest defects per roll. By the time I visited, the number of defects had fallen from 4,000 per roll to less than 20. That staggering improvement came about because the operators were getting immediate feedback on their performance. No foreman was berating them to do better. No bonus system offered incentives. They just wanted to do good work, and now they had the feedback they needed to do their jobs well. Before the defect counter was installed next to the extruder, an engineer might have been able to predict that with excellent operators defects should be as low as 20 per roll. But the operators would have scoffed at the ivory tower approach. "He just doesn't know what it's like to run theextruder all day long," they would likely have said. And they would have been right. It's very difficult to do one's job at the peak level without feedback. Feedback and the Marketplace Feedback often sounds like an engineering concept, but it's central to the working of a competitive economy. A business uses various resources to produce a finished product. Is that endeavor worthwhile? If the value of the finished product is greater than the value of the resources used, then the company has created additional value. It's doing something right and should be given positive feedback. In a free-market economy, that feedback is called profit.

Similarly, a company that uses highly valued resources to produce a product that is not highly valued should get negative feedback. It's called a loss. The case of the plastics extruder is fairly easy. A defect is a defect. It can be counted, and it is tied to the operator's attentiveness. Not all desired results are so easily quantified. The principle of feedback improving quality, however, still applies. Like many business owners, I find that the best feedback I get is from my profit-andloss statement. Black ink is good, red ink bad. But feedback is available in more ways than aggregate profits. Although the bulk of my work is economic research for businesses, I also give speeches to trade associations and other groups. The price I set for my speeches a year ago about matched the value of the time that I put into preparation and presentation. There was no extra value created directly, but I was hoping for some indirect value from the speeches as a marketing tool. That is, I hoped for more consulting projects because of my exposure to more business people. Over time, however, it seemed that all I was getting from the speeches were more speeches. But as a break-even proposition, there wasn't much reason to sell more speeches. In other words, the feedback suggested that the resources I was putting

into the activity just equaled the value of the activity. It wasn't harmful, but neither was it generating extra value. Then at the suggestion of a professional speaker, I tried raising my price. (Funny that an economist should get pricing information from a motivational speaker, but sometimes an outside opinion helps.)The first feedback I watched was sales level. It turned out that the volume of speeches didn't change. (It is possible that the volume would have grown in the absence of the price increase, but at least it didn't fall.) And the second feedback was significant: the time I was spending giving speeches was definitely more valuable than other possible uses of my time. I was adding value. That feels good in the abstract and especially good in the wallet.

Although we often think that low prices are good, prices that are too low generate improperfeedback to producers. They don't realize just how valuable their goods or services are, so they don't produce as much. Feedback is important not just because it gets us producing the right products at the right prices, but because the world is changing. Demand goes up, demand goes down. Technology improves, theprice of key supplies changes, human abilities change, and only a feedback process can get our activities in line with the new reality. Distorting Feedback Writing a tribute to feedback may seem as useful as a tribute to gravity or the beauty of rainbows. But feedback can be stymied, to the great detriment of the people who consume and work. In theplastics company it would not help at all to muffle the clicker. It would make for a quieter work environment, but product quality would certainly fall. Neither would it help to have extra clicking noises added when there were no defects. That would lead operators to adjust their equipment at the wrong time, possibly lowering quality. In other words, the feedback signal has to be undistorted. In my business, as in all other businesses, there are various government-mandated distortions of the feedback process. First, the taxes I have to pay on my income tell me that my time isn't very valuable to others. After federal and state taxes are paid, I get only $60 for every $100 that others are willing to pay me. When I compare the value of my time to other people against the value of the time I spend sitting on my boat, the tax system gives me improper feedback. Thus, I spend more time on my boat than I should (which is exactly the conclusion my wife reached without economic analysis). Price controls on the services I sell would similarly distort the feedback I get; controls would lead me to think that my services were not highly valued. Were I to receive a subsidy, the distorted feedbackwould indicate the resources I used were not very

valuable. Thus I would overuse those resources. Businesses are going to great lengths to increase and speed up feedback. New information systems are getting data to the employees responsible for the results. At many companies, sales people get immediate information on new orders; corporate buyers get immediate information about what products customers are purchasing; inventory managers get immediate information about what products need to be restocked. The value of feedback has helped fuel a large increase in business's use of information technology. Unfortunately, we have not been so fast to eliminate the distortions to feedback caused by government interference in the economy. But there is yet one other feedback mechanism at work. Countries whose governments have less involvement in the economy tend to grow faster, making their populations wealthier. That feedback process may come to dominate global economics. That sounds as good as a silent clicker. -1-

Give Positive Feedback

by Peggy Simonsen When we hear of the need for feedback from managers to employees, we tend to think only of corrective feedback-improving performance problems, for example. Performance problems are barriers that must be overcome of course, for success on the present job as well as for future opportunity. But feedback also needs to be positive to reinforce good behavior. It is the combination of both positive and corrective feedback that results in employees changing their behavior. I once had a manager say to me, "Why should I have to give positive feedback? They're only doing their job. That's what they're paid to do!" My response was, "Do you want them to continue doing it?"

Positive Feedback Motivates Positive feedback supports employee motivation. It is always nice to hear, but is essential when employees have gone the extra mile, overcome a crisis, or put in unusual effort to achieve a goal or meet a deadline. You could argue that is what they are paid to do; but if you value the behavior, positive feedbackrewards corrective feedback make positive feedback. A "good job" kind of comment is better than no response; but specific, targeted comments are more meaningful. For example, if you want to acknowledge an employee's willingness to work overtime when needed, you might just say thank you. But taking a few minutes to say specifically what you appreciated about the employee's efforts, the impact it has on the team or the output of work, plus a comment about the future, make positive feedback much more meaningful. Taking time to provide that kind of positive feedback reinforces the very behavior you value (See side bar). Sometimes positive reinforcement is needed after the employee demonstrates the corrected behavior. Let's look at an example: Joe creates a negative climate in the work group. He's a complainer, particularly about company systems and procedures. His manager recognizes that Joe's attitude and complaints have a negative impact on the team, and must be addressed. As a first step, the manager identifies specific occasions where she observed Joe's behavior that she wants to address. In preparation for giving feedback, she identifies the impact of the negative behavior on the team and on Joe's reputation in the firm. She prepares examples of the changed behavior she wanted Joe to work on and then met with Joe to discuss it. Her purpose in addressing what some might consider "personality characteristics" is to help Joe change the behavior that was getting in the way of the team's work and Joe's success. The Second Step. Joe's manager knows the first step in changing behavior is the awareness of it as a problem, and so the need for corrective feedback. The second step is willingness to work on theproblem, and her approach in giving feedback helps Joe accept the problem instead of becoming defensive. Together they identify actions that could contribute to Joe's efforts to change, because behavior change requires practice of the new, desired behavior. As the manager, she is willing to deal with some of the procedures that cause Joe's frustration and negativism. She knows also that positive reinforcement would be needed each time she observed the desired behavior by Joe. and it reinforces valuable it. and Just as the steps so to giving to more actionable, do the steps

The second

step

of the feedback process is

identifying the impact

of the behavior

on the employee's goals or the team's success. Career discussions with an employee, help the supervisor know what his or her goals are and what must be done to address a behavior change accordingly. It becomes a clear case of "what's in it for me?" In the case of Joe, the impact of the negative attitude might be stated something like this. "Joe, you have identified a goal to have more opportunity to work on cross-functional teams to broaden your base of expertise. If your reputation for complaining precedes you, other team leaders will be less likely to select you. Even with your technical expertise, you may not be seen as a team player." In stating the desirable behavior, Joe's manager might say something like, "What teams need is someone who can help analyze the pros and cons of a situation without sounding critical of others. Let's discuss what you need to do to change your approach." Behavior as a Controllable Factor Giving corrective or positive feedback requires the manager to be aware of individual employee's behavior. It means approaching behavior as a controllable factor, not writing it off as an unchangeable character attempts trait. to After change, providing corrective feedback and change hasn't reinforcing the employee's if the desired

occurred, the manager has a choice to make. Should he or she persevere or put his or her energy into developing someone who is more responsive? The manager is not being asked to be a therapist just to make sure his or her employees are clear about expectations and provide some guidance to move in the right direction. A good manager is one about whom people say, "he or she always gave me good feedback. He or she acknowledged when I did something well and corrected me when I needed it." Reprinted with permission of Insight, the magazine of the Ilinois Society of CPAs -1-

The

Effects of Hands-On Experience onStudents'

Preferences for Assessment Methods.

by Katrien Struyven , Filip Dochy , Steven Janssens Although changes, invention, creation, design, progress, in short, innovation, are concepts that represent the key issues of educational policy, they are often a source of anxiety and concern for many teachereducators and teachers themselves (Hall, 1979; Van den Berg & Ros, 1999). They are reluctant to readdress, redesign, and reorganize the teaching practices with which they feel comfortable. Consequently; proposals for educational innovation are critically appraised and a negative disposition toward the unknown is apparent (Hargraeves, 2000). There is often professional unease about unknown subjects, situations, or innovations. It might be an atavistic fear of change, anxiety about chaos, or professional concern for conserving established values and standards (Cuban, 1998; Kelchtermans, 2005; Waugh & Punch, 1987). It is only when familiarity grows with these subjects or situations that fears are allayed and the new, changed situation is accepted (Van den Bergh, Vandenberghe, & Sleegers, 1999; Van de Ven & Rogers, 1988). Fullan (2001) describes the process as follows:
Real by works and uncertainty subjective change, whether and and result the it can and desired collective uncertainty, in a of sense growth. joys of or not, and of The mastery are represents if mastery, the a serious change of to and the to personal out experience characterized accomplishment

ambivalence

professional meaning

anxieties central change,

educational

success or failure thereof. (p. 32)

Positive

experiences

with

the

change,

embodied

in

the

sense of mastery,

accomplishment, and professional growth, define the success of the change and its continuation in practice. Moreover, ifstudents are thought of as "participants in a process of change and organizational life" rather than as "potential beneficiaries of change" (Fullan, 2001, p. 151), involving students when studying change and understanding educational innovation comes in natural. In fact, during the process of change, studentsmight suffer similar feelings of uncertainty and ambivalence at the start and joys of mastery, accomplishment, and academic growth when change has

proven to work. In this respect, student teachers in teacher training programs are interesting subjects. On one hand, they are students in the process of change when experiencing new teaching methods or assessment modes; on the other hand, they are to serve the function of teachers implementing change in practice. In addition, empirical evidence repeatedly has shown that teachers tend to teach as they were taught (Johnson & Seagull, 1968), often failing to use the innovative techniques that have been advocated in training (Grippin, 1989). Rather than delivering information about engaging and innovative teaching practices through traditional approaches, modeling the use of these teaching methods serves the purpose (Loughran & Berry, 2005). Combining the abovementioned arguments, the assumption is made that the modeling and use of new assessment techniques in teacher training might generate initial fearful dispositions with student teachers toward the changes, fears that might be allayed when familiarity grows and the changes tend to work out, with feelings of mastery, accomplishment, and professional/ academic growth as consequences, defining the (possible) adoption of the change in student teachers' current and future teaching practices. Hence, the effects of student teachers' hands-on (read: actual) experience with new modes of assessment in teacher training on their preferences for evaluation methods in general, and the experienced method in particular, are investigated. This study not only examines the dynamics ofstudents' preferences with respect to four assessment methods that follow a standardized course onchild development but also aligns instruction and assessment as a traditional learning environment is compared to a student-activating type of instruction that is followed by one of four assessment types. In contrast to former research on students' assessment preferences (Birenbaum, 1997; Darling-Hammond & Snyder, 2000; Sambell, McDowell, & Brown, 1997; Zoller & Ben-Chaim, 1997), studies that have adopted case-study designs, qualitative methods of data gathering, or survey inquiries at one moment in time, the present study is unique for its use of a quasi-experimental design by means ofquestionnaires administered in three different occasions, that is, prior to, during, and after student

teachers have had experience with a particular assessment method. In recent years, the repertoire of assessment methods in use in higher education has expanded considerably. New modes of assessment have enriched the conventional evaluation setting, formerly characterized by both the multiple-choice examination and the traditional evaluation by essay (Sambell et al., 1997). More recently, portfolios, selfand peer assessment, simulations, and other innovative methods were introduced in higher educational contexts. The notion of alternative assessment is in this respect often used to denote forms of assessment that favor the integration of assessment, teaching, and learning; the involvement of students as active and informed participants; assessment tasks that are authentic, meaningful, and engaging; assessments that mirror realistic contexts; and assessments that focus on both the process and products of learning and move away from single test scores toward a descriptive assessment based on a range of abilities and outcomes (Sambell et al., 1997). The present study compares four assessment methods: one conventional multiple-choice test and three alternative assessment methods, namely, case-based evaluation, peer assessment, and portfolio assessment, and investigates (the dynamics in) student teachers" assessment preferences. A profound study of educational literature on students' assessment preferences

confronted us with three types of studies. First, there are the studies that require students to express their preferences concerning two contrasting evaluation methods. These studies often relate student characteristics to preferences for particular assessment types. For example, Zeidner (1987) concludes that both female and male high school students prefer the multiple-choice format to the essay type of examination. Also Traub and MacRury's (1990) scholastic students report more positive attitudes toward multiple-choice tests on the grounds that these examinations seem easier to prepare, easier to take, and may produce higher relative scores. Nevertheless, these results do not apply to the entire group of students. Birenbaum and Feldman (1998) discovered that university students (in social sciences and arts) with good learning skills, who have high confidence in their academic ability, tend to prefer the essay

type ofassessment to multiple-choice examinations. Conversely, students with poor learning skills, who tend to have low confidence in their academic ability, prefer multiple-choice testing to the constructed-response type of assessment. Results also show that low test anxiety measures were related to positive attitudes toward the essay format. Students with high test anxiety have more unfavorable attitudes toward openended format assessments and a preference for the choice-response type. In contrast to Zeidner (1987), this study also indicated gender differences, with men having more favorable attitudes toward the choice-response format than do women (Birenbaum & Feldman, 1998; Struyven, Dochy, & Janssens, 2005). The second category of studies also investigates students' assessment preferences for several evaluation methods. In reference to asking students to indicate their appreciation of a whole range ofassessment methods, Birenbaum's (1997) results on university students' assessment preferences demonstrate that the highest preference was for teacher-guided test preparation. Next in order came nonconventional assessment types and student participation, followed by higher order thinking tasks and integrated assessment. The least favored category was oral exams. Similarly, Zoller and Ben-Chaim (1997) found that oral examinations of all types were rated as least preferred by college students in both the United States and Israel. The higher stress level associated with oral exams, and the consequent poor performance, are the main reasons for students' dislike of this type of evaluation. College science students' preferred assessment method is nonconventional, written exams in which time is unlimited and any materials are allowed. Moreover, American students like the traditional writtenexam significantly more than do their Israeli counterparts (Zoller & Ben-Chaim, 1997). Characteristic ofboth categories of studies is that students' assessment preferences are not related to a particular teaching context in students' higher education programs. Moreover, preferences are measured at one moment and are treated as student characteristics that are fairly stable over time. However, there is evidence to suggest that students' preferences change due to handson experience with a particular teaching environment, which is the focus of the present

study. Many researchers have interrogated students about their assessment preferences after experiencing alternative methods ofassessment. By means of the case study methodology, Sambell et al. (1997) tried to unveil universitystudents' interpretations, perceptions, and behaviors when experiencing different forms of alternative assessment. Broadly speaking, they found that students often reacted negatively when they discussed what they regarded as "normal" or traditional assessment. Exams had little to do with the more challenging task of trying to make sense and understand their subject. In contrast, when studentsconsidered new forms of assessment, their views of the educational worth of assessment changed, often quite dramatically. Alternative assessment was perceived to enable, rather than pollute, the quality of learning achieved (Sambell et al., 1997). However, students repeatedly express that a severe workload tends to alter their efforts in studying (Sambell et al., 1997). Similarly, Slater (1996) found thatstudents in their first year of university physics education had come to like portfolio assessment.Students thought that they would remember much better and longer what they were learning, compared with material learned for other assessment formats, because they had internalized the material while working with it, thought about the principles, and applied concepts creatively and extensively for the duration of the course. Students enjoyed the time they spent on creating portfolios and believed it helped them learn. Segers and Dochy (2001) found similar results in students' perceptions about self- and peer assessment in a problem-based learning environment setting at university (with law and educational science students). Students reported that these assessment procedures (or should they be called pedagogical tools) stimulate deeplevel learning and critical thinking. In addition, Macdonald (2002) finds evidence for students' preferences being affected by course experiences at the Open University. In contrast to the feedback received from the undergraduate students, all of the postgraduate students exposed to a project as an end-of-course assessment expressed satisfaction with the project as a final examinable component (Macdonald, 2002). Although it is clear from these results that students' assessment preferences are affected by their experience with a particular assessment method, the lack of pretest measures makes it hard to pinpoint the effect of the experience with evaluation on students'

changes in assessment preferences. The purpose of this study is to provide evidence about the effect of student teachers' hands-onexperience of a particular mode of assessment (e.g., multiple-choice test, casebased exam, peer assessment or portfolio) on their assessment preferences using a threewave longitudinal design. The research project attempts to test the following hypotheses, reported in this article: (a) unknown assessment methods are regarded negatively; (b) as familiarity with an assessment method grows, assessment preferences will change positively; and (c) students' preferences are congruent withstudents' perceptions of the appropriateness of the assessment methods. RESEARCH DESIGN Sample The investigation had a quasi-experimental design because of the authentic educational context in which the study was conducted. Learning materials on the content of child development (a course book, a booklet of assignments, and four evaluation formats) were developed for a set of 10 lessons. These were delivered in five research conditions involving 664 students in their 1st year of the elementaryteacher training program of eight participating institutions in the Flemish part of Belgium (N = 664).Students were primarily women (83%) age 18 to 20 years. Lecturers, responsible for the instruction ofthe course on child development in diverse teacher education institutions in Flanders, were invited to participate in the experiment. Of these, 20 highly motivated, qualified ([greater than or equal to] 5 years of experience in teaching) lecturers participated in the study. Conversations with (the teams of) lecturers in each institution, prior to the experiment, revealed that their interest in active teaching, preliminary (positive) hands-on experiences in class, and the prospects of gaining insight into the effects of student-activating teaching methods and assessment on student learning and perceptions were the foremost intrinsic motives mentioned. The clear-cut student-activating teaching/learning materials to be received was the primary extrinsic motive. During this conversation, each team ofteachers within an institution expressed

their preferences toward the treatments in the study. First choices of assessment-byinstruction could be assigned to the teacher training lecturers. Student teachers, following the course on child development, were randomly distributed between the classes and lecturers. Apart from the lecture-taught students who followed instruction in large groups (approximately 70 students in each group), class sizes in the activating conditions were limited to a maximum of 40. However, due to absenteeism and the phenomenon of dropout in the 1st year of higher education, class sizes were usually smaller (see Table 1). Research Conditions In all, there are five experimental groups in this investigation: one lecture-based setting and four student-activating learning environments characterized by one of four assessment modes. Student-activating teaching methods, as defined in this study, challenge students to construct knowledge by means of authentic assignments (a) that literally require their active participation in the learning/teaching process to incorporate the available information by means of discovery learning (b) and with the assistance of a scaffolding teacher (c) who is continuously available to help and coach the student teachers. These three characteristics are essential features of the teaching methods that are used to activate students in this study. The experimental conditions are as follows: Le: lecture-based learning environment + multiple-choice examination (N = 131) Mc: student-activating learning environment + multiple-choice examination (N = 119) Ca: student-activating learning environment + case-based assessment (N = 126) Pe: student-activating learning environment + peer/ co-assessment (N = 174) Po: student-activating learning environment + portfolio assessment (N = 114) The first group of preservice teachers (Le) was instructed in the content of the child development course within a traditional lecture-based learning environment

characterized by formal lectures and assessed by means of a multiple-choice examination format. The other four groups (Mc, Ca, Pe, Po) learned in the same studentactivating learning environment, characterized by the same learning content and teaching methods. However, the assessment mode that accompanied this learning setting distinguished the conditions from one another. Each group was assigned to a different assessment mode, namely, (a) a multiple-choice test (Mc), (b) a case-based assessment (Ca), (c) a peer/coassessment (Pe), and (d) a portfolio assessment (Po). Because teacher education has a fairly uniform structure in Flanders and a similar intake of 1st-year students (e.g., gender, socioeconomic status, ethnicity, etc.), each institution for teacher education was assigned one of these learning environments. Consequently, all participating student teachers within a particular school were treated the same way by means of standardized learning materials. This procedure precludes bias in the results due to students in different classes comparing one treatment to another. A lack of differences in pretest measures ratifies that groups might be considered comparable (see also Struyven, Dochy, Janssens, & Gielen, 2006). The course on child development is compulsory within the 1st year of the program for preservice elementary school teachers. Table 1 provides a numeric representation of the matching procedures of research conditions, schools, lectures, classes, and student teachers involved in the present study. Learning Materials for the Course on Child Development The learning materials consist of three parts: a course book, a booklet of studentactivating assignments on the contents studied in the course book, and the four assessment methods that were used to evaluate student performance in the course. The course book. The content of the course book (Struyven, Sierens, Dochy, & Janssens, 2003) concerns the development of the child, from conception throughout the school age period until adolescence. Each developmental stage in the book was treated in the same way, by a thematic sequence of the domains that characterize the growth of each child (e.g., linguistic development, physical development, motor development, and cognitive

development). The student-activating assignments. Apart from the lecture-based learning environment, all students in the experiment were instructed in the content by means of studentactivating assignments (e.g., problem-based learning tasks, case studies, teamwork, etc.), in which students were challenged to become active learners who construct their own knowledge. The assignments required students to select, interpret, acquire, and manage the information available, aimed at its application in real-life cases or solutions to authentic, relevant problems. The teachers' role within the student-activating learning environment was restricted to the supervision and the coaching of the students' learning processes while they were tackling these tasks. The assignments were collaborative in nature (6-8students) and required shared responsibilities from students. Ten lessons, each of approximately 1.5 hours, grouped the assignments and were the same in the four student-activating groups. Detailed instructions with the assignments were provided to direct both students and teachers. As well as the pre-constructed, standard learning/teaching materials for students and teachers, randomly selected observations in the classes of participating teachers ensured the intended, standardized implementation of the treatments. The assessment method. Bearing in mind the importance of alignment between instruction and assessment (Biggs, 1996; Segers, Dochy, & Cascallar, 2004), the lecturebased setting was only followed by the multiple-choice test, whereas the case-based evaluation and peer and portfolio assessment required working with the studentactivating assignments. With the exception of one group (Le), students were largely unfamiliar with the four assessment methods and the majority of studentshad no prior hands-on experience of the methods. All methods address the content(s) (and assignments) of the course on child development and aim to measure similar domains of knowledge from acquisition and insight to application and problem-solving skills, the emphasis lying on the latter. Dependent on the sole summative function (Le, Mc, and Ca) or the combined formative function of assessment (Po, [Pe.sup.1]), the purposes of the assessment methods are, respectively, to measure individual differences and to assist

learning in combination with the assessment of individual achievement (Pellegrino, 2002). Each assessment method is discussed in detail and an overview of the characteristics of the tools and procedures is presented in Table 2. The upper half of the table illustrates the characteristics of the end-of-course examination or assessment conversation, whereas the lower part emphasizes the assessment procedure as it is embedded in the course. Multiple-Choice Test The multiple-choice test followed the set of classes and included 20 questions with four multiple-choice answers. Only one choice offers the correct solution to a question. To avoid pick/guess answers, a correction for guessing was used, that is, wrong answers were punished by a withdrawal of 1/3 point (correct answer = 1 point, blank answer = 0 points). Items were equally divided into four categories relating to knowledge, insight, application, and problem solving. Whereas the first category only aims to measure knowledge acquisition, the other categories surpass this level, aiming at the measurement of knowledge construction (Birenbaum, 1996). The examination was a closed book evaluation. Case-Based Assessment The case-based assessment in this study was developed according to the principles of the OverAll Test (Segers, 2003) and concerned the case of Miss Ellen, who is a teacher in the 3rd year of elementary school. The case material involved a set of information and documents on the class and pupils, including the following: the class report for the first trimester, the detailed planning of a week, the thematic planning of the whole school year, the floor plan of the classroom and its play materials, a medical record on Robert, and a letter from Cindy's mother. Students received the case materials after the final lesson so they could use them to prepare for their examination. The examination questions remained confidential and all concerned this case study, which required students to apply the theories concerning child development to this

case. Students were allowed to use all resources available (e.g., course book, case documentation, articles, etc.) to complete their examination. Peer/Coassessment Peer assessment has been used to differentiate between the individual contributions to small group projects (Topping, 2003) and is a cooperative process of peer and teachers' involvement. In particular, three problem-based assignments related to the contents of child development were the subjects of the peer evaluation. These assignments required the preservice teachers to work together in groups of six to eight individuals. The peer assessment involved group members scoring their peers, and themselves, on the learning and collaborative processes within the group and the contribution of each group member to the task's solutions. Scoring was anonymous. Student teachers had to indicate for each individual in the group, including themselves, whether he or she contributed more (score 3), equal to (score 2), or less (score 1) than average or if the student made no contribution to the group's work at all (score 0). A calculation of these scores resulted in an individual peer factor (i.e., an approximate score of 1) for each of the members of the group. A peer factor score lower than I indicates studentteacher's efforts that are below average and affect achievement negatively, whereas a score higher than 1 suggests above-average work and affects the student teacher's performance positively. As already mentioned, the peer assessment method in the present study is a cooperative method of student teachers and the lecturer. The starting point of student teacher's assessment is the product of the group, namely, its solutions to the authentic problem set in the assignment, which is scored by theteacher. Depending on the individual peer factor (cf. approximate score of 1), this teacher's score was increased or decreased, which could lie above or below the average score of the group, resulting in an individual score for the student teacher that was not only dependent on the student's efforts and contribution to the group's work but also was dependent on the quality of the group's work. Correction for favoritism was introduced by eliminating the highest and lowest peer score for each student. To conclude the peer assessment procedure, an oral group assessment

conversation on the group's assignments was arranged. Please note that student teachers were to be informed of their individual peer factor (cf. the approximate score of 1) after each of the three assignments so peer assessment would serve formative assessment purposes as well. However, due to time restrictions and excessive workload, the lecturers were not able to provide student teachers with formative feedback on their peer factor, opportunities of assisting preservice teachers' learning for future tasks being missed. Portfolio Assessment The portfolio assessment consists of a portfolio, including the portfolio-constructing process, and an assessment interview of the student by the teacher. The portfolio document contained a selection of activating assignments tackled during classes, elaborated by students" reflections on their own experiences and learning (Davies & LeMahieu, 2003; Janssens, Boes, & Wante, 2001). For example, remarkable experiences during teaching practice, or from their own childhood, had to be applied to the content studied in the selected assignment. The evaluation of the portfolio served both formative (during the lessons by means of intermediate feedback sessions) and summative purposes (during the final portfolio conversation; Black & William, 1998). Only after the final assessment conversation between student and teacher on the definitive handedin portfolio was a final score given to the quality of work and effort that the student had invested. The scores for the course on child development were solely determined by the score on the examination in the multiple-choice (Le and Mc) and the case-based (Ca) conditions, whereas students' work on the group assignments during the lessons and on the portfolio was included in students' final marks for the peer (Pe) and portfolio conditions (Po). Apart from the multiple-choice examination (Le and Mc), all assessment methods took place by means of an open-book format that allowed the use of resources. The multiple-choice examination (Le and Mc) and case-based evaluation (Ca) were individual writtenexaminations, whereas both the peer (Pe) and portfolio

assessments (Po) were accompanied by an oral assessment conversation, in groups and individually, respectively. These assessment conversations mainly concerned the submitted work, namely, the problem-based assignments and the definitive portfolio. Although the assessment formats included in this study are new to the students, it would be incorrect to conclude that exposing students to unfamiliar assessments in the final exam is an acceptable practice. Thorough preparations were, of course, made for students to become informed and skilled examination participants. For instance, information on the assessment method was given during the first lesson of the course and sample questions were distributed beforehand. The methods, including formative assessment, portfolio (Po), and peer assessment (Pe), also included informative directions from the teacher and a try-out session was run. Although the goals to be achieved and the skills needed during the assessment processes might differentiate between the methods, the assessment methods concern similar contents and assess comparable domains of student knowledge. Notwithstanding the fact that assessment methods are not interchangeable objective-wise, abstraction is made as the influence of each assessment mode onstudents' assessment preferences is central to this study. However, these differences between the assessment methods need to be carefully born in mind given the purposes they serve for explanation. Longitudinal Measurements Data collection on student assessment preferences was obtained using a longitudinal design by means of questionnaires. Three moments of measurement were selected: (a) at the end of the first lesson when students had been briefly introduced to the learning environment and evaluation method for the course (pretest 1, Lesson 1), (b) at the end of the 10th and final lesson before preparing for the examination (pretest 2, Lesson 10), and (c) immediately after students had experienced their expected mode of assessment (posttest, exam). Items on the questionnaires were either closed or open-ended. To control for social desirability in responses, student teachers attending the course on child development were reassured of the anonymous processing of the questionnaires.

Moreover, their lecturers would not have access to their personal responses. The comment was given to student teachers that there were no wrong or correct answers to the items, only honest or dishonest answers. Obviously, the honest answers represented the desirable kinds that are aimed for. Students' preferences for different assessment methods were measured at the end of Lesson 1 and after experiencing their assessment method. On both occasions, students were given an identical list of 13 assessment methods, for each of which they had to indicate their appreciation on a 5-point Likert-type scale ranging from I really like this method (score = 5), I quite like this method (score = 4), I have no explicit opinion, I'm undecided (score = 3), I slightly dislike this method (score = 2), to I really dislike this method (score = 1). The presence of at least five response categories is required for ordinal-treated-as-interval measurement scales (Likert, 1932). Because students may be unfamiliar with or have no distinct preferences for some assessments, a neutral response category was included. The 13 evaluation methods included in the list are as follows: (a) multiple-choice examination, (b) individualwritten (closed book) examination (i.e., typical open questions with short answers), (c) individual oral (closed book) examination, (d) individual (written) open book examination, (e) take-home examination, (f) practical examination (e.g., simulation, internship-exercises, etc.), (g) case-based examination, (h) paper/essay, (i) portfolio assessment, (j) oral examination in a group, (k) peer assessment, (l) selfassessment, and (m) cooperative assessment. The unconventional or alternative assessment methods (such as 5, 6, 7, 9, 10, 11, 12, and 13) were associated with a basic one-sentence definition to avoid misinterpretations and to make sure student teachers understand the methods in the same and intended way. At the end of Lesson 10, students were asked for their perceptions of the expected assessment method, that is, before they had hands-on experience during the examination period. Students had to explain how they felt about the method of assessment that accompanied the course. They had to tick one of the following answers: very good (++, score 5), good (+), moderate (+/-), weak (-), or very weak (- -, score 1). Blank space was

provided for optional, additional comments. After the examination, student teachers were not only asked about their assessment preferences by indicating the extent to which they liked or disliked the methods from the list but students' perceptions of the appropriateness of the assessment method they had experienced (i.e., one of the four examination formats mentioned above) were examined. Student teachers had to indicate whether they found the assessment mode they had experienced appropriate (yes) or inappropriate (no). In addition, if students thought their method to be inappropriate, they could suggest a more suitable assessment format in the blank space provided. RESULTS The results are reported in three discrete sections, corresponding to the hypotheses being tested. Descriptive statistics, (nonparametric) table statistics, and ANOVAs are applied, including Bonferonni comparisons and effect sizes ([R.sup.2] and Cohen's d). Before discussing the three hypotheses central to this article, an overall ranking of students' preferences for the various assessment methods is given to provide an overview. Generally, students prefer the assessment modes in the order represented in Table 3. According to the students in this study, the most familiar and convenient assessment mode, namely, the individual closed book written examination, is generally most popular. This is followed by the practical examination and then the individual closed book oral examination. Preferences were scored on a 5-point Likert-type scale: Consequently, only the methods with a score of less than 3, such as the oral group exam and the peer assessment format, are actually disliked by students. Although most alternative assessment modes and methods that allow for the use of resources at home or at the examination moment are considered to reduce stress and test anxiety (Zoller & Ben-Chaim, 1988), they score only moderately (approximately score 3) in the list of student preferences. Please note that high standard deviations (> 1.06) are reported, suggesting that student teachers' opinions are (more) widespread.

Hypothesis 1: Unknown assessments, unloved assessments. Are unknown assessments regarded negatively? When student teachers were asked to evaluate their course on child development during the final lesson, the answers for assessment method reflect the distribution among the research conditions shown in Table 4. Table 4 clearly shows that the lecturebased students (Le), as well as their fellows in the active learning environment with portfolio assessment (Po), are most satisfied with their expected mode of assessment. In contrast, student teachers in the student-activating learning environments with multiplechoice testing (Mc), case-based examination (Ca), or peer assessment (Pe) think negatively about their forthcoming evaluation method. The Kruskal-Wallis test for k independent groups reveals statistically significant results ([chi square] = 37.34, df = 4, p = < .0001) and the Bonferroni mean comparisons replicate the differences between the lecture-based setting (Le) and three other conditions in the study (Mc, Ca, Pe) and the activated portfolio condition (Po) and the peer assessment condition (Pe). The additional comments associated with this item, and the abovementioned values, reveal interesting arguments in both positive and negative directions. Because the question for additional information was optional, only about half of the students expressed their opinion and their answers might not reflect the group's perceptions accurately. Therefore, the choice was made not to quantify students' additional comments. Nonetheless, these arguments provide valuable information with regard to the characteristics of the tools that are preferred or disliked. Caution is recommended with respect to the generalization of these results. An important positive argument that associates the alternative assessment methods is that there is no real examination or exam that is related to cramming and learning by heart in the perception ofstudents (e.g., "cramming for the exam is unnecessary" [Po], "no 'real' examination" [Pe, Po], "too much information available to study/cram" [Ca]). Another positive argument sounded in both the alternative and the traditional assessment conditions is that theory and (own) practice are interrelated (e.g., "theories need to be put

into practice/applications

of theory are more important" [Po, Ca], "stress

is onapplications" [Le], "compatible with my interests and experiences" [Po]). Both arguments might have given rise to the experienced instructive and informative nacre of the assessments in the present study (e.g., "instructive/ informative" [Po, Pe, Ca], "open book format, looking up information is fascinating" [Ca], "intellectually challenging questions" [Le], "questions cover the range of the course, you know whether you really understand everything" [Le]). In addition, several answers are relegated to the rest category that pinpoints particular features of the diverse methods in the study. For example, for the peer assessment, student teachers mentioned that you learn how to deal with the evaluations of your work by others than the teacher (Pe) and students appreciated in the multiple choice format that "littlewriting is needed" (Mc/Le) and "the correct answer is already shown" (Mc/Le). These arguments contrast highly with the negative comments that were given to ratify poor scale scores. Perhaps the lack of a real examination in the alternative assessment method conditions has led student teachers to feelings of unattained mastery of the contents (e.g., "Do I really know everything about the contents in the course?"/"I did not grasp the information on child development" [Po], "I never actually study the theories and information in the course book" [Ca, Pe, Po]). Moreover, other negative arguments tend to associate the particular procedures of the method of assessment. For example, some student teachers in the portfolio condition (Po) thought working on the portfolio required "much efforts" and associated with "high workloads." Peer assessment student teachers (Pe), on the other hand, were more concerned about the "fairness" of the procedure, about the difficulties they experienced assessing other peers, and reported the problem of absent team members for adequate peer assessment execution (e.g., "subjective"/"favoritism"/"appropriate evaluation method?" [Pe], "difficult" [Pe], "not allstudents attend classes regularly" [Pe]). These students tend to experience unease about the peer assessment scoring procedure and how it is going to affect their scores (e.g., "what is my final score going to be like" [Pe]). This comment might have been strengthened by the poor(er) implementation of the peer assessment procedure because due to time restrictions and high workloads of the lecturer, formative

interim feedback on the peer factor was not given to student teachers while working on the assignments. With regard to the case-based examination (Ca), some student teachers considered the method "easy" but regretted that their efforts on the assignments during the course did not count in their final scores (e.g., "the assignments in class should count for our final score" [Ca]). Finally, the multiple-choice test (Le/Mc) also generated negative arguments that associated poor attitudes toward this evaluation format, such as "answers resemble each other" (Le), "no additional comments and support are allowed" (Mc/Le), and the large amount of contents covered by the course made studentsfeel that there was simply "too much information to study" (Mc/Le). It should be noted that the portfolio (Po) and peer assessment (Pe) conditions had prior hands-onexperience with the formative part of the evaluation method and the lecturetaught group (Le) experienced a partial multiple-choice format examination at the time this question needed to be addressed. Consequently, Hypotheses I and 2 that unknown assessments are unloved tend to be true, with the exception of peer assessment (Pe), which received the lowest appreciation scores despite some hands-on experience (possibly because the formative function was inadequately served?). Hypothesis 2: As familiarity with an assessment method grows, students' preferences for it change positively. Evidence for this hypothesis is provided by means of the pre- and postmeasurements of student teachers' preferences for the listed assessment methods. If students' appreciations of the methods between the first lesson and after their examination are compared by research condition, each experienced assessment format benefits from the experience of the respective group of students, as demonstrated in Table 5. Paired t tests show the statistical significance of the differences. Cohen's d embodies the effect size of the comparisons and the results of the Bonferroni method are included in the notes for Table 5. The actual experienced assessment mode, as well as the related methods, are underlined. Positive t values demonstrate gains in student teachers' preferences, whereas negative scores represent losses of students' appreciation.

The results are remarkable and the four assessment methods and related modes display (significantly) higher preference scores after the student teachers have had handson experience with them during the course on child development. For example, the multiple-choice test at the end of the course is remarkably more favored, compared to the beginning of that same course in the multiple-choice conditions Le and Mc. In effect, the multiple-choice examination becomes the most popular method in the studentactivating multiple-choice condition (Mc) and second best, behind the individual written, closed book exam, in the lecture-taught group (Le). The same results are found in the case-based condition (Ca), where both the open book format and the case-based examination benefit from the hands-on experience of students and become the most popular assessment formats for these students. Although self, peer, and cooperative assessments did not reach the top 3 of preferred evaluation methods according to the peer students (Pe), they all improved in preference scores, as did the oral examination in groups that these students experienced. The related open book format and the take-home exam also gained popularity in this condition. Only the portfolio students (Po) did not express statistically significant gains in preference for the experienced method. Nonetheless, student teachers' preference means rose and the portfolio assessment achieved second place in evaluation preferences behind the individual oral examination, which would approximate to the format of the final assessment conversation on the definitive portfolio were it to allow the use of resources. The preference levels of custom evaluation methods, such as number 2 and 3 in the list, have fallen. In particular, the appreciation for the individual written, closed book examination suffered losses in all research conditions. The preference level of the individual oral (closed book) examination also fell significantly in the lecture-taught (Le), the active multiple-choice (Mc), and case-based (Ca) conditions. A decrease in preference is revealed for the practical examination. Furthermore, Table 5 reveals significant changes in preference that might be provoked by experiences outside the course on child development. For example, lecture-taught students (Le) seem to feel less favorable toward the case-based examination, the

paper/essay type of exam, and portfolio assessment, whereas this trend is not apparent in the active multiple-choice condition (Mc). These latter students tend to like the open book format more at the end of the academic year but tended to dislike peer assessment. Similarly, explanations for the shift in preferences for the paper, portfolio, and cooperative assessment in the case-based (Ca) condition remain unknown. On top of that, students in the peer assessment (Pe) condition tend to prefer the multiple-choice format significantly more after the examination, whereas this trend is not apparent elsewhere. These results are inconsistent with the assumptions and are different for the various research conditions. Therefore, it is likely that other experiences, in other courses, during the student's 1st year of elementary teacher training might explain these phenomena. This table also suggests the first hypothesis, that unknown assessments are negatively regarded, to be true. Additional evidence is provided by the ANOVA analysis of the four concerning assessment methods by research condition for both moments of measurement, represented in Table 6. The ANOVAs clearly demonstrate the increase in preferences after student teachers' hands-onexperience of the assessment mode during the child development course. Whereas at the start of the lessons (L1) students in different research conditions tend to think similarly about the four assessment methods in this study, their preferences after their evaluation experience (Ex) differ between the research conditions. One exception is the active multiple-choice (Mc) students, who tend to dislike their assessment method beforehand more strongly than their fellow students in other conditions, whereas afterward, students consider the multiple-choice test to be their most preferred method. Comparable a posteriori likeliness is shown by the lecture-taught students (Le), who favor this method more highly than their fellow students in the case-based (Ca) and portfolio (Po) conditions. The assessments in the other three conditions (Ca, Pe, Po) gained popularity after students' experiences with that method to the same extent, compared to the initial preference scores. Results are statistically significant. The increasing [R.sup.2]s between Lesson I and the examination moment also support

the effect of the hands-on experience of students' preferences for that particular assessment method. In addition, similarities to the peer assessment (Pe) condition are detected only for the portfolio condition (Po). Hypothesis 3: Students' preferences are congruent with students' perceptions of the appropriateness of the assessment methods. If familiarity grows and preferences become more positive (see Tables 5 and 6) after student teachers' hands-on experience, then it is hypothesized that students' preferences are congruent with their perceptions of the appropriateness of the assessment method. Positive experiences of an assessment will compel positive perceptions of its appropriateness, whereas negative experiences with an assessment method might produce negative perceptions of its appropriateness. Students were asked whether they considered the assessment method that they experienced to be appropriate. Table 7 represents students' responses. Blank space was provided if students wanted to expand on their answer or felt like suggesting a more appropriate assessment format. After the examination, the majority of the students think of their experienced assessment method as appropriate or well suited to the course. Notwithstanding, the Pearson chisquare test of independence reveals significant differences between the research conditions (dr= 4, [chi square] = 51.30, p < .0001) and the Bonferroni comparisons show that the peer assessment (Pe) condition and, to some extent, the lecture-taught (Le) group, are least positive about the appropriateness of the assessment format for this course. Of interest, the lecture-taught students (Le) seem to be not as satisfied with the multiple-choice test as their colleagues in the student-activating condition. When students suggested a different format of assessment as more appropriate, almost all answers suggest the customary methods of the individual written and oral examination formats. Sometimes suggestions for changes to the experienced assessment method were proposed, such as blank space to give additional explanations for the choice of answer in the multiple-choice test or a combination of multiple-choice items with open-ended questions.

DISCUSSION AND CONCLUSIONS In conclusion, the three hypotheses of the study are largely supported by the results, namely, (a) unknown assessments are unloved assessments; (b) as familiarity grows with the assessment tool, students' preferences will change positively; and (c) students' perceptions of the appropriateness of the assessment method will be congruent with their preferences. Of interest, the conclusions apply to both traditional tests, such as multiplechoice testing, and new modes of assessment, such as case-based assessment, peer evaluation, and portfolio assessment. Unknown assessments are negatively regarded assessments and, prior to this study, students' assessment preferences did not differ between research conditions. One exception was the low preference scores of the multiple-choice test in the active Mc condition. Perhaps the realization that the multiple-choice test was going to be their assessment method somehow frightened students and made them think more negatively about their expected examination. However, these critiques faded after the actual experience of the multiple-choice test, with the active Mc students considering this format to be their most preferred method and regarding it more positively than their fellow students, who did not experience this multiple-choice test (Ca, Pe, Po). Notwithstanding these positive attitudes of student teachers, multiple-choice tests have been the target of severe public and professional attack during recent years (Sambell et al., 1997; Zeidner, 1987). Moreover, university students in Sambell et al.'s (1997) study often reacted negatively when they discussed conventional evaluation methods. Exams had little to do with the more challenging task of trying to make sense of, and understand, their subject. In fact, students claimed that traditional evaluation had a detrimental effect on their learning process. Hence, the results of this study are rather surprising. The a priori unfamiliarity with this tool, and the possible good use of the assessment method (cf. student teachers' comments, Hypothesis 1, e.g., good coverage of the contents, emphasis on application) and its orientation toward knowledge construction, including the skills to apply content to practice-relevant situations and to solve real-life problems, might serve to explain these findings. Inversely, the probable

poor use

of the peer

assessment,

in which the

formative

function (cf.

interim feedback sessions on the peer factor) was not met by the lecturers due to time constraints and high workload, tends to give an explanation for the moderate to low effects of this format. Hence, opportunities for adjustments in learning behavior and group processes were missed and students lacked information on the performance consequences of the peer assessment procedure (cf. student teachers' comments, Hypothesis 1). Moreover, student teachers' comments tend to be consistent with the results of Clifford (1999), who found students to resent staff not taking responsibility and feeling unsure about their own, and their peers', abilities to grade work. Similarly, possibly because student teachers lacked interim feedback from the lecturer and therefore insight in the consequences of the peer assessment procedure for their scores, preservice teachers in the present study also highlight reservations about favoritism for friends and subjectivity or incompetence in scoring of their peer assessors. This finding contrasts with other results (e.g., Segers & Dochy, 2001) that have used the same peer assessment procedure but with the interim feedback adequately carried out. Consequently, the use and quality of the assessment method elicit an important limitation of the present study: Conclusions only apply to the instrument or tool that was used to make the assessment method operational. Caution in generalizing the findings to the assessment method is recommended. Replications of this study and verification of the results are needed. In fact, findings could be extended and illuminated further by assessing the quality of implementation of the methods in the study. Moreover, the results in this study consider the participating student teachers grouped in each research condition as a whole. No additional differential effects or covariables were investigated. For example, what are the (intermediate) effects of gender, student performance, individual teachers, and students' approaches to learning, among other factors? Future research that addresses these questions will be valuable. Notwithstanding these reservations, the purpose of this study was to investigate the dynamics of students' preferences for assessment methods, triggered by actual handson experience with assessments, and convincing evidence that is in line with educational literature (Aday & Campbell, 1995; Feigenbaum & Friend, 1992) was found. In

particular, the assessment methods of each of the research conditions are highly valued by student teachers after their examinations. This finding seems to replicate and extend previous findings of Macdonald (2002) and is consistent with research on students changing their dispositions (e.g., Sambell et al., 1997). However, would it be correct to advocate that the effects are caused by the hands-on experience and the associated growing familiarity with the assessment method in the study? An alternative explanation might be that the results are influenced and biased by social desirability in responses (e.g., "the teacher expects this answer from me"). The interim findings at the end of the final lesson of the course tend to support a causal relationship to the biasing effect. Significant differences are found between the unknown assessment methods and the familiar modes. In particular, the lecture-taught students (Le) and their portfolio condition (Po) colleagues were already highly satisfied with their assessment modes. The lecture-taught students had already experienced a multiple-choice test after the autumn semester. Likewise, the portfolio students, who were intensively working on the completion of their definitive portfolios at the time of the Lesson 10 measurement, already had firsthand experience of working on a portfolio, possibly resulting in Fullan's (2001) notion of mastery, achievement, or academic growth with student teachers. Moreover, students also had participated in a formative assessment conversation with their lecturer, so the routine for their final examination conversation with the teacher was already known. Please note that the portfolio assessment method provides preservice teachers with more feedback, support, and direction before the end-of-course assessment during the examination period than the solely summative assessments in the study. That is, the formative function that the portfolio serves might have assisted student teachers' learning (Black & William, 1998; Sadler, 1989) and may have affected students' attitudes toward this assessment method positively. Of interest, these arguments of familiarity and of the beneficial formative function of the method were expected to apply to the peer assessment method as well; however, it did not, as explained above (possibly caused by a poor[er] implementation of the method). This finding also supports the causal relationship thesis better than the social desirability explanation. From this research, we have learned that hands-on experience of assessment does make a

difference and implications for educational practice are indicated. Students' perceptions of learning and assessment are not stable, unchangeable characteristics. This message is encouraging in the midst of the accumulating literature on the resistant nature of prior beliefs and the failure of teacher education programs to produce changes (Kagan, 1992; Pintrich, Marx, & Boyle, 1993). However, if we want to change the traditional perceptions of student teachers and teachers, this study provides evidence that handson experience with new assessment tools serves as an incentive for such a change to occur. This is supported by the finding of Willcoxson (1998) that most academics mentioned their own experiences of being taught as key factors influencing the way they themselves teach. Consequently, university instructors working with preservice teachers ought to be encouraged to use a variety of assessment methods with students in their courses. Student teachers need to be informed of the properties of the assessment methods as early as possible so that familiarity will grow fast. However, changes in perceptions might occur only after actual experience with the assessment and merely giving informationon the method itself will not be sufficient (Richardson, 1990). In fact, continuous improvement, action research, and experimentation are characteristics of successful innovative schools (Little, 1982; Sagor, 2000). Consequently, the recommendation to expose student teachers to many new assessment method experiences, as well as teaching method experiences, is apparent. This type of modeling would benefit the teacher candidates once they have embarked on their teaching careers as they would have more experience with, and inclination to use, a wider variety of classroom assessment options (e.g., formative and/or summative, open vs. closed book, individual vs. group, knowledge acquisition or knowledge construction purposes, etc.). In this way, a wide range of prospective elementary school children will be accommodated and diverse learning outcomes are achieved. Moreover, the use of formative assessments has the ability to raise standards in achievement and in classroom teaching (Black & William, 1998; Sadler, 1989; Stiggins, 2002). Finally, handson experience with unknown assessment methods is an important incentive to trigger changes in students' perceptions and assessment preferences. Hence, teacher educators are strongly recommended to make the most of the assessment experimentation

opportunity in class and constructively contribute to student learning. In addition, the present study shows that students do not prefer a single method of assessment while disliking all others. Instead, students tend to prefer a mix of modes and this mix of preferences shows similarities to, but also considerable differences from, the findings in educational literature on students' assessment preferences. For example, if students' preferences for multiple-choice tests and open-ended questions or examinations are compared, the results of the present study contradict the results of Traub and MacRury (1990) and Zeidner (1987). The explanation underlies the hypothesis of this study that unknown assessment methods are regarded negatively. The use of multiplechoice tests in Belgian (Flemish) Secondary Education (12-18 years) is rare. Consequently, student teachers in the study enter higher education with little knowledge of this assessment mode (Van den Berg & Ros, 1999) and, consequently, low preference scores are obtained. Contrary to the findings of Sander, Stevenson, King, and Coates (2000) and Macdonald (2002), these students do not favor coursework assessment, such as portfolio assessments and papers/ essays, to traditional examinations such as written and oral closed book examinations, at least not initially or prior to actual experiences with the methods. Likewise, research findings that alternative assessment methods are favored above conventional assessment methods (Birenbaum, 1997; Sambell et al., 1997) do not apply to these results. They are more consistent with the finding of the Zoller and Ben-Chaim (1997) study, in which American students were particularly interested in the traditional written exam as a known means to achieve grades. The findings of Birenbaum (1997) and Zoller and Ben-Chaim (1997) that the oral examination is the least favored also does not account for the students in this study. On the contrary, the individual oral examination is in the top 3 of preferred assessment methods. Perhaps student teachers are more talkative than their colleagues in other higher education programs. The oral group examination, on the other hand, does not appeal to the students; moreover, students dislike this assessment method as much as the peer assessment method. Presumably--and the additional comments of student teachers in the study (cf. Hypothesis 1) and Clifford's (1999) results tend to support this statement--the dependence of individual scores on others, apart from the teacher, makes

students feel uncomfortable and uneasy. Please note that comparison of the results of the present study to the findings of previous studies on students' assessment preferences is hardly warranted. Often, students' prior experiences with assessment are not taken under investigation, pretest measures are lacking, and students' assessment preferences are treated as fairly stable learner characteristics. However, the finding that student teachers like a variety of assessment methods asserts teacher education educators to adopt an assortment of evaluation formats. Diverse objectives might be set such as helping to develop positive student teachers' perceptions regarding alternative assessments, satisfying a broad range of pre-service teachers with distinct preferences (and associated learner characteristics), or attempting future elementary school pupils to be assisted in their learning by means of new methods of assessment. Whatever the aims, enriching teacher education programs with alternative (and formative) assessment methods in addition to conventional methods might serve the purpose. REFERENCES Aday, R. H., & Campbell, M. J. (1995). Changes in nursing students' attitudes and work preferences after a gerontology curriculum. Educational Gerontology, 21 (3), 247-260. Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32(3), 347-364. Birenbaum, M. (1996). Assessment 2000: Towards a pluralistic approach to assessment. In M. Birenbaum & F. Dochy (Eds.), Alternatives in assessment of achievements, learning processes and prior knowledge: Evaluation in education and human services (pp. 3-29). Boston: Kluwer Academic. Birenbaum, M. (1997). Assessment preferences and their relationship to learning strategies and orientations. Higher Education, 33(1), 71-84. Birenbaum, M., & Feldman, R. A. (1998). Relationships between learning patterns and attitudes towards two assessment formats. Educational Research, 40(1), 90-97.

Black, P., & William, D. (1998). Inside the black box: Raising standards in classroom assessment. Phi Delta Kappan, 80(2), 139-148. Clifford, V. A. (1999). The development of autonomous learners in a university setting. Higher Education Research and Development, 18(1), 115-128. Cuban, L. (1998). How schools change reforms: Redefining reform success and failure. Teachers College Record, 99(3), 453-477. Darling-Hammond, L., & Snyder, J. (2000). Authentic assessment of teaching in context. Teaching andTeacher Education, 16(5-6), 523-545. Davies, A., & LeMahieu, P. (2003). Reconsidering portfolios and research evidence. In M. Segers, F. Dochy, & E. Cascallar (Eds.), Optimising new modes of assessment: In search of qualities and standards (pp. 141-170). Dordrecht, the Netherlands: Kluwer Academic. Feigenbaum, E., & Friend, R. (1992). A comparison of freshmen and upper division students' preferences for small and large psychology classes. Teaching of Psychology, 19(1), 12-15. Fullan, M. (2001). The new meaning of educational change (3rd ed.). New York: Teachers College Press. Grippin, P. C. (1989, February). Using research on teaching excellence to remodel teacher education. Paper presented at the annual meeting of the Eastern Educational Research Association, Savannah, Georgia. Hall, G. E. (1979). The concerns-based approach to facilitating change. Educational Horizons, 57(4), 202-208. Hargraeves, A. (Ed.). (2000). Learning to change: Teaching beyond subjects and standards. San Francisco: Jossey Bass.

Janssens, S., Boes, W., & Wante, D. (2001). Portfolio: een instrument voor toetsing en begeleiding [Portfolio: An instrument for evaluation and coaching]. In F. Dochy, L. Heylen, & H. Van de Mosselaer (Eds.), Assessment in onderwijs [Assessment in education] (pp. 203-224). Utrecht, the Netherlands: Lemma. Johnson, J. L., & Seagull, A. A. (1968). Form and function in the affective training of teachers. Phi Delta Kappan, 50(3), 166-170. Kagan, D. M. (1992). Implication of research on teacher beliefs. Educational Psychologist, 27(1), 65-90. Kelchtermans, G. (2005). Teachers' emotions in educational reforms: Selfunderstanding, vulnerable commitment, and micropolitical literacy. Teaching and Teacher Education, 21 (8), 995-1006. Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 55. Little, J. W. (1982). Norms of collegiality and experimentation: Workplace conditions of school success. American Educational Research Journal, 19(3), 325-340. Loughran, J., & Berry, A. (2005). Modelling by teacher educators. Teaching and Teacher Education: An International Journal of Research and Studies, 21(2), 193203. Macdonald, J. (2002). Getting it together and being put on the spot: Synopsis, motivation and examination. Studies in Higher Education, 27(3), 329-338. Pellegrino, J. W. (2002). Knowing what students know. Issues in Science and Technology, 19(2), 48-52. Pintrich, P. R., Marx, R. W., & Boyle, R. A. (1993). Beyond cold conceptual change: The role of motivational beliefs and classroom contextual factors in the process of

conceptual change. Review of Educational Research, 63(2), 167-199. Richardson, V. (1990). Significant and worthwhile change in teaching practice. Educational Researcher, 19(7), 10-18. Sadler, R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18, 119-144. Sagor, R. (2000). Guiding school improvement with action research. Alexandria, VA: Association for Supervision and Curriculum Development (ASCD). Sambell, K., McDowell, L., & Brown, S. (1997). "But is it fair?" An exploratory study of student perceptions of the consequential validity of assessment. Studies in Educational Evaluation, 23(4), 349-371. Sander, P., Stevenson, K., King, M., & Coates, D. (2000). University students' expectations of teaching. Studies in Higher Education, 25(3), 309-323. Segers, M. (2003). Evaluating the Over All Test: Looking for multiple validity measures. In M. Segers, F. Dochy, & E. Cascallar (Eds.), Optimising new modes of assessment: In search of qualities and standards (pp. 119-140). Dordrecht, the Netherlands: Kluwer Academic. Segers, M., & Dochy, F. (2001). New assessment forms in problem-based learning: The value-added of the students' perspective. Studies in Higher Education, 26(3), 327-343. Segers, M., Dochy, F., & Cascallar, E. (Eds.). (2004). Optimising new modes of assessment: In search of qualities and standards. Dordrecht, the Netherlands: Kluwer Academic. Slater, T. F. (1996). Portfolio assessment strategies for grading first-year university physics students in the USA. Physics Education, 31(5), 329-333. Stiggins, R. J. (2002). Assessment crisis! The absence of assessment FOR learning. Phi

Delta Kappan, 83, 758-765. Struyven, K., Dochy, F., & Janssens, S. (2005). Students' perceptions about evaluation and assessment in higher education: A review. Assessment and Evaluation in Higher Education, 30(4), 331-347. Struyven, K., Dochy, F., Janssens, S., & Gielen, S. (2006). On the dynamics of students' approaches to learning: The effects of the learning/teaching environment. Learning and Instruction, 16(4), 279-294. Struyven, K., Sierens, E., Dochy, F., & Janssens, S. (2003). Groot worden: De ontwikkeling van baby tot adolescent. Handboek voor toekomstige leerkrachten [Growing: The development from baby to adolescent. Course book for prospective teachers]. Leuven, the Netherlands: LannooCampus. Topping, K. (2003). Self and peer assessment in school and university: Reliability, validity and utility. In M. Segers, F. Dochy, & E. Cascallar (Eds.), Optimising new modes of assessment: In search of qualities and standards (pp. 55-88). Dordrecht, the Netherlands: Kluwer Academic. Traub, R. E., & MacRury, K. (1990). Multiple-choice vs. free response in the testing of scholastic achievement. In K. Ingenkamp & R. S. Jager (Eds.), Test und trends 8: jahrbuch der padagogischen diagnostik [Test and trends 8: Yearbook of educational diagnostics] (pp. 128-159). Weinheim und Base, Germany: Beltz Verlag. Van den Berg, R., & Ros, A. (1999). The permanent importance of the subjective reality of teachers during educational innovation: A concerns based approach. American Educational Research Journal, 36, 879-906. Van den Berg, R., Vandenberghe, R., & Sleegers, P. (1999). Management of school innovations from a cultural-individual perspective. School Effectiveness and School Improvement, 10, 321-351.

Van de Ven, A. H., & Rogers, E. M. (1988). Innovations and organizations: Critical perspectives. Communication Research, 15(5), 632-651. Waugh, R. F., & Punch, K. F. (1987). Teacher receptivity to systemwide change in the implementation stage. Review of Educational Research, 57(3), 237-254. Willcoxson, 23(1), 59-70. Zeidner, M. (1987). Essay versus multiple-choice type classroom exams: The student's perspective. Journal of Educational Research, 80(6), 352-358. Zoller, U., & Ben-Chaim, D. (1988). Interaction between examination-type anxiety state and academic achievement in college science: An action-oriented research. Journal of Research in Science Teaching, 26(1), 65-77. Zoller, U., & Ben-Chaim, D. (1997). Examination type preferences of college science students and their faculty in Israel and USA: A comparative study. School Science and Mathematics, 97(1), 3-12. Katrien Struyven Filip Dochy Steven Janssens University of Leuven, Belgium Katrien Struyven is a postdoctoral research fellow at the Centre for L. (1998). The impact of academics' learning and teaching

preferences on their teaching practices: A pilot study. Studies in Higher Education,

Research on Teaching and Training at the University of Leuven (KULeuven), Belgium. She achieved her doctoral degree on the topic of "The Effects of Student-Activated Teaching/Learning Environments (Instruction and Assessment) on Students' Perceptions, Student Performance, and Preservice Teachers' Teaching" in May 2005. Her research

interests lie within the field of didactics and assessment and their respective effects on student performance, students' approaches to learning, and students' perceptions of the learning environment. Filip Dochy is professor of teaching and training and corporate training at the Centre for Educational Research on Lifelong Learning and Participation and the Centre for Research on Teaching and Training at the University of Leuven (KULeuven), Belgium. His research concentrates on new learning and training environments, new modes of assessment on teaching, and lifelong learning of trainers inteacher training and corporate training settings. He is past president of the European Association for Research on Learning & Instruction (EARLI, www.earli.org) and editor of the Educational Research Review. Steven Janssens is professor at and director of the Centre for Research on Teaching and Training at the University of Leuven (KULeuven), Belgium. His research focuses on subjective theories of teachers, micropolitics in the teacher-pupils relationships, and the use of video and portfolio in teacher training.
TABLE 1 Overview and of Conditions and Number of Schools, in the Lecturers, Study

Classes,

Student

Teachers Treatment

Involved

Lessons Lectures+ Student-activating assignments+ Peer Portfolio Treatment Lessons Lectures+

Assessment Multiple-Choice Multiple-choice Case-based assessment (Po) exam Test

[N.sub.schools] (Le) test (Pe) 2 (Mc) (Ca) 2 2 1 1

[N.sub.Lecturers] Assessment S1 Test (Le) 2 S2 /

Multiple-Choice

Student-activating assignments+ Peer Portfolio Treatment Lessons Lectures+ assignments+ Peer Portfolio Treatment Lessons Lectures+ assignments+ Peer Portfolio NOTE: S1, S2 = School

Multiple-choice Case-based assessment (Po) exam (Pe)

test (Ca)

(Mc) 3 5

5 1 1 1

2 [N.sub.classes]

Assessment Multiple-Choice Multiple-choice Case-based assessment (Po) exam (Pe) Test

S1 (Le) test (Ca) 5 4 [N.sub.students] (Mc) 3 2

S2 / 5 3 3 1 /

Student-activating

Assessment Multiple-Choice Case-based assessment (Po) 1 and School + + 2 Test test exam (Pe)

S1 (Le) (Mc) (Ca) 87 86 in each 54

S2 131 119 72 87 28 condition; examination; Le Mc = = / /

Student-activating

Multiple-choice

lecture-based Ca Pe = = =

learning

environment learning learning equals

multiple-choice + +

student-activating

learning

environment

multiple-choice case-based

examination; assessment; Po In of hard who the class. teachers in

student-activating student-activating cases, condition

environment environment + school an in of on In

peer/co-assessment; assessment. teacher, feature make or it equals

student-activating teacher equals

learning class.

environment school,

portfolio inevitable categories number more students study. of

some

and/or

Although the based effects

quasi-experimental to directly of Numbers might be

research, the are to

overlaps the than

address students the

school,

lecturer,

students

administered

questionnaires. the present

reality,

student involved

subscribed

course

TABLE the

Overview Four

of

the

Characteristics Methods

of

(and in

Differences the

Between) Study

Assessment

Assessment Characteristics

Multiple-Choice Test End-of-course exam assessment conversation or

Case-Based Exam

Open/closed Individual/exam in Written/oral Questions exam in deal the

book examination Individual

Closed

Open Individual

group Written examination Information book in to Case study with? Assessment procedure (in book relation info) Written

course directly

course

as course Preparation students the method the course? Formative/ use Interim of for

part on development Lesson Interim

of

the child 1 Instructions lessons /

assessment during Lesson questions Summative summative of evaluation feedback (formal sessions) No No the Summative 10 Example

Course final Final

work calculated score determined

No in score? Exam by? Assessment

No

Exam

Assessment

Peer

Portfolio

Characteristics End-of-course exam assessment conversation Open/closed Individual/exam in Written/oral Questions exam in the to Group course directly Assessment procedure as course Preparation students the the + method course? Example summative use Interim (formal feedback groups Course final Final score exam Group by? tasks peer (N = 3) + + determined assessments (Yes, after paper) work calculated for assessment of Instructions Case during Case study back, to go (a) Summative of evaluation each (a) Yes, (and Yes in score? Group + tasks, including individual reflections exam 1 interim on Yes session demand) questions on (Interim 3 (home) questions + adolescence part on development Instructions Try-out peer + feedsessions) Example Formative Summative the Instructions feedback session) (a) Example + questions (1 Interim of the child tasks book (in Oral examination relation info) to Portfolio course with? (in book relation info) deal book examination Group group Oral Individual Open Open or

Formative/

(Formative)

sessions)

(portfolio)

(a) Due to time restrictions, teachers were unable to provide students with the interim peer feedback information.

TABLE at

Simple

Statistics of Method written

for the

the

List

of on

Assessment Child SD 4.24

Preferences Development Ranking

the

Start

Course N M 668 667 665 662 660 661 3.29 668 665 669 661 667 666 of

Assessment Individual Practical Individual Portfolio Take Open Self Cooperative Multiple-choice Case-based Oral Peer TABLE 4 The group home book

exam exam

0.73 0.84 1.12 1.14 1.20 1.20 6 8 1.25 1.27 1.06 1.23 1.24 Assessment 561) 1.17

1 2 3 4 5 6

examination oral assessment examination examination 663 assessment test examination exam assessment Overall

4.10 3.78 3.66 3.38 3.29 1.19 3.27 3.22 3.11 3.03 2.70 2.67 the Expected =

Paper/essay assessment

9 10 11 12 13 Method

Appreciation (%)

at the End of Lesson 10 by Research Conditions, Reported by Means of Percentages (N Total Condition Le Mc Ca Pe PO 113 99 92 154 103 N M (SD) + 6.2 7.1 2.2 5.2 9.7 + 47.8 24.2 34.8 23.4 39.8 + +/35.4 33.3 27.2 25.3 21.4 8.8 24.2 27.2 27.9 20.4 1.8 11.1 8.7 18.2 8.7 (%) 100 100 100 100 100

3.48(0.81) 2.92(1.10) 2.95(1.03) 2.69(1.17) 3.21(1.14)

NOTE: very good = + + (score 5), good = +, moderate = +/-, weak =-, very + weak = + (score 1). Le = Mc lecture-based = Ca Pe Po = = = learning environment learning multiple-choice environment environment environment condition + examination; + + case-based portfolio student-activating

environment learning learning learning research

multiple-choice

examination; assessment; assessment.

student-activating student-activating student-activating test for results ([chi

peer/co-assessment; statistically

Kruskall-Wallis

revealed

significant

square] = 37.34; df = 4; p < .0001). If the scale scores are used, the

Bonferonni for TABLE Both Values 5

comparisons Le >

reveal Ca,

significant Mc, Values by Tests 1 Method exam Pe for of

differences and Each Method Po

([alpha] > the

.05)

Pe. List at the 1

Students' of the of

Preference Measurement Paired t

From Minus Exam

Moments

Research

Condition,

Including Lesson

Examination

Lesson Condition Le (N Indiv Indiv Open Take Practical Case-based Paper/essay Portfolio Group Peer Cooperative Mc (N = 99) + + book home exam exam Individual Individual Open Take Practical Case-based Paper/essay Portfolio Group Peer Cooperative Ca (N = 91) + + book home Individual Individual Open Take + oral assessment assessment + oral assessment assessment Multiple-choice written oral exam exam = Evaluation 115) + + book home exam exam

M *

(SD) 3.31(1.25)

(SD) 3.68(1.24)

Multiple-choice written oral exam exam exam exam

4.16(0.71) 3.82(1.08) 3.46(1.25) 3.38(1.24)

3.93(0.90) 3.47(1.27) 3.42(1.21) 3.21(1.30) 3.38(1.17) 2.63(1.07) 2.72(1.32)

4.24(0.68) 3.07(1.08) 3.34(1.24) 3.52(1.12) exam 2.72(1.22) 2.66(1.21) 3.34(1.12) 3.28(1.22) exam exam exam * 4.43(0.61) 3.91(1.14) 3.35(1.13) 3.43(1.08) 4.13(0.96) 2.94(0.99) 3.23(1.23) 3.51(1.25) exam 2.93(1.24) 2.94(1.21) 3.38(1.16) 3.33(1.35) exam exam exam * * 4.15(0.74) 3.87(1.10) 3.40(1.22) 3.59(1.14) 4.10(0.85)

3.02(1.35) 2.58(1.19) 2.49(1.31) 3.03(1.19) 3.08(1.27) 3.95(1.20) 3.57(1.23) 2.88(1.34) 3.80(1.23) 3.65(1.23) 3.24(1.18) 2.64(1.21) 2.78(1.27) 3.01(1.38) 2.70(1.41) 2.37(1.21) 3.12(1.33) 2.93(1.30) 3.11(1.38) 3.87(0.97) 3.34(1.17) 4.20(0.91) 3.58(1.26) 3.48(1.11) 3.10(1.19) 2.61(1.27)

Self-assessment

Self-assessment

Multiple-choice written oral exam exam exam

Practical

Case-based Paper/essay Portfolio Group Peer Cooperative Pe (N = 152) + + book home Individual Individual Open Take +

exam

* 3.75(1.09)

2.94(1.11)

3.97(1.01) 2.55(1.23) 2.79(1.39)

3.04(1.14) oral exam 2.83(1.20) 2.65(1.27) 3.33(1.22) assessment Multiple-choice written oral exam exam exam exam exam exam * * 3.31(1.25) exam

2.82(1.26) 2.48(1.19) 3.06(1.12) 2.73(1.28) 3.42(1.21) 3.94(1.06) 3.64(1.15) 3.83(1.04) 3.79(1.08) 3.79(1.13) 2.95(1.20) 3.25(1.24) 3.46(1.15)

assessment

Self-assessment

3.11(0.77)

4.25(0.77) 3.66(1.13) 3.07(1.18) 3.33(1.20)

Practical Case-based Paper/essay Portfolio Group Peer Self Cooperative Po (N = 58) + + book home Individual Individual Open Take + oral

3.97(0.48) 3.09(1.14) 3.45(1.08) 3.77(1.13) exam * * * exam exam * * * * 2.74(1.29) 3.14(1.18) 3.19(1.23) exam

2.50(1.15)

3.26(1.25) 3.17(1.26) 3.55(1.13) 3.69(0.95) 3.19(1.14) 3.18(1.21) 4.12(0.65) 3.35(1.19) 3.20(1.31) 3.77(1.11) 3.30(1.10)

assessment -assessment assessment written oral exam exam exam exam * oral assessment

Multiple-choice

3.04(1.17) 4.16(0.82) 4.09(0.84)

3.39(1.00) 3.18(1.19) 4.16(0.83) 2.96(1.02)

Practical Case-based Paper/essay Portfolio Group Peer Cooperative Paired (Exam Condition Le (N Indiv Indiv = + + +

3.30(1.21) 4.05(1.07) exam 2.82(1.24) 2.86(1.22) 3.45(1.16) assessment t Evaluation Method exam 3.35(1.31)

3.05(1.11) 3.96(1.11) 2.81(1.26) 2.56(1.18) 3.11(1.23) 2.98(1.25) Tests L1) df * 114 114 104 t 2.44

Self-assessment

115)

Multiple-choice written oral exam exam

-2.49 -3.30

Open Take Practical Case-based

book home exam exam

exam exam 114 113 112 114 oral exam 114 114 assessment

111 109

-0.37 -1.53 -7.66 -3.52 -4.46 -3.72 114 -2.75 -1.24 -1.35 -1.43 * 98 98 92 7.98 -5.89 -7.55 3.62 1.62 -6.64 -2.11 -3.24 -3.12 96 -1.58 -3.99 -2.07

Paper/essay Portfolio Group Peer Cooperative Mc (N Individual Individual Open Take Practical Case-based Paper/essay Portfolio Group Peer Cooperative Ca (N Individual Individual Open Take Practical Case-based Paper/essay Portfolio Group Peer Cooperative Pe (N Individual Individual Open Take Practical book home = 152) + + + Self-assessment oral assessment book home = + + 91) + Self-assessment oral assessment = 99) + + book home exam exam + Self-assessment assessment

110 exam exam exam 94 96 98 95 96

Multiple-choice written oral exam exam

95 exam 96 97 assessment Multiple-choice written oral exam exam exam exam 90 exam 87 89 assessment Multiple-choice written oral exam exam exam exam exam * * 149 88 exam 89 * 90 exam exam * * 89 89 96 exam

-2.82 79 90 89 90 87 0.07 -2.69 -3.50 5.44 -0.08 -4.98 7.37 -3.57 -6.42 -0.07 -1.12 -2.16 -4.20 139 3.02 -3.11 -0.25 7.27 3.70 -2.06 150 150 149 147

Case-based Paper/essay Portfolio Group Peer Self Cooperative Po (N Individual Individual Open Take Practical Case-based Paper/essay Portfolio Group Peer Cooperative Paired (Exam Condition Le (N Indiv Indiv Open Take Practical Case-based Paper/essay Portfolio Group Peer Self-assessment Cooperative Mc (N = 99) + + book Individual Individual Open + = + + 115) + Self-assessment + book home = + + oral

exam 145 145 exam * * *

145

-1.18 -1.70 -2.88

* 151 151

151

5.95 3.26 3.77 4.17 51 0.83 -5.42 0.31 -0.21 0.09 -2.26 1.71

assessment -assessment assessment 58) written oral exam exam exam exam

151 exam 56 * 57 56 55 55 56

Multiple-choice exam exam * *

56 * oral assessment 55 assessment t Evaluation Multiple-choice Method exam 0.001 0.715 0.130 * * * * exam 0.179 0.007 assessment * 0.157 exam * <0.001 <0.001 0.001 * 0.216 0.001 <0.001 <0.001 oral * 54 exam 56 55 56

-1.33 -0.50 -0.10 -1.93 -1.88 -1.99 Tests L1) p 0.016 * * * d 0.297

written oral book home exam exam

exam exam exam exam <0.001

0.014

-0.284 -0.297 -0.033 -0.134 -0.899 -0.409 -0.484 -0.403 -0.116 -0.135 -0.268 -0.160

assessment

Multiple-choice written oral exam exam exam

<0.001 * *

1.085 -0.886 -0.828

0.381

Take Practical Case-based Paper/essay Portfolio Group Peer Cooperative Ca (N = + + book Individual Individual Open Take Practical Case-based Paper/essay Portfolio Group Peer + +

home exam exam

exam <0.001 0.037 0.002 0.002 oral 0.041 exam <0.001

0.109 * * * * 0.117 * * 0.006 exam 0.009 0.001 <0.001 * 0.937 * * * * exam 0.265 0.945 * <0.001 exam 0.002 0.803 * * * 0.241 <0.001 <0.001 exam exam * 0.003 * <0.001 * exam *

0.190 -0.827 -0.271 -0.360 -0.380 -0.173 -0.471 -0.208 -0.302 0.948 * * 0.008 -0.325 -0.467 0.743 -0.008 -0.627 0.971 -0.411 -0.769 -0.008 -0.138 -0.231 -0.473 * 0.252 -0.324 -0.018 0.683 0.403 -0.181 -0.120 -0.172 -0.272 * * * * 0.410 * 0.755 0.633 0.337 0.355 0.455 0.130 -0.948 0.040 -0.036 0.016 -0.398 0.321

assessment assessment 91) written oral exam exam exam exam * 0.001 <0.001 oral assessment

Self-assessment

Multiple-choice exam * <0.001

home

Self-assessment Cooperative Pe (N = 152) + + book home Case-based Paper/essay Portfolio Group Peer Self Cooperative Po (N = + + book home exam Individual Individual Open Take Practical Case-based + oral assessment -assessment assessment 58) written oral exam Individual Individual Open Take

0.033 assessment written oral exam exam exam exam 0.091 0.005 exam * * * exam exam * * 0.028 exam * * * * 0.041 Multiple-choice

Practical

<0.001 0.001 <0.001 <0.001 exam <0.001 * 0.837 0.926 * 0.092

Multiple-choice

exam

Paper/essay Portfolio Group Peer Cooperative NOTE: Le = + Self-assessment * oral assessment

0.188 0.620 exam 0.058 0.066 assessment learning Ca = Pe = The Po 0.051 environment learning = 0.923

-0.215 -0.083 -0.008 -0.250 -0.284 -0.289 + multiple-choice environment + learning learning learning methods the peer within the

lecture-based Mc = examination;

examination; multiple-choice environment environment environment represent (e.g., + + +

student-activating assessment; assessment.

student-activating student-activating student-activating underlined is

case-based portfolio the

peer/co-assessment; evaluation assessment also

assessment

either

method

that an

experienced book

research condition or is part of, or similar to, the assessment method the portfolio procedure at Mc Ca, the the > includes selfopen exam; assessment of = Le Mc, methods .05) > Ca involves and cooperative assessment). ([alpha] exam: Ca > Pe

Cohen's d is used as effect size. Bonferroni comparisons for the list examination research Ca, Le, Pe, Mc; Po; (d) show (b) open significant for (a) + Ca individual book exam: differences written > Le, between and Po > conditions multiple-choice exam: Po,

Po, Le > Po, and Pe > Po; (c) individual + oral exam: Le > Mc, Pe > and and > Le; (e) take-home exam: Pe > Le, Po; (f) practical exam: Pe > Le, Mc; (g) case-based exam: Ca > Le, Mc, Pe, Po, and Po > Le, Mc; (h) paper/essay: Pe > Ca, Le, Mc; (i) portfolio: Pe > Ca and Po > Ca, Le, Mc; Le, G) Mc, group Ca, + Po; Pe are for > oral (l) Ca, Po exam: Le, > prePe Mc, > Le, Po. Mc; Pe At (k) > the exam: the are peer Le; > start Le scores 4). p ANOVA the Moments MoM Method the List Four of of < Assessment by Methods Research .05. of This (MoM) [df.sub.error] Study, for assessment: (m) of Mc are the and used Pe > self-assessment: for Ca, cooperative course, Pe of > Mc the from

assessment: differences and for participants

found whom

multiple-choice Le, and Mc. Only posttest

portfolio:

available

(different

Table * TABLE 6 One-Way From

Abstracted Both

Methods,

Condition

Measurement

Evaluation

[df.sub.model]

(Condition)

Multiple-choice test (Me Case-based examination Portfolio assessment Cooperative assessment Evaluation Method Multiple-choice test (Me Case-based examination Portfolio assessment Cooperative assessment (Pe) Evaluation Method Multiple-choice test (Me Le examination Po Pe Portfolio assessment assessment NOTE: Ex = MoM = (Po) (Pe) Moments + of Po, Pe (Ca) > and > Case-based Ca Mc Le) (Po) (Ca) and 6.01 Le) 0.56 (Pe) F (Po) L1 (Ca) L1 and L1

L1 Le) Ex 4 Ex 4 Ex p 4

4 Ex 4 4

664 588 656 637 657 4 4 636 660 631 [R.sup.2]

(Condition) <0.0001 9.04 <0.0001 0.6907 <0.0001 0.0588 <0.0001 0.8729 <0.0001 * * * * * 0.0349 0.0579 0.0034 0.1653 0.0137 0.0729 0.0019 0.0772

31.54 2.29 12.50 0.31 13.19

Significant (Condition) < Me Le, > Po, / > > / Pe > L1 > Po, = Le at = Me Me, / Le, the = end Me, of Ca lesson 1; Le, Ca Po, Me, Le Pe, Me, Le Le Ca, Pe, Differences Pe, Po, Ca Po Ca

Cooperative

Measurement, the +

immediately

after

examination;

lecture-based examination; case-based

learning Ca =

environment learning

multiple-choice learning

examination; multiple-choice environment +

student-activating assessment;

environment

student-activating

Pe Po

= =

student-activating student-activating (Dunn) t tests

learning learning for each

environment environment couple + of

peer/co-assessment; assessment. ([alpha] =

portfolio conditions

Bonferroni

.05). TABLE the 7 Students' Assessment on N 125 107 96 181 61 = lecture-based Mc = examination; Perceptions Method, Child Yes 71.2 83.2 89.6 56.9 88.5 learning Ca = Pe = Po [chi on After Development No the Appropriateness Completing (N Total 28.8 16.8 10.4 43.1 11.5 environment learning = + = the 570) (%) 100 100 100 100 100 multiple-choice environment + learning learning learning test p < is show Ca > .0001. of

Examination Condition Le Mc Ca Pe Po NOTE: Le

examination; multiple-choice environment environment environment statistically Bonferonni statistically + + +

student-activating assessment;

student-activating student-activating student-activating chi-square = 51.30, for square] by =

case-based portfolio

peer/co-assessment; df = 4,

assessment. proportions

Pearson

significant: significant

comparisons

for

research .05)

condition yes:

differences

([alpha]

Le, Pe, Mc > Pe, and Po > Pe, and for no: Le > Ca, and Pe > Ca, Mc,

Po.

Comments to Comments: Teachers and Students in Written Dialogue about Critical Revision
by Christyne A. Berzsenyi Effective teacher feedback increases students' awareness of the choices they can make and have made in a piece of writing and enables them to discuss those choices with others. Like many writing teachers, I continue to search for the most effective methods of teaching

students revision strategies through my feedback to their writing. However, I have felt that the unidirectional nature of the traditional teacher feedback and student revision of drafts process produces limited results in terms of actively involving students in rhetorical analysis that results in more effective text. As the technical writing student notes in the epigraph above, students are not typically required to articulate a rationale for their choices or offer an explanation, defense, justification and reconsideration of those choices. In turn, they are not taught to critically analyze their texts, on which successful revision practices are based. Even the most provocative and sensitive teacher comments generally ask students to comply with the teacher's evaluations and suggestions in revised texts, often without a genuine understanding of the intent of the teacher's feedback. Simply put, there is no meaningful dialogue about the paper between teacher and student, which means that students do not learn the internal dialogue of self-critique needed for performing critical revision on their own. Instead, students associate revision with dependence on a teacher's authoritative evaluation. Like many teachers, I've used student-teacher conferences to discuss revision with students. However, I've had moderate success engaging them in active, critical discussion. Students have shown difficulty talking freely, specifically, and spontaneously in real-time conversation about revising their texts. Without a conceptual vocabulary to "talk" about writing, students can not critically discuss their writing in terms of thesis statements, topic sentences, language conventions, support, audience, appropriateness, purpose, and so forth. Once students understand this language, they can put this new literacy into practice in intercommunication with teachers and their peers. Creating what Peter L. Mortensen calls a "talk-back" form of interaction between student-writer and teacher-reader could give students the voice and agency to respond to the traditional authority figure. Also, such interaction with teachers engages students more actively and critically in their own writing processes, an important component of successful writing, as Pamela Gay, Kathryn Evans, Andrea Lunsford, Helen Rothschild Ewald, and Richard Beach have noted. However, just as with learning any new form of literacy, students need exposure and practice with the language of revision before they feel confident and proficient enough in the discourse to use it in oral communication. With the goal of developing students' revision literacy and practices, I developed a dynamic, critical revision method that allows students some time to reflect on their texts and to think about responses to my feedback before entering into discussion and collaboration with me. This essay describes the Comments to Comments assignment, an asynchronous, written collaboration between teachers and students that is designed to teach students to develop, analyze, articulate, reconsider, and explain their revision ideas. Comments to Comments begins when students write their first drafts and I write feedback directly on their papers. The feedback provides marginal and end comments to identify and discuss specific rhetorical strategies, organization, syntax, semantics, and mechanics issues in their texts-both global and local revision concerns. Second, I return student papers with feedback and explain the

Comments to Comments assignment requirements with the aid of a handout of instructions, which illustrates how to respond to my feedback in a dialogue format (see Appendix A). Third, after students transcribe verbatim all of my feedback on their papers, they must carefully consider and responsively reply to each prompt. More specifically, students respond with discussion about their choices, justification for agreeing or disagreeing with my interpretation of their writing, interpretations of the assignment, questions, challenges, and so forth. What results is a written document that resembles a play script of dialogue between teacher and student about the student's text. To encourage active participation in the written dialogue, I assign students a grade for completing Comments to Comments. Finally, the fourth step involves teacher's second response in writing to the student's reply comments. Here, Comments to Comments ends but oral conversations and collaboration could continue. Granted, the assignment requires a bit more consideration, time, and writing from both students and teachers in order to achieve a rich and focused discussion and revision process. However, performing Comments to Comments with students once at the beginning of the semester has saved me time later in the course because oral conversations, peer revision, and my feedback are clearer and more meaningful to students as they revise subsequent assignments. While I consider this assignment to be useful for teaching revision to less experienced students, it has not been equally successful in all writing classes. I will discuss the long, tedious process of trial and error that I went through as I refined and adapted the assignment. TEACHERS IN WRITTEN DIALOGUE WITH STUDENTS While Pamela Gay has argued for a similar approach to written dialogue with students in her 1998 essay, "Dialogizing Response in the Writing Classroom: Students Answer Back," my approach to inviting students into a dialogue of critical revision differs. More specifically, Gay argues that basic writing students initially need to "vent" their feelings as a structured form of response in addition to providing a general reaction to the teacher's evaluation of their writing. Similarly, I agree that venting is important for students to move emotionally beyond their initial reactions of frustration, disappointment, and even anger into more productive emotional and mental states for performing critical rhetorical revision. This is why I do not begin revision dialogue with students immediately after returning papers to them with grades. I wait a class session or two, after they've had some time to reflect on their writing and my feedback before I discuss their grades, my comments, or the evaluations. While engaged in writing Comments to Comments, students respond to my comments and discuss their own texts directly and specifically in a less emotion-focused revision dialogue than Gay suggests-one with conviction about rewriting and collaborating. On the one hand, I do agree with Gay that students should have the opportunity to express their concerns, confusion, and frustration about an assignment and their performance of it. On

the other hand, I think such dialogue should be a part of the general discourse of the writing classroom throughout the writing process. Instead of an initial reaction to the paper's grade, I call students to provide responses of disagreement to specific points of my feedback along with reasoning to articulate a developed, critical discussion. If, after writing many points of reasoned disagreement a student feels that my evaluation was too severe, they are invited to make an argument for it. In fact, students have persuaded me to change my assessment through substantiated reasoning, which amounts to successful, critical dialogue about their writing. THE ROUGH EVOLUTION OF COMMENTS TO COMMENTS Over the past six years, my approach to giving feedback in Comments to Comments has developed significantly in accordance with my goals, my evaluations of students' written dialogues, and with students' anonymous critiques of the revision assignment. As Nancy Sommers explains, teacher feedback should motivate students to revisit their texts with curiosity and involvement: "The challenge we face as teachers is to develop comments which will provide an inherent reason for students to revise; it is a sense of revision as discovery, as a repeated process of beginning again, as starting out new, that our students have not learned" (156). Further, feedback should strive to invigorate students' inquiry into concerns of audience, purpose, terminology, conventions, genre, and form to comprise a critical revision process, as James Porter suggests. In addition, teachers should address the language mechanics and format issues. Also, feedback must be given with care to produce generous responses that are specific, reflective, challenging, and critical as opposed to vague, overly general, and ambiguous, as Chris Anson and Michael Robertson have argued in their essays. In turn, students are challenged to think and dialogue about their writing in a substantive manner. The following lists present types of feedback and response that develop into the written interaction of Comments to Comments: Types of Teacher Feedback * Questions that suggest expansion, clarification, explanation, persuasion * Remarks which reveal instructor's understanding of the students' texts so that student writers can evaluate whether that response is in line with their purposes or not * Identification of a mechanical problem in a specific sentence with a request to explain the error of standard English and to correct the sentence * Praise about what is working well accompanied by questions that ask students to explain why that element works well

Types of Student Responses * Revised words or sentences accompanied by an explanation or the student's agreement with the suggestion or interpretation of the text * Reasoning for disagreeing with instructor's suggestion or interpretation of the text * Discussions of successful writing strategies that respond to instructor's praise * New directions for revision that were not initiated by the instructor While these lists represent the current dialogue guidelines, when I began teaching Comments to Comments, my feedback did not elicit a critical reflection on student writing. I will address several of these problems that interfered with productive dialogue and how I modified the assignment to overcome the obstacles. Instructor's Handwriting One problem with written dialogue involved my handwriting. In an effort to be more time efficient while grading papers, I quickly wrote comments to student's writing because I had so much I wanted to share with students. However, what resulted was sloppy handwriting that students could not always read. In addition to hindering students' ability to dialogue with me, carelessly written communication hurt my ethos. In fact, it was quite embarrassing as one business writing student referred to my handwriting as "an ancient Egyptian sign system." Bottom line, my written remarks needed to be more carefully written so that my students could decipher them without translation from me. In turn, students ask for clarification occasionally instead of regularly. Students with Low Revision Literacy Students also had trouble understanding my feedback when they lacked a working knowledge of revision literacy. In other words, students must be able not only to read the words but also to understand the meaning of the questions, rhetorical concepts, and suggestions for revision in order to respond to them appropriately and completely. After having gotten some responses such as "I'm not sure what you mean by this" and "I'm not sure how to respond to this comment," I decided to provide a class session for students to begin the assignment. During this period, I explain the requirements thoroughly and provide a handout of instructions and examples of productive revision dialogue. Further, I review overhead slides that present several types of comments and responses from previous students to illustrate teacher-student collaboration. Encouraging all students to read through each of my comments, I then address each student individually, asking if they have any questions and if I can clarify any aspect of

my feedback. I agree with David L. Wallace; it is important to emphasize that students learn to write purposefully, evaluate their performance in terms of that purpose, and reconsider their strategies to produce more successful writing for a target audience. Ideally, the teacher feedback should help students to assess their rhetorical effectiveness and to revise their writing with purpose and audience in mind. Marginal or End Comments? While I have debated over the placement of feedback on student papers, as other scholars have, I decided that marginal comments are necessary for basic writers, while end comments are effective with more advanced writers, who have fewer sentence-level revision concerns than less experienced writers. For example, those basic writers who write sentence fragments, run-ons, and comma splices have difficulty identifying them on their own in their writing when I wrote end comments. End comments would vaguely indicate the presence of mechanical errors somewhere in the paper without giving enough guidelines for inexperienced writers to revise them. (I'll address revision dialogue about language mechanics a bit later in this essay.) Also, I found that I wrote superfluous description just trying to identify or reference the section of their text to which I had a comment. It was an inefficient use of feedback time and space. The clarity of marginal comments stems from their specificity, and students no longer give me the responses of confusion such as "what part of my essay are you talking about?" or "I'm not sure where you mean." A Teacher Learning to Let Go of Authority Trying to helpful, I tended to be too directive about what I thought students should do to revise their work, instead of letting them think it through. In turn, I limited students' critical thinking processes for revision. Further, I wasn't prompting students to perform rhetorical criticism of their texts; rather, I was making their decisions for them and then telling them what to do, which completely undermined my desire to encourage critical thought. To address this problem, I started to write feedback that was inquisitive and interpretative rather than directive. For example, I commanded a revision regarding focus by writing, "provide a thesis statement." As a change, I have articulated a prompt about thesis statements for a female composition student by writing, "I don't see a thesis statement that indicates the main point of your paper. Could you write one or revise a sentence that you think suggests your main point?" With this shift in sentence function, I received much more elaborate responses from the student who could explain what she thought was her thesis statement and add a new or revised thesis statement to convey the focus of her paper. My goal here was to encourage the student to dialogue with me about what a clear thesis is and why it is clear or not clear for a specific audience in a specific text. Given this, I was able to write a prompt that engaged the

student in a critical decision about how effectively she was communicating the main point of her essay instead of placing her in a passive subject position. Students Not Elaborating When students did not elaborately respond to my feedback, I couldn't gauge whether or not they understood the assignment, conventions of essay writing, mechanical rules, or the like. Further, I could not understand what rationale or learned practice was underlying their decisions, which would help me respond more appropriately to their revision strategies. Therefore, my follow-up reply wasn't very responsive to why the student made specific choices. For example, when I praised students for their writing, students often responded with a simple "thank you." More specifically, when I used the vague praise of "good detail," I couldn't be sure that the writer was consciously providing details that supported topic sentences. The point is that my comment of "good detail" did not evoke a conversation. In fact, it stifled conversation in its definitive evaluation and lack of inquiry into student views or decisions behind the inclusion of a given detail. Now, I follow-up my praise with a question that asks students to explain how an element functions in the essay. For example, "good detail" is followed up with "Why are good details important to topic sentences?" As a result, students provide reasoning that reflects their understanding of how details illustrate and support the sub-points provided in a topic sentence. In another example of praise about the inclusion of a clear thesis statement, I asked a male writing student, "What does your thesis statement let your popular magazine audience know about what your essay argues?" Here, the student must rethink the power and clarity of his thesis statement in terms of the audience, emphasizing a rhetorical perspective. Through this inquiry process about writing and rewriting, I can reinforce conventions of composition and strategies of argumentation to make writing more conscious for each student. Learning to Phrase Questions so That Students Explain After I learned to phrase suggestions as questions, I still received responses that expressed uncritical compliance. For example, I would ask a composition student, "how about including an example of the benefits of training with a professional to be more persuasive?" To this question, the student responded with "ok." Clearly, I have not engaged the student in an act of critical discussion about how and why to revise the text. Therefore, I now add to my suggestions a follow-up question or questions such as "Why do you think I've made that suggestion" or "Do you agree or disagree and why?" More specifically, I provide a suggestion for revision and request a rationale for such a revision strategy. For example, I wrote to a technical writing student on a cover letter for a resume, "Why might explaining the significance of your work experience here help support your topic sentence and increase your credibility?" Instead of supplying students with my rationale for a suggestion, I ask students to

critically think through possible strategies, effects, and reasons for revising the work, which is how I characterize critical revision. As an appropriate and complete response, students must provide verbatim what additions they would include, and make clear to me that they are not simply complying with my suggestion. Rather, they must show conviction through elaboration and reasoning that the decision to revise in a specific manner is a result of their own critical thinking process with their own goals in mind. Without these inquiries, I was inadvertently emphasizing the finished product of writing rather than the process of critical revision. The following are examples of original feedback, student responses, and my follow-up comments: Instructor's Feedback: Perhaps you could make this paragraph more meaningful to less experienced readers by saying what the pieces or some of the pieces of equipment do in terms of the process of beer brewing. What do you think? Student Response: I definitely think it would make the paper more interesting and flow like a well-developed paragraph containing more explicit information. Ex. The first piece of equipment that you might find in your kitchen is a brewpot. A brewpot needs to be made of stainless steel or enamel-coated metal that is not chipped. (nontraditional basic writing student) Instructor Follow-up: Excellent, thorough revision discussion here! Instructor Feedback: Where's your thesis? Would you write one? Student Response: Weather forecasting will always be a challenging occupation, but with the help of radar, Doppler Radar, and computers, predicting weather has become more accurate over the years. Instructor Follow-up: Excellent thesis with clarity about where you are heading with this essay. Having to rationalize a revision decision will increase the internalization of revision inquiry and strategies. In other words, students learn by doing the work of critical revision. Inviting Students to Disagree A few years ago, a couple of students expressed disagreement with my feedback and did so in an intelligent, critical response. After recognizing the value of encouraging this kind of critical thought, I began to invite students to disagree with my feedback, if that's how they felt. While a few students lacked good reasons for their disagreements, most disagreements resulted in productive dialogue when students thought critically outside of my suggestions and in terms of their own rhetorical goals. While disagreement is invited, substantiation or justification for the student's alternate position is required, just as an instructor's comments need justification. Instead of simply expressing compliance and telling me what they think I want to hear, I

encourage students to critically think through what they want for their own writing, what they mean, and how they want readers to think about their subject. Further, such justification and clarification enables me to better understand a student's purpose and reasoning, which may not have been obvious while reading the paper. In some cases, students' disagreement has convinced me that their reasoning makes more sense rhetorically than what I had in mind. In other cases, the students' disagreement indicates some misunderstanding of the assignment or my comment. The following are examples of my original feedback on their papers, their disagreements with my feedback, and my follow-up comments: Instructor's Feedback: Why did you change paragraphs? You continue to talk about regulations regarding basketball baskets in the second paragraph. Explain. Agree/disagree? Student Response: I thought I should change paragraphs because I was talking about the past and then I started to talk about the present. (Basic writing student) Instructor's Follow-up: Ok. Good explanation. For revision, make sure that your topic sentences reflect your change in time frame. Instructor's Feedback: Performance effect of lowering a truck? Does it give a smoother ride or the like? That would mean more to someone like me who doesn't know much about customizing trucks-your target audience. Student Response: I answered the question later in the essay. Therefore, there's no need for revision. (Basic writing student) Instructor's Follow-up: Ok. I see that you do address this later. My comment anticipated the significance of the process of lowering a truck, which you do address successfully at a later point. Instructor's Feedback: Why change paragraphs? Explain why or why not. Student Response: The reason for changing paragraphs is because I got this information from two different sources. Not only that, the one paragraph shows the effects on male smokers and the other shows the effects on female smokers. After your explanation in class, I now know that I need to do some revision on my topic sentences so that I clearly explain that one paragraph is about men and the other is about women. Also, I need to show more details on the subjects "male" and "female" in each paragraph. (Composition student) Instructor's Follow-up: exactly! Instructor's Feedback: Could you elaborate on this theory [of how the mysterious contents in the briefcase might be an Academy Award Oscar statuette]? It seems more realistic.

Student Response: I don't believe in this theory at all. I refuse to believe that the briefcase, something that boggled the greatest movie minds contained an Oscar. It's not something that could cause people to get killed over. (Basic writing student) Instructor's Follow-up: Ok. That's logical reasoning provided here. You sound firm in your belief with a strong voice that is missing from other parts of the essay. Put this in the paper to provide a strong refutation against this theory; it's persuasive. What's a pleasure for me to see is that student disagreements are generally written in a strong voice and reflect students' commitments to their writing and to their rhetorical goals. On the one hand, some disagreements reinforce in students what their goals are and how to achieve them better. On the other hand, other disagreements express a misunderstanding about the assignment or my comment, which I could then clarify for them by referring to the assignment requirements or by explaining what I intended by my comment. Regardless, in disagreement, students show they care enough about their text to disagree with me and negotiate the rewriting, reshaping, and re-imagining of their work. Dealing with Mechanics In terms of discussing the revision of language mechanics, I realized that I needed to address conventions and guidelines with inexperienced writers. As Robert Connors, Mina Shaughnessy, and others have argued, mechanical errors are rhetorically significant in the construction of ethos as writers meet discourse community expectations. Years ago, I would make "x" markings to indicate some form of mechanical error in that sentence without blatantly identifying the error so that the student would have to think about it. However, in response, I received some reactions of confusion that never developed into critical dialogue such as "I'm not sure what's wrong here" or blind guesses such as "Is it a fragment?" Then, I came to believe that inexperienced student writers need more specific feedback than vague circles around an error or markings without information about the mechanical error. However, they also need more active revision work than what comes with the instructor's correction of an error, which students gloss over or completely ignore, and, therefore, do not learn to recognize or correct. As a new strategy, I decided to name the mechanical error in an identified sentence. However, inexperienced writers would still be confused about revising since many don't have the vocabulary for understanding some mechanical errors by name. For example, on one student's essay, I indicated that a sentence was a comma splice, asking the student to edit the sentence. When the student responded to my prompt, he wrote, "I don't know what this is," and provided a new sentence that produced another comma splice error. Clearly, I had to address at least some of the common grammatical errors instead of just assuming or

expecting that writing students know what they are and how to correct them. In fact, I began introducing revision by reviewing some of the common mechanical errors. Then, if they still made the errors in their papers, I would state the error in the sentence and ask students to review by referencing that type of error in their style manuals, explain why it is ungrammatical, and rewrite the sentence with the correction. For further assistance in this endeavor, I suggest going to my office hours and/or to the Writing Center. The following is an example of dialogue with a basic writing student about language mechanics: Instructor's Feedback: Why is this a fragment? Please revise the sentence. Student Response: The subject was missing which is necessary for a complete sentence. The sentence should read, "Baking in the sun all day, I got very burned." Instructor's Follow-up: Good work. Having students take the time to reference resources on mechanics and explain why their writing reflects an error in mechanics helps students to recognize that kind of error and learn the appropriate conventions for the first time. The following is an example of a revision dialogue with a basic writing student who did not fully answer my prompt and, in turn, also didn't really learn what was ungrammatical about his sentence: Instructor's Feedback: Why is this a run-on? Revise the sentence. Student Response: DNA is a perfect tool for law enforcement agencies to have especially with the criminals getting more advanced it gives the police an advantage. Instructor's Follow-up: This is still a run-on. Let's work in my office on how to correct run-on sentences, which combine two or more sentences without proper punctuation. As with the above example, the dialogues about mechanics can reveal a lack of understanding about the type of error and provide an opportunity to develop skill in that area, which might have gone neglected by the student without such prompting from me. Other students may have typos, which simply show a lack of proofing and editing. Generally, students respond to typo comments by bashfully acknowledging a "stupid mistake" that was overlooked in error. Then, they must correct the typo. Calling attention to these surface level mistakes helps reinforce good proofing and editing practices that should occur during the final stages of revision to build a credible ethos. Learning to Convey Suggestions Clearly

When students occasionally gave thin and unreflective responses, I realized that my comments mistakenly conveyed a neutral attitude about a missing, key element. For example, the following exchange with a composition student illustrates my miscommunication: Instructor's Feedback: I'm not sure where your thesis is. Student Response: It's in my first sentence, second paragraph. What I meant by this vague comment was that it was a problem of clarity, purpose, and focus that I could not find this essential feature of an essay, the thesis. The second sentence of his essay was not an arguable claim. What I was hoping the student would have done with my comment was to realize that he had not written a clear thesis statement and that he would provide one for the revision dialogue and subsequent draft. However, the lack of direction and nonchalant tone of my comment did not facilitate such a response. I had not clearly communicated my assessment, suggestions for revision, or my expectations for further dialogue. With some embarrassment, I responded to the student by acknowledging my lack of clarity, requesting revision of the thesis, and stating its importance in an essay: Instructor's Follow-up: I realize now that my comment is vague. What I should have said here is that the fact that I can't identify a thesis means that I need you to clarify what you are arguing by writing a clearer thesis. Why do you think that your lay magazine readers should easily understand the main point of your essay in terms of persuasiveness? In this follow-up comment, I admit my error to the student, which I believe creates a less authoritative relationship and, therefore, further enables more genuine collaboration between us. Also, I ask him to revise so that he would better achieve his own goals of readers' understanding of and adherence to his main argument-a rhetorical analysis of the student text with the student. Praise, Praise, Praise! An important counterpart to discussing weaknesses in a student's writing is to identify strengths as well, a strategy I incorporated into Comments to Comments. Since praise encourages students to overcome writing apprehension (Daiker 105) and offers "the psychology of positive reinforcement" (Irmscher 150), students develop more positive attitudes about their writing. Also, praising student writing lets students know that they are doing things well, which reinforces effective writing strategies, as Macrorie, Hirsch, Shaughnessy, and Diederich have argued. However, my early forms of praise were typically vague comments of "good" or "yes" in the margins. After receiving simple expressions of appreciation for the compliment rather than a critical dialogue, it was obvious to me that students need to develop an understanding of what is "good" writing within specific contexts so that they can use that strategy again in future writing. In turn, I modified my praise

feedback by identifying the strength of the text with comments such as "clear thesis." However, in response, I consistently received unreflective responses such as "thanks" or "The book shows us to do it this way." This student demonstrates attention to the book as a reference to disciplinary writing but does not express an understanding of how or why this strategy operates for the community or purpose. Thus, I needed to modify my praise in order to stimulate critical thought about why a particular sentence or strategy works well in a student's paper. In turn, I started to include a question along with the praise to stimulate discussion of the reasons the rhetorical element served the paper's effectiveness. However, responding to praise often perplexes students because they don't see a need to reply to a compliment with which they are in agreement. Therefore, I had to explain that even my praise necessitated their analytical responses. Addressing my praise of their written work directly helps students to realize how good a paragraph, detail, or the like actually is. I want students to recognize when and how they have moved closer to their rhetorical goals with effective writing, conclusions they may not have realized during the drafting of their papers. Clearly, the realization of one's strengths as a writer enables greater confidence in that writing ability. The following exchanges about praise are with three basic writing students: Instructor's Feedback: This scene came to life for me because of your vivid description of the cabin fire using active verbs and expressive adjectives. How did you decide to write these details? Student's Response: I carefully selected these words in an attempt to recreate for the reader the energy that was apparent at the time and which caused me to act with fear, anger, and finally action. Instructor's Follow-up: Clear reasoning! Instructor Feedback: Good topic sentence! Why are topic sentences important to paragraphs and essays? Student Response: Topic sentences let the reader know that the subject is changing and also keeps them interested. Instructor Follow-up: Yes! Instructor Feedback: Great Title! Why are great titles important? Student Response: The title got your attention and it stated what the essay was going to be about. It is also a fun one! Instructor Follow-up: yes! I agree!

Such student responses indicate analysis of rhetorical strategies, which lead to an understanding of how to produce particular effects for readers in subsequent writing tasks. Also, focusing on strengths celebrates what students have accomplished, which helps to change the instructor's role from authoritative corrector of mistakes to writing collaborator and encourager. What's the Final Word? As a revision strategy, instructors can use Comments to Comments either as the final act of revision on a paper or in conjunction with final rewrites to be submitted for a grade. The decision to follow-up Comments to Comments with a graded, revised draft depends on instructors' time constraints and on their desires to achieve or to do as Julie Jung suggests, disrupt closure to students' revision processes. Worth noting, students typically prefer to revise the draft after Comments to Comments in order to achieve the finished, improved text. In fact, when I have not asked students to turn in a revised essay after Comments to Comments, they report less satisfaction with the assignment and have criticized me for not requiring the follow-up revisions in the text. Several students associate improvements in their writing when they see the evidence before them in their final papers as compared to earlier drafts. As a result of this critique, I started to include a second draft in the project, which further reinforces the importance of revision as an ongoing process. After all, if students feel good about their writing, they will write more and become more confident. COMMENTS TO COMMENTS: MIXED REVIEWS For the past six years, I have taught Comments to Comments in a wide variety of writing courses that included basic writing, standard composition, honors composition, technical writing, and business communication. However, I adjust my feedback and expectations to suit the course and the level of student writing, which produces different forms of written dialogue, collaboration, and revision. In fact, my method of adapting the assignment for each course was developed in part by anonymous student feedback on the assignment. Students from the full range of writing courses answered the question, "Has Comments to Comments been an effective method of learning to revise your own writing? Why or why not?" In the introductory technical writing classes, students overall found the assignment to be useful and could complete the assignment without much difficulty. With more advanced writers, the assignment can feel like busy work because they have already interiorized much of the kind of revision dialogue in which I would engage them. The advanced business writing students, for example, completed Comments to Comments with thorough elaboration about their writing choices and with a greater percentage of disagreements with me than less experienced writers provide. Both of these factors reveal that they have had more experience with writing and that they have more direction in terms of how they wish to formulate their documents and why.

Therefore, I no longer use Comments to Comments with the senior level students. Instead, I've found that Comments to Comments makes a discernable difference with less experienced writers. Discussions of subsequent drafts and assignments reveal experimentation and practice with their new literacy and elevations in their confidence and interest in their writing. Also, in the technical writing classes, students learn the discourse of communities of science and technology, which is unfamiliar, formal rhetorical territory. I find that Comments to Comments calls students to apply the technical writing concepts from our textbook in written dialogue, and, therefore, incorporate those terms into their own work. It's exciting to see student writers' anxieties about writing diminish as they learn to talk about the assignments, ask questions about writing for an audience, and discuss how they plan to fulfill the requirements and their own purposes. While students continue to have criticisms about the assignment, they generally fall into one of two types and are few in numbers. A small percentage of students expressed a preference for speaking with me rather than writing with me their dialogues about revision. Among the basic writing students surveyed, only four students out of eighty either considered Comments to Comments to be as useful as or less useful than oral conferencing. I encourage these more face-to-face communication-- oriented students to come to my office for individual conferencing at any point in the revision process. The second criticism is that Comments to Comments takes a long time to complete, which is accurate if the student is responding specifically and elaborately in revision. For example, one student identified having trouble referencing the style manual to learn about a mechanical rule and explain how to revise the sentence. Another student stated that there were a lot of comments to which to respond, which made the assignment lengthy. Despite these relatively few criticisms, other students reported that Comments to Comments "wasn't that hard," was a "good learning tool," and "was very fun talking about something I know about [myself]." Overall, student feedback suggests that I continue to use Comments to Comments for teaching revision in classes with less experienced writers. The following kinds of positive feedback clearly prevailed among the students' responses and motivated my continued application and modification of the assignment: Yes, I think it makes you think about why you do certain things and how to do them right. Many times you don't know why you are making certain revisions but this makes you think about why you are doing them. Doing this allows you to understand revising better, therefore, being better at it in the future. (Basic writing student on essay assignment) Instead of just reading and probably ignoring your comments, I had to analyze what you were saying, think about the effect of the comment on my document and then decide how to go about either implementing the comment or telling you why I felt the comment wasn't good for

me and my purpose. It made me reason out my decisions for putting certain things in my document. I think this helped me to develop a more readable, scannable, and attractive technical document. (Technical writing student on resume assignment) I believe I learned more from doing the Comments to Comments than I have from just revising rough drafts. It made me think of more than one technique of revising. Instead of coming out and stating how to fix the work, the students are told what is wrong and they must figure out how to fix it. (Basic writing student on essay assignment) I do believe that the Comments to Comments assignment is a good idea because it gives you a good amount of time to answer the questions with a good response as well. I also think that it might be a good idea to revise the resume and have it graded a second time to make it an even better copy. (Technical writing student on resume writing) This feedback gave me the sense that students understood the goals of the assignment as well as the benefits of revision on their writing. Also, they wrote that they enjoyed talking about themselves and in "writing the play script." Further, students indicated satisfaction with the indepth exchange about their writing, which focused on aspects of persuasion and convention. Overall, the main difference between how I've used Comments to Comments in the various writing courses is the kind of feedback I give and, in turn, students' responses. On the one hand, in sophomore writing classes such as the technical writing classes, my feedback is focused on rhetorical issues-meeting audience expectations and document-form conventions, and achieving writing objectives. On the other hand, in firstyear writing courses, my feedback addresses assignment requirements, essay features, paragraphing, mechanical errors, and the use and documentation of research sources. As student writers across courses, cultures, and writing contexts have varying needs for improving their rhetorical effectiveness, even within a given course, I have to adjust the revision dialogue to meet those needs. Those needs become apparent to me as I get to know them in class with writing assignments, one-on-one conferencing, and in-class interaction. For example, with ESL students in my technical writing classes, my comments focused on their language mechanics related to writing in a second language, which differs from basic writing language mechanics issues. In sum, how I practice and implement Comments to Comments in writing classes is always under review and adjustment to individual students and their writing. More on Comments to Comments Future work on this pedagogy includes methods for assessing changes in students' awareness and application of revision. At this point, while I have considered setting up test classes as control groups, too many variables make any attempt to correlate changes in their papers with

their completion of Comments to Comments inconclusive. As it continues to evolve, this creative and critical pedagogy will focus on students' awareness of how they write, why they write, to whom they write, and ways that their writing can become more effective. As Robert Probst argues, teacher feedback should involve students in a "shared commitmentthey are not opponents sparring in a linguistic ring, the student attempting to slip confusions and inadequacies past the teacher, and the teacher attempting to catch, label, and castigate all the flaws" (70). Accordingly, Comments to Comments is a collaborative model of studentteacher revision; students are more likely to perceive instructors as being on their side, working on their behalf, rather than as an obstacle to overcome, psyche out, figure out, or manipulate in order to earn the desired grades. The sense of team spirit becomes a strong part of the complex power relationship between students and instructors. Further, teachers' feedback should suggest that student writing matters enough to warrant a collaborative revision endeavor. Calling students to actively dialogue with instructors about their writing makes it difficult for even resistant students to be apathetic. Such repeated readings are encouraged by the revision dialogue of inquiry, discussion, and negotiation between teacher and student. Through this experience with revision, students develop a literacy that makes face-to-face conferencing and peer review more productive and meaningful. Furthermore, with practice, the language of revision dialogue becomes the internal dialogue of self-- critique, therefore, making a more resourceful writer. Echoing Dawn Dreyer's purposes in teaching revision with students, I designed Comments to Comments to invigorate in students an attitude toward writing that involves self-awareness, effective communication with others, and interest in their own writing-all of which fuel repeated readings, reconsideration of original plans, and assessment of current and new directions in writing. Lehman, Pennsylvania -1-