Indirect Assessment of the First-Year Writing Program: Surveying UL First-Year Writing Students with the Consortium for the

Study of Writing in College (CSWC) Survey Instrument

Dr. Clancy Ratliff Director of First-Year Writing October 2011

Acknowledgements: This project was funded by the Office of Academic Planning and Faculty Development through an Instructional Improvement Mini-Grant.

Introduction
The field of rhetoric and composition studies has, by way of over four decades of research, come to agree on several best practices in the teaching of writing: having students write essays in a series of drafts, assigning brainstorming and prewriting activities, assigning a variety of genres, assigning papers written to a specific audience, holding one-on-one conferences with students about their writing, and more. For the firstyear writing curriculum at the University of Louisiana at Lafayette, I have designed course outlines, outcomes, and assignments with these practices in mind. These pedagogical techniques are the focus of a survey I distributed to students in English 101 and 102 at the end of the spring 2011 semester. The survey is based largely on questions designed in 2007 by Charles Paine, Robert Gonyea, Paul Anderson, and Chris Anson for the Consortium for the Study of Writing in College, a partnership between the National Survey of Student Engagement and the Council of Writing Program Administrators. The questions are intended to serve as a supplement to the NSSE to measure the degree to which students engage in writing throughout all their college coursework in both general education and their majors but the authors of the survey permit professors and administrators to use the instrument for smaller studies at their own universities. I believe that the results of this survey help to contextualize the results of our previous assessment efforts. In the fall of 2008 and the spring of 2009, the first-year writing program conducted course-embedded assessment with the help of Paula Phillips Carson and Carolyn Bruder. During these semesters, we collected samples of student writing from English 102 classes and evaluated them according to a rubric I designed. The results both semesters were disappointing; in neither round of assessment did we meet our benchmark of 70% or more students getting a rating of satisfactory or higher. We conducted another round of course-embedded assessment this semester with writing samples from spring 2010; this time, students from English 115 were also included in the data set, which revealed a more accurate representation of the range of achievement across the freshman class and, I believe, resulted in our meeting the benchmark: 72% of student papers were scored as satisfactory or better. My research assistant, Christal Seahorn, and I visited six sections of English 101 and 23 sections of English 102. Participation was voluntary; we visited classes with the instructors consent, and we offered them the opportunity to read the survey questions before deciding whether or not to let us distribute the survey to their students. One English 101 teacher took us up on this opportunity. One important caveat for these findings, then, is that the teachers self-selected to have their students participate in the assessment. Random selection of teachers and unannounced classroom visits would probably have yielded different results, but for the sake of morale, I decided to let teachers opt into the assessment. A total of 546 students took the survey, 426 of whom were in English 102 classes. In all, we visited six sections of English 101 and 23 sections of English 102.

This report contains results from both the 101 students and the 102 students, along with my commentary on the differences in the findings. It's important to note that students in English 102 taking the survey were asked to reflect on their experiences in both 101 and 102 as they answered the questions.

Which of the following most accurately describes your situation (check one): English 101 I have taken/plan to take all my first-year writing courses (90, 101, 102, or 115 as applicable) at ULL I have taken one or more first-year writing courses at another university and am completing the first-year writing requirement at ULL I have taken one or more first-year writing courses in high school through dual enrollment I have received automatic credit for one or more firstyear writing courses from testing (AP exam, ACT/SAT score, CLEP) and am completing the first-year writing requirement at ULL 108 (90%) English 102 350 (86%)

10 (8%)

36 (9%)

2 (1%)

6 (2%)

14 (3%)

Overall: I asked this question to get a sense of how many students take both courses of the writing sequence here at ULL and to differentiate between transfer students and dualenrollment students. I know from experience that had we done this survey in a fall semester, these numbers would have been different; many of the first-time freshmen who take English 102 in fall semesters have completed English 101 through dual enrollment. 1. During the current school year, for how many of your writing assignments have you done each of the following? Brainstormed (listed ideas, mapped concepts, prepared an outline, etc.) to develop your ideas before you started drafting your assignment: All assignments Most assignments Few assignments One assignment No assignments English 101 29% 31% 22% 13% 3% English 102 29% 33% 28% 4% 6% Difference 0% 2% increase in 102 6% increase in 102 9% decrease in 102 3% increase in 102

Overall: Brainstorming on the whole increased. More students in 102 brainstormed on at least a few assignments, and fewer brainstormed on 0-1 assignment.
Talked with your instructor to develop your ideas before you started drafting your assignment

All assignments Most assignments Few assignments One assignment No assignments

English 101 17% 29% 25% 17% 12%

English 102 15% 23% 37% 16% 9%

Difference 2% decrease in 102 6% decrease in 102 12% increase in 102 1% decrease in 102 3% decrease in 102

Overall: Talking with instructors to develop ideas increased in 102, but only slightly. Talked with a classmate, friend, or family member to develop your ideas before you started drafting your assignment

All assignments Most assignments Few assignments One assignment No assignments

English 101 18% 22% 27% 16% 16%

English 102 26% 23% 27% 12% 12%

Difference 8% increase in 102 1% increase in 102 no change 4% decrease in 102 4% decrease in 102

Overall: Brainstorming with peers increased slightly in 102.

Received feedback from your instructor about a draft before turning in your final assignment

All assignments Most assignments Few assignments One assignment No assignments

English 101 35% 20% 15% 13% 12%

English 102 29% 26% 23% 9% 12%

Difference 6% decrease in 102 6% increase in 102 8% increase in 102 4% decrease in 102 no change

Overall: By the end of English 102, more students reported having received feedback on drafts. Instructor comments on rough drafts are pedagogically powerful, and in the future, I would like to see all students report having received feedback on drafts for at least a few assignments (and instructors requiring submission of rough drafts for feedback and revision). However, that is difficult for teachers given the demands on their time and our class size in 101 and 102. Received feedback from a classmate, friend, or family member about a draft before turning in your final assignment (this includes class peer review activities)

All assignments Most assignments Few assignments One assignment

English 101 36% 23% 15% 13% -

English 102 32% 28% 26% 5%

Difference 4% decrease in 102 5% increase in 102 11% increase in 102 8% decrease in 102

No assignments

12%

9%

3% decrease in 102

Overall: Frequency of getting peer feedback on drafts increased slightly in 102. Visited the Writing Center to get help with your writing assignment before turning it in

All assignments Most assignments Few assignments One assignment No assignments

English 101 1% 2% 7% 11% 76%

English 102 2% 3% 8% 13% 75%

Difference 1% increase in 102 1% increase in 102 1% increase in 102 2% increase in 102 1% decrease in 102

Overall: Frequency of Writing Center use increased slightly in 102. Used an online tutoring service such as SMARTHINKING to get help with your writing assignment before turning it in

All assignments Most assignments Few assignments One assignment No assignments

English 101 0% 0% 3% 2% 93%

English 102 2% 1% 3% 4% 90%

Difference 2% increase in 102 1% increase in 102 no change 2% decrease in 102 3% decrease in 102

Overall: Frequency of online tutoring services increased slightly in 102. For first-year writing, I would prefer that students use our Writing Center rather than SMARTHINKING, as our Writing Center tutors know our writing program's rubric and outcomes. From what I understand, the university no longer has a contract with SMARTHINKING, so my concern is probably moot.
Used the course textbook to help you as you wrote and revised the assignment

All assignments Most assignments Few assignments One assignment No assignments

English 101 7% 11% 18% 11% 51%

English 102 7% 12% 32% 18% 30%

Difference no change 1% increase in 102 14% increase in 102 7% decrease in 102 21% decrease in 102

Overall: Textbook use increased in 102. I am troubled by students' lack of use of the textbook, as it is the main artifact of the writing curriculum. I am going to investigate this matter further.

Proofread your final draft for errors before turning it in

All assignments Most assignments Few assignments One assignment No assignments

English 101 67% 17% 11% 3% 1%

English 102 65% 21% 11% 2% 1%

Difference 2% decrease in 102 4% increase in 102 no change 1% decrease in 102 no change

Overall: Proofreading frequency increased slightly in 102. 2. During the current school year, in how many of your writing assignments did you: Narrate or describe one of your own experiences All assignments Most assignments Few assignments One assignment No assignments English 101 4% 15% 33% 33% 12% English 102 3% 13% 38% 29% 16% Difference 1% decrease in 102 2% decrease in 102 5% increase in 102 4% decrease in 102 4% increase in 102

Overall: Narrative writing based on personal experience decreased in 102, or rather, it remained at English 101 levels. That is to be expected; while personal narratives and personal essays are sometimes assigned in 101, 102 teachers are asked to assign research-based writing. Summarize something you read, such as articles, books, or online publications

All assignments Most assignments Few assignments One assignment No assignments

English 101 8% 21% 41% 22% 7%

English 102 23% 41% 30% 4% 1%

Difference 15% increase in 102 20% increase in 102 11% decrease in 102 18% decrease in 102 6% decrease in 102

Overall: Summary of outside sources increased significantly in 102. That is in keeping with the 102 curriculum in which students are required to engage with/in academic source-based research writing. Analyze or evaluate something you read, researched, or observed

All assignments Most assignments Few assignments One assignment No assignments

English 101 23% 27% 28% 17% 3% -

English 102 26% 43% 26% 4% 1%

Difference 3% increase in 102 16% increase in 102 2% decrease in 102 14% decrease in 102 2% decrease in 102

Overall: Analysis increased in 102, which is to be expected given the greater difficulty of English 102 and, perhaps, students' greater awareness of what it is they are actually doing in their writing.
Do field research (interviews, informal or formal surveys) for research-based writing

All assignments Most assignments Few assignments One assignment No assignments

English 101 0% 5% 12% 22% 59%

English 102 4% 10% 18% 23% 45%

Difference 4% increase in 102 5% increase in 102 6% increase in 102 1% increase in 102 15% decrease in 102

Overall: Field research increased in 102. It still remains at low levels, which is fine; teachers are under no instruction to require this kind of research. Teachers who are themselves experienced field researchers seem to be more confident with teaching (and more likely to teach) field research.

Argue a position using evidence and reasoning

All assignments Most assignments Few assignments One assignment No assignments

English 101 16% 22% 30% 26% 6%

English 102 24% 42% 23% 10% 1%

Difference 8% increase in 102 20% increase in 102 7% decrease in 102 16% decrease in 102 5% decrease in 102

Overall: We see a considerable increase in argument-based writing in 102. Again, I believe this can be attributed to both the 101/102 curricular distinctions and the students' increased sophistication and self-awareness.
Summarize and refute opposing views (counter-arguments) in argument-based writing

All assignments Most assignments Few assignments One assignment No assignments

English 101 11% 16% 28% 28% 15%

English 102 12% 35% 35% 14% 5%

Difference 1% increase in 102 19% increase in 102 7% increase in 102 14% decrease in 102 10% decrease in 102

Overall: Engaging with counterarguments increases in 102, I believe for reasons I've already stated.

Use ULL library resources (databases, books, journals) to find sources for researchbased writing

All assignments Most assignments Few assignments One assignment No assignments

English 101 7% 7% 9% 33% 42%

English 102 30% 28% 26% 11% 4%

Difference 23% increase in 102 21% increase in 102 17% increase in 102 22% decrease in 102 38% decrease in 102

Overall: We see a somewhat dramatic increase in library use in 102; again, this difference has its basis in the First-Year Writing curriculum.
Create the project with multimedia (web page, poster, slide presentation such as PowerPoint, etc.)

All assignments Most assignments Few assignments One assignment No assignments

English 101 0% 4% 3% 11% 80%

English 102 1% 1% 8% 13% 76%

Difference 1% increase in 102 3% increase in 102 5% increase in 102 2% increase in 102 4% decrease in 102

Overall: There's a slight increase in multimedia use in 102. Teachers are not required to assign work that uses some of the examples of multimedia, and our technology resources do not give much support to such instruction. I suspect the increase in 102 is due to teachers' assigning research presentations in conjunction with research-based essays. I included this question because it was in the original CSWC survey instrument.
Include drawings, tables, photos, screen shots, or other visual content into your written assignment

All assignments Most assignments Few assignments One assignment No assignments

English 101 0% 3% 11% 17% 67%

English 102 1% 1% 7% 13% 78%

Difference 1% increase in 102 2% increase in 102 4% decrease in 102 4% decrease in 102 12% increase in 102

Overall: Again, the inclusion of visual content in writing assignments is not something required of teachers or students. I suspect the overall decrease of visual content in 102 has to do with the fact that some English 101 teachers have students do a rhetorical analysis of a visual argument as one of the assignments, but that assignment is not typically given in 102. 3. During the current school year, for how many of your writing assignments have -

your instructors in ULL First-Year Writing courses done each of the following?
Provided instructions describing what he or she wanted you to do

All assignments Most assignments Few assignments One assignment No assignments

English 101 77% 12% 5% 3% 1%

English 102 79% 15% 3% 1% 2%

Difference 2% increase in 102 3% increase in 102 2% decrease in 102 2% decrease in 102 1% increase in 102

Overall: Student perception of having received instructions on assignments increased slightly in 102.
Assigned readings from the course textbooks in conjunction with the assignment

All assignments Most assignments Few assignments One assignment No assignments

English 101 40% 21% 18% 2% 18%

English 102 31% 38% 22% 4% 4%

Difference 9% decrease in 102 17% increase in 102 4% increase in 102 2% increase in 102 14% decrease in 102

Overall: Student perception of having been assigned reading from the course textbooks increased somewhat. I asked these questions about textbook use because occasionally (though less so in the last year and a half), teachers complain that they don't like the required books, but I wanted to get a clear sense of how much the students and teachers actually use the books.
Explained in advance what he or she wanted you to learn

All assignments Most assignments Few assignments One assignment No assignments

English 101 55% 23% 15% 2% 3%

English 102 49% 29% 17% 2% 3%

Difference 6% decrease in 102 6% increase in 102 2% increase in 102 no change no change

Overall: There was no significant change in this number between 101 and 102. I will encourage teachers to explain learning outcomes for particular writing assignments by continuing to give them language to express these learning goals -- referencing the course outcomes and more specific learning goals.

Explained in advance the criteria he or she would use to grade your assignment

All assignments Most assignments Few assignments One assignment No assignments

English 101 65% 12% 11% 3% 7%

English 102 58% 20% 16% 2% 3%

Difference 7% decrease in 102 8% increase in 102 5% increase in 102 1% decrease in 102 4% decrease in 102

Overall: Student perception of instructors' clarity in presenting grade criteria declined slightly in 102. I suspect that this is due to the increased emphasis on research paper writing in 102 and students' confusion about how MLA documentation format is weighted in the grades (based on my interactions with students in grade appeal cases) -as well as their generalized confusion about the genres of academic research writing.
Referenced the First-Year Writing Program grading rubric in the Freshman Guide

All assignments Most assignments Few assignments One assignment No assignments

English 101 40% 15% 14% 5% 23%

English 102 30% 23% 23% 7% 17%

Difference 10% decrease in 102 8% increase in 102 9% increase in 102 2% increase in 102 5% decrease in 102

Overall: Students perceived that their instructors referenced the writing program rubric slightly more frequently in 102, with a greater percentage of students reporting that the instructors referenced the rubric for at least a few assignments. I will continue to remind instructors that the rubric is a useful tool for them (and certainly for students).
Provided a sample of a completed assignment written by the instructor or a student (this includes sample ULL student essays in the Freshman Guide)

All assignments Most assignments Few assignments One assignment No assignments

English 101 31% 17% 31% 12% 7%

English 102 27% 27% 28% 10% 8%

Difference 4% decrease in 102 10% increase in 102 3% decrease in 102 2% decrease in 102 1% increase in 102

Overall: No significant change from 101 to 102. I think it's more important in English 102 for students to see examples of the genres of papers they are expected to write, written by student authors like themselves; while they have ample opportunity to read arguments written by journalists and other professional writers, they rarely have the opportunity to read student research papers. Journalists and professional writers in

popular magazines are not held to the same standards of citing their sources (certainly not in the format students are expected ; it is assumed that the publication has fact-checkers who
Asked you to do short, ungraded, informal writing exercises in or out of class

All assignments Most assignments Few assignments One assignment No assignments

English 101 23% 9% 35% 13% 18%

English 102 17% 21% 34% 11% 16%

Difference 6% decrease in 102 12% increase in 102 1% decrease in 102 2% decrease in 102 2% decrease in 102

Overall: There was a slight increase in 102. It's unclear how to interpret these answers; my sense is that teachers do assign a good bit of informal journal writing and in-class writing, but perhaps students do not see it as directly connected to their major assignments. Sometimes it's not, and the teachers are just having the students write for the sake of practicing writing.
Asked you to give feedback to a classmate about a draft or outline the classmate had written

All assignments Most assignments Few assignments One assignment No assignments

English 101 43% 9% 23% 16% 7%

English 102 32% 29% 27% 5% 7%

Difference 11% decrease in 102 20% increase in 102 4% increase in 102 11% decrease in 102 no change

Overall: There was a slight increase in peer response exercises in 102. While peer response exercises aren't mandatory in first-year writing, they do help students meet our outcome of "engage in writing as a recursive process," and they help students in basic project management -- not putting their essay-writing off until immediately before the deadline. I'm pleased to see that 88% of students reported having done peer response exercises for at least a few of their assignments.

Conclusion
Overall, I am satisfied with these results. While we have conducted direct assessment of student work, I was interested in what a combination of direct and indirect assessment would show. I wanted to see what students perceptions were of what their instructors required them to do. I believe its possible that teachers may require many of these activities but that students may not recall having engaged in them, and this survey shows which activities the students have retained. Now that I have the results of the survey, I am planning closing-the-loop measures: first, teaching workshops devoted to the areas Id -

like to improve, which for me include students getting feedback from peers, tutors, and teachers prior to turning in their papers, as well as doing more reading, particularly the course textbooks, and more analysis and evaluation in their work (though I understand that students may actually be doing analysis and evaluation but not realizing it. Thats another area Id like to address: metacognitive skills in writing). I am also designing curriculum materials targeted toward these areas -- assignment handouts, classroom activities, etc. -- to upload to my Moodle repository of instructional support materials for first-year writing instructors.