Spring2010CEAreport | Rubric (Academic) | Educational Assessment

First-Year Writing Program Spring 2010 Course-Embedded Assessment Report of Results Dr.

Clancy Ratliff, Director of First-Year Writing
Thanks to Amanda Anderson, Joe Baumann, Garnet Branch, Joanna Davis-McElligatt, Jack Ferstel, John Guidry, Marie Hendry, Amanda LaRoche, Katie Manning, Jennifer Page, Christal Seahorn, and Elizabeth Sonnier, who served as readers.

Assessment Spring 2010
The readers scored papers in each of the three categories from 1-6, where 1-2 were poor, 3-4 were satisfactory, and 5-6 were outstanding. The FirstYear Writing Program rubric is on page 7 of this report. After a thorough norming exercise, two readers scored each paper. We met our benchmark of 70% of student papers scoring satisfactory or higher.

Overall results (n=100) Content
mean: 3.4

Poor 24% 25% 29% 28%

Satisfactory | Outstanding 76% 75% 71% 72%

Organization
mean: 3.31 mean: 3.39

Language Issues Overall
mean: 3.36

First-Year Writing Program CEA History
Fall 2008

The First-Year Writing Program conducted rounds of course-embedded assessment (CEA) in Fall 2008, Spring 2009, and Spring 2010. In Fall 2009, in lieu of CEA, students in English 101 took the Collegiate Learning Assessment exam. Students in the Fall 2008 CEA performed below expectations (our benchmark was 70% or more students with an overall score of satisfactory or higher); that semester, only 49% of student papers were scored satisfactory or higher. We attributed these results to unusually high attrition and student distraction in the wake of Hurricane Gustav and Hurricane Ike. Also, we revised the rubric.

Spring 2009
Students in the Spring 2009 CEA performed below expectations, but that semester, 59% of student papers were scored satisfactory or higher.

Spring 2010
For this round of CEA, I made three major changes in the process: 1. Skip Fox, a faculty member in English, suggested including papers from English 115 in the assessment. We decided to do this when we collected our Spring 2010 samples; using samples from one section of English 115, which is an honors course, gives a more accurate measurement of the range of achievement

2

across the freshman class. We had heretofore been scoring only papers from English 102 classes. Of the 100 papers we scored, 87 came from English 102 classes, and 13 came from an English 115 class. I do not know exactly what percentage of the freshman population typically places into honors courses from year to year, so it's unclear whether English 115 students are overrepresented or underrepresented in this sample. 2. Another change I made was to change the scoring range from 1-3 to 1-6. This change follows a method used by Dr. Asao Inoue, professor and Assessment Coordinator for the College of Arts and Humanities at California State University, Fresno, which directs assessment readers to put student work into "conceptual buckets": a score of 1-2 would put student work into the "poor" bucket, 3-4 in the "satisfactory" bucket, and 5-6 in the "outstanding" bucket. Readers now had the discretion to give a paper a "low satisfactory" or "high satisfactory" if there was some overlap between the categories, as there always is. 3. Finally, one key difference between the Spring 2010 CEA and previous rounds was that the First-Year Writing Program rubric had become widely available to students. It is printed in their required textbook, The Freshman Guide to Writing, beginning with the academic year 2009-2010 edition of the book, and it is on the First-Year Writing Program web site. All teachers of First-Year Writing courses are asked to use that rubric when grading student writing, and it is the rubric we use to settle grade disputes. I believe that these changes provided enhanced construct validity.

Future Plans and Concerns
We plan to conduct another round of CEA in Fall 2011. In the Spring 2011 semester, we will be doing a large-scale indirect survey assessment of the First-Year Writing Program, a project that is being supported by the Office of Academic Planning and Faculty Development in the form of an Instructional Improvement MiniGrant. For the next round of CEA, I will attempt to gather data about the percentage of freshmen who qualify for the Honors Program and use exactly that number of English 115 papers in the 100-sample set. Starting in Fall 2010, we made some changes to the First-Year Writing curriculum. We are now using new books that are, I believe, more closely aligned with the outcomes and the rubric we use. Perhaps these changes will raise or lower the scores in the next round of CEA, though I know that a cause-and-effect relationship cannot be proven. An ongoing concern I have is the increasing population of freshmen who take their first-semester writing course in high school dual enrollment programs. I am skeptical that any dual-enrollment program, no matter how rigorous, is equivalent to taking separate high school and college English courses. Students who go straight into English 102 simply have less time to practice writing; I advocate a curriculum that provides students with at least five full years of English courses: four years in high school and one full year in college. Most relevant to the CEA, though, is the question of what exactly is being assessed. Our writing program consists of a two-semester sequence: English 101 and English 102. Admittedly, there will always be transfer students who have taken their first-semester writing course at another institution, but the dual enrollment population is sizable, and some of Louisiana's dual enrollment programs have come under criticism for their lack of rigor above and beyond the regular requirements of high school English 4. One possible method for addressing this issue is to use the special-roster data the English department receives each semester and check the student writing samples against that data set; only students whose records show that they took English 101 at ULL would be included in the CEA, or their papers could be included but given designated numbers (unbeknownst to the readers) in the CEA sample set.

3

Raw Data (R1 and R2 refer to "Reader 1" and "Reader 2")
Paper Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Content Content Score R1 Content Score R2 Average 4 3 3 3 3 1 4 5 5 4 2 3 3 2 3 1 2 4 6 4 5 2 5 3 5 2 3 2 4 3 1 4 3 2 3 2 3 3 4 4 2 3 3 4 5 4 3 3 2 3 3 3 2.5 3.5 3 1 4 4 3.5 3.5 2 3 3 3 3.5 1.5 2.5 3.5 5 4.5 4.5 2.5 4 2.5 4 Organization Score Organization Score Organization R1 R2 Average 3 3 3 4 4 1 3 5 2 4 2 2 2 2 4 2 2 4 5 3 5 4 4 3 4 3 3 3 4 5 1 3 3 2 2 2 4 2 3 5 3 3 3 4 4 4 2 3 5 3 3 3 3 4 4.5 1 3 4 2 3 2 3 2 2.5 4.5 2.5 2.5 3.5 4.5 3.5 4.5 3 3.5 4 3.5 Language Score R1 3 3 2 5 5 1 5 5 6 4 4 3 2 3 4 2 1 5 5 4 4 5 5 3 5 Language Score R2 2 3 3 4 5 1 4 3 3 3 3 3 2 3 5 2 2 3 3 4 3 2 3 5 2 Language Average 2.5 3 2.5 4.5 5 1 4.5 4 4.5 3.5 3.5 3 2 3 4.5 2 1.5 4 4 4 3.5 3.5 4 4 3.5 Overall Average 2.88 3 2.72 3.94 4.06 1 3.75 4 3.19 3.31 2.38 3 2.38 2.81 4.13 2 2.25 3.63 4.56 4 4.25 2.94 3.81 3.44 3.69

4

Paper Number
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

Content Content Score R1 Content Score R2 Average
3 3 2 5 5 2 3 5 5 5 6 4 5 3 2 4 5 2 5 3 3 3 4 2 5 2 3 3 3 3 2 3 2 5 5 4 5 2 2 5 3 4 5 4 3 4 5 4 3 2 2.5 3 2.5 4 4 2 3 3.5 5 5 5 4.5 3.5 2.5 3.5 3.5 4.5 3.5 4.5 3 3.5 4 4 2.5 3.5

Organization Score Organization Score Organization R1 R2 Average
2 3 1 5 5 2 3 4 4 4 6 4 3 4 4 3 5 2 4 3 4 2 4 1 4 3 3 3 2 3 2 3 2 4 4 5 5 2 2 3 3 5 5 5 2 4 5 3 1 2 2.5 3 2 3.5 4 2 3 3 4 4 5.5 4.5 2.5 3 3.5 3 5 3.5 4.5 2.5 4 3.5 3.5 1 3

Language Score R1
3 4 3 3 4 2 3 5 3 5 6 4 3 3 2 5 5 2 4 4 3 3 4 3 4

Language Score R2
1 2 2 3 3 3 3 5 3 4 4 5 3 2 4 4 4 5 5 3 3 5 3 3 2

Language Average
2 3 2.5 3 3.5 2.5 3 5 3 4.5 5 4.5 3 2.5 3 4.5 4.5 3.5 4.5 3.5 3 4 3.5 3 3

Overall Average
2.38 3 2.31 3.56 3.88 2.13 3 3.69 4.13 4.5 5.19 4.5 3 2.69 3.38 3.56 4.69 3.5 4.5 2.94 3.56 3.81 3.69 2.06 3.19

5

Paper Number
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75

Content Content Score R1 Content Score R2 Average
3 4 1 5 4 3 3 3 4 2 3 1 3 2 2 3 4 2 5 3 3 5 1 4 2 3 3 2 4 4 4 4 3 3 1 3 2 1 2 2 3 4 1 2 5 2 6 1 3 3 3 3.5 1.5 4.5 4 3.5 3.5 3 3.5 1.5 3 1.5 2 2 2 3 4 1.5 3.5 4 2.5 5.5 1 3.5 2.5

Organization Score Organization Score Organization R1 R2 Average
3 5 2 3 3 4 3 2 3 2 2 2 2 4 3 3 3 3 4 3 3 5 2 4 2 2 4 2 5 4 5 4 4 2 1 3 2 1 3 3 3 4 3 2 4 3 6 3 2 5 2.5 4.5 2 4 3.5 4.5 3.5 3 2.5 1.5 2.5 2 1.5 3.5 3 3 3.5 3 3 3.5 3 5.5 2.5 3 3.5

Language Score R1
2 4 3 3 3 4 1 1 2 2 2 2 3 4 2 4 4 2 5 4 4 5 4 4 3

Language Score Language R2 Average
2 4 2 4 2 3 4 4 2 1 3 2 1 2 3 3 4 1 2 1 1 6 3 3 4 2 4 2.5 3.5 2.5 3.5 2.5 2.5 2 1.5 2.5 2 2 3 2.5 3.5 4 1.5 3.5 2.5 2.5 5.5 3.5 3.5 3.5

Overall Average
2.56 4 1.94 4.06 3.44 3.88 3.25 2.88 2.75 1.5 2.69 1.81 1.81 2.81 2.5 3.13 3.81 2.06 3.31 3.44 2.69 5.5 2.19 3.31 3.13

6

Paper Number
76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

Content Content Score R1 Content Score R2 Average
3 3 4 5 5 6 4 5 4 3 3 2 5 5 3 2 4 3 4 5 3 5 3 4 3 5 4 4 6 5 6 5 2 3 3 2 5 2 3 2 4 4 3 4 6 3 4 5 3 5 4 3.5 4 5.5 5 6 4.5 3.5 3.5 3 2.5 3.5 3.5 4 2.5 3 4 3 4 5.5 3 4.5 4 3.5 4 3.4

Organization Score Organization Score Organization R1 R2 Average
2 4 4 6 5 5 3 4 2 4 4 3 4 4 3 3 5 3 4 4 3 2 2 4 4 4 4 5 5 5 6 4 2 3 3 2 4 3 2 2 3 4 4 4 5 2 2 4 3 5 3 4 4.5 5.5 5 5.5 3.5 3 2.5 3.5 3 3.5 3.5 3 2.5 3 4.5 3.5 4 4.5 2.5 2 3 3.5 4.5 3.31

Language Score Language Score Language R1 R2 Average
2 4 4 6 6 6 3 3 4 5 3 2 6 5 3 2 5 4 4 4 3 4 3 4 2 3 4 4 6 5 5 2 4 2 3 4 2 4 4 2 2 5 2 4 6 4 3 5 3 5 2.5 4 4 6 5.5 5.5 2.5 3.5 3 4 3.5 2 5 4.5 2.5 2 5 3 4 5 3.5 3.5 4 3.5 3.5 3.39

Overall Average
3.25 3.81 4.19 5.63 5.13 5.69 3.63 3.31 3 3.44 2.94 3.13 3.88 3.75 2.5 2.75 4.44 3.19 4 5 2.94 3.31 3.63 3.5 4.06 3.36

7

First-Year Writing Program Rubric
Poor (1-2)
• Topic is too broad and general for a paper of its length • No clear main idea or sustained position in the paper (or argument is incredible to an academic audience) • Argument, if present, is unsupported, or evidence is insufficient • Paper is overly reliant on cliches or culturally conditioned/ ethnocentric assumptions and bias • Sources, if used, are used inappropriately (data dump, awkward integration, unintentional plagiarism, etc.) • Introduction does not orient the reader to the concerns of the paper or contextualize the subject of the paper • Arrangement of the paper is haphazard and random • Paragraphs do not have transitions that guide the reader from one idea to the next • Conclusion is absent or abrupt

Satisfactory (3-4)
• Topic is manageable for a paper of its length • Position/argument is comprehensible even if not clearly stated • Amount of evidence is sufficient • Demonstrates critical thinking (ability to recognize complexity, biases, and stereotypical representations; distinguishing fact from opinion) • Use of sources is appropriate; no serious problems with integration of other materials • Introduction is recognizable even if it is not always reader-based • Paragraphs generally treat one idea at a time • Attempts at transitions between paragraphs are made, even if they are awkward • Conclusion provides some closure to the argument, even if only a summary of the main points • Grammar, punctuation, and spelling are mostly correct • Student shows command of language (word choice/ vocabulary) • Varied sentence structure

Outstanding (5-6)

Content

• Topic is narrow enough to allow for a rigorous, nuanced treatment of the subject • Thesis is stated clearly • Evidence is ample to support position taken • Contains some acknowledgment of opposing/divergent views • Sources are used purposefully and strategically, integrated seamlessly

Organization

• Paper contains a clear, reader-based introduction, development, conclusion • Logical, smooth transitions between sections • Plan of development stated (forecasting statement, self-announcing structure to argument) • Conclusion does more than just summarize the paper; restates the thesis in a fresh way or includes a gesture (call for action, unresolved questions, etc.) • Paper is virtually free of error • Writer shows an unusual felicity with regard to word choice, turns of phrase (ex. uses obscure words, bon mots) • Sentence structure is complex but not cumbersome

Language Issues

• Frequency of error (of any type) seriously detracts from the content of the paper

8

Sign up to vote on this title
UsefulNot useful