Page 1

A review of rubrics for assessing online discussions

Bobby ElliottScottish Qualifications AuthorityAbstract

This paper explores current practice in the assessment of online forums. It does thisby reviewing the literature relating to this area of online learning, and extracting therubrics contained within that literature to discover best practice as defined by theleading writers in this field.Twenty rubrics were analysed to discover their common characteristics. Theserubrics were also reviewed for their quality in terms of validity, reliability and a new

characteristic called “

fidelity

”

.The results of this study show that there is an inconsistent approach to theexpression and definition of rubrics for the assessment of online discussions; that thepurpose of the rubrics appears to be confused; and that their validity and,particularly, their fidelity are low.The paper concludes by recommending that rubrics continue to be used for theassessment of online discussions but that a more consistent approach is taken totheir construction and definition, and that current practice is changed to improvevalidity and fidelity.

The characteristics of a “good” marking scheme are provided to

help faculty todevelop rubrics.

Keywords

: assessment, learning, discussion, forums, criteria, rubric, markingscheme, online, writing.

A review of rubrics for assessing online discussions

Page 2

Introduction

This introductory section seeks to provide background information on the key themesin this paper, which are: online discussion boards, online writing, and assessment.

Educational benefits of discussion boards

The educational benefits of online discussion boards are well known and longestablished. Jonassen, Davison,

et al

(1995) expressed their suitability for learning asfollows:

“The dialogue serves as an instrument for art

iculation because in the processof explaining, clarifying, elaborating, and defending ideas, cognitive processes

involving integration, elaboration and structurisation took place”.

Newman, Webb and Cochrane (1995) found that students contributed more outsidematerial and experiences, and integrated ideas better, when using online discussions.Hoag and Baldwin (2000) report that students learn more in an online collaborativeclass than in a face-to-face classroom, and they also acquire greater experience of team work, communication, time management, and technology.Some academics have claimed that asynchronous discourse is inherently self-reflective and therefore more conducive to deep learning than synchronous discourse(see Riel, 1995, for example). Berlin

er (1987) reported that the increased “wait time”

(the time available to discuss each topic) increases the opportunities for reflectivelearning.Swan, Shen, & Hiltz (2006) summarise the benefits of online forums:

“Many researchers note that students perc

eive online discussion as more equitable and more democratic than traditional classroom discussions because it gives equal voice to all participants. Online asynchronous discussion also affords participants

the opportunity to reflect on their classmates’ c

ontributions while creating their own, and to reflect on their own writing before posting it. This creates a certain mindfulness and reflection among students. In addition, many researchers have noted the way participants in online discussion perceive the social presence of their colleagues, creating feelings of community. Such findings have led educators to conclude that asynchronous online discussion is a particularly rich vehicle for

supporting collaborative learning.”

Some drawbacks

The reduction in constraints on time and place can result in pressures on studentsand faculty to read and participate in online forums (see, for example, Gabriel, 2004,Hiltz and Wellman, 1997, and Wyatt, 2005). Peters and Hewitt (2009) carried outformal interviews with 57 post-graduate students undertaking an online programmeand discovered that the volume of messages was their greatest frustration.

The lack of visual communication clues, “disembodied learning” as Dreyfus (2002)

put it, can cause conflict and anxiety among learners. Mixed patterns of participationare common, with some learners dominating discussions and some inhibited fromcontributing. The text-only format of traditional discussion boards can also inhibitinteraction and can disadvantage learners with poor writing skills.

A review of rubrics for assessing online discussions

Page 3

Online writing

Online discussion boards are one instance of what has been variously described as

“Web 2.0 writing” , “online writing”, “digital writing” or “digital learning narratives”,

which can take various forms including online forums, blogs, wikis, social networks,instant message logs, and virtual worlds. Many of the issues raised in this paper arerelevant to all forms of online writing. Most of the afore-mentioned benefits of onlinediscussion boards are, in fact, benefits of any form of (asynchronous) online writing.This new medium provides new affordances

–

new ways of utilising the medium tocommunicate and collaborate. For example, online writing makes visible such thingsas co-operation, collaboration, and self-reflection; t

he learner’s thought processes

are also more apparent. The inclusion of multimedia (such as audio and video) isstraight-forward. Referencing (hyperlinking) to related resources or information issimple. The asynchronous nature of many online communications makes the time

and place of contributions more flexible, and provides more “wait time” (see above)

to improve opportunities for reflective writing. The writing may have an audience farbeyond the walls of the university (perhaps a national or global audience). Authenticity can be improved by tackling real-world issues and seeking feedback from peers and experts across the world. The potential for producing authentic, co-constructed, interconnected, continuously improved, media-rich information objectsof national or international interest is unique.These new affordances have significant implications for assessment. They provide anopportunity to assess skills that were previously considered difficult, or impossible, toassess using traditional approaches.

Assessment of online learning

The word “assessment” is derived from its Latin root

assidere

, which means “to sitbeside”. At its most basic level, assessment can be defined as “observing learning”

(Glossary of assessment terms, 2002). For the purposes of this paper, a moredetailed definition will be used:

“Assessment is an on

-going process aimed at understanding and improving student learning. It involves making our expectations explicit and public; setting appropriate criteria and high standards for learning quality; systematically gathering, analyzing, and interpreting evidence to determine how well performance matches those expectations and standards; and using the resulting information to

document, explain, and improve performance.”

(Angelo, 1995)

Characteristics of assessment

Assessment is traditionally viewed from two perspectives



validity



reliabilityFor the purposes of this paper, a third characteristic will be considered:

fidelity

.Sadler (2009) proposed this characteristic of assessment, which is explained below.

“Fairness” (the equality of an assessment) and “practicality” (the feasibility of an

assessment) are sometimes also considered as characteristics of assessment.