A Comparative Study of Paper-and-Pen

Versus Computer-Delivered Assessment

Modes on Students’ Writing Quality: A
Singapore Study

Yin Ling Cheung

The Asia-Pacific Education

Researcher

ISSN 0119-5646
Volume 25
Number 1

Asia-Pacific Edu Res (2016) 25:23-33

DOI 10.1007/s40299-015-0229-2

1 23
Your article is protected by copyright and
all rights are held exclusively by De La Salle
University. This e-offprint is for personal
use only and shall not be self-archived
in electronic repositories. If you wish to
self-archive your article, please use the
accepted manuscript version for posting on
your own website. You may further deposit
the accepted manuscript version in any
repository, provided it is only made publicly
available 12 months after official publication
or later and provided acknowledgement is
given to the original source of publication
and a link is inserted to the published article
on Springer's website. The link must be
accompanied by the following text: "The final
publication is available at link.springer.com”.

1 23
 Author's personal copy
Asia-Pacific Edu Res (2016) 25(1):23–33
DOI 10.1007/s40299-015-0229-2
REGULAR ARTICLE
A Comparative Study of Paper-and-Pen Versus Computer-
Delivered Assessment Modes on Students’ Writing Quality:
A Singapore Study
Yin Ling Cheung
Published online: 30 January 2015
Ó De La Salle University 2015
Abstract Much research has been conducted to investi-
gate the quality of writing and high-level revisions in word
processing-assisted and pen-and-paper writing modes.
Studies that address cognitive aspects, such as experience
and comfort with computers, by which students compose
essays during writing assessments have remained relatively
unexplored. To fill this gap, the present study investigates
the impact of writing using computers versus traditional
writing with pen-and-paper based on 366 undergraduate
students in Singapore. It analyses the students’ thinking
processes, the quality of their written work, and their per-
ceptions of working with computers. The results indicate
that the students demonstrated a positive attitude towards
working with computers. Furthermore, the computer-
delivered writing mode has a positive impact on their
quality of writing in both technical aspects (content,
organization, vocabulary, language use, and mechanics)
and global aspects (setting macro rhetorical goal and
ordering information in achieving the macro rhetorical
goal). On the other hand, the results show that there was no
significant difference between the paper and computer
writing modes in their effects during the planning stage of
writing in-class reflections. The study contributes to writ-
ing research by encouraging teachers to give students an
option of delivery modes, beyond traditional pen-and-
paper, for their writing assessments.
Keywords Writing
Thinking processes

Writing quality
Perceptions of working with computers
Y. L. Cheung (&)
Nanyang Technological University, Singapore, Singapore
e-mail: yinling.cheung@nie.edu.sg
Introduction
In the last two decades, research has compared the effects
of word processing-assisted and pen-and-paper writing
modes on the quality of writing and higher-level revisions
of authors (Lam and Pennington 1995; Lee 2004; Li 2006;
Li and Cumming 2001). In language learning and tech-
nology research, investigators (e.g. Wolfe and Manalo
2004) have called for studies to address the cognitive
mechanisms (e.g. experience and comfort with computers)
by which students compose essays, during writing assess-
ments, using each of the composition media of word pro-
cessing and handwriting. In studies of computer-assisted
language learning, research has traditionally put emphasis
on the quality of writing and higher-level revisions as
outcomes. The participants’ experience and comfort with
computers have received relatively little attention in the
field of computer-assisted language learning. The bulk of
the computer-assisted language learning literature pro-
duced so far has done little in informing us how the cog-
nitive mechanisms impact academic writing activities
(Wolfe and Manalo 2004). The identified research gap
needs to be addressed because experience in working with
computers is important in helping us understand how
computer-delivered versus paper-based modes may impact
thinking processes and therefore the quality of writing, and
in helping writing teachers decide whether they should give
students the choice of writing medium in writing
assessments.
Given that computers have become commonplace in
teaching and learning in tertiary education, and given the
lack of comparative knowledge between students’ written
tests in classrooms using computer-delivered and paper-
based modes, a study that encompasses students’ computer
experience can provide us with insights into how their
123
 Author's personal copy
24
thinking processes and the quality of their in-class reflec-
tions differ according to the writing medium. In Singapore
and other ESL contexts, a study that focuses on writing
tests using both computer-delivered and paper-based
modes of writing in tertiary classroom settings can add to
our understanding of how advanced students can maximize
their performance with a best choice of writing medium in
writing assessments. It is a synergistic area of work
between academic writing and computer-assisted language
learning that merits research. In what follows, first I situate
the study within current literature about quality and higher-
level revisions of written products between paper and
computer modes, thinking processes in composing, and
attitude towards working with computers. I state the
research questions, context and the participants, and
methodology. Then, I describe and discuss the key findings
of this study. Last, I suggest limitations of the study and
further research directions.
Quality and Higher-Level Revisions of Written
Products Between Paper and Computer Modes
A number of studies have explored the quality of written
work with the use of pen-and-paper and word processors in
primary, secondary, and university settings. Two of these
studies published in the 1990s, by Lam and Pennington
(1995) and Owston and Wideman (1997), indicated that
pre-university students who used computers to produce
their compositions demonstrated work of high quality, as
indicated in the test scores. Lam and Pennington compared
the results of compositions administered to eight 9th grade
students in a pen-and-paper group and nine 9th grade stu-
dents in a computer group for 70 min in every 6-day lesson
cycle, over one academic year in Hong Kong. The findings
showed that students in the computer group performed
better in content, language use, mechanics, organization,
and vocabulary compared with those writing in pen-and-
paper. Owston and Wideman (1997) investigated the
compositions of 110 3rd grade students over a three-year
period in Canada in terms of (1) number of words per text,
(2) number of texts per students, and (3) number of texts
written with the use of computers and with pen-and-paper.
The results showed that those who used word processors
exhibited texts of high quality in terms of surface and deep
structures. Three studies, published after year 2000, by Li
and Cumming (2001), Lee (2004), and Li (2006), also
showed that university students who composed their essays
using word processors exhibited work of better quality
compared with those writing in pen-and-paper. Li and
Cumming (2001) looked at the 14 essays produced by
Hsin, a 29-year-old Taiwanese male speaker living in
Toronto, over a period of 8 months with alternate use of
pen-and-paper and a word processor. The subject’s
123
Y. L. Cheung
computer-assisted word processing essays scored higher in
content and language use. It was because ‘‘Hsin altered his
thinking processes while composing on the computer,
making more higher-level revisions, which contributed to
the improved quality of his essays’’ (p. 145). Li (2006)
compared the effect of word processing and pen-and-paper
on the writing produced by 21 Mandarin students in Tor-
onto who possessed advanced English language profi-
ciency. Li’s findings indicated that students who composed
essays with a word processor showed better quality work in
terms of argumentation as well as higher- and lower-level
revisions. Lee (2004) investigated whether English-as-a-
second-language students at a public university in the
Midwestern region of the United States scored better in
computer-based writing assessment versus pen-and-paper
writing assessment. The results of Lee’s (2004) study
showed that students scored higher in all the analytic
components for the computer-assisted essays, compared
with the pen-and-paper essays. The reasons for such results
can be attributed to the ‘‘authenticity of the writing envi-
ronment’’ (p. 18) and students’ positive attitude towards the
use of computers.
Contrary to the findings of the above-mentioned studies
that students who used computer-assisted mode showed
better quality writing compared with those using a paper-
and-pen mode, the studies conducted by Dalton and Han-
nafin (1987), Hawisher (1987), Teichman and Poris (1989),
and Lee (2002) discovered that the difference in writing
quality produced by pen-and-paper and word processors
was insignificant. In a year-long study, Dalton and Han-
nafin (1987) found that high-proficiency 11th grade stu-
dents, who were not trained in keyboarding skills, did not
show improvement in the quality of their computer-assisted
writing. They also did not pay much effort in planning
when they composed on computers. Another study in
which the use of word processors did not seem to lead to
better quality writing is the one by Hawisher (1987). In a
freshman rhetoric course in Illinois, Hawisher (1987)
compared the effects of the use of pen-and-paper versus
word processors in a control group’s and an experimental
group’s writing quality and frequency of revision, among
20 freshmen of advanced writing ability. Despite the fact
that the pen-and-paper group did more revisions than the
computer group, both groups did not show marked
improvement in the quality of writing. Likewise, in a
semester-long study in New York, Teichman and Poris’s
(1989) found that freshmen of average writing ability did
not show better quality of writing when composing their
essays on computers, compared with another group of
students using traditional pen-and-paper, given that both
groups had a similar level of apprehension about writing
when they enrolled in a College Writing course. Lee (2002)
asserted that the quality of computer-assisted essays was
 Author's personal copy
A Singapore Study
not better than that of pen-and-paper essays, as measured
by the test scores. In Lee’s study, it is worthy of note that
students who composed by word processor wrote longer
essays, but the longer sentences did not imply writing of
better quality. To sum up, the studies reviewed above have
produced inconsistent findings regarding the impact of pen-
and-paper versus word processors on the quality of written
work, based as they are on different sample sizes, different
contexts, different linguistic background of the partici-
pants, different training with regard to keyboarding skills,
and different lengths of study. Since writing is a compli-
cated activity, in the next section, I will discuss thinking
processes behind composing.
Thinking Processes in Producing Written Texts
Over the last four decades, there have been paradigm shifts
regarding thinking processes in writing. Rohman (1965)
and Britton et al. (1975) suggested the Pre-Write/Write/Re-
Write stage model and the Conception/Incubation/Produc-
tion stage model of composing, respectively. These stage
models describe writing as a linear process and as a series
of clean-cut stages. However, the assumption that writing
is a linear process overlooks some deeper aspects of writ-
ing. In reality, writers ‘‘constantly plan and revise as they
compose’’ (Flower and Hayes 1981, p. 367). Moreover,
stage models focus on the development of written products
from planning, writing, and revision; they neglect ‘‘the
inner processes of decision and choice’’ (Flower and Hayes
1981, p. 367). Essentially, in stage models of writing, the
thinking processes in composing are not explicitly
considered.
Drawing on a cognitive process model of writing,
Flower and Hayes (1981) argued that writing comprises the
rhetorical problem, the writer’s long-term memory, and the
writing process. The rhetorical problem includes the topic,
an awareness of the audience, and the writer’s role. The
writer’s long-term memory concerns knowledge about the
rhetorical problem that can be retrieved from the writer’s
mind or secondary sources such as books. Flower and
Hayes (1981) suggested that the writing process includes
planning, goal-setting, translating, and reviewing. Planning
includes generating ideas from long-term memory as well
as organizing ideas. Goal-setting includes establishing both
the global goal and sub-goals in writing. Throughout the
composing process, ideas are developed and revised to fit
the global goals and sub-goals. Translating is about putting
the ideas into words by considering lexical and syntactic
elements. Revising is reviewing the written product, and it
may lead to further planning and organizing the ideas as
well as translating. Unlike stage models of writing, which
perceive writing as a linear activity, the revision compo-
nent in cognitive process theory of writing can occur at any
25
time during composition and is ‘‘able to interrupt any other
process’’ (Flower and Hayes 1981, p. 374). Importantly, a
cognitive process model of writing suggests that writing is
a recursive activity. Writers constantly need to go back to
revise before they can move forward.
Based on constructivist theories of learning science,
Bereiter and Scardamalia (1987) suggested a knowledge-
transforming model of writing that includes the thinking
processes in planning, organizing, writing, and revising the
essays. In contrast to stage models of writing and the
cognitive process theory of writing, the knowledge-trans-
forming model places a strong emphasis on setting the
macro rhetorical goal and the sub-goals by the writers.
Throughout the composing process, skilled writers make
sure that the content and the vocabulary that they select
will help achieve the sub-goals and the macro rhetorical
goal of their written work. They also organize the infor-
mation so as to achieve the macro rhetorical goal of the
paper. Like a cognitive process theory of writing, the act of
writing is recursive in the knowledge-transforming model
because planning, organizing, writing, and revising an
essay may not be in sequence. In view of the macro rhe-
torical goal of the paper, the writer may need to plan new
ideas and constantly revise information to suit the global
goal. Nowadays, given that the use of computers is com-
mon among university students, it is worthwhile to study
their attitudes towards the use of computers because this
attitude will influence their learning outcome.
Attitude Towards Working with Computers
Researchers (e.g. Simonson 1995; Teo 2006, 2008) have
been interested in student-teachers’ attitude towards com-
puters. Teo (2008) suggested that understanding pre-ser-
vice teachers’ attitude towards the use of computers ‘‘may
provide useful insights into acceptance and usage of tech-
nology in learning’’ (p. 413). Simonson (1995) claimed that
it is important to understand students’ attitudes towards
computer use because positive attitudes will facilitate
positive learning outcome and achievement. Teo (2008)
investigated 139 Singaporean student-teachers who enrol-
led in two Postgraduate Diploma of Education programmes
at a university. The results show that these participants
exhibited positive attitudes towards computer use. In a
diploma course offered by a post-secondary institution in
Singapore, Teo (2006) concluded that there was a fairly
high level of positive attitudes with computer use, reiter-
ating the findings by Liu et al. (1998), and suggesting that
students with favourable attitudes towards computer use in
general correlated positively with their favourable attitudes
towards computer use for their learning in particular. In
Teo’s (2006) study, the students’ positive attitude with
computer is attributed to the Singapore Ministry of
123
 Author's personal copy
26
Education’s initiative to popularize the use of information
technology in teaching and learning at the primary, sec-
ondary, and tertiary levels since two decades ago (MOE
1997), in addition to widespread home computer ownership
with internet access. It is worth noting that the two studies
reviewed above have focused on Singaporean student-
teachers enrolled in a diploma and a postgraduate diploma
programme, and both programmes had a relatively small
student population. The studies did not examine student-
teachers in undergraduate degree programmes, which are
usually larger programmes in a university.
The Study
Research Questions
The purpose of the study was to examine possible differ-
ences in thinking processes during composition, and dif-
ferences in the quality of written products between paper-
and computer-based modes in the ALS 101 Academic
Discourse Skills course. The following three questions
were addressed in this study:
i. In what ways does the quality of the in-class reflections
differ across paper and computer modes?
ii. What differences exist in the thinking processes (e.g.
pre-planning, in-process planning, searching for
words/phrases, and reasoning about linguistic choices)
for composition on paper versus computer?
iii. What are the attitudes of the ALS101 students
towards working with computers?
Context and the Participants
In the January 2012 semester, 366 ALS101 Academic
Discourse Skills first-year undergraduate students volun-
teered to take part in the study. They ranged in age from 19
to 50 years old. There were more females than males,
which is typical for Education majors. The ALS101 course,
which focused on the writing of research papers, was a
compulsory course taken by all first-year undergraduates in
BA/BSc (Education) programme at a large public univer-
sity in Singapore. English is the first language and the
medium of instruction. Students learned research-paper
writing skills such as different key components of a
research paper; information-based versus rhetorical-based
approach to writing including the statement of the rhetor-
ical situation (i.e. purpose, audience, and context) and
macro rhetorical goal; ordering of information in achieving
the stated macro rhetorical goals; and choosing vocabulary
that suits the rhetorical situation. Instruction took the form
of 12 weekly classroom lessons, with each lesson lasting
123
Y. L. Cheung
180 min. Since data were collected as part of a course, the
submission of the in-class reflections was part of the
required coursework. If the students did not wish to par-
ticipate in the study, they would still be required to do the
reflections and surveys, but their data would not be inclu-
ded in the project. Students had the right to withdraw from
the study at any time. This project had no impact on stu-
dents’ grades, whether the students took part in it or not.
Confidentiality of all the participants was maintained at all
times.
Data Collection
In order to collect data to evaluate the effects of word
processing-assisted and pen-and-paper writing modes on
the quality of writing, four types of instruments were used:
(a) a computer attitude survey, (b) an in-class reflection
composed on a computer, (c) an in-class reflection written
with pen-and-paper, and (d) a survey on the thinking pro-
cesses during composition. Each instrument went through
piloting with an ALS 101 tutorial group in September–
November 2011. Piloting was being done as part of my
usual teaching practice and not for research or publication
purposes. The instruments being used were part of my
regular teaching. Student responses during the pilot
allowed me to make any necessary changes to the instru-
ments before final data for the research were collected. It
should be noted that the collected data were anonymised
before being entered into SPSS for analysis.
Computer Attitude Survey (See ‘‘Appendix 1’’ section)
The questionnaire utilized in this study was adapted from
Gressard and Loyd’s (1986) ‘‘Computer Attitude Scale’’. It
included 31 items rated on a 4-point scale (1 = strongly
disagree to 4 = strongly agree). The items covered the
subjects’ perceptions of computers, including fondness for
computers, computer anxiety, computer confidence, and
computer usefulness.
In-Class Reflection (Composed on Computer)
Student participants were given an hour in class to plan and
write up a reflection (300–400 words) on their portable
computers. This in-class reflection made up 10 % of the
course grade. They were asked to reflect on the most
important or insightful thing they learned from the ALS101
class that week (i.e. week 6). The class topic for that week
was writing the introduction and methodology sections of a
research paper. To ensure that the computer-assisted in-
class reflections would be carried out successfully, a week
before the students were to write the computer-assisted
reflection formally, I (in my capacity as the ALS101 course
 Author's personal copy
A Singapore Study
coordinator) asked them to bring their laptops to class for a
dress rehearsal of the writing. I also asked them to email
their finished work to their peers to simulate the submission
procedure. In the dress rehearsal, some students encoun-
tered technical problems in the emailing process. As a
solution, I requested all the tutors to bring their jump drives
and let the students save their work in the jump drives.
In-Class Reflection (Written with Pen-and-Paper)
The same group of students as the above were given an
hour in class to plan and write up a reflection (300–400
words) with pen-and-paper on the most important or
insightful thing they learned from the ALS101 class that
week (week 9). The class topic for that week was writing
the findings, discussion, and conclusion sections of a
research paper. Like the computer-delivered in-class
reflection, this pen-and-paper in-class reflection counted
for 10 % of the course grade.
Survey on Thinking Processes While Composing (See
‘‘Appendix 2’’ section)
The survey on thinking processes utilized in this study was
adapted from Li’s (2006) ‘‘The Six Types of Decision
Making while Composing’’. The ALS101 Academic Dis-
course Skills students were invited to voluntarily partici-
pate in the qualitative component of this study. The surveys
with open-ended questions were conducted at two points in
time: the surveys were conducted on the same day after the
students had completed their in-class reflections (i) on
computer (in week 6) and (ii) with pen-and-paper (in week
9), respectively. The interviewees were asked to report
their experience in terms of pre-planning [i.e. planning
before the writing starts, see Li (2006, p. 17)], in-process
planning [i.e. planning after the writing has started, e.g.
planning a paragraph, see Li (2006, p. 18)], searching for
the right words or phrases, reasoning about linguistic
choices [e.g. using linguistic rules or checking syntax or
morphology, see Li (2006, p. 18)], evaluation behaviours
[e.g. repeating orally what has been written down, Li
(2006, p. 18)], and checking spelling (Li 2006, p. 19).
Data Analysis
For quantitative data analysis, SPSS was used to input and
categorize the closed-ended survey responses. Concerning
the qualitative data, the method of analysis is as follows.
After reading through the open-ended responses from the
survey data for a few times to gain familiarity with the
content, my research assistants and I developed a set of
categories. First, we assigned each student a number. Then,
27
we categorized each student’s responses within the cate-
gories that we developed as they related to the research
questions. By comparing across subcategories, we were
able to identify commonality and idiosyncrasies in the
open-ended responses.
Regarding in-class reflections, we conducted two rounds
of analysis. For the first measure of the quality of writing,
we analysed five technical aspects, namely content, orga-
nization, vocabulary, language use, and mechanics. These
aspects were evaluated, as suggested by Lam and Pen-
nington (1995). I was the first rater for this task. The sec-
ond rater has a PhD in Curriculum and Instruction, with an
ESL emphasis, from a U.S. university. I trained the second
rater and monitored her work procedure, but did not
influence her rating. For the pen-and paper reflection, the
inter-rater agreement between me and the second reader
was 81 %. For the computer-assisted reflection, the inter-
rater agreement was 90 %.
With regard to the second measure of the quality of
writing, we scored four global aspects that are important
in the development of arguments at a global level, namely
stating the rhetorical situation (i.e. purpose, audience, and
context), setting macro rhetorical goal, ordering infor-
mation in achieving the macro rhetorical goal, and
choosing vocabulary that suits the rhetorical situation.
Analysis of the reflections was done by two research
assistants working independently. One assistant has a
Masters’ degree and the other has a PhD degree. They
were trained and monitored by me. The first step in the
identification of the above-mentioned four aspects was to
read all the reflections quickly. Subsequently, the sen-
tences were coded with reference to rhetorical situation,
macro rhetorical goal, ordering information in achieving
the macro rhetorical goal, and choosing the vocabulary
which suited the rhetorical situation. The aggregate scores
of each of these four aspects were computed separately.
Then, the averages of the scores by the two research
assistants, in each of the four global aspects, were
computed.
Results
The Quality of the In-Class Reflections Between Paper
and Computer Modes
We begin by comparing the quality of the writing samples
produced on computers versus pen-and-paper in terms of
the five technical aspects. The following patterns have
emerged from the analyses based on data from both the
computer-assisted (computer group henceforth) and pen-
and-paper (pen-and-paper group henceforth) reflections.
The data were analysed using five t tests for independent
123
 Author's personal copy
28 Y. L. Cheung

Table 1 Mean difference of the 5 aspects of composition group) were able to set the macro rhetorical goal, organize
Group Mean Standard
information to achieve the macro rhetorical goal, and choose
deviation words that suit the rhetorical situation (Table 3).

Content
Thinking Processes in Composing Across Paper
Computer 25.9372 3.36930 and Computer Modes
Pen-and-paper 22.5710 5.10752
Organization We analyse results from the survey on thinking processes
Computer 18.1694 1.59751 while composing (‘‘Appendix 2’’). The data analysis shows
Pen-and-paper 15.5164 3.12278 that there was no significant difference between paper and
Vocabulary computer modes in the aspects of in-process planning. Indeed,
Computer 19.6503 .87779 whether they used paper or computer to compose, the students
Pen-and-paper 18.4590 2.37503 all wrote down their initial ideas on a piece of paper during the
Language planning stage. With regard to searching for the right words or
Computer 24.0164 1.91741 phrases, the majority of students in either group stated that
Pen-and-paper 22.5792 3.41600 they had no difficulties in finding appropriate words and
Mechanics phrases. The two groups also acted similarly in proofreading
Computer 4.5328 .61734 their in-class reflections. Specifically, they commented that
Pen-and-paper 3.9426 1.15841 they focused on surface-level errors, rather than the devel-
opment of ideas at a global level while revising their in-class
reflections. With regard to reasoning about linguistic choices,
samples.1 The five independent t tests were performed at students who composed on computer demonstrated different
alpha = 0.01 through Bonferroni adjustment of the initial kinds of thinking processes compared with those who com-
alpha level of 0.05 to control for the family-wise error rate, posed by pen-and-paper. In particular, those who used com-
following Howell (2007). The dependent variable was the puter had no problem with linguistic rule and syntax. Those
subjects’ scores on the five aspects of composition, including who composed in pen-and-paper reported to have more dif-
content, organization, vocabulary, language use, and ficulties with respect to linguistic choices (e.g. finding the
mechanics. Prior to the test, Levene’s tests for homogeneous right syntax, grammar, and tenses) in the composition process.
variances showed that the variances of the two groups were Students who composed using either pen-and-paper or com-
equal, namely p = 1.00. This allowed for the analyses to be puter did not have significant issues with spelling.
performed using statistics for equal variances. More specif-
ically, the t tests showed a significant difference between the Student Perceptions of Working with Computers
groups for all 5 aspects of composition in which
t(730) = 9.92, p \ 0.01, with the computer group scoring Upon completion of the computer-assisted reflection, stu-
statistically higher than the pen-and-paper group (see dents responded to a four-point Likert scale survey
Table 1). In terms of the total score, the computer group with (‘‘Appendix 1’’ section) about their attitude towards com-
a mean of 92.31 performed significantly better than the pen- puters. The following pattern has merged from analyses
and-paper group with a mean of 83.34 (see Table 2). based on data from this computer attitude survey.
We now compare the quality of writing samples produced Regardless of their scores on the computer-assisted
on computer versus pen-and-paper in terms of the four global reflection, the results revealed that the students had positive
aspects of composition. The results are shown in Table 3. attitude towards working with computers. As indicated in
From the results, notice that the students in the two groups Table 4, the survey consisted of positively keyed items
performed similarly in identifying the rhetorical situation, but (N = 12) and negatively keyed items (N = 19). The stu-
more students in the computer group (than the pen-and-paper dents responded with ‘‘strongly agree’’ or ‘‘agree’’ to 12 out
of 31 items in the survey (i.e. items 1, 3, 4, 6, 9, 11, 12, 14,
1
I thank the reviewer for the suggestion. ANOVA is a statistical 16, 22, 25, 28). These items are positively keyed, as an
method for analysing the correlations between groups of data and is
agreement or strong agreement with the item indicates a
most helpful when more than two groups of data are involved. I agree
that it could be used for my analysis. However, the t test that I employ relatively high level of liking to working with computers.
in this paper is also a commonly accepted method for analysing the With regard to negatively keyed items, it is assumed that a
relationship between two groups of data, or pair-wise relationships in student who disagrees/strongly disagrees to a statement that
a larger set of data groups. It has been widely employed in language
s/he is not good with computers (e.g. item 2) actually works
writing research in particular. In this paper, I use the t-test to compare
the means between my two main groups of data, namely the computer well with computers. That is, a disagreement to ‘‘I’m no good
group and the pen-and-paper group, in several aspects. with computers’’ is logically similar to an agreement to ‘‘I’m

123
 Author's personal copy
A Singapore Study 29

Table 2 Average marks of the 5 aspects of composition

Aspect Pen and paper (N = 366) Computer (N = 366) t value Df Sig. 2-tailed

Content 22.57 25.94 10.53 730 .000***

Organization 15.52 18.17 14.47 730 .000***
Vocabulary 18.46 19.65 9.00 730 .000***
Language Use 22.85 24.02 7.02 730 .000***
Mechanics 3.94 4.53 8.60 730 .000***
Total 83.34 92.31 9.92 730 .000***
*** Significantly different at 0.0005 level

Table 3 Average scores in four global aspects of composition

Identifying the Setting the macro Organizing the information Choosing words to suit the
rhetorical situation rhetorical goal to achieve the macro rhetorical goal rhetorical situation

Computer 245 285 256 375

Pen-and-paper 243 254 197 309

good with computers’’. The students in this study responded
with ‘‘strongly disagree’’ or ‘‘disagree’’ to 19 out of 31 items
in the survey (i.e. negatively keyed items 2, 5, 7, 8, 10, 13, 15,
17, 18, 19, 20, 21, 23, 24, 26, 27, 29, 30, 31; Table 4).
Discussion
The Quality of the In-Class Reflections Between Paper
and Computer Modes
The finding that students using computer to compose per-
formed better than those using pen-and-paper is consistent
with the research results reported by Lam and Pennington
(1995) and Lee (2004). It should be noted that Lam and
Pennington’s (1995) participants, who were students in a
technical secondary school in Hong Kong, were divided
into separate control and experimental groups. The control
group completed their writing tests with pen-and-paper,
whereas the experimental group completed theirs with
computers. By contrast, my study compares the quality of
writing samples between writing modes (computer versus
pen-and-paper) but produced by the same group of par-
ticipants. The subjects in Lee’s (2004) study were five
undergraduate and 37 graduate students, in the context of
an ESL Placement Test at the University of Illinois at
Urbana–Champaign. My results complement those of Lam
and Pennington (1995) and Lee (2004) using a significantly
different population of 366 undergraduate students.
My findings seem to be contradictory to those of Wolfe
and Manalo (2004), who solicited their writing samples from
TOEFL tests in the U.S. The main finding of Wolfe and
Manalo’s (2004) is that there was no significant difference in
the quality of samples produced on computers versus and
pen-and-paper. However, it must be noted that Wolfe and
Manalo (2004) allowed the participants to choose the writing
medium, and this freedom of choice factored in the partici-
pants’ performance in the assessment (e.g. those who were
skilled in computers were likely to choose computer as the
writing medium). Hence, the two sets of results are not
directly comparable. As we will see in ‘‘Student perceptions
of working with computers’’ section, most students who took
part in my study had high levels of comfort and experience
with computers. This may provide a good explanation why
they received higher scores in computer-delivered writing
assessment than in paper-based assessment.
My study shows that those who composed on computers
did better in the global aspects (i.e. stating rhetorical sit-
uation, setting the macro rhetorical goal, organizing the
information to achieve the macro rhetorical goal, and
choosing the words in order to suit the rhetorical situation)
of higher-order thinking processes. This finding is consis-
tent with the research results reported by Li (2006). The
similarity could be due to the fact that participants in both
my study and Li’s study were used to working with com-
puters. It is therefore relatively easier for them to make
global-level revisions on computers than in pen-and-paper.
Thinking Processes in Composing Across Paper
and Computer Modes
The thinking processes exhibited by the students are related
to local revisions (Cumming and So 1996), that is, revi-
sions made at the word level (Li 2006). Specifically, stu-
dents in my study reported to have placed more emphasis
on correctness, surface-level grammar, and spelling in the
writing, rather than global revisions (i.e. higher-order
thinking processes such as an intention to communicate

123
 Author's personal copy
30 Y. L. Cheung

Table 4 Mean and standard deviation of the computer attitude survey (N = 366)
Items Mean Standard deviation

1. Computers do not scare me at all 3.47 .87

2. I’m no good with computers 1.85 .80
3. I would like working with computers 3.28 .75
4. I will use computers many ways in my life 3.53 .73
5. Working with a computer would make me very nervous 1.60 .79
6. Generally, I would feel OK about trying a new problem on the computer 3.04 .73
7. The challenge of solving problems with computers does not appeal to me 2.42 .90
8. Learning about computers is a waste of time 1.60 .79
9. I do not feel threatened when others talk about computers 3.13 .88
10. I don’t feel I would do advanced computer work 2.58 .94
11. I think working with computers would be enjoyable and stimulating 3.17 .72
12. Learning about computers is worthwhile 3.40 .69
13. I feel aggressive and hostile toward computers 1.41 .69
14. I am sure I could do work with computers 3.46 .70
15. Figuring out computer problems does not appeal to me 2.61 .95
16. I’ll need a firm mastery of computers for my future work 3.14 .78
17. It wouldn’t bother me at all to take computer courses 2.94 .91
18. I’m not the type to do well with computers 2.00 .86
19. I expect to have little use for computers in my daily life 1.59 .83
20. Computers make me feel uncomfortable 1.52 .76
21. I can’t think of any way that I will use computers in my career 1.39 .81
22. I would feel at ease in a computer class 3.14 .73
23. I think using a computer would be very hard for me 1.68 .72
24. Once I start to work with the computer, I would find it hard to stop 2.70 .84
25. Knowing how to work with computers will increase my job possibilities 3.57 .62
26. I get a sinking feeling when I think of trying to use a computer 1.64 .79
27. I will do as little work with computers as possible 1.86 .79
28. I would feel comfortable with a computer 3.32 .70
29. Computers make me feel uneasy and confused 1.61 .76
30. I do not enjoy talking with others about computers 2.17 .84
31. Working with computers will not be important to me in my life’s work 1.62 .83

with the readers) in the composing process. It is interesting
to note that, although students were explicitly taught the
rhetorical-based approach to writing (which is reader-
focused), the majority of them still revised their reflections
by focusing on surface-level errors, which are a feature of
the information-based (as opposed to rhetorical-based)
approach to writing. Most students remained more familiar
with the information-based approach despite explicit
instruction on the rhetorical-based alternative.
Notice that in my study there was no major difference
between paper and computer modes of composition in
terms of in-process planning and searching for appropriate
words and phrases. These findings are consistent with the
research results reported by Li (2006) on the influence of
word processing on writing assessment and on the writing
of English-as-a-second-language students. A major reason
for this similarity is that both Li’s (2006) study and my
study involved participants who were proficient in English.
The participants in both Li’s (2006) study and my study
learned English essay writing skills in Asian contexts (i.e.
Mainland China and Singapore, respectively). The students
might have been taught similar thinking processes about
writing procedures and rhetorical conventions of written
texts in their English lessons.
Student Perceptions of Working with Computers
Students expressed a high level of liking to working with
computers, as given in Table 4. This finding is the same as
the findings reported by Lee (2004) for his subjects on the
impact of word processor on an English-as-a-second-lan-
guage writing assessment in the U.S. The reason for the
similarity in the results could be attributed to the fact that
both Lee and I conducted the studies in universities where

123
 Author's personal copy
A Singapore Study 31

students are accustomed to writing on computers. Students in
both studies are used to completing their academic writing
tasks on computers, given that ‘‘computers become an
authentic mode for writing’’ (Lee 2004, p. 5). More impor-
tantly, the results of the computer survey supported the
adoption of computer-assisted in-class reflections, because
they increase ‘‘the chance for a better performance and
authenticity of the writing environment’’ (Lee 2004, p. 18).
Indeed, students got better scores in computer-delivered
writing tests in my study. This suggests that giving students
the choice to compose on computers, in addition to tradi-
tional pen-and-paper, during writing tests is a right direction.
university students an option to complete their writing
assessments using means other than traditional pen-and-paper.
The finding echoes Li’s (2006) recommendation that ‘‘edu-
cators involved in assessing learners’ writing should give
learners the option to choose the writing medium’’ (p. 17). It
agrees with Li’s (2006) contention that ‘‘as educators involved
in assessing learners’ writing ability, we must accept the
computer as a new writing tool and as such accept computer-
based writing when assessing learners’ writing ability’’ (p. 17).
Acknowledgments I would like to thank the ALS101 students who
took part in this study, and Singapore National Institute of Education
Academic Research Fund (RI 1/11 CYL) for funding this project.

Conclusion Appendix 1

This study examined the effects of using computers versus Computer Attitude Scale (Adapted from Gressard
traditional pen-and-paper on the writing of 366 first-year and Loyd 1986)
Singaporean undergraduates in an academic writing course
(i.e. an academic writing course). Compared with previous The following statements are designed to permit you to
paper-based versus computer-delivered writing research, this indicate the extent to which you agree or disagree with the
study investigated (1) students’ thinking processes in com- ideas expressed. Place a checkmark in the space under the
posing, (2) the quality of their writing, and (3) their perception label which is closest to your agreement or disagreement
of working with computers. The results of the study concluded with the statements.
that students showed a positive attitude towards working with
computers, and the computer-delivered writing mode has a Strongly Slightly Slightly Strongly
positive impact on students’ quality of writing. Existing evi- agree agree disagree disagree
dence-based research is scarce that investigates the students’ (4) (3) (2) (1)
experience and comfort with computers, and the quality of
1. Computers do not scare me
writing as it is influenced by the mode of writing, the work by at all
Lee (2004) being the only other example I am aware of. The 2. I’m no good with
current study adds to this important but largely neglected computers
evidence-based approach for the subject problem. 3. I would like working with
The research has a number of limitations. First, this project computers
was conducted at one public university, among first-year 4. I will use computers many
Singaporean students. The conclusions drawn may not be ways in my life
immediately generalizable to other student populations. Fur- 5. Working with a computer
would make me very
ther research targeted at other institutions of higher learning, nervous
for students in different stages of their studies, will advance 6. Generally, I would feel OK
our understanding of students’ attitude towards writing on about trying a new problem
computers in broader contexts. Another limitation is the on the computer
absence of interviews with student participants, due to insuf- 7. The challenge of solving
ficient manpower and busy schedules of the students who had problems with computers
does not appeal to me
teaching practicum assignments at the end of the semester.
8. Learning about computers
Future research studies could solicit more in-depth input from is a waste of time
the student participants through interviews, particularly email
9. I do not feel threatened
interviews which could be more efficient. This information when others talk about
might uncover further valuable insights into nuances of their computers
behaviours and attitudes towards computer-delivered versus 10. I don’t feel I would do
paper modes of completing their academic writing tasks. advanced computer work
This study was an attempt to help broaden the state of 11. I think working with
computers would be
knowledge about the application of computers to writing
enjoyable and stimulating
assessments, and to investigate the benefits of giving

123

32
Strongly Slightly Slightly Strongly
agree
(4)
12. Learning about computers
is worthwhile
13. I feel aggressive and
hostile towards computers
14. I am sure I could do work
with computers
15. Figuring out computer
problems does not appeal to
me
16. I’ll need a firm mastery of
computers for my future
work
17. It wouldn’t bother me at
all to take computer courses
18. I’m not the type to do
well with computers
19. I expect to have little use
for computers in my daily
life
20. Computers make me feel
uncomfortable
21. I can’t think of any way
that I will use computers in
my career.
22. I would feel at ease in a
computer class
23. I think using a computer
would be very hard for me
24. Once I start to work with
the computer, I would find
it hard to stop
25. Knowing how to work
with computers will
increase my job
possibilities
26. I get a sinking feeling
when I think of trying to use
a computer
27. I will do as little work
with computers as possible
28. I would feel comfortable
with a computer
29. Computers make me feel
uneasy and confused
30. I do not enjoy talking
with others about
computers
31. Working with computers
will not be important to me
in my life’s work
123
Author's personal copy
Y. L. Cheung
Appendix 2
agree disagree disagree
(3) (2) (1) Survey on Thinking Processes While Composing
(Adapted from Li 2006)
Q1 Could you tell me how you planned your reflections?
Q2 Could you tell me what you did in the in-process planning?
Q3 Could you tell me whether you had any difficulties in searching
for the right words or phrases? Give examples
Q4 Could you tell me whether you had any difficulties in making
linguistic choices (any problems in terms of coming up with the
right sentence structures, singular vs plural forms, past tense vs
present tense, etc.)?
Q5 Could you tell me how you proofread your work? Did you read
(orally) what had been written down? What did you revise
during your proofreading?
References
Bereiter, C., & Scardamalia, M. (1987). The psychology of written
composition. Hillsdale, NJ: Lawrence Erlbaum.
Britton, J., Burgess, T., Martin, N., McLeod, A., & Rosen, H. (1975).
The development of writing abilities. London: Macmillan.
Cumming, A., & So, S. (1996). Tutoring second language text
revisions: Does the approach to instruction or the language of
communication make a difference? Journal of Second Language
Writing, 5(3), 197–226.
Dalton, D. W., & Hannafin, M. J. (1987). The effect of word
processing on written composition. Journal of Educational
Research, 80, 338–342.
Flower, L., & Hayes, J. R. (1981). A cognitive process theory of
writing. College Composition and Communication, 32(4),
365–387.
Gressard, C. P., & Loyd, B. H. (1986). Validation studies of a new
computer attitude scale. Association for Education Data System
Journal, 18, 295–301.
Hawisher, G. E. (1987). The effects of word processing on the
revision strategies of college freshmen. Research in the Teaching
of English, 21, 145–160.
Howell, D. C. (2007). Statistical methods in psychology. Belmont:
Wadsworth.
Lam, F. S., & Pennington, M. C. (1995). The computer vs. the pen: A
comparative study of word processing in a Hong Kong
secondary classroom. Computer-Assisted Language Learning,
8, 75–92.
Lee, Y. (2002). A comparison of composing processes and written
products in timed essay tests across paper-and-pencil and
computer modes. Assessing Writing, 8, 135–257.
Lee, H. K. (2004). A comparative study of ESL writers’ performance
in a paper-based and a computer-delivered writing test. Assess-
ing Writing, 9, 4–26.
Li, J. (2006). The mediation of technology in ESL writing and its
implications for writing assessment. Assessing Writing, 11, 5–21.
 Author's personal copy
A Singapore Study
Li, J., & Cumming, A. (2001). Word processing and ESL writing: A
longitudinal case study. International Journal of English Studies,
1, 127–152.
Liu, X., Macmillan, R. B., & Timmons, V. (1998). Integration of
computers into the curriculum: How teachers may hinder
students’ use of computers. McGill Journal of Education,
33(1), 51–69.
Ministry of Education. (1997). Masterplan for information technol-
ogy. Singapore: Ministry of Education.
Owston, R. D., & Wideman, H. H. (1997). Word processors and
children’s writing in a high-computer-access setting. Journal of
Research on Computing in Education, 30(2), 202–220.
Rohman, G. (1965). Pre-writing: The stage of discovery in the writing
process. College Composition and Communication, 16, 106–112.
Simonson, M. (1995). Instructional technology and attitude change. In
G. J. Aglin (Ed.), Instructional technology: Past, present, and
future (pp. 365–373). Englewood, CO: Libraries Unlimited.
33
Teichman, M., & Poris, M. (1989). Initial effects of word processing
on writing quality and writing anxiety in freshman writers.
Computers and Humanities, 23(2), 93–103.
Teo, T. (2006). Attitudes toward computers: A study of post-
secondary students in Singapore. Interactive Learning Environ-
ments, 14(1), 17–24.
Teo, T. (2008). Pre-service teachers’ attitudes towards computer use:
A Singapore survey. Australasian Journal of Educational
Technology, 28(4), 413–424.
Wolfe, E. W., & Manalo, J. R. (2004). Computer medium compa-
rability in a direct writing assessment. Language Learning and
Technology, 8(1), 53–65.
123