151

Chapter IX
Validation of E-Learning
Courses in Computer Science
and Humanities:
A Matter of Context

Robert S. Friedman
New Jersey Institute of Technology, USA

Fadi P. Deek
New Jersey Institute of Technology, USA

Norbert Elliot
New Jersey Institute of Technology, USA

ABSTRACT

In order to offer a unified framework for the empirical assessment of e-learning (EL), this chapter presents
findings from three studies conducted at a comprehensive technological university. The first, an archival
study, centers on student performance in undergraduate computer science and humanities courses. The
second study, a survey given three times within EL classes, investigates the variables of learning style,
general expectation, and interaction in student performance. The third study investigates student per-
formance on computer-mediated information literacy. Taken together, these three studies—focusing on
archival, process, and performance-based techniques—suggest that a comprehensive assessment model
has the potential to yield a depth of knowledge allowing shareholders to make informed decisions on
the complexities of asynchronous learning in post-secondary education.

Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
 Validation of E-Learning Courses in Computer Science and Humanities
INTRODUCTION
The aim of this chapter is to present evidence
derived from three studies of e-learning (EL)
targeting student outcomes. The studies were
undertaken to identify a profile for potential suc-
cess and support policy guidelines for limiting
registration and course enrollment. This aim is
achieved through discussion of three studies that
were conducted at a comprehensive technological
university in the United States, where EL has his-
torically been offered as an asynchronous, online
alternative to traditional face-to-face classes, with
content management systems such as Webboard,
WebCT and Moodle providing a “virtual class-
room” environment;; the difference in persistence
and success rates between the two modes of course
delivery, however, are comparatively lower for
online sections. The first study is archival and
centers on student performance in undergraduate
computer science and humanities courses. The
second study, a survey distributed three times
within humanities EL courses, investigates the
variables of learning style, general expectation,
and interaction in student performance. The
third study investigates student performance on
computer-mediated information literacy tasks.
Our findings confirm not only that students in EL
sections lack the required technical and informa-
tion literacy skills to succeed. Taken together,
these archival, process, and performance-based
techniques warrant a unified framework for the
empirical assessment of EL. Our findings suggest
that a comprehensive assessment model has the
potential to yield a depth of knowledge allowing
shareholders to make informed decisions on the
complexities of asynchronous learning in post-
secondary education.
New Jersey Institute of Technology (NJIT) has
a history in e-learning (EL) beginning in the
1980s when researchers in the university’s De-
partment of Computer and Information Science
created and deployed the Electronic Information
Exchange System for use in the original Virtual
ClassroomTM (Hiltz & Turoff, 1993). In 2002, with
computing education at its enrollment height,
152
approximately three-quarters of the 150 EL class
sections (15-week semester cohorts) offered were
in computer science or information systems at the
undergraduate and graduate levels. Today, with
5,585 undergraduate students and 2,822 graduate
students enrolled in the fall of 2008, demand for
EL remains strong, and the disciplinary distribu-
tion of the approximately 60 EL sections offered
during the current academic year is balanced
among majors in computer science, information
systems, information technology, and manage-
ment; several of the university’s general university
requirements in humanities are, as well, offered in
an EL format. All class teachers are historically
committed to delivering high quality instruction,
and all are committed to an empirical base for
decision-making regarding the evaluation of these
courses to facilitate positive student outcomes in
terms of success with courses and degree comple-
tion (Foster, Bower, & Watson, 2002).
This chapter presents our efforts—through
archival studies, surveys, and performance mea-
sures—to come to terms with the complexities of
offering E-Learning courses. Within an environ-
ment that would be, in mission and vision, ideally
suited to successful asynchronous instruction, it
might be imagined that all measures would report
incomparable success, yet such is not the uniform
case. While much has been gained in our under-
standing of the complex variables and validation
processes involved in justifying information,
much remains to be done (Millwood & Terrell,
2005). Our studies reveal that a triangulated model
is promising when a variety of shareholders in-
vestigate what really happens in asynchronously
offered undergraduate courses.
Archihiudy: Two
Diciplii
Research on rates of student success in EL classes
tends to be drawn from single samples. Yet com-
parison of two disciplines—one invested in re-
Validation of E-Learning Courses in Computer Science and Humanities

sponding to empirically-oriented tasks in a limited
response format (responding to multiple-choice
questions or creating computer code), the other
invested in responding to verbally-oriented tasks
in a open response format (participating in on-
line discussions or submitting essays)—seemed
ideal in allowing more to be known about the
specifics of EL across disciplinary frameworks
(Elliot, Friedman, & Briller, 2005).
Analysis of Undergraduates Enrolled
in Computer Science Courses
In 2002, administrators and faculty, attempting to
address the existence of low student success rates
in EL classes—and, subsequently, lower time to
completion and graduation rates—learned they
were not alone. In the USA, retention in E-Learn-
ing is a recurring issue. Low perseverance rates
are found to fall in disproportional ranges when
compared to traditionally taught classes (Institute
for Higher Education Policy, 1999; Zimmerman,
2002). A recent study regarding perseverance in
an EL introductory computer science (CS) class,
for example, noted that the “dropout rate” (i.e., a
student who submitted some work and then either
withdrew from the class or failed to take the final
examination) was 42%, a rate much higher than
the 12% and the 26% for the same class, taught
in prior years by the same instructor in a face-to-
face format (Mock, 2003). These numbers were
not greatly different from those within computing
programs at NJIT.
During the spring of 2003, administrators
in the College of Computing Sciences (the ad-
ministrative transformation of the then-expand-
ing Department of Computer and Information
Science) determined to address the disparity
between EL classes’ low passing rates relative
to sections offered in a traditional face-to-face
(FTF) format. To validate that perception and
to warrant changes in enrollment policy, student
records were extracted from fall of 1996 to spring
of 2002. The total sample of students for EL
classes was 2,554, as Table 1 demonstrates, while
the total number of students for the FTF classes
was 15,468. Academic preparedness, withdrawal
rates categorized according to grade point average
(GPA), and class repetition were addressed in the
archival study.
Taking the college admission SAT Reason-
ing Test, which tests students’ subject matter
knowledge in reading, writing, and mathematics
(College Board, 2008a) as a standard measure of
preparedness, the archival study revealed that the
SAT score for the students in EL courses (M =
1097) was comparable to the average FTF score
(M = 1106). Similarly, the SAT scores for students

Table 1. Computer science courses: An archival analysis of grade point average (GPA)

All CS Courses Total Seats Cumulative GPA Cumulative GPA from Cumulative GPA
Less than 2.0 2.1 to 3.0 from 3.1 to 4.0

Total enrollment 18,022 955 9,872 7,195

Number withdrew 1,9101 185 1, 264 452
Percent withdrew 10.5% 19.4% 12.8% 6.28%
Face to Face Enrollment 15,468 833 8,534 7,110
Number withdrew 1479 142 1009 328
Percent withdrew 9.6% 17% 11.8% 4.6%
ELearning Enrollment 2,554 122 1,338 1,094
Number withdrew 422 43 255 124
Percent withdrew 16.5% 35.2% 19.1% 11.3%

153
 Validation of E-Learning Courses in Computer Science and Humanities
who withdrew from the EL courses (M = 1095)
and students who completed the EL course (M =
1099) were also comparable. With the acknowl-
edgement of the College Board (2008b) that a
range of 30 to 40 points reflects ability, the SAT
scores of students who completed the FTF courses
(M = 1131) may be understood to be just within
the Board’s established range of difference than
those who withdrew (M = 1098).
If the academic preparation was similar in a
comparison of EL and FTF undergraduate classes,
the withdrawal rate revealed differences. Overall,
the withdrawal rates from the EL classes were
higher (16.5%) than from the FTF courses (9.6%).
Examination of the GPA of the students revealed
striking differences: for 955 students with GPAs
below 2.0, 35.2% withdrew from EL classes, while
only 17% withdrew from FTF classes. Further
study revealed that 66.7% of students with a GPA
below 1.0 withdrew from E-Learning classes,
while only 16.7% of that same group withdrew
from FTF classes; in a similar pattern, 70% of
students with a GPA between 1.1 and 2.0 withdrew
from E-Learning classes, while only 21.3% with-
drew from FTF classes. These were the highest
withdrawal rates identified in the study.
For the 9,872 students with GPAs between
2.1 and 3.0, 11.8% withdrew from FTF classes,
while 19.l% withdrew from EL classes. On the
highest end of the spectrum, the withdrawal pat-
terns were retained: for the 7,195 students with
GPAs between 3.1 and 4.0, only 4.6% withdrew
from FTF classes, while 11.3% withdrew from
EL classes. Further analysis revealed that for the
very best students—those with GPAs between 3.5
and 4.0, only 2.9% withdrew FTF classes, while
7.6% withdrew from EL classes. Further analysis
using a Pearson correlation revealed a statistically
significant correlation between student GPA and
grades in both EL (r = .542, p < .01) and FTF
(.532, p < .01) CS classes.
Were students who withdrew or failed the
EL classes more likely to pass if they repeated
those classes in EL or FTF format? Of the 422
154
EL students who withdrew from their classes,
only 30% repeated those classes in that format.
Those students then passed at a 57% success rate.
Those 70% who took the course in a FTF mode
passed at a similar rate of 55%. However, of the
200 students who failed their EL classes, only
60% of those students were successful in their
second EL classes attempt; of the 71% who took
FTF classes on their next attempt to pass the class,
84% succeeded.
Such archival analysis problematizes the tra-
ditional academic impulse: admit only students
who have higher GPAs to EL courses. Students in
both EL and FTF classes were of similar academic
ability, and withdrawal rates were similar across
GPAs. Taking a second EL classes after failing
it resulted in a higher failure rate, while taking
the course in a FTF mode increased the rate of
success. Indeed, had a policy been designed to
admit only students with high GPAs—a conclu-
sion warranted by the data—then 39.6% (n = 1012)
of the 2,554 students in EL classes would have
been denied enrollment; had the cut off GPA been
increased to 3.0, then 57.2% of the students (n =
1460) would have been denied enrollment. Such
a policy appears especially problematic when we
recognize that the very best students—those with
GPAs of 3.5 to 4.0 (n = 2395)—withdrew from
FTF courses at a noticeably lower rate (2.9%) than
those enrolled in EL classes (7.6%). Clearly, more
large scale studies need to be undertaken regard-
ing the potential uniqueness of withdrawal rates,
occurring regardless of student preparation and
GPA, to the discipline of Computer Science.
Analysis of Undergraduates Enrolled
in Humanities Courses
A concurrent study extracted the records of all stu-
dents (n = 384) enrolled in EL humanities courses
offered as electives to all undergraduates to fulfill
humanities general university requirement, from
1994 to 2001. Developed under funding from the
Alfred P. Sloan foundation—a web-based consor-
Validation of E-Learning Courses in Computer Science and Humanities
tium that “encourages the collaborative sharing
of knowledge and effective practices to improve
online education in learning effectiveness, ac-
cess, affordability for learners and providers, and
student and faculty satisfaction” (SloanC, 2008)
—two world literature courses were of special
interest. Specifically developed to be offered in
an EL environment, the courses feature lectures
by the humanities faculty distributed to students
(originally on VHS tape; currently streamed via
the Internet) and asynchronous conferences in
a WebCT format, with full use of well-articu-
lated assignments leading to group projects and
researched essays. These courses had reached
their maturity in 2001 and 2002, and the 115 en-
rolled students provided a look at EL at its most
advanced levels of delivery and student-centered-
ness. Unlike the majority of computing classes,
offering content that is quantitative in nature and
empirical in origin (making possible widespread
use of multiple choice exams and other binary-
optioned assessment measures), all humanities
classes anticipated highly subjective responses to
complex readings, a process requiring significant
peer-peer and peer-instructor interaction. The
two world literature classes offered heightened
interaction with students.
While the junior-level students were far re-
moved from any relevance to be gained from SAT
comparisons (and many lacked SAT scores due to
transfer status from two-year colleges), archival
analysis demonstrated that the average GPA was
similar between all the EL humanities courses (M
= 2.85) and the two EL world literature classes (M
= 2.81). Hence, the academic preparation appears
to be nearly identical across groups.
Nevertheless, as was the case in the CS analy-
sis, retention complexities are apparent. The EL
world literature classes began with a total of 115
students. Within the first two weeks of the class, 52
students withdrew. As the deadline for withdrawal
approached, 26 students more students withdrew.
By the end of the class, only 63 students were
awarded final grades. Thus, there was a 42.5% loss
of students in a class taught by those instructors
most prepared to work within an EL environment.
Ironically, the featured class, incorporating those
techniques and values most central to asynchro-
nous learning, produced a higher withdrawal rate
than had the CS courses.
Taken together, the large CS archival study and
the smaller world literature study are evocative;
indeed, it may be useful to think of the CS study as
the larger population that is representative of EL,
with the literature classes part of that larger world.
The similarities between student performances in
the two worlds are noteworthy. There appears to
be little difference in the academic preparedness
of those who elect to take EL classes. Neverthe-
less,, even with similar academic preparation,
there are striking differences in the performance
of the EL students. In the EL classes, regardless
of disciplines, the perseverance rates are unac-
ceptably low. In order to understand this outcome,
a student survey was developed, with questions
practical and applied in nature, to allow further
inquiry (DeTure, 2004).
Survey Study: One Disciplii
To learn more about the process of student reac-
tion to EL instruction, a survey was designed and
administered to students enrolled in humanities
classes over a two-semester period. Humanities
classes were selected for survey because they are,
at the present writing, designed to follow a best
practices model of EL course delivery (Friedman,
Elliot, & Haggerty, in press).
Specifically, the instructional model currently
employed is best described as a socio-technical
system. In terms of education, Socio-Technical
Systems (STSs) are computer technologies that
enable social interaction of on-line learning,
whether conversation (email), group discus-
sion (chat), or group writing (wiki). STSs allow
social networking, as well as collaborative idea
generation and the sharing of knowledge through
155
 Validation of E-Learning Courses in Computer Science and Humanities
academic journals (Whitworth, 2006; Whitworth
& Friedman, 2008). As well, the humanities
classes were selected because they are all writ-
ing intensive in nature. Students participate in
discussions, draft, and submit writing in which
referential (comprising scientific, informative,
and exploratory discourse) and persuasive writ-
ing (comprising the systematic application of
logical models) are both required (Kinneavy,
1971). Furthermore, the courses are informed
by traditional and current directions in writing
theory (Flower, 1994; MacArthur, Graham, and
Fitzgerald, 2006), the ways that writing shapes
and is shaped by cognition (Bazerman, 2008), and
the state-of-the-art methods by which writing is
assessed (Elliot, Briller, & Joshi, 2007).
As Langdon Winner (1980) long ago correctly
argued, artifacts have politics. While general
principles such as the Sloan Consortium’s cat-
egories for success in on-line learning (Lorenzo
& Moore, 2002) were in mind as the system was
emerging, we sought to know the specifics of
learning effectiveness, student satisfaction, and
access. A model of student learning was there-
fore designed in the survey around the variables
of student demographics, learning style, general
expectations, and interaction.
During the summer of 2006, Survey 1 (n =
108) yielded a 41.53% return, Survey 2, (n = 89)
yielded a 34.23% return, and Survey 3 (adminis-
tered after the withdrawal date during the closing
days of the class, n = 62) yielded a 23.85% return.
Surveys were given in classes with the following
titles: The Pre-Modern World, the Making of the
Modern World, World Literature I (2 sections),
World Literature II, Writing about Science, Tech-
nology, and Society (two sections), and Esthetics
and Modern Technology. During the fall of 2006,
Survey 1 (n = 68) yielded a 50.75% return, Survey
2 (n = 58) yielded a 43.28% return, and Survey 3
(administered after the withdrawal date during
the closing days of the class, n = 58) yielded a
52.25% return. Surveys were given in classes
with the following titles: Technical Writing (two
156
sections), Literature and Medicine (two sections),
and Engineering Ethics. Each instructor in each
class had previously taught in an E-Learning
environment and had volunteered to teach these
classes, electives for all students that satisfy the
general university requirements.
Demographics
Nineteen questions on Survey 1 in our study were
designed to obtain information about our students.
While questions of gender, age, and ethnicity were
standard, questions were also asked about the best
language for writing and hours worked per week.
To determine if the students had the technology
at hand for these digitally intensive E-Learning
classes, questions were included regarding the
availability of high-speed connection and com-
puters. Information regarding experiences with
previous on-line courses was also requested.
Broadly viewed as one of the nation’s most
diverse campuses, the NJIT undergraduate stu-
dents enrollment composition for 2005—the time
after which all of the students in the survey were
enrolled—reveals that 34.8% of the students are
white, non-Hispanic, 20.2% are Asian/Pacific
Islander, 31.2% are Hispanic, and 10.8% are Black,
non-Hispanic. (14.8% declined to state their
ethnicity, and 5.9% identified themselves as non-
citizens.) 20 % of the 5, 360 undergraduates were
women, while 80% of undergraduates are male.
During the summer, 84.3% (n = 91) of the
students reported that their best language for
writing was English (χ2 (1, N = 108) = 50.704, p =
.01.) as opposed to another language. When asked
about the hours they were working either on or off
campus, 47.2% (n = 51) reported that they were
working 35 or more hours per week, while only
13.9% (n = 15) reported that they were working
less than 5 hours per week. When asked if they
used a DSL or cable connection when engaged in
online learning, 94.4% (n = 102) reported that they
did (χ2 (1, N = 108) = 85.333, p = .01.); asked if a
computer was reserved primarily or exclusively
Validation of E-Learning Courses in Computer Science and Humanities
for their use, 89.9% (n = 97) replied that there
was (χ2 (1, N = 108) = 156.167, p = .01.). Asked if
this was their first on-line learning class, 39.8%
of the students replied that it was, while 32.4%
(n = 35) replied that they had enrolled in two or
fewer classes; only 27.8% (n = 30) of the students
were experienced on-line learners, having taken
three or more such classes.
During the fall, 77.9% of the students (n = 53)
reported that their best language for writing was
English (χ2 (1, N = 68) = 21.235, p = .01.). When
asked about the hours they worked, 27.9% (n = 19)
reported that they were working 35 or more hours
per week, while 27.9% (n = 19) reported that they
were working less than 5 hours per week. When
asked if they used high-speed connections, 97.1%
(n = 66) replied that they did (χ2 (1, N = 68) =
60.235, p = .01.); and 97.1% (n = 66) replied that
they had a computer reserved for their use (χ2 2(1,
N = 68) = 60.235, p = .01.). Twenty five percent (n
= 17) of the students replied that this was their first
experience with on-line learning, while 33.8% (n
= 23) had enrolled in two or fewer classes. Forty
one percent (n = 28) of the students reported that
they had taken three or more on-line classes.
Among the most striking results associated
with the demographic portion of the survey were
the perseverance rates revealed by the WebCT’s
“Quizzes/Surveys” tool. The tool captures each
student enrolled in the class at the time the survey
was taken and maintains the students’ names in
the database. Hence, the tool records students who
enroll yet withdraw before the registrar’s deadline.
These students may be termed “shoppers,” as they
take up seats in the E-Learning classes beyond
the deadline to drop or add a class, preventing
other students from enrolling.
During the summer semester 260 names
appeared for the first survey; by the end of the
semester, only 176 of these names appeared on the
roster. Thus, 84 students—47.72 % of the avail-
able seats—were taken by those whose names
never appeared on the final roster. During the fall
semester, 134 names appeared on the first survey;
by the end of the semester, 111 names appeared on
the final roster—a 17.16% loss of available seats
due to the presence of shoppers. With 47% of
students working nearly full-time in the summer
and 28% working that same amount in the fall, it
is easy to see how the convenience of asynchro-
nous access can be confused with ease of course
content (Williams & Hellman, 2004).
Learning Style
To provide information about learning styles,
the survey included nine questions that asked
students to select between viewing the instructor
as the class expert or the class facilitator, along
with questions on motivation and preferences in
deadlines. Analysis of survey 3, the summative
summer set of responses, revealed that 62.9%
of the responding students felt that they wanted
the instructor to serve as an expert, rather than a
facilitator, in class. That response lessened during
the longer fall term, yet 58.6% remained steadfast
in their views that the instructor should serve
as expert, not facilitator. Regarding motivation,
however, students did not feel that the instruc-
tor was the greatest motivator in class. In the
summer 61% said that their own self-motivation
was most important, followed by the instructor
(32.3%), with fellow classmates placing a distant
third (6.5%). A similar set of responses was given
in the fall: self-motivation (67.2%), the instructor
(29.3%), and classmates (3.4%). Overwhelmingly,
students preferred hard deadlines over no deadline
for work to be submitted. The preference for hard
deadlines in the summer (79%) was reflected again
in the fall survey (70.7%).
General Expectations
While the survey included seventeen questions
asking students to identify the degree of difficulty
of the class, the amount of time, and the degree
of engagement required for success, the survey
also asked if the students found the writing de-
157
 Validation of E-Learning Courses in Computer Science and Humanities
mands of E-Learning courses different than those
encountered in face-to-face classes.
During the summer, 91.9% of the students
reported that they felt that the EL class was either
the same as, or harder than, a FTF class, and in
the fall 82.6% of the students reported the same
level of difficulty. In the summer, 92.9% of the
students reported that the class required either the
same or more time than an FTF class, and 86.3%
of the fall students reported the same level of time
required; indeed, in the spring, 64.4% reported
that more time was needed, and in the fall 50%
reported the same. Again, the survey proved
sensitive to the time of class; in the summer,
54.4% of the students reported that the course
materials needed to be engaged four or five days
a week, while in the fall, 50% of the students
reported that the materials needed to be engaged
two or three times a week. When that engage-
ment occurred during the summer, 37.1% of the
students reported that the engagement required
two to three hours of work, and 61.3% reported
that more than 3 hours were required; in the fall,
25.9% of the students reported that between one
and two hours were required, 39.9% reported
that between two to three hours were required,
and 32.8% reported that three or more hours of
engagement were required. During the summer,
82.3% of the students reported that their attention
was either focused or completely focused, while
in the fall 75.9% of the students reported similar
levels of attention. Regarding the level of writing
time, 66.1% of the summer students reported that
the E-Learning class required more writing time,
and 62.1% of the fall students reported that more
writing time was required.
Interaction
Nine questions were designed to gather informa-
tion about the importance of interaction with the
instructor, with classmates, with the digital video
material, and with the on-line reading material.
Two of the questions employed a six-point Likert
158
scale constructed according to a balanced model
in which the labels formed an equal interval con-
tinuum bounded by opposite poles with a neutral
midpoint (Dunham & Davidson, 1991), 65.5%
of the summer students reported that they either
very strongly agreed or strongly agreed that the
instructor was an important part of the class (M =
4.98, SD = .98). In the fall, 51.7% of the students
reached the same conclusion (M = 4.62, SD =
1.14). In the summer, 37.1% of the students agreed
with the statement that collaborative classmate
interaction was an important part of the class (M
= 4.16, SD = 1.27), while in the fall 44.8% of the
students agreed with this statement (M = 4.1, SD =
1.22). Overwhelmingly, students reported that the
digital video material was either easy or foolproof
(89.1% in the summer; 74.1% in the fall). A similar
finding was true of the on-line reading material,
stored both in the course management system
and the university’s digital databases: 95.3% of
summer students reported that access was either
easy or foolproof, and 89.7% of the fall students
reported the same levels of access.
Learning Style and General
Epectations Variable Interactions
The relationships between learning style and
general expectations would, we hoped, reveal
more about the variables of E-Learning in under-
graduate humanities as they were hosted within
our STS framework. Studies on general expec-
tations within an E-Learning environment have
also become central to E-Learning research. As
enrollments have continued to grow—the National
Center for Education Statistics (2003) has reported
that during the 12-month 2000–2001 academic
year, 56 percent (2,320) of all 2-year and 4-year
granting institutions offered distance education
courses, with an enrollment of 3,077,000 during
that period—topics such as instructor prepared-
ness have come to the forefront of research. Shav-
elson & Huang (2003), for example, found that
many universities lack faculty who are properly
Validation of E-Learning Courses in Computer Science and Humanities
trained for creating and delivering online classes,
resulting in lower student success rates.
Elements for success identified by Frey, Faul &
Yankelov (2003) include online posting of grades,
sufficiently detailed and accurate lecture notes,
and well-defined guidelines on how to finish
the assignments, as well as consistent, constant
interaction with the instructor. Such contact and
interaction can take the form of virtual office hours
and phone availability, lecturer clarity and depend-
ability, according to Memon, Shih & Thinger
(2006). In terms of behaviors that foster success,
Ley & Young (2001) find that self-regulation is
an essential component. This research is comple-
mentary to Dabbagh & Kitsantas (2005), who
suggest that instructors spend more time building
up and designing classes that actually promote
self-regulation. In their comparative study, Leners
& Sitzman (2006) find that caring in face-to-face
classrooms is experienced through voice, body
language, facial expressions, and behaviors that
translate, according to Dillon & Stines (1996), to
the E-Learning classroom through means of an
empathetic perspective, timeliness of communi-
cations, and a tone of appreciation. This finding
is in accord with Simonson (1996), who stresses
that the relationship between the teacher and
student may also matter in the student’s success.
For Cereijo (2006), a similarly optimistic attitude
and a strong willpower is required for student
success, which is facilitated by self-discipline
and enthusiasm reinforced by faculty-student
interactions. Generally, as Jamison (2003) has
Table 2. Cronbach’s Alpha correlations for learning style, general expectations, and interaction
scales
Summer 2006 (n = 62) / Fall 2006 (n = 58)
1. Learning Style (variable with 6 questions)
2. General Expectations (variable with 4 questions on E-Learning vs. FTF comparison and 3 questions
on time)
3. General Expectations Subscale 1: E-Learning vs. FTF (variable with 4 questions)
4. General Expectations Subscale 2: Time (variable with 3 questions)
5. Interaction (variable with 4 questions)
found, capability beliefs, a responsive environ-
ment, goal-oriented curriculum, mutual respect,
enthusiasm, and diplomacy are strong factors of
student completion.
In the context of the success factors above, scale
construction proved promising. Survey 3, given
at the end of the classes, was used to construct
the scale. Questions were grouped and combined
to produce the scale and subscale. Questions of
interest, including the ones discussed above, were
combined to produce The Learning Style Scale.
The General Expectations Scale also employed
key questions, and two sub-scales were devised,
focusing on E-Learning versus face-to-face
comparisons and on comparative issues of time.
The Interaction Scale was based, as well, on the
questions of interest discussed above. If relation-
ships were identified among the scales, then a
coherent model could be developed. Overall, as
Table 2 demonstrates, the internal consistencies
of the scales were essentially strong, ranging
from .698 to .787. The relationship of all scales
to subscales, as measured by Cronbach’s alpha,
was .549 during the summer semester and .515
in the fall semester.
In the summer, the General Expectations Scale
and its Subscale 1 demonstrated a high Pearson
product moment correlation and high significance
(summer, r = .759, p < .01, r = .797, p < .01). In
the fall, the General Expectations Scale and both
Subscales 1 and 2 demonstrated similarly high
correlations and high significance (fall, r = .833,
p < .01, r = .858, p < .01, r = .431, p < .01). The
.73 .741
.698 .735
.764 .787
.74 .758
.744 .719
159
 Validation of E-Learning Courses in Computer Science and Humanities
key to these strong within-scale relationships
is to be found in the writing intensive nature of
the E-Learning classes. As the high correlations
between the General Expectations Scale and its
two Subscales demonstrate, it is justified to think
about student writing as being the cohesive vehicle
by which the STS framework is mediated.
However, while the General Expectations
proved cohesive, no similar cohesiveness was
demonstrated among the scales. Only a single
statistically significant positive correlation
(.394, p < .01) was found in the summer semester
between the Interaction Scale and the General
Expectation Scale; no other statistically significant
positive correlations were observed. Of interest
is the statistically significant negative correla-
tion observed in the summer semester between
the Learning Style Scale and the Interaction
Scale (r = -.394, p < .01). A hint as to the cause
of this finding is found in comparison of ques-
tions regarding the significance of the instructor
and the role of motivation. During the summer
semester, only 32% of the students held that the
instructor was the greatest motivation, and 29%
of the fall students answered in the same fashion.
Yet 94% of the students agreed, strongly agreed,
or very strongly agreed that interaction with the
instructor was an important part of the course in
the summer, and 89% of the fall students reported
the same perception. The Learning Style Scale,
thus, appears to be reporting a very different
set of beliefs than the Interaction Scale: while
learning style may be varied—students were
split on the issue of instructor-centered versus
learner-centered approaches—interaction with
the instructor remains the most important factor
in E-Learning classes (Andrysyszn, Cragg, &
Humbert, 2001; Swan et al 2001; Paloff & Pratt,
2003; Wallace, 2005).
Gyles as
Models
Question 21 of Survey 3 asked students to iden-
tify their expected grade in the class on a 9-point
160
scale (A = 9, B+ = 8, B = 7, C+ = 6, C = 5, D = 4,
W = 3, I = 2, F = 1). While we could not match
survey to student, we nevertheless knew who had
completed Survey 3 and, thus, could obtain their
final grade from the class roster. In the summer
semester, these students (n = 62) expected a final
grade with a mean of 7.23 (SD = 2.13). The actual
final grade mean of those students was 7.58 (SD =
1.4). The expected and final grades were correlated
(.286, p < .05), although there was no statistically
different difference (t = 42.36(61), p < .01). In the
fall semester, students who completed survey 3 (n
= 58) expected a final grade with a mean of 7.74
(SD = 12.9). The actual final grade mean of those
students was 7.63 (SD = 1.6). Again, the expected
and final grades were correlated (.286, p < .05),
although there was no statistically different dif-
ference (t = 36.45(58), p < .001). Using question 21
as a proxy performance-based dependent variable
appeared conceptually warranted because of the
traditional use of course grades as a performance
measure and because of the correlations between
expected and final grades.
The regression model, with the Learning
Scale, the General Expectations Scale (and its
two Subscales), and the Interaction Scale serving
as the independent variables, was not statistically
significant (R2 = .12 F(4, 56) = 1.97, p = .112), with
only 12% of the variability of the expected grade
representing the proportion of variance explained
by the interaction of the independent variables.
Similar results were seen in a regression analysis
of the fall semester (R2 = .17, F(5, 51) = 2.092,
p = .1). Clearly, there were factors influencing
the model that were not explained by the usual
criterion variable of the final grade.
A dependent variable constructed was based
on Type II learning styles—those expressing a
preference for structure, cognitive simplicity, and
conformity (Zhang & Sternberg, 2005, 2006).
Using these questions as a constructed variable,
we performed a regression analysis—with the
General Expectations Scale (and its two Sub-
scales), and the Interaction Scale serving as the
Validation of E-Learning Courses in Computer Science and Humanities
independent variables—for the fall semester with
the following results: R2 = .227, F(4, 52) = 3.816,
p < .01. The level of significance was high, with
23% of the variance in the Type II dependent
variable represented by the interaction of the
independent variables.
This finding is among the most important of
this third study. We offer three reasons for the
significance of this finding. First, an outcome vari-
able capturing a structured type of learning style
provides a cohesive model for the STS framework
examined in this study. This is not to say that
grades, the universal proxy for performance, do
not matter, but it appears as if the construct of
structure is more important to the present model’s
coherence (Felder & Brent, 1996). Second, struc-
ture appears to be dependent on factors attributed
to the instructor. The syllabus, class materials,
assignment content, and grading criteria—all
provided by the instructor—are overwhelmingly
found by the students to be adhered to, on hand,
known, and articulated. As other researchers
have found, the role of the instructor is key to
E-Learning (Andrysyszn, Cragg, & Humbert,
2001; Swan et al, 2001; Paloff & Pratt, 2003; Wal-
lace, 2005). Third, in that these are humanities
classes under investigation, it might be assumed
that an enrolled student might prefer reflective
observation and active experimentation. In fact,
however, the students might be said to prefer con-
crete experience and abstract conceptualization.
As students in a technological university, their
learning styles may—or may not—vary according
to subject matter and may be mediated by both
the STS framework (incorporating structure) and
the humanities instructors (fostering divergent
thinking) (Sahin, 2008). Hence, with Zhang &
Sternberg (2005) we believe that an orientation
toward learning styles should be based not on
rigid categories but on individual differences as
they are found in context. In sum: learning styles
matter and must be considered as they emerge
within contexts in which structure, provided by
the instructor, rests at the center of the E-Learn-
ing environment.
Information Literacy :
A Laent Variaia
Borsboom, Mellenbergh, & van Heerden (2003)
correctly argue that a “realist account of the latent
variable is required to maintain a consistent con-
nection between the formal and empirical con-
cept” of the third variable (p. 204). Conceptually
following these researchers, we seek to answer
the relevant research question: Are there latent
variable processes that generate the behaviors we
are observing in our E-Learning studies? As the
above analyses demonstrate, these behaviors may
be summarized as follows: a very diverse group
of CS undergraduate students with solid SAT
Reasoning Test scores appear to be withdrawing
from E-Learning courses at unacceptable rates
that increase regardless of high or low GPAs.
When sub-groups are examined in writing inten-
sive E-Learning courses, these withdrawal rates
increase. These students report no difficulty with
their English language proficiently, nor do they
report difficulties with the technological aspects
of course management systems. While they be-
lieve that their own motivation is important, they
also report, in seeming contradiction, that the
instructor’s presence in very important. When
outcomes models are developed, course grade
does not appear to be as important as learning
styles. Yet, although the new media’s intersection
with writing is frequently described as part of
the process of electracy (Ulmer, 2003)—a term
describing rhetorical stances of chora, approba-
tion, juxtaposition, commutation, nonlinearity,
and imagery (Rice, 2007)—the student learning
styles suggest they are anelectrate in their desire
for structure, cognitive simplicity, and conformity.
Are there individual processes, we must then ask,
that exist as third variables involved in E-Learn-
ing that, if identified, will help us to frame the
often contradictory relationships we are observ-
ing? Is there, as Sanford, Lajbcygier, and Spratt
have proposed in this book (Chapter 8), a priori
background knowledge that students bring, or do
not bring, to the E-Learning tasks at hand?
161
 Validation of E-Learning Courses in Computer Science and Humanities
In the fall of 2004, librarians and faculty at
NJIT began a formal investigation of the infor-
mation literacy skills of undergraduate students.
Working with specialists in research and infor-
mation literacy at the university’s Robert Van
Houten Library, instructors in the department
of humanities worked to design an information
literacy model based on standards derived from
the Association of College & Research Libraries
(ACRL). In that the faculty had been assessing
the writing skills of students enrolled in general
undergraduate requirements (GUR) in humani-
ties since 1996, a traditional portfolio assessment
system had emerged that allowed reliable and valid
programmatic information to be gained about
student writing (Elliot, Briller, & Joshi, 2007).
A new portfolio assessment system launched
in spring 2005—termed the NJIT Information
Literacy Scale (ILS)—shifted the assessment
focus from writing to information literacy as-
sessment (Scharf, Elliot, Huey, Briller, & Joshi,
2007). The NJIT ILS was designed to investigate
student ability to identify, find, understand,
and use information in drafting, revising, and
finalizing researched, persuasive writing. While
allowing similarly strong validity evidence to
be warranted as the original portfolio system,
the information literacy scores were lower than
anticipated, documenting marginal to unaccept-
able levels of student information literacy skills.
Instructional and library faculty were interested
in learning more about the information literacy
skills of their students.
In fall 2005, NJIT and the Educational Testing
Service (ETS) undertook a collaborative research
agreement to investigate more fully—by means
of multiple approaches—the variables of infor-
mation literacy as they were evidenced within
student performance at a public comprehensive
technological university (Katz, et al., 2008). The
collaboration would bring together the portfo-
lio-based assessment approach of NJIT with the
performance-based, automatically scored iSkills
assessment, which was designed to measure
162
information literacy skills as they appear in
technological environments (Katz, 2007b). The in-
formation and communication technology skills,
complementary to the NJIT Literacy Scales, were
designed assess student ability to appropriately use
digital technology, communication tools, and/or
networks to solve information problems in order
to function in an information society. This student
ability includes employing information as a tool to
research, organize, and communicate information
and having a fundamental understanding of the
ethical/legal issues surrounding accessing and
using information. Key to the ETS assessment is
its Internet-delivered, performance-based assess-
ment basis. Assessment administration takes ap-
proximately 75 minutes, divided into two sections
lasting 35 and 40 minutes, respectively. During
this time, students respond to 15 interactive tasks
comprising a real-world scenario, such as a class
or work assignment, that frames the information
task. Students solve the tasks in the context of a
simulation (for example, e-mail, Web browser,
or library database) having the look and feel of
typical applications.
Thus, the study embodied two elements central
to E-Learning within our specific institutional
site: the use of writing as a process to deepen
knowledge, and the use of computer-mediated
environments to deliver content information. If
students could demonstrate proficiency in these
information literacy measures, we would be able
to rule out the presence of a latent variable in
the E-Learning process; that is, students were
able to demonstrate, in FTF settings (where tra-
ditional instructional means were in place) that
they possessed the skills necessary to retrieve
and integrate information from divergent sources
into their work. As well, if students were able to
demonstrate, in the iSkills simulation, that they
could research, organize, and communicate such
information in a timed setting, then the essential
skills necessary for success in E-Learning could
be demonstrated.
Validation of E-Learning Courses in Computer Science and Humanities

While first-year students were assessed, these
students do not take asynchronously offered
E-Learning courses. For the present purposes,
information gained from the sophomores, juniors,
and seniors is most relevant. A simple random
sample of upper-division students was created
across each section of two representative humani-
ties writing classes: cultural history and technical
writing. These students, along with students from
the senior seminar who were selected as described
below, were identified for portfolio submission.
The senior seminar students consisted of a census
of all whose transcripts revealed that they had
never taken any class outside of NJIT; hence, these
students, while small in number, represented a
meaningful population of NJIT students. Overall,
students were found in cultural history (n = 95),
as well as in technical writing (n = 48) and the
senior seminars (n = 33). The sample resulting
from the sampling plan closely matched the NJIT
student population. Students were tested in a
proctored computer lab on the iSkills assessment
in late March 2006 and in May 2006 portfolios
of targeted students were evaluated according to
the NJIT ILS. Table 3 presents the results of the
ETS / NJIT collaborative research.
The scores for students on the iSkills Advanced
Test enrolled in cultural history (M = 548.5, SD
= 36.9), technical writing (M = 547.2, SD = 39.5),
and the senior seminars (M = 568.3, SD = 28.2)
were, at best, average. With a score range from
400 to 700, the students in the study performed at
a level similar to that reported on a national level.
Of 3,571 students tested on the Advanced iSkills
test between January 2006 and May 2007, a score
of 555 (SD = 32) was the reported as the average.
That is, students were only able to answer 55% of
the questions correctly. Nationally, across several
tasks, few test takers could adapt material for a new
audience. In a web search task, only 40% entered
multiple search terms to narrow the results. When
asked to organize a large amount of information
efficiently, more than half the students failed to
sort the information to clarify related material.
When searching a large database, only 50% of test
takers used a strategy that minimized irrelevant
results (Katz, 2007a).
Regarding the NJIT scale, it is clear that much
remains to be done in helping students integrate
sources meaningfully into their essays. While
students in cultural history could achieve basic
documentation forms—almost universally those
advocated by the Modern Language Association
(Gibaldi, 2003)—they had problems demonstrat-
ing that they had provided sources beyond those
noted in the syllabus (evidence of research),
identifying sources germane to their topic (in-
tegration), and achieving an overall sense of a
researched essay (holistic score). In that a score
below 7 is taken at NJIT to be a demonstration
of weak performance, it appeared that only the
citation variable—a mechanistic adherence to

Table 3. Information literacy scores: Advanced level ETS iskills assessment and NJIT information literacy
scale (means and standard deviations)

Cultural History Technical Writing Senior Seminar

(n=95) (n=48) (n=33)
ETS Advanced iSkills Score 548.5 (36.9) 547.2 (39.5) 568.3 (28.2)
NJIT Citation 7.8 (2.8) 5.3 (2.7) 8.3 (2.1)
NJIT Evidence of Research 6.9 (2.8) 5.8 (2.49) 6.8 (2.5)
NJIT Appropriateness 6.7 (2.7) n/a 7.2 (2.3)
NJIT Integration 6.5 (2.7) n/a 6.7 (2.3)
NJIT Holistic Score 6.8 (2.4) n/a 7.0 (2.2)

163
 Validation of E-Learning Courses in Computer Science and Humanities
citation format—could be taken as a measure of
competency.
The technical writing students’ abilities to
cite sources and provide evidence of independent
research were profoundly weak. Although only
these two variables were assessed among other
variables important to technical writing teachers,
it is distressing that the lowest performance on the
NJIT scales was identified in this group of students.
Less distressing, however, were the scores of the
senior seminar students. Scharf, et al. (2007), in-
vestigating a similar group of students at NJIT (n
= 100), found ILS scores below the cut score of 7
on each of variables that make up the component
ILS variable used in the current study. One year
later, the citation, appropriateness, and holistic
scores had met the competency score of 7.
Surely, the skills addressed by both the ETS
and NJIT measures are integral to successful
E-Learning. If, as the ETS research confirms,
the vast majority of students are performing
poorly on the iSkills measure, then we must
wonder about the impact of this lack of skill on
E-Learning students. As well, if students in writ-
ing-intensive classes are not able to cite sources,
extend searches beyond a syllabus, identify ap-
propriate sources, and integrate information into
their work—processes concurrent with drafting,
revising, and finalizing researched, persuasive
writing—then these students will surely either
fail in E-Learning classes requiring such skills,
or they will withdraw at high levels. It may indeed
be the case that the essential E-Learning skills
are not there in the first place.
Di
As Markham and Hurst have reminded us in
Chapter I, new conceptualizations of the process
of gathering validity evidence are now emerging
(Kane, 2006; Dialogue on Validity, 2007). The
present shift from the term “validity” (a noun
suggesting a concretized fact) to “validate” (a
164
transitive verb suggesting recurrent process) has
been epitomized in Michael T. Kane’s conclu-
sion about the nature of evidence in assessment
contexts: “Validation has a contingent charac-
ter; the evidence required to justify a proposed
interpretation or use depends on the proposed
interpretation or use” (2006, p. 60). With validity
now viewed as a series of integrated judgments
(Messick, 1989; AERA, APA, & NCME, 1999;
Brennan, 2006), context-specific arguments
(Mislevy & Brennan, 2006; Mislevy, 2007) and
procedurally-dependent observations resembling
science itself (Embretson, 2007), a new world of
evaluation is before us.
The three kinds of studies performed in this
study—archival, process, and outcomes—serve
to help us better understand the complexities in-
volved in the assessment of E-Learning. Indeed,
any one of the methods may not, in itself, prove
sufficient: trend data, student response, and stu-
dent performance (Lane & Stone, 2006) should
each be taken into account when effectiveness
studies are to be designed, conducted, and reported
(Watkins & Corry, 2005).
As our three studies demonstrate, there are
complex variables, often contradictory, in play
within a specific institutional setting. It is, there-
fore, difficult indeed to come to conclusions about
what must be done regarding the success of any
system. The kinds of tasks required for general
E-Learning success at NJIT may be, for instance,
similar to those found in the computer-mediated
iSkills assessment. Yet those skills may not be
sufficient to ensure success in specific classes,
such as world literature, that are writing intensive.
If students in FTF classes can demonstrate, at the
very end of a class, only marginal skills, then how
can similar students be expected to demonstrate
those skills in the already complex asynchronous
environment?
To end with a case in point regarding the
complexities of assessing E-Learning, consider-
ation of the presence of cognitive complexity is
in order. Writing assignments are socio-cogni-
Validation of E-Learning Courses in Computer Science and Humanities
tive activities that are congruent with many of
the best practice values of E-Learning. As they
write, students participate in discussions, draft,
and submit assignments in which referential (com-
prising scientific, informative, and exploratory
discourse) and persuasive writing (comprising
the systematic application of logical models) are
both required (Kinneavy, 1971). At their best,
writing tasks are informed by traditional and
current directions in writing theory (MacArthur,
Graham, & Fitzgerald, 2006), demonstrate both
the ways that writing shapes and is shaped by
cognition (Bazerman, 2008), and the state-of-the-
art methods by which writing is assessed (Elliot,
Briller & Joshi, 2007). As well, writing serves
to lessen transactional distance. As proposed by
Moore (1973; Moore, 2007), activities yielding
high transactional distance, such as reading a
textbook, afford little dialogue and less com-
munication; activities yielding low transactional
distance, such as independent studies, yield highly
individualized communication. In the writing-
enhanced E-Learning environment, all transac-
tions are writing-intensive, thus allowing for the
desired low transactional distance. Hence, student
writing is seen as the vehicle by which the STS
framework is mediated (Bolter & Grusin, 2000)
into an individualistic, student-centered context
allowing low transactional distance—and, thus,
greater communication. In that student writing
plays such a central role in class design, the con-
cept of lessening transactional distance through
writing tasks should be of interest to all E-Learn-
ing instructors.
Specifically, a student in a writing-intensive
E-Learning class would likely be involved in the
following potential case: planning a researched
essay on, for example, the measurement of hu-
man intelligence, searching databases such as
Academic Search Premiere and JSTOR to find
the very best articles, reading and synthesizing
those identified sources, documenting them in
a standardized format, drafting and submit-
ting the work for review, and then making final
revisions based on that review. If, however, the
recent results of the iSkills assessment are taken
into account—less than half the tested students
could enter multiple search terms to narrow the
results, more than half the students failed to sort
the information to clarify related material, and
only half of the test takers used a strategy that
minimized irrelevant results—then the essential
skills are lacking in the first place. Never mind
about grammar and mechanics, about essay orga-
nization and writer’s aim, about documentation
and editing; the assignment breaks down at the
initial point of researched writing: the database
search itself.
Future Trends
Although the three studies presented in this
chapter offer evidence obtained from recent
experiences with E-Learning at NJIT, it is clear
to us that the resulting implications affect all
contemporary learning, regardless of its form.
The rapid integration of course management
tools and asynchronous instruction, originally
deemed exclusive to EL, into FTF teaching has
blurred the distinction between the two modes
of learning. In fact, the amalgamation of the two
communities is becoming the norm. Furthermore,
ubiquitous technologies, both wearable and wire-
less, promptly embraced by this generation of
learners, are assertively shifting campus interac-
tion from exclusively physical locations to their
digital alternatives. It is not difficult to imagine
the emergent applications of this evolving “smart
campus” to education (Jones & Grandhi, 2005).
Thus, protesting EL and issuing calls for limit-
ing its enrollment, which were unsuccessful in
the past, are even more unrealistic now. Instead,
technological and pedagogical policies need
to converge in order to enhance the quality of
student learning and ensure that the literacy and
skills “needed to fully participate in an increas-
ingly competitive work environment” are learned
165
 Validation of E-Learning Courses in Computer Science and Humanities
(Kirsch, Braun, Yamamoto, & Sum, 2007),
whether in physical or virtual settings.
C Education, 16, 227-244.
Regardless of SAT score or GPA, students simply
lack the fundamental skills to deal with the new
media. Alone, this fact helps us come to terms with
the stunning withdrawal rates and the conflicted
survey information presented in this chapter.
Without the critical skills of identifying, find-
ing, understanding, and using information—as
well as rapidly and effectively researching and
organizing information responsibly—the student
will certainly be lost upon point of entry – in class
and in the information workplace. Our attempts to
validate our processes may thus achieve method-
ological soundness but fail to provide directions
to deal with this emerging context. As our notions
of validity are presently being re-imagined to ac-
knowledge process, context, and contingency, it is
perhaps helpful to imagine E-Learning itself as an
emerging construct, one that mirrors the realities
of students’ environments far more than it does
the protocols of brick and mortar universities. In
this view, there is much yet to learn about this
critical socio-technical system and the contexts
in which it operates.
REFERENCES
American Educational Research Association,
American Psychological Association, & National
Council on Measurement in Education. (1999).
Standards for educational and psychological
testing. Washington, DC: American Psychologi-
cal Association.
Andrysyszyn, M. A., Cragg, B., & Humbert, J.
(2001). Nurse practitioner preferences for distance
education methods related to learning style, course
content and achievement. Journal of Nursing
Education, 40, 163- 170.
166
Aragon, S. R., Johnson, S. D., & Shaik, N. (2002).
The influence of learning style preferences on
student success in online versus face-to-face
environments. American Journal of Distance
Bazerman, C. (Ed.). (2008). Handbook of research
on writing: History, society, school, individual,
text. New York: Earlbaum.
Bolter, J. D., & Grusin, R. (2000). Remediation:
Understanding the new media. Cambridge: MA,
MIT Press.
Borsboom, D., Mellenbergh, G. J., & van Heerden,
J. (2003). The theoretical status of latent variables.
Psychological Review, 110, 203-219.
Brennan, R. L. (2006). Perspectives on the evolu-
tion and future of educational measurement. In R.
L. Brennan (Ed.). Educational measurement (4th
edition, pp. 1-16). Westport, CT: Praeger.
College Board. (2008a). SAT Reasoning Test.
Retrieved October 4, 2008, from http://www.col-
legeboard.com/student/testing/sat/about/SATI.
html
College Board. (2008b). Score range. Retrieved
October 4, 2008, from http://www.collegeboard.
com/student/testing/sat/scores/understanding/
scorerange.html.
Cereijo, M. V. P. (2006). Attitude as predictor of
success in online training. International Journal
on E-Learning, 5, 623-639.
Dabbagh, N., & Kitsantas, A. (2005). The role of
web-based pedagogical tools in supporting student
self-regulation in distributed learning environ-
ments. Instructional Science, 25, 24-37.
DeTure, M. (2004). Cognitive style and self-ef-
ficacy: Predicting student success in online dis-
tance education. American Journal of Distance
Education, 18, 21-38.
Dialogue on Validity [Special issue]. 2007. Edu-
cational Researcher, 36(8).
Validation of E-Learning Courses in Computer Science and Humanities
Dillon, R. S., & Stines, R. W. (1996). A phe-
nomenological study of faculty-student caring
interactions. Journal of Nursing Education, 35,
113-118.
Dunham, T. C., & Davidson, M. L. (1991). Effect
of scale anchors on student rating of instructors.
Applied Measurement in Education, 4, 23-35.
Elliot, N., Briller, V., & Joshi, K. (2007). Portfo-
lio assessment: Quantification and community.
Journal of Writing Assessment, 3, 5-29.
Elliot, N., Friedman, R. S., & Briller, V. (2005,
June-July). Irony and asychronicity: Interpret-
ing withdrawal rates in E-Learning courses.
Paper presented at the meeting of the Ed-Media
World Conference on Educational Multimedia,
Hypermedia, and Telecommunications, Montreal,
Canada.
Elwert, B., & Hitch, L. (Eds.). (2002). Motivating
and retaining adult learners online. Essex Junc-
tion, Vt.: GetEducated.com.
Embretson, S. E. (2007). Construct validity: A
universal validity system or just another test
evaluation procedure? Educational Researcher,
36, 449-455.
Felder, R. M., & Brent, R. (1996). Navigating
the bumpy road to student-centered instruction.
College Teaching, 44, 43-47.
Flower, L. (1994). The construction of negotiated
meaning: A social cognitive theory of writing.
Carbondale: Southern Illinois University Press.
Foster, L., Bower, B. L., & Watson, L. W. (Eds.).
(2002). Western cooperative for educational tele-
communications. Best practices for electronically
offered degree and certificate programs. ASHE
Reader: Distance Education: Teaching and Learn-
ing in Higher Education. Boston: Pearson.
Friedman R., Elliot, N, & Haggerty, B. (in press).
E-Learning in undergraduate humanities classes:
Unpacking the variables. International Journal
on E-Learning.
Frey, A., Faul, A., & Yankelov, P. (2003). Student
strategies of web-assisted teaching strategies.
Journal of Social Work Education, 39, 443-457.
Gibaldi, J. (2003). MLA handbook for writers of
research papers. (6th ed.). New York: Modern
Language Association.
Hiltz, R. S., & Turoff, M. (1993). Network nation
- revised edition: Human communication via
computer. Cambridge, MA: MIT Press.
Institute for Higher Education Policy. (1999). A
review of contemporary research on the effec-
tiveness of distance learning in higher educa-
tion. Washington, DC: The Institute for Higher
Education Research.
Jamison, T. M. (2003). Ebb from the web: Using
motivational systems theory to predict student
completion of asynchronous web-based distance
education courses. Unpublished doctoral disserta-
tion, George Mason University.
Jones, Q., & Grandhi, S. (2005). P3 systems:
putting the place back into social networks. IEEE
Internet Computing, 9, 38-46.
Katz, I. R. (2007a, June). Digital natives but not
information fluent: Results from ETS’s iSkills
assessment. Presented at the Alliance for a Media
Literate American Research Summit, St. Louis,
MO.
Katz, I. R. (2007b). Testing information literacy
in digital environments: ETS’s iSkills™ assess-
ment. Information Technology and Libraries,
26(3), 3–12.
Katz, I. R., Elliot, N., Attali, Y., Scharf, D., Pow-
ers, D., Huey, H., Joshi, K., & Briller, V. (2008).
The assessment of information literacy: A case
study (ETS Research Rpt. No 08-33.) Princeton,
NJ: Educational Testing Service.
167
 Validation of E-Learning Courses in Computer Science and Humanities
Kane, M. T. (2006) Validation. In R. L. Brennan
(Ed.), Educational measurement (4th edition,
pp. 17-64). Westport, CT: American Council on
Education/Praeger.
Kinneavy, J. L. (1971). A theory of discourse.
New York: Norton.
Kirsch, I., Braun, H., Yamamoto, K., & Sum, A.
(2007). America’s perfect storm: Three forces
changing our nation’s future. New Jersey: Edu-
cational Testing Service.
Lane, S., & Stone, C. A. (2006). Performance
assessment. In R. L. Brennan (Ed.), Educational
measurement (4th edition, pp. 387-431). Westport,
CT: American Council on Education/Praeger.
Leners, D. W., & Sitzman, K. (2006). Graduate
student perceptions: Feeling the passion of car-
ing online. Nursing Education Perspectives, 27,
315-319.
Ley, K., & Young, D. B. (2001). Instructional prin-
ciples for self-regulation. Educational Technology
Research and Development, 49, 93–103.
Lorenzo, J., & Moore, J. (2002). The Sloan Con-
sortium report to the nation: Five pillars of quality
online education. Retrieved from http://www.
sloan-c.org/effectivepractices/pillarreport1.pdf
MacArthur, C. A., Graham, S., & Fitzgerald, J.
(2006). Handbook of writing research. New York:
Guilford Press.
Memon, A., Shih, L., & Thinger, B. (2006, June).
Development and delivery of nuclear engineering
online courses: The Excelsior College experi-
ence. Paper presented at the meeting of the ASEE
Annual Conference & Exposition: Excellence in
Education, Chicago, IL.
Messick, S. (1989). Validity. In R. L. Linn (Ed.).
Educational measurement (3rd edition, pp. 13-
103). New York: American Council on Educa-
tion/Macmillan.
168
Millwood, R., & Terrell, I. (2005). Overview: New
technology, learning and assessment in higher
education. Innovations in Education & Teaching
International, 42, 195-204.
Mislevy, R. J., & Brennan, R. L. (2006).Cognitive
psychology and educational assessment. In R. L.
Brennan (Ed.). Educational measurement (4th
edition, pp. 257-305). Westport, CT: Praeger.
Mislevy, R. J. (2007). Validity by design. Educa-
tional Researcher 36, 463-469.
Mock, K. (2003). The development of a CSO
course for distance delivery. The Journal of Com-
puting Sciences in Colleges, 19(2), 30-38.
Moore, M. G. (1973). Toward a theory of inde-
pendent teaching and learning. The Journal of
Higher Education, 44, 661-679.
Moore, M. G. (2007). The theory of transactional
distance. In M. G. Moore (Ed.), Handbook of
Distance Education (2nd edition, pp. 89-105).
New Jersey: Earlbaum.
National Center for Education Statistics (2003).
Distance learning in higher education institu-
tions: 2000-2001. (Rep. No. NCES 2003-017).
Washington, DC: National Center for Education
Statistics.
Paloff, R., & Pratt, K. (2003). The virtual student:
A profile and guide to working with online learn-
ers. San Francisco: Jossey-Bass.
Rice, J. (2007). The rhetoric of cool: Composition
studies and the new media. Carbondale: Southern
Illinois University Press.
Sahin, S. (2008). The relationship between student
characteristics, including learning styles, and
their perceptions and satisfactions in web-based
courses in higher education. Turkish Journal of
Distance Education, 9, 123-138.
Scharf, D., Elliot, N., Huey, H. A., Briller, V., &
Joshi, K. (2007). Direct assessment of informa-
Validation of E-Learning Courses in Computer Science and Humanities
tion literacy using writing portfolios. Journal of
Academic Librarianship, 33, 462-477.
Shavelson, R. J., & Huang, L. (2003, January/
February). Responding responsibly to the frenzy
to assess learning in higher education. Change,
11–19.
Simonson, C. L. S. (1996). Teaching caring to
nursing students. Journal of Nursing Education,
35, 100-104.
Sloan-C. (2008). A consortium of institutions and
organizations committed to quality online educa-
tion. Retrieved October 4, 2008, from http://www.
sloan-c.org/
Swan, K., Shea, P., Fredericksen, E., Pickett, F.,
Pelz, W. & Maher, G. (2001). Building Knowledge
Building Communities: Consistency, Contact
and Communication in the Virtual Classroom,
Journal of Educational Computing Research,
23(4/2000), 359 - 383.
Ulmer, G. L. (2003). Internet invention: From
literacy to electracy. New York: Longman.
Wallace, R. M. (2005). Online learning in higher
education: A review of research on interactions
among teachers and students. Education, Com-
munication, and Information, 3, 241-280.
Watkins, R., & Corry, M. (2005). E-Learning
companion: A student’s guide to online success.
Boston: Houghton-Mifflin.
Whitworth, B. (2006). Social-technical systems.
In C. Ghaoui (Ed.), Encyclopedia of Human
Computer Interaction (pp. 533-541). London:
Idea Group Reference.
Whitworth, B., & Friedman, R. (2008). The chal-
lenge of modern academic knowledge exchange.
SIGITE Newsletter, 5(2), 4-10.
Williams, P. E., & Hellman, C. M. (2004). Dif-
ferences in self-regulation for online learning
between first- and second-generation college
students. Research in Higher Education, 45,
71-82.
Winner, L. (1980). Do artifacts have politics?
Daedalus, 101, 121-136.
Zhang, L., & Sternberg, R. J. (2005). A threefold
model of intellectual styles. Educational Psychol-
ogy Review, 17(1), 1-53.
Zhang, L., & Sternberg, R. J. (2006) The nature
of intellectual styles. New Jersey: Erlbaum.
Zimmerman, M. C. (2002). Academic self-regu-
lation explains persistence and attrition in web-
based courses: A grounded theory. Unpublished
doctoral dissertation, Northern Arizona Univer-
sity, Flagstaff.
169