You are on page 1of 7

Volume 2 No.

2 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences

©2010-11 CIS Journal. All rights reserved.

The Effectiveness of Learning Management System (LMS) Case Study

at Open University Malaysia (OUM), Kota Bharu Campus
Khadijah Abdul Rahman, Siti Aswani Mohd Ghazali, Dr Mohd Nasir Ismail
Information Management Faculty, University of Teknologi MARA, Machang Campus, Kelantan,,

Learning Management System (LMS) provides an integrated platform for content, delivery and management of learning as
well as accessibility by a range of users that may include learners, content creators and administrators. Open University
Malaysia, the seventh private university in Malaysia that fully operated as an Open and Distance Learning environment
had successfully implemented the LMS to ease the learning process of their students as well as their administrators.
Although LMS is an important element in OUM, most of the students do not fully utilize the system to support for online
submission of course works and discuss any matters in discussion board or online discussion as well as to self-evaluate
their performance in learning. They tend to do all those things manually such as discuss any subject face to face meeting
with their peers and tutors, they prefer to have paper based tests rather than online quizzes and so on. These factors
contribute a major reason why LMS is not used effectively .T his paper is measuring the effectiveness of Learning
Management System of Open University Malaysia by testing the LMS done by three categories of users which are novice,
knowledge intermittent and expert users along with the distributions of hundred set of questionnaires to randomly selected
users in order to collect data regarding the effectiveness of OUM’s Learning Management System. The findings and result
of this survey and testing is also discussed.

Keywords: LMS, Open and Distance Learning, OUM

I. INTRODUCTION  The distance education and institute train a

sufficient number of teaching staff who are
Maxwell (1995) said that Open learning is qualified in developing distance educations
defined as a student-centered approach to education that programs.
removes all barriers to access while providing a high
degree of learner autonomy. Distance education refers to Considering the importance of all these
a mode of delivering a course of study in which the purposes, and believing that distance education will
majority of communication between teachers and play a more important role in the education system,
students occurs non-contiguously, and the two-way Open University Malaysia (Universiti Terbuka
communication between teacher and student necessary Malaysia) or better known as OU Malaysia was
for the educational process is technologically mediated. established on August 10, 2000 as the seventh private
Distance education may or may not be based on open- university in Malaysia. Although incorporated as a
learning ideals. Distance education programs are private university under the Private Higher Education
becoming more common in traditional higher Institutions Act 1996, the University leverages on the
education institutions. According to Aybay and Oguz quality, prestige and capabilities of its strategic partners (a
(2002), in the last ten years, many universities and consortium of the 11 public universities), OUM adopts the
higher education institutions have started offering motto `University for All' which is consistent with its
distance education courses for their on-campus philosophy on democratization of education. This
students due to the following purposes: philosophy underlies the belief that education should be
made available to all, regardless of time, place, age and
 The university gains experience on online social economic background. Open and Distance Learning
course development. (ODL) mode is specifically targeted at working adults who
 The university gains experience on the need to support of their families as well as manage their
management of online programs and this career while pursuing higher education. Located in the
possibly leads to the establishment of an heart of Kuala Lumpur, the university opens to all
institute or center specialized in distance qualified candidates to further their study in various fields
education method. with the OUM's vision to be a leader and innovator in
 The students taking online courses return open learning. From its first intake of 753 students in
quick feedback on the program. August 2001, it boosts to a total of 25,000 students. There


Volume 2 No. 2 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences

©2010-11 CIS Journal. All rights reserved.

are 16 programs offered, 2 Diplomas, 11 Bachelor degrees e-learning refers to the process of learning or training
and 3 Masters. online, meaning acquiring knowledge via Internet and the
The objective towards accomplishing the study Web. There are three main factors that encourage the
which are: higher education institutions to get involved with e-
To measure the effectiveness of the OUM's Learning learning such as the increasing number of students, the
Management System using four factors below need for life-long learning and the need to prepare
students with knowledge and skills for succeeding in the
 Easy to learn knowledge economy ( Harasim, 1995) as cited in Iahad et
 Error Tolerance al. (2004).
 Speed According to Relan and Gillani (as cited in
 Quality Lewis, 2002) the online learning environment are broadly
interpreted as an instructional delivery method in which
These four factors derived from Shackel’s the Internet is the medium of the delivery. There are two
Usability Model. According to Shackel (1991) usability is ways to delivery online learning such as:
a property of a system or a piece of equipment. The
property is not constant, being relative in relation to users,  Synchronous: communication in which interaction
their training and support, tasks and environments. Thus, between learner-instructor and learner-learner is
the evaluation is context-dependent. The system or piece simultaneous.
of equipment may be usable if it matches the combination  Asynchronous: communication in which
of users, tasks, and environment. For a system to be usable interaction between learner-instructor and learner-
it has to achieve defined levels on the following scales: learner does not take place simultaneously (anytime,
 Effectiveness, meaning the results of
interaction in terms of speed and errors; Open and Distance Learning is a new way of
 Learnability, meaning the relation of learning in Malaysia. It began in 1972 when University
performance to training and frequency of use, i.e. Sains Malaysia introduced open and distance learning.
the novice user's learning time with specified Since today, ten public universities in Malaysia offered
training and retention on the part of casual users; open and distance learning, followed by Open University
 Flexibility, allowing adaptation to tasks and Malaysia (OUM), the seventh private university in
Malaysia that fully based operate on open and distance
environments beyond those first specified;
learning environment. To define Open and Distance
 Attitude, meaning "acceptable levels of human Learning, we must divide the term into two parts. Open
costs in terms of tiredness, discomfort, frustration learning and distance learning, two systems combined
and personal effort". together.
Open learning means that the chance for learning
So, researcher chooses only one characteristic of is opened widely to anyone who want to learn in the
Shackel’s Model which is effectiveness. This effectiveness tertiary stage. The needs for tertiary stage of education
consists of four factors such as easy to learn, error have given way for open learning. According to Harry
tolerance, speed and quality. (1997), the term `Open Learning' stands for general aim of
opening up education and training more widely. Openness
II. PREVIOUS RESEARCH has many dimensions and most projects that describe
themselves as open learning.
According to Porter, L. (1997), on-line learning According to Linda (2001), traditional classroom
may take place in the classroom or workplace, alongside lessons are often teacher directed, cannot be interesting
other activities, or may be undertaken at home or at online and engaging, often limited to textbooks, sometimes
access centers in the community. On-line learning thought of as boring by students, can be found on the web
provides hands-on experience using information but resources are not readily available, and teachers
technologies and makes it easier for learners to access sometimes lack the initiative to experiment with varied
learning and customize study. It may also refer to a range methods of teaching the curriculum. A traditional
of approaches from making resources available classroom might be found in the workspace or an office
electronically, to the creation of rich interactive on-line building as well as at home or at field site. What is
experiences involving synchronous and asynchronous traditional as that there is an established place, where
communication. learners meet with one or more teaches or the material
In Malaysia, the wave of e-learning is blowing they have create and setup to work interactively with
across the entire education system from public and private students and learning occurs. It may contain computer
institutions (Ndubisi, 2004). According to Iahad, N., equipment but it may be picture as an older whiteboard,
Kalaitzakis, E., Dafoulas, G.A and Macaulay, L.A.(2004), classroom full of desks or table with teacher lecturing

Volume 2 No. 2 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences

©2010-11 CIS Journal. All rights reserved.

students face to face. One of the things in a traditional implementation. According to Schar and Krueger
class is that the student can ask questions, make (2000), an LMS must be capable of handling various
comments, interact in verbal discussions, or collaborate delivery modes and automate the cumbersome
with small groups. process of learner enrollment, registration, records,
transcripts, schedules, and reports, and it must
incorporate evaluation, assessment, and testing
capabilities. Learning Management Systems have
been widely adopted by institutions and instructional
designers in order to fulfill certain needs and
requirements in a field of ever increasing demands
for effective, fast and pedagogically correct
education and training. LMS that are in use today are
either commercial products such as WebCT,
Blackboard, Intralearn, open-source projects such as
customized software systems that serve the
instructional purposes of particular organizations
(Avgeriou, Papasalouros, Retalis, Skordalakis 2003).
The users of LMS can be classified into three
 The learners that use the system in order to
participate through distance (in place and/or
time) to the educational process. In fact, the
learners are the focal users of LMS, in the
sense that these systems are being developed
in order to satisfy some of their needs and
resolve their problems.
Figure 1: The comparison between e-learning and  The instructors, being the teachers and their
traditional method. (Adapted from assistants that use the system in order to coach, supervise, assist and evaluate the
students (e.g. notify for important issues on
Figure 1 above shows that the comparison in an electronic notice board, engage in
both traditional and e-learning method. It is can be seen discussions in electronic for a communicate
clearly that the process of learning can be done easily by and exchange personal messages with
internet. Students can interact with their lecturer students, collect, assess and return
interactively 24 hours per day and 7 days per week via e- deliverables, etc. ).
mail and on-line discussion asynchronously, by chatting  The administrators of the system, who
synchronously as well as can confidentially collect undertake the support of all the other users of
programs and respond via LMS, otherwise in traditional the system and safeguard its proper
classroom , the contents of the lectures are manually operational status
driven. Once the class ends the lectures cannot be
archived by students. Effectiveness is measured in The research framework for this research is
terms of speed and error, refers to levels of user based on the Shackel's Model of Usabiliry (1991) as
performance (Lindgaard, 1994; Shackel, 1991). shown in Figure 2. But the researcher study based on
Effectiveness quantitatively measures the ease of one of the characteristic of Shackel's model which is
using the system, either by speed of performance or effectiveness. The model is derived from Shackel's
error rate. Effectiveness means the capability of, or definition of usability as illustrated below. Shackel
success in, achieving a given goal. Contrary to was one of the first Human Computer Interaction
efficiency the focus of effectiveness is the (HCI) researchers to seriously tackle the question on
achievement as such, not the resources spent, so "What is a usable product or application?" According
anything that is effective is not necessarily efficient, to Kissane and Finn (2003), he offered the following
but anything that is efficient also has to be effective. simple definition;
Learning Management System provides an
integrated platform for content, delivery, and "A usable product is one that users find satisfaction
management of learning, as well as accessibility by a for the tasks for which it was designed".
range of users that may include learners, content
creators, and administrators. An LMS acts as the
central component of an enterprise e-learning


Volume 2 No. 2 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences

©2010-11 CIS Journal. All rights reserved.

Figure 2: Shackel's definition of Usability (1991)

Shackel proposes that usability can be seen in terms of
four operational criteria such as effectiveness; learn
ability, flexibility and attitude (Benyon, 1993, Folmer &
Bosch, 2004).
 Effectiveness - performance in accomplishment
of tasks by some percentage of the users within
some proportion of the environments in which Figure 3: The Research Theoretical Framework
the systems will operate and measure by a
characteristic of the interaction such as the time III. COMPONENTS OF WORK
taken to complete the task or the number of errors SITUATION
 Learn ability - degree of learning to accomplish
The product acceptance is determined by the
tasks depends on some specified level of
context in which it is used such as the particular users,
competence whether it is given a specific amount
tasks and environment. In Human Computer Interaction
of training or not. Learning also includes the time
(HCI), the system is the computer system. Meanwhile,
taken to relearn the system if details are
environment includes physical aspects such as appropriate
heating, lighting, equipment layout, operating
 Flexibility - adaptation to variation in tasks and
circumstances as well as psychological aspects such as the
environments which can be accommodated by the
provision of help and training and socio-political features
such as the organizational environment in which the
 Attitude - user satisfaction with the system interaction take place. The relationship between the
whether to continue use the system or enhance Shackel's Model of Usability and the four components is
their use of the system illustrated in Figure 3. The product acceptance is
measured by the usability factors such as effectiveness,
This study involves the testing on the learnability, flexibility and attitude in particular
effectiveness of the Learning Management System. At this environment.
phase, tasks list and questionnaire are used to gather the
data. The task list is translated into scenarios based on the
objectives of the testing. Shackel (1991) identified four IV. RESEARCH DESIGN
factors on usability which are effectiveness; learn ability,
flexibility and attitude. So, based on this model, the The researcher conducted a testing to evaluate the
researcher chooses the effectiveness, one characteristic in effectiveness of OUM's LMS. The testing approach
Shackel's Model to complete this research. Four types of requires students or users of the LMS to work on typical
questionnaire are designed to determine the effectiveness tasks using the real system. The result from the testing can
of the Learning Management System. Figure 3 indicates be used to show how the user interface, speed, quality and
that the research methodology framework used to evaluate the overall of the LMS can supports the users in their
the OUM's Learning Management System which is learning process as an Open and Distance Learners.
categorized into two areas of study, theoretical and Furthermore, Genise (2002) stated the results of the testing
exploratory. In theoretical study, the researcher review a can also be used by the designers to make any changes on
lot of literatures from various sources such as book, the system design. The testing technique used in this study
journal, articles whether printed or on the web. Whereas in is questionnaires.
exploratory study, the researcher formulate the research by
identifying variables, constructs questionnaire based n one V. VARIABLES
of the usability attributes of Shackel's Model (1991) which
is effectiveness and then the tasks list for testing is The variables used in this study can be classified
designed. Data collection is gathered by conducting the into two types such as independent variables and
testing and questionnaire given to the respondents.


Volume 2 No. 2 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences

©2010-11 CIS Journal. All rights reserved.

dependent variable. Table 6 lists all the independent and as to test student's performance by doing an on-line quiz
dependent variables which will be used for analysis. for the course that they registered, to join the discussion
forum by posting and response to their peer's message and
VI. TESTING finally to find one book by giving them the title of the
book. All the tasks should be completed within fifteen
In evaluating the OUM's LMS, three main tasks minutes or five minutes each tasks. After completing
with a total of four to five sub tasks were formulated based every section of the task the subject has to give the
on the three main features or functions of the system such comments.

Table 1: The set of tasks

Task 1 Task 2 Task 3

1. Select one of the course 1. Post one message asking your 1. Click myLibrary
that you have registered tutor about the subject that button
2. Click on the Quizzes you don’t understand (you 2. Find a book entitled
button in LMS can choose any subject that “Taking Control with
3. Answer the test within 5 you’ registered) Time Management” in
minutes 2. Response to one of your e-Books collection
4. Submit the result friend’s message in LMS and (24x7 database)
5. Comment on Task 1 discuss about the topic raised 3. Display your result
(you can choose any subject 4. Search the book within
that you’ registered) one minutes
3. Do both question 1 and 2 in 5 5. Comment on Task 3
4. Comment on Task 2

VII. SUMMARY AND CONCLUDING any problem in using the Learning Management
System. From the study also, it showed the OUM's
REMARKS Learning Management System is easy to learn for
different levels of users, can easily recover from
The result of this research indicates that the errors once users made errors, can speed up learning
overall level of the effectiveness construct in student’s process as well as has a quality for students of Open
point of view is in the acceptable level. The mean score University Malaysia.
for the overall of the effectiveness on easy to learn factor
is 3.8193 while on the error tolerance factor the mean
score for the effectiveness of OUM's LMS is 3.7383. For
the effectiveness of LMS in OUM on speed factor the
overall mean score is 3.8925 and 4.110 mean score's for [1] Avgeriou, P., Papasalouros, A., Retalis, .S,
the effectiveness on quality factor which is the main factor Skordalakis, M. (2003). Towards a Pattern
that contribute to the effectiveness the OUM's LMS. Language for Learning Management Systems.
From this study it showed that the Learning Educational Technology & Society, 6(2), 11-
Management System of Open University Malaysia is 24, Available at
used effectively by students although students came
from various levels of educational background, age,
marital status, course, level of thinking, computer [2] Aybay, I & Oguz,D. (2002).A Learning
skills an d so on, although novice users at first they Management System Developed at the Eastern
seemed lost and cannot move around the system but Mediterranean University: The Turkish Online
lastly they can used it effectively parallel with their Journal of Educational Technology, Vol 2(2),
title as open and distance learners. The intermittent pp: 1303-652
users at first done several errors but they can recover
the errors easily as well as the expert users, they [3] Barnum, C.M. (2002). Usability Testing and
were frequently used the system so they do not have Research. USA: Pearson Education.


Volume 2 No. 2 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences

©2010-11 CIS Journal. All rights reserved.

[4] Beck, E.(2005). Education Research Institute

University of Oslo In: Berthelsen. Bouvin, [14] Iahad, N., Kalaitzakis, E., Dafoulas, G.A., &
Krogh and Kyng (Eds.) Critical Computing- Macaulay, L.A. (2004). Evaluation of Online
Between Sense and Sensibilities. Proceedings Assessment : The role of feedback in Learner-
of the Fourth Decennial Aarhus Conference, Centered elearning. Proceedings of the 37~'
Aarhus, August 20-24, 2005 (ACM, New York, Hawaii International Conference of System
2005, pp. 173-176). Sciences. Available in IEEE database.

[5] Bouras et al (1999). Efficient Data [15] Kissane, A., & Finn, E. (2003), Development
Management to Support Educational Activities. of a Usability Model for VLE Deployment.
In: International Workshop ICL99, Interactive ILTA, EdTech 2003.
Computer aided Learning - Tools and
Application, Villach, Austria, 6-8 October. [16] Lewis, B.A. (2002). Learning Effectiveness:
Efficacy of Quizzes vs.Discussion in Online
[6] Carter, B. (1999) Effective Integration of Learning. Master Thesis from UMI
Multimedia Technology Into Scientific Dissertation Services.
Thematic Units Brannon Carter ( ED 608) -
Expanding Reading Ability Through Content [17] Lindgaard, G. ls` ed. (1994). Usability
Area Instruction The University of Alabama in Testing and System Evaluation: A guide for
Huntsville designing useful computer system. UK;
Chapman & Hall.
[7] Chapanis, A. (1991). Evaluating Usability. [18] Maish Nichani (2001) e-Learning Post.
Human Factors for Informatics Usability Eds. Retrieved from; maish[
S. B. and S. J. Richardson. Cambridge, citied October 11, 2005]
Cambridge University Press: 359-395.
[19] Maxwell, L. 1995. Integrating Open Learning
[8] Feldman, M.(2002) Media Reviews: LMS and Distance Education. Educational
Breakdown in Ebsco Publishing Folmer, E._ & Technology November-December, 43-48.
Bosch, J. (2004). Architecting for Usability ; A
Survey. Journal of Systems and Software, 70 [20] McCormack, C., & Jones, J. D. (1997).Building
(1-2), 61-7 a Web-based Education System, New York:
Wiley Computer Publishing.
[9] Gaines, B.R. (1988). A Conceptual Framework
for Person-Computer Interaction in Distributed [21] Moore, M. G., & Kearsley, G. (1996). Distance
Systems. IEEE Transactions Systems, Man & Education: A Systems View, Belmont, CA:
Cybernetics SMC-18(4) 532-541. Wadsworth Publishing Company.

[10] Genise, P. (2002). Usability Evaluation: [22] Ndubisi, N.O. (2004). Factors Influencing e-
Methods and Techniques. Version 2.0 Learning Adoption Intention : Examining the
Retrieved August 19, 2005 from determinant structure of the decomposed theory of planned behaviour constructs, 252-
/elvisino/usaEval.html 262.

[11] Hall, J (2003 ) Assessing Learning [23] Nielsen, J. (1990a). Evaluating Hypertext
Management System, senior vice president of Usability. In: D. H. Jonassen, & H. Mandl
Oracle University, the Education and Training (Eds.), Designing hypermedia for learning,
Division of Oracle Corporation. Berlin: SpringerVerlag, 147-168

[12] Harasim, L, (1995) Learning Networks; a Field [24] Oakes, K (2002) e-Learning, Aspen Enterprise
Guide to Teaching and learning Online Learning Paltform. Retrieved from:
Cambridge, Ma: MIT Kevin.oakes (~a,[citied
October 4, 2005 ]
[13] Harry K (1997b) Research Reviews and
Agendas, Section 5, Researching the Literature [25] Ponta, D., Scapolla, A.M., Taini, M. (1996).
on Open and Distance Education, Block 3, Telematics for Education: The Design of a
H801, Foundations of Open and Distance distributed Computer-based Collaborative
Education, The Open University. Learning System. In: Proceedings of ED-

Volume 2 No. 2 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences

©2010-11 CIS Journal. All rights reserved.

TELECOM 96, Boston, MA,USA, 17-22 June [31] Shneiderman, B. (1998) Designing the User
1996, pp. 252-257~ Interface: Strategies for Effective Human-
Computer Interaction (3rd ed ), Reading, MA:
[26] Quint, V (2001) The user Interface Domain Addison-Wesley
retrieved from available on a retrieved 14 April [32] Tella S. 1997. An 'Uneasy Alliance' of Media
2006 Education and Multiculturalism, with a View to
Foreign Language Learning Methodology. Media
[27] Ravi Dharbamulla & Pamela Lawhed, (2004) Education Centre. University of Helsinki.
Paving the Way Towards an Efficient Department of Teacher Education. OLE
Learning Management System ACMSE'04, Publications 4, pp. 14-17].
April 2-3, 2004, Huntsville, Alabama,
USA.Copyright 2004 ACM 1-58113-870- [33] Welsch, E.(2005) SCORM: Clarity or Calamity?
9/04/04 Available at
[28] Sandra ,C.(2005)Distance Education Report : Retrieved on December 12, 2005
Benefits of Upgrading to LMS, May 2005
[34] Prasert Sudjaipraparat (2006), E-learning:
[29] Salter, D., Richards, L & Carey,T (2004). concepts and practice.Available at
Educational Media International: The'TS' http//
design Model: An Instructional Model and Retrieved on December 12, 2006
learning Environment to support the
Integration of Online and Campus-Based
Courses, Council for Educational Media

[30] Shackel, B. (1991). Usability--context,

Framework, Definition, Design and
Evaluation. Shackel, B. and Richardson, S.,
Ed. Human Factors for Informatics Usability.
pp.21-37. Cambridge, UK, Cambridge University