See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/280485420

Gap Analysis Of LMS Profile Requirements

Research · July 2015

DOI: 10.13140/RG.2.1.2195.1847

CITATIONS READS
0 1,332

2 authors:

Gerry Paul Cacho Genove Cecilia Mercado

Saint Louis University, Baguio City Saint Louis University, Baguio City
4 PUBLICATIONS   5 CITATIONS    15 PUBLICATIONS   36 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Information System Strategies for SME's: SLU-EISSIF Inc. View project

UNDERSTANDING THE READINESS OF IMPLEMENTING OUTCOME-BASED EDUCATION AMONG SELECTED HIGHER EDUCATION INSTITUTIONS IN PHILIPPINES View project

All content following this page was uploaded by Gerry Paul Cacho Genove on 27 July 2015.

The user has requested enhancement of the downloaded file.

 GAP ANALYSIS OF LMS PROFILE REQUIREMENTS
USING STANDARDS

Gerry Paul C. Genove Cecilia A. Mercado

gepaul_genove@yahoo.com cicsdean@slu.edu.ph

ABSTRACT

One of the major setbacks raised by the continually changing developments in

technology solutions is the capacity of the organizations to keep up with these changes. An
upgrade in features and the introduction of additional modules of technology solutions do not
necessitate an immediate upgrade of an application. The proliferation of Learning Management
System (LMS) solutions both proprietary and open source raises a growing concern on
organizations implementing LMS particularly on whether existing LMS solutions are operating
within acceptable service standards both in the perspective of the users and the industry
perspectives. This study aimed to identify the standards-based requirements within which any
existing LMS may be assessed using gap analysis. Requirements include 1) features, 2)
technical and 3) support structure. The study profiled the current state of the My Classes LMS in
the following areas: features, technical, support structure and utilization.

The features profile used an Assessment Methodology identified by Spikesource et. al

(2005) where a list of criteria were identified with their corresponding weights. This was used to
identify a rating for the LMS software used. The technical profile identifies the hardware,
software, network and internet infrastructure utilized by the My Classes LMS. The network and
internet infrastructure was subjected to the Mohktar et. al’s rubric to evaluate the network and
internet infrastructure of an HEI. The support structure profile identified the key responsibilities,
tools and information used to provide support for the current LMS. A rubric to evaluate the
support structure was created using useful information presented by Towns et. al [1] and
Working Group Experts [2]. The study also presents a framework to identify the LMS standards-
based profile in the following areas: features, technical and support structure. The assessment
methodologies used in detailing the features, technical and support structure profiles were used
to define the standards for these 3 areas. A review of best practices and related literature were
done in order to define the standards. A gap analysis between the current profile of the My
Classes LMS and the standards-based profile in these 3 areas identified the gaps which were
inputs in identifying the standard-based requirements. Moreover, an analysis of the LMS server
logs, the student and faculty surveys identified the factors that affected the stakeholders’
utilization of the LMS. These factors were used to identify additional requirements of the current
LMS.

Result of such can equip educational and technology administrators informed decisions
on upgrade or purchase or development of new LMS solutions.

CATEGORIES and SUBJECT

ICT in Education, eLearning

KEYWORDS
Learning Management System, LMS features, technical features, support structure, LMS, Open
BBR, profiling
1. INTRODUCTION

Education has become a commodity in which people seek to invest for their own
personal gain, to ensure equality of opportunity and as a route to a better life[3]. The
introduction of e-learning poised the Philippine government to effect changes in the landscape
of education in the Philippines. It created a number of initiatives in partnership with external
agencies since the year 2000. One form of e-learning tool that is widespread today is the use of
a Learning Management System. This is sometimes referred to as Course Management
Systems(CMS) or Virtual Learning Environments(VLE). A Learning Management System(LMS)
is a software application or Web-based technology used to plan, implement, and assess a
specific learning process[4]. Vovides et. al [5] noted that LMS are utilized in education in
different ways and are evolving. It can be used as a supplement to the traditional classroom
curriculum, i.e., as an electronic repository of course materials. Instructors who teach in-class
courses may also choose to use a ‘blended’ approach by utilizing the CMS as a tool to deliver
additional or supplemental course materials to students. Traditional teaching environments tend
to be teacher-centered while a blended approach allows instructors to mix things up and to offer
students a more intellectually engaging learning experience by combining in-class time with on-
line components through the use of synchronous and asynchronous tools. Finally, a CMS can
be used in distance education for the delivery of fully online courses.
The implementation of CMS in universities followed on the revolution of educational
technology that promised better quality, learner-centered education and stipulates that it would
deliver more independent and active students[6]. There are evidences that LMS is a preferred
e-learning tool that is continuously gaining popularity and is being embraced as a major platform
in the educational technology revolution.
The presence of e-learning methods and technologies would not be sufficient means of
enhancing the learning process. According to Wagner et. al [7], Successful implementation of
e-learning is dependent on the extent to which the needs and concerns of the stakeholder
groups involved are addressed. Problems in the implementation and utilization of an LMS exist.
According to Evans [8] “There are many reasons for determining why teaching staff is not
integrating technology into their classroom lessons”. The purpose of this study to present a
framework that will maximize the potentials of the existing LMS by identifying current
implementation issues and problems in line with features, technical setup, support structure and
utilization through the process of benchmarking with the standards. By using the gap analysis,
a set of requirements can be formulated wherein the real essence of its introduction will be
realized in an efficient manner.

2. METHODOLOGY

2.1 Establishing the Current and Standard Profile of the LMS

2.1.1 Features Profile

Evaluating software is a significant task for corporate IT managers, but probable users of
open source software lack an easy, effective, and trustworthy process for decision making.
There is no widely used model for assessment. This complicates open source adoption, as
companies assessing open source software can rarely learn from each other’s experiences [9].
Several methodologies exist in evaluating open source software. These often include a set of
criteria for the evaluation. Some examples of these evaluation methodologies are the Open
Source Maturity Model by CapGemini (OSMM CapGemini), Open Source Maturity Model
developed by Navicasoft’s Bernard Golden (OSMM Navica) and Qualification and Selection of
Open Source Software (QSOS) developed by Atos Origin. Another methodology for an Open
Standard in evaluating OSS is the Open Business Readiness Rating (Open BRR) developed by
Carnegie Mellon West, SpikeSource, O'Reilly, and Intel. Its goal is to enable the entire
community (enterprise adopters and developers) to rate software in an open and standardized
way. By adopting a framework for evaluating software, the current LMS features profile and the
standard LMS features profile can be created. This study implements the OpenBRR model as
an example to establish the features profile.
The profiling method for the current LMS includes review of product documentation, the
LMS software website and web searches for related documents can be used to complete the
needed information. Moreover, the profiling method for the standard LMS includes identification
of properties and features that should be present in an LMS software by reviewing related
documents pertaining to best practices in the implementation of an LMS software.

2.1.2 LMS Technical Profile

This section profiles the Hardware and Software components being utilized by the
current LMS. The hardware running the LMS, the server technologies that were utilized and the
operating system of the server formed this profile. Moreover, the standard hardware and
software LMS profile will be created using hardware and software recommendation from the
LMS vendor or developer.
The rubric for assessing ICT infrastructure [10] was used to identify the network and
internet connection profile and identify acceptable standards which were used in the
assessment in this area. Mohktar et. al [10] presented in his study an evaluation tool that will
evaluate the readiness of an institution for e-learning. Rubrics are set of categories that define
and describe important components of the areas being assessed. Each category contains a
gradation of levels of implementation with a score assigned to each level [11]. The rubric
adopts a 3-point scale to differentiate the levels of ICT implementation. The three categories of
ICT implementation are descriptively labeled as low, moderate and high to represent the lower,
middle and upper tier of the rubric scale. Part of this rubric particularly the area to measure the
network and internet was used to evaluate the network and internet infrastructure utilized by the
current LMS. Defining the standard for each criterion entails a review of acceptable network and
internet connection standards applied to HEIs. The rubric that is used to measure the network
and internet infrastructure is shown on Table 1.
Table 1 : Network and Internet Assessment based on the Rubric formulated by Mohktar[10]
Levels of implementation
Network and Internet Indicators
Low Moderate High

100 Mbps Fast

Network Specification 10 Mbps Ethernet or less Gigabit Ethernet or better
Ethernet

Dialup or broadband up to Broadband, 2 to 7

Internet bandwidth Broadband, 8 Mbps or better
1 mega bits per seconds Mbps

Less than 25% of learning 25% to 50% of

Wireless coverage More than 50% of learning area
area learning area

Slowness/unreliability a Generally works well Always smooth without

Network performance
frequent problem but slow at busy times appreciable delay

Slowness/unreliability a Generally works well Always smooth without

Internet performance
frequent problem but slow at busy times appreciable delay

Source :http://eprints.utm.my/3115/1/Rubric_For_Assessing_ICT_Infrastructure_in_Malaysia_Higher_Education_2006.pdf
2.1.3 Support Structure Profile

This section identified the support structure for the administration and support related
activities involving the end-users. The rubric for assessing the support structure shown in table
2 was used to create the current LMS support structure and LMS standard support structure
profile. The support structure assessment rubric was derived from User Support Best practices
identified by Towns et. al [1] and variables found on the ICT Maturity Tool specifically the ICT
Organizational Support Assessment identified by Working Groups Experts [2]. The ICT Maturity
tool was developed by Carnegie Mellon and International Development Research Center
(IDRC) to assess the readiness of an institution in integrating ICT in HEIs. The assessment is
based on knowledge base of global trends in ICT applications in different universities and
studies of academic experiences and best practices. A component of this tool addresses the
ICT Organizational Support which is aimed in measuring key indicators addressing the user
support structure. The ICT Organizational Support determines the success or failure of ICT
applications in higher education. Since the study was limited to the support structure concerns
of the LMS system, other components found on the tool were not included. Towns et. al [1]
identified a support model that enumerates key components that should be present in a
support system. These components were used to identify the profile of the LMS system
particularly addressing the support model components. The support model components
identify functional areas or variables that are critical in the implementation of a support system
structure. Best practices and experiences by LMS experts in the industry identified by the
eLearning Guild were reviewed to formulate metrics for each component found on the rubric.
Table 2 : Rubric for Assessing the Support Structure

Scoring
Component /
Description
Variable
1-
5 – Excellent 4 - Very Good 3 - Acceptable 2 - Poor
Unacceptable

Support Model Components

User Information Availability of Complete; Regularly Available and Available but Not available
and Tools Frequently asked Available online; updated, almost almost complete incomplete (less
questions, discussion Regularly complete(above (above 50%) but than 50%) and not
forums and online updated 75%) and not updated on a updated
documentation. available online regular basis
and updated

Service Level Appropriately set the Complete set of Majority of the Around half of the Less than 50% None
Agreements & shared expectations SLAs and policies users adopts users adopts adopts SLAs and
Policies the users of the LMS SLAs and SLAs and policies policies
and those providing policies(above
75%)
Support

User Accounts All users need to Supported online Supported online None
and Allocation obtain some type of and measures with limited
Procedures account and some are in place in capabilities/
form of authorization to case problems capacity
use specific occur.
Resources
Education and The users of need to Regular and Scheduled Scheduled On-demand and none
Training be educated and scheduled training but limited training but limited attended by
trained in their use training; available to the personnel to the critical selected or few
to all users. New who acts as 1st or personnel who users
faculty and 2nd line of support acts as 1st line of
students are in the department. support in the
subjected to He is then department
training. Trained responsible in Teachers train
teachers train training his peers their students
their students in the department.
Trained teachers
train their
students

Help Desk No support function Available 24/7 Limited to office Limited to office Limited to none
Process would be complete Full time school- hours but online / hours availability of
without the core of staff based or agency electronic or auto- Part-time school- support staff
providing day-to-day support with mated alterna- based or agency Technical support
assistance to additional staff tives are available support response time
the users of the (including faculty) Full time school- greater than 24
based or agency Most technical
resources and services Most technical support response hours
available support capable
support response of trouble time is less than
time is less than shooting basic 24 hours
4 hours network and
hardware repair
including assistive
technologies
Technical support
response time is
less than 8 hours

Support Staff The support staff must Complete set of Complete set of Most Information Incomplete set of none
Information and have at their disposal a information which information which are available tools and
Tools number of “tools of the is digitally is digitally (more than information
trade” and information available and available and 75%)and
resources to effectively support tools are support tools(more than
provide support to the complete and tools(more than 60%) are
user community available 80%) are available
available

Measuring A support group needs An online An online An online An online feedback No Feedback
Success through some way to determine feedback system feedback is feedback is is available. System is
a Feedback success or failure of is available. available. available. Technology not available.
System problem solving and Technology Technology Technology used used to review
support regularly used to frequently used to infrequently to student
methods review user review user review student assessment
assessment assessment assessment information.
information which information information.
results in needed
changes.

Support Staff

Support Where is end-user A support system Presence of 1st, Presence of 1st, Centrally available None exists
responsibilities support available? Can that is and 2nd level of level of support with limited capacity
either be or a systematically & support;
combination of the efficiently
following ; Centralized structured
support, decentralized combines the
support(1st and 2nd strengths of the
level support), different end-user
presence of Service support;
Level Presence of 1st,
Agreements(SLAs) 2nd and 3rd level
of support;
Defines SLAs
 Staff in the The staff caters to the Network Ma- Network Network Network Very limited /
following following set of nagement only management only management management only None exists
technical ICT responsibilities plus administra- plus only plus
areas Network management, tive system administrative administrative
Administrative system analysis and system system
analysis and design,
design plus analysis and analysis and
Intranet and internet
Hardware main-
application design plus design
tenance and
development, Hardware
repair and Data-
Database maintenance
base manage-
management,
ment plus Intra- and repair
Hardware maintenance
net and Internet and Database
and repair, Help desk
application management
development
and help desk

Staff in the Systems System 1st and/or 2nd level maintenance -Systems Very limited /
following ICT administration, System administrator of user support and control administration only None exists
functional areas maintenance and (e.g. library and
control, Primary for online courses (e.g. library
system of user support instructional and archives archives system,
technology (to finance,
system,
combine finance, student student Registration
pedagogy with system,
registration
technology) human
system, human
resources, etc)
resources,
1st and/or 2nd
level of user etc)
support

2.1.4 My Classes Utilization Profile

This section profiles the utilization of the current LMS. This identifies the utilization level
of the LMS for both faculty and students. Server statistics coming from the LMS can be used to
capture how the system was utilized on the previous semesters and how it is was utilized in the
current semester. A survey on how the LMS is being utilized and key factors affecting their use
of the LMS will help profile the current utilization of the LMS. Information like the distribution of
faculty and student users, the tools and the time the LMS is utilized should be presented to
better understand how the current LMS is utilized. A survey on how the LMS is being utilized
which identifies key utilization issues will help profile the current utilization of the LMS.
Benchmarking with HEIs who successfully implemented an LMS with high utilization levels and
interviews with school administrators defining the ideal utilization level will we inputs in coming
up with the standard utilization profile of the LMS.

2.2 Using Gap Analysis to Identify Standard requirements for the LMS

Gap analysis consists of defining the present state, the desired or `target' state and
hence the gap between them. In the later stages of problem solving the aim is to look at ways to
bridge the gap defined and this may often be accomplished by backward-chaining logical
sequences of actions or intermediate states from the desired state to the present state [12]. The
gap analysis was used for the features, technical and support structure assessment. Using
these inputs, gaps can be identified and measures or actions on how this gap can be bridged.
The assessment of the utilization profile of the current LMS is also presented in this section.

2.2.1 Features Assessment

An evaluation of the LMS software was made in order to prove the competence of the
current LMS used. Gauging the competence of the LMS software is important to prove the
merit of the software being capable in meeting the needs of its users and intended purpose. A
gap analysis for each criterion as well as the overall rating of the current LMS was benchmarked
with the identified standard rating for each category and the overall standard rating. This
determined whether the current LMS software was fit and capable in meeting the competency
level of a standard LMS. Table 3 presents an assessment example of the features profile of
Saint Louis University(SLU) myClasses LMS with the features profile of the standard LMS.
Table 3 : Gap Analysis on the LMS Features
Target
Current
Percen- State –
Categories/Metrics State - My Gap Description
tage Stan-
Classes
dards
The standard set for the functionality
Functionality (25%) 4.4 4.1 0.3
criterion is satisfied

My Classes LMS exceeds standard that

General Features of LMS 70 5 4 1
was set for this sub-criterion

My Classes LMS meets standard that was

Special Features 0 3 3 0
set for this sub-criterion

Manageability of My Classes LMS meets standard that was

20 5 5 0
Functions set for this sub-criterion

The standard set for the usability

Usability (20%) 4.75 4 0.75
criterion is satisfied

My Classes LMS meets standard that was

End-user UI experience 50 5 5 0
set for this sub-criterion

Time for setup pre-

My Classes LMS exceeds standard that
requisites For installing 25 4 3 1
was set for this sub-criterion
open source software

Time for vanilla My Classes LMS exceeds standard that

25 5 3 2
installation/ configuration was set for this sub-criterion

The standard set for the documentation

Documentation (15%) 5 5 0
criterion is satisfied
Existence of various kinds My Classes LMS meets standard that was
70 5 5 0
of documentation set for this sub-criterion

User contribution My Classes LMS meets standard that was

30 5 5 0
framework set for this sub-criterion
The standard set for the community
Community (12%) 4.2 3 1.2
criterion is satisfied
Average volume of
My Classes LMS meets standard that was
general mailing list in the 40 3 3 0
set for this sub-criterion
last 6 months
Number of unique code
My Classes LMS exceeds standard that
contributors in the last 6 60 5 3 2
was set for this sub-criterion
months
The standard set for the support
Support (10%) 4 4 0
criterion is satisfied

Average volume of
My Classes LMS meets standard that was
general mailing list in the 50 3 3 0
set for this sub-criterion
last 6 months (50%)

Quality of professional My Classes LMS meets standard that was

50 5 5 0
support set for this sub-criterion

The standard set for the security

Security (10%) 4.5 3.5 1
criterion is satisfied
Number of security
vulnerabilities in the last 6
My Classes LMS exceeds standard that
months that are 50 4 3 1
was set for this sub-criterion
moderately to extremely
critical (50%)

Number of security
My Classes LMS exceeds standard that
vulnerabilities still open 25 5 3 2
was set for this sub-criterion
(unpatched)

Is there a dedicated
My Classes LMS meets standard that was
information (web page or 25 5 5 0
set for this sub-criterion
wiki) for security? (25%)
The standard set for the adoption
Adoption (8%) 4.4 4.6 -0.2
criterion was not satisfied
How many book titles
does Amazon.com give The number of books listed in Amazon
for Power Search query: 20 2 3 -1 does not meet the standard for this sub-
“subject: computer and criterion
title:component name”?

My Classes LMS meets standard that was

Reference deployment 80 5 5 0
set for this sub-criterion

Current LMS software meets standard

Overall Rating (100%) 4.506 4.085 0.421
that was set

2.2.2 Technical Assessment

2.2.2.1 Hardware and Software Assessment

This section presents the assessment of the hardware and software used by the My
Classes LMS benchmarked with the standards. The Gap analysis was used for this purpose.
The assessment was done in order to identify whether a specified requirement meets the
minimum requirement specified by the vendor or distributor of the LMS software based on the
released HW/SW documentation. Table 4 presents an assessment example of HW and SW
utilized by the SLU myClasses LMS with the standards using Gap Analysis.
Table 4 : Gap Analysis on the Hardware and Software Requirements
Target State
Requirement / Current State (My
(Recommended Gap Description
Property Classes LMS)
Specification)
Hardware
Dual Core AMD
1 Quad-Core Intel
Opteron 2210; Speed
Xeon Processor The processor of the My Classes LMS is
Processor : 1.8GHz, Level 2 +
X5450 (3.0GHz 12MB better than the standard that was set
cache :2X1MB
L2 1333MHz 120W
Cache,1Ghz

1GB DDR2, 667MHz, Memory requirement for My Classes LMS

4GB Fully Buffered
Memory 2x512MB Single + exceeds the minimum memory
DIMM 667MHz
Ranked DIMMs requirement

2 73GB 10K 2.5" Hot-

Swap SAS HDD
36GB, SAS, 2.5-inch, The disk requirement exceeds the
Disk operating in raid 0 + 1 +
10K RPM Hard Drive minimum that was set
146GB 10K 2.5" Hot-
Swap SAS HDD
Dual Embedded
2 1 GB Ethernet;
Broadcom NetXtreme Network controller meets the requirement
Ethernet / Network Integrated dual 0
II 5708 Gigabit / standard that was set
Gigabit Ethernet
Ethernet NIC
 Product Product documentation requirement is
Available Available 0
Documentation satisfied
IBM 16X RAM-Read
DVD-ROM IDE Drive In terms of the classification of the
IBM C117 17 CRT No monitor, keyboard machine, since the standard hardware is
Monitor and mouse a rack-mount type of hardware as
IBM Preferred Pro Remote compared to a tower type of hardware,
Other components +
Keyboard USB - US administration through these other components are necessary
English 103P SSH, PHPmyAdmin for the My Classes LMS server and are
IBM 2 Button Optical and SFTP. optional components for the rack-mount
Wheel Mouse - Black type of server
- USB
Software
CENTOS 5.1 Debian Linux The Linux operating system is common to
Operating Systems (Community Windows and Mac OS 0 both the My Classes LMS and the
ENTerprise OS) is also supported standard OS that was set

SW Technology : The version used is newer than those

Version 2.2.3 Version : 1.x or 2.x +
Apache that was set as standard
SW Technology : The version used is newer than those that
Version 5.0.22 Version : 3.23.6 +
MySQL was set as standard
The version used is newer than those that
SW Technology : PHP Version 5.1.6 Version : 4.3.2 +
was set as standard
SW: Administrative
All of these
Tools (SSH, mySQL The version for the tools used for
technologies are Any version 0
Adminis-trator & Query administration is met
present
Browser, SFTP)

2.2.2.2 Network and Internet Assessment

A gap analysis was performed to identify the gaps between the current network and
internet structure utilized by the LMS with the standards that were defined. Table 5 shows an
example of SLU’s myClasses LMS Gap Analysis on the Network and Internet Infrastructure.
Table 5 : Gap Analysis on the Network and Internet Requirements

Network and Internet Current State (My Target State

Gap Description
Indicator Classes ) (Standards)

Meets the standard requirement of providing

Network Specification 2 2 0
100 mbps ethernet
Exceeds the minimum requirement where
Internet bandwidth 3 2 1 14mbps is classified as high level of
implementation
Meets the standard requirement of covering 25
Wireless coverage 2 2 0
to 50% of learning area
Exceeds standard requirement where the level
Network performance 3 2 1 of implementation of the SLU network is
classified as HIGH
Exceeds standard requirement where the level
Internet performance 3 2 1 of implementation of the SLU internet
infrastructure is classified as HIGH
2.2.3 Support Structure Assessment
Using Gap Analysis, the support structure was assessed using the score obtained in the
assessment of the My Classes support structure with the support structure standards. The
current support structure profile and the standards for support structure were identified using a
scoring system provided by the rubric for support structure. Table 6 shows an example of a Gap
analysis on SLU’s myClasses LMS Support Structure.
Table 6 : Gap Analysis on the LMS Support Structure
Current Target
Support Structure Component State - My State -
Gap Description
/ Variable Classes Stan-
LMS dards

Support Model Components

The following properties were not present on the My

User Information and Tools
Classes LMS:
Complete set of user information and tools
2 5 -3
End-user information and tools that is locally customized
and available online
End-user information and tools that is regularly updated.

The following properties were not present on the My

Classes LMS:
Service Level Agreements &
1 5 -4 Complete set of SLAs covering operational activities of
Policies
all end-users
Complete set of Policies

User Accounts and Allocation Current setup meets the required standard for user
5 5 0
Procedures accounts and allocation procedures

The following properties were not present on the My

Classes LMS:
Education and Training 2 5 -3 Regular and scheduled training
Training for all types of users (faculty and students)
Training for support staff
The following properties were not present on the My
Classes LMS:
Help Desk Process 2 3 -1 Technical response time that is less than 24 hours
Period for help desk process covers the duration of
office hours

Support Staff Information and The following were not present on the My Classes LMS:
4 5 -1
Tools Complete set of support staff information and tools

The following were not present on the My Classes LMS:

Measuring Success through a A reliable online feedback system
2 5 -3
Feedback System Technology regularly used to review user assessment
information which results to needed changes.

Support Staff
Support responsibilities
Staff in the following technical
ICT areas
Staff in the following ICT
functional areas
The following were not present on the My Classes LMS:
2 3 -1 st
presence of 1 level of support
Current setup meets the required standard for user
4 3 +1 accounts and allocation procedures
The following were not present on the My Classes LMS:
st nd
1 and/or 2 level of user support
3 5 -2 System administrator for online courses instructional
technology (to combine pedagogy with technology)
acting in full-time capacity
2.2.4 Utilization Assessment
The assessment of the My Classes utilization was performed through a quantitative and
qualitative analysis of the information collected. Factors affecting the stakeholders’ utilization of
the LMS were identified after a methodical analysis of data gathered. A qualitative analysis of
the comments given by the respondents further helped in identifying these factors. A review of
the current utilization profile and the optimal utilization goal that was defined through the review
of successful LMS implementation and goals set by school administrators can be used to
identify the gaps between the current utilization of the current LMS and the target(standard).
The information gathered after the analysis was used to create recommendations on possible
actions to improve the utilization of the My Classes LMS.

2.3 Standards-based requirements to optimize the deployment of My

Classes – Learning Management System

This section identified the standards-based requirements on the following areas:

features, technical, support structure and utilization aspects of the current LMS. A set of
requirements can be formulated whose goal is to improve the deployment of the current LMS
and address issues that were identified as problem areas or gaps. The assessment done on
Section 3.2 was used to identify these gaps. Table 7 presents an example summary of the
standard-based requirements for the LMS applied to the case of SLU’s MyClasses LMS.
Table 7 : Standard-based Requirements for the LMS

Implementation Area Identified Gaps Requirements

Features none -

Technical none -

Support Structure

The complete set of user information and tools provided by Dokeos should
Complete set of user be available on the SLU Network.
information and tools
A Frequently Asked Questions page should be created and reflect basic
answers to common questions asked by users.

End-user information
User Information Information and tools are customized according to the needs of the My
and tools that is locally
and Tools Classes LMS users. Online manuals for students and teachers that are
customized which is
localized should be created.
available online

End-user information Review and update end-user information and tools. This shall be done
and tools that is specially during times where changes or upgrades are done to the software
regularly updated or when a need for additional documentation is requested by the end-users

Complete set of SLAs

Service Level Create SLAs with end-users and offices assisting the end-users. This will
covering operational
Agreements & guide the users on how problems are to be resolved and the personnel to
activities of all end-
Policies approach once LMS support services are needed.
users
 There should be a motivational push from the administration for the LMS to
be mainstreamed by setting policies and offering incentives to faculty who
will be adopting the LMS.
Policies about LMS use and service should be introduced. This shall include
the scope of how the LMS shall be utilized. These policies should be
channelled on effective communication channels to inform all users. Some
possible policies may include the following as suggested by Waterhouse &
Rogers (2010):
course policies in the course syllabus
student privacy policies
Complete set of
email policies
Policies
forum discussion policies
software standard policies
assignment policies
feedback policies
getting technical help policies
student code of conduct policies
intellectual property rights policy
Policies and procedures should be available in printable form
For large policy changes that are likely to be controversial, giving users an
opportunity to voice questions and discuss issues may be useful.

Standardize training schedules on a school-year or semester basis.

For new faculty, a scheduled training of the LMS should be done by the LMS
staff through the personnel office for newly hired faculty. The training should
introduce the features of the LMS, creation and administration of online
courses, and adding content to courses. The training should emphasize the
teacher’s role as prime movers for their students’ adoption of the LMS and
their active role of training their students in their use of the LMS.

Regular and scheduled For freshmen students, a brief introduction of the LMS can be done during
Education and training for student and their scheduled orientation. It is also suggested that an LMS primer be
Training faculty users included on the student handbook or primer that is distributed during
enrolment.

A scheduled training for all departments of all colleges should be done by the
LMS staff to ensure that every faculty receives LMS training.

An effective approach of teachers training their students would be a more

feasible than the LMS staff handling all training needs of students.

It is recommended that regular walk-in trainings for students and teachers be

conducted on a weekly or monthly basis.

During system upgrades or updates, all support staff are re-trained and
informed of significant changes

Ongoing training of support service staff to handle user needs. In order to

produce a dynamic team of support staff members, enrichment training
Education and
Training for support through attendance in LMS related seminars or trainings should be done.
Training
staff
(continued)
An exchange of knowledge through regular discussion of user problems and
experiences and how they were handled is suggested for people who are
part of the support group. This can be done through a discussion forum
which can be placed on the LMS. An FAQ should be created and updated to
reflect the topics found on the forum.

Technical response Create a dedicated support group to handle this function. The LMS support
time is less than 24 group should be able to handle all problems within the day wherein the
hours response time is less than 24 hours.
Help Desk Process
Duration of the whole
Availability of help desk process during office hours by delegating the
office hours period for
responsibility to the support staff.
help desk process
 Set of support staff information should be well documented and available
online. The information should be updated frequently to reflect important
documentation about troubleshooting known end-user as well as technical
Support Staff Complete set of support
LMS problems.
Information and staff information and
Tools tools
Support tools such as browsers or browser plug-ins should be complete and
up to date. Other technical and networking tools should be available using a
common online repository like an FTP server.
A feedback mechanism through regular surveys conducted at least once a
year should replace the existing feedback mechanism. A survey to measure
user insights about the LMS should be done in order to get useful user
A reliable online
feedback information.
feedback system is
available.
Measuring Success A structured form of feedback mechanism through an online form document
through a Feedback should be available to replace feedback through e-mail which can be
System accomplished voluntarily.

Technology regularly
Feedback information is assessed and recommendations are created for the
used to review user
improvement of the LMS. This information should be benchmarked with
assessment information
similar studies to provide the best recommendations in further improving the
which results in needed
LMS.
changes.
A structure of support for end-users should be created. The responsibilities
of the support groups shall delineate key responsibilities for services
expected by the end-users. This should be embodied on the SLA that will be
formulated for students and teachers.
st st
Support Presence of 1 level of The 1 level of support for students shall be their instructor conducting the
responsibilities support online course. In case problems are unresolved, the faculty may seek
advice or help from his department head or may seek the help of the LMS
support group.
st
The 1 level of support for teachers/faculty shall be the department head.
Faculty may seek advice from the LMS staff if issues are unresolved.
st nd
1 and/or 2 level of A structure of support for end-users should be created to provide support
user support services to the end-users.
Staff in the System administrator
following ICT for online courses
functional areas A dedicated LMS system administrator should be appointed on a full time
instructional technology
basis or a review of the setup of functions that will allow the current LMS
(to combine pedagogy
administrator to achieve the same objective.
with technology) acting
in full-time capacity
Implement education and training requirements identified for support
Training structure particularly on training and education identified in the support
structure.

Support services Implement requirements identified under support structure

Lack of information to
Implement requirements identified under support structure
promote the LMS
Utilization Improve staffing responsibilities detailed under support structure
Further improve technical facilities to produce a more reliable system which
LMS reliability
encompasses all hardware equipments handled by the ICTR Lab and
SLUNet Office and IT equipments used by end-users in the university.
Address these issues by looking into the recommendations of the following
studies:
IT equipment problems Mercado (2008)
Parilla-Ferrer (2007)
Tagimacruz (2005)
3. CONCLUSION

A standards-based requirement in the implementation of LMS addresses a number of

issues. These may include underutilization of an existing LMS, formalization of its
implementation and meeting a standard implementation of an LMS. This study focused only on
four aspects where standards along LMS may be measured. When existing LMS are in place,
and that a need for its evaluation along its utilization and its assessment along existing
standards, the methodology presented in this study is useful and that evaluation using gap
analysis method is most appropriate.
A set of requirements based on the gaps and factors affecting the stakeholders’ utilization
can be identified by considering the results areas of the gap analysis. In areas where no gaps
were determined, no requirements were formulated but recommendations were given to further
solidify the strengths of the LMS in the coming semesters and school years. Using the gaps
that were identified in the assessment, a detailed set of requirements was identified. These
requirements were based from the deficiencies identified in the gap analysis. Using the
standard-based requirements and recommendations a profound solution to the underlying
problem was presented.

4. REFERENCES

[1] Towns, J., Ferguson, J., Fredrick, D. and Myers, G., 2001,‘Grid User Support Best
Practices’, viewed 2009 October, <http://www.ggf1.nl/abstracts/GUS/ GridUserServicesBest
Practices-02221.pdf>
[2] Working Groups Experts 2000, ‘Guidelines for Institutional Self-Assessment of ICT maturity
in African Universities’, viewed 2009 September, <http://www.aau.org/english/ documents/ICT-
GUID.pdf>
[3] Davies, D. 1998, ‘The Virtual University: A Learning University’, The Journal of Workplace
Learning, vol. 10, no. 4, pp. 175 – 213.
[4] Paulsen, M, 2002, ‘Online Education Systems in Scandanavian and Australian Universities:
A Comparative Study’, The International Review of Research in Open and
Distance Learning Journal, vol. 3, no. 2., viewed 2009 July, <http://www.irrodl.org/
index.php/irrodl/article/view/104/559. Retrieved July 10, 2009>
[5] Vovides, S., Sanchez-Alonzo, S., Mitropoulou,V. and Nickmans, G., 2007. ‘The Use of e-
Learning Course Strategies and to Improve Self-regulated Learning’, Education Research
Review, vol. 2, pp. 64–74.
[6] Swinney, L. A. , 2004, ‘Why Faculty Use a Course Management System(blackboard) to
Supplement their Teaching of Traditional Undergraduate Courses’, Doctor of
Philosophy Dissertation, University of North Dakota
[7] Wagner, N., Hassanein, K. and Head, M., 2008, ‘Who is responsible for E-Learning Success
in Higher Education: A Stakeholders’ Analysis’, Journal of Educational Technology and
Society, vol. 11, no. 3, pp. 26 - 36.
[8] Evans, K., 2005, ‘Front End Analysis Plan for the Underutilization of Technology at Lincoln
Middle School’, viewed 2009 September, <http://www.kristenevans.info/su/
reports/FEAplan.pdf>
[9] SpikeSource, the Center for Open Source Investigation at Carnegie Mellon West, and Intel
Corporation, 2005, ‘Business Readiness Rating for Open source; A Proposed
Open Standard to Facilitate Assesment and Adoption of Open Source Software, RFC1’ viewed
2009 July, http://www.openbrr.org/docs/BRR_whitepaper_2005RFC1.pdf
 [10] Mokhtar, S., Alias R., Rahman A., 2007, ‘Rubric for Assessing ICT Infrastructure in
Malaysia Higher Education’, viewed 2009 September, <http://eprints.utm.my/3115/1/
Rubric_For_Assessing_ICT_Infrastructure_in_Malaysia_Higher_Education_2006.pdf
[11] Pickett, N., 1998, Creating Rubrics [online], viewed 9 February 2006,
<http://teacher.esuhsd.org/rubrics/ >
[12] UC 2008, ‘University of Canterbury LMS Review – Final Report and Recommendations’,
viewed 2009 August, http://uctl.canterbury.ac.nz/
files/staff/moodle/Final%20Report%20of%20the%20LMS%20Review%20Steering%20
Group%20-%20public%20version.pdf
[13] Deprez, J.C. and Alexandre, S., 2008, ‘Comparing Assessment Methodologies for
Free/Open Source Software: OpenBRR & QSOS’, viewed 2009 October,
<http://www.qualoss.org/dissemination/DEPREZ_CompareFlOSS AssessMethodo-Camera-
02.pdf
[14] Letellier F., 2009, ‘FOSS-Bridge’, viewed 2009 October,<http://netnam.vn/foss-
bridge/uploads/Main/Foss-Bridge-Block27.pdf >
[15] De Silva, C., 2009, ‘Open source software assessment methodologies’, viewed 2009
October, <http://www.itpro.lk/?q=node/2814>
[16] Center for learning & Performance technology for Instructional Course tools, 2009,
‘Learning Tools Compendium’, viewed 2009 December,http://www.c4lpt.co.uk/Directory/
Tools/instructional.html
[17] Kvavik, R., Caruso, J., Morgan, G., 2004, ‘ECAR Study of Students and Information
Technology, 2005:Convenience, Connection, Control and Learning’, EDUCAUSE Center for
Applied Research, vol. 5
[18] Edutools 2005, ‘EduTolls Website’, viewed 2009 December,
http://www.edutools.info/glossary.jsp?pj=4
[18] Smith, S., Salaway, G., Caruso, J., Katz, R., 2009, ‘ECAR Study of Undergraduate
Students and Information Technology, 2009’, EDUCAUSE Center for Applied
Research, vol 6 [18] McHenry, B., 2010, ‘New Features for Learning
Management Systems’, viewed 2010 January, http://www.sloanconsortium.org/
publications/magazine/v3n2/mchenry.asp

View publication stats