Professional Documents
Culture Documents
LDC E-Learning Guides
LDC E-Learning Guides
Evaluating E-Learning
Ref: Eval-EL-G3
May-04
At a departmental meeting, the Chair says: We have all these new skills
policies. We need to make sure all curricula are explicit about what employable
skills are being developed by our students. We need all tutors to review their
courses in terms of using e-learning to support skills development.
A lecturer has just been given a set of multimedia web materials that support
students in critically evaluating X (problem/issue/topic). S(he) is now tasked
to ensure they know how to use it and what to do with it.
May-04
It is fairly well accepted that e-learning developments that are valuable and
sustainable in the longer term are those based on the use of small-scale, incremental,
non-revolutionary technologies, i.e. mainstream worldware tools (Ehrmann, 2000).
These require far lower investment in terms of cost, maintenance, updating and skills.
In todays climate, the Web is the ultimate worldware tool; this includes web-based
communication tools, but also analysis type applications, such as spreadsheets, maths
tools, design software etc.). The challenge to the lecturer then is to package these
basic tools in pedagogically viable ways. This is where needs analysis can assist in
providing diagnostic evaluation that is, scoping out the objectives of what your
development seeks to achieve against the requirements of the intended end-users.
May-04
Involving students
Involving students in the needs analysis process helps you to engage with them as the
main beneficiaries (or sufferers!) of any new e-learning approaches you put in place.
It can encourage more active participation in the development process. You might
consider holding an initial focus group with students explaining your aims and the elearning being developed and ask for their ideas and feedback as things progress. The
students can also propose areas for investigation and can give feedback on the
effectiveness of your evaluation questions.
Many techniques are available for eliciting student (user) needs. One problem is that
students often do not know or cannot articulate what they want in the course context
For example, new students who have not studied a subject beforehand may not have
the necessary language/terminology. They may lack the knowledge that comes from
experience of using technology: selecting those that have experience of e-learning
activities. Often students who volunteer for focus groups or interviews or return
questionnaires are the technophiles, the technophobes not wishing to publicise their
lack of skills. This can skew your analysis considerably. Selection of students to
involve in a needs analysis therefore requires careful thought, to ensure you obtain
responses that are reasonably representative of the whole student group. You may
have to use elicitation techniques that build knowledge as well as keep a focus on the
users own needs.
May-04
An example of this exercise given in the table below from a session with a group of
academic tutors analysing the broad benefits and constraints of e-learning.
Strengths
Weaknesses
Opportunities
Profiling
Simple documentation templates can be created for recording and developing user
needs statements from an interview or focus group. Questions and discussion might
be used to gather information on the student group:
Students availability or preference of study environments
Student characteristics: entry qualifications, employment aspirations,
language, disability, etc.
Current student problems or concerns
Refining an existing methods or provision
Establishing what students see as appropriate or helpful activities
Recording such information systematically over time helps develop a culture of
considering student needs and experiences as well as gauging successes of a new
course or method.
Undertaking a needs analysis for e-learning
May-04
Curriculum design
A needs analysis for a new method or course is likely to include a review of overall
teaching strategies. E-learning facilitates a whole range of teaching and learning
activities see LDC e-guide E-learning solutions for teaching and learning. However,
a learning activity does not take place in isolation to the teaching and study
environments, assessment tasks, tools used and so forth. It is therefore important to
ensure that all components of the curriculum, including any uses of e-learning
methods or materials, are properly integrated and the purpose of a particular
component is then clear to the student.
Pedagogical models are useful for making explicit the intentions of the e-learning
approach. The needs of the tutors and students will be different in each case.
Looking at needs across different types of development models, Robin Mason
(1998) contends that:
Current approaches to teaching and learning in higher education are
dominated by the following: the importance of interactivity in the learning
process, the changing role of the teacher from sage to guide, the need for
knowledge management skills and for team working abilities, and the
move towards resource-based rather than packaged learning.
She suggests three development models:
1. CONTENT SUPPORT MODEL
Course content and tutorial support are dealt with separately, particularly used where
content does not change significantly or where courses are tutored by external staff.
Collaborative activity (peer commenting, online assessment, computer conferencing)
amongst students is rudimentary, in most cases less than 20% study time, and added
onto the course as supplementary rather than core. With increasing use of the web for
delivering content, there is more scope to extend the balance of content to
collaborative activity.
2. WRAP-AROUND MODEL
The course content consists of tailor made materials (study guide, activities and
discussion) wrapped around existing materials (textbooks, CD-ROM resources or
tutorials) and representing around 50% study time. The remaining 50% is comprised
of online interactions and discussions, including real time online events and screen
sharing with increasing audio/video components. The tutor role is more extensive as
less of the course is pre-determined and students take more responsibility for their
learning.
3. INTEGRATED MODEL
The third model is at the opposite end of the spectrum from the first. The heart of the
course involves collaborative activities, learning resources and joint assignments.
These take place online through discussion, accessing and processing information and
carrying out tasks. The course contents are fluid and dynamic as they are largely
determined by the individual and group activity. In a sense, the integrated model
dissolves the distinction between content and support, and is dependent on the
creation of a learning community.
May-04
The range of technologies that encourage active learning is staggering. Many fall into
one of three categories: tools and resources for learning by doing, time-delayed
exchange, and real-time conversation.
Building on existing course models, it can be useful to map tools and technologies
onto learning activities and to consider how these enhance or enrich what you already
do. A simple list to use might look like this:
Lecture presentation
Knowledge dissemination
Communication with and between students
Assessment for feedback and monitoring
Labs and tutorial activities
Course management
Pedagogically, you might consider what learning theories are being supported (see EGuide Pedagogies for E-Learning). For instance:
Instructivist
Constructivist
Situated learning
May-04
Author
Dr Jay Dempster, Learning Development Centre
Tel: 024 76524670 Email: jay.dempster@warwick.ac.uk
References
Chickering, A. and Ehrmann, S. (1995) Implementing the Seven Principles:
Technology as Levers
Compton, Philip (1997) Evaluation: A practical guide to methods, Philip, in LTDI
Implementing Learning Technology: http://www.icbl.hw.ac.uk/ltdi/implementingit/eval.htm
Ehrmann, Stephen C., (2000) Technology and Revolution in Education: Ending the
Cycle of Failure, Liberal Education, Fall, pp. 40-49. Available on the web:
http://www.tltgroup.org/resources/V_Cycle_of_Failure.html
Gilb, T. and Finzi, S. (1988) Principles of Software Engineering, Addison Wesley.
Mason, R. (1998) Models for online courses. ALT Magazine online at:
http://www.aln.org/publications/magazine/v2n2/mason.asp
Further resources
Pedagogies for E-Learning, LDC e-learning guide:
http://www.warwick.ac.uk/go/LDC/resources/eguides/eskills/pedagogies/
E-Tutoring: Teaching, Supporting, Managing and Assessing Students Online, LDC elearning guide:
http://www.warwick.ac.uk/go/LDC/resources/eguides/eskills/etutoring/
Biggs, J. (2002) Aligning the curriculum to promote good learning. LTSN Generic
Centre paper from Constructive Alignment in Action Imaginative Curriculum
Symposium Nov 2002. Available on the web at: http://www.ltsn.ac.uk/
application.asp?app=resources.asp&process=full_record§ion=generic&id=156
Collis, B. (1996). Tele-learning in a Digital World. Thomson Computer Press, London.
Petersen, R. (1992) Training needs analysis in the workplace, London: Kogan Page.
Stoner, Greg (1996) A conceptual framework for the integration of learning
technology, chapter 3 in Implementing Learning Technology, LTDI publication.
http://www.icbl.hw.ac.uk/ltdi/implementing-it/implt.pdf, pp. 6-8.
May-04
Consider this from your own point of view and from the point of view of the people
you deal with. Be realistic. If you are having any difficulty with this, try writing down a
list of the features of the technology. Some of these will hopefully be strengths!
In looking at the strengths, think about them in relation to the alternative methods for example, how does an online discussion compare to a seminar for achieving
existing learning goals and possible new learning goals?
Weaknesses:
What
What
What
What
Again, consider this from an internal and external basis: Do other people seem to
perceive weaknesses that you do not see? Is the traditional method actually better at
delivering the learning goals? It is best to be realistic now, and face any unpleasant
truths as soon as possible.
Opportunities:
May-04