NHS Shared Learning

Quality Assurance Checklists for Evaluating Learning Objects and Online Courses

February 2009

Page | 1
Note
This document provides an outline of the Resource workflow within NHS Shared Learning and also provides the checklists of
evaluating learning objects and online courses before they are uploaded and published within NHS Shared Learning.

To provide high quality learning resources, all resources in NHS Shared Learning (NSL) should be subject to a quality assurance
process requiring that they meet the majority of the evaluation checklist criteria outlined below.

The following checklists are built upon the peer review checklist and development guidelines produced by the e-Learning
Alliance NHS Special Interest Group and the evaluation criteria produced by the Southern Regional Education Board
(www.sreb.org). They aim to cover the technical and educational quality of e-Learning resources.

Page | 2
Table of Content

Resource Workflow......................................................................................................................4

Who does what?..........................................................................................................................7

Contributor’s Quality Assurance Checklist for evaluating learning objects .................................8

Standard A: Content quality .....................................................................................................................9
Standard B: Learning objective alignment ................................................................................................10
Standard C: Feedback ...........................................................................................................................11
Standard D: Motivation ..........................................................................................................................11
Standard E: Design and Usability.............................................................................................................12
Standard F: Accessibility .......................................................................................................................13
Standard G: Reusability and standards compliance ....................................................................................14
Standard H: Intellectual property and copyright .......................................................................................15

Contributor’s Quality Assurance Checklist for evaluating online courses..................................16

Standard A: Content..............................................................................................................................16
Standard B: Instructional design..............................................................................................................17
Standard C: Assessment ........................................................................................................................19
Standard D: Technology.........................................................................................................................20

Contributor’s Quality Assurance Checklist for resources during the upload process within
Intralibrary................................................................................................................................21

How can Review Panel help in Peer review?..............................................................................22

Page | 3
Resource Workflow
A basic workflow* for a resource before it is published within NHS Shared Learning will be as
follows:

Step 1 – Upload resource

Contributor (whether organization or individual) uploads the resource after completing the basic Metadata and
assigning license rights. The resource is not published at this point. The Contributor is asked to indicate in the description
field whether the below QA checklist(s) have been applied.

Step 2 – Catalogue and Publish

The cataloguer completes the full metadata record and publishes the resource to NHS Shared Learning where it can be
retrieved by searching and browsing by users.

Communication between Contributor and Cataloguer

There are 2 instances when a contributor will receive an email alert from the cataloguer

1. A resource has been published to NHS Shared Learning

After the contributor has successfully completed the upload process, the Cataloguer will receive an email alert
indicating that a resource is available to be published. The cataloguer completes the full metadata record, checks the
validity of the resource as per the cataloguer checklist and publishes the resource to the portal. At this point the
contributor will receive an email alert stating that their resource has been published.

* A step by step guide to contributing resources to NHS Shared Learning can be found here

Page | 4
 2. A resource has been returned for clarification or the resource has been found unsuitable* for NHS Shared
Learning
When a cataloguer receives an email alert about a resource being uploaded and begin to complete the metadata
record, he/she might find that some important information about the resource is missing. In such a scenario the
contributor will receive an email alert with the appropriate query/ comment. There after the contributor will follow
the below steps to fix the issue:

• Contributor logs into Intralibrary to view the comments sent by the Cataloguer

• Contributor then adds an appropriate reply/ comment to the resource after the appropriate change has been
made to the resource. At this point the cataloguer will again be indicated with an email alert with the comment
added by the Contributor.

*
Please refer the Collection policy to view the Resource types and categories that may be contributed to NHS Shared Learning

Page | 5
 A flowchart illustrating the above process is given below

Contributor UPLOADS resource via Intralibrary

Contributor ADDS basic Metadata information

and License rights to complete the upload
process
Email alert to Cataloguer

Cataloguer RESERVES the resource within

Intralibrary
Email alert to Contributor

Email alert to Contributor

Cataloguer VALIDATES the resource as per
the cataloguer checklist

Resource No
COMPLIES
as per
checklist
Yes

Cataloguer completes full METADATA

PUBLISH to NHS Shared Learning

Page | 6
Who does what?
It is the responsibility of both the Contributor and Cataloguer to perform Quality assurance as per the resource workflow
mentioned above and the roles defined below. The checklists defined in the following pages have been designed to assist
contributors in determining the quality and effectiveness of learning objects. This evaluation is essential to ensure that
learning objects placed in NHS Shared Learning are of high quality.

Contributors (whether organization or individual)

1. Every contributor will apply the essential elements of the ‘Checklist for evaluating Learning Objects’ or ‘Checklist
for evaluating online courses’ before uploading resources to NSL.
2. Contributors are responsible for ensuring that both technical and educational quality assurance, and, where appropriate,
quality assurance from a clinical or subject perspective is carried out before uploading the resource to NSL.
3. Contributors are responsible for indicating in the description field of the metadata the nature of the quality assurance
carried out. Please refer ‘Contributor’s Quality Assurance Checklist for resources during the upload process within
Intralibrary’ for more details.
4. Boards have the option of using the Review Panel tool to carry out local peer review and consultation on e-Learning
resources during the development stage, including use of the QA checklists.
NOTE: If the contributor themselves are not qualified to perform the Quality Assurance, it is the responsibility of the
Contributor to ensure that it is done by an appropriately qualified evaluator before uploading it to NHS Shared Learning.

Cataloguer

1. Every Cataloguer will check to ensure that all essential items in the metadata record has been filled before publishing
the resource to NSL
2. The cataloguer will apply a metadata value within the catalogue record to indicate that the Quality Assurance process
has been applied to the resource. This information will then be apparent in the record as retrieved by searching and
browsing of NHS Shared Learning.
NOTE: If no evidence is provided by the contributor of carrying out the quality check then the cataloguer can request this
information from the Contributor.

Page | 7
Contributor’s Quality Assurance Checklist for evaluating learning
objects
The checklist given below will be applied by Contributors before uploading resources to NHS Shared Learning portal via the
Intralibrary tool.

The IEEE (2002) defines Learning Objects as “any entity, digital or non-digital, which can be used, re-used or referenced
during technology supported learning.”

Learning objects exist and interoperate at different levels of granularity. The simplest level is content, information or knowledge
object. This could be

 a simple text document,

 a photograph,
 a video clip,
 a two- or three- dimensional image
 A single topic(s) with a learning objective
 an online course (s)

Standards B – E may not be appropriate for simple files e.g. images.

Page | 8
Standard A: Content quality - The content is accurate and grammatically correct, and
the scope is sufficient for the intended use.

Criteria Meets criterion

Fully Partiall NA More information Comments
y needed
Essential criteria
1. Content is accurate
2. Content is fair and unbiased
3. The Learning Object is appropriate
for the scope it aims to cover.

Desirable criteria
4. Content is sequenced logically and
effectively
5. Learning experiences are sufficient in
number and scope to support the
targeted outcomes.
6. Content has been reviewed by a
subject-matter expert.
7. Content and learning activities are
clearly aligned with learning outcomes.

Page | 9
Standard B: Learning objective alignment - Learning goals and objectives are provided
to outline learning expectations and are applicable and relevant to the subject matter and
the audience.

Criteria Meets criterion

Fully Partially NA More information Comments
needed
Essential criteria
1. Learning objectives* are appropriate
to learning outcomes*.
2. Learning objectives are clearly
stated.

*
Learning objectives: Statements about what a student will gain from a course or activity. These are specific statements about exactly what a student should
know, be able to do, or value as a result of accomplishing a learning goal. Learning objectives form the basis for curriculum and course development as well as
testing.
*
Learning outcome: A concrete action that a student demonstrates as a result of learning. A learning outcome can be a demonstration of knowledge, a skill,
or a value. Generally, learning outcomes are assessed at the course and/or program levels.

Page | 10
Standard C: Feedback - Learners are provided with constructive, relevant and frequent
feedback based on their activities within the learning object.

Criteria Meets criterion

Fully Partiall NA More information Comments
y needed
Essential criteria
1. Feedback supports positive learning
outcomes.
2. Learners are provided with timely
responses and feedback when asked to
answer questions or provide
information.
3. Feedback compares learner
performance with the relevant criteria
and explains how performance can be
improved.
Desirable criteria
4. Feedback uses language that
encourages learners.

Standard D: Motivation - The learning environment is engaging, interactive and

relevant to the intended learner.

Criteria Meets criterion

Fully Partiall NA More information Comments
y needed
Essential criteria
1. The learning environment provides
appropriate and engaging learning
opportunities.

Page | 11
2. The learning object provides true-to-
life learning activities and interactivity
whenever possible.
3. The learning object defines realistic
expectations and standards for
success.
4. Learners are given adequate
directions and support to engage in the
learning object activities.

Standard E: Design and Usability

Criteria Meets criterion

Fully Partially NA More information Comments
needed
Essential criteria
1. Design is clear, consistent and
provides appropriate instructions for
navigation and interaction.
2. Design and interface is consistent
and predictable.
3. Navigation provides users with a way
to return to the start menu, navigate
within and exit from the learning
object.
4. If audio and video components are
present, user control is allowed.
5. Hyperlinks and buttons function
effectively.
Desirable criteria
6. Type and typeface are appropriate.
7. Images and graphics are used
appropriately.

Page | 12
 8. Colour is used appropriately
throughout.
9. Plain English is used as far as
possible.

Standard F: Accessibility The learning object provides accommodation for learners with
sensory and/or motor disabilities.

Criteria Meets criterion

Fully Partiall NA More information Comments
y needed
Essential criteria
1. Learning objects should comply with
W3C AA accessibility standards.

Desirable criteria
1. Provide information for learners to
change their default settings to make
the application accessible*

*
Refer www.saifscotland.org.uk for guidance on how to provide accessible information and how to ensure that hardcopy and electronic documents are
accessible to disabled people.

Page | 13
Standard G: Reusability and standards compliance - The learning object can be used
in varying learning contexts with learners from diverse backgrounds and supports
international standards and specifications.

Criteria Meets criterion

Fully Partiall NA More information Comments
y needed
Essential criteria
1. The learning object consists of one or
more self-contained learning units,
each addressing a single topic or
learning objective, and structured as a
standalone resource.
2. Hardware and software requirements
are defined.
3. Learning objects are constructed in
compliance with technical
interoperability standards allowing
sharing of content and assessments
among different learning management
systems / virtual learning
environments. Content should be
SCORM1 1.2 or IMS Content Packaging2
1.1.2 compliant at a minimum.
Assessments should be IMS QTI
conformant3.

1
SCORM, the Sharable Content Object Reference Model, is a technical specification that governs how online training (or "e-learning") is created and delivered
to learners. For more details visit http://www.adlnet.gov/about/index.aspx
2
The IMS Content Packaging Information Model describes data structures that are used to provide interoperability of Internet based content with content
creation tools, learning management systems (LMS), and run time environments. For more details visit http://www.imsglobal.org
3
The IMS Question & Test Interoperability (QTI) specification describes a data model for the representation of question (assessmentItem) and their
corresponding results reports. For more details visit http://www.imsglobal.org

Page | 14
Standard H: Intellectual property and copyright - The learning object metadata
address the rights of the owner and the conditions for use.

Criteria Meets criterion

Fully Partiall NA More information Comments
y needed
Essential criteria
1. Issues associated with copyright or
IPR of learning materials are clearly
addressed.

Desirable criteria
2. If the content is developed and
owned by the individual or organization
submitting the learning object, a
Creative Commons* license or similar is
attached.
3. All quoted materials are cited
correctly by adhering to one of the
standard citation formats.

*
Refer www.creativecommons.org for more details

Page | 15
Contributor’s Quality Assurance Checklist for evaluating online courses
Standard A: Content

Criteria Meets criterion

Fully Partially NA More information Comments
needed
Essential criteria
1. Course objectives are measurable
and state clearly what participants will
know or be able to do at end of the
course.
2. A clear, complete course overview
or syllabus is provided.
3. Objectives are matched to content
requirements and to the grade and
skill levels of the intended audience.
4. Course requirements (e.g.
timeframes, expectations for
communication, activities and
assignments, and assessments) are
consistent with course objectives and
are clearly stated.
5. Issues associated with copyright or
IPR of learning materials are clearly
addressed.
6. Data protection policies are clearly
stated where required – e.g. if course
requires user identification.
7. Assessment of learning is included
and answers and explanations are
available.
8. The course content is accurate,
current and free of bias. This includes
Equality and Diversity impact
assessment.
Desirable criteria
9. Generic information literacy and
communication skills are incorporated
as an integral part of the course.
10. Learning resources are made
available to help learners acquire
prerequisite knowledge and skills
before starting the course
11. Notes and resources for instructors
are included.

Standard B: Instructional design

Criteria Meets criterion

Fully Partiall NA More information Comments
y needed
Essential criteria
1. Course design reflects a clear
understanding of learner needs, and
incorporates varied ways to learn and
multiple levels of mastery (novice to
expert)
2. The course is organized into units
and lessons.
3. The course unit overview describes
the objectives, activities, and
resources within the unit.
4. Each lesson includes a lesson
overview, concepts and activities,
assignments and assessments,
providing multiple learning
opportunities for students to master
the content.
5. The course engages learners in
activities that address a variety of
learning styles and preferences.
6. Readability levels, written
assignments and numeracy
requirements are appropriate for the
course content and the learners.
Desirable criteria
7. The course is designed to teach
concepts and skills that learners will
retain over time - i.e. they are relevant
to the real world in which the learning
is to be applied. Meaningful and
authentic learning experiences are
provided that will help learners to
apply course concepts.
8. Course instruction includes activities
that engage learners in active learning.
9. Instruction provides learners with
alternative learning pathways to
master the content, based on learner
needs.
10. The course provides opportunities
for learners to engage in higher-order
thinking, including critical reasoning,
problem solving, developing mental
models, forming opinions.
11. Instructional design enables the
course to be adapted to accommodate
special learner needs – e.g. learners
with physical disabilities; low literacy
or numeracy skills.
12. Course design provides appropriate
opportunities for instructor-learner
interaction, including timely feedback
about student progress.
13. Learners have access to
information and learning materials that
enrich the course content.
14. Course offers opportunities for
students to give feedback on course
content

Standard C: Assessment

Criteria Meets criterion

Fully Partiall NA More information Comments
y needed
Essential criteria
1. Assessment strategies are consistent
with course objectives and are clearly
stated.
2. The course structure includes
adequate and appropriate methods and
procedures to assess learners’ mastery
of learning objectives.

Desirable criteria
3. Ongoing assessments are conducted
to verify the learner’s readiness for the
next lesson.
4. Assessment strategies make the
learner aware of his/her progress.
5. Assessment strategies are
sufficiently flexible to assess learners in
a variety of ways.
6. Grading schemes and models are
available to tutors assessing learners.

Standard D: Technology

Criteria Meets criterion

Fully Partially NA More information Comments
needed
Essential criteria
1. Course architecture permits tutors
to add content, activities and
assessments to extend learning
opportunities.
2. The course is constructed in
compliance with technical
interoperability standards allowing
sharing of content and assessments
among different learning management
systems / virtual learning
environments. Content should be
SCORM 1.2 or IMS Content Packaging
1.1.2 compliant at a minimum.
Assessments should be IMS QTI
conformant.
3. Course adheres to W3C AA
accessibility standards.
4. Hardware, web browser and
software requirements are specified.
5. Prerequisite technology skills are
explicitly stated.
Desirable criteria
1. The course is easy to navigate.
2. The course provider offers
 assistance with technical and course
management.

Contributor’s Quality Assurance Checklist for resources during the

upload process within Intralibrary

These checklists applies only when the contributor uploads the resource* to NHS Shared Learning portal and adds
metadata via the Intralibrary system

1. Check if the resource uploaded through Intralibrary has an appropriate title, description and keywords

2. Check if the description field indicates what QA process has been applied. This should include the checklists presented in
this document (which cover technical and educational QA) plus any quality assurance conducted from a subject/clinical
perspective. E.g., you can state in the description, this resource has been Educationally/ Technically Quality assured.

3. Check if the description field indicates what KSF mapping applies to the resource, if applicable

4. Check if you have indicated within the description field when the resource was created or published, if known. If the
exact date is unknown you can also state the year of creation/publication

5. Check if you have indicated the author and the publisher of the resource if the you yourself/organization has not
authored or published the resource

6. Check if you have selected the appropriate License rights associated with the resource while uploading the resource

*
A step by step guide to Contributing resources to NHS Shared Learning can be found here
How can Review Panel help in Peer review?
Review Panel is a space where you can share your work with others and invite everyone involved in your project to make
contributions and have their say.

It offers you a collaborative online forum where anybody with a vested interest in your work can assess and review it quickly
and easily. Reviewers can help to guide the development of your project.

You can track, preview and feedback on:

• PDFs
• Multimedia
• Websites
• e-Learning materials
• Training courses
• Word documents
• And much more!

Here are the basic steps involved to invite reviewers to your project
1. Login Review Panel using your ATHENS details

2. Load the project you would like to be reviewed

i. You can also create a new project that will allow you to upload a totally new project

3. Select the potential reviewers by ticking the check boxes next to their name.
i. Note: Your reviewer will have to login once for you to be able to see their names
ii. An email will be sent to your reviewers when you select/add them to the project.

*
A step by step guide to Reviewing resources online can be found here
How to add comments and suggestions?
1. Load up the project you wish to review by clicking on its name.
2. Select the item you wish to comment on by clicking its name.
3. Type your comment into the space provided on left hand side and click Add.
4. Your comment will appear along with your name and the correct date.

Your projects screen will let you know how many reviewers have been assigned to your project and how many
comments have been posted.

The reviewer can then indicate via email of telephone that the review has been completed for the project owner to
analyze the comments and address them.