Educational Innovation in Economics and Business Pedagogy, Technology and Innovation

Educational Innovation in Economics and Business VIII
Educational Innovation in Economics and Business

Volume 8
The titles published in this series are listed at the end of this volume.
Educational Innovation in
Economics and Business
VIII
Pedagogy, Technology and Innovation
Edited by
Roger Ottewill
Centre for Learning and Teaching,
University of Southampton, U.K.
Liz Borredon
EDHEC Business School,
Lille, France
Laurent Falque
EDHEC Business School,
Lille-Nice, France
Bruce Macfarlane
Educational Development Centre,
City University, London, U.K.
and
Ann Wall
School of Business and Finance,
Sheffield Hallam University, U.K.
SPRINGER-SCIENCE+BUSINESS MEDIA, B.V.

A C.I.P. Catalogue record for this book is available from the Library of Congress.
ISBN 978-90-481-6505-6 ISBN 978-94-017-1386-3 (eBook)

DOI 10.1007/978-94-017-1386-3
Printed on acid-free paper
All Rights Reserved

© 2004 Springer Science+Business Media Dordrecht
Originally published by Kluwer Academic Publishers in 2004
Softcover reprint of the hardcover 1st edition 2004
No part of this work may be reproduced, stored in a retrieval system, or transmitted
in any form or by any means, electronic, mechanical, photocopying, microfilming, recording
or otherwise, without written permission from the Publisher, with the exception
of any material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work.
Contents
Contributors ix
Acknowledgements xiii
Editors xv
Preface xvii
Part I: The Business Context 1
How E-Learning Businesses Meet Client and End User Needs:

Analysing the Collaborative Contexts 3
DAVID RUSSELL, DAVID CALVEY & MARK BANKS
Business Process, Experience and Memory: Educational Approaches

and Technology Tools for a Global Workforce 19
DANIEL M. CARCHIDI & DAVID J. MCCARTHY
v
vi Contents
Part II: Challenges for the Academy 41
From High Level Clarity to Ground Level Confusion: Exactly Where

do IT-Mediated Education Policies Fit? 43
STEPHEN D. REEVE & STEPHEN H. FLOWERS
Moving a University Toward On-line Learning: Opportunities,

Challenges, and Technologies 61
DREW PARKER & ANDREW GEMINO
Linking Pedagogical Innovation and Information Technology to

Enhance Business Education 77
SERGIO VASQUEZ BRONFMAN
The Experience of Self-Organized Learning Through the Use of

Learning Plans for Knowledge Management 93
VIVIEN LEE LOOI CHNG & STEVEN COOMBS
Part III: Team and Collaborative Learning 111
Using Student Consulting Team Assignments as a Vehicle to Teach a

Systems Development Course 113
SYLNOVIE MERCHANT
Using Team Learning in the Classroom: Experiences and Lessons 129

LEROY F. CHRIST, MARY Y. CHRIST, A. STEVEN GRAHAM,
MICHAEL K. MCCUDDY & WENDY L. PIRIE
Using Teams in the Classroom: Meeting the Challenge of Evaluating

Students’ Work 147
MICHAEL K. MCCUDDY & WENDY L. PIRIE
International Management: Early Experience in Multicultural Virtual

Team Interaction 161
KEN MORSE
Contents vii
Part IV : Problem Based Learning 173
PRAXIS: A Practice-Based Instructional System in the First Year of

an Industrial Engineering Program 175
WILLEM M. VAN WOERDEN & NYNKE JO SMIT
Using the Web for Problem-Based Learning 189

NYNKE JO SMIT, MAARTEN VAN RIEMSDIJK & JAN VAN DER VEEN
Effects of Problem-Based Learning in Business Education: A

Comparison Between a PBL and a Conventional Educational
Approach 205
PIET VAN DEN BOSSCHE, MIEN SEGERS, DAVID GIJBELS &
FILIP DOCHY
Learning about Teaching Information Systems in a Problem-Based

Curriculum: An Exploratory Study of the Impact of Students’
Individual Differences on their Conception and Perception of
Problem Tasks 229
JAN NIJHUIS, MIEN SEGERS & WIM GIJSELAERS
Part V : Distance and On Line Learning 253
The Use of a Virtual Learning Environment to Support Learners

on Work-Based Learning Programs 255
LEN BIRD
Distance Learning: The Experience of Accounting at the University

of Natal (Durban), South Africa 273
ANTHONY B. LUMBY & ADRIAN D. SAVILLE
Creating and Improving a “Virtual Object” Through Web-Mediated

Discourse 289
GORDON WELLS
Testing Social Information Processing Theories in Distance

Education 315
WM. BENJAMIN MARTZ, JR.& MORGAN M. SHEPHERD
viii Contents
PART VI : Specific Applications of Learning Technology 331
Electronic Tutorial for Moral Reasoning in Business Education:

A Technological Learning Tool to Facilitate Students’
Moral Reasoning 333
MARGARITA ALEMÁN VARGAS
The Five Key Benefits of On-line Final Examinations

(with Three Free Bonus Benefits) 347
JEREMY B. WILLIAMS
Assertion-Reason Assessment in Formative and Summative Tests:

Results from Two Graduate Case Studies 359
LUKE B. CONNELLY
Exploring the Implications of Videoconferencing for Management

Learning 379
VIVIEN HODGSON & MIREIA ASENSIO
Index 393
Contributors
Margarita Aleman Vargas, Faculty Member of Bilingual High

School, The Monterrey Institute of Technology Campus, Guadalajara,
México. maleman@itesm.mx
Mireia Asensio Department of Management Learning, The School

of Management, Lancaster University, UK. m.asensio@lancaster.ac.uk
Mark Banks, Manchester Institute for Popular Culture,

Manchester Metropolitan University, M15 6LL, UK.
m.o.banks@mmu.ac.uk
Len Bird, Work-based Learning Unit, Coventry Business School,

Coventry University, Priory Street, Coventry. CV1 5FB, UK.
l.bird@coventry.ac.uk
David Calvey, Centre for Employment Research, Manchester

Institute for Telematics and Employment Research, Manchester
Metropolitan University, M15 6LL, UK. d.calvey@mmu.ac.uk
Daniel Carchidi, Knowledge Systems GE Capital, Performance

Technology Solutions, USA. daniel.carchidi@gecapital.com
Leroy Christ, College of Business Administration, Valparaiso

University, Indiana, USA. leroy.christ@valpo.edu
ix
x Contributors
Mary Christ, College of Business Administration, Valparaiso

University, Indiana, USA. mary.christ@valpo.edu
Luke Connelly, Brisbane Graduate School of Business,

Queensland University of Technology, 2 George Street, Brisbane, Q
4001, Australia. l.connelly@qut.edu.au
Steven Coombs, Department of Curriculum Studies & Secondary

Education, School of Education, Sonoma State University, 1801 East
Cotati Avenue, Rohnert Park Ca. 94928-3609, USA.
steven.coombs@sonoma.edu
Filip Dochy. Educational Innovation and Information Technology

(EDIT), Faculty of Law, University of Maastricht, the Netherlands.
Filip.dochy@edit.unimaas.nl
Stephen Flowers, University of Brighton Business School, UK.

shf@bton.ac.uk
Andrew Gemino, Faculty of Business Administration, Simon

Fraser University, 8888 University Drive, Burnaby, British Columbia,
V5A 1S6, Canada. gemino@sfu.ca
David Gijbels, Educational Innovation and Information

Technology (EDIT), Faculty of Law, University of Maastricht, the
Netherlands. David.gijbels@edit.unimaas.nl
Wim Gijselaers, Department of Educational Development and

Educational Research, University of Maastricht, PO Box 616, 6200
MD, Maastricht, the Netherlands. w.gijselaers@educ.unimaas.nl
Steven Graham, Purdue University North Central, Indiana, USA.

stevenvalpo@hotmail.com
Vivien Hodgson, Department of Management Learning, The

School of Management, Lancaster University, UK.
v.hodgson@lancaster.ac.uk
Vivien Lee Looi Chng, Temasek Polytechnic 21 Temasek Avenue 1

Singapore 529 757. looichng@tp.edu.sg
Contributors xi
Anthony Lumby, Faculty of Management Studies, University of

Natal (Durban), South Africa. lumby@nu.ac.za
Wm Benjamin Martz, Information Systems Department, College of

Business, University of Colorado at Colorado Springs, USA.
wmartz@uccs.edu
Sylnovie Merchant Department of Management, California State

University, Sacramento, California, USA. smerchant@csus.edu
David McCarthy, Run Fast Inc, USA. djmrunfast@aol.com
Michael McCuddy, College of Business Administration,

Valparaiso University, Indiana, USA. mike.mccuddy@valpo.edu
Kenneth Morse, Department of Marketing & International

Management, Waikato Management School, University of Waikato,
Private Bag 3105, Hamilton, New Zealand.
k.morse@mngt.waikato.ac.nz
Jan Nijhuis, Department of Management Science, Faculty of

Economics and Business Administration, University of Maastricht, PO
Box 616, 6200 MD Maastricht, the Netherlands.
j.nijhuis@mw.unimaas.nl
Drew Parker, Faculty of Business Administration, Simon Fraser

University, 8888 University Drive, Burnaby, British Columbia, V5A
1S6, Canada. drew@sfu.ca
Wendy Pirie, College of Business Administration, Valparaiso

University, Indiana, USA. wendy.pirie@valpo.edu
Stephen Reeve, University of Brighton Business School, UK.

s.d.reeve@bton.ac.uk
Maarten van Riemsdijk, Faculty of Technology & Management,

University of Twente, Enschede, the Netherlands.
m.j.vanriemsdijk@sms.utwente.nl
David Russell, Centre for Employment Research, Manchester

Institute for Telematics and Employment Research, Manchester
Metropolitan University, M15 6LL, UK. d.russell@mmu.ac.uk
xii Contributors
Adrian Saville, School of Economics and Management, University

of Natal (Durban), South Africa. saville@nu.ac.za
Mien Segers, Department of Educational Development and

Research, Faculty of Economics and Business Administration,
University of Maastricht, PO Box 616, 6200 MD Maastricht, the
Netherlands. m.segers@educ.unimaas.nl
Morgan Shepherd, Information Systems Department, College of

Business, University of Colorado at Colorado Springs, USA.
mshepher@ uccs.edu
Nynke Smit, Faculty of Technology & Management, University of

Twente, Enschede, the Netherlands, n.j.smit@sms.utwente.nl
Piet van den Bossche, Department of Educational Research and

Development, Faculty of Economics and Business Administration,
University of Maastricht, the Netherlands.
piet.vandenbossche@educ.unimaas.nl
Jan van der Veen, University of Twente, Enschede, the

Netherlands. j.t.vanderveen@dinkel.utwente.nl
Sergio Vasquez Bronfman, ESCP-EAP, 79 Avenue de la

République; 75011 Paris; France. vasquez@escp-eap.net
Gordon Wells, Dept. of Education, University of California, 1156

High St.,Santa Cruz, CA 95064, USA. gwells@cats.ucsc.edu
Jeremy Williams, Brisbane Graduate School of Business,

Queensland University of Technology, GPO Box 2434, Brisbane,
QLD 4001, Australia. jb.williams@qut.edu.au
Willem van Woerden, Faculty of Technology & Management,

University of Twente, Enschede, the Netherlands.
w.m.vanwoerden@sms.utwente.nl
Acknowledgements
It would not have been possible to produce this book without support and
help from a wide variety of sources. First, we are very grateful to all those
who attended the EDiNEB VIII conference in Nice and submitted papers for
our consideration. While this presented us with many challenges in drawing
up a short list of those to include, it clearly demonstrated the high degree of
interest in, and the wealth of experience of, educational innovation amongst
the membership of the EDiNEB network. Second, thanks are due to staff of
EDHEC School of Management who hosted and helped to organize such a
successful conference in a very attractive location. Last, but by no means
least, we greatly appreciate the help received from EDiNEB personnel. We
particularly wish to acknowledge the contribution of Bob Janssen Steenberg
and Henny Dankers who coped admirably with the very difficult task of
getting the book into a camera-ready format.
Roger Ottewill
Liz Borredon
Laurent Falque
Bruce Macfarlane
Ann Wall
xiii
The Editors
Roger Ottewill (BSc Econ, University of London) currently combines a

part-time research post in the Centre for Learning and Teaching (CLT) at the
University of Southampton with a very active semi-retirement. Prior to
joining the CLT on 1st October 2001, he was employed as a lecturer for 28
years by Sheffield Hallam University. During this period he supported
learners on a wide variety of courses at many levels from sub-degree to
postgraduate. His principal subject areas were in the fields of public
administration and business and management. From the early 1990s he
became increasingly involved in education research and development and
played a key role in projects relating to aspects of resource based learning,
vocational education, course evaluation, language learning and cross-cultural
skill development. These resulted in conference papers, including
contributions to two papers included in earlier volumes of this series; articles
in a wide variety of academic journals; and co-editorship (with Bruce
Macfarlane) of a book Effective Learning and Teaching in Business and
Management, published by Kogan Page in 2001. As a member of the CLT
he is helping to enhance the quality of learning, teaching and assessment by
promoting good practice and fostering innovation. He retains his links with
Sheffield Hallam University through a Visiting Fellowship with the School
of Business and Finance.
Liz Borredon (MA in Management Education by Research, Lancaster
University) is a Professor in the Department of Management and Strategy
and Deputy Head of Centre for Languages and International
Communication at the EDHEC Business School, Lille, France. She is on the
Board of Directors of the European Mentoring and Coaching Centre and on
the Mentoring Committee of the Academy of Management (AoM) USA.
xv
xvi Editors
Her research is in mentoring, dialogue and organisational learning. Her

present studies focus on the role of mentor within collaborative learning and
knowledge creation processes. She has published numerous articles and
regularly contributes to AoM annual meetings.
Laurent Falque (PhD, University of Tours) is a Professor in Human
Resource Management in the EDHEC School of Management, Lille.
Previously, he worked as human resources manager for 11 years. At EDHEC
he introduced problem-based learning in 1997. He is a member of the
EDiNEB board. During his studies for his PhD he moved gradually into the
field of decision-making research. Currently, he is focusing on discernment
and deliberation in decision making and acting as a coach.
Bruce Macfarlane (PhD, University of London) is Reader in Higher
Education at City University, London. Before joining City University in
October, 2000 he worked for 13 years as a business and management
lecturer in higher education. His research interests incorporate values in
higher education and the pedagogy of business and management education.
He recently co-edited (with Roger Ottewill) the first book in a new Institute
for Learning and Teaching/Kogan Page subject series entitled Effective
Learning and Teaching in Business and Management (2001). His
professional activities incorporate work as an accreditor for the Institute for
Learning and Teaching in Higher Education, membership of the editorial
boards of Teaching in Higher Education; the Journal of Management
Development; and the Journal of Business Ethics Education and the
organisation of conferences on the teaching of business ethics in
collaboration with the European Business Ethics Network (UK). Currently,
he is writing a book on the ethics of teaching in higher education.
Ann Wall (BA, University of Nottingham) is currently a Senior Lecturer
at Sheffield Hallam University in the School of Business and Finance. For
more than twenty years she has been involved with students on a variety of
courses at a variety of levels, from sub-degree to postgraduate. Her principal
subject areas are public and social administration, particularly health care
policy. Since the early 1990s she has also been interested in education
research and development and has contributed to several projects relating to
aspects of resource based learning, vocational education and course
evaluation. She has presented papers at three EDiNEB conferences, one of
which was included in an earlier volume in this series. Other publications
include four textbooks on the NHS and community health services; and a
number of contributions to pedagogic journals such as, Assessment and
Evaluation in Higher Education, Journal of Vocational Education and
Training, Quality Assurance in Education and Education and Training. She
has also written case studies on general practice for use in GP training and
open learning materials for the Open University.
Preface
As business and management educators seek to respond positively to the

ever-changing environment within which they practice, so there is an
increasing need to keep abreast of developments in pedagogy and
technology. Without an appreciation of current pedagogic and/or
technological thinking it is unlikely that innovations in educational practice
will be as robust as might otherwise be the case. Thus, it is incumbent on
educators to ensure that in seeking to improve the learning experience of
their students they are prepared to be reflective and evidence-driven in their
approach. The papers in this edited volume illustrate many different aspects
of such a process. They have been contributed by educators from a variety of
countries thereby providing a truly international perspective on pedagogy,
technology and innovation.
Some of the papers focus on the broader institutional and corporate
context, others on more specific aspects of pedagogy and approaches to
learning, teaching and assessment. In a number there is a strong research
focus with both qualitative and quantitative methodologies being
represented. Others are essentially evaluative reports of a particular
innovation based on the personal reflections and practice-based analysis of
those involved. The papers are also illustrative of the range and variety of
subject areas in which EDiNEB members have an interest, from economics
to business ethics and from knowledge management to accountancy.
Likewise, there are differences relating to level and context. Some relate to
undergraduate education others to postgraduate. Many are concerned with
campus-based provision, but distance education and learning in the
workplace are also covered. Notwithstanding this diversity, what all the
papers have in common is that the underlying motivation for what they
xvii
xviii Preface
report is an enhancement of the quality of the learning environment for those

seeking a deeper knowledge and understanding of the world of business,
management, economics and finance.
Naturally, within this collection there is a strong emphasis on the
application of learning technology but this is not at the expense of non-
technologically informed aspects of pedagogy. Indeed one of the important
messages is that technology should be seen, at least in an educational
context, as a means to an end and not as an end in itself. That end, as
indicated earlier, should always be the enrichment of the student learning
experience and/or an extension of the range of learning outcomes being
pursued.
In organising the papers selected for inclusion in the book, the editors
have “worked with the grain” of what was submitted for consideration by
contributors to the eighth EDiNEB international conference held in Nice
rather than a pre-determined plan. What has emerged is a structure in which
the earlier papers tend to deal with macro-level issues, reflecting business
and institution-wide change, and the later papers with more micro-level
topics, focusing on specific teaching contexts and strategies.
To assist readers the book has been divided into 6 parts. However, these
should not be regarded as watertight compartments since a number of papers
deal with a variety of topics and could easily be allocated to more than one
part of the book. That said, as far as possible, papers have been grouped
according to their major theme.
Part I: The Business Context

The two papers which make up this part of the book both, in their
different ways, illustrate developments within the business world. One by
Russell, Calvey and Banks focuses on e-learning businesses that have
emerged to meet particular training needs of the business community.
Depending on perspective, these can be seen as either complementing or
competing with traditional academic providers, such as universities and
colleges. The other paper by McCarthy and Carchidi considers how a large
conglomerate, General Electric, is using technology to meet the development
needs of its staff. Together, these papers are a reminder to those in academic
institutions that for business related subjects, at least, the educational
community extends well beyond the hallowed walls of academe.
Additionally, they indicate some lessons and potential challenges for
colleagues working in universities and colleges.
Preface xix
Part II: Challenges for the Academy

This part of the book includes papers that deal with challenges facing
those working in an academic environment. All have a particular resonance
for business and management educators and those in related disciplines.
Here, however, the focus is more inward looking. Most of the challenges
relate to the learning environment, in particular technological enhancement,
and to the needs of educators and of learners in this respect.
Two papers by Reeve and Flowers and Parker and Gemino adopt a sector
level and an institutional level perspective respectively. Reeve and Flowers
raise some very pertinent questions regarding the way forward for IT
mediated learning in higher education. While most of their empirical
material relates to the UK, it does have a worldwide significance. Arguably
Parker and Gemino are more positive but still recognise the difficulties
involved in “moving a university towards on-line learning”. Again, while
this is a case study of one Canadian university, the lessons are universal.
The other three papers concentrate on challenges that relate more directly
to students and their learning. Vasquz Bronfman, writing from a French
perspective, seeks to address the question of how to “link information
technology and active learning methods in order to add value to business
education”. Drawing upon evidence from three case studies he reaches the
conclusion that in exploiting the potential of information technology, it is
important to avoid the dangers of technocentrism and put pedagogical
considerations first.
While the World Wide Web represents a significant resource for learning
one of the key issues facing educators is how to get students to use the
information it provides in an informed and critical way. Lee and Coombs
address this challenge arguing that self-organized learning is an important
way to overcome the “‘cut and paste’ mentality”. Using economic concepts
and principles as examples, they recommend learning plans as a way of
building the competence of students in this respect.
Part III: Team and Collaborative Learning

In this part of the book various aspects of one type of response to some of
the developments/challenges highlighted in Parts I and II are explored. As
the title indicates, this involves the use of teamwork to enrich the learning
experiences of students.
Marchant’s contribution is intended as a guide to those wishing to
improve teaching effectiveness and students learning within the
xx Preface
Management Information Science/Systems disciplines. In the case described,

student-consulting teams were used to prepare students for the realities of
designing systems for actual organisations. The success of the programme
explained lies in the course reflecting what systems analysts do, together
with the tools and techniques used in this discipline. A complementary
strength of the courses is learning about team management and team
dynamics.
Pirie, Christ and colleagues encourage faculty to develop interactive team
learning situations. They explain how courses conducted at Valparaiso and
Purdue Universities provide “exceptional learning opportunities”. Students’
subject mastery was enhanced and, in addition, their time management,
study skills and behavioural competencies were developed as a result of
learning collaboratively. Their analysis is complemented by that of
McCuddy and Pirie, who demonstrate how peer assessment enhances
learning and, within this process, the need to focus on concrete behaviour
rather than abstractions. The authors are adamant about the faculty guiding
students in developing capacities in this area.
Morse focuses on the successes and limitations of virtual team exercises
in multicultural environments. In terms of success, he highlights how a
diverse group from different locations can be brought together without
significant expense enhancing the participants’ perception of the realistic
nature of their education and, in the process, creating a flexible virtual
learning space that facilitates communication. The difficulties are primarily
technological, exacerbated by reliance on a single language (English); by
cultural behaviour patterns which differ with regard to ease and openness in
seeking assistance; and by the amount of time participants spend on
familiarising themselves with personal isolation embodied in electronic
communication.
Part IV: Problem-Based Learning (PBL)

One pedagogic innovation that has had a considerable impact in the
Netherlands and with which EDiNEB has been particularly associated is
PBL. Thus, not surprisingly, the four papers, which comprise this part of the
book, all have Dutch authors. Each deals with different aspects of this very
influential approach in contemporary business education.
Learners in the field of business, management and economics are often
attracted by the prospect of a qualification that will lead directly to material
success. For educators working with students impatient to see the relevance
of learning to a practical context this can represent a significant motivational
challenge. Prompted by concerns about the enthusiasm and success rate of
their students, Van Woerden and Smit contend that a highly structured
Preface xxi
approach to problem-based group learning is a way of motivating students to

work harder and improving their academic success. It is also, they argue, a
means of easing the difficulties students experience in adapting to the
demands of university education.
Smit and colleagues demonstrate how the introduction of web support
has stimulated the use of theoretical material in case study exercises. They
also explain how on-campus students have gained “added value” from
having a shared electronic workspace facility.
In their paper, a group from the University of Maastricht (Van den
Bossche, Segers, Gijbels and Dochy) compare PBL with a conventional
educational approach. Using a quasi-experimental research methodology
their results suggest that students learning in a problem based environment
score perform better than their counterparts in a conventional environment
with respect to knowledge acquisition and case studies. However, they found
no statistically significant difference for the application of knowledge or
skill development. They argue that the results confirm PBL as a “powerful
learning environment”.
The paper by Nijhuis and colleagues reports the results of research into a
problem based, information systems course. The findings suggest that
motivation and guidance are important task elements regardless of the
preferred learning style or personality of students and that this should be
taken into account by tutors when designing problem based tasks. However,
what is also needed is similar research in other subject areas in order to
contain further insights into the relevance of course content in the design of
tasks.
Part V: Distance and On-Line Learning

This group of papers is particularly international in composition
including contributions from the UK, South Africa and the USA. They also
carry a clear and optimistic message. Distance and on-line learning can, if
used thoughtfully and in conjunction with face-to-face learning, not only
enhance student learning but can compensate for some of the shortcomings
of traditional methods.
Bird describes how a virtual learning environment was used to create a
“community of practice” for work-based students that successfully combined
the university, workplace and wider professional community. A distance
learning programme for accountants, described by Lumby and Saville,
increased the participation of part time, non- white students in accountancy
education. Wells explores the use of the Web Knowledge Forum to generate
discourse amongst student and teachers in between their class meetings. In
the final paper, by Martz and Shepherd, a comparison is made between
xxii Preface
distance and on-campus students, in particular the affiliation needs of

students and how these can be met in a distance learning context.
Part VI: Specific Applications of Learning

Technology
In the final part of the book, a number of very specific applications of
learning technology are analysed/or evaluated.
Aleman Vargas’ contribution deals with a very important, although
sometimes sadly neglected, aspect of business education, namely business
ethics. She demonstrates how an electronic tutorial allows students to
develop and practice moral reasoning in their own time and at their own
pace, released from the “threats” which sometimes inhibit students in the
classroom. Although the setting is the Monterrey Institute of Technology in
Mexico, the innovation could well inspire developments elsewhere.
The authors of the next two papers, Williams and Connelly, are from the
Queensland Institute of Technology in Australia. Their particular interest is
the role that technology can play in student assessment. Williams agues that
appropriately designed and implemented on-line examinations can present
educators and students with a number of important benefits including a
reduction in cheating and stress and an enhancement in the quality of
learning. Connelly sees on-line assessment as facilitating formative, as well
as summative, assessment. This is particularly important given the increasing
pressure on the time available for tutors to support learning by providing
students with opportunities to practice and obtain constructive feedback.
One very specialised type of technology, videoconferencing, and its
application in educational settings is the theme of a final paper by Hodgson
and Ascensio. Perceived problems with videoconferencing, the authors
claim, are more to do with the way the learning event has been designed than
with the shortcomings of the media itself. Two aspects are explained as
critical: the first is “social presence”, the second the degree of student
“engagement”. Hodgson and Asensio provide a framework for
understanding learners’ needs and programme design components for
entering a new era of programme delivery.
It is anticipated that this collection of papers will inspire and encourage
others to innovate and will contribute to the forging of partnerships across
national boundaries. There is much to be gained from international
collaboration in those spheres of education of particular concern to EDiNEB
members and, indeed, all involved in business and economics education
worldwide. Internationalisation in the business world needs to be matched by
developments in the academic world. Technology is eroding old divisions,
such as those between the academic and corporate worlds, thereby
Preface xxiii
contributing to the globalisation of business knowledge and educational

processes. Thus, it is vital for educators to keep abreast of what is
happening. Clearly, in this respect, EDiNEB has played and will continue to
play an increasingly important role.
PART I
THE BUSINESS CONTEXT

How E-Learning Businesses Meet Client and End User
Needs: Analysing the Collaborative Contexts
David Russell1, David Calvey2 & Mark Banks2

1
Centre for Employment Research, Manchester Institute for Telematics and Employment
Research, Manchester Metropolitan Unversity, UK; 2Department of Sociology, Manchester
Metropolitan University, UK
1. INTRODUCTION
This paper reports on research undertaken into the emerging e-learning

industry in the North West of England. It is based on findings from SMILE
(Skills for the Missing Industry’s Leaders and Enterprises), a research
project part sponsored by the ESF/Adapt-University for Industry. The
research has assessed and evaluated management skills within small and
medium sized enterprises (SMEs), specifically those producing digital media
based educational and training materials, hereafter known as e-learning
products, a fast growing and economically important sub-sector which has
emerged to meet increased demand for multimedia based learning and
training materials in educational and workplace environments. Previous
research has argued that creative management skills may be lacking in the
UK industry context – potentially leading to job losses and thus a “missing
industry” (Speake & Powell, 1997). Our research has aimed to identify just
how, and how far, managers in such firms can ensure that the creators of
education and training materials are able to meet demands for new and
innovative products, that not only utilize the best of convergent technologies,
but can respond to consumer demands for both bespoke and generic learning
products. The chapter examines the ways in which SMEs that provide e-
learning products collaborate and communicate with their clients, customers
and/or end users.
3
4 David Russell, David Calvey & Mark Banks
Our premise is this: given the increased demands for more sophisticated
and “learning centered” products, it is becoming increasingly crucial for
firms to source and exploit content, education, knowledge and expertise that
is external to the traditional boundaries of the firm. This predominantly
occurs in three ways. Firstly, there is the sourcing of content from the client
– material that can be shaped into or inform the e-learning product in
question. Secondly, and increasingly, there is the need to draw advice from
external learning experts. It is now necessary for firms to open up their
organization to exterior knowledge to create new collaborations that can
position learning in the production process. Thirdly, given the necessity of
providing effective learning that is sensitive to end user needs, it is now
more common to involve or conceptualize the end user within the
development process. Ensuring that learning products are able to engage
with and enhance the learning of the end user involves more consultation,
partnership and interaction with the learner than ever before. These shifts
raise a set of problems related to how firms can effectively interact,
exchange and collaborate with external agencies in order to create, distribute
and evolve effective learning tools and products.
Using case study examples from our qualitative research of over 20
companies, we show how firms are attempting to expand (and expound) new
“learning communities” in order to effect progressive e-learning products.
The phrase “learning community” we use to describe the interactions
between the collection of “communities of practice” integral to the firm, and
the range of external experts, clients and end-users implicated in the creation
of an e-learning product. We feel that given the need for flexibility and
creativity in this sub-sector (Swanson & Wise, 1997), the more firms can
exploit or integrate external expertise, client creativity and learners’
knowledge and viewpoints, the more effective these learning communities
and their products will be. However as we will reveal, while some successes
have been identified, the strategies and pathways adopted in forming these
new communities are often partial and uncertain. We conclude by observing
that firms in the digital education and training sector need to more fully
conceptualize and engage with the possibilities of expanding their learning
communities to ensure the continued production of innovative e-learning
products.
2. WHY LEARNING IS CRUCIAL
It is expected that the number of SMEs in the e-learning field will grow,
yet businesses vary in the extent to which they possess detailed knowledge
or interest in learning. We have identified a number of advertising,
E-Learning Businesses 5
management training and publishing companies now adopting e-learning as

a new delivery channel but many lack a grounding or understanding, or
willingness to engage with, the philosophy or practices of learning. While
some see issues of learning as central, amongst the majority of firms we
studied, learning is often a secondary consideration to the provision of a
“technical solution” or “design impact”. The lack of recognition of the
specific ways in which firms can proactively shape or deliver education and
learning activities, for varied constituencies of end users, was a recurrent
finding of our research. Firms were either largely content to leave the
learning input to the providers of content (e.g. academic departments,
corporate trainers) or to assume that learning would proceed
unproblematically at the point of delivery (in schools, workplace etc.). But
we argue that the demand for more effective bespoke and tailored learning
products, now demands that firms pay more attention to learning design,
processes and outcomes. Increasingly, learning about learning is good
business sense.
Learning is necessary but, we argue, cannot be contained within the
conventional understanding of the firm as a bounded set of communities of
practice. A more open ended conceptualization is required, we feel, in order
to capture the layers of communities that must inter-relate to produce quality
e-learning materials. We want to show how extending communities of
practice into learning communities is one route that firms can follow to
enhance the quality of e-learning products and help create of a more open
and reflexive attitude to learning within the firm itself.
3. FIRMS AS COMMUNITIES OF PRACTICE
The firm is often seen as a key organization that can house the expertise,
skills and knowledge necessary for efficient and effective e-learning
production. Fransman’s (1994) conception of the firm as a “processor of
knowledge” (see Amin, 2000) is perhaps an apposite description of how
digital media firms operate. The firms we have been involved with are
processing knowledge for their own uses but they also trade in knowledge.
They draw upon the knowledge of in-house experts and build this knowledge
into products. They comprise sections, each with responsibility for, or
claiming to own, part of the design and production process.
With these issues in mind, in recent years it has become common to refer
to firms as a composite or collection of different communities of practice.
The idea of a community of practice was developed by Lave and Wenger
(1990) as a theory for practice-based learning in which one could undertake
“legitimate participation”, to serve a kind of apprenticeship with a group of
insiders in an organization. The theory was referred to by Brown and Duguid

(1991) to support their contention that the separation of knowledge from
practice is unsound. They argued that the ways in which people actually
work differ from official descriptions and that “learning in working” is a
better way to improve performance and most characteristic of how firms
share and develop knowledge. Wenger and Snyder (2000) later described a
community of practice as a “group of people informally bound together by
shared expertise and passion for a joint enterprise”, with members inevitably
sharing knowledge in order to solve problems in their organization. Yet,
while these communities might be informal and resistant to supervision, they
cannot exist without management support and structure. They are bounded,
to varying degrees, within the bureaucratic organization of the firm.
Given the ways in which communities of practice are often associated
with firms working in the creative or cultural industries (Raffo, et al., 2000),
we felt bound to test out how far our digital media firms were acting as
communities of practice in the production of e-learning products. However,
while the theory could be observed in action, at a number of levels, we
began to stumble over some key questions.
• Can communities of practice operate across different organizations/
groups?
• Can they operate without geographical proximity?
• How far does the involvement of the external expert, client or learner
constitute enhancement, supervision or interference in the community of
practice?
We became concerned with how far agencies and forces external to the
organization – whether in conflict, co-operation or collaboration with the
firm – are able to penetrate or impact upon Wenger’s (1998) “locally
negotiated regime of competence” and “shared histories of learning” that
make up the community of practice (Amin, 2000). We observed that trading
in knowledge and putting in the “learning” involves several complex
processes of negotiation, particularly around companies that produce
speculative and bespoke products. The firm negotiates with clients who
commission products, with the experts who are selling their knowledge, with
the end users and with the firm’s own staff who design and produce the
learning products. The types of exchanges and relative bargaining positions
of the parties to these negotiations vary according to the stage in the
production, but there is an officially encouraged series of challenges to the
material as it passes through the production process. Knowledge is
constantly passing through these firms and we considered that firms ought be
able to benefit from these internal and external challenges in terms of
enhancing their own performance. We thus concluded that as long as we
considered the community of practice to be synonymous only with the firm,
we were unable to fully account for the complexities of the production

process and understanding of learning that we were detecting in our digital
media SMEs.
From the existing literature, communities of practice as defined cannot be
multi or inter-organizational. The groups studied by writers on communities
of practice are usually involved in discrete firms or task based activities, for
instance Wenger’s (1998) dealing with insurance claims, where there is a set
of closed procedures as a framework but decisions can be made within this
framework on the basis of tacit knowledge or unwritten convention. The
emphasis is on close study of single organizations, rather than on the diffuse
and diverse networks that make up the production process in, for example,
the e-learning sector. In a sector where management skills and knowledge
for effective production can be acquired through extended and exterior
communities of clients, experts and learners, where mechanisms for the
creation, exchange and possession of knowledge are much harder to define
and where relying on others (non-firm members) is a necessary imperative,
the received notion of a community of practice begins to unravel. Thus we
needed to develop a more appropriate understanding of how communities of
practice operate within the context of production in the e-learning SME
sector.
Examples from our case studies are chosen to shed light on the extended
and more complex role of community in the e-learning production process.
We build on Brown and Duguid’s (1991) argument that an organization can
be perceived as a “collective of communities” and extend this into our
analysis of e-learning producers - small firms who must engage with a wider
collection of communities in addition to the ones internally embedded within
the firm.
4. DEVELOPING LEARNING PRODUCTS
Before we examine the components of the learning community with

examples from our case studies, it is useful to reflect on the particular kinds
of e-learning products that our companies were producing. Most commonly,
firms were producing interactive CD ROM’s or web based products,
sometimes converging the two, and while the content of products differed
markedly, we identified three broad product “types”. Each involves a
different level of community involvement from the four key parties
involved: the firm, external experts, the client and the end-user or learner.
These categories broadly describe the main parties involved. Often,
however, the boundaries are blurred. For example the client and the end-user
may be one and the same; the firm may merely be a single operator who acts
as a hub for a range of external experts. We use the categories schematically

for analytical clarity.
4.1 Company-Specific : Rule Based
In certain cases, the material is ready made and it is converted into

technology based training (TBT). In turn, some of this is rule based, for
example the conversion of client companies’ internal regulations or
induction procedures. The appreciation of how learners learn should not be
underestimated, but the client companies’ needs and those of their learners
are relatively easy to define. Much of the work concerns creating rule sets
and programming. There are many models for doing TBT and a growing
literature on instructional design (Christian-Carter, 2001). In these products
the learning and learning design are largely the province of the firm and the
client. External experts and learners are less likely to be involved in this
learning community.
4.2 Company-Specific: Non-Rule Based
In the production of bespoke business learning materials, there is a

growing emphasis upon client and end-user needs, which are often difficult
to establish or easily build into a client brief. Hence open ended and non-rule
based learning is becoming more popular. We argue that this is one factor
that differentiates this sub-sector from other creative media businesses such
as advertising. Firms are therefore beginning to involve the client or end user
in the creative process and this requires the involvement of figures such as
the learning designer (see below), in order to create or facilitate specific
products and/or forms of delivery. The concentration of firms on the clients’
or end users’ learning requirements leads us to argue for a complex
assessment of the types of creativity necessary for effective production, one
that incorporates understanding of the interpretation of client needs, the
design of the learning approach and the graphic design and technical
elements. In these products the firm, the client and the end-user are often
closely involved in the design, development and evaluation – potentially a
creative learning community.
4.3 Non-Company-Specific
Some e-learning products may be generic, often developed for use across
a range of clients’ companies or divisions, or targeted at a particular market,
education or industry sector. In other cases the product is produced for retail
in the high street. Whichever the case, the company producing the products
has to acquire knowledge and decide on the outcomes and objectives of the
learning. In generic high street retail products, the firm and its designers will
usually not have any sustained engagement or full knowledge of the client or
end users themselves. Thus, the notion that end users are engaged in the
wider learning community is limited. Although depending on what that
product is, they may of course decide to form their own community of
practice. In this kind of speculative production, the notion of a learning
community may be firm-led, drawing more upon historical or established
models of design and learning delivery or market research.
It is in the case of “high–end” corporate or management generic learning
products, supported by a range of tools and e-coaches, that the learning
community may come more into play. Individual or small groups of learners
might be invited to register from different organizations at different times.
Their learning is not situated in an organization, as in a community of
practice, but in their professional arena. It may also be subject to evaluation
and formal review.
There is no necessary relationship between the type of product produced
and the strength and scale of the learning community. However, as a
learning-led approach emerges it is becoming increasingly necessary for
firms to strengthen their learning community, whether involved in the
creation of more tailored or bespoke non–rule based products or to cope with
the increased potential for interactivity even in generic or rule based sets.
5. EXTENDING COMMUNITIES OF PRACTICE

INTO LEARNING COMMUNITIES
As argued, in e-learning production the extension of a community of

practice into a learning community involves integration and exchange
between the firm and its internal communities with the external world – most
notably freelance or exterior experts, the client and end-users. While client
and end-users may be one and the same (such as someone who buys an e-
learning product in the high street) it is more common for large firms,
schools or universities to act as the client and its staff or students to be
identified as the end users. By showing how each of these constituencies
have a role in the production of e-learning products we hope to show how
effective e-learning production can be better obtained through a more open
and open ended approach to learning, one embedded within informal
learning communities.
5.1 The Firm in the Learning Community
While the firm, as a set of communities of practice, is engaged in formal,

structured, but also informal and tacit, modes of learning (Amin, 2000), for
e-learning firms, learning must take on a more upfront and central role. It is,
after all, the raw material and raison d’etre of the firm. We find that the
successful e-learning SME is one that is pushing the learning dimension
much more to the fore, both in terms of product and production process.
In terms of product, the way in which this is most evident is in the
increased creation of specific roles for an in-house expert in the design and
delivery of learning. Often referred to as the “learning designer”, this figure
acts as the designer, director and evaluator of learning needs and outcomes
in the production process.
The role of a learning designer will vary from firm to firm but, most
crucially, an outward looking, experimental approach to learning is key. For
the learning designer the parameters of learning are worked out, in and
through the production process, in a manner contingent on a range of issues
including client needs, resource constraints and pedagogic (and andragogic)
principles. It is against the background of this process of negotiation with
external agencies that firms have increased the development of learning
dialogues and attempted to redefine and reposition the role of a learning
designer.
Good designers have been described by one respondent as “experts in not
knowing” and can ideally work with any type of content. We wonder if this
is true, but clearly a figure that can manage and manipulate any kind of
content to effect and implement an effective learning process must clearly be
attractive in a burgeoning e-learning marketplace.
As well as employing learning designers, in terms of production firms are
becoming more outward looking and experimental in their search to create a
“learning organization”, involving wider engagement with prospective
members of a learning community. This involves integrating external experts
into the learning dialogue.
5.2 External Experts in the Learning Community
Given that many SMEs are unable to employ a full time learning designer
or expert, and with the need for ultimate flexibility in a fast changing and
nascent market place, the role of external experts, such as freelance learning
designers, evaluators and educators becomes more crucial. As this
respondent offered:
“We have recently, with the training, we’ve had a group of experts in the
field writing materials for us, to put together, to create a program, which
has been quite interesting because they’ve all been very, very much their
own people, very much with their own ideas.”
The role of these external experts has grown in recent years as digital firms
with strong technical and design competence but little in the way of
educational expertise are looking to factor the learning dimension into the
product and production process. These experts are often members of close
knit and informal networks, often, but not necessarily, geographically
clustered around the commissioning firm and bound together by a history of
collaboration, shared experience and know how. Given the high levels of
self-employment and freelance work in this sector, the role of the external
expert is more pronounced and vital than in other, more traditional, industry
sectors, making them key members of any learning partnerships.
5.3 The Clients in the Learning Community
The client must play a central role in the formation and maintenance of
an operative learning community. Attitudes to clients varied among the firms
we interviewed, with some of them talking about managing their clients’
expectations and the “whole process being managing the client to accept the
creative”. These remarks were often from firms with advertising
backgrounds. However the ones with their origins in education or
management consultancy took a different view, seeing the clients’ creative
contribution as a central part of developing learning products and enhancing
learning practices.
For the firms that we would call “learning-led”, the negotiations with
clients and the meeting of their needs assumed a central role:
“… there may be a group of people who are involved on the project and
I’d like to meet with them, so I’d ask them if I could get together with
them and it might be that there’d be two or three meetings there in order
just to begin to get a handle on, what are the specs, what do they want the
project to be able to do at the end of it and what, what do they want?”
While the extent to which clients are engaged in a learning dialogue with
firms will vary, it was clearly evident that more successful and progressive
firms understood the client as central in the definition and delivery of
learning, not merely a hindrance to be “managed” out of the production
equation. This ensures that the issue of learning, for both parties, remains
open and subject to creative development.
5.4 The End Users in the Learning Community
While firms, external experts and clients can often generate productive
learning communities, the involvement of end users/learners is an area that
needs more work. The needs of both individual, and groups of, learners are
often subordinated to concerns over development and design, timescale,
budget, distribution, price and so on. But there are a number of other, more
hidden, reasons why end user learning needs and experiences are often
secondary concerns. It may be that in the provision of tried and tested rule
based software, learners’ needs are assumed to be simple and straightforward
and unworthy of detailed consideration. Further, it is often the case that
clients assume that their staff or student learning needs are homogenous or
easily predetermined. Consultation and testing at the point of learning may
not therefore occur. Also, firms themselves may feel uncertain in
challenging clients’ identification of what constitute the key learner issues in
practice, particularly in cases where, for example, the client is an educational
institution. Even when partnerships involve the end user, there is no
guarantee that the learners input will be as valued as that of the firm or the
experts. In short, we observe much of what constitutes the debate over
learning has often missed out the learners themselves.
However for a growing number of firms, the integration of the end user
into the learning dialogue is now deemed crucial. We found firms where it
was considered vital to know the learner and develop learning technology
that is positively learner centered, and seek to develop partnerships with
them in a collaborative learning network.
“If you look at it from three points of view, the first point of view is,
what does the learner require? What are the specific requirements of the
individual learner? The second point of view is, do we want to make a
profit from this? The third point of view is what level of quality should
we be aiming for? In all cases the primary focus of attention needs to be
on the needs of the learner.”
Not only are firms looking to expand their testing, evaluation and
feedback strategies, an increasing tendency now is to generate a dynamic
among learners in “softer” skill areas and encourage them to share learning
experiences and stories and to generally engage more widely with the
providers and producers of e-learning materials and developments in the
learning community. The idea of “recursive loops” for learning is suggested,
with the goal being to create a set of mechanisms whereby all parties can
provide feedback and engage for the duration of the production process.
How this can be achieved, however, remains a crucial question.
6. MECHANISMS FOR MANAGING THE

LEARNING COMMUNITY
How are these four disparate constituents of learning communities able to

bind and act to effect the development of learning and learning products?
Our respondents indicated that certain emotional and conceptual leaps must
be made to open out the organization to the influence of the other – an
openness which requires the cultivation of new relations of trust, as well as
the management of both face-to-face and virtual or distance relationships.
6.1 The Importance of Trust
It was apparent that many of our e-learning firms were driven by pure
economic considerations, employing low cost strategies and traditional
approaches to management that, for them, meant the necessity of remaining
afloat in a competitive and uncertain market. But in more progressive e-
learning SMEs, the common management style is an empowered one where
facilitation, trust and ethics are valued. In this case the sharing and common
exploitation of knowledge was frequently referred to:
“The guys are constantly – well the way we’ve got the office set up in a
huge great big circle – they’re constantly skidding around to one
another’s machines, there are no prima donnas in there and nobody’s
precious about knowledge and if somebody has knowledge they share it.
It’s as simple as that.”
The role of trust in maintaining a learning community is clearly important.
Once the community of practice extends beyond the traditional boundaries
of the firm then it becomes more of an imperative to create and sustain
active mechanisms of trust and reciprocal exchange (Banks, et al., 2000). As
Giddens (1991) notes, since social relations are now less dependent upon
physical locality or place than previously, new “relational communities”
(Amin, 2000) must be developed and sustained across space and time.
6.2 Face-to-Face Community
Geographical proximity remains important in production; to involve

clients, external experts and learners in face-to-face exchanges remains the
best way for identifying and agreeing objectives, processes and outcomes
and for helping to facilitate the levels of “active trust” (Giddens, 1991)
deemed necessary to bind the learning community. It was noted that face-to-
face meetings avoided “endless e-mails”, course content was largely written
within face-to-face meetings and problems caused by the lack of

understanding of each other’s functions and of the development process
itself could be overcome with face-to-face exchanges. It was stated by many
firms that personal meetings are essential throughout the development
process:
“I like to be able to see the actual learners and meet with them and
discuss the kinds of things they do, so, step one then is, I like to be able
to define what the problem is and what the product has to be able to do.”
“I’ve got to define what the client needs and that will involve probably
some meetings with the people who are responsible for the project
themselves. I’d also like to see and meet with some of the end users and
I’d like to go into their workplace, like to see them working in their
situation.”
There was also evidence that firms wished to strengthen local networks in
order to provide some context and comparison for their activities with other
firms in the sub-sector – geographical proximity then still being seen to
provide a strong basis for networking and collaborative mechanisms.
6.3 Relational Communities
Where it is not possible to meet the clients or external experts directly, a

secondary option is to use information and communication technologies to
enable relationships to be maintained across distance. More commonly, it is
often impossible to meet or engage with the learners (at least not all of
them). But if learners are to become more central to the learning community
then mechanisms for integration must be found. Ensuring integration into the
learning community over geographical space becomes a key issue for the
firm. This is one example of the mechanisms one firm provided to support
relational communities of learners from different organizations over space:
“From the home page, you can go to the parts you are allowed access to,
for instance, course material, discussion groups, chat, where you are up
to individually, syllabus, personal pages. [There is access to] course
material – accessed via the menu bar – material, management of material,
links and live unit activities. There are five levels of collaboration open
to individual learners – to other individuals, to the present activity, to the
cohort [group of learners on course], to the curriculum [others on the
same program but at a different stage], to all others on all programs.”
Additionally, “e-coaches” now assume a central role in cementing the
dispersed body of learners into the learning community. For example, to link
the coaches to the learners there may be devices such as a “personal adviser”
tool. This also links the coaches to the units they are facilitating and they can
check on the progress of individual learners.
Some of our SMEs now intend to make it a priority to update knowledge
and to provide toolkits, interactive resource centers, with archived
documents and links to material, and more opportunities for learners to do
things themselves. Learners may be more engaged in feedback activities, or
even privy to on-going consultation and development discussions. One
respondent referred to this as the provision of “facilitated communities of
application”. This involves allowing learners to create content and put it on
line themselves and to facilitate networking with other learners and
community members, across cultures and continents, thereby meeting the
overall aim of raising the profile and enhancing the contribution of learners
in the e-learning process.
7. BARRIERS TO THE FORMATION OF

LEARNING COMMUNITIES
We have, quite selectively, revealed some of the possibilities and

strategies of new learning communities, as they take shape around our e-
learning SMEs. However, it would be misleading to suggest that this is a
general or even a widespread process. Many firms remain locked into the
traditional separation of the firm from the external world, the prioritization
of the firm’s expertise over external knowledge and giving short shrift to
opening up the debate on learning within the firm or problematising the
learning qualities of their products:
“Everyone has a first class honors here, they get the training they require
when [they] switch the computer on, that’s it. Go and learn it….You need
the Internet to learn and we don’t need anything else.”
“We shout and they learn it.”

Clients too are often treated as distractions, or even irritations, rather than
potential partners in the development of appropriate learning:
“I don’t think many clients understand the work that’s needed on their
side in order to hand us the content that has to go into it…If you want a
quote they think we’ve got a magical content generator in the building.”
One of the firms where we interviewed spoke of encouraging its staff to
experiment and make mistakes; it wanted to avoid the “blame culture” it had
encountered among its clients. Another, however, while claiming that its size
and attitude allowed it the “luxury of debate” on learning theory, was later
criticized by its own staff for its concentration on deadlines and profits
which discouraged staff from being innovative. It seems clear that while
learners are encouraged to use tools experimentally to construct their
learning, create their own content and upload whatever they like into their
shared space, the capacity of many SMEs to experiment, make mistakes and
survive is limited. The economic pressures placed upon a small firm may
make it difficult to enjoy the luxury of debate on learning – this is accepted.
All activities take place within budgetary, human resource and time
limitations. Even if the firm wishes to develop a network of external experts,
clients and learners a decision has to be made on whether to formalize and
structure this network and incur maintenance costs through communication,
meetings, events and so on, or to leave it open ended and informal, but
potentially losing impetus and the leakage of its key asset – productive
encounters between different knowledges and communities. Finally, even
when learning communities are in evidence, the power relationships within
that network may be asymmetrical. Not everyone is able to shape the
direction or definition of learning exactly as they would wish.
8. CONCLUSION
The formation of e-learning businesses and the creation of the learning

products involve new convergences of technology, media, skills and,
increasingly, individuals and organizations themselves. The management of
firms and production processes must now focus on the human aspects of
these convergences – the learning community. In our sample, this has led to
the emergence of learning designers, as well as the increased utilization of
client knowledge and, where required, the involvement of the external
experts, clients and end-users in the creative learning process, providing an
additional, challenging dimension.
Furthermore, the issues of distance and “what happens when you cannot
meet” apply not only to the users of the material but also to the providers of
the knowledge. There are differences among the tools and procedures firms
and learners use to acquire their knowledge. As more firms enter this
business sub-sector and as learning design has the potential to become a
profession in its own right, the need to acquire knowledge from outside
sources will increase. This question of knowledge acquisition over distance
therefore applies to developers as well as learners and is one of those
addressed in our project. There seemed to be an irony in that firms find face-
to-face meetings essential in the development of distance learning products –
a necessity perhaps only partially offset by new relational communities that
are emerging across geographical space. The end result is a loosely bounded
learning community comprising members from a variety of organizations
and groups interacting face-to-face and at distance.
As more and more firms look to enter the e-learning market, not all will
come ready equipped with learning designers or a discrete learning
philosophy. We argue that this may, but certainly not always, undermine the
quality and effectiveness of learning products. But by engaging in reflexive
and extensive debate with those who do have an understanding of the power
and potential of a learning led approach (other firms, learning and education
experts) and by engaging with the wider communities in the e-learning
marketplace – clients, experts and crucially learners – firms can go a long
way to resolving some of the current dilemmas of this emergent industry,
namely how to provide materials that are sufficiently researched and piloted,
pedagogically and andragogically appropriate and serve the needs of clients
and the diverse, and fast expanding, communities of end users.
REFERENCES
Amin, A. (2000). Organisational learning through communities of practice. Paper presented
at Millennium Schumpeter Conference, University of Manchester, June.
Banks, M., Lovatt, A., O’Connor, J., & Raffo, C. (2000). Risk and trust in the cultural
industries. Geoforum, 31 (4), 453-464.
Brown, J.S., & Duguid, P. (1991). Organizational knowledge and communities of practice.
Organization Science, 2 (1), 40-57.
Christian-Carter, J. (2001). Mastering Instructional Design in Technology-Based Training.
London: Chartered Institute of Personnel and Development.
Fransman, M. (1994). Information, knowledge, vision and theories of the firm. Industrial and
Corporate Change, 3 (3).
Giddens, A. (1991). Modernity and Self Identity. Cambridge: Polity Press.
Lave, J., & Wenger, E. (1990). Situated learning: Legitimate peripheral participation.
Cambridge: Cambridge University Press.
Raffo, C., O’Connor, J., Lovatt, A., & Banks, M. (2000). Attitudes to formal business training
and learning amongst entrepreneurs in the cultural industries: Situated learning through
‘doing with others’. Journal of Education and Work, 13, 215-230.
Speake, T., & Powell, J. (1997). Skills for the missing industry: An exploratory study.
University of Salford: Training Technology Unit, Dep. for Education and Employment.
Swanson, G., & Wise, P. (1997). Digital futures: Women’s employment in the multimedia
industries. Brisbane: Australian Key Centre for Cultural and Media Policy.
Wenger, E.C. (1998). Communities of practice: Learning, meaning and identity. Cambridge:
Cambridge University Press.
Wenger, E.C. & Snyder, W.M. (2000). Communities of practice: The organisational frontier.
Harvard Business Review, Jan 1st.
Business Process, Experience and Memory:
Educational Approaches and Technology Tools for a
Global Workforce
Daniel M. Carchidi1 & David J. McCarthy2

1
GE Capital, Learning Services, USA, 2Run Fast, Inc, USA.
1. INTRODUCTION
The great Internet shakeout of 2000 has made all of us take a step back
and think hard about the net-based economy and the type of ventures that
will succeed or fail in turbulent times. In many cases, the success or failure
of a net-based venture or industry segment is not necessarily based on the
underlying technologies, but rather the application of technology in an
appropriate context. What is clear is that there seems to be little abatement in
the application of new technologies in all types of business contexts.
The early 21st Century is an era marked by rapid technological
innovation that is fuelling globalization. The trend toward globalization has
sparked an upswing in cross-border mergers and acquisitions. These cross-
border mergers and acquisitions have created a host of cultural and political
issues that must be addressed by training communities that are increasingly
being asked to do more with less. The e-learning trend is certainly not a US
only phenomenon as efforts such as the “eLearning: Designing tomorrow’s
education” initiative (European Commission, 2000) and the recently
released, “eLearning Action Plan” (European Commission, 2001) suggest.
The European Commission is getting serious about the role that technology-
mediated learning can have on the economic development of that region.
For many in the training community, e-learning may provide tangible
benefits such as cutting costs for travel and living expenses. A study by
19
20 Daniel M. Carchidi & David J. McCarthy
Hambrecht and Company (2000) reveals that $500 million was spent on
Internet training in 1999. Spending projections for 2002 are expected to
explode to over $7 billion. While cost savings may be on the minds of many,
the results from the shift may actually improve learning.
As Hall (2000) has pointed out, e-learning is beginning to demonstrate
clear benefits for management education. He notes the value in consistency
of information and content integrity, as advantages over instructor led
training, since the dynamics of interaction that the student experiences are
more uniform in an e-learning format. E-learning also allows students of
varied capability levels to learn at their own pace, which can speed the time
to competency for some students. Mastering knowledge in a rapidly
changing business environment is a challenging task as there is less and less
time available for training. Hall explains that learning that is accessible
“anytime, anywhere,” “in the right amount” and “suited to the individual’s
needs” makes sense given the pace of business today. While acknowledging
that technology cannot solve all training issues, Hall does make a strong case
for e-learning development.
In this paper, we will consider two ideas for understanding e-learning and
targeting processes that will help e-learning efforts succeed. The first
presents a method for linking business process, experience and institutional
memory, which supports the e-learning development process. The second
concerns linkages between the learning or cognitive level and the extent of
the business process being modeled, which we term the isomorphic model.
The isomorphic model is intended to enable trainers to select the most
appropriate educational approach and technology to address e-learning
development in an environment marked by globalization, mergers and
acquisitions and shifts in culture and politics. We believe that concentrating
on the extent of business process being modeled and the level of learning
will provide educators with an effective approach for selecting the
appropriate e-learning solution. Three e-learning projects from the General
Electric Company (GE) are then examined. Finally, we offer lessons learned
and implications for educators considering e-learning development efforts.
2. THE GENERAL ELECTRIC BUSINESS (AND

LEARNING) ENVIRONMENT
For over 120 years GE has been a major player in the US and global
economies. In 2000, GE’s net earnings were $12.7 billion on revenues of
$129.9 billion. GE has been named the “world's most respected company”
by The Financial Times in 1998, 1999 and 2000, “America’s most admired
company” by Fortune in 1998, 1999, 2000 & 2001 and e-business of the
Business Process, Experience and Memory 21
Year by InternetWeek in 2000, among other honors. The company consists

of 12 businesses, which span financial services, appliances, lighting, aircraft
engines, media, plastics, aviation, transportation, power, industrial and
medical systems industries. As such, GE has been a leader in facilitating and
managing the globalization, merger and acquisition and cross-cultural
integration of diverse businesses and people since its founding.
The learning environment at GE has a long history of developing some of
the best leaders in business and for reinventing itself with the changing
times. GE invests over $1 billion annually on training and education
programs around the world to insure the highest levels of competency in its
workforce. Although the scope of its activities is global, the focus of GE’s
commitment to excellence in leadership development and cultural change is
Crotonville, the world’s first major corporate university. Crotonville
provides the forum for GE people to tackle new business problems from
around the world and share their knowledge with customers, suppliers and
colleagues throughout GE. Increasingly, GE training leaders spurred by
digitization efforts and e-business initiatives company-wide, have focused
their efforts on e-learning development. Beyond potential cost reductions,
there is a belief in some circles that technology tools that incorporate the best
practices and experiences of GE’s top performers can enhance performance.
In the next section, we will examine the link between business process,
experience and institutional memory and their connection to performance
enhancing technology tools.
3. BUSINESS PROCESS, EXPERIENCE AND

MEMORY: A STARTING POINT FOR E-
LEARNING DEVELOPMENT
The e-learning solutions discussed in this paper originate from a

consistent approach for linking business process, experience and memory
within an organization. The relationship between business process,
individual experience and memory are depicted in figure 1.
The four-stage process begins with identifying the business process, then
systematically extracting the experiences and culling the institutional
memory and tacit information of the organization’s experts. Once the
institutional memory is made explicit, the development team can then
perform the learning design and construct an appropriate intervention.
Figure 1: The Relationship between Business Process, Experience and

Memory.
3.1 Stage 1: Needs Assessment
This involves a thorough rendering of the business process and includes a

detailed description of the sub-processes and business tasks contained within
the process. During this stage, the development team ensures that the process
is stable. Before people learn to execute on a process, we need to ensure that
it is viable. The team begins to determine whether a performance gap in
conducting the business process exists within the target audience. Processes
that are found to be ineffective, in need of re-engineering or insufficiently
documented are deemed inappropriate for a learning intervention since the
solution would be based upon a potentially flawed or immature process. For
example, it would be costly and highly ineffective to design an educational
intervention around a process that key sub-process owners had not agreed to.
The outcomes of this stage are the determination as to whether a learning
need exists and to what extent the business process needs to be modeled. The
extent of the business process to be modeled provides a critical scoping
parameter for the development effort.
3.2 Stage 2: Solution Definition
Beginning with the end in mind, the development team conducts a series
of interviews with business process experts as well as those that enable and
support performance of the business process. The team extracts the
experiences of these subject matter experts and documents such areas as the
business need, performance need, content areas and technology environment
of the organization. Subject matter experts may include process owners, high
performers, content experts, technology experts and target users of the
proposed solution. The isomorphic model of e-learning development (see
figure 2) is used to determine aspects of the appropriate learning solution.
The outcome of this stage is a comprehensive solution description that
delineates the learning objectives, content topic list, business case, high level
design concept, project timeline, target audience characteristics and solution
cost.
3.3 Stage 3: Content Analysis
This begins the development phase of the e-learning solution effort.

Beginning with the performance gaps established in Stage 2, the
development team develops learning objectives to address them. In this
phase extensive interviews are also conducted with subject matter experts
that possess strong content knowledge. The subject matter experts aid the
development team in extracting the most valuable aspects of the
organization’s institutional memory. For example, a business development
leader who recently conducted a particularly challenging acquisition may
have valuable insights on successful strategies to incorporate or pitfalls to
avoid for future acquisitions. Depending on the extent of the solution, this
may involve detailed explanations of minute elements of the business
process and take several months to develop.
3.4 Stage 4: Solution Development
This stage involves constructing and producing the e-learning

intervention. Based upon the input of experts on business process, content,
learning design, technology and deployment environment, the learning
intervention is rolled out to the target audience. Once deployed the solution
is assessed in the context of the business process to ensure that the
performance goals are being achieved.
4. THE ISOMORPHIC MODEL OF E-LEARNING

DEVELOPMENT
This is an approach for identifying critical elements of the e-learning

intervention/solution based upon learning and business process
specifications (see figure 2). The model allows e-learning developers to
compare the desired learning or performance objectives with the structural
dimensions of the business process they are trying to simulate. By aligning
these dimensions, learning designers and content developers are able to
select the appropriate e-learning application (i.e. knowledge system, activity-
based learning or business simulation – see below) to meet the learning
objectives and business process complexity required by their target audience.
The model has been used to identify solution development parameters for
several organizations within GE, which are discussed in subsequent sections
of this paper.
Figure 2: The Isomorphic Model for E-learning Development.
A starting point for selecting the appropriate e-learning solution begins

by establishing concrete learning objectives for the target learner group.
Learning objectives are often obtained through reviews of previously
documented learning objectives, job task analysis and interviews with

business process and content subject matter experts. Once documented, these
objectives become the guideposts for segmenting the desired learning
objectives for target audiences.
The first dimension of figure 2 shows gradations of the extent of the
business process being modeled. This dimension extends from the
explanation of a task or concept to modeling performance of specific
business tasks through modeling performance of the entire business process,
an outcome of Stage 1 above. Explaining a task or concept is the lowest level
of business process being modeled. Learners do not perform any aspect of
the business process. Rather, a task or concept pertaining to the business task
is explained or represented for the learner (i.e. a description of how to
evaluate a cash flow statement during an acquisition due diligence process).
For performance of business task, the designer will model a portion of
the business process related to a specific task or group of tasks. The learner
will be expected to complete the task successfully in order to demonstrate
competency. For example, putting the learner through the process of
evaluating a cash flow statement with appropriate performance support and
coaching.
Performance of the business process is the portion of the model that has
the greatest depth and breadth by encompassing an entire business process.
Designers document all aspects of the business process. This is typically an
extensive and expensive process. Learners are immersed within a
comprehensive learning environment in which they complete learning
assignments corresponding to the entire business process. An example would
be the execution of an end-to-end acquisition due diligence process
culminating in decision-making responsibilities.
The second dimension of figure 2, level of learning, recognizes that
effectiveness extends from understanding through demonstrated decision
making ability, an outcome of Stage 2 above. Understanding corresponds to
Bloom’s taxonomy of the cognitive domain’s knowledge and comprehension
levels in which remembering and explaining the meaning of information are
emphasized (Bloom, 1956). Demonstrated ability corresponds to application
and analysis levels of Bloom’s taxonomy. Here, the ability to use
abstractions in concrete situations and break down a whole entity into its
component parts is the focus. Decision-making, the most cognitively
complex learning level, corresponds to Bloom’s synthesis and evaluation
levels. Decision-making emphasizes putting parts together to form a new and
integrated whole and making judgments about the qualities of ideas,
resources, or events.
Solutions may vary in design and deployment characteristics, but are
typically one of three types:
• Knowledge system, an inquiry-based system in which experts share

industry and company knowledge through an index of multimedia
delivered stories, concepts, terminology and demonstrations of business
process tools.
• Activity-based learning environment, where learners develop skills to
execute specific business tasks that are customized to an organization’s
learning needs and processes.
• Business simulation, a comprehensive environment in which learners
build generic foundational learning through a simulation of a real-world,
end-to-end process or transaction.
Learning objectives within the isomorphic model may build or scaffold upon
each other. For example, the learning design of an activity-based solution
that focuses on application and analysis level learning would incorporate an
expert knowledge system, which concentrates on knowledge and
comprehension level learning. Consider an activity-based scenario in which
a learner is asked to develop skills in entering information correctly in a
customer relationship management tool, such as Siebel. The learning
application emphasizes data entry techniques and is supported by a system of
walkthroughs and glossary of terms that support and demonstrate how to use
the application. The knowledge system also provides the learner with a
business context for using the tool, which is captured in streaming video
expert stories. The next section provides examples of products developed
using the isomorphic model.
5. PRODUCT EXAMPLES IN BUSINESS CONTEXT
5.1 Case 1: GE Capital - Euro InSIGHT Knowledge

System
5.1.1 Business Context
Because GE is a multifaceted company with diverse businesses across the

European Union, the company realized that the introduction of the Euro
would affect each business unit differently. Therefore, each business unit
within the Euro zone began working on the appropriate response for its
territory and market. Areas of similarity across businesses, such as payroll or
interaction with shared service centers, were addressed as a group of
businesses on a country-by-country basis. To ensure high standards, each
business’s approach for converting to the Euro was based on the Six Sigma
Quality process - GE’s Six Sigma Quality methodology. A project
management office (PMO) was established in 1998 to facilitate cross
business information sharing, drive common practices, conduct project
monitoring and ensure that the same solution was not being done multiple
times across the 13 businesses within the Euro zone.
5.1.2 Business Problem
By the end of 1999, businesses were well engaged in planning for the
Euro transition. Early in the planning process, the businesses identified a
common need for training. Two types of training needs were identified. The
first dealt with general awareness of what Euro transition would mean to the
individual and what affects the new currency would have in a business
context. The second concerned function specific training; that is training that
was specific to sales, human resources, etc. As Dawn Johnson, the leader of
the Euro PMO pointed out:
“Where we saw an immediate need was in the area of awareness training.
We saw not only that it was an immediate need, but also that it was
common to everybody.”
Figure 3: Euro InSIGHT 1 Indexing System.

The PMO was charged with creating a consistent and easily accessible
approach for disseminating general awareness training across all GE Capital
businesses.
5.1.3 Business Solution
By identifying the business need, process complexity and performance

objectives, the development team was able to structure a solution that was
appropriate for the needs of the target audience. Interviews with process and
content subject matter experts revealed that the immediate training need
should address relatively low cognitive complexity (i.e. Bloom’s knowledge
and comprehension levels) and the business process should focus on
explanation of Euro concepts and implications of the transition. The figures
below illustrate how the development team designed the knowledge system
to interact with users.
Figure 4: Euro InSIGHT 2 Concept Explanation in Animated Format.

5.1.4 Business Outcomes
The sponsor of the project found that deployment of the knowledge

system initially was not well timed for the target audience. The urgency of
the Euro transition had not captured the attention of the target audience,
although the Euro team leaders in each of the Euro zone businesses found
the solution highly engaging. The project sponsor continued her efforts to
expand awareness of the system by enlisting high-level business leaders to
communicate the importance of using the technology tool. Over time, the
necessity of learning more about the Euro transition and its effects on GE
Capital are driving up usage of the knowledge system. The ubiquity of the
on-line system is coming to be viewed as a major advantage over instructor
led training since the demands of the business environment and the travel
and living costs are drastically reduced. More importantly, access to learning
is available at point of need, around the clock.
5.2 Case 2: Card Services EZBinder System –

Knowledge System
GE Card Services is a leader in providing credit services to retailers and

consumers. Formed in 1932 as a provider of consumer financing for GE
Appliances, Card Services provides private-label credit cards, commercial
programs and card-related financial services for hundreds of retailers and
manufacturers across North America. Card Services also issues and services
corporate cards for commercial customers, including purchasing, travel and
fleet vehicle cards. Their direct-to-consumer lending group offers debt
consolidation and home equity loans. Card Services offers clients a full
range of operational, financial and analytic support, and develops
customized marketing programs designed to increase sales and customer
loyalty. This philosophy of partnership has helped it grow to nearly $25
billion in total assets and to serve more than 100 million cardholders. They
have 29 offices throughout the world.
The credit card market in the USA and globally is marked by increased
competition and eroding margins. The maturation of the credit card industry
has resulted in significant scaling of credit card operations and the need to
reduce time to market for innovations in order to remain competitive.
Card Services new product introduction (NPI) cycle was lagging their
competition in terms of cycle times and differentiation of products once
introduced. “Our products were taking longer to get to market and often of a
‘me-too’ nature,” noted Babs Ryan, Director of Marketing for Card Services
and project sponsor.
NPI processes, although thoughtfully designed, were marked by
inefficient hand-offs from department to department. As Ryan commented,
“Sometimes our process was facilitated by nothing more than a string of
emails from the project manager”. NPI teams are run by project managers
within the Marketing Group but tasks are completed by individuals with only
task level responsibilities. The finance department handles pricing, IT does
systems and legal writes the contracts. The project manager had
responsibility for overall process execution but not the authority over the
resources required to complete the individual tasks.
Another complicating factor in Card Services’ NPI approach was that
although the process is done many times, the same people rarely do it twice.
There was no centralized repository for capturing and exchanging
experiences and best practices. Lack of process repeatability led to a lack of
leverage of experiences.
In addition to the issues discussed above, Ryan was adamant that the
software portion of the solution should not look like typical web-based
applications. The software had to look and operate like the current operating
environment - filing cabinets, binders, documents, etc. This requirement was
intended to eliminate application training needs and increase adaptation and
usage by the user community.
The process for developing the solution started with confirming the NPI
process and the tactical-level operating model for the NPI project teams.
Once the overall process and operating model was documented, the concepts
of the isomorphic model were applied to determine the most appropriate
means for effecting the required changes within the organization. Based on
the analysis, we identified two key factors to drive the solution definition:
knowledge of task level concepts and streamlined communication.
Based on these two factors, the team developed the EZBinder system.
EZBinder is an Internet-based process and knowledge management system
used to capture, retain and reuse best practices in an effort to cut NPI cycle
times in half and increase end product differentiation. The Card Services
instance of EZBinder was for NPI, but it was created in a generic manner so
that it can be used for any repeatable process.
The EZBinder system acts as a codified expert knowledge system for the
company’s best practices related to NPI. EZBinder contains a virtual
template binder that is copied and shared for each specific NPI project. The
system maintains copies of all NPI projects so that best practices from prior
projects can be easily integrated into the template binder for reuse by
projects going forward. The EZBinder solution embeds institutional memory
within the best practice documents to be leverage by each subsequent user of
the EZBinder application.
In the balance of process and user control versus flexibility, the processes
and software were built using a “structured flexibility” philosophy. The
overall system is structured around best practice documents in a templates
binder that appear in the NPI process order with the same name in each
project. This allows users to easily recognize pertinent documents across
projects but at the same time allows them to retrieve and post documents in
any order.
The figures below illustrate how the team built in the requirement that the
system mirror the look and feel of the documents.
Figure 5: EZBinder 1 Filing Cabinet Metaphor One filing cabinet per

product line.
Figure 6: EZBinder 2 Open Drawer One binder per project.
Figure 7: EZBinder 3 Open Binder One tab per process step.

Figure 8: EZBinder 4 Process-Driven Best Practice Documents One or

many templates per process step.
The EZBinder application has been successful in limited rollout to date in

the Card Services business. For the projects that have been run through the
EZBinder system, best practices have been re-used from project to project
and communication among team members has been streamlined.
In addition, the EZBinder system has been “productized” by the
Performance Technology Solutions business unit and has been resold to
other business units within GE Capital. Lessons learned for the development
and rollout of the EZBinder system are the following:
• Rollout Sponsorship: Training system rollout, like any application
rollout, needs to be done under the auspices of strong executive
sponsorship. The senior executive sponsor for the project left before the
system was rolled out limiting the initial uptake of the system.
• Departmental IT Support: Technology-based solutions require
departmental technology support. Whether we as educators and corporate
trainers like it or not, our finest learning designs and taxonomies can go
unused or worse misused if the proper departmental technology support is
not there to help implement the solution and take over ongoing
maintenance. In the case of EZBinder, there were several breaks in the

project continuity due to lack of support from departmental IT.
5.3 Case 3: CAS Simulation Suite – Business Simulation
The GE Corporate Audit staff are responsible for evaluating whether or

not the financial position of the company is fairly stated according to
generally accepted accounting principles and GE corporate standards.
Auditing the GE businesses requires a unique blend of technical accounting
skills and customer management diplomacy. First, finding potential audit
issues requires a thorough knowledge of the business being audited,
including a deep knowledge of the technical accounting involved. Also,
being able to navigate the politics involved in potentially restating corporate
earnings once an issue is identified is equally important.
Adding complexity to an already challenging learner environment is
GE’s growth path. GE experiences significant organic growth and growth
through acquisition each year. The proposed Honeywell acquisition was a
prime example of the acquisition-driven growth seen at GE. This merger
would have nearly doubled the size of the audit staff if the deal had closed.
All auditors are required to audit using the same philosophies,
methodologies and techniques.
There are several significant issues associated with training a group like
the audit staff, as noted by their e-learning Manager. First, there is a wide
disparity of knowledge and skill base across new auditors. Some auditors
come to the audit staff from prestigious MBA with finance degrees. Some
come to the audit staff with no financial background. Many others arrive as a
result of acquisition. The second critical issue in training the audit staff is
geography. Learners and experts are distributed literally all over the globe.
The cost of executing a synchronous face-to-face learning experience for
new auditors is prohibitive. The third issue is time. Auditor’s workloads run
60 to 70 hours per week before you add in training time. Any training rolled
out to audit staff learners has to be tailored to the individual learner;
distributed across the world efficiently; and low incremental cost per
conduct in terms of expert resources.
Through our analyses based on the concepts of the isomorphic model, we

determined that a series of process simulations with user-driven pacing and
depth (see figure 9 below) to be distributed via the Internet was the best
approach. We employed the notion of “self-scaffolding”, allowing the
flexibility for slower learners to explore and drill down on basic task
knowledge and concepts at their own pace and allow faster learners to move
quickly past subjects that they have already mastered.
Figure 9: CAS 1 User Driven Navigation Floor plan.
To start the training development, we created a basic model for all

applications to follow. The same learning design and navigation was used
across applications. We also strove to make the learning experience mirror
the on-the-job experience, what we call fidelity (see figure 10 below).
Figure 10: CAS 2 Fidelity Prior Year Work papers.
The model and fidelity concepts were stressed so that learners would not
have to “learn to learn” and could immediately get right into the application.
Out of deference to subject matter experts’ time, we employed role-plays to
capture the expert knowledge during the development effort and then
embedded that knowledge into the asynchronous training applications (see
figure 11 below). Different subject matter experts were approached for the
different technical accounting areas.
The business process that we chose to simulate was the scoping of the
audit team’s audit, with the learner responsible for deciding on the scope and
then defending the scoping decisions to the audit managers (see figure 12
below). Regardless of uniqueness of the user experience, all learners still
have the goal of demonstrating competency through execution of the scoping
process.
Figure 11: CAS 3 Task Concepts and Basic Knowledge Audit Manager
Meeting.
The response from the clients has been very positive. We have not
collected hard data on the costing saving or the increase in competency of
users. Our evidence is anecdotal in nature. After the initial contract to build
the learning design model and three specific applications, we have built five
more modules on the same model and three more are budgeted after that.
6. IMPLICATIONS FOR E-LEARNING SUCCESS
As a result of our work in developing experienced-based learning

solutions within GE, we offer the following implications for e-learning
success.
6.1 Determine a Baseline of Workforce Performance
Efforts to digitize learning are becoming increasingly more prevalent in

large corporations like GE. The value of the digitization effort can be
undermined if the goal is merely to move content to the web. As we have
discovered in our e-learning development efforts, a needs assessment should
be conducted to verify that the process and content that are being digitized
are effective for meeting the organization’s business goals, prior to the start
of content digitization.
Figure 12: CAS 4 Competency through Demonstration, Process

Execution Scoping Meeting.
6.2 Establish an Infrastructure that Supports Global

Distribution
Attention to detail in constructing the appropriate solution for a given

performance gap can be undermined by a solution that neglects the
deployment environment of the target users. Delivering media rich
applications, via CD-ROM or the web, should include a thorough

understanding of the deployment requirements at the beginning of the
project. These should include areas such as the hardware, operating system
and network environment characteristics. In addition, it is important to
understand the experience level of the target audience and any past issues
that may have been encountered when rolling out technology tools.
6.3 Gain Clear Acceptance from and Ensure Continuity

in a Project Sponsor
It is essential from both acceptance and continuity perspectives to

designate a project sponsor and have him or her continue as sponsor
throughout the lifecycle of the project. We have discovered that changing
priorities may occur when ownership of an e-learning project is transferred
to a new project sponsor, which can undermine the solution development
effort. Assurances should be in place prior to launching the project that the
sponsor or his or her designee will see a project through to completion.
Those considering an e-learning project should select a sponsor who will
provide the stability to see the project through to completion. From top-level
executive sponsorship to departmental IT support, there must be a desire to
improve the existing training models.
6.4 Plan for the Entire E-learning Development Process
As discussed earlier, successful design and development of e-learning

technologies involves a comprehensive approach. In several efforts that we
have been involved in, we have observed that projects can run out of steam
when the final stages of product development and acceptance are neglected.
Inattention to deployment issues can sink an e-learning product. We
recommend setting aside at least 10% of the development budget for
deployment and rollout activities. Schedule feedback sessions, in addition to
integrating feedback options, into the e-learning application itself in order to
get user impression once the application is more widely deployed.
6.5 Projects Need to be ‘Quick-Hit.’
If initial project results cannot be tangibly realized within three months,

the expectation must be set that market forces, priorities and team members
will change. Changes in priorities and personnel often result in scaling back
or de-funding of initiatives. The current corporate competitive landscape
demands that training solutions keep up with the pace of change faced by
businesses.
6.6 Training Has Never Been and Is Still Not Free
The more complex the learning needs, the more costly the solution. The
average on-line training experience costs from $20,000 to $50,000 per
learner hour. Significant economies of scale can be achieved however
through thoughtful programmed reuse of training assets. Asynchronous
courses also eliminate the need to have an expert present for each course
conduct.
7. CONCLUSION
We believe the benefits of a structured approach to linking business

processes, experience and institutional memory can lead to enhanced
learning experiences that support the core processes of the organization with
the most effective experiences and knowledge of the organization’s subject
matter experts. And by focusing on the level of learning and the extent of the
business process being modeled, e-learning managers and developers can
select the appropriate solution for their target audience. Using case examples
from GE that highlight the value and limitations of e-learning, we have
contextualized how technology-based management education is being
designed, deployed and developed in a large, multinational corporation. Our
intent has been to offer both a conceptual approach as well as practical
examples for faculty members, training administrators or course developers
whose educational portfolio may benefit from e-learning courses or
programs.
REFERENCES
Bloom, B.S. (Ed.) (1956). Taxonomy of educational objectives: The classification of
educational goals: Handbook I, cognitive domain. New York: Longmans Green.
European Commission (2000). eLearning: Designing tomorrow’s education [On-line].
(Accessed 15/07/02). URL: http://www.europa.eu.int/comm./elearning
European Commission (2001). The eLearning action plan: Designing tomorrow’s education
[On-line]. (Accessed 15/07/02). URL:
http://europa.eu.int/comm/education/elearning/planen.pdf
Hall, B. (2000). eLearning: Building competitive advantage through people and technology.
Forbes Special Advertising Section [On-line]. (Accessed 15/07/02).
URL: http://www.forbes.com/specialsections/elearning/e-04.htm
PART II
CHALLENGES FOR THE ACADEMY

From High Level Clarity to Ground Level Confusion:
Exactly Where do IT-Mediated Education Policies
Fit?
Stephen D. Reeve & Stephen H. Flowers

University of Brighton Business School, UK.
1. INTRODUCTION
Time for inclusive debate on policies regarding the structure and use of
IT mediated learning (ITML) within higher education (HE) is rapidly
running out. Currently, particularly in the UK, institutional debate remains at
an early stage and clear policy is marked more by asymmetry than any
coherent pattern. The pace of technology escalation and installation however
is forcing staff, students and administrators to make some important resource
allocative decisions at national, regional, institutional and departmental
level. Whether foreshadowed in the rhetorical call for a response to the
“mega” on-line institutions in the US (Newby, 2000) or the dire predictions
of a commoditized future (Noble, 1997), there are clear signs that the
benevolent dot.edu bubble may be about to burst.
The overall shift within institutions to ITML leads inexorably to a very
different universe. The kind of environment that actually comes about,
however, will depend very much on how actors at the micro level embrace
this change. The existence of resource availability at higher levels, coupled
with exhortational pressure within an expanding e-economy present a clear
image of change and success about to happen. The results of a more
fragmented and ragged transitions to varying amounts of ITML prefigure a
43
44 Stephen D. Reeve & Stephen H. Flowers
more sinister dynamic, whereby not only there are individual winners and
losers but the whole HE sector itself may stand to lose.
As institutions begin to grapple with the difficult realities of assigning
resources away from traditional practice and into the new, largely uncharted,
arena of technology mediation, stark choices will be faced. For some time
now, the experimentation and pump-priming phase of on-line education has
provided a sense that both new infrastructure and conventional practice may
co-exist. Indeed in most experimental projects, concepts such as learning
enhancement, more effective learning and supportive processes are declared
as the main priority or driver. What may not have been so obvious is the
state of play after the special funding initiatives run out. As we emerge from
this experiential phase of technology use it is likely that the policy
imperatives at all levels within the debate will now begin to change and the
focus of debate will move from the issue of: “Can it be done?” towards the
more fundamental question of: “What value does it add to the educational
process?” In allocational terms, the real opportunity cost of more spending
and investment on IT will emerge, as will clarity as to what will in future be
foregone.
As control of the policy agenda moves from the higher, more explicitly
visionary, levels to the institutional level, complexity sets in. Not
technological complexity per se but political and allocational uncertainty
which trail in their wake the intractable problems often faced by HE
institutions. Powerful questions emerge as reality hits home in terms of cost,
value, cultural change and implications. In an apparently fractal manner,
such a scene is replayed within institutions, as strategies are set and passed
further down towards the practitioner structures – departments, schools,
research groups. Eventually individual academics are reached and finally the
reality of the vision stands or falls according to the engagement, enthusiasm
and actions of the teaching staff. This paper intends to explore this chain of
communication and make some attempt at predicting possible future
scenarios. The aim is to catalyze debate in an area where many feel
disempowered or not competent. Some of the critical and uncertain factors
key to scenario construction lead to images which do not concur with the
more usual “up beat” view of the technological future. To borrow rather
heavily from the world of the media, this “Blade Runner” imagery should be
contrasted with the utopias often represented as the future of IT mediated
education.
From High Level Clarity to Ground Level Confusion 45
2. THE CHANGED CONTEXT FOR DOT.EDU
The opportunities and choices that now face HE must be seen against a
specific contextual backdrop, which has become clearer over recent years
and shares many attributes with commercial and industrial experience. As
the context of HE has become ever more explicitly competitive, the search
for a defining competitive advantage has become more meaningful and the
use of IT within the learning and teaching environment has become prime
candidate. Whether for “first strike” advantage (Reeve & Flowers, 1999) or
simply to avoid competitive disadvantage, HE institutions are seeking to
position themselves as players, either real or nominal, in the evolving
dot.edu environment. Just as no company can be without a web site or an e-
commerce portal, no university can be without an on-line learning initiative.
This is also leading universities to consider with whom they should forge
partnerships from outside of the usual HE environment, so as to be in some
kind of position to take part even though lacking key capabilities; to fall in
with the UK government’s desire that HE should become the mainspring of
the emerging e-entrepreneurial economy; and to obtain potential revenue
from any future “edu sales” which may come about.
As an interesting analysis points out (Monk, 2000), the changed
commercial environment may have had a marked effect on the type and style
of training currently proffered by employers to their employees. Similarly,
the onset of fierce global competition within the HE environment has forced
institutions to consider their modus operandi, illuminating ITML which
resonates with the discourse of the time around the Internet, World Wide
Web, dotcoms etc. In a critical view of the relationship between dominant
rhetoric and actions, Monk poses the awkward question as to the actual use
and effectiveness of flexible training methodologies compared to the
discourse of the time, finding real world activity to be in stark contrast to the
espoused descriptions. The idea that the use of open learning materials
within institutions may not reflect the rhetorical high ground has been
explored before (Reeve, 1993) and such reflections may point to a
discrepancy between the anticipated effects and outcomes of ITML and the
eventual reality. Nevertheless, the direct connection between a changed,
more competitive HE environment and the embrace of ITML as a strategic
tool cannot be ignored.
In this context, following private corporate practice (Strassman, 1997),
universities are pushed toward allocating resources with less than
conventional debate or knowledge, out of a general sense of unease. If
everyone else is doing it and it has become part of the rhetorical and
marketing offer, then there is almost no alternative for an institution other
than to follow suit. Given the resource implications mentioned earlier, it is
not difficult to discern the build up of a wave of IT expenditure that attempts

to reach further and further into previously untouched traditional practice.
However, there is no real clarity that such resource reallocation will pay off
in terms of lower operating costs or effectively higher student numbers or
educational capital sales. Furthermore, there is a danger that the previously
relatively uniform standards of provision quality will begin to splinter with
“struggling” institutions forced to push increasing amounts of finance toward
IT investment hoping to achieve operational cost savings, with continued
insecurity as to eventual pay off. By contrast “successful” institutions, that is
those who have either gained meaningful increases in student registrations,
course fees or have effectively lowered operating costs or all three, will
increase market share, consequent promotion spend and reputation. Such
winners retain the luxury of a balanced portfolio of traditional face-to-face
and ITML provision.
Trapped in the vice between increasing competition and the need to be
more cost effective, the further and deeper IT spending goes, the further
away the institution drifts from being a viable traditional operation. Not only
do the shores of fondly remembered academic practice begin to look distant,
the chances of ever reaching them again become ever more remote. Indeed,
if some of the commentaries as to the eventual cost impact of ITML are
proved to be accurate, parts of the sector will find themselves in a double-
bind where the expected efficiencies and cost reductions prove illusory:
“the thinking was that a school could enroll hundreds, maybe thousands
of students in each online course, collecting huge revenues even while
keeping the per-student cost below traditional tuition levels. In reality
they’ve found that providing a quality online education involves a
tremendous amount of preparation on the part of professors, and it’s
nearly impossible when class sizes exceed 40 … many of the more
reputable online programs cost even more than their traditional
counterparts.” (Los Angeles Times, 2000).
Whether this would go so far as to reach a distinction between an elite
group of universities and the classic “diploma mills” (Noble, 1997; Flowers
& Reeve, 1998), or whether there will simply be a more stretched continuum
of provision, remains to be seen. Proponents of the new systems, typically
early adopters and technology convinced advocates, state clearly that there
are only likely to be benefits from the movement to such allocative choices.
Any reading of the convinced advocates within the distance learning field or
corporate partnership circles (e.g. Thompson, 1999) suggests that the
investment in the new technologies within the learning field can only
enhance learning, lower operating cost and increase registrations. For
beleaguered senior staff having to consider the financial and investment
options, the more likely position is one of confusion combined with a lack of
knowledge and hard data.
In a sense the sector becomes trapped in a kind of “Nash Equilibrium” of
fear, where no individual institution in the absence of others would commit
to the unevaluated spend necessary, yet across the sector as a whole no
institution knowing how the others would act, can do anything other than
commit to that level of spend, afraid of being left behind in the eyes of
potential students, sponsors and political authority. This bounded rational,
and sub optimal, approach will clearly carry implications beyond the
straightforwardly economic, as the evidence is still far from clear that the
learning enhancement so often posited for ITML exists in any meaningful
sense and that distinct dangers in terms of lost social learning, false
democratization and an impoverished virtual experience may lie further
down the line (Flowers & Reeve, 1998).
Unease is further heightened by the pace of on-line alliance
announcements. Whatever the reality, the reporting and PR suggests that
workable, cohesive, alliance-based distance education will work (The
Guardian, 2000) provoking a radical change in public perception. Groups of
nationally elite institutions in close collaboration with global technology
companies present a truly impressive “new kid on the block”. The emphasis
on leading edge activity in transmission/ interaction terms to be gained from
the large network suppliers coupled with the reputations of famous
universities will certainly present a very seductive package. Alliance looks
set to be the theme for the early part of the 21st Century, with groups
forming worldwide, region-wide, inter-sectoral networks of course
construction, mediation, maintenance and delivery.
The more frequently sets of IT mediated alliances of presumably
different classes of provider occur, the more extreme becomes the “lock-in”
of an oligopolistic group of large technology companies to the provision of
HE. A set of global through to local stratified oligopolies could be
envisaged. As natural monopoly, as a result of population size and
geography for institutions, begins to decay and competition intensifies, the
hierarchically organized sets of alliances will cross physical space and
instead be defined by their “brand space”. The most illustrious and famous
will form a premium oligopolistic structure; sets of institutions of lesser
reputations will form lower tiers, with each alliance level catering for the
price level and demand of their “virtual” client group. Such a model may
present some interesting considerations in terms of competition policy, if
markets are defined as within a brand space where by definition there is no
longer competition but overt collusion. Institutions that are not clear about
their role or competitive offering within such a universe may be in some
danger. For example, there is a general sense that there are too many
business schools within the UK. Alliances may be beneficial for the
consumer providing additional subjects, access to different faculty and a
more international dimension. They might also play a role in reducing the
number of UK schools. As the Director General of the Association of
MBAs, the UK accreditation body has been quoted as saying:
“There are far too many business schools in the UK given the diversity of
quality. I think we will see more of these mergers and acquisitions which we
are already seeing internationally, which will then help to cut the number of
UK schools.” (Anderson, 2001)
It is not only the award bearing sector which is exhibiting more overt
marketing signals, the recently announced “virtual college” comprising
Oxford, Princeton, Stanford and Yale is surely an archetype of the new
global premier league (The Financial Times, 2000). Here the non-profit
making “University Alliance for Life Long Learning” will essentially target
alumni of the above institutions in order to provide an ongoing educational
system on-line allowing the 500,000 or so members to “widen their
knowledge and obtain specialist updates” (Princeton University, 2000)
It may be that for any individual student the actual experience is
markedly shallower than the offer of a conventionally based program, but
this will be overridden by the brand of the completed award. Thus the
differential branding may start to become more important than the reality of
the quality of experience or learning offered. As marketeers know well, a
form of tacit collusion will set in whereby students on a branded program
might not be willing to acknowledge weakness in educational or process
terms and an alliance will knowingly be retailing a product whose relative
experiential poverty it is consciously aware of. The more this kind of
collusion spreads, then the more damage is eventually done to the
philosophy and standards traditionally associated and explicitly enshrined
within a previously independent and “unbranded” university sector.
In terms of the policy debate, institutions should be clear on where they
stand in their views on the educational effectiveness of ITML and further
how and where they might factor in many of the other traditional ingredients
of HE, such as the intellectual socialization of (particularly young) people;
emphasis on good citizenship within civil society; the responsible cut and
thrust of emotionally charged debate; and genuinely social or collaborative
learning practice. Despite the continued discussion surrounding the “no
significant difference” phenomenon, there still remains a great deal of
ambiguity as to how well traditionally structured distance learning could
engage with some of the above issues, let alone how a predominantly virtual
provision might cope. Although clearly not at the top of the technologists’
agenda, the implications of generations of students failing to engage with the
wider social dynamics commonly associated with conventional HE might

come back to haunt politicians and administrators later.
3. THE POLICY BACKGROUND – DRIVING THE

DOT.EDU REVOLUTION?
At national policy level the vision of technology enhanced HE is clear

and there has been a strong and well-funded policy steer encouraging
engagement with learning technologies. For many years government and
commerce, the venture capitalists in this scenario, have been content to
pump enormous funds into these activities with a series of interlocking
initiatives and infrastructures designed to support and encourage academics
in the adoption of ITML. At this level of policy, channeled funding and
exhortation are the main imperatives with technology placed at the heart of a
vision of a learning and developing society.
Centrally-funded initiatives such as, in the UK, the Computers in
Teaching Initiative and the Teaching and Learning Technology Programme
(TLTP) channel resources to the dot.edu start-ups and are surrounded by a
whole host of dependent organizations that either advise, support or actively
encourage the use of learning technologies. Such organizations, including
the Association of Learning Technology, Combined Higher Education
Software Team, Joint Information Systems Committee and Electronic
Libraries, are both the brokers and investors in this dot.edu activity and are
largely dependent on its continuation. The dot.edu investment analysts are
organizations such as the Universities and Colleges Information Systems
Association (UCISA) whose aim is to promote high standards of education
particularly in the provision and development of computing and information
systems in UK universities and acts to liaise with the major funding, research
academic and administrative bodies within the sector. The position of these
bodies is clear and is neatly summed up by a statement of the then UCISA
Chair, who said:
“New applications of communications and information technology have
the power to transform the future provision of HE. This will require
ongoing investment in infrastructure at national, regional and local levels.
It is unlikely that increased use of IT will substantially reduce overall
costs, but the rewards in enhanced quality and improved access to HE
will be huge.” (Price, 1996).
At the national and sectoral policy level the vision is thus clear and well
articulated.
Moving to the institutional level the picture becomes more complex as

differential adoption rates, strategic aims and competitive contexts result in a
patchwork quilt of activity. Examples of such activity within the UK include
the creation and implementation of an “on-line learning environment” at
Coventry University; regional initiatives, such as the Scottish Clyde virtual-
U project; and transnational programs, such as Europace 2000. Perhaps the
most significant indicator would be the government’s commitment to, and
part funding of, the e-university within the UK. In contrast to such wholesale
technological upgrading is the more cautious encouragement of broad-based
innovation with ITML, with smaller numbers of specific projects supported
through a central Learning Technologies Group. Such a “wait and see”
approach, may benefit from the learning of first movers, but may
demonstrate all the characteristics of differential rates of adoption.
As the focus of enquiry moves to the level of individual academics the
picture becomes far more complex. Here the issues change from the
infrastructural, exhortative or facilitative to the rather more mundane issues
surrounding the adoption and implementation of specific technologies. At
this level the issues are no longer about the potential of ITML to
“revolutionize HE”, rather it is about the specific value such technologies
can add to their work. It is at this level that the self-referential R&D activity
and the fractal policy positions that exist at national, sectoral or institutional
level break down and the pixilation of policy is complete. In effect, a large
number of faculties have simply not engaged with the policy debate
surrounding the use of ITML at all.
At national level, partly in response to the lack of progress at grass roots
level, policy positions have begun to soften with calls for “pedagogical
research to inform the educational use of C&IT” in the provision of evidence
to the National Committee of Inquiry into Higher Education (Price, 1996)
and the recognition that:
“New technology has an important part to play, but is only one method of
delivery. Exclusive use of new technology could exclude some learners
(the poor, those without the necessary IT skills). We believe that a multi-
stranded approach is needed” (UCISA, 1998).
The changing focus of the TLTP is a clear demonstration of this shift.
Established by the central funding council in 1992 the first two phases of
TLTP funded 76 projects to develop new materials across a wide range of
subject areas and examine the implementation of learning technologies
within HE. The third and final phase of this program was launched in 1998
funding a further 32 projects over three years, but with a specific remit to
focus on “implementation, embedding the use of new technologies into HE
and evaluating its effectiveness”.
4. NEW STRUCTURES EMERGE
Perhaps ironically, just as the national attempts to drive the creation of

embedded and diffuse dot.edu environments may be faltering to some degree
as “grand strategy”; the appearance of newer, less tangibly central forms of
activity, in some cases outside of the traditional HE sector, are beginning to
emerge. The area of partnerships between institutions, private companies,
and voluntary bodies is clearly one much vaunted in the financial press. The
partnership between the London School of Economics, Stanford, Chicago,
Columbia, Carnegie Mellon and Unext.com provides a clear example of both
the tiring of an elite transnational system and the potential of such a
partnership to operate globally should it so choose in the teaching and
learning field. (Bradshaw, 2001). The connections with a large publishing
and educational software house and its connections on into a particular
platform model, demonstrate the possible strength and depth of such
alliances as indicators of future models. Market width is generated laterally
through the alliance whilst an almost complete IT based educational supply
chain flows downstream and out. Specifically elite business schools:
“are faced with the need to internationalize, develop strategic alliances
and use technology effectively while protecting their intellectual
property, the one thing that differentiates them from other schools…”
(Bradshaw, 1999).
The purer forms of corporate universities are also evolving stronger links
with the traditional sector. Despite being outnumbered some 18:1 by their U
S counterparts (The Financial Times, 2000), European corporate universities
look set to consolidate and expand. The Cap Gemini Ernst and Young
University for example has links with Insead, London Business School and
Cambridge. Other links among leading European corporate and public
universities would include the Lufthansa School of Business with Insead,
London Business School, McGill and (in China) CEIBS; the University for
Lloyds TSB, with Warwick and Nottingham Trent University.
As other business schools rush to join alliances and partnerships, further
structures involving global consultancies, IT companies and HE institutions
may be envisaged. Once such structures are established as precedents, the
field will widen beyond the remit of business education and encompass
many other traditional fields and disciplines. Such spontaneous
developments, whilst outside of specific policy control, may well receive the
largesse and interest of national and regional Government. For example the
link between Cambridge and MIT received £70 million of British state
funding (Kelly, 1999) and was not without controversy. A new kind of
supply chain is being formed as companies like Thomson, McGraw Hill and
Pearson, long involved as academic publishers, purchase and create course

management companies and mediation channels so as to provide end-to-end
on-line provision in an e-learning market estimated to be worth $11.4 billion
by 2003 (Mayfield, 2000). Clearly, this may or may not feed into
conventional HE institutions. There is some degree of symbiosis, as such
publishing based giants still need the reputation of the HE delivery sector as
their outlet, and the sector will increasingly need the convenience and after
sales provided by such a supply chain, but once again branding may
eventually play a part in terms of who gets what and who is involved with
whom.
The contribution expected of HE by governments as “arms of the
knowledge based economy” (Watson, 2000) may allow the proliferation of
structures and systems that flourish at the expense of elements of the more
traditional sector. It is only once certain aspects of the sector have
disappeared, or changed beyond recognition, that the overall systemic loss
might be accounted for in terms of access, provision and quality. As such an
overall sectoral balance sheet is unlikely. There are dangers in the
incremental modifications in provision, as more and more educational output
becomes “privatized” and “virtualized”.
That ITML has a fundamentally important role in teaching and learning
within HE in the future is not in doubt. The importance of this issue places it
at the centre of the debate over the future role and practice of HE. The
government has funded the creation of a world-class communications
infrastructure and has also funded successive initiatives in this area in order
to ensure that the UK remains at the forefront in ITML use. The HE sector as
a whole strongly supports this policy and a wide range of initiatives is
underway within individual institutions. It is only when the focus of enquiry
moves to the faculty level that it becomes clearer that the adoption of such
technologies is far more uneven and that it is here that the real challenge for
change exists. However, given the rapid alliance building and development
of “for profit” and independent e-learning institutions, it may be that this
need to change becomes both more urgent and, paradoxically, entirely
irrelevant as a tide of commercial dot.edu activity sweeps over the HE
sector.
5. THE DOT.EDU PHENOMENON – SOME

ALTERNATIVE PERSPECTIVES
A potentially challenging perspective on the adoption of IT in teaching

and learning is provided by the work of Moore (1991) and Rogers (1995),
amongst others. If this activity is viewed as an example of the adoption of a
technological innovation, it provides an insight into some of the challenges

that face any attempt to embed IT into the mainstream activities of HE.
Moore and Rogers propose a well-defined pattern in the way in which an
innovation is likely to be adopted and that adoption by certain groups will
proceed through a predictable sequence. For the sake of simplicity these can
be summarized into three main groups: early adopters, mainstream and
laggards. As a group early adopters will include individuals who are willing
to try out any new idea, often for its own sake, as well as those who will look
for ways to make dramatic improvements by applying it to core activities.
This group accounts for about 16% of the total population. In contrast the
mainstream group is more conservative, made up of those more interested in
incremental change and the concrete benefits that can be obtained, as well as
more skeptical individuals who are likely to be late adopters. This group
accounts for about 66% of the total population. Finally, the laggard group is
very resistant to change and is only likely to adopt an innovation very late, if
at all. This group accounts for about 16% of the total population.
Applying this model to the adoption of ITML within HE it could be
argued that much of the activity, and thus the comment, discussion and
policy advice is flowing from the early adopter group. However, for the
innovation to succeed it must be taken up by those in the mainstream, a
move that is by no means automatically assured. Indeed, the differences in
needs and wants are so great between these two groups that the possibility of
a “chasm” between early adopters and the mainstream (Moore, 1991) has
been proposed. It is into this chasm that many high-technology innovations
have fallen as attempts are made to move the technology from early adopters
to the mainstream.
However, there are causeways and bridges being launched across this
chasm, which themselves bring new challenges. In the face of potential
chaos and the mixed record of experimentation and enthusiastic amateur
activity in the realm of IT based education, senior administrators are being
offered a clear, coherent new product. Platform e-learning systems present
order where there is confusion; hierarchy where there is anarchy; and
emphasize manageability and simplicity where there is dangerous
complexity. Above all, standardization is being offered to institutions:
“NextEd has built and operates a fully integrated, scaleable and
replicable software platform. It comprises course teaching and learning
software, complete online campus functionality, electronic
communications, student activity tracking, e-commerce applications, call
centre/IVR technology for service and support, and software for online
marketing including database mining …” (Hartland, 2000, p. 1).
There are relatively few platform providers and all contractual

relationships exhibit degrees of sunk cost and “Williamsonian
transformation” with its attendant dangers for the unwary institution in terms
of switching cost and exit strategies. However, the clarity of service provides
a bridge over the maelstrom of the chasm that not only mitigates some of the
confusion to be found there but actually channels the mainstream into a set
of processes and procedures determined by the shape and operation of the
chosen platform itself.
Choice of platform would seem to emphasize brand, that is of one
company’s product over another, but what should not be overlooked, and is
of huge importance here, is that the issue is fundamentally one of
standardization of course provision. In effect this is an industrial model,
finally arriving in a sector that has managed to hold on to craft traditions for
a very long time. It is the industrialization of course provision in HE which
the platform systems provide, with all the one-way, revolutionary, paradigm
shifting effects. This is of far greater significance than the mere choice of
brand as to the “look” of education in institution “x” or “y”. Thus, Moore’s
chasm may be being artificially and rapidly breached. The hearts and minds
of the mainstream may not be being won over, but their approaches and
practices are being industrialized, possibly without debate, possibly without
agreement and maybe without anyone really realizing.
More recently the concept of the “blended” program has become a
dominant theme within advanced or complex corporate training
environments as a solution to the “thinness” of web based distance
approaches, when compared to the richness of video production
environments. Ironically, it is the politics of education which may be forcing
the proprietary e-learning platforms down the same blended path within HE
institutions, as institutional roll out of a particular platform concurs with the
rhetoric of support for, not replacement of, traditional face-to-face
methodologies. Despite the educational sense and value of such a
compromise, it should not be overlooked how easily a functional system,
when in place as a support structure, can be positioned in such a way that it
becomes the single or main vehicle for virtual provision. The industrial logic
of the “automated” system is then in a position to drive out relatively
expensive traditional provision even within a given institution.
Another perspective on adoption of ITML within HE is provided by the
McFarlan and McKenney (1983) strategic grid. The strategic grid is a long-
established and widely used framework within the information systems
management field for focusing attention on the issue of the balance of
expenditure within the IT portfolio. A version of the grid is shown in figure
1.
LOW future importance of L&T technology HIGH
Support Turnaround
current importance of L&T technology
L&T technologies that may be improved L&T technologies that have potential
but are not critical to future success strategic importance
Factory Strategic
L&T technologies that sustain the L&T technologies that are critical for
existing operation future success
HIGH
Figure 1: Strategic L&T Technology Matrix.
The strategic grid is useful for classifying an organization’s portfolio of

information systems and, as a result of further discussion, making strategic
decisions as to where investments should be made and may thus be a key
enabler in determining the shape of the future IT system portfolio. The
strategic grid can help to demonstrate that investments are not being made
into core systems or that there is the potential for further speculative
investments within the overall portfolio.
In this context it may be usefully re-named the strategic learning and
teaching technology matrix since it will be used as a means of placing ITML
within the overall portfolio of learning and teaching methods currently in use
within HE. It could be argued that, for very many HE institutions, the
traditional learning and teaching technologies of the lecture, seminar and
tutorial all reside in either the support or factory quadrants. In contrast, the
new web-based interactive approaches are firmly located in the turnaround
quadrant since although they are undoubtedly of huge potential importance;
they have yet to demonstrate clearly their critical strategic importance.
Perhaps more interestingly, it could also be argued that the strategic quadrant
is largely vacant for very many universities.
This apparent paradox is possibly at the core of the debate surrounding

the role of ITML within HE. Universities have traditionally concentrated far
more on the quality of the content delivered rather than the quality of the
method of delivery, the focus was on the message, not the medium and
competition was based on the quality of research and substance of what was
taught, not the quality of lectures. The emergence and growth of web-based
approaches has meant that there is a huge impetus for this traditional
relationship to change such that the medium is elevated to assume a very
high level of importance in the educational process. Whilst it is to be
expected that within the innovator community the medium largely is the
message, this shift in priorities does raise a series of fundamental
institutional questions, not least amongst these is the ability to maintain and
update an increasingly expensive IT-based learning infrastructure in an era
in which (potentially) institutions are judged as much on the quality of their
medium as on the quality of their message.
In an impressive attempt to analyze the costs and benefits of ITML, Luke
declares:
“Digitalization costs more not less; it takes more people not fewer;
optimal class size on line often falls, not increases. Computer technology
is a rapidly obsolescing permanent cost, not long paying investment.”
(1998, p. 32)
This is from a convinced advocate of IT based learning, but demonstrates
the scale of the debate yet to take place. Further, the move to compete on the
basis of technology-based systems in an era of diminishing resource and
without special initiative funding can only raise significant internal resource-
allocation issues. Given this situation, the demand for promoters of ITML to
clearly demonstrate the educational value of such systems in order to justify
the resources they will consume will be overwhelming.
6. CONCLUSION – THE EMERGING POLICY

FUTURE
The similar patterns of investment and expectations between the bull

phase of dot.com companies and much of the activity surrounding ITML
within HE are striking. Both areas promised huge future returns but
effectively deliver little if anything in the present. Very large amounts of
money have been pumped into both with little regard to the normal rules of
investment appraisal, and both are driven by a powerful dream that is fuelled
by the media and the innovator and early adopter groups. Despite a “shake-
out” in the private e-learning sector, as the after shocks of the dot.com
collapse rumbled into the early 2000’s, all the signs are that the market will
continue to grow strongly. New supply chains are being formed; old players
are in networks with new organizations; and platform providers are moving
into key and powerfully strategic spaces.
Many of the strands identified in this paper, when taken together, point to
an uncertain future for HE in its classic role. Although a great deal of the
discussion around ITML focuses, quite rightly, on the exciting, the new and
the potential, some of the very processes pulled in train by this dynamic
bring standardization, hierarchy, one-way structural change and
rationalization. Further, the focus on commercial provision, large
oligopolistic organizations and commoditization levers HE into new and
potentially uncomfortable relationships with private sector commerce and
industry. Content may “thin out” as it is standardized; room for lively debate
may be narrowed; and the social and civilizing roles for HE may decay to
make room for much more specific knowledge training.
The danger exists for HE to walk potentially, in a trance, to the edge of a
cliff and then step off, without anyone consciously desiring this to happen
yet all are powerless to stop it. The standard, and perhaps only, available
form of mass HE would be virtual and probably cobbled together with
components or bits from the production labs of the branded players. Face-to-
face conventional practice will have reached a critically small size and with
consequent loss of scale economies, become infeasible for non-elite
institutions; leaving high quality/high personal interaction as a premium
priced activity available to a relative few at prestigious sites. In some senses,
HE would never have been more available, yet individual experiences of it
would never have been so poor.
HE may now be at the end of the beginning of the IT era. Enough has
now become known to allow practitioners to contribute effectively to policy
development. HE should not and cannot match the speed of response, nor the
privately focused purpose of the “for profit” dot.edus; rather the
concentration should be on the further development of sound and enabling
infrastructures, which support and enhance, but not automate or replace, the
wider roles and duties of effective teaching and learning within our
institutions. Here is the focal point and crucible for informed debate as to
how this revolution is to be managed. If this debate does not ignite in the
short time remaining, the opportunity will have been lost to moderate or
control the industrial dynamic which will overwhelm traditional practice. It
is time academic and technical staff started to ask awkward questions,
initiate discussion, and evaluate relevant research in an effort to take hold of
the debate and influence their future. We hope to have made some useful
contribution to that debate.
REFERENCES
Anderson, L. (2001). Survey - business education UK: Alliances set to rise as a chill wind
blows. The Financial Times, Sep 10th.
Bradshaw, D. (1999). Schools drawn into new webs, The Financial Times, Oct 11th.
Bradshaw, D. (2001). UNext looks for a better blend, The Financial Times, Dec 24th.
The Financial Times (2001). Corporate Universities Xchange. Corporate University Awards
[On-line]. (Accessed 24/07/02) URL: http://www.corpu.com
Flowers, S.H., & Reeve, S.D. (1998). Positioning web-based learning in the higher education
portfolio: Too much too soon? In L. Lloyd (Ed.), Teaching with technology: Rethinking
traditions (pp. 135-151). Florida: Rollins College Press.
The Guardian (2000). Universities forge alliance for global teaching on net, Feb 15th.
Hartland, S. (2000). Global alliance to deliver on-line education. RMIT University,
Melbourne [On-line]. (Accessed 24/07/02) URL: http://www.rmit.edu.au/html
Kelly, J., (1999). Universities to gain from role in economy. Financial Times, Dec 1st.
Los Angeles Times (2000). A good online education is anything but cheap and easy, Mar 3rd.
Luke, T. (1998). Digital discourses, on-line classes, electronic documents: Developing new
university techno cultures. Paper presented to Learning Online ‘98: Building the Virtual
University, Roanoke, VA, June, [On-line]. (Accessed 24/07/02) URL:
http://ultibase.rmit.edu.au/Articles/oct98/luke1.htm
McFarlan , F. & McKenney, J. (1983). Corporate information systems management: The
issues facing senior executives. New York: Dow Jones Irwin.
Mayfield, K. (2000). Publishers yearn to e-learn. Wired News, Sept 18th [On-line]. (Accessed
24/07/02). URL: http://www.wired.com/news/culture/0.1284.38771.00.html
Moore, G.A. (1991). Crossing the chasm. New York: Harper Business.
Monk, D. (2000). New training methods: A giant leap of faith. In L. Borghans, W. Gijselaers,
R. Milter, & J. Stinson (Eds.) Educational Innovation in Economics and Business V (pp.
31-55). Dordrecht: Kluwer Academic Publishers.
Newby, H. (2000). The Guardian, Mar 21st.
Noble, D. (1997). Digital diploma mills, Part I. [Online]. (Accessed 24/07/02). URL:
http://www.communication.ucsd.edu/ dl/ddm1.html
Price, M. (1996). UCISA submission to Dearing report Press release Nov 15th [On-line].
(Accessed 24/07/02). URL: http://www.ucisa.ac.uk/docs/pr/pr1.htm
Princeton University (2000). Press release. Office of Communications. Stanhope Hall, Sep
20th.
Reeve, S.D. (1993). Open learning as curriculum strategy. In N. Graves (Ed.) Learner
managed learning: Policy, theory and practice (pp. 149-156). Leeds: Higher Education for
Capability.
Reeve S., & Flowers, S. (1999). The Shape of Things To Come: a Discussion on the Nature of
the Technologised Higher Education Institution. Paper presented to 10th annual CTI-AFM
conference. Brighton.
Rogers, E.M. (1995). Diffusion of innovations (4th ed.). New York : The Free Press.
Strassmann, P. (1997). The squandered computer: Evaluating the business alignment of
information technologies. Connecticut: The Information Economics Press.
Thompson, M. (1999). The effectiveness of distance learning. Penn State World campus [On-
line]. (Accessed 24/07/02) URL: http://www.worldcampus.psu.edu/pub/news/
effectiveness.html
Watson, D. (2000). The business of borderless education, Closing address to QEII Conference
Centre London, March.
UCISA (1998). Response to the learning age – A renaissance for a new Britain [WWW
document]. [On-line]. (Accessed 24/07/02). URL: http://www.ucisa.ac.uk/docs/
reports/11resp.html
Moving a University Toward On-line Learning:
Opportunities, Challenges, and Technologies
Drew Parker & Andrew Gemino

Faculty of Business Administration, Simon Fraser University, British Columbia, Canada
1. VIRTUAL CLASSROOMS
The concept of the “virtual classroom” has been emerging for decades.
The roots of the concept began in the post-war correspondence phenomenon.
Since then, technological advances in distance learning have increased
possibilities and modified the form of delivery of the virtual classroom.
Current developments in computer technology and telecommunications have
accelerated this rate of change. It is now possible to offer a fully interactive
course, globally, using a variety of multimedia options. Universities have
begun to offer courses based on these structural options. Traditional place-
based institutions are embracing the web as a tool to support classroom
instruction and many institutions are ramping up on-line courses to
supplement traditional post-secondary offerings. Further, completely on-line
programs within traditional universities, and even completely virtual
universities, are coming onstream.
2. ISSUES SURROUNDING THE VIRTUAL

CLASSROOM
Potential advantages of distance learning appeal to the looming crisis

faced by contemporary universities in Canada. Funding cuts have led to
fewer resources; greater demands for accountability; and newfound
61
62 Drew Parker & Andrew Gemino
competition by private educators and other “virtual” groundbreakers.

Queen’s University (http://business.queensu.ca - Accessed 10/06/02) in
Canada now offers its Executive Master of Business Administration (MBA)
programs via teleconferencing at locations across Canada, Asia and Europe.
Athabasca University (http://www.athabascau.ca/ - Accessed 10/06/02)
offers complete programs with technology support, as do several
international institutions, such as the University of Phoenix
(http://online.uophx.edu/ - Accessed 10/06/02) in the U.S.A. and the Open
University (http://www.open .ac.uk/frames.html - Accessed 10/06/02) in the
United Kingdom. Most universities now have a growing distance education
component. The debate over the appropriateness of course delivery options
without face-to-face contact can be a heated one (e.g. see Noble, 2000),
particularly if university policies are not updated to consider new issues
arising from alternative teaching configurations. Issues such as faculty
workload, quality of the educational product and intellectual property rights
offer mixed benefits to different institutional stakeholders. Pioneers in this
area have been tending toward the implementation of entire programs (e.g.
Simon Fraser University’s Graduate Diploma Program in Business
Administration at: http://www.gdba. sfu.ca/gdba/ - Accessed 10/06/02),
rather than offering a mixed mode of place-based and distance programs.
There have even been “for profit” startup projects in Canada offering
Internet-based programs exclusively, often without the traditional
accreditation process. Several hybrid offerings, however, are beginning to
emerge.
3. BACKGROUND
Colleges and universities make up some of the longest standing

institutions in the world. Government and post-secondary education have
changed drastically, but the fundamental structure of a contemporary
university has changed minimally. As a place for learning, traditional post-
secondary institutions remain widely accepted as the de facto educational
instrument. Universities and colleges have survived large changes in the
educational environment, to some degree, by maintaining a tension between
traditional time-tested pedagogy and the need to adapt to changes in
competitive institutions, students and public and private funding. Borrowing
from Porter and Millar’s Competitive Forces Model (1985), figure 1
highlights the factors that influence the delivery of education by post-
secondary institutions.
Moving a University Toward On-line Learning 63
Threat of New
Entrants
Power of
Public
Funding
Bargaining
Power of Universities
Students
Power of
Private
Funding
Threat of
Substitutes
Figure 1: Competitive Forces Affecting the Educational Environment.
It is important to note that these forces impact upon the university in a

particular environment as defined by available technology. A discussion of
the strength and importance of each of these factors requires large
generalizations across institutions and is well beyond the scope of this paper.
Instead, the focus will be to discuss impacts that changes in technology have
on the relative strength of the forces. Viewed in this way, the Internet and
the birth of on-line offerings is a fundamental technological shift that will
require universities and colleges to adapt to this new technological
landscape.
4. THE IMPACT OF TECHNOLOGY
Universities have been adapting to technology from their beginning.

Writing instruments, paper, ink, lights and text books all initiated changes in
the delivery of education. Chalkboards, a mainstay of educational delivery
for hundreds of years, provided an increase in the ability to express and
temporarily store an instructor’s ideas. The mail system enabled post-war
universities to move into correspondence programs when demand for
education exceeded the supply of classrooms. Overhead projectors and
microphones enabled universities to respond to the increased demand for
education by enabling large increases in student to teacher ratios. The
telephone and television further enabled the delivery of distance education.
The advent of personal computers and graphic applications, such as
Microsoft PowerPoint, provided instructors with a tool to deliver more

polished presentations that were also more easily stored, retrieved, modified,
and shared. In all these cases, the technological change enabled universities
to respond to the forces that were present in their environment.
It is important to note that throughout all of this change, what remained at
the core of the delivery, and remains in large part today, was the concept of a
lecture and a classroom where a group of students saw and listened to an
instructor at the same time in the same place. While correspondence and
distance education drove a wedge into the concept of same time - same
place, the Internet has clearly challenged this concept. Universities have
little choice but to adapt to the new technology. Figure 1 provides a
foundation for describing how the Universities will make this transition.
4.1 The Internet and Students
Students are increasingly familiar with computing technology. It follows

that students will want to interact with the University using the web. With
the advent of the Internet, students now have more power to demand more
technology to be used to deliver knowledge. The “digital divide” argument
suggesting Internet users will have different communication opportunities
and options is waning, as today over 83% of Canadians over the age of 15 at
least have access to email (http://www.statcan.ca - Accessed 10/06/02).
4.2 The Internet and New Entrants
Completely on-line courses do not require faculty buildings, heat, light

and other related overhead. Considering only physical infrastructure costs,
the argument has been made that on-line computing can potentially lower
fixed costs associated with course delivery. Assuming equal variable costs,
new universities may be able to successfully compete with traditional
universities using larger amounts of on-line components. This success may
be short-lived, however, if traditional Universities respond. The first
Canadian university to enter into this arena, Unexus University, had a short-
lived tenure of less than three months.
4.3 Current On-line Offerings
With the increased technological literacy of students comes greater

demand for on-line components even in traditional place-based
environments. In addition network technologies are blurring the lines
between close-up and distance education. In response to these forces,
traditional universities have been growing their distance education

components and providing more on-line teaching and learning opportunities.
The key difference between distance and place-base is interactivity. New
modes of delivery are emerging all centered around the concept of on-line
conferencing.
Business administration faculties are among the earlier adopters of IT-
enhanced programs. It could be argued that, since the material under study
incorporates IT within its pedagogy, experimentation with IT-enhanced
delivery is a natural evolution. Piccoli et al. reinforce this assertion:
“Technology savvy students and instructors are early adopters of technology
and represent a high proportion of users of web-based courses.” (2001, p.
402). This lead, however, will be short-lived if anticipated successes are
realized and incorporated into institution-wide programs. Alavi and Leidner
conclude: “For a university that desires to truly embrace the digital age,
greater coordination of acquisition, support, and knowledge sharing is in
order.” (2001, p. 8).
4.4 On-line Conferencing
On-line conferencing tools fundamentally support the creation and

management of discussion groups. Tools offered include organized
discussions, such as weekly topics or modules, data about participants, such
as short resumes and personal information, and organization tools, such as
grade records and access to files for up and downloading. This area is newly
supported by tools available on the Internet, which are evolving rapidly.
4.5 Virtual Seminars
The correspondence course format, which emerged after the Second

World War, most closely serves as a forerunner of a contemporary virtual
seminar. The virtual seminar differs from the correspondence course because
the objectives, interaction and timing are more closely representative of a
regularly scheduled classroom seminar. Students can submit assignments
according to particular due dates, exams can be offered and participation can
be expected and graded as a component of the course itself. The Internet has
offered many novel opportunities to handle transactions and educational
delivery is not exempt. The distinction between distance education and on-
line offerings is being blurred by the growth of innovative technological
options and creative new approaches to teaching.
4.6 Hybrid Models
Novel technological opportunities are providing an increasing number of

hybrid alternatives. Seminars can be enhanced with media, such as on-line
activities, notes and multimedia materials distributed in various ways,
including the web. Faculty members are then made more available through
telephone contact, electronic mail, some form of computer supported
conferencing, or on-line chat/discussion sessions. Further along this
continuum is a course where actual seminar content is substituted by a form
of on-line learning. Several computer conferencing applications exist to
support this concept, such as First Class (First Class Systems
http://www.centrinity.com - Accessed 10/06/02) or the Virtual University
(Simon Fraser University http://virtual-u.cs.sfu.ca/vuweb/ VUenglish/ -
Accessed 10/06/02). These applications offer a comprehensive environment
for on-line discussion based seminars. The seminars can be enhanced by
technological support, providing levels of access to learning materials and
faculty similar to that available to a placed-based seminar. Differences are
substantial in terms of preparation and delivery, but learning objectives and
possibilities appear, based on preliminary results, to be very similar.
5. EXPERIENCES
In the spring term of 1996 the Faculty of Business Administration at

Simon Fraser University offered a “virtual” seminar. This was the first
offering of an upper division Business Administration seminar through a
format managed and run jointly by the Faculties of Business Administration
and Continuing Studies. Previous offerings were limited to lower division,
where a student is in their first two years of undergraduate study and have
not yet been admitted to a degree specialization, such as the Faculty of
Business Administration’s Bachelor of Business Administration. Upper
division courses are specialized in the Faculty and offered to senior
undergraduate students at least in their third year. The course was offered as
a one-time experiment. The application was considered successful and the
course was offered for the second time in this format one year later.
This experiment made sense for Simon Fraser University for several
reasons. First, the timing was appropriate. The Internet, and the web
specifically, were maturing to a stage where they could be depended upon to
support a regularly scheduled series of communications. Students were also
sufficiently knowledgeable about technology and the web to use such an
application without undue time and effort being spent on the mechanisms of
communication rather than course content. They also had access to
microcomputer technology sufficient to handle the required

communications, either off-campus or on-campus for a few who chose this
format in a residential format. The issue of student-owned computer systems
was both an opportunity and a challenge, in that the lack of access to
standardized computer facilities required a “lowest common denominator”
approach to novel technological options.
Simon Fraser University is one of the largest co-operative post-secondary
institutions in Canada. A “co-operative” program is one in which a student
completes approved work terms in conjunction with their academic studies.
A work term is completed and reviewed by the university for accreditation
toward a degree designated as “co-op.” Commensurate with this, the
majority of the students attending the institution are non-traditional, in that
they do not take a full course load for eight months of the year and do other
activities in the summer. An issue that had been expressed by both students
and faculty was the separation between work and studies during a four or
eight month co-op term. Several students expressed an interest in taking one
or two courses during this term, but often were working schedules or
locations that would preclude regular trips to the Burnaby mountain campus.
The ability to “telecommute” was a strong motivating factor for the
consideration of novel forms of educational delivery.
5.1 Building a Virtual Seminar
A virtual seminar tends to be built around some form of conferencing

system. In the 1996 offering of the seminar, FirstClass was the conference
system of choice. It offered both direct dial up and Internet access, with
many redeeming features. First, and probably foremost at the time, it was
reasonably stable. With 25 students accessing the system 24 hours per day a
minimum of twice per week, a 13-week term was handled without any non-
scheduled down time. Of the 25 students, one was located in Europe.
Internet access was critical for this individual. A significant advantage of
FirstClass was its ease of use. FirstClass was an early entrant into the market
as a comprehensive software tool to support an on-line learning
environment. FirstClass had an overall management view allowing a series
of course activities to be encapsulated into a single location for a student or
instructor to place and manage on-line course activities. It was, at the time,
text based and offered conferences where participants could post and
respond to messages; an interactive chat facility for synchronous
discussions; and the ability to place a personal resume on the system. Key
disadvantages were the inability to download any new additions to be read
off-line and the requirement that a software client had to be downloaded and
installed in a student’s computer. Students had to either participate
completely on-line or download discussions to be read one at a time.

Another drawback was the inability to directly reference web-based
materials. There was no interface between FirstClass and web browsers.
The lack of integration with the web proved cumbersome and, in some
cases where Internet service providers were used with a connect time charge,
expensive. The second disadvantage experienced was the inability for
FirstClass to offer dynamic hypertext links to an Internet browser. If a web
location was identified, the best a conference participant could do was
highlight the location, copy it to the clipboard, switch to an Internet browser,
and paste the location into an access command. The interface proved flexible
and sufficient for course dialogues, although exhibits were supplemented in
a paper-based study guide, which was distributed at the outset of the course.
In addition to the conferencing system, office hours were offered in both
place-based and virtual formats. Internet relay chat was used to provide
interactive access among participants and faculty. A discussion channel was
opened with scheduled times that could accommodate both the Canadian and
European students.
FirstClass was very oriented toward collaborative learning, with such
tools as on-line resumes where students could both learn to use the interface
by completing a familiar task and post information about themselves to
personalize subsequent dialogue. The lack of media richness, however,
eventually led to a decision to move toward the web. The first experiment
had been fruitful, and many elements of the general approach remained the
same.
5.2 Subsequent Iterations
After deeming the experiment a success, the decision was made to

continue to offer this course once per year in the virtual format and twice per
year (in the other two trimesters) in a traditional place-based format.
Additionally, a marketing course was designed for a virtual offering in 1998
to experiment with generalizability of the format across different course
specializations. Similar results were obtained. Finally, the notion of a
complete program offered entirely on-line was considered. This notion
required considerable planning and discussion, but was eventually embraced
by the Faculty. The program is a stand-alone graduate diploma program
designed for a niche market. Simon Fraser University’s full-time MBA
program requires an undergraduate degree or equivalent in business
administration, but there was considerable demand from graduates of other
programs. Additionally, some professionals expressed interest in the core
subjects commonly offered in an MBA program without the need for the
degree itself nor the desire to become a full-time student for an eight or
twelve month period. The MBA was streamlined, eliminating some

foundation courses for non-business majors, and the foundation courses were
designed and offered as a separate, stand-alone Graduate Diploma in
Business Administration (GDBA). Courses were first offered in the spring of
1999. The program currently continues to succeed in its objectives (the
program website is at http://www.gdba.sfu.ca - Accessed 10/06/02). Future
courses are envisioned in the undergraduate program and some graduate
courses have been discussed as possibilities for a partial or total virtual
format.
5.3 Format Observations
The courses were offered in the same thirteen-week term as a place-based

offering. Lecture material was posted in a text and graphic format for the
students to read. Each week the students would read the required chapter(s)
in the text and the corresponding lecture material. Students were assigned to
groups of three or four to moderate one week. The moderators would post a
“top ten” list of the issues or concepts they felt were most important for the
week. The other students would then each comment on this list. Toward the
end of the week the moderators would offer a summary of the dialog and the
professor would wrap up the discussion; open a new dialog box for the next
week’s material; and post the lecture notes. The subsequent week’s group
would then have the weekend to prepare their introduction to moderate.
There were also assignments based on techniques throughout the term.
Students would learn a particular technique as part of the weekly material
and be given an assignment to complete using that technique. In this
particular course, the techniques were mostly diagramming tools to support
the role of a systems analyst, such as a data flow diagram or a use-case
model. The students would submit assignments by fax, email, mail or hand-
delivery to the university.
Most of the communication was synchronized only by the week. Students
could participate within 24-hour windows, which suited both individual
schedules and preferences and international time zones. One weekly “office
hours” session was scheduled where students could interactively ask
questions and comment on the material or structure of the course. This
proved a valuable and popular addition to the format. It also facilitated a
social component to the course, an element of considerable concern to both
students and faculty members (e.g. see Parker & Rossner-Merrill, 1998).
FirstClass was used in the 1996 offering, followed by experimentation
with a proprietary web-based interface called Virtual University in 1997.
The 1998 and subsequent offerings returned to FirstClass and subsequent
offerings are planned in the GDBA program using WebCT
(http://www.webct.com/ - Accessed 10/06/02). The undergraduate offerings

remain inconclusive with regard to platform selection. The ability to
download material for study and interaction off-line remains a major
objective, enhanced by a dedicated web server where on-line interaction is
necessary. Microsoft Exchange Server (http://www.microsoft.com -
Accessed 10/06/02) is currently being considered for implementation in
2002, supported by such tools as Netmeeting11 for office hour interactions
and PowerPoint-based tutorials.
5.4 Students’ Performance
Recent studies show that on-line students tend to perform at least as well
as place-based counterparts in many situations (Parker & Gemino, 2001;
Starr-Hiltz et al., 2001). Results from both the FirstClass and Virtual
University offerings were similar. The students in the virtual seminar had an
above average performance compared with their place-based counterparts in
the written examination segment, but did not do as well in the technique-
oriented exercises.
On reflection, both of these results make intuitive sense. The “forced
march” weekly participation necessitated the students keeping pace with
readings and participating at a much higher level than would be expected or
accepted in a place-based setting. For example, if each of 51 students were to
be required to read out loud the comments they personally posted weekly,
they would require between five and ten minutes. Ignoring the management
overhead required for the dialogue, this dictates between four and nine hours
per week of student commentary. Add the lecture material to that list and
you have the equivalent of a two-day seminar rather than the three-hour
vector allotted weekly in the place-based session. Conversely, the place-
based course includes a one-hour tutorial weekly in small groups where
students can ask questions and collaboratively work on techniques, such as
drawing technical diagrams. The lack of this interaction led to a comparative
weakness in this portion of the on-line course. This difference was
hypothesized and tracked, subsequently showing statistically significant
results (Parker & Gemino, 2001).
With student input, efforts have been made to enhance the technique
components of the virtual seminar. Research suggests that differing learning
techniques, such as dual-coding theory where both visual and sound stimulus
are offered simultaneously, can potentially enhance the learning process
(Paivio, 1971). To utilize this, tutorials are being developed on the web using
Microsoft PowerPoint with voice-overs to summarize and provide examples
of techniques to be learned. An example of this is on the web at
http://parker.bus.sfu.ca/bus362/dfdnarrated.htm (Accessed 10/06/02).
The participation component in the grading scheme had some interesting

implications. First, it caught the attention of the university administrators
very quickly. A correspondence course with a participation grade did not fit
the stereotypical offering and warranted attention. Considerable concern was
also experienced with a readings package, normally well over 100 pages, of
only a few pages containing a course outline and on-line access instructions.
A second issue was the ability to grade more objectively on participative
performance. A student, for example, who received a lower grade than
would normally be allocated for average participation could be shown that
they were graded fairly on participation when they did participate, but that a
couple of weeks of inactivity were costly in this format. Several discussions
ensued in the course offerings, with none of them leading anywhere after a
review of participation calculations and verification of postings in the
conferences.
5.5 Findings
A key stumbling block for on-line design is in the initial motivation for
the format. Confusion among the objectives of innovation, economies of
scale and reach lead to challenges in justifying on-line offerings on an
efficiency basis when effectiveness gains were the more likely result.
Moving a course on-line, in our experience, rarely saves net resources.
Benefits fall into the following categories:
• Physical space is saved.
• Reach, or the ability to access a course remotely, is facilitated.
• Discussion opportunities and requirements per student are enhanced.
• The opportunity to meet and learn about globalization in virtually any
context emerges by having a global classroom contingent.
• Guest lecturers and other course visitors can be accessed globally.
Issues affecting conversion of a course to a completely on-line format
include several key decisions. First, class size is an important attribute.
Socialization and discussions can increase dramatically from a traditional
seminar. Experience suggests groups of approximately 25 students can get to
know one another on-line and work effectively to drive a discussion.
Scalability requires decisions about the role of the teacher and discussion
leaders and the format of each learning module. Second, group work on-line
needs to be reexamined. Groups can manage a discussion well, but
assignments tend to be handled individually. New technologies offer
opportunities to change this, but they remain preliminary.
Selecting a course to move on-line remains more art than science, but
three key parameters stand out. First, the content to be learned must be
examined. Concepts, rather than techniques, are better suited to an on-line
discussion format. Second, the learning method requires course designers to

choose how to motivate learning on-line. A course can, for example, adopt a
problem-based approach to learning as opposed to a cognitive flexibility
approach. Finally, the way that learning materials are presented, the
presentation method, must be examined. Presentation methods include text-
based material, discussions, animations or, in certain situations, streaming of
audio or video.
6. MOVING A UNIVERSITY ON-LINE
Several policy-oriented issues came forward in the implementation of

these course offerings. The remainder of this paper will outline these
experiences.
6.1 Selecting an On-line Component
How to teach effectively is a complex topic. Similarly, research is

starting to suggest that, as mentioned earlier, conceptual material is more
easily suited to an on-line format than technique. Many courses require a
balance between these forms of learning. Determining optimal forms for
information presentation in each format will take time, experience and
experimentation. The possibility of hybrid offerings is also starting to be
considered by various institutions. Questions such as: “What are the learning
requirements for this course?” and “How can on-line interaction supplement
or replace face-to-face modules?” must be asked.
In creating a virtual seminar with an interactive, socially rich component,
both format and corresponding technology choices need to be made.
Technology changes rapidly, but most interaction can be categorized into a
framework surrounding the Internet and web. Components of the course
offerings within Simon Fraser University can be summarized as:
• Lecture material. Typically a combination of text and graphics that can
be downloaded and read asynchronously by students.
• Supplemental tutorial materials. These explain the function of the course
and can include techniques that are placed on the web for interaction in a
multimedia format.
• Weekly conference. Courses run through the standard trimester system.
Students read and participate in a conference where issues related to the
weekly topics are discussed and moderated by a group of students. Our
experience limits participants to 25 students per conference. Multiples of
this are divided into separate sections with limited cross-posting
allowed.
• Office hours. The professor is available for synchronous communication

for a fixed time frame each week. This allows for interaction and
common questions to be sorted out.
• Virtual Visitors. Using audio streaming tools, guests can be interviewed
on-line. The guest is interviewed by the professor over a telephone and
students listen over the web and can either call a toll-free number or post
questions on-line.
• Private chats. Students often have group assignments or will wish to
have a discussion with a limited number of people on a specific topic.
The professor typically keeps an office chat facility open for questions
and discussion. Private matters can be handled by email. Moderators, for
example, will use a chat facility to prepare their weekly material for
presentation to the larger group.
6.2 Corresponding Technological Options
No single tool supports all elements of an on-line course. Past experience

has suggested that there is a need for a single utility to locate course
discussions and conferences, but that a suite of tools will likely offer the best
set of opportunities for a virtual seminar. Students must have sufficient
technology to handle course requirements, but this issue is waning rapidly
with the literacy and access of adult learners. The suite of tools typically
follows the applications outlined above:
• Conferencing software. This needs to be affordable, reliable and easy to
learn and use. Offerings include FirstClass, WebCT, Microsoft
Exchange Server, Lotus Notes and Virtual University. Features sought
include web integration, off-line storage and interaction capabilities,
multimedia and interactive chat support.
• Interaction software. Students interact in small groups and with the
professor in office hours. Text has proven sufficient most times. Higher
speed access and more multimedia standards for computer systems are
offering opportunities for voice and video streaming, but bandwidth and
discussion management become complex issues with audio or video
conferencing.
• Broadcast Software. Not all elements of a course need to be two-way.
Lecture notes, tutorials, and guest speakers are all handled in a broadcast
medium. The web facilitates much of this, through web pages that can
include text, graphics and audio and video streaming.
6.3 Universities and Policy Change
Arguably, university environments offer an arena for experimentation,

research, and learning. The form and structure of the educational process,
however, has changed little. Seminars are offered using chalkboards,
whiteboards, overhead projectors or even computer-generated slides and
animations. The format, however, remains place-based in a central university
setting. Several discoveries were made when the idea of a virtual seminar
was originally proposed.
Continuing education tends to be for non-credit offerings. On-line course
expertise tends to reside in the Continuing Studies Department of the
university, but faculties own for-credit courses. Articulation with other post-
secondary institutions is common, but typically for lower-division courses.
The course offered in this experiment was an upper division offering in the
Faculty of Business Administration, and had to receive special dispensation
to be offered through the Department of Continuing Studies.
Faculty offer courses through Continuing Studies on a contract basis.
Regular teaching load is done for courses offered in the home department or
faculty. When this seminar was proposed, the question of contact hours
became particularly contentious. The Dean of the Faculty of Business
Administration had to make an exceptional decision. Policy remains
ambiguous on this point.
Development time and effort is largely ignored. Experience has dictated
that the Continuing Studies model tends to incorporate consideration of the
time it takes to conceptualize and design on-line courses, whereas faculties
and departments tend to consider the design of new formats or offerings as
part of a professor’s regular duties. This tendency, left unmanaged, will
create a considerable barrier to the design and offering of on-line or virtual
courses, since the status quo format is known, understood and accepted.
Foreign students create another new policy problem. Simon Fraser
University, for example, can allow foreign students to apply through the
traditional registration process or they may be allowed to audit a course with
permission of the Faculty member. Since auditing does not allow for a letter
grade and the registration process is lengthy and expensive for a single
course, neither facility is suited to single-course, off-campus visiting
students. Fees and registration procedures need to be reconsidered in light of
the possibility of a foreign student taking a single course for transfer credit to
their home institution without ever being resident at the offering university.
This issue had to be resolved, for example, by creating an identically named
course at both institutions and offering them simultaneously.
A related issue is the notion of the core of a university. If upper division
courses can be articulated across global university offerings, the question
becomes: “At what point does a student no longer have a degree from a
particular university?” A university needs to consider whether they must
provide a core teaching program or if they must perform a host quality
control role. The hypothetical extreme case would be where a student
registered in a university and took a degree program entirely through
articulated transfers, many on-line. A residency requirement would be
unable to define this potential problem, so new thought and corresponding
policy is required.
Electronic access to a university is currently an issue getting considerable
attention. Modem or cable access is expensive for the university to support
and “normal” standards are typically set. Current conferencing alternatives
either have considerable on-line time requirements, such as FirstClass or
Virtual University, or have per-unit software and related technology costs
that may prohibit high-volume undergraduate student use, such as Lotus
Notes (IBM).
7. CONCLUSION
Virtual seminars are a reality. Some universities exist with remote

programs exclusively and traditional universities continue to offer place-
based seminars in centralized settings. Hybrid teaching models, however, are
now possible. These models utilize new hardware, software and
telecommunications developments and offer much promise for a rich,
effective, and efficient educational delivery format.
The major learning curves needing to be managed are the support and
reward structure for the design of virtual seminars by faculty members;
measurement of the effectiveness of alternative modes of delivery for
particular course types; infrastructure support for this mode of education;
and continued research and development into the area. These issues will be
forced to the fore by both technology push and by demand-pull. Technology
continues to offer richer alternatives, while competitive pressures demand
that these formats be considered by faculty and administration of modern
universities. Potential benefits are the ability to compete in a global
educational market; the ability to keep a relationship with increasingly
mobile students; and enhanced cost/benefit of physical plant and related
university infrastructure.
REFERENCES
Alavi, M., & Leidner, D.E. (2001). Research commentary: Technology mediated learning – a
call for greater depth and breadth of research, MIS Quarterly, 12 (1), 1-10.
Noble, D. (2000). Digital diploma mills. In J. Turk (Ed.) The corporate campus:
Commercialization and the dangers to Canada’s colleges and universities (pp. 101-121),
Toronto: James Lorimer and Co Ltd.
Paivio, A. (1971). Imagery and cognitive processes, New York: Holt, Rinehart and Winston.
Parker, D., & Gemino, A. (2001). Inside online learning: Comparing conceptual and
technique learning performance in place-based and ALN formats, Journal of
Asynchronous Learning Networks, 5 (2), 64-74.
Parker, D.C., & Rossner-Merrill, V. (1998). Socialization of distance education: The web as
enabler. Paper presented at WebNet ’98 Proceedings, Association for the Advancement of
Computing in Education, Orlando, FLA.
Piccoli, G., Rami A., & Ives, B. (2001). Web-based virtual learning environments: A research
framework and a preliminary assessment of effectiveness in basic IT skills training, MIS
Quarterly, 25 (4), 401-426.
Porter, M.E. & Millar V.E. (1985). How information gives you competitive advantage,
Harvard Business Review, 63 (4), 149-160.
Starr-Hiltz, R., Coppola, N., Rotter, N., & Todd, M. (2001). Measuring the importance of
collaborative learning for the effectiveness of ALN: A multi-measure, multi-method
approach, Journal of Asynchronous Learning Networks, 4 (2), 1-20.
Linking Pedagogical Innovation and Information
Technology to Enhance Business Education
Sergio Vasquez Bronfman

ESCP-EAP, Paris, France
1. EFFECTIVE LEARNING AND THE ROLE OF

INFORMATION TECHNOLOGY
Business education has long been suffering from a gap between theory
taught and daily professional practice (Schön, 1983; Schön, 1987;
Mintzberg, 1988; Mintzberg, 1996; Spinosa, Flores & Dreyfus, 1997). As a
business school professor I am committed in my research and practice to
bridge this gap. I think that our professional activity is an opportunity for
innovation and hence I design and implement educational experiences to add
value to business education by using educational technologies and applying
active learning methods (Hacker et al., 1996; Vasquez Bronfman, 1998;
Vasquez Bronfman, 2000).
The use of IT in business education has been reported by many scholars
in recent years (Leidner & Jarvenpaa, 1993; Alavi, 1994; Knoll & Jarvenpaa,
1995; Ives & Jarvenpaa, 1996; Alavi, Yoo & Vogel, 1997; Angehrn &
Nabeth, 1997). Most of these studies report the automation of information
flows between teachers and students or among students themselves and
address questions of software and hardware architecture, IT infrastructures
or logistics issues (EDEN, 1998; NTICF, 1998). These automation efforts
may have led to efficiency gains but they have not led to warrantable
improvements in learning. In Papert’s words, instructors and decision-
makers in the field of education seem to be immersed in a techno centric
perspective. “ techno centrism” is the fallacy of referring all questions to
77
78 Sergio Vasquez Bronfman
technology (Papert, 1990a; Papert, 1990b). In the field of educational

technologies, techno centrism gives rise to questions about the number of
computers per student, the number of computers connected to the Internet
and the functionalities of the e-learning platform. In educational technology
research, techno centrism leads to questions such as: “Will the Internet have
this or that effect on management learning, by comparison with a traditional
classroom?” “Will a CD-ROM lead to a mechanical method of thinking in
accounting?” Scholars then run evaluation studies comparing, for instance, a
course delivered via the Internet with a traditional face-to-face course,
identifying IT as the only variable and assuming that the hypothesis “all else
being equal” (ceteris paribus) is true. Of course it is false, because in fact
everything changes: the professor, the students, the classroom and the
technology.
What is missed in techno centric thinking is learning. Instead of focusing
on IT questions, we have to concentrate on learning. And, from a business
school professors’ perspective, we have to focus in particular on business
learning. That leads us to address the following questions: “What is effective
learning (in general)?” and “What is effective business learning (in
particular)?”
Concerning learning in general, I build on pedagogical perspectives such
as constructivism (Piaget, 1985; Piaget, 1992), constructionism (Papert,
1990b; Harel & Papert, 1991) and the work of well known educational
thinkers such as Vygotsky (1985) and Bruner (1987; 1996). Constructivism
is based on the assumption that knowledge is created by learners, rather than
transmitted by teachers like information in a pipeline, and that they discover
and construct meaning from their environments. Constructionism suggests
that learners are particularly likely to create knowledge when they are
actively engaged in making something that is also personally meaningful and
they can share with others, such as video games, robots, computer
animations and written stories or, closer to business learning, e-commerce
web pages and export plans.
Constructionism is also close to the work of Vygotsky (1985) and Bruner
(1987; 1996), in the sense that both state that learning is a social process and
stems from cooperative activities, from making something collectively.
Moreover, they state that effective learning occurs when this process
happens within transactions between learners and members of their culture
more experienced than them, hence leading to the concept of coaching,
mentoring, etc.
In addressing the question of effective business learning, we need first to
have an interpretation of what business really is. Concerning management
(one of the disciplines of business administration), I agree with Drucker’s
stance that “Management is a practice rather than a science. It is not
Pedagogical Innovation and Information Technology 79
knowledge but performance” (Drucker, 1974). I also share Mintzberg’s view

that management is not a technical profession, certainly not a science, not
even applied science, but a practice, a job (Mintzberg, 1996). Following his
ideas on the nature of management, Mintzberg says that effective
management learning must be turned to practice, to the cultivation of skills
(Mintzberg, 1988). I am convinced that this interpretation can be generalized
to all business disciplines (strategy, finance, marketing and sales,
information management). This interpretation of business and of business
learning lead us to a new question: “What is effective learning when one
needs to learn a practice?”
To answer this question I have been inspired by the educational thinkers
already quoted and especially by Schön’s ideas on reflective practice and on
education of reflective practitioners (Schön, 1983; Schön, 1987). Schön is
well known for his remarkable work on practitioners’ education. He
observes that there is an artistry which is inherent in competent professional
practice, an art of problem statement, an art of implementation, an art of
improvisation, and that this artistry contains, but goes beyond, the rationality
of applied science and of research-based techniques. Therefore, the question
of how to learn this artistry is posed. Schön then states that when we observe
how competent practitioners learn their artistry we find different educational
traditions that stand outside or alongside the normative curricula of
universities and business schools. There is athletics coaching; apprenticeship
in industry; conservatoires of music and dance; and studios of visual and
plastic arts. In medical schools there are interns and residents learning under
the guidance of senior clinicians by working with real patients on the wards.
In this kind of learning, students cannot be taught but they can be coached by
experienced practitioners while involved in a project or other meaningful
activity.
In order to train students in the artistry of competent practitioners, Schön
suggests that they must enter a “practicum”. This is a setting designed for the
task of learning a practice. A reflective practicum is a practicum aimed at
helping students acquire the kinds of artistry essential to competence in the
indeterminate zones of practice. The main goal of the practicum is to learn
the know how of practitioners, as opposed to the knowledge accumulated in
the field, which is the goal of traditional university education. In a
practicum, or studio for learning, students run a project and gain knowledge
of the project’s field by being involved in solving the problems posed by the
project itself. In other words, students are immersed in a “learning by doing”
environment under the guidance of senior practitioners.
We have now a robust framework of ideas in the field of business
learning. Summarizing this framework we can say that effective business
learning needs a learning by doing environment where students make things
collectively, tackling real problems under the guidance of experienced

practitioners; where they can share ideas with others hence working in
teams; where coaching helps students reflect on their projects and
assumptions; where lecturing felicitously complements learning by doing by
opening students to new and powerful interpretations of the reality they are
facing.
It is within these new educational perspectives where questions about the
role of IT in enhancing business education must be addressed. I think that
information technologies are not educational tools, they are just tools. As
every technology in history, information technologies are possibilities-
openers. They make new things possible hence stimulating our creativity in
how we use the technology. Therefore, a good question concerning business
education is: “What new possibilities can we invent with IT in order to add
value to business education?” For instance, information technologies, and
especially the Internet, offer us the possibility of:
• accessing information wherever the information source is and wherever
people seeking for information are;
• transferring information to distant people wherever these people are and
regardless of whether they are connected at the moment the information
is transferred;
• asking distant people to make something;
• sharing ideas with people regardless of time and distance constraints.
In order to bridge the gap between theory and practice in business
education, I have designed and implemented some educational innovations
using information technologies. This paper reports my action research on
these experiences and address my main research question: “How can we link
IT and active learning methods in order to add value to business education?”
Three of these learning experiences will be reported and discussed here.
They are:
• a case study on sexual harassment discussed on-line by students of the
Ecole Supérieure de Commerce de Paris (ESCP), City University
Business School (CUBS), London and California State University at
Long Beach (CSULB), California;
• an import/export project between ESCP students and Mexican students
at ITESM;
• an on-line course in leadership intended for practitioners committed to
the process of development in Latin-American countries.
Two of them use IT to enhance face-to-face education, while the last one
is a distance learning course. These learning experiences are part of the
fieldwork of my research program which is to apply the educational views
described above to virtual environments and to observe how they have to
adapt.
2. RESEARCH METHOD
Because I am interested in transformation and not simply in knowledge, I

chose the action research approach to study these IT-supported pedagogical
innovations. Action research may be defined as action and research (Gill &
Johnson, 1991; Gumesson, 1991; Dick, 1998). Action in order to run a
change process, here active learning methods using IT, and research in order
to gain knowledge of the field. Instead of a hypothesis to test I have
outcomes to achieve (e.g. to train students in the import/export business, to
make them enter new work practices using the Internet). In order to do this, I
need to gain knowledge of the field, such as the main benefits of the project
for students, the problems they encountered, the perceived transformations
of the professor’s role and the effectiveness of the learning process.
In order to enhance rigor in our research, we need a declared-in-advance
epistemological framework. Following Checkland and Holwell (1997) we
have to declare a Framework of ideas (F), a Methodology (M) and an Area
of concern (A). In my research program the A is a real-world problem
situation, namely: “How can we link information technologies and active
learning methods to add value to business education?” This area of concern
is studied within the framework of ideas (F) I have summarized above, with
action research being my chosen M. Action research projects take the form
of case studies with data being accessed through participant observation;
analysis of the messages sent by students and professors; students’ final
presentations; interviews with students; and evaluation questionnaires.
3. CASES AND FINDINGS
3.1 Nancy and Sam
3.1.1 Case Description
In 1994 I implemented the Nancy and Sam on-line case study because I
was interested in the possible uses of the Internet to enhance my information
systems courses at ESCP. My intuition was that this new tool could allow us
to run projects between students located in different universities and
business schools throughout the world. I wanted my students learning new
work practices supported by IT (e.g. IT-supported distant teamwork) and
thinking about new organizational possibilities (e.g. network organizations)
opened up by the Internet. But above all I wanted them to learn from direct
experience, not simply from my lecture on these topics. In other words,

learning by doing rather than teaching by telling.
In discussions with my colleagues at CUBS, we thought of running a
case study discussion via the Internet. Finally, three business schools were
involved in this learning experience: ESCP in Paris, CUBS in London and
the CSULB in California, who provided the case. In California and London
participants were MBA students, with some years of professional practice.
CSULB students were participating in the case in the context of a human
resources management course, while CUBS students were involved as part
of an information management course. In Paris, we started with
undergraduate students, around 20 years old, without any professional
experience, and in the context of an information systems course, but in 1997
and until the end (autumn 1998) we moved this learning experience to
students of an intercultural management course at Masters level.
The case study describes a supposed situation of sexual harassment in a
Californian bank. Sam is one of the vice-presidents. In his late thirties, he is
handsome and, since his divorce, he has had many liaisons but he is always
reluctant to commit. Nancy is a 26 years old executive, attractive, who lived
with her boyfriend for two years but has not a stable partner since the
breaking-off of their relationship. She likes the good life (a nice house on the
sea front, a Toyota sport car, etc.), but she has some cash flow problems. She
has a very good professional reputation. Sam knows that Nancy’s boss will
retire soon and he thinks she is well suited for the job. She is obviously very
interested in this promotion. Sam then proposes that they go together to La
Jolla for a bank staff meeting; he will book two rooms at the hotel and after
the meeting they can discuss Nancy’s promotion. But she feels
uncomfortable about this proposal and makes an excuse. Sam then takes her
out for dinner in a pizzeria close to the bank. The dinner goes well and they
talk about everything but Nancy’s promotion. At the end of the dinner, Sam
asks Nancy to have a last whisky at his place in order to discuss her
promotion. The case study question is: “What would you do if you were
Nancy?” Students must answer the question and discuss the issues raised by
the case through participation in an Internet forum with students from the
two other locations.
The case takes four to five weeks. Students meet their professors once a
week in lectures and they connect to the electronic forum outside of
classroom hours. Initially (1994, 1995) they interacted via a mailing list
accessed through standard e-mail software. Progressively we implemented
different technologies: web forums, web mail packages, etc.
Students always interacted quickly and frequently. After a week’s
discussion things are very clear. There is no agreement on what Nancy
should do and, what is more, on the issue of sexual harassment. American
students think that this is obviously sexual harassment and the only question
to discuss is if Nancy should sue Sam tomorrow or later. By contrast, French
students, both men and women, do not see what the problem is. The British
have an intermediary point of view, but one that is closer to the French.
At this point one of the three professors summarizes the discussion and
poses new questions to the students. While it is obvious that there is no
agreement in the interpretation of facts, it is interesting to observe that these
facts are the same for everybody. Why is there this difference in the
interpretation of facts? The professor suggests looking at the differences in
law concerning sexual harassment; the influence of religion in society; the
local cultures in relation to sexual affairs; and related matters. Students find
and report important differences between France, UK and the USA in these
matters. Usually, the discussion then moves towards a debate on flirting at
work. Is it acceptable? What are the “rules of this game” in the workplace?
French students cannot imagine how flirting at work can be outlawed. By
contrast, American students do not want to mix business (work) with
pleasure.
Finally, after four or five weeks of discussion and while there is still no
agreement on what Nancy should do, professors pose further questions.
Imagine that a French Sam goes to California to work in this bank or that
Nancy goes to Paris to work in the French branch of this bank, what should
be the banks’ policy on the issue? Which culture should have priority, the
local culture or the multinational company’s culture? The case ends with a
summary of the answers to these final questions, with a large majority of the
students agreeing on the primacy of the local culture.
3.1.2 Main Benefits
At ESCP evaluation was undertaken systematically during the first two

years (1994-95 and 1995-96) of this learning experience by means of a
qualitative questionnaire completed by the students. From the beginning
until the end of this experience I also analyzed the messages sent by the
students to the discussion forum. In the early years the case helped students
discover the Internet. They learnt how to use it and, by using it, they
discovered the new possibilities it offered for organizational design and for
new work practices. For the duration of the learning experience, students
became aware of the cultural differences between French and American
people with regard to the matters raised by the case, hence becoming aware
of the importance of intercultural management issues.
The on-line discussion also helped students understand that
“communication” is not the same as “transmission of information”. As a
matter of fact, huge amounts of information were transmitted over the
Atlantic and the Channel, but it did not lead to an agreement on what action
should be taken. Students learnt from direct experience that it is one thing to
exchange information and a very different one to communicate; that is to
share a context and take common action.
In order to be successful in this kind of learning experience “one must
think learning, not technology”. The on-line case study was a huge success
because it put pedagogy first. Technology is not an end in itself; rather it is a
tool that opens up new possibilities. Students experience the Internet as a
tool that allows them to consider new work practices, a tool that enables
them to think about intercultural management issues. For professors, the
Internet is a tool that allows us to create a “microworld for learning” (Papert,
1980). Microworlds are explorative learning environments, a small, but
complete, version of some domain of interest that one can explore and
construct. They may well be conceived as a play area that gives students a
chance to experiment, without risk, with concepts that do not otherwise exist
in the world in that combination. Therefore, professors did not have to
describe management problems in an intercultural environment, nor stories
of network organizations. Instead students explored the field in the micro
world and constructed their knowledge by participating in the on-line
discussion. It was on the basis of this discussion that professors could
facilitate critical reflexion and make theoretical contributions.
3.2 Doing Business between Mexico and France
3.2.1 Case Description
Following an initiative of Prof. Carlos Ruy Martinez, from ITESM

(Monterrey, Mexico), I implemented (with the collaboration of other ESCP
colleagues) a practicum for learning international business called Doing
Business between Mexico and France. This practicum is aimed at training
students in the art of import/export consulting. In accordance with my
educational perspective, the way we do this is by involving them in a
learning by doing experience. More precisely, students have to find a
company for which they can prepare an export plan. Hence, at the end of the
three month course students will be able to say to the market: “we know
something about import/export consulting, not mainly because we have been
taught it, but because we have already done it!”
In Paris, students were ESCP full-time students from a Major in
International Business (having more than 10 nationalities), or students from
a Masters in International Project Management. ESCP students met their
professor once a week in an international marketing course and spent
between 30 and 60 minutes discussing the developments of the project. In

Mexico there was a very different scenario. Students were Executive MBA
students spread throughout the Mexican Republic. They attended courses in
the different ITESM locations and participated once a week in an
international marketing course taught at a distance through a
videoconference where Doing Business between Mexico and France was the
main project for credit.
Students worked in teams, usually three to six students in each. The
objectives of the project were to prepare an export plan for a company
wanting to export to Mexico or to France, with students acting as export
consultants, and to help a foreign company (French or Mexican) to export to
Mexico or to France, with students acting as import consultants. Students
used the Internet and other sources of information in order to find relevant
information on the Mexican and French markets. They also found, on the
course web page, a template of an export plan.
For information that is impossible to find on the web (e.g. retailers, price
policy), students developed collaborative working via the Internet (e-mail,
forums) with their “other-side-of-the-ocean” partners. For instance, French
students asked their Mexican colleagues about good retailers for the product
they were trying to export to Mexico and suggested advertising policies.
Students should also have sent a sample of the product to their counterparts
in order to test its cultural acceptance via focus groups.
To help the students in dealing with delays, there was a detailed schedule
on the web page of the course. Students were required to send advancement
reports to their professors every two to three weeks. These reports, and the
weekly meeting, helped professors to coordinate the whole project between
themselves and with students.
We ran this project twice a year from Fall 1997 until spring 2000. Each
time there were 30-35 students in Paris and around 20 students in Mexico.
3.2.2 Evaluation
As expected, students reported that the main benefit of this project was
the fact of learning through practice. “It’s a project in touch with reality”;
“It’s not just an academic exercise but something real” were typical
comments. Concerning the specific benefits of working with the Internet,
students said that they learnt to develop intercultural relations on the net.
I was especially interested in discovering if students perceived that they
had learnt the know-how of import/export practitioners, as opposed to
learning only the accumulated knowledge in the field. The majority of the
students reported that they had learnt the know-how of practitioners and
valued this as one of the main achievements of their learning experience,
because by doing hands-on work rather than reading about it they could gain
much more knowledge and skills and retain them. However, some students
stated that time was too short for them to become “real” practitioners, while
others limited their learning to “knowing how to preparing an export plan”.
Nevertheless, students faced some problems that practitioners encounter,
such as lack of information, decisions based on feelings and difficulty in
understanding their counterparts. This is consistent with the research
literature which shows that for practicing managers information is not
provided in advance in a well-structured report, but has to be collected and
then interpreted against its social context (Mintzberg, 1988; Anghern &
Nabeth, 1997).
Finally, time constraint was a recurrent complaint. Three months are
probably too short when students are also involved in many other academic
activities. Therefore, time extension was the main improvement they
requested.
3.3 An On-line Course in Leadership
3.3.1 Description of the Learning Experience
This course is completely on-line. It is delivered via the Internet and

interactions between people are facilitated electronically. It was intended for
practitioners committed to the process of development in Latin-American
countries. It was offered in Fall 2000 by the International Institute of
Governance based in Barcelona, as part of it’s Virtual School of
Governance, and was operated by the Open University of Catalonia. Almost
30 students, spread throughout Spain and Latin America, followed this
course for a duration of almost two months.
“How can one apply our pedagogical framework to a completely on-line
course?” Usually, on-line courses are organized as a sequence of web-based
documents and a series of traditional assignments in support of these
lectures: questionnaires, exercises, etc. Active learning methods, such as
case studies, discussions and reflexion-on-action (i.e. a learning process
where the learner thinks back on a breakdown encountered when performing
an action thus questioning his/her assumptions.), have a marginal place in
this approach to on-line learning. Our main source of innovation here was to
bring what was marginal to the center (Spinosa, Flores & Dreyfus, 1997).
Therefore we structured the course as a set of individual and collaborative
activities that prompt students to reflect on the nature of leadership and on
the skills needed to become leaders of their communities.
For their first activity students were required to introduce themselves via
the forum of the course and write about 20 lines on their first thoughts about
what leadership is, thereby starting a short discussion on this topic. The
second activity was to identify people they considered to be leaders, around
them (at work, in their families, among their friends, at school or at the
university) and in the world. After identifying these persons, students had to
send a message to the forum describing the identified leaders and telling
others why they look upon these persons as leaders, how their behavior
makes them leaders, etc.
The third activity started by reading a paper summarizing different views
on the nature of leadership (e.g. those of Weber, Drucker, De Vries, Kotter,
Bennis). This was followed by a new discussion on students’ first thoughts
about leadership in the light of what they read in the paper. Students were
then given a second paper which provided them with a new interpretation of
the phenomenon of leadership, a linguistic approach to leadership, based on
the work done by Flores and his colleagues (BDA, 1996), thereby enriching
the discussion in the forum. Finally, a third paper, The Basic Competences
of the Leader (BDA, 1996), was sent to students and they were asked to
observe these competences in the persons they had identified as leaders and
then report their observations in the forum. The last activity was to evaluate
themselves as leaders in the light of what they had learnt throughout the
course. In this way we tried to achieve the main goals of the course, namely
to change the views that students had of the phenomenon of leadership and
make them aware of the skills they need to learn in order to become leaders
of their communities.
3.3.2 Evaluation
Even though the majority of the students actively participated in the

discussion forum, some did not participate very frequently. Moreover, a
number did not participate at all. The main problems faced by students
seeking to participate were:
• lack of technological infrastructure (hardware, software,
telecommunications) in order to properly connect to the server of the
Open University of Catalonia. This was particularly true in the case of
some Latin-American students.
• lack of time to connect frequently to the forum of the course, due to
overwork periods or new work priorities, that made it impossible for
them to keep up with the pace of the course.
In spite of these difficulties more that 20 students participated actively in
the discussions that structured this course. These students reported
interesting ideas on leadership; made clever observations on the persons they
identified as leaders; and contributed realistic evaluations of the

competences they needed to develop in order to become leaders of their
communities.
4. MAIN LESSONS LEARNED AND CONCLUSIONS
4.1 Comparing Practice with Theory
At the beginning of this paper I set out a framework of powerful ideas to

underpin the design of effective IT-supported learning experiences. I said
that “effective business learning needs a learning by doing environment
where students make things collectively tackling real problems under the
guidance of experienced practitioners; where they can share ideas with
others hence working in teams; where coaching helps students to reflect on
their projects and assumptions; where lecturing felicitously complements
learning by doing hence opening students to new and powerful
interpretations of the reality they are facing”. Within this framework of ideas
I also stated that information technologies can open up possibilities, such as
access to information wherever the information source is and sharing ideas
regardless time and distance constraints. I will now see whether the learning
experiences described in this paper match with this framework.
In the Nancy and Sam case students were actively participating in an
international discussion, a learning by doing environment where they were
tackling real problems through an Internet forum. Hence, they were sharing
ideas with others regardless time and distance constraints. Professors met
their students once a week in classroom and could then lecture to introduce
them to new interpretations on intercultural management issues.
The Doing Business between Mexico and France project was a real
practicum, where students had to run a real project, a learning by doing
environment where they were tackling real problems and working in teams
(doing things collectively, sharing ideas with others). Either on-line or when
meeting students once a week in classroom, professors coached them in
preparing their export plans. Thus, students were doing things under the
guidance of experienced practitioners and coaching helped students to reflect
on their projects. The Internet allowed students to work with their
counterparts on the other side of the Atlantic by using e-mail and
asynchronous fora to transfer information to distant people wherever these
people are and regardless of whether or not they are connected at the
moment the information is transferred.
In the leadership on-line course, students were observing people they

considered to be leaders, then reporting their observations and ideas to the
course’s forum (i.e. learning by doing, sharing ideas with others). The
professor was constantly participating in the discussion providing new
interpretations of the leadership phenomenon. Thus, coaching helped
students to reflect on their assumptions, opening them to new and powerful
interpretations. Obviously, without the Internet this course could not exist.
The Internet opened up all the possibilities described above.
4.2 Changes in the Role of Professors
As expected, the most immediate impact observed was on the role of

professors, moving from an information disseminator to a guide, a facilitator
and a coach. Instead of giving lectures and supervising case study
discussions, professors moved from one who teaches to one who facilitates;
from one who transfers information to one who coaches teams through
presentations and partial reports; from one who is an academic to one who is
a senior consultant helping juniors to cultivate skills; from one who marks
assignments to one who helps students to reinterpret their daily practice.
Also, professors must always be supportive in answering students’ questions,
either face-to-face or by e-mail, hence spending probably much more time
with them than in a traditional lecture. This change of role may be difficult
for many academics because they have to show their “knowing-in-action”
every time they meet their students (Schön, 1987). They have to show the
knowledge of the academic and the know-how of the practitioner.
Over the coming years business education will face many challenges to
the traditional ways of learning. The use of information technologies is seen
as a panacea for meeting these challenges. I strongly believe that there is a
risk for the quality of education because when implemented within a
traditional educational framework, information technologies can reinforce
the worst features. This paper has shown that there is an alternative
approach, shared (at least partially) by other scholars (Knoll & Jarvenpaa,
1995; Alavi, Yoo & Vogel, 1997), which provides an answer to the main
research question of this paper (“How can we link IT and active learning
methods in order to add value to business education?”). Summarizing, it is
necessary to reject techno centrism and to put pedagogical innovation first
while making good use of the potential of IT. In addition, I have provided
clues as to how to apply the innovative educational views described in this
paper to virtual environments and how they have to adapt. In order to make
progress, further research is needed on collaborative learning at a distance
via the Internet and on distant coaching techniques.
REFERENCES
Alavi, M. (1994). Computer-mediated collaborative learning: An empirical evaluation. MIS
Quarterly, 18 (2), 159-174.
Alavi, M., Yoo, Y., & Vogel, D.R. (1997). Using information technology to add value to
management education. Academy of Management Journal, 40 (6), 1310-1333.
Anghern, A. & Nabeth T. (1997). Leveraging emerging technologies in management
education: Research and experiences. European Management Journal, 15 (3), 275-285.
BDA - Business Design Associates (1996). Leadership. Unpublished paper.
Bruner, J. (1987). Comment les enfants apprennent à parler. Paris: Retz.
Bruner, J. (1996). Education, entrée dans la culture. Paris: Retz.
Checkland, P. & Holwell, S. (1997). Information, systems, and information systems : Making
sense of the field. London: Wiley.
Dick, B. (1998). Applications. Session 1 of Areol - action research and evaluation on line
[WWW document]. [On-line]. (Accessed 26/07/02) URL:
http://www.scu.edu.au/schools/sawd/areol/areol-session01.html
Drucker, P. (1974). Management. London: Heinemann.
EDEN (1998). Universities in a Digital Era. Paper presented at 7th EDEN Conference.
University of Bologna, Italy.
Gill, J. & Johnson, P. (1991). Research methods for managers. London: Paul Chapman.
Gumesson, E. (1991). Qualitative methods in management research. Newbury Park, CA:
Sage.
Hacker, K.L., Lewis, R., Treadwell, T., Rich, M., Vasquez Bronfman, S., & Wignall, D.
(1996). Building Inter-Campus Computer-Mediated Communication: A Case Study of
COMCONF and Its Possible Contributions. Paper presented at the Conference of the
Western States Communication Association, Communication and Instruction Interest
Group. Pasadena, CA, February.
Harel, I. & Papert, S. (Eds.) (1991). Constructionism. Norwood, NJ: Ablex.
Ives, B. & Jarvenpaa, S.L. (1996). Will the Internet revolutionize business education and
research. Sloan Management Review, Spring.
Knoll, K. & Jarvenpaa, S.L. (1995). Learning to work in distributed global teams.
Proceedings of the 28th Hawaii Conference on Systems Sciences, Vol. 4, Los Alamitos,
CA: IEEE Computer Society Press.
Leidner, D.E., & Jarvenpaa, S.L. (1995). The use of information technology to enhance
management school education: A theoretical view. MIS Quarterly, 19 (3), 265-291.
Mintzberg, H. (1988). Formons des managers, non des MBA!. Harvard-L’Expansion, Winter,
Paris.
Mintzberg, H. (1996). Musings on management. Harvard Business Review, July-August.
NTICF (1998). Proceedings of the conference nouvelles technologies de l’information et de la
communication dans les formations d’ingénieurs et dans l’industrie, INSA Rouen, France.
Papert, S. (1980). Mindstorms: children, computers, and powerful ideas. New York: Basic
Books.
Papert, S. (1990a). Computer criticism vs. techno centric thinking. E&L Memo N° 1,
Epistemology and Learning Group, MIT Media Laboratory.
Papert, S. (1990b). A critique of techno centrism in thinking about the school of the future.
E&L Memo N° 2, Epistemology and Learning Group, MIT Media Laboratory.
Piaget, J. (1985). Psychologie et pédagogie. Paris: Gallimard (Collection Folio).
Piaget, J. (1992). La naissance de l’intelligence chez l’enfant. Paris : Delachaux & Niestle.
Schön, D. (1983). The reflective practitioner. New York: Basic Books.
Schön, D. (1987). Educating the reflective practitioner. San Francisco, CA: Jossey-Bass
Publishers.
Spinosa C., Flores F., & Dreyfus H. (1997). Disclosing new worlds. Cambridge, MA: MIT
Press.
Vasquez Bronfman, S. (1998). The virtual campus of educational technologies: A coaching at
a distance system. Paper presented at 7th EDEN conference, University of Bologna, Italy.
Vasquez Bronfman, S. (2000). Le practicum réflexif : un cadre pour l’apprentissage de savoir-
faire. Sciences et Techniques Educatives, 7 (1), 227-243.
Vygotsky, L.S. (1985). Pensée et langage. Paris: La Dispute.
The Experience of Self-Organized Learning Through
the Use of Learning Plans for Knowledge
Management
Vivien Lee Looi Chng1 & Steven Coombs2

1
Temasek Polytechnic, Singapore; 2Department of Curriculum Studies & Secondary Education,
Sonoma State University, USA
1. INTRODUCTION
The call for educators to develop students who are capable of coping with
change, who are apt at higher order thinking for problem solving,
communication and teamwork, whilst exhibiting qualities, such as
adaptability and creativity, is internationally agreed upon. Many agree that
instilling in students an independent spirit of lifelong learning and equipping
them with the skills to reflect on information would work towards these aims
and prepare them for the complex problem solving work required in a
technological age (Browne & Hoag, 1995; Heijke & Ramaekers, 1999;
Keizer, 1999; McEwen, 1994). The situation is no different in Singapore as
Prime Minister, Goh Chok Tong explains:
“Our collective tolerance for change, and willingness to invest in learning
as a continuous activity will determine how well we cope with an
uncertain future … LEARNING NATION will require innovation at
every level of society.” (Goh, 1997).
This paper evaluates the critical thinking pedagogy of self-organized
learning (S-o-L). We begin with an overview of the theoretical framework
that explains how critical thinkers can be developed through the use of a new
reflective technology that we call “learning plans”. Specifically, we consider
93
94 Vivien Lee Looi Chng & Steven Coombs
how these can improve the writing skills of economics students through
adopting critical thinking schemas. Finally, we review the findings of an
action research project conducted with Singapore polytechnic students.
2. SELF-ORGANIZED LEARNING
Whilst there are many definitions of critical thinking, there are some
common characteristics across all the various models and theories. Teaching
and learning that is focused on encouraging critical thinking is generally
characterized by exploration, interaction, discussion and an evaluation of
alternatives to arrive at a sound conclusion (Browne & Freeman, 2000;
McEwen, 1994). One such approach is S-o-L, which is valuable because it
offers both a theoretical basis for action and practical “conversational” tools
to scaffold the learner’s critical thinking abilities.
S-o-L is based on various learning theories, which combine social
constructivism with cybernetics (systems thinking), conversational theory
and Kelly’s (1955) personal construct theory (PCT). The S-o-L
conversational learning theory provides instructional design axioms and a
practical set of thinking tools that raises the learner's awareness to enable
systematic reflection on his/her experience to construe personal learning. In
S-o-L, human learning is defined as the “conversational construction,
reconstruction and exchange of personally significant, relevant and viable
meanings with awareness and controlled purposiveness” (Harri-Augstein &
Thomas, 1991, p. 23).
In S-o-L, the conversational individual constructs knowledge through a
“learning conversation” that has dual components. One is a conversation
from within our self to our self and the other is externally with others
(Thomas & Harri-Augstein, 1985). From a psychological perspective of
systems-based thinking, it is suggested that individuals self-manage their
inner reflective process, constructing knowledge and modeling concepts of
the world experienced through a complex process of personal hypothesis
testing between past and present experiences. In a continual process of
deconstructing and reconstructing one’s experiences as personal constructs,
the learner as scientist adopts a holistic world-view, linking one’s personal
experience with societal influences and behaviors (Coombs & Smith, 1998).
Bannister (1981) explores the instructional design schema of PCT by
considering Kelly’s (1955) creativity cycle, which states that in achieving
any kind of learning an individual undergoes the three recursive phases of
circumspection, pre-emption and control. The three-phase creativity cycle as
three steps of reflective modeling is elaborated upon by Coombs (1995). An
idea capture phase (a loose construing process) is followed by an idea
Experience of Self-Organized Learning 95
development phase (focusing issues into an operation strategy) and, finally,

there is an operational management phase of project control through a
recursive cycle. This critical thinking model has provided a solution for the
construction of pedagogical tools based on the principles of S-o-L.
2.1 Self-Organized Learning for the Students
As mentioned earlier, one S-o-L conversational tool is called a “learning

plan” (LP). This operates as a critical thinking scaffold for project
management (see Appendix 1). The LP is described as a conversational tool
that allows critical thinking skills to be modeled through practical tasks
simulating the real world. LPs operate as a flexible and experientially
content-free technology that breaks learning events down into task
manageable activities with opportunities for students to model their
understanding at reflective milestones. Authentic simulations play the
important social function of helping the learner to personally identify with
abstract concepts and, hence, model personal knowledge from a meaningful
experiential event. Such a task-controlled reflective process gives personal
voice to prior knowledge, designing experiential linkages between past and
present learning, thus increasing meaning making to a greater depth of
personal relevance. Student-centered scaffolding is workable as the LP
defines discrete learning pathways that gain access to what Coombs (2002)
defines as the principle curriculum content resources, which are located at
“learning nodes”.
Teachers negotiate the framework of LPs with students to arrive at
customized solutions that define the scope and nature of the self-directed
learning activity (Coombs & Smith, 1998). They help students come to an
awareness of the problem solving skills necessary for independent learning
and inculcate a positive attitude towards critical thinking through
empowering student control of the curriculum learning tasks to be achieved.
LPs are thus flexible project management critical thinking tools with built-in
curriculum and assessment goals that guide the student towards thinking
critically about knowledge application (Coombs, 1995).
3. THEORETICAL BACKGROUND
3.1 Reflective Learning
Reflective learning has its basis in student-centered constructivism,

where there are many meanings and personal perspectives from which to
experientially structure and frame the world (Duffy & Jonassen, 1992).
Situating cognitive experiences in socially authentic tasks and increasing
transfer between in-school and out-of-school experiences is an educational
goal whereby the emphasis is on the learning of reflective skills geared
towards problem solving and meaning making in social situations (Resnick,
Levine & Teasley, 1991). S-o-L is a curriculum philosophy that underpins
action learning through developing transferable critical thinking skills as part
of a reflective learning personal schema.
Reflective learning is also described as a means of developing the ability
of students to self-assess. To ensure this, three measures are recommended.
First, make clear the assessment criteria and objectives to be used for any
learning activity. Second, ensure understanding of what constitutes a high
quality of learning. Last, provide a sense of what constitutes further action to
be taken on any learning task or process. In-built into this structure of
learning is formative feedback that provides students with a genuine sense of
task ownership (Stefani, Clarke & Littlejohn, 2000).
Reflective learning is also related to Kolb’s experiential learning cycle
(Stefani, Clarke & Littlejohn, 2000). Experiential learning is described as
four recursive steps: experiencing, reflecting, abstracting and applying. It is
reflection on concrete experience and the opportunity to apply learning to
new situations that brings about meaningful knowledge construction (Kolb,
1984). The notion of S-o-L conversational constructivism is further related
to reflection on personal social experience with knowledge construction
(Coombs & Smith, 1998).
3.2 Social Constructivism and Writing Skills
Flower (1979) offers a model of writing that demonstrates quality critical

thinking. Writing is described as a powerful tool for idea generation and
experiential linkages can be attributed to the S-o-L inner conversation of
rationalization (Cohen & Spencer, 1993). In S-o-L the inner conversation is
described as a conversational personal meaning construing process, which
underpins the stages of circumspection and pre-emption of Kelly’s creativity
cycle (Harri-Augstein & Thomas, 1991).
Flower's second stage of writing development considers reader-based

prose, where the writer ensures the readability of the article by organizing,
structuring and presenting arguments clearly. This relates to Harri-
Augstein’s and Thomas’ external conversation that takes into consideration
the influence of societal needs and behavior. In the act of meaningful student
writing, the Kelly “control phase” describes the prose as being operationally
ready, suiting its target audience whilst meeting assessment criteria. Figure 1
diagrammatically synthesizes Flower’s model of writing with S-o-L, with
two-headed arrows depicting a two-way relationship.
Conversational Process Writing Product
Inner Conversation Stage 1: Writer -

based prose
External Stage 2: Reader -

Conversation based prose
Figure 1: Relating the Conversational Process to the Writing Product.
3.3 Social Constructivism and Reflective Critical

Thinking in Economics Education
The literature on critical thinking in economics education offers a range

of perspectives as to how skills can be developed. Perry’s (1970) taxonomy
of cognitive development lists four stages of critical thinking. Firstly, lower
order “dualistic” thinking, where responses are of an objective nature
requiring little deliberation. At the second stage students are able to perceive
knowledge as subjective, but are not equipped to evaluate this knowledge.
Next, students begin to recognize the criteria and methods for evaluation.
Finally, at the highest level of thinking subjectivity is introduced (Thoma,
1993). As explained, decision-making is “inherently comparative and self-
consciously value-laden” (Feiner & Roberts, 1995, p. 367). Students
therefore need to learn how to prioritize criteria for making judgments. Table
1 presents a variant of the framework by describing the transition between

each stage of cognitive development as applied to economics education. The
last column provides examples of students’ writing samples obtained from
this action research project.
Table 1: Nelson Variant of Perry’s Taxonomy of Cognitive Development Applied to

Economics Education.
Perry Scheme Pedagogical Practice Sample from Student Writing
Mode 1: Helping students recognize “The rise in February was 1.3%, a 0.7%
Dualism positive economics as having drop in Consumer Price Index compared
an objective nature. to January”.
Transition 1: Can be achieved by having “Economist Daniel Lian reported that

Uncertainty and students summarize a news Singapore’s economy would grow 6.6%
Ambiguity article about real world events this year due to the fact that firstly,
and recognizing that different citizens save less and therefore have
points of views exist. higher marginal propensity to spend. This
is further encouraged by tax cuts and an
increase in government spending. …. JP
Morgan’s Tan Tzu Ping however says
that GDP will not be as high as 6.6% this
year due to asset appreciation. Due to the
high cost of housing and luxury items,
people are unwilling to spend”.
Mode 2: Students are able to “Prof Tan … suggested that the

Multiplicity differentiate normative government should keep business and
economics as having a wage costs down till the economy gains
subjective nature. strength in order for moderate growth to
take place”.
Transition 2: Achieved by having students “However even if inflation is low, these

Opinion as differentiate opinions by figures (estimates) may not be accurate as
insufficient establishing criteria to evaluate the values used are nominal GDP values,
various opinions. The end which are the value of the current
result is that students take a period’s production measured at current
position and are able to support market prices. Instead, real GDP values,
it. Examples, models and which are the value of current output
diagrams are useful at this using prices of a selected base year,
stage. should be used”.
Mode 3: Students are able to recognize “I agree with BOJ governor Masura
Perry Scheme Pedagogical Practice Sample from Student Writing

Contextual that different disciplines use Hayami that Japan should promote
relativism various evaluation criteria and structural reforms and implement
are able to apply the critical monetary policy to ease the effects of the
thinking skills to different deflationary spiral”.
contexts.
Transition 3: Achieved by having students “Nevertheless, I think that people

Joining values see the relevance of critical nowadays are more affluent and are
and analysis thinking in the real world. willing to give up their money in
They should recognize that exchange for status, image and a higher
differing values exist, for standard of living”.
example, progressive and
regressive tax structures and
the benefits it brings to
different economic groups.
Mode 4: Students are able to choose “I encourage the workers to take up this
Contextually from competing choices based training program as firstly, they will be
Appropriate on sound economic thinking able to learn new skills and also they will
Decisions and personal values. have an advantage over others by having
upgraded their skills and knowledge. In
this way, they will not be easily
retrenched and they can even compete
with others should they switch jobs due to
reasons such as a salary mismatch or the
relocation of the firm”.
Mode 1: Helping students recognize “The rise in February was 1.3%, a 0.7%
Dualism positive economics as having drop in Consumer Price Index compared
an objective nature. to January”.
Source: Adapted from Thoma (1993).
Advocating a student-centered approach, Shackelford emphasizes

cooperation and discovery learning through reflection on personal
experience. “Dialogue allows students to explore the language, argument,
and discourse of economics, and at the same time to create knowledge as
part of an emerging community of learners” (Shackelford, 1992, p. 573).
Writing assignments focus upon this type of exploration and internalization.
The constructivist opportunities of writing activities expounded upon
incorporate five writing proficiencies: (i) accessing existing knowledge; (ii)
displaying command of existing knowledge; (iii) interpreting existing

knowledge, (iv) applying existing knowledge; and, (v) creating new
knowledge. The use of subject specific terminology and methods guides the
student writing process toward an economic way of thinking in both analysis
and creation of new knowledge (Hansen, 1988). It is explained that the
active constructivist process links the building of knowledge units and the
social environment:
“What is built is an always changeable notion of the way the world works
... Writing is one method of accomplishing the requisite interaction of
(the) student with (the) world” (Petr, 1990, p. 129).
The evidence of student interaction with the world can be viewed through
the writing portfolio, which promotes constructivism in that it encourages
reflection that investigates connections between what is known, read and
felt. As an assessment tool, the portfolio increases student accountability by
requiring active personal reflection when selecting written evidence. It is
found that student writing improved the students’ economic way of thinking
(Cohen & Spencer, 1993). Tschudi (1986) also expounds on writing as a
constructivist knowledge building learning activity.
Critical thinking is more than just imparting skills to learners. Critical
thinking is defined as “the use of those cognitive skills or strategies that
increase the probability of a desirable outcome … describ(ing) thinking that
is purposeful, reasoned, and goal directed” (Halpern, 1997, p. 4). However,
she further suggests that for critical thinking to be effective it is the attitudes
and dispositions towards critical thinking that need to be developed. Critical
thinkers exhibit the dispositions of a willingness to plan, flexibility,
persistence, self-correction, mindfulness and consensus seeking (Halpern,
1997, pp. 11-12). Such attitudes make up the building blocks of intellectual
self-consciousness, resulting in the scholarly ownership of learning. The call
promoting higher-order thinking as incorporating attitudinal change has also
been echoed by economics educators (Thoma, 1993).
4. ACTION RESEARCH PROJECT
4.1 Current Economics Teaching Practice and

Curriculum Reform Needs
First year business students at a polytechnic in Singapore study

microeconomics in semester 1 and macroeconomics in semester 2. The
purpose of the macroeconomics course is to introduce the students to the

fundamental economic objectives of any government and the tools and
policies used to achieve these aims.
As a course requirement, students compile an economics portfolio that
constitutes 15% of their final grade. This assignment requires students to
prepare eight analyses of newspaper clippings over a period of 12 weeks.
Students are free to select articles of interest from any of the taught topics.
Generally, students are encouraged to write analyses that include a brief
summary followed by an explanation and application of the relevant
economic concepts and principles, along with their own comments and
opinions. During the interim submission in week 8, students are required to
submit three draft essays to the tutor. This process of submitting draft essays
provides students' with an opportunity to clarify their thinking.
The purpose of introducing portfolio writing is to encourage independent
student learning of macroeconomic concepts; how they can be applied to a
range of real world situations; and the decision-making processes of policy
makers and economists. Qualitative student evidence supports this, with
typical comments:
“By applying the topics to the articles, I will be able to see the practical
side of learning economics rather than just memorizing the facts”.
“Doing economic analysis of news articles enables me to apply concepts

which I learnt during lectures. It also helps me to understand those topics
which I’m unsure of before I analyzed the news articles. This is because I
can clarify my doubts with my friends before I can analyze those news
articles”.
This method of actively engaging students in an economic way of
thinking by using newspaper articles has also been used by other economics
educators (Cohen & Spencer, 1993; Hansen, 1988; Petr, 1990).
The Internet age now provides news clippings from on-line sources. With
the massive availability of news articles available from the internet students
are generally not too discerning when selecting articles. They often select
articles that provide more of a socio-political commentary rather than
economic perspectives. Students often adopt a “cut and paste” mentality and
simply re-produce the article verbatim by writing long summaries with little
attempt to engage in economic analysis. Students appeared unable to identify
the relevant economic concepts and principles to use or simply applied the
concepts wrongly. Economic illiteracy was further illuminated via students
who were unable to interpret the data provided. An action research study
seeking to enable change of this classroom practice was identified as the
main goal.
A focus group interview with five students provided insights that

confirmed observations of students’ difficulties. Typical responses were:
“Actually, we are not very sure which topics we should focus on. Like,
maybe on this topic, we can relate to many other topics, we don’t know
which sub-topic to relate the article to”.
“By analyzing, sometimes you worry that you are analyzing it wrongly.
You may think that it is an increase in aggregate demand, but actually it
is a decrease in aggregate demand. But actually, if you know the concept,
you can easily do the analysis”.
4.2 Action
A preliminary LP scaffold was designed and introduced to two classes

following the submission of the first draft (see Appendix 1). The LP was
designed to aid the process of reading, summarizing and finally analyzing
the article, that is from low order to high order task-based thinking activities.
In designing successful writing assignments, the sequencing of the writing
assignment implies the need for a systems thinking protocol that becomes a
critical thinking rubric (Davidson & Gumnior, 1993). In facilitating
internalization, the emphasis is on the need to focus on the process rather
than the finished product. (Cohen & Spencer, 1993)
Four pointers for the design of economics writing assignments can be
used to evaluate the quality of LPs. Firstly, negotiate manageable
assignments with students. In this aspect LPs are made up of different bite-
size learning tasks presented in order of increasing difficulty, making each
stage of the process achievable by engendering student awareness of the
learning process. As Appendix 1 illustrates: task 1 requires student reading
to achieve familiarization of the article; task 2 entails writing a summary;
and task 3 involves higher-order thinking in the form of analysis. Achievable
goals are clearly defined and assessed at each learning stage. This is further
complemented by student opportunities for consulting their tutors on
collaborative assistance over the writing process.
Secondly, a checklist of assessment benchmarks should be provided to
give students a sense of the achievable standards. Such a checklist was
incorporated into the LP, as each task was broken up into systematic action
steps. For example, in summary writing the student should begin with a one-
sentence thesis statement; followed by a one-paragraph summary; and,
finally, provide examples of evidence to support the paragraph.
The third pointer is that a writing assignment should provide incentives
to secure high quality performance. This is ensured through a 15% weighted
portfolio assessment score. Self-motivated students thus have clear

assessment guidance of the effort required to submit quality work. Lastly,
involve students in peer discussions, as conversational social collaboration
increases S-o-L capabilities and reduces students’ dependence on lecturers.
The LP is designed as a task-based learning scaffold, providing small
manageable steps as a guide to enable student management of their own
group discussions (Hansen, 1988).
The preliminary LP for economic news analysis was introduced to two
tutorial groups in weeks 5 and 6. Students were provided with a news article
to complete one written analysis in class. The written work was reviewed to
assess if instructions were clear and to identify areas needing further
improvement. It was also discovered that students had no clear idea about
how to select an appropriate news article. This finding later became the
second stage of the action research cycle. Following the period of interim
portfolio submission in week 8, the new improved LP was distributed to the
same two classes for use.
4.3 Review of Action
Four weeks after the final LP was introduced, 37 students completed a

questionnaire about their experience of having used the LPs. The questions
assessed if students had found the LPs useful toward encouraging reflective
critical thinking, while providing opportunities for social constructivism.
The preliminary results are summarized in table 2. With at least 73% or
higher indicating agreement, it is clear that LPs do have the potential to
promote skilful critical thinking through scaffolding continuous reflective
learning, thereby enabling S-o-L competency.
However, these are early stage findings collected from the use of LPs
over a short four-week period. Students needed to adjust to the change of
using the LP scaffold as an instructional tool, as one student explains: “I am
used to doing the news articles on my own so I encountered difficulties to
adjust myself with the guidelines given to me”. More rigorous use of the LP
will be required at the start of the semester 1 microeconomics course. The
students themselves suggested the use of newspaper clippings as examples to
elaborate upon each step of the LP. It is explained that examples provide
students with a good sense of what makes for quality writing and it
facilitates self-assessment (Stefani, Clarke & Littlejohn, 2000). As for
evidence of improved thinking abilities, in-depth interviews with students
will provide further insight into their experience of using LPs. An evaluation
of the writing samples gathered from the portfolio is also necessary.
As for the benefit of writing to learn, the potential gains of portfolio
writing in economics are more than just cognitive benefits. Social-emotional
and moral growth is also possible (Kish & Sheehan, 1997). In another study,
five benefits of writing were identified. These are the opportunity for
learning using a “productive learning tool”; emphasis on “enhanced
cognitive functions”; an avenue for teachers to diagnose the mental
processes of students; an opportunity for students to evaluate their own
progress; and, most importantly, an active engagement in constructivist
learning (Petr, 1990).
Table 2: Results of Survey Indicating Percentage in Strong Agreement or Agreement.

The learning plan is useful as it is organized in a way that helps me think 81.1%
systematically.
The learning plan provides guidelines and yet gives me the freedom to complete 86.5%
the analysis in a way I am comfortable with.
The learning plan provides guidelines that my friends and I can use for our 73.0%
discussion to analyze news articles.
The learning plan provides guidelines but I can still approach my lecturer for help 94.5%
in completing the news analysis.
The learning plan is useful as it organizes the tasks to be completed in order of 75.7%
difficulty.
The skills I acquire from using the Learning Plan will help me think more critically 72.9%
about other subject areas.
A learning plan is most useful when introduced to students at the beginning of the 89.1%
microeconomics course.
Further improvements suggest that the external environment, where the

external conversation takes place, could have been better stimulated. The
instructions and assessment outcomes of the LPs could have the social
context for which the student is writing. It is suggested that “create(ing) an
imaginary reader to serve as a more ‘realistic’ audience for the student”
would work towards this aim (Cohen & Spencer, 1993, p. 221).
For example, the question could be re-written as: “Select an appropriate
news article on a topic of your choice and by following the checklist below,
complete a 200-300 word analysis of it to be printed in a business newsletter
for all polytechnic students”.
A transparent description of the learning task requirements provided to
all students is reinforced via clear linkage to the assessment criteria.
Ambiguity is reduced and students become more confident. The writing
outcomes are improved and grading the portfolios also became a less
arduous task.
4.4 Educational Implications
The implications for instructional design and curriculum planning are

that S-o-L amplifies the learning experience through raising awareness of the
conversational process rather than concentrating on the subject content
outcome. As advocated, “to fully exploit a writing-to-learn approach,
instructors must devote energy not just to the students’ finished papers but to
the students’ writing processes, the strategies and procedures followed in the
act of writing. It is also important to devote attention to assignment design,
to clarifying the appropriate audience for the paper, and to responding to
student writing” (Cohen & Spencer, 1993, p. 219).
This implies that there are new competencies required of educators to be
equipped as competent and confident facilitators of learning. They should be
able to play their role as task supervisors, learning coaches and
“intentionality managers” (Harri-Augstein & Thomas, 1991). It is explained
that LPs instructionally designed in a manner that is user-friendly would
enable “the user to interact in a meaningful and conversationally fluent
manner, so as to maximize one’s creative learning potential” (Coombs, 2000,
p. 21). A recommendation that can be made for the effective authoring of
LPs to influence constructivist learning efforts is that teachers should
diagnose the students, subject matter and environment to design
educationally sound LPs. As thinking professionals engaged in active
scrutiny of how situations can be improved, they bring with them practical
know-how for the adaptation of pedagogical theory to the needs of students
whom they know best, setting appropriate curriculum activities in a
supportive learning environment. A competent facilitator of instructional
design should thus be able to demonstrate a comprehensive understanding of
students’ needs; content area of the subject taught; and the social context in
which learning will take place. The facilitator requires a complete overview
of all factors influencing learning, hence the need to “diagnose pedagogy”
(Solomon & Morocco, 1999). Professional development requires that
teachers’ pedagogical knowledge should also be constantly updated. Figure
2 defines the task management roles as key facilitator competencies to
ensure successful S-o-L.
Table 3 compares the essential differences between reflective real-world
practice and existing pedagogical practice. The recommendations for
improvement are derived from reflection on the evidence collected from
having introduced the LPs and the literature review, which suggests new
ways of improving pedagogical practice.
SeridJ"hw-P:V.a-:..1~-----
n;~ ~d~~~~
Farq_ila' r t--=-=»JcJ.:Jm&!Dn:..
TME~
~~
DA.QIC&S£XnKc:I'DiiEX[
£1iiED UauiATOII.
~~~--~
pGUHIH~~
.._
~l:ww:Stqio l."'nE'al<IQUIYJUNAGElt
~&e~~d
lliAC:I'lOSf!S~GOGY ..&!a ezwo i.c.c:w:r~
~.-!.......,_...,.,
IIAGaOSES smuECI
v.,...q'XPl,
r-=o:i""-;:sn-'7!~~ ~
==~6w-1 ~.a:!~
fa d::::ii;::::e ~~
DIAC!IOSESSI
illnlJEJIU$r-n~~~ ~ ~
-'~~·
:..:::z:;p~~...:iil:;j
Figure 2: Re-defined Components of Task-management Roles with

Facilitator Competencies for a S-o-L Environment (Adapted from Coombs
and Smith, 1998).
Table 3: Role for the Facilitator in the Economics Classroom.

Facilitator Existing Practice Recommended Improvement
Competency
Task Supervisors No formal opportunities Build in cooperative learning
Organize the social for social learning. LPs opportunities to encourage peer review,
domain. introduced were mostly thus providing more opportunities for
used by students social constructivism.
independently.
Learning Coaches Currently available for Initiate students into use of LPs by
Conversational consultation at students’ initially providing examples of news
learning assistant, request. articles and demonstrating how each
initiating scaffolding Responding to emails. learning activity can be carried out.
exercises using Giving hints when a The next stage could involve providing
conversational tools suitable news article is news articles and asking leading
and reflective sourced. questions.
techniques.
Intentionality Designed the first version Author supporting learning materials such
Managers of the LP for economic as examples and leading questions
Designs the learning news analysis. outlined above.
opportunities and Design the LP to incorporate more
authors the resources negotiated learning with student,
i.e. conversational reviewing success of LP used during
tools and courseware. period of interim portfolio submission.
5. CONCLUSION
Starting with the need to develop critical thinkers, this research study
reviewed the theory of S-o-L and the LP conversational tool to consider how
it can be integrated into economics instruction. Specifically, the LP, based
on reflective, conversational learning and social constructivism, was
introduced to enhance students’ critical thinking abilities when writing an
economic analysis of a news article. The action research project has had
favorable preliminary findings in that students now have clear criteria and
points of reference in organizing and structuring their written work. Through
achievable goals, the improved critical thinking attitudes of students can be
expected to facilitate learning across contexts. If indeed, developing such
conversational learning organizations is the way to go in this knowledge age,
then introducing S-o-L would be a sensible pedagogical approach. As an
experiential content-free technology, the LP is an easily adaptable template
that can be recruited into any discipline. By fluently scaffolding the student
learning process encounter, the LP levers and maximizes the experiential
gain of the planned event.
APPENDIX: LEARNING PLAN FOR ECONOMIC

ANALYSIS OF A NEWS ARTICLE
Instructions
Select an appropriate news article on a topic of your choice and complete an analysis of it
using the steps laid out below. Each step can be viewed as a component of a checklist.
Learning Objectives
By the end of this project, you should be able to:
Summarize a news article.
Analyze the news article from an economic perspective.
Identify appropriate economic concepts to apply in the economic analysis of the news article.
Task 1 Reading the News Article
1.1 Survey the article
Read the heading and sub-headings.
Read the introductory and concluding paragraphs.
Review any pictures/graphics, noting the captions.
1.2 Question
Turn the heading/sub-headings into questions.
Ask: What did the lecturer say about this topic when it was assigned?
Ask: What do I already know about this topic?
Read
Read each paragraph one at a time with your questions in mind.
Look for answers, making up new questions when necessary.
Make a list of the important points; underlining or highlighting them.
Task 2 Summarizing the News Article
Using the main points identified in Task 1, write a one-sentence thesis statement that sums up
the article.
Condense the article into a one-paragraph summary by including the major points.
Do include one or more of the author’s examples and/or evidences.
Task 3 Writing the News Analysis
Identify all the economic concepts and principles used in the article by highlighting them.
Categorize the concepts and principles highlighted above according to chapters and sub-
chapters.
Identify the chapter and sub-chapter you would like to do further analysis on.
Explain the relevant concepts briefly.
Use diagrams to illustrate the main points where necessary.
Ask: Are there any hidden assumptions in the article?
Ask: Are there any weak arguments in the article?
Ask: Are there other points of views that can be considered?
REFERENCES
Bannister, D. (1981). Personal construct theory and research method. In P. Reason and J.
Rowan (Eds.), Human inquiry: A sourcebook of new paradigm research (pp. 191-200).
Chichester, UK: John Wiley and Sons.
Browne, N., & Freeman, K. (2000). Distinguishing features of critical thinking classrooms.
Teaching in Higher Education, 5 (3), 301-310.
Browne, M.N., & Hoag, J.H. (1995). Critical thinking in graduate economic programs: A
study of faculty perceptions. Journal of Economics Education, 26 (2), 171-181.
Cohen, A.J., & Spencer, J. (1993). Using writing across the curriculum in economics: Is
taking the plunge worth it? Journal of Economics Education, 24 (3), 219-231.
Coombs, S.J. (1995). Design and conversational evaluation of an information technology
learning environment based on self-organized learning. Unpublished doctoral thesis,
Centre for the Study of Human Learning, Brunel University, London.
Coombs, S.J. (2000). The psychology of user-friendliness: The use of information technology
as a reflective learning medium. The Korean Journal of Thinking & Problem Solving, 10
(2), 19-31.
Coombs, S.J. (2002, in press). The teacher's role in using IT to manage student-centred
learning. In A. Chang, & I. Smith (Eds.), Critical and creative thinking, Vol. 2. Singapore:
Pearson Sprint, National Institute of Education.
Coombs, S.J., & Smith, I. (1998). Designing a self-organized conversational learning
environment. Educational Technology, 38 (3), 17-28.
Davidson, L.S., & Gumnior, E.C. (1993). Writing to learn in a business economics class.
Journal of Economics Education, 24 (3), 237-244.
Duffy, T.M. & Jonassen, D.H. (1992). Constructivism: New implications for instructional
technology. In T.M. Duffy, & D.H. Jonassen (Eds.), Constructivism and the technology of
instruction: A conversation (pp. 1-16). Hillsdale, NJ: Lawrence Erlbaum Associates,
Publishers.
Feiner, S. & Roberts, B. (1995). Using alternative paradigms to teach about race and gender:
A critical thinking approach to introductory economics. American Economic Review, 85
(2), 367-371.
Flower, L.S. (1979). Writer-based prose: A cognitive basis for problems in writing. In College
English, Vol. 41. In Cohen, A.J. and Spencer, J. (1993). Using writing across the
curriculum in economics: Is taking the plunge worth it? Journal of Economics Education,
24 (3), 219-231.
Goh (1997). Speech by Prime Minister Goh Chok Tong at the opening of the 7th international
conference on thinking on Monday 2nd June. Singapore: Ministry of Education [On-line].
(Accessed 16/07/02). URL: http://www1.moe.edu.sg/ speeches/020697.htm
Halpern, D. (1997). Critical thinking across the curriculum: A brief edition of thought and
knowledge. Mahwah, NJ: Lawrence Erlbaum.
Hansen, W.L. (1988). Integrating the practice of writing into economics instruction. In W.E.
Becker, & M. Watts (Eds.), Teaching economics to undergraduates: Alternatives to chalk
and talk (pp. 79-118). UK: Edward Elgar.
Harri-Augstein, E. & Thomas, L.F. (1991). Learning conversations: The self-organized
learning way to personal and organizational growth. London: Routledge & Kegan Paul.
Heijke, H. & Ramaekers, G. (1999). Importance of informatics skills for economics
graduates. In J. Hommes, P.K. Keizer, M. Pettigrew & J. Troy (Eds.), Educational
Innovation in Economics and Business IV: Learning in a changing environment (pp. 61-
82). Dordrecht: Kluwer Academic Publishers.
Keizer, P.K. (1999). How to prepare graduates for the changing workplace? In J. Hommes,
P.K. Keizer, M. Pettigrew & J. Troy (Eds.), Educational Innovation in Economics and
Business IV: Learning in a changing environment (pp. 1-27). Dordrecht: Kluwer
Academic Publishers.
Kelly, G. (1955). The psychology of personal constructs, Volumes 1 & 2. New York: Norton.
Kish, C.K. & Sheehan, J.K. (1997). Portfolios in the classroom: A vehicle for developing
reflective thinking. High School Journal, 80 (4), 254-261.
Kolb, D.A. (1984). Experiential learning: Experience as the source of learning and
development. Englewood Cliffs, NJ: Prentice-Hall.
McEwen, B.C. (1994). Teaching critical thinking skills in business education. Journal of
Education for Business, 70 (2), 99-103.
Perry, W. (1970). Forms of intellectual and ethical development in the college years: A
scheme. New York: Holt, Rinehart and Winston.
Petr, J.L. (1990). Student writing as a guide to student thinking. In P. Saunders & W.B.
Walstad (Eds.), The principles of economics course: A handbook for instructors (pp. 127-
140). New York: McGraw Hill Publishing Company.
Resnick, L.B., Levine, J.M. & Teasley, S.D. (Eds.). (1991). Perspectives on socially shared
cognition. Washington, DC: American Psychological Association.
Shackelford, J. (1992). Feminist pedagogy: A means for bringing critical thinking and
creativity to the economics classroom. American Economic Review, 82 (3), 570-576.
Solomon, M.Z., & Morocco, C.C. (1999). The diagnostic teacher. In M.Z. Solomon (Ed.), The
diagnostic teacher: Constructing new approaches to professional development (pp.231-
246). New York: Teachers College Press.
Stefani, L.A.J., Clarke, J., & Littlejohn, A.H. (2000). Developing a student-centred approach
to reflective learning. Innovations in Education and Training International, 37 (2), 163-
171.
Tschudi, S. (1986). Teaching writing in the content areas: College level. Washington, D.C.:
National Education Association of the United States.
Thoma, G.A. (1993). The Perry framework and tactics for teaching critical thinking in
economics. Journal of Economics Education, 24 (2), 128-136.
Thomas, L.F., & Harri-Augstein, S. (1985). Self-organized learning: Foundations of a
conversational science for psychology. London: Routledge and Kegan Paul.
PART III
TEAM AND COLLABORATIVE LEARNING

Using Student Consulting Team Assignments as a
Vehicle to Teach a Systems Development Course
Sylnovie Merchant
California State University, Sacramento, CA, USA
1. BACKGROUND
In 1925, Henri Fayol (as cited in Donnely et al., 1981) introduced

principles of management that would determine the effectiveness of an
organization. His concern was finding principles that were flexible and
adaptable to the circumstances and events at the time. His chief desire was to
elevate the status of management practices by supplying a framework for
analysis. His framework included a statement of principles and functions.
His list of principles included 14 that were designed to guide the thinking of
managers in resolving concrete problems or the day-to-day activities under
review. His five functions, however, were the ones that he felt managers
should engage in and which he elaborated upon the most. These principles
were:
• Planning - includes all those activities of a manager which result in a
course of action. The manager should make the best possible forecast of
future events that affect the firm and draw up an operating plan that
guides future decisions.
• Organizing - includes all activities which result in a structure of tasks
and authority. This managerial function determines the appropriate
machines, material and human mix which are necessary to accomplish
the task.
• Commanding - directing the activities of subordinates. The manager
should set a good example and know thoroughly the personnel and the
agreements made between the personnel and the organization. The
113
114 Sylnovie Merchant
managers should have direct, two-way communication with

subordinates. Furthermore, managers should continually evaluate the
organizational structure and subordinates and they should not hesitate to
change the structure if they consider it faulty or to fire subordinates if
they are incompetent.
• Coordinating - activities which bind together all individual efforts and
direct them toward a common objective.
• Controlling - assuring that actual activities are consistent with planned
activities (Donnelly, Gibson & Ivancevich, 1981).
Subsequent authors in the field of management increased Fayol’s five
principles to seven or reduced the five to three. Yet, since 1925 the
principles of management, offered as “universal” principles, still revolve
around the five set forth by Fayol. For the last two decades, management
classes have introduced students to the five management functions of
planning, organizing, staffing, directing and controlling. These have been the
underpinnings of the modern organization’s analysis and success.
1.1 Current Requirements
Since the time of Fayol, the modern organization has evolved and
typically consists of interrelated departments and units. In planning and
controlling them, management must depend upon various sources of
information, both external and internal to the organization. This has become
necessary as organizations grow in complexity, thereby increasing the
number of points at which decisions must be made, ranging from individual
decision makers at the lowest operating levels to policy-making groups in
top management.
Over the years computer systems, designed to provide information to
managers to support the process of decision making, are increasingly more
important for an organization to be successful. Effective planning and
controlling in any organization requires relevant information. The quality of
a decision depends greatly on an understanding of the circumstances
surrounding an issue and knowledge of the available alternatives, states of
nature and competitive strategies. The better the information, the better will
be the resulting decision. Unfortunately, an organization has no memory
other than the memory of the people who manage it. Because individuals
come and go, managers must, out of necessity, develop some type of
information system.
The basic problem appears to be that the information requirements of
today’s managers have changed greatly from past decades but the basic
information arrangements within most organizations have remained
essentially the same. The solution is to design, develop and implement an
Student Consulting Team Assignments 115
information system that leads to a competitive advantage. That is, the issue
of generating the right information at the right time must now be viewed in a
much broader perspective than previously. The task of developing data for
managerial decisions must be viewed as the function of a management
information system (MIS) rather than as solely the role of individual
managers in the various functional areas of an organization. The importance
of information becomes obvious when one realizes that managers rarely
work with “things" but rather with “information about things." Thus, MISs
have one primary purpose, namely to provide the manager with the
necessary data for making intelligent decisions.
The term information system (IS) aptly describes its function, which are
systems for providing information to management. More specifically, we can
define IS as: “An organized, structured complex of individuals, machines,
and procedures for providing pertinent information from both external and
internal sources. It supports the planning, control, and operations functions
of an organization by providing uniform information for use as the bases for
decision making” (Kennevan, 1970, p. 63).
In this context, then, when viewed as the conversion of information into
action through the process of decision-making, an organization can be
thought of as an information-decision system. This, then, makes the
requirement of a valid assessment of the system’s needs and objectives of
utmost importance. The performance of management depends largely on the
availability and timely utilization of information at all levels in the
organization. Information is the glue that holds organizations together. That
is, information-decision systems need to be considered in conjunction with
the fundamental managerial functions of planning, organizing, and
controlling. If organizations are to implement planning and control, if
organizations are tied to communication, and if communication is
represented by an information-decision system, then the key to success in
planning and controlling any operation lies in the information-decision
system (Johnson, Fast & Rosenzweig, 1973). Therefore, we are dealing with
three types of information - planning, controlling, and operational.
1.1.1 Planning Information
This type of information relates to the top management tasks of

formulating objectives for the organization; the amounts and kinds of
resources necessary to attain the objectives; and the policies that govern the
rise of these objectives. Much of this information will come from external
sources and will relate to such factors as the present and predicted state of
the economy, availability of resources (non-human as well as human) and
the political and regulatory environment.
1.1.2 Control Information
This information aids managers in making decisions that are consistent

with the achievement of organizational objectives as well as the efficient use
of resources. It enables middle managers to determine if actual results are
meeting planned-for-results (objectives). It relies heavily on internal sources
of information (often interdepartmental) and involves such matters as
developing budgets and measuring the performance of first-line supervisors.
1.1.3 Operational Information
This information relates to the day-to-day activities of the organization. It

includes routine and necessary types of information, such as financial
accounting, inventory control and production scheduling. It is generated
internally and, since it usually relates to specific tasks, it often comes from
one designated department. First-line supervisors are the primary users of
this information.
It is obvious from this discussion that information is vital for
organizational effectiveness. The challenge for instructors is to devise and
present a curriculum for students in the area of ISs that will help them make
a contribution to their future organizations.
1.2 Course Challenges
While management education and training has been taught for many
years, in the field of management information science/systems (MIS) or
computer information science (CIS), the newness of the discipline attracts
many recommendations on what methodology is the best to obtain course
objectives. Regardless of these recommendations, the key issue remains the
same (i.e. “What is the value of the course to the student?”).
This is uniquely true for students in MIS/CIS, as the demand for high
technology workers has been steadily increasing (Hodges, 1987; Silvestri &
Lukasiewicz, 1987). The number of students enrolling in this major is also
increasing at a steady rate. To businesses that may be hiring these students, it
is important that they possess the necessary skills. These skills include
communication, both verbal and written, and the ability to work in teams. In
the area of ISs development these skills are crucial and, while many students
acquire these skills in different business courses, the problem lies in how to
use them while, at the same time, gaining experience in their chosen field.
2. FRAMEWORKS FOR TEACHING

INFORMATION SYSTEMS
Various authors have set forth their views on the nature of ISs and their
design and their proposed frameworks clearly illustrate the complexity and
comprehensiveness that instructors need to consider in teaching in such a
field. For example, Ackoff’s 1967 paper discusses the design of an MIS
through a five-step process. He also points out how these steps would assist
others within the system (specialists, researchers, managers) to collaborate in
order to build a system that is effective and usable. Churchman (1971)
devotes an entire book to systems design, three chapters of which outlined
objectives, resources, components and management of the system for
processing information. In 1970, Zani detailed a layout for an IS through the
discussion of a general framework, key success variables, decision analysis
and information technology. Dearden (1972) lays out a fundamental
rationalization for the design of a MIS. He argues that management
information is a homogenous subject for study and specialization where a set
of principles and practices can be established uniformly and that the systems
approach can and should be used to analyze management’s information
requirements.
Benbasat and Schroeder (1977) address the internal processes of the
organization, such as cost and time performance and the number of reports
requested via the various functional areas. Their study was extremely useful
and appropriate for the information on “interactions” taking place within an
organization. Ives, Hamilton, and Davis (1980) also investigated inter-
relationships within the system. That is their concern revolved around the
resources and constraints, which dictate the scope and form of each
information subsystem and their study concentrates on the internal
environment as well as the use, development and operation process. In 1980,
Sprague provided a framework for the development of a decision support
system dedicated to improving the performance of knowledge workers in
organizations through IT. His emphasis is clearly on the subsystems within
an organization and how best their resources can be combined to be more
effective. Bariff and Ginzberg (1982) made an important contribution to the
literature in exploring the relationship between processes used in an
organization and the levels of analysis applied, thereby combining
individual, groups, organizational and societal variables in the makeup of an
organization. In essence, to be effective the inner workings of the
organization must be taken into consideration and can be evaluated by
understanding the process from the standpoint of psychology, sociology,
organizational behavior, etc.
In 1971 Mitroff proposed a model that would provide the user with the
relevant information that is needed. His proposal is the Hegelian or
dialectical inquiring system as a means of presenting pro and con arguments
on an issue of importance to the user. Another user model is offered by
Zmud (1979). His model looked at the elements of individual differences,
personality, cognitive behavior, design characteristics and the attitude of the
user. His argument was that all of these variables impact on the IS in place.
Benbasat and Dexter (1979) emphasized the psychological type of the user.
Their conclusion, which supports the findings of others, is that the
psychological type of the user does have an impact on the design of ISs. An
interesting article by Dickson, Benbasat and King (1982) addresses the needs
of users from a training perspective. That is schools need to provide vigorous
and effective academic programs that will produce people who will fill
forecasted needs in the field of ISs. These individuals must also be equipped
with the appropriate skills that will aid organizations in achieving their
objectives.
McKenny and Keen (1974) demonstrated the importance of taking into
account the different styles of thinking when designing an IS. In particular,
since most system designers are systematic thinkers, and not all managers
are, they need to consider these differences. Others in this category include
Swanson (1974) who argues that there has been widespread failure in the
implementation of MIS ideals and a contributing factor has been the
assumption by designers that a manager does not need to understand how the
IS works. King and Cleland (1975) propose a methodology for developing
ISs that focus on the information requirements of managers in a formal, but
participatory manner.
2.1 Previous Studies on Methods of Teaching

Information Systems
In a recent approach to improve student learning in this field of ISs, it

was found that there were significant benefits when students work in small
groups. These included: (1) students gaining insight into group dynamics; (2)
setting more comprehensive assignments than is possible when students
work on their own; (3) students developing interpersonal skills; (4) students
being exposed to other points of view; and (5) students being prepared for
the real world (Lejk, Weyvill & Farrow, 1999).
Another instructor found that his course on spreadsheets provided an
ideal opportunity to integrate desktop technology, application software
packages, business problem-solving expertise and programming at the macro
level. By using this method of addressing all these issues in a single course,
the student was presented with the opportunity to observe the relationship
between technology and business decision-making (Chou & Gensler, 1993).
One instructor used a role playing technique to teach systems analysis
and design and was able to conclude that, in addition to providing the
benefits typically associated with the techniques of role play, the approach
provided an experience-based format that has been shown to better prepare
students for their professional careers in a real environment (Kirs, 1994).
In a final example, an instructor integrated total quality management, in
the form of total quality involvement, into a systems analysis and design
course and found that it was an effective instructional methodology. Not
only did class attendance and participation improve and increase, but
students’ reactions to the course were overwhelmingly positive
(Killingsworth, Harden & Dellana, 1999).
3. AN INNOVATIVE APPROACH
Instructors in higher education have been challenged by businesses to

change how they deliver their product in order to produce students who have
the skills and self-confidence to work in teams when they enter the
workforce. With the objective of introducing reality into the curriculum and
offering students the opportunity to assume the role of a consultant while
working with a team, an innovative approach was introduced in the teaching
of an MIS undergraduate course in systems analysis and design. The goal of
the course is to provide students with the opportunity to design and develop
an IS for use by a real business in the local area. The challenge of teaching
MIS can be met more successfully by creating a situation where students can
practice the concepts and techniques that are taught in the classroom.
Historically, the systems analysis and design course included the theory
and principles involved in ISs design and development, but without much
practical application to a real business. The course is now split over two
semesters where the goal is the analysis, design, development and
implementation of a complete IS for an actual business in the community.
These two courses are currently listed as MIS 116A and MIS 116B. Table 1
shows the major activities covered during the two semesters.
Based upon this framework students are taught to understand that they
can approach their systems analysis and design activities by analyzing user
requirements; defining user needs; developing processes and procedures;
implementing and testing; and training the users. Feedback will occur, not
only from me as the instructor, but also from their client on what is desired
and/or what is being done correctly.
Table 1: Course Activities.

MIS 116A (Semester 1) MIS 116B (Semester 2)
Analysis and Design Development and Implementation
Identify and Define Problems Create Usable System
Relate Issues to Mission/Objectives Develop Operations Manual
Learn Correct Processes Testing the System (team and others)
Learn to Use a CASE Tools Implementation Issues
Define IS Requirements Train the Users
Constraints can come in various forms from the external environment,

aspects such as time constraints, costs, degree of assistance/resistance, etc.
Regardless of these, students can pursue a highly structured framework that
basically walks them through the process for both semesters. Activities that
occur in each course are detailed in the following sections.
Constraints
Analyze User Define User Develop Processes

Requirements Needs and Procedures
Feedback
and
Interaction
Train Users Test and

Implement
Constraints
Figure 1: Systems Analysis and Design.
3.1 First Semester - MIS 116A
In the MIS 116A course students are introduced to the principles

involved in analyzing and designing an IS. While design concepts are
discussed, the emphasis is primarily on analysis. Students are taught
diagramming techniques and assessment methods used to determine the

requirements of the user.
The students are also required to form teams of approximately four to six
individuals and select a project manager. The teams are then required to
locate a company or agency that is interested in automating a manually-
based IS or enhancing its current computer-based system. They cannot
choose a system with which any member of their team is currently working
or has worked on in the past.
Students may choose any type or size of company or agency. If they
choose a small company, they must try to cover a fairly wide range of
business functions. If they choose a medium or large company, they must
narrow the scope to a defined business application (e.g. customer billing and
collection). The students are required to explain to their potential client that
the project team is willing to help them define their requirements for a new
computer system. This service is provided free in return for the client’s
cooperation for the project, keeping in mind that consulting services could
cost the business at least $15,000.
Throughout the semester the students are analyzing their clients’ needs
through interviews, observations and other information gathering techniques
learned in the course. They use the tools and techniques learned in the course
to understand and diagram the current system (manual or computerized).
They also learn how to use a computer aided software engineering (CASE)
tool to assist them in their analysis. Throughout the course they produce
several deliverables (reports). These are shown in table 2.
Table 2: MIS 116A Deliverables.

Deliverable Description
Information on all team members as well as policy and
Project Team Composition
procedures for the team to follow.
Name and description of client; contact information for
Project Team Proposal
client; preliminary diagrams.
Breakdown of major activities; Gantt chart with semester
Initial Project Schedule
schedule.
Overview of company; problems company is facing;
Current System Description
appropriate diagrams and supporting material.
Revised Project Schedule Tasks completed; any revisions made and reasons why.
How analysis conducted; user requirements; system
User Requirements Document constraints; description and diagrams of current system;
alternative solutions.
Synopsis of all previous studies; proposed system;
Final Systems Study appropriate diagrams and supporting material; prototype of
proposed system.
By the end of the first semester each team has completed the analysis of
the current system and put together a proposed design for a future system.
To help facilitate the design of a new system the teams also create a
prototype using a software package that is a skeleton of the proposed system.
The students do a presentation of their analysis and proposed design at the
end of the course.
3.2 Second Semester - MIS 116B
In most cases, the same teams continue on to MIS 116B and continue
working with the same client. During the second semester the emphasis is on
the design of the new system. This is very much the capstone course for the
MIS students as they apply all of the concepts they learned in their database
and programming courses as well as the previous section of the systems
analysis and design course (MIS 116A) to create a working, real-life system
usable by their client. The students start with their initial prototype and
revise and build the system throughout the semester. Both the team that
designed it and another team test this system. Each team must also create a
user’s guide and develop both implementation and user training plans.
Table 3: MIS 116B Deliverables.

Deliverable Description
Description of business; overview of problem; methodology
Project Organization
used.
Description of current system; scope, nature, and impact of
System Definition problems detected; feasibility analysis; appropriate diagrams
and documentation; Level 1 prototype.
Alternatives considered; proposed system; breakeven and
Analysis & Physical Design payback analysis; performance standards; control
specifications; Level 2 prototype.
Menu layouts; detailed programming logic; program listing;
Programming
Level 3 prototype.
System Testing Results of testing of completed system; detailed user manual.
Results of another team’s testing of the completed system (as
Acceptance Test
reported by the testing team)
Installation and operation plan; training plan for users;
Installation & Operation
method of support and duration.
In addition to designing the final system, students produce several reports

as they progress through each phase of the design as shown in table 3. Each
student must take part in at least two presentations during the semester, thus
helping to enhance their oral communication skills. They must also lead at
least one phase of the project thereby gaining experience of performing a
leadership role.
The project manager must evaluate each team member at least twice
during the semester. Not only must they fill out an evaluation form, they
must also discuss the evaluation with the individual team member. Both the
project manager and the team member must sign the form before handing it
in.
An interesting aspect of the MIS 116B course is the ability of any team to
fire a member who does not adequately fulfill his/her obligations. In order to
do so, proper documentation must be kept on the problems with the
individual’s performance. Once fired, the student has to find another team
that is willing to accept him/her or accept a fail grade for the course. This
situation has occurred, although infrequently.
3.3 Projects
The projects completed by the students have encompassed both the

private and the public sector. Some of the projects completed by students are
summarized below.
• A reservation system for a restaurant that allows the user to input
reservations as they are requested. The user can also update each
reservation. The system has helped the client track reservations and has
eliminated the problem of more than one reservation booked for a given
date and room.
• An online tracking system created through the use of the Internet that
allows a state agency to accept orders online and track those orders.
• Horse statistics reporting system that enables an equine organization to
track statistics on horses that take part in their competitions. This
information is also included in their annual report.
• Inventory tracking system for a store that sells a variety of unique
products.
• A system that tracks rentals for a video store by registering the
individuals who rent a video, DVD, or viewing equipment, date and time
of rental, etc.
• A system that keeps track of international students at a community
college. The data on each student includes their contact information in
both the USA and their country of origin.
4. TEACHING EFFECTIVENESS OF THE COURSE
In order to assess the effectiveness of the MIS 116A and MIS 116B
course a survey was done of the students after the completion of each
semester. The primary purpose of the survey was to ask students to assess
the value of the courses. The survey was developed based upon the major
activities shown in table 1. Table 4 provides demographic information on the
respondents.
Table 4: Demographics.
Demographics Freq.
Male 67
Gender
Female 62
21 - 25 70
Age
Over 25 59
United States 65
Place of birth
Other 64
Less than 1 year 25
1-3 years 38
Work Experience
3-5 years 25
5 or more years 41
Table 5: MIS 116A Survey Results (n=57).

Items to Evaluate Valuable and Extremely Valuable (%)
Working in teams 91.2
Understand details of a business 100
Working with a real company 93.0
Defining IS requirements 98.2
Learning correct process steps 94.7
How to identify/define problems 94.7
Relate issues to mission/objective 87.7
Use CASE Tools to analyze issues 68.4
Written/oral communication skills 89.5
Training for the real world 96.5
Feedback from the students was overwhelmingly positive. The value of

the approach from the students’ standpoint found that all but a few of the
variables were rated 90% or more as being valuable or extremely valuable to
them. A key question was: “Does this project prepare you for the real
world?” Over 90% of the students on both courses rated the experience as
valuable or extremely valuable with respect to their training for the real
world. Table 5 and table 6 show the aspects evaluated and the students’
responses based upon their experiences of the course. There were a total of
57 usable responses for MIS 116A and 72 usable responses for MIS 116B.
Table 6: MIS 116B Survey Results (n=72).

Items to Evaluate Valuable and Extremely Valuable (%)
Working in teams 97.2
Creating a usable, working system 95.8
Working with a real company 90.3
Testing your system 93.1
Have others test your system 90.3
Implementation/conversion Issues 86.1
Training users 87.5
Developing operations manual 91.7
Written/Oral communication skills 95.8
Training for the real world 93.1
It is interesting to note that students rated the CASE tool relatively low.
This is probably due to the fact that the CASE tool used for the course was
adopted primarily because of its low cost. The tool itself is difficult for many
students to learn and the documentation provided for it is limited. It is also
interesting that the students on the MIS 116B course rated the importance of
written/oral communication higher than those on MIS 116A. This is no
doubt due to the fact that there is more emphasis on written/oral
communication in MIS 116B than MIS 116A.
5. CONCLUSION
The challenge for instructors, particularly those in the IS’s field, is to

teach students in a manner which fulfils the objectives of the course, while
relating the concepts taught to the real world. The MIS 116A and MIS 116B
courses have been very successful in both these aspects. First, the courses
reflect much of what systems analysts are doing. The students are confronted
with real clients in real world situations. They are made to understand how
different tools and techniques are used in this discipline. At the same time,
they are made aware of the variety of requirements in systems analysis and
design and the fact that there is more to this area than theory and tools. This
approach also shows how flexible the area of systems analysis and design is
and that it is not simply a rigid set of rules and procedures taught in a
classroom.
Most importantly, this course teaches students how to work in teams and
how to deal with the dynamics of managing individual team members while
still meeting the project requirements. In addition, students also learn the
importance of communication skills, both written and verbal, and how to
apply them. Although this approach does not guarantee similar results in
other courses taught at other institutions, the evidence supports the
likelihood of similar success. This serves as a guide for others who wish to
improve teaching effectiveness and student learning.
REFERENCES
Ackoff, R. L. (1967). Management misinformation systems. Management Science, 14 (4),
B147-156.
Bariff, M.L., & Ginzberg, M.J. (1982). MIS and the behavioral sciences: Research patterns
and prescriptions. DATA BASE, 14 (1), 19-26.
Benbasat, I.W., & Dexter, A.S. (1979). Value and events approaches to accounting: An
experimental evaluation. Accounting Review, 54 (4), 735-749.
Benbasat, I.W., & Schroeder, R. (1977). An experimental investigation of some MIS design
variables. MIS Quarterly, 2 (2), 43-45.
Chou, D.C., & Gensler, P.J. (1993). Using spreadsheets to teach decision support systems in
business schools. Journal of Education for Business, 69 (2), 116-121.
Churchman, C.W. (1971). The design of inquiring systems. New York: Basic Books.
Dearden, J. (1972). MIS is a mirage. Harvard Business Review, Jan-Feb, 90-99.
Dickson, G.W. (1981). Management information systems: Evolution and status, Advances in
Computers, 20, 1-37.
Dickson, G.W, Benbasat, I.W., & King, W.A (1982). The MIS area: Problems, challenges and
opportunities. DATABASE, 14 (1), 7-12.
Donnelly, J.H., Gibson, J.L., & Ivancevich, J.M. (1981). Fundamentals of management.
Texas: BPI.
Hodges, P. (1987). Three decades by the numbers. Datamation, 33 (18), 77-87.
Ives, B., Hamilton S., & Davis, G.B. (1980). A framework for research in computer-based
management information systems. Management Science, 26 (9), 910-934.
Johnson, R.A., Kast, F.E., & Rosenzweig, J.E. (1973). The theory and management of
systems. New York: McGraw-Hill.
Kennevan, W.J. (1970). MIS Universe. Data Management, Sept., 63.
Killingsworth, B.L., Harden, M.B., & Dellana, S.A. (1999). Total quality involvement in the
classroom: Integrating TQM in a systems analysis and design course. College Student
Journal, 33 (3), 465-477.
King, W., & Cleland D. (1975). The design of management information systems,
Management Science, 22 (3), 286-297.
Kirs, P.J. (1994). A role playing approach to the instruction of information systems analysis
and design courses. Journal of Education for Business, 69 (6), 317-325.
Lejk, M., Wyvill, M., & Farrow, S. (1999). Group assessment in systems analysis and design:
A comparison of the performance of streamed and mixed-ability groups. Assessment and
Evaluation in Higher Education, 24 (1), 5-15.
McKenny, J.L., & Keen, P.G.W. (1974). How manager’s minds work, Harvard Business
Review, 52 (3), 79-90.
Mitroff, I.I. (1971). A communication model of dialectical inquiring systems: A strategy for
strategic planning. Management Science, 17 (10), B 634-B648.
Silvestri, G.T., & Lukasiewicz, J.M (1987). Projections 2000: A look at occupational
employment trends to the year 2000. Monthly Labour Review, Sept, 46-63.
Sprague, R.H. (1980). A framework for the development of decision support systems. MIS
Quarterly, 4 (4), 1-26.
Swanson, E.B. (1974). Management information systems: Appreciation and involvement.
Management Science, 20 (2), 178-188.
Zani, W.M. (1970). Blueprint for MIS, Harvard Business Review, Nov-Dec, 95-100.
Zmud, R.W. (1979). Individual differences and MIS success: A review of the empirical
literature. Management Science, 25 (2), 966-979.
Using Team Learning in the Classroom: Experiences
and Lessons
Leroy F. Christ1, Mary Y. Christ1, A. Steven Graham2, Michael K. McCuddy1

& Wendy L. Pirie1
1
College of Business Administration, Valparaiso University, Indiana, USA; 2Purdue University
North Central, Westville, Indiana, USA
1. INTRODUCTION
An important skill for success in today’s workplace is the ability to work

effectively in a team. “Many organizations are moving toward a
management system where teams of employees, rather than individuals, are
responsible for achieving organizational goals” (Kaplan & Welker, 2001,
p.15). This paper considers how teamwork skills can be developed in the
classroom by using a team learning approach. Team learning is an
innovative method that places greater responsibility on the students for the
teaching/learning process. Team learning emphasizes a high level of active
involvement and a great deal of self-management by students. It furthers the
objectives of developing teamwork skills and providing students with the
competencies and desire for lifelong learning. The professor ceases to be the
“sage on the stage” and becomes the “guide on the side.”
Michaelson and Price (1999), in describing the team learning process,
contend that it ensures content coverage while freeing up time for other
important areas. Team learning ensures that students attain an appropriate
level of proficiency in cognitive/technical knowledge while providing class
time for the development of important competencies, such as leadership,
personal initiative, planning, teamwork, communication, synthesis and
integration. In contrast, traditional education focuses almost exclusively on
the development of cognitive/technical knowledge.
129
130 Leroy F. Christ et al.
In the process of learning, the steps might be summarized as initial

exposure to the subject matter, understanding and application and mastery.
In team learning, the students are actively involved at the first exposure
stage. Prior to class, each student reads the assigned material. Students are
held accountable for assigned material through individual and/or team
quizzes that are usually administered prior to class discussion of a topic.
Discussion and class activities focus on developing a conceptual framework
and on topics where students experience difficulty or desire additional
coverage. As a result, the initial exposure to basic knowledge occurs
primarily outside the classroom and class time focuses more on the
application of that knowledge. This results in more effective use of the
professor’s expertise and of the limited student-professor time together in
class.
The author team has experimented with team learning on a variety of
courses at two different universities. Each university and course is described
briefly. Diverse team learning approaches were used. Drawing on our
experiences, we explore six issues in team learning. Three of these issues -
forming teams, training students in teamwork skills, and classroom
management and evaluation of student performance - reflect considerable
variation among our different approaches to implementing team learning.
The other three issues - faculty frustrations and concerns, student
performance, and student feedback - share considerable commonalties.
Finally, we offer conclusions regarding the team learning approach.
2. UNIVERSITY CONTEXT
2.1 Valparaiso University
Valparaiso University (VU) is a co-ed four-year private institution in

Northwest Indiana with a traditional residential student body. About 12% of
the 3,600 students are enrolled in the College of Business Administration.
Approximately 40% of the students come from Indiana and another 40%
come from other states surrounding the Great Lakes. About 69% of the
students come from the upper fifth of their high school graduating class.
About an eighth of the student body is involved in intercollegiate athletics.
Almost all the students are involved in intramural athletics and/or other
extracurricular activities. As a result, many students form strong collegial
bonds.
Team Learning in the Classroom 131
2.2 Purdue University-North Central
Purdue University North Central (PUNC), located in Northwest Indiana,

is a branch campus of the state supported university system. About one-
quarter of the 3,450 students take a four-year Baccalaureate, while the
remainder are registered in two-year associate programs. Approximately
one-third of the student body is registered in business programs. Many of
them are non-traditional students working full-time while taking one or two
courses per semester. In 1997, the average credit hours per student was 9.12.
Approximately half the students are full-time (12 or more hours) and half are
part-time. Even the full-time students are generally working more than 20
hours per week. A high proportion of the student population has families.
Attendance at classes is problematic due to work requirements, as well as
family and other commitments.
PUNC is a non-residential campus. Most students commute from three
towns about 10 miles away, although some commute more than 30 miles to
classes. For this reason, there is very little interaction between the students
outside of class. There are no intramural sports programs, few student
organizations and virtually no social activities. Therefore, students do not
form collegial bonds.
3. COURSE CONTEXT
At VU, team learning was used in the following courses: Financial

Management (Fin), Management and Organizational Behavior (MOB),
Management of Financial Institutions (Institutions), Intermediate Accounting
I and II (Acc I and Acc II), Advanced Accounting (Adv Acc), Tax Research
(TaxR) and Auditing (Aud). At PUNC, team learning was used in
Introduction to Business (IntB) and Small Business/Entrepreneurial Ventures
(Small Bus). Course level, objectives and content influenced the manner in
which team learning was implemented.
3.1 VU Junior-Level Courses
Fin is a highly quantitative course required of all business majors and

minors. Course objectives include developing familiarity with the financial
environment and basic terminology of finance; introducing the financial
concepts necessary to conduct business and to manage personal finances;
and enhancing communication and teamwork skills. Team learning was used
in Summer 2000 and Fall 2000. Class sizes ranged from 10 to 29 students.
The initial use of team learning in summer 2000 was very positive.
However, the uniqueness of the class, in terms of timing and composition,

raised questions as to whether team learning would work in a more typical
class. Summer courses are taught over a six-week period as opposed to the
standard term of fifteen weeks. Five of the ten students enrolled in Summer
2000 were repeating the course and had an average cumulative grade point
average (GPA) of 2.162 versus 3.159 for the first-time students. English was
a second language for five of the students; four of the five were extremely
fluent. The two sections in Fall 2000 had 29 and 27 students respectively. In
total, only four students were repeating the course and eight students in one
section had English as a second language.
MOB is required of all business majors and minors. The course
objectives are to foster development of the behavioral, communication and
self-management competencies that are necessary for success in an
increasingly dynamic, competitive and global environment. Team learning
was used in MOB in Fall 2000 and Spring 2001 with two sections each term.
About 20 to 30 students, typically consisting of an approximately equal
number of business majors and business minors, enroll in each section.
Institution is required of all finance majors. The primary objective of this
course focuses on measuring and managing risk in financial institutions.
Team learning was used in this course in the Spring 2001 term with 29
students.
Acc I and II are required of all accounting majors. Course objectives
include understanding and interpreting real-world financial statements,
exploring the concepts and theories underlying financial accounting and
reporting and applying the concepts and theories to complex economic
transactions. The team learning approach was used in Fall 2000 (Acc I) and
in Spring 2001 (Acc II) with 11 students.
3.2 VU Senior-Level Courses
Adv Acc is taken by accounting majors. The course objective is to

understand advanced accounting topics and their application to specific
factual situations. Team learning was used in the Spring 2001 term with 9
students.
TaxR is taken by accounting majors. The course objective is to examine
the application of accounting principles to tax issues and problems in
specific factual situations. Team learning was used in TaxR during the
Spring 2001 term with 12 students.
Aud is taken by accounting majors. Course objectives include
understanding the responsibilities and challenges facing the public
accounting profession and learning the basic concepts, principles and
standards of audit practice. Team learning was used in Auditing in Fall 2000
with 11 students.
3.3 PUNC Courses
IntB, a freshman level course, is required of all two-year associate and

four-year baccalaureate students in business. Course objectives are to
introduce the functional areas of business and to explore contemporary
business issues. Team learning was used in Fall 2000 in two sections of 21
and 30.
Small Bus is taken by juniors and seniors in the four-year baccalaureate
program. Course objectives focus on developing teamwork, problem solving,
presentation and persuasion skills. Major emphasis is on strategic planning,
organization, leadership and control of the small business. Team learning
was used in Fall 2000 and Spring 2001 with two sections each term. Section
sizes were 6, 24, 19, and 20.
4. FORMING TEAMS
One of the primary features of team learning is purposeful exposure to

heterogeneous work teams (Kaplan & Welker, 2001, p.16). A common goal
of team formation in our courses was to provide team members with the
opportunity to learn how to work effectively with a more diverse group of
people. Variations among courses occurred because there is not a single
optimal way of forming teams nor is there a single optimal team size.
In Summer 2000, the Fin students introduced themselves, discussed their

strengths and weaknesses and then self-selected into three groups - two
groups of three students and one of four. In Fall 2000, students introduced
themselves as above and then self-selected into one of two groups, those
comfortable with numbers and those who were not. The professor then
placed students into groups of four in one class and groups of three in the
other, balancing mathematical orientation, gender and ethnicity.
In Fall 2000, the MOB students participated in an ice breaking/team
formation exercise at the beginning of the semester. The students separated
into two groups, business majors and business minors, and then paired up
with an unfamiliar member of the opposite group. If the business major and
business minor groups were unequal in size, some triads were formed. After
an initial get-acquainted conversation, the dyads (and triads) were then

aggregated so that each learning team comprised approximately one-sixth of
the class. The resulting learning teams had a mix of competencies, values,
perspectives, orientations and supervisory experience.
In Spring 2001, psycho-geometrics (Dellinger, 1996) was used for the
purpose of team formation. Psycho-geometrics use shape selection to
differentiate people based on key personality traits. The MOB students chose
a circle, square, rectangle, triangle or squiggle, based on the shape with
which they felt the most affinity. Using these self-selections, the professor
formed heterogeneous learning teams.
In Spring 2001, Institutions students introduced themselves and chose a
shape as above. The students self-selected into groups balanced by shape.
There were six groups of four and one group of five. While the groups were
heterogeneous by shape, friendship and previous academic achievement
influenced group formation thereby introducing some homogeneity.
In Acc I and Acc II, teams of three or four were balanced on the basis of
gender. In Acc II, relative ability was also a factor. Teams were altered from
Acc I to Acc II so that no more than two members of a team worked together
in both semesters.
In Adv Acc, students, who knew one another well, self-selected three
member teams and the instructor assigned uncommitted students to teams.
In TaxR, students self-selected, typically on the basis of friendship, into
dyads to take quizzes and do the tax return problems. For the tax research
problems, teams of different personality types were formed using psycho-
geometrics as described above.
In Aud, students were assigned to teams on the basis of ability, as
evidenced by previous course grades, and gender.
4.3 PUNC Courses
In the PUNC environment, it is a challenge to effectively form teams. In

general, groups (rather than teams) were formed randomly by the students
themselves, typically based on seating proximity. Limited group
restructuring occurred throughout the semester resulting in greater
homogeneity.
5. TRAINING STUDENTS IN TEAMWORK SKILLS
The extent of team training varied from course to course, from none to
very extensive. We have become increasingly convinced that some degree of
training in teamwork skills is essential for successful team learning to occur
(Christ, McCuddy & Pirie, 2001).
In Fin and Institutions, a discussion of teams and team learning took

place on the first day of classes. This included contrasting teams versus
groups, team formation stages, skills necessary for effective teamwork, roles
in teams and expectations regarding feedback to fellow team members.
In MOB, early exposure of students to fundamental concepts and ideas
about teams and teamwork is an integral part of the course. Most of the first
five 75-minute classes are devoted to the nature of management and
organizational behavior in contemporary businesses; the roles of teams,
particularly self-directed work teams, in modern businesses; and the skills
required for success in the contemporary workplace.
At the beginning of Acc I, the purpose of the team learning approach was
discussed and a handout on characteristics of effective teams was distributed.
Approximately half a class period was spent discussing teamwork skills. In
Acc II, a continuation of Acc I, no additional training was provided.
No training in teamwork skills was given to students in Adv Acc or

TaxR. The students already had MOB in which the use and development of
teamwork competencies was a significant component. The benefits of
working in teams were discussed, but no time was spent on the specifics of
developing team competencies. Students in Aud had previously taken MOB.
The purpose of the team learning approach was discussed but no teamwork
training occurred.
5.3 PUNC Courses
No teamwork training occurred. On projects completed outside of class,

group work, not teamwork, occurred. Generally, the groups split
responsibilities for the term projects. The students functioned independently
of each other. A student assigned responsibility was chastised if the work
was not completed on time and/or if the grade was lower than the group
deemed satisfactory. This behavior persisted in spite of repeated admonitions

about the advantages with respect to learning and quality of work that
typically result from true teamwork. Some groups did spontaneously form
teams with bonds that persisted outside of class time. These groups appeared
to be more motivated and achieved a better rate of learning.
6. CLASSROOM MANAGEMENT AND

EVALUATION OF STUDENT PERFORMANCE
According to Kaplan and Welker (2001), the primary features of team

learning include:
• a grade based on individual and team outputs and peer evaluation of
performance within the team;
• a large amount of class time devoted to team activities; and
• class time focused on the application of concepts.
As noted in the introduction to this paper, faculty members are most
involved at the understanding and application stage while initial exposure to
basic knowledge occurs primarily outside the classroom. Discussion and
class activities focus on developing concepts and topics where students
experience difficulty or desire additional coverage. The use of the limited
student-professor time together is thereby optimized.
Common to each course, students were responsible for reading material
prior to class. Individual quizzes and/or team quizzes (subsequently referred
to as pre-quizzes) based on this reading were administered. Students had an
opportunity to ask questions about the assigned material prior to taking the
quizzes. Initially there were few questions, but soon students recognized the
benefit of asking questions.
When both individual and team pre-quizzes were used, the students
turned in the individual quiz prior to taking the team quiz. In all instances,
during team quizzes, the professor walked around the room and answered
questions. The nature of a question dictated whether the response was
addressed to the individual team or the entire class. Interactions among team
members were observed. Usually, each member was involved in developing
the team answer, especially when there was a difference of opinion.
After the pre-quiz(zes), in-depth discussion of the material occurred.
Discussion addressed areas of difficulty evident from the pre-quizzes and
additional student questions and generally advanced the material beyond the
textbook’s content. Although discussion was partially based on students’
questions, the professor implicitly influenced it by the selection of pre-quiz
questions.
6.1.1 Financial Management
Individual pre-quizzes of a short-problem nature were used. The teams

then took the same quiz. Subsequent to class discussion, an individual post-
quiz was administered. Each team also applied course concepts in analyzing
two companies. During the term team members provided each other with
feedback and at the end of term each team submitted to the professor a team
evaluation of individual members. Students took an individual
comprehensive final exam.
6.1.2 Management and Organizational Behavior
MOB was structured around six course content segments. Reading

assignments were selected to fit with each segment. Lectures did not cover
the reading assignments per se but focused on fundamental teamwork
concepts and knowledge (course segment 1) and the application of an
integrative conceptual model of organizational management (course
segments 2 through 6). Lecturing and discussion typically occurred at the
beginning of each segment and usually consumed about two 75-minute class
periods. Subsequently, answers were provided to further student questions.
There were several assignments with respect to the six content segments,
some team-based and some individually-based. In Spring 2001 the activities
included the following: a multiple-choice team quiz after the initial content
lecture/discussion on a segment; an individual multiple-choice quiz after the
grading and review of each team quiz; and a team analysis of a company that
applied the conceptual model developed in content segments 2 to 6. In
addition, each student drew on learning team experiences to write a paper
reflecting on, and critically appraising, his/her leadership, followership and
teamwork experiences and competencies. In the first iteration of team
learning in MOB, individual pre-quizzes were used but were dropped due to
the excessive amount of time consumed by in-class quizzes.
6.1.3 Management of Financial Institutions
The general procedure for Institutions was similar to that of Fin.

However, the pre-quizzes in this more advanced course were objective in
nature and focused on terminology and conceptual issues. The teams were
also assigned problem sets of a technical nature for discussion in class and
for grading. The post-quizzes typically covered a number of chapters and
were problem-oriented. A significant portion of the course grade was based

on an individual problem-oriented midterm and final exam. At the end of the
semester, each team submitted to the professor an evaluation of individual
members for the team portion of the grade.
6.1.4 Intermediate Accounting I and II
Individual pre-quizzes were mostly multiple-choice with some matching

or short answer questions and problems. The students also completed the
same quiz in teams. After the team quiz, students graded their own quizzes
and asked questions regarding areas of difficulty.
On non pre-quiz days, discussion of assigned homework problems
occurred. Additional problems were also completed in class by the teams.
The material in Acc II is significantly more difficult than in Acc I.
Therefore, some changes were implemented to the above approach. In order
to reduce quiz time, chapters were grouped for quizzing purposes. Students
compared their solutions to the homework problems within their teams
before the solutions were reviewed as a class. These team discussions helped
focus the questions on the homework. These two changes allowed the class
to devote more time to concentrating on challenging concepts and working
on in-class problems.
In both Acc I and II there were significant individual and team projects as
well as three individual tests. At the end of the semester, each student
completed an evaluation of the team members’ contributions to the team
efforts.
6.2.1 Advanced Accounting
For Adv Acc, the original intent was to have individual and team
multiple-choice quizzes prior to discussion of the chapter material and
assigned problems. However, many of the assigned problems were very time
consuming and, in the interest of preserving class time for discussion, only
the team quiz was given for most chapters. The quizzes were a mix of
conceptual and computational questions. After completion of the team quiz,
the correct answers were discussed before addressing the homework
problems. No quiz was given at the conclusion of a chapter but students
were assigned individual computer projects that applied the material. Three
individual exams and two research papers were required as well.
6.2.2 Tax Research
For TaxR teams did tax research problems and tax returns outside of
class and took in-class quizzes. Prior to discussing the assigned problems,
the professor responded to specific questions over the chapter material.
Teams provided conceptual and numerical answers to the assigned problems.
Quizzes, consisting of true-false questions (with an explanation required for
any false answer), were administered after discussing the chapter and
assigned problems. Three individual exams were also administered.
6.2.3 Auditing
The approach in Aud mirrored that of Acc I for pre-quizzes. On non pre-
quiz days individual assignments consisted of conceptual essay questions,
application problems and/or case studies. Unless there were questions, little
class time was spent discussing responses to essay questions. For problems
and case studies, teams reviewed the individual responses and then each
team presented the team solution to the class. Upon completion of a chapter
the students took an individual post-quiz similar in format to the pre-quizzes
but not identical. There were significant individual and team projects as well
as individual midterm and final exams. At the end of the semester, each
student completed an evaluation of the team members’ contributions to the
team efforts.
6.3 PUNC Courses
6.3.1 Introduction to Business
The beginning of each class was spent discussing current business affairs
in the context of the course material. Students were given the opportunity to
ask about text material that they had not understood. Responses to these
questions were solicited from the class rather than just having the instructor
answer. Some excellent ideas were often forthcoming from the students. The
students then took an individual pre-quiz followed by the same quiz in
teams. After the team quiz a case that applied concepts from the chapter was
discussed. Students took midterm and final exams and completed term
projects.
6.3.2 Small Business/Entrepreneurial Ventures
The classroom management approach for Small Bus mirrored that of

IntB. In addition to the pre-quizzes, teams submitted case write-ups. Each
student prepared a business plan and took three tests, one of which was a
comprehensive final.
7. FACULTY FRUSTRATIONS AND CONCERNS
There were three sources of frustration common to all the professors. One
was a sense of underutilization of faculty knowledge; another was the
challenge of student preparation; and a third was the time consumed in
quizzes.
When the learning teams were working on the in-class quizzes, the
professor sometimes felt underutilized. There is an inherent desire to impart
“wisdom,” rather than allowing students to learn collaboratively. This is a
manifestation of the latent, but nonetheless powerful, desire to be the “sage
on the stage.” These moments are, in fact, signs that self-management is
alive and well in these learning teams and that the learning teams are
operating as intended, as collaborative vehicles for students to learn
effectively on their own. A related frustration is that other faculty might
perceive that we are not working because we are not lecturing. There is
debate as to which approach, lecture or team learning, better facilitates
student learning, retention and synthesis of the material.
Student learning, retention and synthesis of the material will occur with
team learning only if the students assume responsibility and prepare in
advance. For example, a PUNC student, a single mother with a young child,
for the first few weeks felt overwhelmed by the time commitment required
to read the text, but finally dealt with the issue by allocating the required
preparation time. An unintended consequence of team learning is the
development of time management skills. At VU, the students are generally
bright students who have never had to study very hard. Team learning forces
the students to develop better study habits and to come to class adequately
prepared. This promotes the joint objectives of developing independent
learning and lifelong learning skills.
While the pre-quizzes may promote independent and lifelong learning
skills, the amount of time spent on quizzes as opposed to discussion was a
major source of frustration. This time management concern prompted a
reduction in the number of quizzes in some courses. However, it became
evident that the pre-quizzes forced students to expend more effort on reading
and comprehending the material. When pre-quizzes were eliminated, some
students seemed to assume the professor would take responsibility for

explaining the material. To deal with this in future semesters, some
individual quizzes will be completed outside of class, either through on-line
testing or take-home quizzes. This will leave the team quizzes to be
completed in class, thereby allowing adequate class time for discussion and
other team activities.
8. STUDENT PERFORMANCE
All the authors sensed that student performance improved with team
learning as opposed to lectures. This manifested itself in a number of ways.
At PUNC, attendance in team learning courses was higher than in lecture
courses, greater than 75% compared to less than 60%. At VU, attendance
typically exceeds 90% regardless of teaching method. At both universities
the level of preparation for class increased dramatically. The number of
students participating in class and the quality of participation increased. All
of the above led to grades on the individual components of courses that
appeared to be higher than in lecture-oriented courses, indicating a better
grasp of the material.
With respect to the level of preparation, it was evident from students’
questions that they had read and studied the assigned material. There is a
great deal of information in a textbook, much more than can be imparted
effectively, given time constraints, in a lecture course. Team learning helps
overcome this constraint by ensuring that a portion of the text material is
learned by the students working independently. Questions directed the
discussion and generally advanced the material beyond the textbook content.
The use of teams encouraged everyone’s involvement in the discussions.
Students with less self-assurance gained confidence interacting with their
team and thus felt sufficiently comfortable to participate in class discussions.
The class discussions were definitely of a higher quality than normal.
Students asked insightful questions and gave insightful answers, thereby
demonstrating a sound grasp of the material. Faculty frequently acted as
discussion facilitators rather than as experts imparting knowledge. The
authors believed that the increased student responsibility for learning led to
improved performance.
Consistent with this perception, performance on the applied analyses in
MOB was superior to that on the same assignments using different
instructional methods. Final exam grades and the ultimate course grades in
all courses appeared to be higher and thus indicated that the students had in
fact learned the material better than students had in previous semesters.
A regression analysis was run to test the faculty perception at VU that

students learned the material better with team learning. There were 661
observations across eight courses. Table 1 provides details on the number of
observations (students) per course. Each course had an individual professor
who taught it in more than one way - using team learning, lectures or
lectures as part of the integrated functional core (IFC). This was an
experimental team-taught course that combined four stand-alone courses into
a 12-hour block, including Fin and MOB. The dependent variable measuring
the individual's concept mastery was either the grade on the final exam (Fin,
Institutions, Adv Acc, TaxR, and Aud); the skills paper (MOB); or the
average of three exams in the course (Acc I and II). No baselines without
team learning were available for the PUNC courses.
The overall grade for the course was also used as an alternative
dependent variable. This allowed for a comparison across courses
unhindered by the individual professor’s grading scheme. The final course
letter grade was converted to a percentage.
Table 1: Number of Observations.

Course Name Lecture IFC Team Learning Totals
Acc I 17 10 27
Acc II 18 11 29
Adv Acc 12 9 21
TaxR 10 12 22
Aud 13 11 24
Fin 121 89 61 271
Institutions 33 29 62
MOB 15 90 100 205
Totals 239 179 243 661
Notes
The data consist of the entire class rosters for which final grades were submitted with 20
student observations dropped for the following reasons:
Transfer students whose prior GPAs were unavailable (10).
Students who did not complete the course (5).
Visiting students whose prior GPAs were unavailable (4).
Access denied to student records (1).
Cumulative GPA prior to the course was an independent variable, used as

a surrogate for the differential ability of the students. Dummy variables were
created for classes using team learning and for classes that were part of the
integrated functional core. Dummy variables were also created for each
course because each professor used a different grading system. No intercept
was estimated and the coefficient on each course dummy variable is in effect
an intercept term for each course, the grade that would be achieved for a
student with a prior GPA of zero who was in neither a team learning nor
integrated learning course.
The regression equation is as follows:
Dependent variable = β1 GPA + β2 Dummy Team + β3 Dummy IFC + β4
Dummy Acc I + β5 Dummy Acc II + β6 Dummy Adv Acc + β7 Dummy TaxR
+ β8 Dummy Aud + β9 Dummy Fin + β10 Dummy Institutions + β11 Dummy
MOB
The results of the regression analysis are shown in tables 2 and 3. For
example, the regression for the skills paper in MOB in a team learning
format is as follows:
Grade on skills paper = 15.89 GPA + 2.36 (1) + 1.15 (0) + 21.15 (0) +
25.91 (0) + 39.60 (0) + 17.87 (0) + 23.56 (0) + 25.09 (0) + 32.23 (0) +
34.11 (1)
Table 2: Regression Results with the Grade on the Final Exam, Skills Paper or Three Exams
as the Dependent Variable
Estimated Standard Error 2-tailed
Coefficient of the Estimate T-Statistic P value1
Prior GPA 15.89 0.85 18.80 0.00%
Team Learning 2.36 1.12 2.11 3.52%
IFC Course 1.15 1.28 0.90 36.84%
Acc I 21.15 3.37 6.27 0.00%
Acc II 25.91 3.31 7.82 0.00%
Adv Acc 39.60 3.74 10.58 0.00%
TaxR 17.87 3.64 4.91 0.00%
Aud 23.56 3.58 6.57 0.00%
Fin 25.09 2.58 9.73 0.00%
Institutions 32.23 2.98 10.82 0.00%
MOB 34.11 2.73 12.47 0.00%
Notes
1
Probability of the null hypothesis that the coefficient could be zero
Adjusted R2 = 0.4416
F-statistic = 53.1945
Team learning leads to improved individual learning. The grade on the

final exam, the skills paper or the average of three exams was 2.36% higher
than in lecture format classes at the 95% confidence level. The overall
course grade was 3.17% higher (significant at 0.0002%). Clearly, the level of
proficiency in cognitive/technical knowledge was increased while providing
class time for the development of other important competencies. An
improvement in the level of proficiency in cognitive/technical knowledge
exceeded the original objective, which was to provide time for and promote
the development of other important competencies.
Table 3: Regression Results with the Overall Grade for the Course as the Dependent Variable.
Estimated Standard Error 2-tailed
Coefficient of the Estimate T-Statistic P value1
Prior GPA 10.05 0.50 20.15 0.00%
Team Learning 3.17 0.66 4.81 0.00%
IFC Course 2.82 0.75 3.75 0.02%
Acc I 50.92 1.99 25.56 0.00%
Acc II 51.00 1.96 26.07 0.00%
Adv Acc 47.33 2.21 21.41 0.00%
TaxR 48.48 2.15 22.56 0.00%
Aud 54.27 2.12 25.64 0.00%
Fin 51.27 1.52 33.67 0.00%
Institutions 52.88 1.76 30.06 0.00%
MOB 53.92 1.62 33.38 0.00%
Notes
1
Probability of the null hypothesis that the coefficient could be zero.
Adjusted R2 = 0.4292
F-statistic = 50.6335
9. STUDENT FEEDBACK
Generally, student feedback was very positive. Informal discussions with

students indicated that they enjoyed the approach and particularly liked the
team interactions. They felt this provided a stimulating classroom
experience. Several students commented that team learning was their most
successful educational experience, having learned much more than in other
courses they had taken. Students indicated that they had learned how to
study for the first time; learned time management skills; gained confidence
in their ability to learn on their own; and learned teamwork skills.
Students generally approved of not discussing material in the book that
was straightforward and instead preferred to concentrate on the material that
was less clear and more challenging. However, there was some student
concern about being tested on material prior to a lecture on it. As the
semester progressed, students became more comfortable with identifying
assigned material that was causing them difficulty and asking questions
about it before the quiz was given. Some professors alleviated this concern
by curving the results on the pre-quiz.
There was a positive student response to working in teams because it
provided them with multiple perspectives on a problem or issue and allowed
them to clarify their understanding of the material. Students of varying
abilities benefited from this approach. Those requiring assistance in
comprehending the material had it readily available and those who easily
understood the material had the opportunity to strengthen their

understanding through explanation. As one student said:
“Everyone has something to contribute to a team, no matter how large or
small. While working in teams pushed my patience at times, I think that
greater knowledge can come out of teamwork. On my team, I had five
teachers that helped me, instead of just myself.”
The students’ primary concerns were with team formation and team
member evaluation. On occasion the students were upset that the professors
formed the teams rather than letting the students form their own teams.
However, allowing student self-selection jeopardizes the team heterogeneity
that is critical to the development of the skills necessary for working
successfully with members of an increasingly diverse workforce.
A related concern in working with unfamiliar team members is that a
non-performing team member might drag down the team. There were limited
complaints about team members shirking their responsibilities. The teams
generally established fairly homogeneous expectations of the effort required
by each team member although each team established different expectations.
In addition, team members had input regarding the evaluation of their peers.
According to another student:
“The realization that my grade would be dependent on four strangers
increased my uneasiness about the course. This uneasiness was caused by
my past experiences with group work in classes, and the lack of control I
had over my grade. Despite my initial fears, the team activities in this
class were beneficial and led to a positive educational experience.”
The students also expressed some concern with the amount of time spent
on quizzes. One student wrote on the faculty evaluation that: “I am not
paying all this money to teach myself!” The negative comments were in the
minority, however. In cases where a vocal minority complained about team
learning, given the opportunity no concrete recommendations were advanced
by the minority for an alternative that the other students were prepared to
accept.
The overall power of the learning team approach might be captured
through the experience of a high school senior. She visited the IntB class
twice and took the tests with no preparation. She enjoyed the experience to
such an extent that she intends to enroll in it. This indicates that even
prospective students recognize that team learning is an “exciting” way to
engage students in the learning process. As a recent student noted:
“I felt that this experience was not always fun, but most definitely a great
learning experience.”
10. CONCLUDING OBSERVATIONS
Team learning results exceeded our expectations. Student performance

on tests of cognitive/technical knowledge was superior to that of students
taught in a lecture format. Observations of students interacting in teams
indicated that behavioral skills were developing as desired. An advantage of
team learning is the professor’s opportunity to observe team members
interacting. Student feedback indicated that additional skills such as time
management and study skills were enhanced as well.
There is transference of responsibility for learning from the
professor/lecturer to the student/team learner. Using the team learning
format the students are in charge of their own learning, particularly with
respect to behavioral skill development; understanding of the reading
material; and ability to apply conceptual models to real-world situations. The
professor’s role is one of explaining and elaborating on the material in
response to questions, evaluating student performance and providing
feedback. While the quizzes potentially guide students’ questions, the
professor has to be well prepared and able to respond to a variety of
questions. It is a source of delight and challenge when student interest is
sparked in an unexpected area. Team learning is a stimulating experience for
all participants, and one that we would recommend heartily.
REFERENCES
Christ, M.Y., McCuddy, M.K., & Pirie, W.L. (2001). Effective teamwork doesn’t just happen:
Preparing students for meaningful teamwork experiences. Paper presented at the 8th
Annual EDiNEB International Conference, Nice France, June.
Dellinger, S. (1996). Communicating beyond our differences: Introducing the psycho-
geometrics system. Tampa, FL: Jade Ink.
Kaplan, J.E., & Welker, M.G. (2001). Team learning is here to stay! New Accountant, 16 ( 3),
14-16.
Michaelson, L., & Price, M. (1999). Integrating the undergraduate business core. Paper
presented at the AACSB Continuous Improvement Seminar, Minneapolis, September.
Using Teams in the Classroom: Meeting the Challenge
of Evaluating Students’ Work
Michael K. McCuddy & Wendy L. Pirie

College of Business Administration, Valparaiso University, Indiana, USA
1. INTRODUCTION
Using groups and teams in the classroom seems to have a momentum of

its own as more and more educators incorporate various forms of group or
team activities into their courses (e.g. Aiken, 1991; Alie, Beam, & Carey,
1998; Boyatiz, 1994; Confessare & Confessare, 1992; Johnson, Johnson, &
Smith, 1994; Knowles, 1975; Langrehr et al., 1998; Malinger, 1998;
Malinger & Elden, 1987; McCuddy, 1995; McCuddy & Pirie, 2000;
Michaelson, 1992; Michaelson & Price, 1999; Miller, 1991; Ramsey &
Couch, 1994). As educators embrace teamwork in the classroom, they
encounter new and different challenges in attempting to make the
teaching/learning enterprise an effective venture. Included among these
challenges are the determination of effective team member role behaviors
and skills (Christ, McCuddy & Pirie, 2000); dealing with “free riders”, that
is those students who do not carry their fair share of the team’s burden; and
the evaluation of student performance (Malinger, 1998).
We believe that one of the more important challenges associated with the
use of groups and teams in the classroom is the evaluation of student
performance, both individually and collectively. The evaluation of student
performance has two objectives when teams are used. One objective is to
provide an appropriate grade to each student based on his/her contributions
to the team project. The second objective is to provide feedback of a
developmental nature to the students, since a fundamental purpose of
assigning team projects is to help students improve their teamwork skills.
147
148 Michael K. McCuddy & Wendy L. Pirie
Fellow team members are the most knowledgeable as to the individual’s

performance within the team. Thus, it is imperative that information from
them is used in the evaluation process, both for grading and feedback. This
paper explores the nature of this performance evaluation challenge.
2. EVALUATING PERFORMANCE IN DIFFERENT

TEAM PROJECT SETTINGS
We have used student project teams in many different settings. These

projects can be broadly divided into three categories: (1) applying concepts
to a hypothetical situation; (2) acting as student consultants to a real
organization; and (3) using concepts to analyze real organizations.
2.1 Applying Concepts to a Hypothetical Situation
2.1.1 Plan for a Single Major Event
A project that was used for several terms involved an entire Management
and Organizational Behavior (MOB) class working together as a single
organization to develop a plan for staging a hypothetical “National Sports
Festival” at some specified location in the USA. Developing the event plan
required the students to address all relevant aspects of staging a sports
festival, including but not limited to:
• developing a mission statement that clearly specified the festival’s
purpose while achieving financial solvency and profitability;
• selecting events to be included in the festival;
• selecting and developing venues for staging the events;
• securing and managing human resources to operate the festival;
• managing relationships with the relevant sports associations, corporate
sponsors, government, media, and other interested parties;
• securing financing for the festival;
• developing plans for the daily operation of the festival;
• providing transportation, housing and other logistical support for
participants and spectators;
• marketing the festival; and
• dismantling the organization once the festival has been staged.
In summary, this project was somewhat akin to developing a plan for
staging the quadrennial Olympic Games, although not on as grand a scale.
Teams in the Classroom 149
Fundamentally, the students were charged with working together in

creating and operating a self-managing organization that was responsible for
developing a business plan. The required output of the organization
consisted of a detailed written plan, this typically was 200-250 pages in
length, and an oral presentation of the plan to a panel of faculty and outside
business persons.
In some semesters two sections of the course worked together on one
project. In other semesters each of two sections of the course worked on
separate event plans, though both were of similar magnitude. A section of
the course usually contained approximately 30 students.
The project grade constituted 50% of the course grade. In some semesters
35% of the project grade was allocated to the performance of the entire
project organization; 35% to performance of the student’s primary work
team; and the remaining 30% to the individual student’s own performance.
In other semesters 25% of the project grade was for organization-wide
performance; 25% for team performance; and 50% for individual
performance.
Every student was required to evaluate his/her own contributions as well
as those of the other members of his/her primary project team. They also had
the option of evaluating the performance of members of other project teams
with whom they had significant interactions during the project. These
evaluations were then used in determining the individual component of the
project grade. The professor determined the team and organization-wide
components.
2.1.2 Plan for a Business
Another approach, similar to the event plan discussed above, was a

project that focused on developing a detailed plan for establishing and
operating a hypothetical business. This format focused on creating a plan for
establishing an on-going business concern with long-term implications. The
event planning project, in contrast, focused on a staging a one-time event
with shorter-term implications.
This project format was used one semester with a single class of
approximately 20 students. The entire class worked together in a self-
managing organization that was required to identify, create, and/or develop a
product or service that did not exist in the marketplace; develop a detailed
business plan for producing and marketing this product or service; utilize
lessons contained in the book NUTS! Southwest Airlines’ Crazy Recipe for
Business and Personal Success (Freiberg & Freiberg, 1996) in developing
the business plan; and ensure that the plan considered the implications of
maintaining the business as an ongoing commercial entity.
The business plan covered all conceivable aspects of business operations,

including relevant human resources, marketing, finance, operations, legal
issues, etc. The students chose to develop a plan for establishing a private,
mass transit system in and around the city of Valparaiso, Indiana in the mid-
western USA.
The students made an oral presentation of the business plan to a panel of
faculty and outside business people. The project grade constituted 50% of
the course grade, with 50% of the project grade allocated to individual
performance, 25% to team performance and 25% to project organization
performance. The peer evaluation requirements were the same as with the
event-planning project discussed above.
2.2 Acting as Student Consultants to a Real

Organization
2.2.1 Special University Event
In the Fall 1996 semester, two sections of MOB worked together as a

self-managing project organization to develop, in conjunction with an ad hoc
university committee, a detailed plan for a staging a major, special event in
October 1997. The event was a football game between Valparaiso and Yale
Universities at Soldiers Field in Chicago, Illinois. There were numerous
ancillary activities associated with the football game.
The self-managing project organization was charged with developing a
plan that would satisfy the needs and interests of many different university
units as well as being financially successful. The project organization
developed a detailed written plan for this event and orally presented the
highlights of the plan to members of the ad hoc committee and several
university administrators.
The self-managing project organization was divided into eight teams,
each with different responsibilities. Each team did a periodic assessment of
its members. Each team had to reach consensus on evaluating each of its
members in terms of contributions to fulfilling the team’s project
responsibilities. These assessments determined the individual grade, which
was worth 50% of the project grade. Team performance and project
organization performance, both of which were evaluated by the professor,
were each 25% of the project grade. Taken together, the components of the
project grade constituted 50% of the course grade.
2.2.2 External Clients
In the Integrated Functional Core (IFC), an experimental twelve-hour

block of classes that combined the four stand-alone courses of Financial
Management (Finance), MOB, Marketing Management (Marketing) and
Management Information Systems (MIS), the students were required to
perform a major consulting project for an external client. The IFC was
offered four times in a three-year period. Each IFC class had a single client.
The clients were Camp Arcadia, a small Lutheran residential camp;
Canterbury Place, a long-term care facility; and Maria Elena’s Restaurant, a
small restaurant being opened in a disadvantaged neighborhood with the
assistance of a non-profit group organized to help the neighborhood. Two
different projects were undertaken for Canterbury Place.
The client project counted for 18% of the final grade in each of the four
component classes of the IFC, so there was considerable incentive to
perform well. The complexity of the projects undertaken resulted in the class
dividing the work amongst sub-teams. This involved a lot of coordination. A
final grade was given to the class as a whole. This grade was then adjusted
for each individual student based on evaluations by the class of the
performance of each sub-team and each individual. The class as a whole met
and agreed on the evaluations that were then passed on to the professors.
This final exercise, one of peer evaluation, was a very difficult and time-
consuming one for the class. In all four instances, the class initially indicated
to the professors that it would take about five minutes to do but they found
that it took well over three hours. A lot of thought went into the final peer
evaluations.
2.3 Using Concepts to Analyze Real Organizations
2.3.1 Matched-Pairs Application
In the IFC, the Finance and MOB professors used a combined midterm
and an extended project throughout the term, which applied the concepts in
the courses to two companies. The class was split into teams with each team
responsible for their own set of companies. Companies were compared
within industries by the teams for the written reports and orally across
industries within class.
The following is an example of the instructions given to each team
regarding grading of team projects in Finance.
“When turning in each project, each team will decide the division of the
project’s marks within the team based on contribution. The team will list
the students in the team and allocate 100% amongst the group based on
contribution. Example of Allocation and Its Result: The team decides to
allocate the marks as follows: M1- 25%, M2-25%, M3-20% and M4-30%
(note these add up to 100%). A team grade of 87 will result in M1 and
M2 getting 87, M3 getting 70 (4 x 87 x .2 = 70), and M4 getting 104.
Therefore, do not think you are going to get by on the team’s efforts.
Further, I reserve the right to administer an oral examination on the team
project and to adjust the mark allocation based on the member’s
familiarity with the team project.”
In MOB, it was indicated that individual performance within the team
would be taken into account. The specifics of the MOB allocation are
discussed in the following section of the paper. In both MOB and Finance,
the intent was to eliminate the free rider problem. Interestingly, the students
viewed this not as an opportunity to punish but as an opportunity to reward.
Rather than viewing an uneven split as recognition of inferior efforts, they
viewed a higher allocation to one individual as a reward for superlative
effort. They were grateful for the opportunity to thank outstanding team
members. In effect, they gave some members a bonus for their efforts.
In other finance courses a similar approach was used. In Management of
Financial Institutions, student teams applied the concepts to a single
company and an industry sector. In International Finance, the concepts were
applied to individual companies and to individual countries. Student teams
presented their findings to the class throughout the term. Bringing in real
data added life and relevance to the course, and also demonstrated the
difficulty of applying theoretical concepts to complex and ambiguous
situations.
2.3.2 Applying a Conceptual Model to Specific Companies
MOB was structured around six course content segments: teamwork

concepts and knowledge (a stand-alone segment) and the application of an
integrated contingency system (ICS) of organizational management
(segments 1 through 5 of the ICS). Student teams were required to do an
applied analysis, using the ICS in whole or in part. The extent of ICS usage
varied across semesters. In four semesters, teams comparatively analyzed
two companies using the first two segments of the ICS. (This is the MOB
component of the matched-pairs application discussed above.) In another
semester, student teams of four or five members wrote two applied analyses
using the ICS. In the first assignment, each team comparatively analyzed two
companies in terms of segments one and two of the ICS. In the second
applied assignment, each team analyzed a single company, though different
from the two companies used in the first assignment, using all five segments
of the ICS. In a further semester, student teams of three to six members
prepared a written analysis of an assigned company for each segment of the
ICS.
The approach to evaluating team members also varied across semesters.
Two different evaluative approaches were used with the matched-pairs
format. One approach evaluated each team member’s individual company
analysis and the team’s comparative analysis. Half of each person’s project
grade was allocated to the individual component and half to the team
component. The other evaluative approach awarded an overall grade that
was then allocated amongst the team members. For example, if a team’s
analysis assignment received a grade of 90 out of 100 points and the team
consisted of five members, the team received a total of 450 points (5
members x 90 points earned) to distribute among the members. The team
members were required to discuss the situation and come to a consensus
allocation of points.
When the teams wrote two applied analyses, the second evaluative
approach discussed above was used, and each analysis was worth 20% of the
course grade. When the teams wrote five applied analyses, each analysis was
worth 7% of the course grade. There was no differential grading amongst
team members. Having 35% of the course grade based on the analyses was a
significant motivating factor for most students. While formal peer
evaluations were not utilized, the comments of many students in an
individual paper written at the end of the semester indicated that informal
peer evaluations occurred within the team. This self-monitoring behavior
mirrors the behavior of effective teams in non-academic settings.
3. APPROACHES TO PEER ASSESSMENT
The team projects described above each use one of the two basic
approaches to peer assessment: individual assessments that the professor
aggregates and averages or consensus assessment. The former has the
advantage of requiring little time but fails to achieve the objective of
providing feedback of a nature that is meaningful and useful to the student
for behavioral development. For the latter approach to work, students must
be given guidance on peer assessment. Guidance should focus on the nature
of feedback, both corrective and supportive, and consensus building.
Corrective feedback focuses on a person’s developmental needs; it indicates
that a change in behavior is appropriate. Supportive feedback is targeted
toward a person’s strengths; it is intended to encourage and reinforce

behavior.
Three forms are attached in Appendices A through C that provide
students with focus in conducting more effective peer assessments.
Appendix A contains a form that is targeted toward individual assessments
that are of a global behavioral nature. Appendix B focuses on very specific
behavioral and performance criteria. Appendix C is targeted toward
developing a consensus evaluation of how members helped or hindered team
functioning. While each of these forms is targeted toward a specific purpose,
all can be readily adapted for use in either individual peer assessments or
consensus peer assessments. Moreover, all three could be used as a total peer
assessment package.
4. THE CHALLENGES OF USING PEER

ASSESSMENT IN EVALUATING TEAM
MEMBERS
Students generally receive little guidance as to how to assess peers but

are simply told to provide an evaluation for each team member. Peer
assessment is a challenge to experienced individuals and can be a daunting
task for the uninitiated. Instructions to students should cover the following
three areas at a minimum: focusing on concrete behavior rather than general
abstractions, considering effort as well as ability, and arriving at a
consensus.
4.1 Concrete Behavior vs. General Abstractions
This issue ties in very well with general points with respect to providing
constructive feedback. In providing such feedback, the following should be
done:
• The description of the behavior should be objective.
• Specific details rather than generalities should be stated.
• The feedback should deal only with behavior that the person can
strengthen or change.
• If relevant, a description of how the behavior affects other team
members should be included.
The following should not be done:
• Providing feedback for the purpose of scoring points.
• Embarrassing a person in a public forum.
• Providing feedback when either party is emotionally upset.
It should be conveyed to students that peer assessment is not being done

purely for grading purposes, but also to provide appropriate feedback to
team members in order to improve in teamwork skills. Students should be
encouraged to provide feedback to each other even when no formal feedback
is required by the professor. This did occur in the situation where no formal
assessment was required by the professor. To assist students in formulating
concrete behavioral feedback, Appendices A or B could be used.
4.2 Effort versus Ability
Students tend to focus on effort when providing corrective feedback and

on ability when providing supportive feedback. Criticism of fellow team
members to professors is generally related to behavior that hindered the
performance of the team, such as the failure to attend meetings or to do
assigned work. Praise of fellow team members to professors usually
concentrates on contributions of an intellectual nature such as helping fellow
team members grasp a difficult point or arriving at a creative solution to a
problem. In our experience, evaluative responses on a form such as that
shown in Appendix C support this perspective.
Students properly concentrate on providing corrective feedback that
focuses on behavior that can be changed. These behaviors reflect the effort
that students make rather than their ability to perform a given task. Even
without training in peer assessment, it appears that students intuitively know
that corrective feedback is inappropriate given an ability deficiency.
Students are very supportive of fellow team members who lack ability but
put forth effort. The corollary is that students are very critical of team
members who fail to make an effort. By extension, we might infer that they
would be most critical of team members who have ability and fail to put
forth effort.
4.3 Reaching Consensus
It is important that teams reach consensus rather than submitting

individual evaluations of fellow team members that are averaged by the
professor. A critical element of the consensus-seeking process is the giving
and receiving of corrective and supportive feedback. Consensus-seeking
enables a team member to learn what he/she needs to do to effectively
interface with different people. This does not occur when peer evaluations
are simply given to the professor.
In seeking consensus, all team members have input into the evaluation
process and the feedback reflects the collective wisdom of the team.
Individual evaluations, when used for grading purposes, can be unduly
influenced by one or two biased individuals within the group. Biased

evaluations are outliers in the averaging process. Feedback filtered through
the professor’s lens of reality, rather than coming directly from fellow team
members, can be biased, watered down, etc., thereby failing to capture the
full intent or emotional impact of the team members’ feedback.
Consensus-seeking requires considerably more effort in evaluation than
does the individual peer evaluation forms. Team members must justify
numerical ratings rather than simply awarding them. The evaluation form
presented in Appendix C is consistent with this observation. Reaching
agreement on justified, as opposed to, arbitrary ratings can consume a great
deal of time. The evaluation instrument in Appendix C requires team
members to reach agreement on the content of a 100-150 word statement
supporting each team member’s numerical rating. It takes considerable time
to craft a statement that accurately reflects the collective sentiments of the
team. The time involved in consensus-seeking was also evident in the
evaluations performed by the IFC classes, as mentioned earlier.
One of the difficulties in seeking consensus is that feedback occurs in a
semi-public forum. It is important to emphasize to the students that in this
context it is even more critical that feedback be objective and deal with
specific examples of behaviors. Students must be trained in providing
feedback.
5. CONCLUDING OBSERVATIONS
Team projects have become a popular learning/teaching tool. Our

experiences with a variety of team projects have reinforced a long-held
concern, namely how to effectively and meaningfully evaluate students’
performance in team projects. An essential component of this evaluation
process must be peer assessment. In recent years, we have experimented
with a variety of individual and consensus-seeking approaches to peer
evaluation. We have concluded that, at a minimum, students must address
three challenges when conducting peer evaluations. These are focusing on
concrete behavior rather than general abstractions; considering effort as well
as ability; and arriving at a consensus. It is essential that faculty provide
guidance to the students with respect to these challenges.
Further, faculty should stress that there are two objectives when
evaluating the performance of a student in a teamwork setting. The first
objective, a short-run objective, is to provide an appropriate grade to each
student based on his/her contributions to the team project. The second
objective, a long-term and more significant objective, is to provide relevant
feedback of a developmental nature to the students because a fundamental
purpose of assigning team projects is to help students improve their

teamwork skills. Most students attach more importance at this stage in their
life to the first objective. Therefore, the evaluation percentage as a portion of
the course grade needs to be sufficiently high to make it a significant factor
in each student’s course grade. This applies to both individual and team
components of the course grade.
It is up to the faculty member to weigh the relative advantages and
disadvantages of different methods of grade allocation in choosing the
appropriate method for the project or group of students being evaluated.
Likewise, in selecting the method of peer assessment the relative advantages
and disadvantages of each approach should be considered. Hopefully, this
will result in the students being sufficiently motivated by grading
implications that they take the teamwork seriously and thus the second
objective will be achieved. Ultimately, unless the students take the
evaluative feedback seriously there is little likelihood that meaningful
development of teamwork skills will occur.
APPENDIX A: PEER ASSESSMENT FOR TEAM

PROJECTS
On this evaluation, be accurate and honest; don’t give anyone a rating — either high or low
— that he/she truly does not deserve.
Use the rating scale below to evaluate yourself and each of the other members of the project
team. If you believe that you do not have enough information to evaluate someone, indicate
that in the comment area under the person’s name and explain why.
Along with the numerical rating for each person you must provide a brief written justification
for the rating. Write this justification in the comment area under the person’s name.
Rating Scale
5 = Excellent Contributor: Self-motivated; hard worker; takes initiative; makes sacrifices to
ensure the success of the team and the project; highly effective team member; clearly exceeds
normal performance expectations.
4 = Above Average Contributor: Self-motivated; hard worker; takes initiative; usually helpful
to others; very good team member; usually exceeds normal performance expectations.
3 = Average Contributor: Sometimes takes initiative but more often than not requires some
direction; contributions are usually helpful but uninspired; works reasonably well in a team;
meets normal performance expectations most of the time.
2 = Below Average Contributor: Rarely takes initiative; nearly always waits to be told what to
do, then may not carry through on task assignments; questionable effectiveness as a team
member; falls short of normal performance expectations; may blame others for own lack of
adequate performance.
1 = Poor Contributor: Does not take initiative; more of a hindrance than a help; acts in a self-
serving way rather than as a team member; does absolute minimum to avoid being fired from
the project organization; often blames others for own lack of adequate performance.
0 = A Real Loser: Did not do enough to justify being rated as even a poor contributor to the
project.
Name of Person Completing this Form (Please Print)
__________________________________________
Note well: While anonymity may minimize the potential for retaliation against an evaluator, it
also permits the evaluator to be irresponsible. I strongly believe in encouraging responsible
action on the part of both the evaluator and the person being evaluated. Therefore, you must
identify yourself in order for your evaluations to have an impact on the individual grade
determination. Under normal circumstances I am the only person who sees the final peer
evaluations.
Student Name: _____________________________________________
Score (circle one): 0 1 2 3 4 5
Comments:
APPENDIX B: RATING SHEET FOR TEAM

PROJECTS
Team Member Being Rated ______________________________ Date ________
Use this sheet to rate the performance of all the members of your team. All members must be
rated.
If a person receives a rating below 4 on any of the first eight items, you must attach a written
statement of what the person must do to improve his/her final evaluation to 5 or above. These
must be specific, implementable activities. Copies of these evaluations will be given to the
person being rated.
This rating sheet serves as a guide to the grade for each team member’s contribution to the
project. The team should arrive at a consensus grade for each member.
Please rate the person on the following:
1 2 3 4 5 6 7
Always attended meetings Never attended meetings
Available when needed Unavailable when needed
High quality ideas Low quality ideas
Dependable Undependable
High quality work Low quality work
Facilitated goal achievement Hindered goal achievement
Did more than fair share Did less than fair share
Easy to work with Difficult to work with
Overall evaluation high Overall evaluation low
Source: Beatty, J.R., Haas, R.W., & Sciglimpaglia, D. (1996)
APPENDIX C: EVALUATION OF TEAM

MEMBER’S CONTRIBUTIONS
Each of the project teams will meet and engage in a constructive evaluation of the
contributions of each team member thus far on the course project. Focus on:
What each member does that helps in the successful completion of the project.
What each member does that hinders the successful completion of the project.
Come to a consensus on a grade for each individual, reflecting contributions to date. You may
collectively assign any grade between 0% and 100%.
Use the following scale for guidance:
91% to 100% = excellent contribution
81% to 90% = above average contribution
71% to 80% = average contribution
61% to 70% = below average contribution
60% or less = poor contribution
Turn in documentation for your project team that provides:
The grade assigned to each team member.
A statement that justifies each grade (the statement pertaining to each grade should be 100-
150 words in length).
REFERENCES
Aiken, G. (1991). Self-directed learning in introductory management. Journal of Management
Education, 15, 295-312.
Alie, R.E., Beam, H.H., & Carey, T.A. (1998). The use of teams in an undergraduate
management program. Journal of Management Education, 22, 707-719.
Beatty, J.R., Haas, R.W., & Sciglimpaglia, D. (1996). Using peer evaluations to assess
individual performances in group class projects. Journal of Marketing Education, 18 (2),
17-27.
Boyatiz, R. (1994). Stimulating self-directed learning through the managerial assessment and
development course. Journal of Management Education, 18, 304-323.
Christ, M.Y., McCuddy, M.K., & Pirie, W.L. (2000) Teams in the Classroom. Paper
presented at the 7th EDiNEB international conference, Newport Beach, CA, June.
Confessare, G., & Confessare, S. (1992). Guideposts to self-directed learning. King of
Prussia, PA: OD Design and Development.
Freiberg, K., & Freiberg, J. (1996). NUTS! Southwest Airlines’ crazy recipe for business and
personal success. Austin, TX: Bard Press, Inc.
Johnson, D., Johnson, R., & Smith, K. (1994). Cooperative learning: Increasing college
faculty instructional productivity (NTLF, ERIC-HE). Washington, DC: George
Washington University.
Knowles, M. (1975). Self-directed learning: A guide for learners and teachers. Englewood
Cliffs, NJ: Prentice-Hall.
Langrehr, F.W., McCuddy, M.K., Pirie, W.L., Schroeder, D.L., & Strasser, S.E. (1998)
Contributing to a skilled workforce: Valparaiso University’s approach. Paper presented at
the AACSB Undergraduate Program Seminar: Cycles of Change in Undergraduate
Management Education, Charlottesville, Va, February.
Malinger, M. (1998). Maintaining control in the classroom by giving up control. Journal of
Management Education, 22, 472-483.
Malinger, M., & Elden, M. (1987). Improving the quality of work life in the classroom: QWL
as self-managed learning. Organizational Behavior Teaching Review, 11, 43-56.
McCuddy, M.K. (1995). Skill development for future success: Using a self-managing project
organization to foster student learning. Proceedings of the 1995 Annual Meeting of the
National Decision Sciences Institute, 1, 314-316.
McCuddy, M.K., & Pirie, W.L (2000) Issues in program integration—Three cycles of
revision, implementation and evaluation. Paper presented at the AACSB Undergraduate
Program Seminar, Atlanta, February.
Michaelson, L. (1992). Team learning: A comprehensive approach for harnessing the power
of small groups in higher education. In D. Wuluff & J. Nyquist (Eds.), To improve the
academy (pp. 107-122). Stillwater, OK: New Forums Press.
Michaelson, L., & Price, M. (1999). Integrating the core business experience. Paper presented
at the AACSB Continuous Improvement Seminar, Minneapolis, September.
Miller, J.A. (1991). Experiencing management: A comprehensive, ‘hands-on’ model for the
introductory undergraduate management course. Journal of Management Education, 15,
151-169.
Ramsey, V., & Couch, P. (1994) Beyond self-directed learning: A partnership model of
teaching and learning. Journal of Management Education, 18, 139-161.
International Management: Early Experience in
Multicultural Virtual Team Interaction
Ken Morse
Waikato Management School, University of Waikato, Hamilton, New Zealand
1. INTRODUCTION
Proliferation of cross border communication nodes and advances in the

speed and convenience of international transportation has coincided with the
rapid growth of international trade in the past few decades. For firms
pursuing international markets, the level of international travel to meet
market satisfaction demands has increased commensurately (Hill, 2000). At
the same time, the environment of management has also changed.
“Downsizing” and “rightsizing” have resulted in flatter organizations with
fewer levels of information flow and decision-making, requiring more
interpersonal skills and an increasing emphasis on teamwork in
organizations (Donnellon, 1996). The widespread deployment of computer
technology and the development of computer skills has generated an
alternate means of achieving much of this market presence and coordination
ability through the application of IT to the team effort, substituting for an
increasing portion of the cost of transnational employee movement.
As the international environment increasingly turns to electronic
communication technologies (Hill, 2000), experience in manipulating such
technologies is in itself a critical experiential learning objective for the 21st
Century. Some of these changes are reflected in the curriculum, with
increasing group work and team building exercises as integral components
of course requirements (Jacobs, 2000). As a result, international best practice
increasingly demands the ability of executives/ employees, as well as
students and faculty, to work cooperatively over communication media and
161
162 Ken Morse
computer networks. To that end, this paper reports on the wider implications
resulting from one such exercise.
2. BACKGROUND
When enrolling in a course in international management, students bring a

set of preconceptions of that subject with them. Generally, those
preconceptions fall into three groups. Group 1 is composed of those students
pursuing a degree, certificate or diploma in an international field with the
idea of full time employment in an international position in a firm. Usually,
this preconception sees the student as an offshore manager, living in a
foreign country, traveling between their work location and the home
headquarters, and periodically advancing from one offshore position to
another. Group 2 includes those students who have a definite interest in a
specific foreign country and are likely pursuing some form of area studies
program, including extensive language training. Usually, the student’s
preconception is to see the study of international management as a platform
for employment either in a firm whose primary target market is in the
student’s geographic area of study or alternatively as a platform for
increased attractiveness to an employer in that geographic area or country.
Group 3 is composed of those students who fall into neither of the above
groups, but decide to enroll on an international management course for any
number of reasons, all of which might be summed up as “filling an elective
requirement” in their academic program.
Unfortunately for those students in Groups 1 and 2, while their
preconceptions may perhaps match the reality of international management
as it was conducted in the 1970’s and 1980’s, the era of their parents’
employment experience, the reality of international management
increasingly diverges from their preconceptions (The Economist, 1999).
With the recent technological revolution in computerization and
communication and the spread of email and the Internet, offshore
management can be done through electronic means, such as video
conferencing, email and a firm’s intranet. The attractiveness of technology
has resulted in the increasing employment of host country nationals, at the
expense of expatriates, primarily because it is a less expensive option for
firms increasingly pressured by the competitive “bottom line” (Punnett &
Ricks, 1997). It is difficult to predict future trends in this area, but it is
equally difficult to envision a return to the type of international management
supported by these preconceptions.
To correct these misconceptions, students must be shown the capabilities
of electronic communication, to include the ability to create, coordinate and
Multicultural Virtual Team Interaction 163
cooperate in a virtual environment. Already conversant with the technologies

associated with email and “surfing the net”, students must be shown some of
the other, e-business related capabilities and limitations of this exploding
technology. Coupled with this demonstration is the need for sensitization of
the difficulties of communication without the physical presence of other
participants and the difficulties of transmitting/receiving associated
communication cues (Page & Platt, 2000). In the international arena these
difficulties are exacerbated by the difficulties of cross-cultural
communication in such a non-contact environment. Finally, it is useful,
although of lesser importance, to demonstrate the sometime unreliability of
technology as a panacea for interoperability.
2.1 Concept of Simulation
Simulations are defined as “1. imitation or enactment, as of conditions

anticipated; …4. the representation of the behavior or characteristics of one
system through the use of another, esp. using a computer” (Random House,
1997, p. 1205) or alternatively as “3a. imitate the conditions of (a situation,
etc) e.g. for training; b. produce a computer model of (a process)”
(Thompson, 1995, p. 1294). These can run the gamut from world
encompassing, programmable computer models (Hughes, 1999) to total
enterprise simulations (Anderson & Lawton, 1992) to single event role
playing situations (Malik, Howard & Morse, 1997). The commonality
among them all is the intended reduction of extraneous environmental
influences that have little relevance to the issue at hand, whatever that may
be. In a similar vein, experiential learning is “… the process whereby
knowledge is created through the transformation of experience” (Morse &
Malik, 1998, p. 308). As identified previously, simulations provide a very
real supplement to experiential learning exercises (and vice versa) in
preparing students for the anomalies of the post-collegiate environment
(Malik, Howard & Morse 1997).
2.2 The Virtual Team
A virtual team is a group of discrete individuals assigned to achieve a

common goal or purpose, identical to other teams. However, a virtual team
employs electronic communication to enhance mobility and speed of
information by operating both asynchronously and without collocation
(Lipnack & Stamps, 1997).
One key to effective virtual teams is alignment of three separate, but
overlapping characteristics. The goal must be well defined and clearly
specified. The team must be functional and interactive. And, complementing
164 Ken Morse
the above, the technology must be facilitative and reliable. If these three
characteristics coincide, the potential exists to develop the “virtual space” in
which the virtual team functions as indicated in figure 1 (Hornett, 2000).
Th e
G oal
Th e
V irtu a l
Spac e
Te a m
Th e
Te c h n o lo g y
Figure 1: Virtual Team Space.
2.3 Design
Thus it has become necessary to design an experiential exercise that

would simulate the current technological environment, while at the same
time highlighting the difficulties inherent with this new wave of international
management. Such an exercise must incorporate, to the greatest extent
practicable, the virtual environment that would be expected to exist for the
foreseeable future (The Economist, 1999) and provide opportunities for
developing the capabilities today’s students would be expected to offer in the
employment market. To facilitate these design concerns, a term-length
exercise in virtual team management was created. The initial objectives of
this exercise were to:
“Show that teamwork can be accomplished electronically, while
demonstrating the technical difficulty of electronic communication,
including:
• the difficulties associated with non-verbal communication cues in a non-
contact environment, especially the nuances of “face-to-face”
communication;
• the communication difficulty imposed by cultural differences within a

team;
• the added complexity of coordinating activities “on-line”.
Show the effect of using technology in place of international travel
including:
• the ease of communication using electronic resources;
• the relaxation of the time constraints associated with “in-person”
international management.
Develop a sense of skills development through successful completion of a
virtual exercise, to include:
• positive reinforcement of existing “on-line” skills
• exposure to capabilities not previously used in student exercises (i.e.
video conferencing, synchronous editing, etc.).”
Because the three characteristics identified by Hornett (2000), the goal,
the team and the technology coincide, the potential thus exists to develop the
virtual space in which the virtual team may function.
2.4 Operation
To provide a framework for accomplishing these objectives, teams were

formed and assigned a major international feasibility study to complete.
Each study required detailed investigation to determine the potentially most
profitable offshore location for a new production (manufacturing) facility.
Upon completion, an oral presentation, videotaped in a TV studio, and a
formal written feasibility report were the expected deliverables.
As with any simulation, some artificialities are incorporated in this
design. Based on responses to a participant questionnaire, teams were
constructed whose members were chosen to nearly equally match a series of
demographic characteristics (foreign student as opposed to domestic), work
experience (yes as opposed to no), teamwork skills/experience
(much/some/none) and gender, such that each team would encompass
roughly similar levels of this range of characteristics.
The resulting virtual teams were composed of three to five members who
were only allowed to communicate via electronic means, as illustrated in
figure 2. Technical capabilities for each individual team included email
(deposit and wait), a live chat facility (interactive communication) and a
synchronous editing capability (multiple simultaneous access to an open
document). Such capabilities were deemed the minimum necessary, given
the restriction of no face to face communication. Teams were formed during
the second week of the term, with deliverable due during the eleventh week
of the term, thus allowing 10 weeks for the full simulation exercise.
166 Ken Morse
Figure 2: The Virtual Team.
Heavy reliance is placed on student honesty in limiting communication

activity to electronic means, given that those students may be co-located on a
single university campus. In addition, physical handling of document
transfer was made through an intermediary as a substitute for either facsimile
or postal/courier movement. This was done to simulate the time lag
associated with the physical movement of paper resources and was thought
to encourage greater use of electronic means of information flow (i.e. use of
scanners for graphics, etc.).
3. RESULTS
At the conclusion of the exercise, students were asked to comment on

their experience. They were given a six open-ended question instrument for
the collection of qualitative data. The qualitative responses were then
collated to obtain collective impressions of their experience. The participants
were given an additional four demographic questions. The initial simulation
exercise consisted of five teams, ranging in size from three to five and
included 20 third or fourth year students, of whom 40% were “foreign”. All
had been participants in at least one team or group project prior to this
simulation exercise. Their responses are reported below in two ways: firstly,
collective responses from the group as a whole, then secondly, alternate
impressions based on a very rough measure of culture.
3.1 Collective Feedback
Collectively, the group reported several positives aspects of the exercise.

They unanimously saw the exercise as a “real world” application, in part
based on collective background and in part based on their impressions as

developed during the course. Further, they perceived the exercise as an
opportunity to use skills they had developed in their previous coursework,
including computer skills, research and analysis skills, as well as time
management and workflow coordination skills. Additionally, they perceived
the development of new skills, primarily in the manipulation of information
to create the required deliverables and the development of “live” interaction
skills (both asynchronous and synchronous communications) as being an
outcome of this exercise. An unexpected positive outcome reported by 80%
of the participants was a perception that the exercise resulted in an
improvement of their keyboarding skills. As one participant put it: “it was
essential to improve my typing speed to keep up with the rest of the team”.
However, the collective perception of all the participants was that the
technology itself was a major hurdle. The exercise required development of
synchronous communications skills, in both the chat mode and the online
editing mode. Surprisingly, both were new to these students, which imposed
a steep learning curve on team members. Due to the nature of teamwork,
progress could only flow at the rate of the slowest participant, thereby
leaving them with the impression that this skill was both unusually difficult
and potentially detrimental to their progress.
As expected, the participants unanimously suggested that the virtual team
exercise was more difficult than their previous team experiences, a difficulty
which they attributed to the virtual nature of the project. They perceived the
lack of body language and the inability to “read” the tone of written text as
significant problems in understanding what their team members really meant
when offering a contribution. Another unexpected perception reported by
75% of the participants was that the virtual team project left them with the
impression that the team seemed to be disorganized and as a result they
found the team exercise unusually stressful as compared to their live team
experience.
3.2 Cultural Feedback
Because the key to virtual teamwork is communication, and because

much of language, the mechanism of virtual team communication, is
culturally based, the key distinguishing variable used as a surrogate of
culture was that of primary language. This was preferred over either
nationality or ethnicity because of its inherent applicability to the virtual
team concept (Morse, 2000). The demographic data indicated whether
English, the teaching language, was the participants’ primary (mother)
language or a secondary (other than mother) language. The assumption was
made that primary cultural behaviour and interpretation techniques would be
168 Ken Morse
based on the participants’ native culture rather than a learned culture.

Although there are potential difficulties in making such an assumption, given
the demographics of this participant group, the resulting differentiation was
quite clear.
Using this assumption, the participant group was composed of 60% (12)
primary English speakers (EPL), 1 American, 1 Canadian, 1 Briton and 9
New Zealanders and 40% (8) secondary English speakers (ESL), 4 Chinese
(2 PRC, 1 Taiwanese, 1 Singaporean), 2 Malaysian (1 Malay, 1 Tamil), 1
Sinhalese and 1 Chilean. The differences in perceptions of these two groups
of the same exercise follow.
3.2.1 Primary English Speakers
This group of participants suggested that, compared with other

groups/teams in which they had participated, the virtual team exercise
demonstrated a lack of apparent culture, that is each participant would
interpret their participation within their own cultural framework. There was
no need to adjust to the potential differences in cultural behaviour of the
team, as there was no direct contact.
These participants unanimously recognized the increase in speed of
communication and the ability to more rapidly develop a research plan as a
positive aspect of the exercise. They found that it was easier to simply
deposit their comments/inputs in the discussion group (asynchronous
communication) and review other inputs when they were posted. Thus, they
suggest that, because communication was more convenient, they participated
more often that they would have done in a physical team environment. To
this extent, the virtual team environment was considered to be less inhibiting
to these participants.
At the same time, however, they perceived this communication speed to
be a drawback, as they suggested it was much more difficult to interpret the
contribution on the video screen. To this group, increased communication
speed served to further confuse the nuances of English language
communication, perhaps because the context of that communication was less
determinate.
Further, they suggested that development of a team personality or feeling
of group cohesiveness was much more difficult to obtain. In fact, two of the
five groups suggested that, unlike their previous team experiences, the
virtual team never developed its own group cohesiveness, but simply
remained a set of on-screen responses to keyboard inputs.
Another unexpected impression provided by a majority (66% or 8 of 12)
of participants was the perception that it was easier to become a “free rider”
in a virtual team environment than in a physical one. Two surprising points
emerged from this impression. Firstly, the participants attributed free riding
to the inability to use the typical group control mechanisms of tone and
context in discussion to send verbal messages regarding lack of participation.
As one participant stated:
“You never knew if they got the message, as they could easily ignore
your efforts to generate a response. You could never be sure whether they
had actually received the message (a technology issue), or simply chose to
ignore the message once they had received it”.
Secondly, these participants unanimously attributed this free riding to the
ESL students in the group, thus suggesting that it was much easier for these
team members to build stereotypes of cultural behaviour, whether they were
justified or not.
Despite these difficulties, the EPL participants perceived the virtual team
environment as a positive learning experience. They suggested that the
positive results from exercise participation significantly outweighed the
negative and that their exposure to the negative factors associated with such
an experience was in itself a positive experience.
3.2.2 Secondary English Speakers
This group of participants was much less positive about their virtual team
experience. The single unanimously positive reaction from these students
was, surprisingly, a language related advantage. These participants suggested
that the ability to compose and edit their contributions off-line and then post
them to a discussion group (asynchronous communication) allowed them to
more accurately express their meaning.
However, this raised two immediate negative reactions from this group.
Firstly, this type of editing required them to take more time than they would
have chosen to spend on a similar contact exercise. Thus, for these
participants the advantage of accurate communication was offset by an
increased time commitment. Secondly, the language difficulty resulted in the
unanimous view that a chat capability (synchronous communication) and an
on-line editing system (synchronous file access) was too fast for them,
leading to consistent misunderstanding. As one participant said: “In the chat
mode, I couldn’t understand what the rest of the team was saying, because
by the time I interpreted one idea, they had moved on to three others and I
was left behind”.
This difficulty led these participants to identify another problem for
them, that of misunderstanding English. Many of the terms are technical in
nature and, with a lack of either an English language academic background
or business experience, they found it hard to comprehend the meaning of
many of the technical English terms used in their team communication.
170 Ken Morse
Because of their cultural background, they suggested that they were “too
embarrassed to continually interrupt the team communication flow to ask for
explanations and clarifications”. Thus they simply became passive
participants in the exercise, which may be a partial explanation for the
perception of increased free riding, as mentioned by the EPL participants.
Finally, for these participants both the technology and their personal
usage skills became an issue. They unanimously suggested that it was
difficult for them to learn to use the new instructions/ programs as they could
not ask their team members for assistance as they could have done in a
contact environment. Thus, they had to spend additional time learning
program navigation and operation on their own. As with the English
language, they suggested that they were too embarrassed to continuously ask
the lecturer for assistance and thus half the participants indicated that they
did not become comfortable with the technology during the entire exercise.
Added to this technology difficulty was a skills problem, as they suggested
an unfamiliarity with high speed keyboarding techniques, which put them at
another disadvantage during synchronous communication sessions and
required additional time commitment during asynchronous communication.
For those participants for whom English was a secondary, as opposed to
primary, language, this virtual team exercise was not a positive experience.
For them, the exercise served the purpose of reinforcing the distinct
differences between the two groups and their perception that they had to do
more than their EPL counterparts to achieve the same result.
4. CONCLUSIONS/IMPLICATIONS
Due to the relatively small size of the group and the one-off nature of the
simulation, this assessment is only indicative in nature. Yet the exercise
highlights some significant successes and some inherent limitations of
virtual team exercises in a multicultural environment. In terms of successes,
the exercise enhances participants’ perceptions of the realistic nature of their
education. Further, the exercise highlights for participants the potential
flexibility of the virtual environment as an operating space, wherein
participants can take advantage of flexible communications paths. Finally,
the exercise indicates the ability to bring together a diverse group from
multiple locations without the significant expense of international
transportation.
At the same time, this simulation highlights potential difficulties in using
the virtual team construct in a multicultural environment. Firstly, the
technology may be highly developed but individual usage skills differ
around the world. This is exacerbated by a reliance on a single language
(English), which is inexact in its nature and which is a significant hurdle to

non-English speaking participants. Secondly, cultural behaviour patterns
differ with regards to frankness/openness to seek assistance, which can lead
to strengthened stereotypes of participation or lack thereof. And finally,
though participants are increasingly familiar with electronic communication,
familiarity with the virtual environment is an additional difficulty, resulting
in participants spending significant amounts of time familiarizing themselves
with the personal isolation necessarily embodied in electronic
communication.
There are a number of potentially significant implications from this
indicative exercise. Virtual teamwork in a multicultural environment may be
more difficult than in a single, mono-cultural environment, which much of
the current literature addresses (Donnellon, 1999; Lipnack & Stamps, 1997).
Team building in a multicultural environment may include additional
language clarification requirements, further skills training and increased
technology familiarization efforts to be as effective as similar team
operations in a physical or contact environment.
Obviously, further research is required to verify these indicative findings.
However, they are offered here with the hope of encouraging further
interaction among business and management educators in the development,
use and assessment of multi-cultural virtual teams.
REFERENCES
Anderson, P., & Lawton, L. (1992). A survey of methods used for evaluating student
performance on business simulations. Simulation & Gaming, 23 (4), 490-498.
Donnellon, A. (1996). Team talk: The power of language in team dynamics. Boston, MA:
Harvard Business School Press.
The Economist (1999). The net imperative: Business and the internet. 351 (8125), Jun 26-Jul
2.
Hill, C.W. (2000). International Business. Boston, MA: Irwin/McGraw Hill.
Hornett, A. (2000). E-team management. Paper presented at 7th EDiNEB international
conference, Newport Beach, CA, June.
Hughes, B. (1999). The international futures (IF) modeling project. Simulation and Gaming,
30 (3), 304-326.
Jacobs, N. (2000). Issues concerning incorporation of virtual team skills into the curriculum.
Paper presented at 7th EDiNEB international conference, Newport Beach, CA, June.
Lipnack, J., & Stamps, J. (1997). Virtual teams: Reaching across space, time and
organizations with technology. New York, NY: John Wiley & Sons.
Malik, S., Howard, B., & Morse, K. (1997). Business plans, case studies and total enterprise
simulations: A natural coexistence, Developments in Business Simulation and Experiential
Learning, 24 (1), 158-163.
172 Ken Morse
Morse, K. (2000). Educating knowledge workers for corporate leadership: Assessing the
efficacy of experiential learning. Paper presented at 7th EDiNEB international conference,
Newport Beach, CA, June.
Morse, K., & Malik, S. (1998). “Who’s on first?” Exploring the concepts of problem-based
learning, experiential learning, and lifelong learning. Developments in Business Simulation
and Experiential Learning, 26 (1), 303-310.
Page, D., & Platt, R. (2000). Virtual teams: Meeting the next challenge for experiential
education, Developments in Business Simulation and Experiential Learning, 27 (1), 201-
202.
Punnett, B.J. & Ricks, D.A. (1997). International business (2nd ed.). New York, NY:
Wadsworth Publishing.
Random House, (1997). Webster’s College Dictionary. New York, NY: Random House
Publishers.
Thompson, Della, ed. (1995). The Concise Oxford Dictionary of Current English. Oxford,
UK: Oxford University Press.
PART IV
PROBLEM BASED LEARNING

PRAXIS: A Practice-Based Instructional System in
the First Year of an Industrial Engineering Program
Willem M. van Woerden & Nynke Jo Smit

Faculty of Technology & Management, University of Twente, the Netherlands
1. INTRODUCTION
If students are not studying seriously and not performing well, many
factors can be responsible for this inadequate study behavior. For example,
the contents of the program may be boring to the students and not appealing
to their interests; the students may find the lecturers dull because the
teachers rattle off their lectures in the way they are accustomed to; the
program has no interesting assignments in which students are challenged; or
the social program of the students is so demanding that there is little time left
for study. These, and many other, factors can affect the study behavior of
students. From evaluation studies it became apparent that all the factors
mentioned above had had a negative influence in the past.
As a consequence, we decided to reform the instructional system and
created a so-called “consistent learning environment” (Dam ten et al., 2000).
A learning environment is consistent if the instructional process and the
learning process are in tune with each other and reinforce each other. This
can be done, for instance, by giving students assignments immediately
following each instructional sequence and by implementing intermediate
tests as a prerequisite for further progress. To that end, we built an
instructional system in which block teaching and project work were
combined.
Block teaching enables students to concentrate their attention on a
restricted number of subjects, and encourages them to start studying these
subjects quickly because of the restricted time (study blocks are six weeks
175
176 Willem M. van Woerden & Nynke Jo Smit
long). By allowing quick resits, students are able to overturn their failures.
These elements can be seen as features of mastery learning (Block, 1971).
Project work was implemented in the instructional system to challenge
students to work together in small groups on problems related to the subjects
being studied. The various tasks are mutually dependent and need co-
ordination in order to produce a common solution to the problem. These are
elements characteristic of co-operative learning (Sharan et al., 1980).
2. DESCRIPTION OF THE PREVIOUS

INSTRUCTIONAL SYSTEM
The first-year program included a significant number of individual

subjects, all unconnected to each other. In each trimester, the students had to
master five subjects taken from the three main disciplines of the program
(i.e. mathematics, management and technology). Most of the subjects were
taught through lectures. Only in the technological domain were the lectures
followed by practicals. At the end of each trimester, the students were
examined in the five subjects. As a consequence of this arrangement,
students tended to postpone their study activities to the end of each trimester.
They then discovered that there was insufficient time to prepare for the
examinations and so they would drop one or two subjects. As a consequence,
only about 10% of students managed to pass all the examinations within the
first year, whereas the Faculty aimed at a 40% pass rate.
3. DESCRIPTION OF THE NEW INSTRUCTIONAL

SYSTEM
We will now describe the new first-year program, which we call PRAXIS
(practice-based instructional system). The PRAXIS concept was derived
from the theory of Donald Schön (1987), who characterizes the engineer as a
“reflective practitioner”. Students in management and technology will
become practitioners and will have to manage technological innovations in
organizations using scientific methods for solving industrial problems and
reflecting on the impact of their actions. We, therefore, implemented this
concept in the instructional system as a kind of simulation of their future
professional practice.
The objectives of PRAXIS are to:
Practice-Based Instructional System 177
• provide students, during their first year, with an orientation towards the
overall program of Industrial Engineering & Management and to select
the appropriate students;
• improve the study behavior of the students;
• raise their study motivation; and
• improve the success rate.
3.1 Orientation of the Program
The first year encompasses the three different disciplines that represent
the total program: management, mathematics/physics and technology.
Exercises and practical training support every course in mathematics,
physics, and technology, while each course in management is followed by
case studies or by a project.
3.2 Improvement of Study Behavior, Motivation and

Success Rate
Block teaching has been introduced in order to encourage a regular

pattern of study behavior. The academic year is divided into six blocks. Each
block consists of six weeks of teaching plus one week for examinations and
three subjects, one from each of the above mentioned disciplines. After each
pair of blocks, the students can be re-examined to improve any inadequate
marks. A project runs in parallel to a subject over two blocks. In the project,
the students learn to use the knowledge just taught by solving problems
within a small group.
This instructional system deviates from the traditional system in many
respects. Traditionally, students have been free to spend their study time as
they see fit, to attend lectures and tutorials or not. They are not punished for
not showing up. They frequently postponed their study activities, such as
reading the course material and completing exercises, until just before the
date of the examinations.
By dividing the year’s schedule into six blocks of seven weeks, the
students are forced to be active in order to pass the examinations. Because
group work has become a core study activity, a kind of social control has
been built into the program. Moreover, working on a project, on a concrete
problem derived from practice, motivates students to act more like an
engineer in a project team.
4. EVALUATION METHODS
PRAXIS has been evaluated systematically by

• measuring the study time;
• investigating the study behavior; and
• determining the achievements.
Most of the outcomes of the PRAXIS evaluation study could be
compared with data collected from the traditional instructional system.
4.1 Measuring the Study Time
Students were asked to fill in a checklist on the way they spent their time
(24 hours/day). We were interested in the amount of time they spent in every
week of every block on attending lectures, tutorials, practical training
sessions, case studies, project work and self-study.
4.2 Investigation of Study Behavior
Halfway through the year, the students were asked to complete a

questionnaire about their study habits, their study attitudes and their
motivation for studying. Questions were also posed about time spent on
social activities and doing part-time jobs. In addition we interviewed them
about their experiences in the project groups.
4.3 Achievements
The students’ achievement scores in the different subjects of the program

(i.e. success rates and credit points) have been gathered and compared with
the results of previous generations.
5. RESULTS OF THE EVALUATION STUDY
The results of the evaluation study are presented here in the following
three categories:
• time spent studying (scores on the checklist);
• study behavior (scores on the questionnaire and data from the group
interviews); and
• study achievements (subject scores and figures for the overall success
rate).
5.1 Time Spent Studying
Although the program has been based on 40 hours of study in a week,

students spend on average only between 25 and 30 hours. There is some
variation over time, with fewer hours being devoted to study at the beginning
of a block and more hours at the end, when the students are preparing for the
examinations (see figure 1).
Average number of hours that students spend studying
20
percentage of
15
students
10
5
0
5 7.5 10 12.5 15 17.5 20 22.5 25 27.5 30 32.5 35 37.5 40 42.5 45 47.5
hours per week
Figure 1: Average Number of Hours that Students Spend on Studying per

Week.
When we inquired about this discrepancy, students reported that they

needed time for part-time employment, about eight hours a week on average.
Management students are also involved in social and sports activities. This
results in them spending, on average, about six hours a day on studying,
instead of the programmed eight.
This figure has not changed much from those in the past. However, we
did observe that the block system stimulated a more regular pattern of study,
more equally divided over the weeks.
5.2 Study Behavior
The more structured PRAXIS instructional system has been implemented

to accomplish a smoother changeover from the secondary school level to the
university level. To some extent the block system pushes students to start
work in time; to complete the assignments; to read the study books; and to
prepare for the examinations. Although some students complained about the
lack of “academic freedom” with PRAXIS, most of them appreciated the
instructional system. They claim that it is well organized and that it focuses
their study efforts on only three subjects at a time. We were able to compare
the students’ responses to the question: “Was it tiring to do all subjects at the
same time?” in the old and the new system. In the PRAXIS system students
are more positive on this aspect (see figure 2).
It is tiring to keep up all subjects at the same time

percentage of students
50
40
strongly agree
30
no strong opinion
20
strongly disagree
10
0
1995 1997
Figure 2: Opinions on Studying All Subjects Simultaneously.
A second radical change in the instructional system is related to the

implementation of group work as a central study activity in the program. The
students, in groups of six, have to work on a case study or a project twice a
week. Both demand output every week (i.e. reports on the small
investigations into the given problems). The groups have to divide the tasks
between the members in order to meet the demands on time. The project
groups have to deliver group products, so they really have to discuss the
individually gathered data.
I feel support from the project group to my study
pe 40
rce
nt 30
ag
e 20
of
stu 10
de
nts 0
strongly agree strongly disagree
Figure 3: The Project Group Supports my Studying.

The project groups appear to function not only as a working group but
also as a social group. They function as a frame of reference in the study
activities and often helped in making new friends. When the program
coordinator planned for students to change groups, the students protested
strongly against this move. They were reluctant to quit a group in which they
were accustomed to working together. On the other hand, we think it is
important that our students learn to work and co-operate in different project
groups. Therefore the groups were split up after the second trimester,
although a lot of students would have liked to stay in their old group (see
figures 3 and 4).
I was pleased to be able to change project group after two

blocks
80
percentage of
60
students
40
20
0
strongly agree strongly disagree
Figure 4: I Was Pleased to be Able to Change Project Group after Two

Blocks.
5.3 Study Achievements
Study achievement has been measured by examining the number of credit

points that students achieve in the first year. The course-year contains a
potential 42 credit points, with the program consisting of 14 subjects each
worth 3 credit points.
In this study, the figures from six successive years have been collected,
starting with 1994, two years before PRAXIS was implemented. Of course
we look at the results of our enrolment group as a whole, but for evaluation
results we are especially interested in the so-called criterion group, that is
those students with a good level in mathematics and physics (at least an
average score of 7 out of 10 in the final secondary school examinations). If

this group fails, we are reasonably sure that something is wrong with our
study program. Figure 5 shows the success rate over these six years,
whereby we define success rate as the percentage of students passing all
examinations within the first year. Since 1995 the success rate has nearly
doubled.
Figure 5: Success Rate of the First-year Programme after One Year of Study
(%).
It is clear that all students did profit from PRAXIS. The so-called
criterion group continues to perform better than the overall enrolment of
first-year students. Nearly half of this criterion group qualifies to advance
after one year of study.
Further, the students obtain nowadays, on average, more study points
through PRAXIS than in the old instructional system (year 1995), but only
the criterion group meets the minimum requirement of 80% of the 42 study
points at the end of the year (see figure 6). This minimum requirement is a
consequence of the Dutch financing system. Students can have a grant for
five years for our four-year program in Industrial Engineering &
Management. Therefore, a study speed of 80% would allow the students to
finish their study within the five years of funding.
We have observed that students perform better in the management
subjects than in the mathematics and technology subjects. However, we have
also concluded that more students have mastered the mathematics subject
under the PRAXIS system. In the technology field we could see only a slight
improvement in the scores.
The most striking improvements have been found in the three subjects
where the students work with problem-based projects. These are Business
Economics, People and Technology and Production Systems.
Students also performed, on average, better in the first trimester than in

the last trimester.
Figure 6: Average Number of Credit Points after One Year of Study for the
Overall Group and the Criterion Group.
6. DISCUSSION
The results of the evaluation study are discussed under the same three
categories, namely time spent studying, study behavior and study
achievements.
6.1 Time Spent Studying
The instructional system was radically changed by dividing the year into
six blocks, each of six weeks of study followed by one week of
examinations, instead of three periods (trimesters) of fourteen weeks. This
change was made to encourage students to start studying at the beginning of
each trimester. The results show that indeed students are now actively
studying from the beginning of each block. However, in general, students do
not spend as much time as expected on the PRAXIS programmed hours. We
have the impression that our students spend a regular number of hours each
week on their studies (25-30 hours). The rest of the time, they have part-time
employment and participate in all forms of social activities. Research
indicates that this is typical of social science students in the Netherlands. An
earlier research study showed that students have a “fixed” number of study
hours a week, albeit with small differences among disciplines. When the
average study time reaches this boundary, any increase in the number of
contact hours will be compensated for by a corresponding decline in the
hours of self–study (Vos, 1992).
Although the time spent studying has remained the same as in the old
program, we believe that the study hours in PRAXIS are spent more
effectively. We concluded this because the students have performed better
since the implementation of this instructional innovation. In all the various
courses students have to work on assignments and this forces them to read
the textbooks, to use their knowledge and exercise their skills.
6.2 Study Behavior
One of the objectives of PRAXIS was to improve the study behavior of

the students. We expected that the block system would ease the changeover
from the secondary school level to the university level. Further, working on
only three subjects in the block system, instead of five subjects in the
trimester system, is less demanding on students. Figure 2 shows a slight
improvement in that direction, if one compares the findings for 1995 and
1997, which is before and after the implementation of PRAXIS.
We also expected that working as a project group, on group tasks every
week, would have a positive influence on study behavior. Naturally there are
always students who prefer to work alone, but since the students are only
working a quarter of their time in project groups, most of the time they
continue to work individually in practicals and while studying. Furthermore,
some students try to “hide” themselves in groups leaving the work to be
done by others. Demanding individual contributions in the group reports and
the group presentations can discourage this so-called “free rider” behavior
(Jacques, 1991).
From the questionnaire analysis, we have concluded that the students do
like to work with PRAXIS for the above-mentioned reasons. The 1995 and
1997 students worked under different instructional systems. Although they
cannot compare these systems themselves, we have observed that the
PRAXIS generation is more positive about their system. It is likely that this
appreciation is based on their experiences in the project groups. The project
group turns out to be more than simply a study group. It also fulfils some of
the social needs of the students. In our view, group work is an important
factor in motivating students in their study. However, not all the project
groups function well automatically. Some coaching is necessary to ensure
good performance.
6.3 Study Achievements
When we compare the students from before and after the change in the
instructional system, we see that the success rate has doubled. It has also
been observed that students nowadays gain significantly more study points
in their first year. Especially the so-called criterion group achieves the
minimum target of 80% of all the study points available in one year. In
particular, the three subjects in which students work on projects contribute to
the improvement, that is Business Economics, People and Technology and
Production Systems. In these projects, students learn to use their recently
acquired knowledge, taught in lectures and gleaned from textbooks. They
discuss in their project groups the subject matter in order to solve the stated
problems. They expand on their knowledge and learn how to integrate new
knowledge with their existing knowledge–repertoire (Schmidt & De Volder,
1984). In this respect, project work profits from the advantages of problem-
based learning (Van Woerden, 1991).
Also some of the mathematics subjects contribute to the general
improvement in the performance of the students. Previously, all the
mathematical subjects had a selective function in the first year. After the
introduction of PRAXIS, only Operations Research continues to do so. The
special difficulty with this subject is that determining operations presupposes
the ability to use certain mathematical principles just learnt. Only a few
students are able to manage this.
The courses in mathematics (Mathematics I, II and III) have undergone
some changes since the implementation of PRAXIS. All lectures are now
followed by tutorials in which students are encouraged to use their recently
gained knowledge. Previously, the lecturers showed on the blackboard how
to tackle certain problems. Now students have to do this themselves and they
receive feedback on their efforts. This student-led approach seems
responsible for the improved performance.
Finally, the criterion group continues to perform far better than the other
students. In particular, they achieve better scores in the mathematical and
technology subjects. This is quite understandable, given the chosen criterion,
namely high levels of achievement in mathematics and physics in the final
secondary school examinations. With regards to the management subjects
there is hardly a difference between the two groups of students (Van
Woerden, Ruijter & Smit, 1998).
One often hears that block teaching has the drawback that although
knowledge and skills are rapidly obtained they do not stick. Therefore, we
designed a so-called “Integration Case” to take place at the end of the first
year. For this case study, the students have to brush up all the knowledge and
skills gained during the first year’s courses since they are challenged to
apply them to a new and rather complex industrial problem.
7. CONCLUSIONS
The implementation of the new instructional system, PRAXIS, has

resulted in a general improvement in the study behavior of the students.
They seem to study more regularly with the block system and achieve more
by working on problems in their project groups. The block system forces
students to start studying textbooks and completing exercises earlier than
they used to, while the project work motivates them in their studies and
helps them to process the recently gained knowledge.
It can appear as if the new working environment is more structured and
more controlled than before and that this is contradictory to the ideals of
university education about creating the ability to learn independently. In our
opinion, the educational concept of PRAXIS reflects the professional
practice of the industrial manager by combining block teaching with co-
operative learning. An industrial manager has to work under the pressure of
time, in a team, producing creative solutions to complex problems.
Students spend no more time studying with the PRAXIS approach than
with the old instructional system. They spend a given number of hours
studying. The rest of their time is spent working or on social activities.
However, students do seem to study more regularly and therefore more
effectively.
The success rate has improved. The number of credit points gained has
grown slowly over time. The criterion group students continue to perform
better than the other students, especially in mathematics and in technology
subjects. Even with PRAXIS, only the criterion group students meet the
norm of 80% of the maximum 42 credit points in the first year. Although we
did gain important results, we realize that we cannot sit back contentedly.
There is a lot more to do!
REFERENCES
Block, J.H. (1971). Mastery Learning: theory and practice. New York : Holt, Rinehart and
Winston.
Dam, G.T.M ten, Hout, J.F.M.J. van, Terlouw, C., & Willems, J.M. (2000). Onderwijskunde
hoger onderwijs. Handboek voor docenten. Assen: Van Gorcum.
Jacques, D. (1991). Learning in groups. London: Kogan Page.
Schmidt, H.G., & De Volder, M.L. (Eds.) (1984). Tutorials in problem-based learning.
Assen: Van Gorcum.
Schön, D.A. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass Inc.
Sharan, S., Hare P., Webb, C.D., & Hertz –Lazarowitz, R. (1980). Co-operation in education.
Provo, Utah: Brigham Young University Press.
Vos, P. (1992). Het ritme van het rooster. Onderzoek van Onderwijs, 21 (4), 51-53.
Woerden, W.M. van (1991). Het projectonderwijs onderzocht (Research on the project
method of teaching (academic thesis). Enschede: University of Twente.
Woerden, W.M. van, Ruijter, C.T.A., & Smit, N.J. (1998). Effecten van het blok- en
projectonderwijssysteem “PRAXIS”. Paper presented at ORD’98 – Symposium Hoger
Onderwijs. Enschede: Educational Centre/ University of Twente.
Using the Web for Problem-Based Learning
Nynke Jo Smit, Maarten van Riemsdijk & Jan van der Veen
Faculty Technology & Management, University of Twente, the Netherlands
1. INTRODUCTION
In 1997 we designed an organization theory and design course, People,

Technology, and Organization-II. We did evaluate this course intensively.
From the evaluations we learned that we had to introduce web support and in
1998 we came to the conclusion that we had to modify this web support
system. We think that by now we do have an efficient way to stimulate our
students to read theoretical texts that could be of interest for others.
In this paper we will first give a description of the course, then a short
description of the evaluation results and the accessory measures taken. Then
we come to the core of our paper, the profit of the web design. As we are
practitioners, our focus is on quality enhancement of our courses rather than
on research on education. Nevertheless we think that our evaluations are a
good example of applied research and that others could profit from our
findings.
2. THE MANAGEMENT SCIENCE COURSE
The organization theory and design course, People, Technology, and

Organization-II (MTO-II) is taken in the first trimester of the second year of
a five year program in Industrial Engineering and Management at the
University of Twente. MTO-II has a workload of 200 hours for each student
(7.5 ECTS credit points). The course focuses on organization theory and its
189
190 Nynke Jo Smit, Maarten van Riemsdijk & Jan van der Veen
relevance for designing business organizations. It is taught each year to

between 150 and 200 undergraduates.
The course replaced two former courses. The primary goals of the new
course was that students:
• should gain knowledge and insight in organization theory and in the
basic disciplines like psychology and sociology; and
• they must be able to use this knowledge and insight to solve an
engineering design problem.
In Appendix A you will find the description of the course as set out in the
study guide.
3. PROBLEM-BASED LEARNING
Right from the start in 1997 a problem-based learning (PBL) approach

has been adopted with the intention of activating and motivating the students
through participation in realistic case studies. The prime objective of PBL is
that students acquire new “actionable” knowledge. Groups of students work
on case study assignments, which link to realistic problems in that domain.
The case studies are pre-structured to some extent in order to ensure that
selected domain knowledge will be used. The tasks to be performed by the
group are more or less prescribed; the instructors often know a solution to
the problem and the learners working as a group are meant to come to this
solution (Delhoofen, 1996). To structure the work of the groups, seven
consecutive steps can be identified (e.g. Smit et al., 1994). These steps are
reproduced in figure 1.
1 Clarification of unclear aspects of the problem;

2 Definition of the problem;
3 Problem analysis;
4 Inventory of sub-problem and solution alternatives;
5 Formulation of learning goals and self-study tasks;
6 Working on self-study assignments;
7 Reporting and conclusions about the initial problem.
Figure 1: The Seven Steps of PBL.
In PBL students are also engaged in a number of tasks related to the

planning of their work; the operationalization of these plans; and the
monitoring or evaluation of what actually takes place. Planning refers to
activities whereby students or instructors define their objectives and prepare

for tasks in advance of doing them by specifying the time of execution of
tasks, their temporal ordering, their pre- and post-conditions and the
resources needed for proper execution. Operationalisation refers to activities
whereby the tasks defined in the planning are carried out by individuals or
sub-groups of students or instructors and the way these separate tasks are
interrelated and coordinated. Monitoring refers to activities in which an
instructor or student keeps track of progress in terms of the timely and
successful completion of tasks and of the quality of the group process. In
fact, a lot of different educational methods are used (see figure 2).
Educational Methods used in People, Technology & Organisation II

• A leading thread, running through it, consisting of
• A textbook (Daft) with basic knowledge and as a frame of reference for additional articles,
• Three project tasks, each referring to different aspects of the same case study assignment ,
• Final project, encompassing all former project tasks.
• Case study assignments and project tasks
• Project teams
• Theory tests
• Additional articles (to be divided amongst the project group)
• Tutorials
• Competition
• Team teaching.
Figure 2: Educational Methods Used on the Course.
The students work in project groups each consisting of six or seven

students. Over a period of ten weeks the groups study theory and work on
exercises that involve pre-defined case studies. The theoretical component
consists of a textbook, currently Daft (2000), and three sets of eight
theoretical articles. The case studies address a number of organizational
issues in car manufacturing. Parallel to the group work there are a number of
class sessions of around 40 students. In these sessions the student groups
present their findings, after which discussion takes place moderated by an
instructor.
The instructor also decides in the class sessions which student has to
present a specific article. Because the students do not know beforehand if
they have to present a theoretical article to the class, all students must be
able to give a presentation on every article, not only on the articles they had
read themselves (see below). Therefore it is necessary that the students read
the summaries of all articles. The final mark for the course is a combination
of the individual mark for the textbook-based examination and the group
mark for the report on the case study.
4. EVALUATION OF THE 1997 COURSE
We evaluated this course intensively. We asked all students to fill in a

questionnaire on the course; we made observations in the lectures and in the
project groups; and we organized panel discussions with students and
teachers. From this evaluation we demonstrated flaws in the theory part of
the group work (Smit & Van Riemsdijk, 1998). It was apparent from the
evaluation that acquisition and application of the theoretical principles of the
course by student groups was below expectation. In the case-study exercise
reports written by the groups we saw very sparse use of relevant theory. A
lot of students were looking for efficient methods of gaining credit points.
Reading of theoretical articles in pairs, though prescribed, did not take place
and therefore in many groups there was insufficient transfer of what students
had read individually to other group members. This meant that the intended
discussion in the project groups did not take place. The teachers thought the
discussion in the project groups about the articles and possible design
solutions to be essential. Besides, we noticed that working in a team is not an
ability that every student had mastered. We have to stress, however, that the
pass rate was higher than for the former courses. Moreover students did
study more intensively and did gain far more knowledge than before. The
expectations, however, were rather high, and the teachers were striving for
more. So two major changes were planned. First, teaching assistants were
trained in managing group processes. Second, the Internet was introduced as
a platform for discussion. In this paper we will focus on web-based support.
5. WEB SUPPORT
Although we used a lot of educational methods to activate the students,

these proved to be insufficient. We wanted the students to engage more fully
with the theoretical materials. In our opinion the most dominant problems
that hindered instructors and students in performing these tasks in an
efficient way were:
• groups did not have a clear picture of what is expected of them;
• groups had problems with organizing work between meetings;
• groups had problems with access to deliverables and comments;
• group members did not take a fair share of the work; and
• instructors lacked an overview of the progress of groups.
The decision was made to overcome these problems by building a tailor-
made web application. In 1997 building a tailor-made web application was
quite reasonable. Nowadays we should rather look for a ready-to-use system
on the market.
Table 1: Possible Instructional Remedies and Telematic Support Needs in the MTO-II Course
Instructional remedies Telematic support needs
The instructor can predefine some Instructors need a tool that can build and
with access to deliverables with organizing work clear picture of what is
Groups have problems Groups do not have a
structure for the group work by publish the task structure. The
specifying the main steps. representation of tasks shows what is
expected of them
Instructors can provide model finished and what is still to be done.

answers.
Groups should start with an Students need access to the task description
orientation or clarification of what and task structure.
their task is.
Groups are urged to decide about The group needs a group memory where
tasks and responsibilities, in decisions and tasks can be found.
between meetings
particular between meetings.

For some tasks, some group Students need a tool to share proposals,
members may decide to correspond, drafts and other documents, including
communicate or meet in between control of versions.
two group meetings. Students at a distance need synchronous
tools in order to co-ordinate joint tasks.
A group member can be assigned the The manager of the group needs a tool to
(manager) role, which includes schedule tasks and monitor progress.
Instructors lack overview of the Group members do not take Groups have problems
monitoring of group member’s

efforts.
Instructors should plan how students The instructor needs a tool that can make
and comments
will have access to which their products directly available to the other
information. group members.
Students and instructors need some
notification that new material is available.
The group should make decisions Students need a tool that can make their
how and when new information will products directly available to the other
a fair share of the work
be available. A group member can be group members.

assigned the task to co ordinate the Students and instructors need some
archiving. notification that new material is available.
Groups are supposed to address the Groups and instructors need a history of
individual group members’ who contributed what and when.
responsibilities.
If this is not sufficient, the instructor
can address the “free rider”.
Assessment strategies can modify Groups need scoring templates as a support
the group and individual marks so as to implement their self-assessment.
to motivate equal commitment.
Instructors and groups meet Instructors need up-to-date status overviews
regularly and discuss progress. that identify which groups are late with
progress of groups
If a number of deadlines are their work.

scheduled, this can help monitor the
progress.
Special attention can be given to
those groups that are behind
schedule.
We considered using some form of web support as a possible means of

helping to stimulate the reading of theoretical articles through monitoring of
deliverables and to improve the reflection on theory by offering students the
option of reading each others’ work. The advantage of web-based solutions
would be flexible access via the computers in the project rooms, in the
Faculty computer rooms and in the student houses. A requirement was that
any web solution suggested, to fit the planning, operationalization and
monitoring needs of the course, should be as efficient as possible for the
instructors, contributing no additions to the high quantity of paper-based
deliverables in the course and preferably no extra instructor workload.
In table 1 we give an overview of the possible instructional remedies and
the telematic support needs for these problems (Veen van der, 2001).
6. THE 1998 COURSE
A web site was set up in 1998 with the aim of building up a “theory
repository” and of stimulating the students to read theoretical articles and
reflect on the theoretical issues. The division of reading tasks was left to the
groups. For each of the 24 articles on organizational theory, every group had
to submit a reflective contribution that targeted the core of the article. This
contribution took the form of two questions about the article along with a
model answer. The contributions of a group formed a group resource. Only
if the questions were answered in a satisfactory way would the group gain
entrance to a site where they could read the approved answers of the other
groups.
The students were also asked to rank the submissions of other groups,
indicating per article which five contributions they judged to be of highest
quality. This top-5 exercise was intended to further enhance reflection on
theory. The instructor was thus able to monitor the students’ discipline of
reading, assuming that a submitted contribution was indeed related to a
thoughtful reading of an article. The planned relation between the learning
goals and the different work forms is given in table 2.
Table 2: Relating Course Components and Learning Goals.

Course component Theory & Class sessions Group work &
Learning goal Web-support discussion
Knowledge acquisition primary goal
Improving insight secondary goal primary goal secondary goal
Application of theory in problem solving secondary goal primary goal
In addition to the web-support for theory-related purposes, a shared-

workspace facility was made available. This was expected to be useful for
the groups when working between course meetings. The shared workspace
allowed file uploading via a browser so that all members of a group could
access and use the same set of files.
7. EVALUATION OF THE 1998 COURSE
In 1998 we used the same questionnaire for evaluation as we did in 1997,

with additional questions for the web support. There were student panels and
in addition we used the log files for evaluation.
The evaluation of the 1998-course showed that the discipline of reading
articles had been enhanced, but the students felt that the formulation of
questions and model answers did not help them very much in their group
work. The students reported that, after finishing this assignment, they still
had to write summaries for their group. The students thought the top-5
exercise to be a waste of time. The reasons were that, on the one hand,
retrieving other groups’ contributions from the web-site was too difficult and
slow and, on the other hand, ranking the other contributions required close
reading while the material often proved to be hardly relevant. Moreover, the
satisfactory contributions were not screened any further for quality by the
instructors. So, the ranking exercise of other students’ contributions was too
time-consuming and lacked relevance for most groups.
We did see a positive change in the sense that the students were more
aware of what was expected of them via both the course manual and the web
site. The groups divided the work between meetings according to their own
plans. They split up the reading tasks to a large extent. The problem of group
members not taking a fair share of the work persisted.
The facility for instructors and teaching assistants to monitor progress
was not used much. The student’s awareness of being monitored was
sufficient.
Despite provision being made for a web-based group archive, only 5 out
of the 28 groups took advantage of this. A part of the explanation is that the
archive facility was not introduced at the beginning of the course, but only in
the third week. By then, most of the groups had already arranged their own
way of handling information by e-mail or diskette.
As a result of the evaluation of the 1998 course two options were
discussed with the instructors. These were either to drop the telematic
support or to change the set-up in such a way that not only the discipline of
reading was supported, but also benefits for the groups in the
operationalization of their work would be offered. The instructors and the
Faculty management decided an improved set-up would be implemented and
evaluated to see if it was possible to realize improved telematic support and

to design tasks that better supported the groups’ way of working.
8. THE 1999 COURSE
Revised set-ups for the web site and for the instructional tasks were
introduced in 1999. Instead of questions and model-answers, groups were
now asked to contribute one short summary per article, of ten lines at the
most. The summaries formed a collection that the groups could use in their
sharing of expertise. Ranking was now included in the grading by teaching
assistants, with group contribution being assessed as “not ok”, “ok” or
“excellent”. Instead of an obligatory top-5 exercise, the students were now
offered the option of reading a small number of excellent summaries written
by other groups, again only after a serious contribution on that same article
by the group themselves. The goal of this re-designed cross-group exchange
of expertise was to stimulate more efficient reflection on theory. This new
option proved to work much better. Students were eager to gain access to the
“hidden” site on which the excellent contributions were placed. This even
resulted in peer pressure on those group members that had not contributed
sufficiently good summaries, thus excluding the group from the hidden site.
Such pressure forced them to improve their performance. The telematic
support was changed for the 1999 course in three ways:
• login and personalized screens;
• improved ergonomics; and
• integrated and simplified web-based group archives.
Because the operating system, Windows NT, did not provide a
hierarchical group mechanism, a dedicated user administration add-on was
set up (Van der Veen et al., 2000a,b). This solution permitted the formation
of groups at the course and project-group levels via a web browser, as well
as the assignment of roles and accompanying privileges. When students
logged into the course site a personalized screen was presented, offering
only those options that were currently accessible to that user plus some
personalized status information (see figure 3). In contrast, in 1998 students
had been presented with a screen showing all articles, including those to
which they had not yet earned access rights. This had led to frustration for
students when attempting to access non-accessible articles.
Figure 3: Example of a Personalized Screen from the 1999 Course Offering

the Student Access to Six Out of Eight Listings of Excellent Summaries by
Other Groups. Access Depends on Group Performance.
The use of the web-site was simplified in a number of respects. The

excellent contributions for example were now shown immediately in the
main frame of the screen listing of all available contributions, whereas
previously each excellent contribution had to be requested one at a time.
Furthermore, students were no longer required to type in their names when
submitting a summary, as their details were known to the web-based
repository via the login. Also readability of the group contributions was
improved. In addition to the web-based theory repository, each group was
automatically assigned a folder on the server for file archiving.
The 1999 evaluation resembled that of 1998; once again all students were
asked to fill in the questionnaire, we organized student panels and we used
the log files. The evaluation results are summarized in table 3.
Table 3: Summary of the 1998 and 1999 Findings.

Problems that hindered Summary of findings 1998 Summary of findings 1999
instructors and students in
performing their tasks in an
efficient way in MTO-II
Groups do not have a clear Via the course information Via the course information
picture of what is expected of and the web site students and the web site students
them were well aware of what was were well aware of what was
expected. expected.
Groups have problems with Groups were now free to Groups split up tasks to a
organising work between organize work between large extent. Web-based
meetings meetings themselves. They group archives and e-mail
split up reading tasks to a support joint tasks between
large extent. meetings.
The repository helps in
preparing for group meetings,
and as a summary archive
that can be accessed when
needed.
Groups have problems with Web-based group archives All groups use their web-
access to deliverables and were introduced (week 3), but based group archives, e-mail
comments only used by a few groups or both.
Group members do not take a Problem is persistent. Group Most students were satisfied
fair share of the work accounts did not allow with share taking. Web site
monitoring at the individual supported monitoring by the
level via the repository. group themselves.
Instructors lack overview of Progress overview was Progress overview was
the progress of groups enabled but hardly used. enabled but hardly used.
Students do not start using Top-5 exercise was ignored. Both the web-based theory
the telematic support Little use was made of the repository and the web-based
web-based group archives. group archives were used
during this course.
9. RESULTS: WHAT ARE THE PROFITS OF THE

WEB DESIGN
As we did evaluate the MTO-II course very intensively in 1997, 1998

and 1999, we are able to tell what the effect of the web support in this course
has been. We will focus on four aspects:
• the theory repository did help improve the discipline of reading theory;
• students are reading the work of other groups;
• students perceive the reading of other groups’ work as helpful; and
• students do use the groups’ archives.
9.1 The Theory Repository Did Help Improve the

Discipline of Reading Theory
In 1999 groups having received a “not ok” assessment did submit revised
summaries more often than in 1998. The percentage of “excellent lists” for
which groups received access is higher in 1999 compared to 1998 (76% and
65%, respectively). This is due to the additional submissions in 1999.
On the student questionnaire students indicated that most needed more
than 30 minutes when reading an article. In the interview some students
stated that they could now easily monitor whether each student had
performed his or her reading task. The interdependence of group members
on each other’s performance is clear. We conclude that some discipline of
reading has been enforced and can now be monitored. Both the 1998 and
1999 version of the web-application helped to stimulate the students to work
more actively with the theoretical materials offered.
The 1999 evaluation shows that the revised set-up of the web-based
theory repository was perceived as more helpful by students than the 1998
version. The principal reasons for this are summarized below.
• In 1998 the contributions consisted of sets of questions and answers.
Summaries may be more helpful when students want to learn about an
article without reading it.
• The pre-selection of the excellent contributions in 1999 made this option
more efficient for the students.
• The difference of control by students, choosing themselves to read work
of others or not, compared to being forced to read others work and then
rank it as in 1998.
• The improved user interface making the use of the web site more
efficient.
9.2 Students are Reading the Work of Other Groups
In 1999 substantial reading of the work of other groups took place. We

used the log mechanism in the evaluation of the use of all collected
contributions by other students. In 1998 the event-log mechanism showed a
total of 309 reading events, an average of 11 events per group. The number
of events decreased with time when the (unpopular) top-5 exercise was
cancelled. In 1999 this number rose to 1465 reading events, an average of 64
events per group. The actual number of reading events is higher as the
logging mechanism was only available from the fourth week of the course.
Whereas in 1998 we only saw one event per group per article, in 1999 an
average of four individual group members used the option to read summaries
rated as excellent.
The median reading time was 37 seconds. This may seem short, but it
turns out to be sufficient for a fast reading of a 10-line summary in which the
student is interested.
In the evaluation interview of the 1999 course students indicated that
they did read the excellent contributions to get information about the content
of the different articles; to see how other groups were doing; and to get an
indication of what an excellent summary should look like. Students preferred
to read just the excellent summaries instead of all the summaries, because
this would take too much time.
Apart from reading the articles themselves, a majority of students in 1999
used the web site to read summaries written by students from both their own
and other groups. Compared with the situation in 1997, the discipline of
reading can now be monitored by the instructors to a larger extent. A paper-
based solution that accomplishes the same result is difficult to imagine. The
students are willing to work with the web site tasks if they feel they benefit
from this in terms of their assessment in the course.
9.3 Students Perceive the Reading of Other Groups’

Work as Helpful
The 1998 evaluation results (Van der Veen et al., 1999) showed that the
top-5 exercise was felt to be “a waste of time” as reading sometimes more
than 20 contributions on the same article was very time consuming but gave
the students little added value.
By comparison with 1998, the 1999 web site shows significantly higher
appreciation scores for its contribution to knowledge acquisition and insight
improvement. The small-group sessions are clearly highly appreciated. Also
the group discussions are perceived as important for learning purposes. In
the 1999 course the students indicated that the fact that not everybody reads
every article creates an interdependency that has positive effects on
discussion and collaboration. The production of summaries for each other
was thought to be a highly relevant task, as these summaries are a good
introduction for those group members who have not read the article. Access
to other groups’ work also allows students to compare their work with that of
others.
9.4 Students do Use the Groups’ Archives
When looking at the actual use of the group archives, we saw that in
1998 only five out of 28 groups used the group archives, partly because the
group archives were not introduced directly at the start of the course. Some
students indicated that they did not know what the group archive was. Other
students who knew about the group archive but did not use it reported that
they did not want to invest time in learning to use software from which they
expected only limited benefits. Sharing files was mainly organized by using
diskettes and e-mail messages with attachments. Via the student
questionnaire two out of three students reported using e-mail messages for
the 1998 course.
In 1999 the group archives were available from the start of the course.
The interface was simplified using just one directory, which listed all files in
alphabetical order. The group archives were used by all but one group,
storing an average number of 20 files, compared with an average of only two
files the year before. However analysis shows that during the second half of
the course the number of groups still using the group archives dropped from
22 to 14, indicating that some groups abandoned using the archive part of the
web site altogether. The groups that did use the archive asked for some
improvements, such as the possibility of making subdirectories. These
groups had amassed a considerable number of files and some reported losing
an overview because only one single directory was available. Two out of
three students, in the context of the course again, used e-mail. However the
rate of e-mail usage differed between the students using the group archives
and those who did not. Only 54% of students using the archive used e-mail,
while 83% of students reporting that they had not used the group archives
said they used e-mail.
All student groups made use of telematic tools to share information,
showing a preference for using e-mail and/or the group archive. Possible
reasons for groups to prefer e-mail could be that it allows the combination of
exchanging files and communication at the same time. Also, for the time
being, e-mail is a more familiar application to the user than group archives.
10. DOES THE WEB SITE IMPROVE STUDENT

RESULTS?
The effect of use of the web site on the student results is difficult to
measure. Instructors reported that it is very difficult to attribute outcomes to
specific learning events. The course is a complex integration of different
elements. The instructors considered the overall result as a strong
combination of aspects. An analysis was carried out to check for correlation
between the theory examination and case-study grades and web-site related
activities. It was found that:
• groups using the web site more frequently for reading the work of other
groups did not score better in the grading of the case-study report; and
• a correlation was found between the students’ web-related activities and

the theory examination results, but not between web-related activities
and the final case-study results.
Transfer from the theory parts of the course to the case-study problem
solving tasks is thus not as straightforward as anticipated.
The combination of changes in instructional design with an improved
user interface makes it difficult to factor out the dominant cause of the
improvement. However, based on discussions with students, we believe that
the students’ appreciation is caused mainly by the way web support can help
them organize their tasks in an efficient way. Although the web support thus
activates the learning behavior of the students, students appreciate most
highly those learning settings in which they can interact with their peers and
the instructors. However, we believe that these discussions are more fruitful
when the students have been better introduced to the relevant theory.
Figure 4: Project Group at Work in the MTO-II Course.
11. CONCLUSION
In 1997 we designed a new course in organisation theory and design. We

wanted students to work in project groups and divide the reading of articles
among the group members and make summaries for the other group
members. In this way the project groups could read a considerable amount of
relevant literature and the group members needed to co-operate to share the
information.
When we observed that a lot of students were looking for more efficient
ways of gaining their credit points than for co-operating, as we wanted them
to do, we introduced a form of web support. From our elaborate evaluations
we learned that the web support really helped students to share knowledge
and organize their tasks in an efficient way. Future development will focus
on the opportunities to integrate student options for sharing of information,
as well as instructor options for assessment and monitoring, into the generic
web-learning environment.
APPENDIX A: DESCRIPTION OF “PEOPLE,

TECHNOLOGY & ORGANISATION II” IN THE
STUDY GUIDE
This course pays attention to the following subjects: views on organisations and
organisation theory; organisation structure, culture, strategy and goals; contingency factors
(environment, technology, size, age, power) and their influence on the design and change of
organisations; decision-making processes; organisational innovation and change. The course
starts and finishes with a plenary session. In between (individual) reading tests are used to
assess the theoretical knowledge of participants. Using various case study assignments
centred on one common theme, the participants' capabilities to apply that knowledge is
developed and tested. For this the participants collaborate in small project teams of 5-6
people. During a range of workshop-type sessions, these groups are requested to both verbally
and in paper present and discuss their findings from the case analysis and (re-) design.
Course structure:
2 lectures of 2 hours each, case study, project work, self study.
Literature:
Daft, “Organisation Theory and Design”, West College Publishing, 7th edition, Cincinnati,
2000; a reader comprising case study material and a number of additional papers/chapters;
and possibly also a number of handouts.
Examination:
The course is graded on the basis of the results from the reading tests (33%) and the report on
the final assignment (67%).
Language:
Dutch.
REFERENCES
Daft, R.L. (2000). Organisation theory and design. Cincinnati: South Western College
Publishing.
Delhoofen, P. (1996). De student centraal: Handboek zelfgestuurd onderwijs. Groningen:

Wolters-Noordhoff.
Smit, N.J., Hulshof, M.J.F. & Willems, J.M.H.M. (red) (1994). Beïnvloeden van
studeergedrag. Utrecht: VSNU.
Smit, N.J., & Van Riemsdijk, M.J. (1998). Evaluation of the 1996/1997 edition of the MTO2
course. Paper presented at the Educational Research Days ORD’98, Enschede. (in Dutch).
Veen, J.T. van der, Van Riemsdijk, M., Slabbekoorn, H., & Kamp, I.N. van de. (1999).
Stimulating reflection on theory using a Web-based repository of student reviews. In B.A.
Collis & R. Oliver (Eds.), Proceedings of the Edmedia (pp. 172-177). Charlottesville, VA:
AACE.
Veen, J.T. van der, Van Riemsdijk, M., Jones, V., Collis, B. (2000a). Theory repositories via
the Web for problem-based learning. Interactive Learning Environments, 8 (3), 257-277.
Veen, J.T. van der, Van Riemsdijk, M., Laagland, E.F., Gommer, E.M., & Jones, V. (2000b).
Web support for activating use of theory in group based learning. Paper submitted to
WebNet2000, San Antonio.
Veen, J.T. van der (2001). Telematic support for group-based learning (diss.). Enschede:
Twente University Press..
Effects of Problem-Based Learning in Business
Education: A Comparison Between a PBL and a
Conventional Educational Approach
Piet van den Bossche1, Mien Segers1, David Gijbels2 & Filip Dochy2
1
Department of Educational Research and Development, Faculty of Economics and Business
Administration, University of Maastricht, the Netherlands; 2Educational Innovation and
Information Technology, Faculty of Law, University of Maastricht, the Netherlands
1. INTRODUCTION
Over the past decade, business schools have been increasingly criticized
for being too theoretical, too specialized, not internationally oriented and not
engaged in developing the necessary expert knowledge in students
(Gijselaers & Woltjer, 1998). In the workplace of the current “Age of Mind”
(Todd, 1999), knowledge is a valuable asset of the contemporary
organization. In order to be successful in the current dynamic and
competitive environment, the use of existing knowledge and the
development of new knowledge becomes a prominent prerequisite for
solving the complex problems faced today. In view of these developments,
the question that needs to be asked is: “What qualities must graduates of
business education possess?” Many answers to this question have been
formulated in the literature. These share a common view. Society demands
graduates who are capable of efficiently resolving complex problems (Engel,
1997; Poikela & Poikela, 1997; Segers, 1997).
Concomitant with recent developments in society, insights have been
formulated in cognitive psychology and instructional theories, such as
constructivism, which have consequences for educational practice. Research
on expertise has made clear that an enormous amount of knowledge needs to
be mastered to serve as a basis for expert performance (Gijselaers & Woltjer,
205
206 Piet van den Bossche, Mien Segers, David Gijbels & Filip Dochy
1998). At the same time, research has made clear that expertise development
is not simply a matter of adding new information to existing knowledge. For
expert knowledge is organized in such a way that it is readily applicable in
practical settings (Van de Wiel, 1997). Therefore in education the focus
should be not only on acquiring knowledge, but also on students being
trained in using knowledge efficiently (Arts, Gijselaers & Boshuizen, 2000).
What can business education learn from this research on expertise
development (Gijselaers & Woltjer, 1998)? During knowledge acquisition,
students should be stimulated to analyze business cases and relate business
theory to these cases. Thereby, students should be stimulated to conduct their
own information searches and ask their own questions. Small group work
enables this process. If teaching for expertise is to be considered as shifting
away from learning factual knowledge and principles to learning how to use
this knowledge in complex problem-solving situations, curricula should
consist of a considerable amount of problems that are not disciplinary pre-
packaged.
2. PBL IN BUSINESS EDUCATION
As a result of these developments in society and educational science, a

number of concrete innovations have been introduced in business education,
which may be labelled as “student-centred education”. Changes are mainly
happening in the following three directions: from knowledge as a goal to
knowledge as a tool; and from passive to active and independent learners;
and from directive to coaching teachers. Students’ activities and learning
processes are always at the centre (Vermetten, 1999). An example of such a
student-centered approach in higher education is problem-based learning
(PBL). Although originally developed for medical training in Canada, the
orthodox version of PBL is applied globally in many disciplines (Gijselaers,
1995), including business education. The desire to implement PBL as an
alternative for existing teaching practices inevitably raises the question of
whether PBL is an alternative capable of effectively replacing conventional
curricula (Albanese & Mitchell, 1993) in educating business experts.
Although contemporary educational practice increasingly consists of a blend
of conventional and student-centered, it seems a good idea to compare the
merits of the two approaches.
Van den Bossche, Gijbels and Dochy (2000) sought to answer this
question by conducting a meta-analysis. The results of this survey suggest, in
all instances, a strong and significant effect of PBL on the application of
student knowledge. As to student knowledge, after a single course based on
PBL virtually no difference was observed by comparison with students
Effects of Problem-Based Learning in Business Education 207
learning by conventional methods. Learning by “conventional methods”

must be taken to mean those forms of education, which are strongly teacher-
centered, in which transfer of knowledge by the teacher is the predominant
method. In most instances, teaching takes place in large groups. As to the
effect of a fully problem-based curriculum on student knowledge, the
outcome showed a slight tendency towards negative results.
However, only one of the 43 studies included in the statistical meta-
analysis was situated in the domain of business education (Son & Van
Sickle, 2000). This study found small positive effects on the knowledge
acquisition of the implementation of PBL in two business courses.
Given the fact that PBL becomes more and more implemented in
business-curricula (Gijselaers, 1995), further research should be undertaken
into the possibility of generalization as a result of the effects found for
domains other than medicine. In the study discussed below, the effects on
business students’ knowledge and knowledge application of a curriculum-
wide implementation of PBL were investigated. The following core sub-
questions needed to be answered.
• “To what degree do students studying in a problem-based curriculum
possess an accessible knowledge base of the subject studied in
comparison with students to whom the subject was presented in the
conventional way?”
• “Are students who have been subjected to PBL better equipped to apply
the knowledge of the subject studied than students to whom the subject
was presented in the conventional way?”
2.1 Hypotheses
The research questions deal with knowledge acquisition and application

of knowledge issues. For these categories, both developmental aspects and
differences between educational formats are discussed.
2.1.1 Knowledge Acquisition
During the first two years of their higher education, students are exposed
to a lot of new information. The acquisition of a scientific knowledge base of
the respective domain is a main goal. In conventional education, the first two
years of training are mainly devoted to the acquisition of knowledge. In a
problem-based curriculum knowledge application is an important goal from
the start. Here, knowledge is seen as a tool (Dochy et al., 2002). The results
of the meta-analysis by Van den Bossche and colleagues (2000) indicated
that in the starting years the knowledge base of students in a conventional
medical curriculum is not as large as the knowledge base of students

following a problem-based curriculum.
However, it can be assumed that the characteristics of PBL as a powerful
learning environment bring about a “better structured” knowledge base. This
makes the acquired knowledge better available in the future (Gagné, 1993).
Results from the meta-analysis seem to confirm this (Van den Bossche,
Gijbels & Dochy, 2000). This is a possible explanation for the fact that PBL
students in the later years of their training score at least as good as the
students out of a conventional curriculum on a knowledge reproduction test.
The expectation is that business students following a problem-based
curriculum will do slightly worse than their counterparts in a conventional
curriculum on a knowledge reproduction test. However, the difference
between these two groups will become smaller and possibly even culminate
in a higher score for the PBL students at the end of their program.
2.1.2 Knowledge Application
It can be assumed that all students become more adept at the application
of acquired knowledge through their training. In PBL authentic problems are
the starting point for acquiring and applying knowledge. In the conventional
curriculum one has no explicit requirement for the application of knowledge
until after the second year. This means that the acquisition of basic
knowledge precedes learning how to apply it.
In line with the above, research indicates that a curriculum-wide
implementation of PBL in the medical domain has clear positive effects on
the ability of students to apply the acquired knowledge (Van den Bossche,
Gijbels & Dochy, 2000). This effect is noticeable in every year of the
curriculum. The expectation is that business students pursuing a problem-
based curriculum will be more capable of applying the acquired knowledge
than students following a conventional curriculum and this should be true in
every year of the curriculum.
3. METHODOLOGY
3.1 Research Population
The research population consisted of business students at two institutes.

The first institute delivers an educational program that can be characterized
as conventional. It is discipline-oriented with formal lectures as main part of
the educational activities and is located in a Belgian university. A
characteristic of this program is that of free entrance for all students. The
other institute has a problem-based curriculum. Students meet each other
twice a week in small tutorial groups, of between 12 and 15, guided by a
tutor. In these groups they are confronted with authentic problems, problems
which they might encounter in real life situations. Because authentic
problems are often not solvable within mono-disciplinary constraints, the
curriculum is organized on a multidisciplinary basis. This implies that
problems are discussed from different disciplinary points of view. The
problems are the context in which the students study the basic concepts and
models within the fields of economics (Segers, 1997). A Dutch Faculty of
Economics organizes this problem-based education. At this institute there is
a kind of “selection at the gate”. Students are only admitted when they have
completed certain preliminary training.
3.2 Sample and Procedure
The specific situation of both institutes compelled us to undertake an

adapted sampling in both institutes.
3.2.1 The Conventional Institute
All the subjects of the population were part of the sample, except at the
start of Year 1 where a random sample was taken. This resulted in 50
participants for the pretest, while 16, 15 and 22 students, respectively,
participated in the tests at the end of Year 1, Year 2 and in Years 3 and 4.
3.2.2 The PBL Institute
All tests were taken from a random sample of the population, except at
the start from Year 1 where a quota sample procedure was used. 13 students
participated in the pre-test. There were, respectively, 21 and 46 students for
the test at the end of Year 1 and Year 2. Also 54 students from Years 3 and 4
participated in the study.
3.3 Research Design
Because the research was conducted in a “real-life” context a random

distribution of the subjects over experimental and conventional groups was
not feasible. In using this quasi-experimental design internal validity is more
vulnerable than in an experimental design and generalizability can be merely
presumed. Its great advantage lies in its ecological validity that is the extent
to which findings can be generalized to the “real world” (Campbell &

Stanley, 1963).
In this research the focus was on the subject of macroeconomics, as part
of the economic curriculum. A curriculum-analysis was conducted in order
to find out when and to what extent macroeconomics had a place in both
curricula. Particularly in the first two years a difference was found between
the two curricula.
In the disciplinary-oriented curriculum of the conventional institute a
general economics course, in which the macroeconomic context is covered,
is taught in the first year. In the second year of their studies, the students
have to follow a course dedicated to macroeconomic theories and insights. In
the third and fourth year students follow a range of courses in which the
application of macroeconomic content is required.
As a consequence of the multi-disciplinary approach of the curriculum in
the PBL institute, it is not always possible to ascertain the points at which
macroeconomic concepts need to be applied. However, in the first year of
the course, the macroeconomic way of seeing is already explicitly placed at
the center. Also in the second year attention is given to the macroeconomic
perspective.
This difference in the structure of the curriculum is important for a
correct interpretation of the results. A pretest was taken in order to check the
prior knowledge of the students. Next, tests were taken after one year. For
PBL students (i.e. those selected at the gate) this was at the end of Year 1
and for those receiving conventional education (selection in Year 1) at the
start of the second year. This difference in the point of assessment was
introduced to adjust for the difference in prior knowledge that could be
recorded as a consequence of the difference in the moment of selection. A
third test was conducted at the end of Year 2. As a result of the structure of
the programs, in Years 3 and 4 of both programs (a student can choose when
s/he follows certain courses), the fourth test was taken from a mixed group
of third and fourth year students.
3.4 Research Instrument
Both the acquired knowledge and the degree to which students are
capable of applying the knowledge were evaluated. Therefore, two
instruments were used: a knowledge-test and a case-based test.
3.4.1 The Knowledge Test
The knowledge test measures primarily factual knowledge, the meaning

of symbols and the concepts and principles pertaining to a particular theme.
This type of knowledge is called declarative knowledge (Anderson, 1983;

Dochy & Alexander, 1995). The items contained in the knowledge test
require the students to reproduce their knowledge and/or demonstrate
insight. It is not sufficient for students to remember or even understand
isolated definitions of domain-related concepts. They need to understand the
frame of reference that organizes them (Segers, 1997; Segers, Dochy & De
Corte, 1999).
To assure relatively even coverage of the domain, an analytic grid is used
for the construction of the test.
Item 1:
True/?/false The Gross domestic product measures the total spending of all residents of an
economy.
(false)
Item 2:
A car manufacturer takes more employees in service and pays their wages. As a consequence the
production increases, which is held in stock for selling later.
True/?/false The gross domestic product (GDP) increases.
(true)
Item 3:
Suppose there is inflation, and the Central Bank changes the growth rate of the money supply so as to
equal the long term annual growth rate of production. Suppose also that people believe this money
growth rate will continue to equal the growth rate of production. In the following several immediate
effects are mentioned.
True/?/false An immediate effect would be that the nominal interest rate would fall. (true)
True/?/false An immediate effect would be that actual inflation would temporarily be

negative. (true)
Figure 1: Examples of Items from the Knowledge Test.
These examples assess conceptual understanding. The first and the

second questions require students to be able to recognize the definition of the
gross domestic product. The second question is embedded in a simplified
authentic situation. It asks for more than merely factual recall. Students have
not only to reproduce the definition of the concept, but also apply it to the
case of the car manufacturer. Since only the relevant variables are
mentioned, students do not need to retrieve the relevant information from the
case in order to be able to answer the question. The third question starts from
a macroeconomics case which, like the second question, presents the critical
elements for solving the problem. To answer the questions, students need to
understand the various relevant concepts (nominal interest rate, inflation,
rate of growth of the money supply, long-run annual growth rate of

production). Additionally, they are required to master the interconnections
between these concepts.
As is clear from the examples, we used the “question mark” option. This
option allows the students “to pass” on a particular question. Students who
circle the question mark option indicate that they have not mastered the
subject. This option allows them not to give an answer and therefore to avoid
guessing. They are not punished for not knowing. Choosing the question
mark option gives a score of 0 points. On the other hand, they lose one point
(-1) when giving the wrong answer. Circling the right answer means +1
score. The introduction of this scoring system makes guessing only attractive
for students who are reasonably sure of the answer. Therefore, in most cases
choice of the wrong answer reveals that students have misunderstood the
objective measured (Segers, 1997; Segers, Dochy & De Corte, 1999).
3.4.2 Case-based Test
In this test, students are confronted with a case that describes a

macroeconomic problem. Essay-type questions are posed about this
problem. The case-based test measures whether students are able to retrieve
the relevant concept (model, principle) for the problem. Furthermore, it
measures if they can use these instruments to solve the problem. It measures
if the knowledge is usable (Glaser, 1990) or whether students know “when
and where” (conditional knowledge). In short, the test measures the extent to
which students are able to analyze problems and contribute to their solution
by applying the relevant tools (knowledge). Figure 2 gives an example of
such a question.
Case: THE NV BELGIUM (for a complete description of the case: contact the first author)
Question:
Give two actions the Belgian government can take as part of a policy in the tradition of Keynes and
describe the consequences of the actions by means of the IS/LM model.
Figure 2: An Example of a Case-based Question.
3.4.3 Difference Between the Knowledge and the Case-based Test
The knowledge test aspires to measures what the students know. The test
appeals to their reproductive thinking. Besides, and this is the maximum, the
test measures if they are capable of using the learned subject matter, but only
in situations where solution and solution strategy is defined in advance. This
is known as convergent thinking (Tempelaar, 1993).
The case-based test measures the ability to apply. In other words, can the
student also apply the learned subject matter in situations where different
ways to reach the solution are possible? This is known as divergent thinking
(Tempelaar, 1993) or creative application (Block de & Heene, 1992). In
addition, the case creates a more complex context than any knowledge test
ever can.
The difference between knowledge and application is best seen as a
continuum, with the knowledge test tending towards the knowledge pole and
the case-based test towards the application pole.
3.5 Concrete Operationalisation and Construction of

Instruments
3.5.1 Knowledge Test
Different parallel tests were used in this research. Teachers of both

institutes screened these tests for content and instructional validity.
Instructional validity refers to the similarity between the operational
curriculum and what is being tested (McClung, 1979). None-relevant items
were removed from the tests. The content validity is assured by the use of
the analytic grid for the construction of the test. This was also double-
checked by a teacher in the conventional institute.
Furthermore, the tests were comparable concerning difficulty. To assure
this, the p-values of the tests as a whole were calculated. The p-value is
defined as the proportion students that choose the correct alternative
(Dousma & Horsten, 1995). To ascertain the p-value of the different test,
data were used from the summative use of a different test in the curriculum
of the PBL institute. The tests were all filled in by comparable groups of
approximately 280 students. The p-values of the different tests were
respectively 0.60, 0.52, 0.57 and 0.59. One can conclude that all these tests
have the same degree of difficulty.
3.5.2 Case-based Test
An authentic case was formulated by experts in the field of

macroeconomics. The construction and review of the problems were guided
by a set of criteria for case writing (Leenders & Erskine, 1989; Van
Vilsteren, Van der Heijden & Arts, 1993). The described economic problem
starts with an introduction, presenting information about the context and the
position of the student. The specific problem situation is described next. The
problem description ends with two questions.
The schemes for the analysis of the responses were based on a detailed
model for problem-analysis by the expert-constructors. A specialist in the
content matter evaluated the responses of the students. One of the authors of
this article acted as second reviewer. Central criteria for the coding were the
number of correct concepts, relationships between the concepts used for
problem analysis and the correctness of the product (i.e. solution of the
problem).
4. RESULTS
In the first part of this section the results of the knowledge-test are
presented. In the second part a description of the results for the case-based
test is provided. To begin the pre-test is analyzed by independent t-tests to
discern differences in the prior knowledge of the students. Then the expertise
(-development) of the students in both learning environments is analyzed by
a 2 x 3 analysis of variance. The interpretation of the main effects will be
done by analyzing the marginal means and their respective 95% - confidence
intervals. The interpretation of the interactions will be done by studying the
pattern of means (and their confidence intervals) and the effect-sizes per
expertise-level (d-index; Cooper, 1989).
4.1 Knowledge Acquisition
4.1.1 Pre-knowledge Test
Both groups score very low on the knowledge test at the moment of the
pre-test. The students of the conventional institute have a mean total score of
2.88%. The mean total score of the students of the problem-based institute is
7.37%. This difference (4.48%) reaches a significant level (t = 2.14, df = 61,
p = .036, CI: +/- 4.18%).
As a consequence of the specific context of the institutes (selection at the
gate or after the first year of training) a difference in results could be
expected. The selection in the conventional institute happens at the end of
the first year. From that moment on, the two groups can be considered as
equal. By accommodating the second moment of measurement to these
conditions (cfr. supra), the groups can be compared from that moment on.
4.1.2 Knowledge Development Through the Curricula
4.1.2.1 Total-score
The results of the analysis with the total score as a dependent variable are
presented in table 1.
Table 1: 2 X 3 Analysis of Variance of the True Minus False Score in Terms of Percentage
(Total Score)
SS Df MS F P
Learning env (L) 1204.62 1 1204.62 8.50 .004
Expertise (E) 2826.13 2 1413.06 9.97 .000
LXE 1028.05 2 514.03 3.63 .029
Within 22812.40 161 141.69
Total 73584.55 167
Both the main effects and the interaction effect turn out to be statistical
significant. The results between the expertise-level and the results on the
knowledge-test is the most significant, followed by the effect of the learning
environment and then by the expertise-learning environment interaction. The
interpretation of these effects is guided by the plot of the means in figure 3,
with the means and the effect-sizes based on the differences per expertise-
level.
The significant main effect of expertise-level (F(2,161) = 9.97, p = .000,
MSw = 141.69) is caused by the decline of the score of the total group. At
the end of the first year they score 23.64% (CI: +/- 3.97%), at the end of the
second year this is merely 15.31% (CI: +/- 3.81%) and in the third and fourth
year 12.19% (CI: +/- 3.13%).
There is a significant relation between the learning environment and the
result on the knowledge test, also when this is controlled for the level of
expertise (F(1, 161) = 8.50, p = .004). The students of the PBL institute
score higher than students of the conventional institute. Their scores are,
respectively, 20.20% (CI: +/- 2.32%) and 13.96 (CI: +/- 3.53%). This results
in an effect size of 3.49.
The analysis shows that a complete description of the results on the
knowledge test must not only comprise the main effects, but also the
interaction between the learning environment and the expertise-level. Also
this effect is significant (F(2, 161)= 3.63, p = .029). The mean difference
between the two learning environments is strikingly larger at the end of the
first year than at the end of the second year and in the third and the fourth
years.
The students of the PBL institute in Year 1 scored 31.00% (CI: +/-
5.13%). This is almost twice as good as the students of the conventional
institute who scored 16.27% (CI: +/- 6.07%). This resulted in an effect size
of 4.79. The two groups converged at the end of Year 2, when the difference
is only 1.69% (ES = 0.49). The students of the PBL institute have a mean
score of 16.15% (CI: +/- 16.15%) which is still higher than the students of
the conventional institute with a score of 14.46% (CI: +/- 6.79). Also in the
third and the fourth year the students of the PBL institute score higher
(13.43%, CI: +/- 3.20%) than the students of the conventional institute
(11.15%, BI: +/- 5.39%). The difference is 2.29% (ES = 0.84).
Figure 3: Mean Total Score (true minus false) as a Function of the Learning
Environment and the Expertise-Level (the error bars show the 95%
confidence intervals of the means).
The interaction effect is mainly due to the strong higher score of the PBL
students at the end of Year 1. This difference disappears at the other
expertise-levels. However, the tendency that the students in a PBL
environment get higher scores than students in a conventional learning
environment (cfr. also the main effects) stays.
Actually, from the curriculum-analysis we learned that, due to the
structure of the curriculum, the results of the students in the PBL
environment at the end of Year 1 need to be compared with the results of the
students in the conventional learning environment at the end of Year 2. At
that moment, both groups have had a complete introduction to
macroeconomics. Comparing these two, one can see that the first group
scores higher than the latter. This difference seems almost to disappear in the
following year(s).
4.1.2.2 True-score
Figure 4 gives a plot of the mean number of correct scores in function of
the learning environment and the expertise-level. In this plot also the results
of the pre-test are depicted, however these are not included in the analysis of
variance.
Table 2: 2 x 3 Analysis of Variance of the Number of Correct Scores on the Knowledge Test.
SS df MS F P
Learning env. (L) 1663.31 1 1663.31 18.25 .000
Expertise (E) 858.254 2 429.13 4.71 .010
LXE 1620.90 2 810.45 8.89 .000
Within 14671.85 161 91.129
Total 35030.47 167
The analysis of variance presented in table 2 shows that both the main
effects and the interaction effect is significant.
The mean correct scores differ between the three levels of expertise (F(2,
161) = 4.71, MSw = 91.13, p = .010). The first and the second year do not
differ that much, with a score of 45.46% (CI: +/- 3.19%) and 46.38% (CI:
+/- 3.06%) respectively. In Years 3 and 4 the students give on average less
correct answers (40.79%, CI: +/-2.51%).
There is also a significant relationship between the learning environment
and the number of correct scores (F 91, 161)= 18.25, p = .000). The students
of the conventional institute get at average 40.55% of the answers correct
(CI: +/- 2.82%). The students of the PBL institute do significantly better
with a mean score of 47.88 (CI: +/- 1.86). This difference results also in a
strong practical effect size of 5.12.
Also the interaction effect is very significant (F(2, 161)= 8.89, p = .000).
As shown in figure 4 the students of the PBL institute do better at the end of
Year 1 with a score of 53.97% (CI: +/- 4.11%), compared with the score of
39.93% (CI: +/- 4.87%) from the students of the conventional institute. This
difference is not found at the end of Year 2. At this point in time the students
of the conventional institute (47.30%, CI: +/- 5.44%) score slightly better
than the students of the PBL institute. In the third and the fourth year the
students of the PBL institute (44.20%, CI: +/- 2.57%) do again much better
than the students of the conventional institute (37.38%, CI: +/- 4.32%).
These results indicate that the students of the PBL institute know the
most at the end of Year 1 and the students of the conventional institute seem
to know the most at the end of the second year. This seems logical, knowing
the structure of the curriculum of both institutes. In the second year, the
number of correct answers of both groups is comparable. This in spite of a

decrease in the results of the PBL group compared with their results in Year
1. However, in the case of the students of the problem-based curriculum, this
is one year after the study of the basic macroeconomic principles. For the
students of the conventional curriculum, this is immediately after the
thorough studying of macroeconomics. The students of the conventional
institute score not as high in the third and the fourth year as they did in the
second year. In other words, they also have a “drop” after the year of
intensive study of the basic macroeconomic knowledge. The PBL students
stay at the same level as in the second year.
Figure 4: Mean Number of Correct Scores in Function of the Learning

Environment and the Expertise-Level (the error bars show the 95%
confidence intervals of the means).
4.2 Knowledge Application
4.2.1 Pre-test Scores
The average scores on the case-based test of the students of the

conventional and the PBL institute are respectively 0.33 and 0.94 (with a
maximum score of 10). An independent two-sided t-test showed that these

scores differ significantly (t = 4.03, df = 91, p = .000, CI: +/- 0.30). At the
moment of entry the students of the PBL institute score higher than the
students of the conventional institute.
Also in this case the difference in admission policy needs to be
considered. The explanation that the difference in the pre-test is caused by
this difference in admission policy allows us to consider the two groups as
comparable from the moment the selection has also taken place in the
conventional curriculum.
4.2.2 Development of the Ability to Apply Knowledge Through the

Curricula
Table 3 presents the results of the 2x3 analysis of variance with two
levels of learning environment and three levels of expertise. The results are
interpreted using the plot of means in figure 5, the marginal means and the
effect sizes.
Table 3: 2 x 3 Analysis of Variance of the Scores on the Case-Based Test.

SS Df MS F
Learning env. (L) 16.93 1 16.94 2.60
Expertise (E) 7.61 2 3.80 0.58
LXE 8.91 2 4.46 0.68
Within 1049.99 161 6.52
Total 3409.94 167
The results presented in table 3 indicate that there is no significant

relationship between the learning environment and the results on the case-
based test (F(1, 161) = 2.60, p = .109, MSw = 6.52). However, one can
observe that the students of the PBL institute (4.07, CI: +/- 0.489) score on
average higher than the students of the conventional institute (3.33, CI: +/-
0.76). This difference, although it is not statistically significant, results in a
practical significant effect size of 1.92.
A comparison of the mean results on the different levels of expertise
(controlled for learning environment) does not indicate any significant
differences (F(2, 161) = 0.58, p = .559). On average the students score 4.06
(CI: +/- 0.85) at the end of Year 1. In the second year and in the third and the
fourth year the tendency to get lower results is observed, respectively 3.51
(CI: +/- 0.82) and 3.52 (CI: +/- 0.67).
The plot of the means in figure 5 suggests an interaction effect. The
difference between the mean scores of the students of the conventional
institute (3.30, CI: +/- 1.30) and of the PBL institute (4.82, CI: +/- 1.10) at
the end of the first year is larger than the small difference between the mean
scores of the students of the conventional institute (3.42, CI: +/- 1.46) and
the PBL institute (3.60, CI: +/- 0.74) at the end of the second year. Whereas
the difference in Year 1 is 1.52, resulting in an effect size of 2.31, there is
almost no difference in Year 2 (0.19, ES = 0.25).
Figure 5: Mean scores on the case-based test in function of the learning

environment and the expertis-level (the error bars show the 95% confidence
intervals of the means).
Compared with Year 2, the difference increases in the third and fourth
year. The students of the conventional institute score 3.26 (CI: +/- 1.16) and
those of the PBL institute core 3.77 (CI: +/- 1.16); a difference of 0.51,
resulting in an effect size of 0.87.
However, in spite of all those differences, no significant interaction effect
was established (F(2, 161) = 0.68, p = .506). This analysis indicates that
there are only small differences between the different levels of expertise.
Also this analysis indicates that the differences between the scores of the
students in the PBL environment and the conventional environment are
comparable. Nevertheless, the students have a tendency to get higher scores
(also the effect sizes point in that direction), especially in the first year.
The content matter necessary for solving the case was in both institutes
covered in the first year of the curriculum. So this cannot be an alternative
explanation for the differences found at the end of the first year. At the end
of the second year no differences are found. The converging of the scores at
the end of year two is foremost a result of the decline in scores of the
students at the PBL institute. At the end of Years 3 and 4 there is again a
practical significant difference between the two groups of students. The
tendency is for the results of the PBL students to increase again, after a
decline in Year two, and that the results of the conventional students slightly
decrease.
5. CONCLUSION AND DISCUSSION
In this closing part the answers the research has given to the posed
research questions will be considered. Next, some critical reflections and
suggestions for further research will be made.
5.1 Knowledge Acquisition
First, we wished to find the answer to the question about the extent to
which students following a problem-based curriculum possess an accessible
knowledge base for the subject matter compared to students who studied in
the conventional way. Not only differences in learning environments were
studied, but also developmental aspects, such as retention.
The students from the PBL institute have a higher mean total score on the
knowledge test at every moment. This results in a statistical main effect of
the learning environment, this difference is also practically very significant
(ES = 3.49).
A clearer image of the knowledge base can be drawn if the number of
correct answers is also taken into consideration. In general, here a practical
and statistically significant higher score of the students from the PBL
institute is also observed. The results indicate that whereas the students of
the PBL institute know the most at the end of Year 1, the students of the
conventional institute seem to know the most at the end of the second year.
This is in line with the structure of the curriculum. In the second year, the
number of correct answers of both groups is comparable. This in spite of a
decrease in the results of the PBL group compared to their results of Year 1.
However, for the students of the problem-based curriculum this is one year
after the study of the basic macroeconomic principles. For the students of the
conventional curriculum this is immediately after the thorough studying of
macroeconomics. The students of the conventional institute do not score as
high in the third and the fourth year as in the second year. In other words,
they also have a “drop” after the year of intensive study of the basic
macroeconomic knowledge. The PBL students stay at the same level as in
the second year.
In general, the analyses with the number of correct answers suggest the
same conclusion as those with the total score, only more pronounced. Also
they correct for the apparent conclusion, on the basis of the total scores, that
the students of the conventional institute do not make any progress after
following an important macroeconomic course. This is due to the higher
number of incorrect answers they also make after this period. This points to
the existence of a lot of misconceptions.
These results do not confirm the stated hypotheses. The students in a
problem-based curriculum score better on the test measuring knowledge
acquisition from the first moment on. This conclusion is primarily based on
the finding that the students of the PBL institute at the end of Year 1 score
higher than the students of the conventional institute at the end of Year 2.
This is the moment when they were both thoroughly confronted with
macroeconomics.
With these results, this research is in line with some other research that
has found positive effects of PBL on the knowledge of students (Antepohl &
Herzig, 1997, 1999; Distlehorst & Robbs, 1998; Doucet et al., 1998; Lewis
& Tamblyn, 1987; Richards et al., 1996; Son & Van Sickle, 2000; Tans et
al., 1986; Verhoeven et al., 1998). A possible explanation for the better than
expected results of the PBL students is the nature of the research instrument.
Van den Bossche and colleagues (2000) ascertained that the more an
instrument appeals to the retrieval strategies and skill of students, the larger
the established effect of PBL. From the description of the instrument used in
foregoing research we learn this instruments appeals to a certain degree to
the skills of the students. This is a possible explanation for the effects found.
Anyway, these results confirm the character of PBL as a learning
environment that promotes the structuring and elaboration of knowledge
resulting in a more accessible knowledge base.
The collected data do not permit any conclusion on the retention of the
subject matter in the long-term. Both groups show a decline in their results
on the test a year after their “real” confrontation with macroeconomics. A
positive observation is that the students of the PBL curriculum do not
experience any further decline in the third and the fourth year. However, we
cannot compare these results with the knowledge development of the
students in the conventional curriculum, because we have no data for two
years after their “peak”.
5.2 Knowledge Application
This research was also looking for an answer to the question as to

whether the students in a problem-based curriculum were able to better apply
their knowledge than students who studied the subject matter in the
conventional way. The analysis of the scores on the case-based test indicated
no statistically significant effect of the learning environment on the scores of
the students. However, the students of the PBL institute have the strong
tendency to score higher at each level of expertise. Also the effect sizes in
Year 1 (ES = 2.31) and in the third and the fourth year (ES = 0.87) indicate a
practically significant effect. Besides that, the general effect of the learning
environment is practically significant.
If we confront these results with the stated hypotheses that students in a
problem-based curriculum are more capable in applying their knowledge on
every moment in that curriculum, we have to conclude that the tendency in
the results confirms this hypothesis. However, this tendency is not
statistically significant.
One can conclude very carefully that a PBL environment is capable of
promoting the application of the acquired knowledge by students through
setting the adagio “knowledge as an instrument” (Norman & Schmidt, 1992)
central. Already at the end of the first year the influence of this powerful,
student-based learning environment is noticeable. The students of the
problem-based institute score better than the students of the conventional
institute, at the end of Year 2.
A remarkable phenomenon is that the students in both groups never get
higher results on the test after the first year. In the PBL-group this is possibly
due to the decrease in knowledge. However, the conventional group also
makes no progression, although their knowledge of macroeconomics
increases in the second year. Apparently, the students are not capable of
applying the newly acquired knowledge; the new knowledge stays inert
(Mandl, Gruber & Renkl, 1996).
5.3 Critical Reflection and Suggestions for Further

Research
Below four critical comments on our own research are made, followed by
suggestions for further research. The first concerns the critical success
factors of PBL; the second deals with the one-sided attention given to the
cognitive end of education; and the third comment points to the lack of data
on long-term effects. Finally, some remarks are made on the validity of the
study.
5.3.1 Critical Success Factors
The study examines the effects of a PBL learning environment. Even if

the differences that were found were to be seen as a genuine effect of the
learning environment (cf. infra), it remains difficult to single out those
aspects of the learning environment, which caused those differences. It also

remains unclear what aspects play a crucial role in establishing a powerful
learning environment. Nevertheless, such information could play an
important role in successfully implementing PBL and teacher training.
Further research into the critical success factors of PBL is indicated.
5.3.2 Limitation to Cognitive Effects
Although the literature on education pointed out the importance of

affective and motivational elements (Boekaerts, 1993), the above study was
limited to the cognitive effects of PBL, in particular knowledge acquisition
and application. For this reason, it seems necessary that future research also
aims at the affective and motivational aspects of learning, since powerful
learning environments, in particular PBL environments, have an impact on
this area as well in the sense that students have been shown to acquire a
more positive attitude (Albanese & Mitchell, 1993; Lieberman et al., 1997;
Percac & Armstrong, 1998; Vernon & Blake, 1993).
5.3.3 Long-Term Effects
When explaining the results, we referred to the fact that PBL students in
some instances acquire less factual knowledge, but that they organize this
knowledge better so that more of it is retrievable if availed upon. This
implies that PBL students will possess this knowledge for the long-term and
that they will also be better able to apply it after completing their studies.
The results from the meta-analysis by Van den Bossche et al. (2000), which
had retention period as a mediating variable, pointed in that direction. The
data gathered from our own research does not permit, however, to reach a
substantiated conclusion about subject matter retention. In order to chart the
effects of PBL in the long term a longitudinal study is required, which
continues to follow the students after having graduated.
5.3.4 Internal and External Validity
The results of this study do not lend themselves to generalization, as a

number of remarks about the validity of the study will show. Statements
about internal validity must be interpreted cautiously as a result of the fact
that the students were not distributed at random over the experimental and
control conditions. Although efforts were made to examine the
comparability of the groups, the best guarantee of countering selection bias
is random group distribution. This, however, is hardly possible in
ecologically valid research.
In view of the specific context of the institutes in which the study was
conducted and the limited range of subject matters studied, caution must also
be exercised where external validity is concerned. In addition, the limited
number of participants possibly endangers the external validity.
The study was a first step towards investigating the effects of PBL in
various domains. The validity of the results of the study must be evaluated in
the light of future similar studies (Son & Van Sickle, 2000).
REFERENCES
Albanese, M.A., & Mitchell, S. (1993). Problem-based learning: A review of literature on its
outcomes and implementation issues. Academic Medicine, 68, 52-813.
Anderson, J.R. (1983). The architecture of cognition. Cambridge, MA: Harvard University
Press.
Antepohl, W., & Herzig, S. (1997). Problem-based learning supplementing in the course of
basic pharmacology-results and perspectives from two medical schools. Naunyn-
Schmiedeberg’s Archives of Pharmacology, 355, R18.
Antepohl, W., & Herzig, S. (1999). Problem-based learning versus lecture-based learning in a
course of basic pharmacology: A controlled, randomized study. Medical Education, 33 (2),
106-113.
Arts, J.A.R.M., Gijselaers, W.H., & Boshuizen, H.P.A. (2000). Expertise development in
managerial sciences: The use of knowledge types in problem-solving. Paper presented at
the Annual meeting of the American Educational Research Association, New Orleans,
Louisiana.
Block De, A., & Heene, J. (1992). Inleiding tot de algemene didactiek [Introduction to
general didactics]. Antwerpen : Standaard Educatieve Uitgeverij.
Boekaerts, M. (1993). Being concerned with well-being and with learning. Educational
Psychologist, 28, 149-167.
Campbell, D.T., & Stanley, J.C. (1963). Experimental and quasi-experimental designs for
research. Chicago: Rand McNally.
Cooper, H.M. (1989). Integrating research. A guide for literature reviews: Vol 2. London:
Sage Publications.
Distlehorst, L.H., & Robbs, R.S. (1998). A comparison of problem-based learning and
standard curriculum students: Three years of retrospective data. Teaching and Learning in
Medicine, 10 (3), 131-137.
Dochy, F., & Alexander, P.A. (1995). Mapping prior knowledge: A framework for discussion
among researchers. European Journal for Psychology of Education, 10, (3), 225-242.
Dochy, F., Segers, M., Gijbels, D., & Van den Bossche, P. (2002). Studentgericht onderwijs
& probleemgestuurd onderwijs: betekenis, achtergronden en effecten [Student-centered
education & problem-based learning: Meaning, background and effects]. Utrecht:
Lemma.
Doucet, M.D., Purdy, R.A., Kaufman, D.M., & Langille, D.B. (1998). Comparison of
problem-based learning and lecture format in continuing medical education on headache
diagnosis and management. Medical Education, 32 (6), 590-596.
Dousma, T., & Horsten, A. (1995). Tentamineren [Examinations]. Hoger Onderwijs Reeks.
Groningen : Wolters-Noordhoff.
Engel, C.E. (1997). Not just a method but a way of learning. In D. Boud & G. Feletti (Eds.),
The challenge of problem-based learning (2nd ed.) (pp. 17-27). London: Kogan Page.
Gagné, E.D. (1993). The cognitive psychology of school Learning (2nd ed.). New York:
HarperCollins Publishers.
Gijselaers, W.H. (1995). Perspectives on problem-based learning. In W. Gijselaers, D.
Tempelaar, P. Keizer, J. Blommaert, E. Bernard & H. Kasper (Eds.), Educational
Innovation in Economics and Business I: The case of problem-based learning (pp. 39-52).
Dordrecht: Kluwer.
Gijselaers, W.H., & Woltjer, G. (1998). Cognitive science perspectives on learning and
instructional quality. In D.T. Tempelaar, F. Wiedersheim-Paul & E. Gunnarsson (Eds.),
Educational Innovation in Economics and Business II. In search of quality (pp. 345-354).
Dordrecht: Kluwer.
Glaser, R. (1990). Toward new models for assessment. International Journal of Educational
Research, 14, 475-483.
Leenders, M.R., & Erskine, J.A. (1989). Case research: The case writing process. Ontario:
University of Western Ontario.
Lewis, K.E., & Tamblyn, R.M. (1987). The problem-based learning approach in
Baccalaureate nursing education: How effective is it? Nursing Papers, 19 (2), 17-26.
Lieberman, S.A., Stroup-Beham, C.A., Peelm J.L., & Camp, M.G. (1997). Medical student
perception of the academic environment: A prospective comparison of traditional and
problem-based curricula. Academic Medicine, 72 (10), s13-s15.
Mandl, H., Gruber, H., & Renkl, A. (1996). Communities of practice toward expertise: Social
foundation of university instruction. In P.B. Baltes & U.M. Staudinger, Interactive minds.
Life-span perspectives on the social foundation of cognition (pp. 394-412). Cambridge:
Cambridge University Press.
McClung, M.S. (1979). Competency testing programs: Legal and educational issues.
Fordham Law Review, 47, 6511-712.
Norman, G.R., & Schmidt, H.G. (1992). The psychological basis of problem-based learning:
A review of the evidence. Academic Medicine, 67 (9), 557-565.
Percac, S. & Armstrong, E.G. (1998). Introducing a problem-based anatomy course in a
traditional curriculum: A Croatian experience. Medical Teacher, 20 (2), 114-117.
Poikela, E. & Poikela, S. (1997). Conceptions of learning and knowledge – impacts on the
implementation of problem-based learning. Zeitschrift fur Hochschuldidactik, 21 (1), 8-21.
Richards, B.F., Ober, P., Cariaga-Lo, L., Camp, M.G., Philp, J., McFarlane, M., Rupp, R., &
Zaccaro, D.J. (1996). Rating of students’ performances in a third-year internal medicine
clerkship: A comparison between problem-based and lecture-based curricula. Academic
Medicine, 71(2): 187-189.
Segers, M. (1997). An alternative for assessing problem-solving skills: The overall test.
Studies in Educational Evaluation, 23 (4), 373-398.
Segers, M., Dochy, F., & De Corte, E. (1999). Assessment practices and students’ knowledge
profiles in a problem-based curriculum. Learning Environments Research, 12 (2), 191-
213.
Son, B., & Van Sickle, R.L. (2000). Problem-solving instruction and students' acquisition,
retention, and structuring of economics knowledge. Journal of Research and Development
in Education, 33(2), 95-105.
Tans, R.W., Schmidt, H.G., Schade-Hoogeveen, B.E.J., & Gijselaers, W.H. (1986). Sturing
van het onderwijsleerproces door middel van problemen: een veldexperiment [Guiding the
learning process through problems: A field experiment]. Tijdschrift voor
Onderwijsresearch, 11, 35-46.
Tempelaar, D. (Ed., 1993). Handleiding Toetsconstructie [Manual testconstruction].

Maastricht: Rijksuniversiteit Limburg, Faculteit der Economische Wetenschappen.
Todd, R. (1999). Reading the future: Knowledge management 4: Towards a community of
learning. Scan, 18 (3), 34-37.
Van den Bossche, P., Gijbels, D., & Dochy, F. (2000). Does Problem-based learning educate
problem-solvers? A meta-analysis on the effects of problem-based learning. Paper
presented at the seventh EDiNEB conference, Newport Beach, USA, June.
Verhoeven, B.H., Verwijnen, G.M., Scherpbier, A.J.J.A., Holdrinet, R.S.G., Oeseburg, B.,
Bulte, J.A., & Van der Vleuten, C.P.M. (1998). An analysis of progress test results of PBL
and non-PBL students. Medical Teacher, 20 (4), 310-316.
Vermetten, Y.J.M. (1999). Consistency and variability of student learning in higher
education. Ph. D. Thesis. Catholique University Brabant, The Netherlands.
Vernon, D.T.A., & Blake, R.L. (1993). Does problem-based learning work? A meta-analysis
of evaluative research. Academic Medicine, 68, 550-563.
Vilsteren Van, P.P.M., Van der Heijden, M.P., & Arts, A.R.M. (1993). Het gebruik van
casussen in cursussen van de Open Universiteit [The use of cases at the Open University].
Heerlen: Open Universiteit.
Wiel van de, M. (1997). Knowledge encapsulation. Studies on the development of medical
expertise. Unpublished PhD. Thesis, University of Maastricht, The Netherlands.
Learning about Teaching Information Systems in a
Problem-Based Curriculum: An Exploratory Study of
the Impact of Students’ Individual Differences on
their Conception and Perception of Problem Tasks
Jan Nijhuis, Mien Segers & Wim Gijselaers

Department of Management Science, Faculty of Economics and Business Administration,
University of Maastricht, the Netherlands
1. INTRODUCTION
Todd (1999) describes the late 20th Century as the “Age of Information”,
where an emphasis is placed on the external organization, transformation and
communication of information. He argues that the 21st Century will be the
“Age of the Mind”, where the key success factors of organizations will be
the extent to which they are able to use, share and create knowledge. It is
well acknowledged that in order to comply with the demands of the Age of
the Mind, it is imperative for educators at all levels to develop learners with
cognitive and meta-cognitive, as well as social, competencies. As Birenbaum
and Dochy (1996) indicate, cognitive competencies include problem solving,
critical thinking, formulating questions, searching for relevant information,
making informal judgments and ensuring the efficient use of information.
Since schools and universities are preparing students for society, both the
new role of information and the new required competences should be taken
into account when developing courses. Within a variety of disciplines,
innovations in education are implemented. For informatics, new ways of
teaching are described by Davey and Tatnall (1994), Jurema et al. (1995)
and Penjam (1997) and are discussed at gatherings organized by the
Information Systems Education Conference (ISECON).
229
230 Jan Nijhuis, Mien Segers & Wim Gijselaers
The innovations in education are mainly based on constructivist

principles of learning and teaching (Savery & Duffy, 1995). They are
characterized by the view that learning means actively constructing
knowledge and skills on the basis of prior knowledge, embedded in contexts
that are authentic and offer ample opportunities for social interaction. One
example of this kind of instruction is problem-based learning (PBL),
nowadays implemented in a variety of institutions. In the field of
informatics, for example, Ma (1994) reported on the implementation of PBL
in a course on information systems. In PBL the problem or task is the
starting point for students’ learning. A task is a short description or
compilation of phenomena and events that can be perceived in reality. This
has to be analyzed, explained and/or solved by the tutorial group in terms of
underlying principles, mechanisms and processes (Moust, Bouhuijs &
Schmidt, 1989). The importance of the problem task as the core element in
PBL is emphasized by the research of Gijselaers and Schmidt (1990). On the
basis of this research, Norman and Schmidt (2000) present a path model
indicating the importance of the problem task in PBL and its relation with
other variables of the PBL environment.
Nevertheless, up till now, limited empirical research has been conducted
in this area. In their review, Albanese and Mitchell conclude that although
the core of PBL is the use of problem tasks to focus learning: “little research
has been conducted that will provide more than intuitive guidelines for
development of problems” (1993, p. 72). Dolmans and colleagues refer to
studies of Coulson and Osborne (1984), Shahabudin (1987) and Dolmans et
al. (1993). They summarize that: “most studies investigating effective cases
have focused on the relationship between student-generated learning issues
and faculty objectives.”(1997, p. 185). Although there is a growing interest
in the influence of students’ conceptions and perceptions on learning, it is
surprising that none of the studies on problem tasks in PBL have tackled this
issue from this perspective. This implies questions such as: “How do
students conceive problem tasks?” “How do they experience the use of
problem tasks?” “Which variables account for their conceptions and
perceptions?” As Prosser and Trigwell state: “University teachers need to try
to look at their designs through their students’ eyes” (1999, p. 59). Also
Birenbaum (2000) refers to a network of relations between perceptions and
conceptions, both of which are influenced by individual characteristics, such
as personality, attitudes, prior knowledge and learning styles. This paper
focuses on the interrelations of a set of student characteristics within a PBL
instructional context and its relations with students’ conceptions and
perceptions of problem tasks. The context is a second year course on
information systems within a Business university program. Central elements
are students’ conceptions and perceptions and their relation with student
Learning about Teaching Information Systems in PBL 231
attitudes, personality and learning style. Because problem tasks are the main
driving forces for learning in PBL this research concentrates on this aspect
of the learning environment.
The underlying research question is: “What is the link between students’
individual differences and their conceptions and perceptions of problem
tasks in a course in information systems?” The paper first outlines the
theoretical framework, followed by a description of the course in
information systems. Next the method of data gathering is presented.
Subsequently, the research findings are specified and interpreted. The paper
concludes with some implications for instruction and further research.
2. STUDENTS’ INDIVIDUAL DIFFERENCES AND

THEIR CONCEPTIONS AND PERCEPTIONS OF
PROBLEM TASKS
PBL is frequently advanced as a powerful instructional approach that is

engaging and leads to sustained and transferable learning (Bowden &
Marton, 1998; De Corte, 1990; Mergendoller, Bellisimo & Maxwell, 1999).
It is an instructional approach where learning is conceived as (a) an active,
constructive process of the learner; (b) contextualized; (c) initiated by a
cognitive stimulus; (d) a matter of extensive exercising in a variety of
contexts; (e) and a social process. In this respect, PBL learning environments
fit the constructivist principles of learning. This view of learning is
expressed in the instructional principles that guide the practice of learning
and teaching. The PBL students acquire knowledge and skills by handling
authentic problems in small tutorial groups, coached by a tutor. On the basis
of authentic problems, the students critically reflect on their prior knowledge
and, as a result, develop their own learning objectives. Authentic refers to
the cognitive demands of the problems tasks. The thinking required is
consistent with the cognitive demands of the environment for which we are
preparing the students (Savery & Duffy, 1995). By discussing with their
peers the information gathered with respect to the learning objectives, the
students elaborate their initial knowledge base. They test their ideas against
alternative views and provide and receive reflection on both the content
learned and the learning process.
Schmidt et al. (1995) conducted a study using structural equation
modeling to clarify the relations between different instructional variables of
a PBL environment and students’ outcomes. The model is based on the
students’ perceptions of the different PBL variables. The key question was:
“To what extent do the students’ perceptions of the quality of a variety of
PBL variables influence the students’ learning outcomes?” The variables

measured were the prior knowledge of the students and students’ perceptions
of the following variables: quality of the problems studied; tutor functioning;
tutorial-group functioning, self-study time, interest in subject matter and
learning outcomes. Students’ perceptions were measured by means of
student surveys. The variables were measured by a series of items. In figure
1 a graphical presentation of the model is given.
.19
.51 .11
Amount of prior knowledge Quality of problems Tutor performance
.26 .46 .27
.69
Group functioning E1
.22
.98
Time spent on
E2 Individual Study .82
.63
.58
.78 Interest in
E3 Achievement E4
Subject matter
Figure 1: The Influence of the PBL Environment on Learning Outcomes.
As indicated by the figures accompanying the arrows, the path

coefficients indicate that the direct influence of the PBL-environment
variables on learning outcomes is small. The findings suggest that the
influence of the different PBL environment variables is a complex one, with
the group functioning, the quality of the problems and the tutor performance
having the most influence on other variables in the model. The research
presented here elaborates on the variable problem tasks.
Prosser and Trigwell (1999) refer to a series of studies investigating the
relation between students’ conceptions of learning and their perceptions.
They advocate that individual characteristics account for differences in
students’ conceptions and perceptions. The present study intends to explore
some of these individual characteristics in the context of PBL and the
problem tasks as a core element. The research focuses on the relation of the
students’ conceptions and perceptions with some student characteristics,
specifically learning style, personality characteristics, commitment to PBL,

attitude towards information systems and prior knowledge. Attention is also
given to students’ perceptions of the problem tasks within the course on
information systems. The relations between the different variables are
schematized in figure 2.
Characteristics of students:
• Learning style, personality,
attitude to problem-based
learning
• Prior knowledge of business
Conception of Perception important

important task task elements in a
elements in a course course business
business informatics informatics
Figure 2: Student Variables Affecting Students’ Perceptions of Problem

Tasks.
2.1 Students’ Conceptions
Conceptions can be described as the possible way in which learning, the

learning environment and the subject matter studied can be construed or
experienced. They refer to the way a phenomenon is valued within a specific
context (Marton & Booth, 1997). To link the conceptions of important task
elements with the person related and content related factors six research
questions have been formulated. Relevant theoretical background supports
each research question:
1. How do PBL students conceive problem tasks as having a set of
characteristics in order to drive learning?
The learning principles of constructivism (Savery & Duffy, 1995; Poikela
& Poikela, 1997) and more general research on learning and cognition
(Koschmann et al., 1994; Schmidt, 1993; Regehr & Norman, 1996;
Boekaerts, 1997) have led to a set of principles for effective problem
tasks. In addition, the analyses of the nature of today’s problem situations
have led to some characteristics being identified (Stoy, 1999). They can
be summarized as follows: adapted to the students’ prior knowledge and
experiences; relevant to the future profession; integrated in a real-life
context; having a relevant level of complexity; stimulating for self-
directed formulation of learning goals; stimulating to search for
information; stimulating to elaborate; and sustaining discussions on
alternative views. The key question is: how do students value these
different characteristics of problem tasks.
2. How are students’ conceptions of important task elements related to their
learning styles?
Learning styles refer to students’ preferences for particular kinds of
learning activities. On the basis of a research review, Matthews (1991)
concludes that researchers tend to agree that persons with some learning
styles have a greater potential for success. Matthews argues that, among
other factors, one reason for this difference in performance is that
instruction more nearly matches the learning styles of those groups who
find success. Gentry & Helgsesen argue that: “when given a choice,
individuals prefer decisions situations and problem types that are
consistent with their own learning style” (1999, p. 61). They used
learning style information to improve the core financial management
course. Prosser & Trigwell (1999) suggest that interrelations between
students’ conceptions, students’ learning styles and students’ perceptions
of the learning environment may explain differences in learning
outcomes.
In the research presented in this paper, we used the Kolb concept of
learning style as described in Osland, Kolb and Rubin (2001). The
concept of learning style is associated with the successive stages in
experiential learning: concrete experience, reflective observation, abstract
conceptualization and active experimentation. Because the research
focuses on problem tasks, the Kolb experiential learning model and his
concept of learning style are considered as appropriate. Problem tasks
stimulate students for a continuous transaction with the environment, as
expressed in the problem task, on the basis of prior individual’s
experiences. Kolb uses two factors to describe the way persons perceive
new experiences and process experiences. The scores on these two
factors result in four basic learning styles: converging, diverging,
assimilating and accommodating. The convergent learning style relies
primarily on the dominant learning abilities of abstract conceptualization
and active experimentation. The greatest strength of this approach lies in
problem solving, decision-making and the practical application of ideas.
The divergent learning style has the opposite learning strengths from
convergence, emphasizing concrete experience and reflective
observation. The greatest strength lies in imaginative ability and

awareness of meaning and values. In assimilation, the dominant learning
abilities are abstract conceptualization and reflective observation. The
greatest strength of this orientation lies in inductive reasoning and the
ability to create theoretical models, in assimilating disparate observations
into an integrated explanation. The accommodative learning style has the
opposite strengths form assimilation, emphasizing concrete experience
and active experimentation. The greatest strength of this orientation lies
in doing things, in carrying out plans and tasks and getting involved in
new experiences. Because of differences in ways of learning, each with
its own preference for dealing with problems, conceptions of different
task elements could differ per learning style.
3. How are students’ conceptions of important task elements related to their
personality characteristics?
Prosser et al. (1994), as well as Biggs (1978, 1985), developed models of
student learning. Both indicate the interaction between personality
characteristics of the student, students’ conceptions and perceptions of
the learning environment, students’ approaches to learning and their
learning outcomes.
Within research, personality traits are measured in different ways, such as
the big five, locus of control and self-efficacy. The big five incorporates
five traits (Piedmont, 1998), based on Costa & McCrae, 1985): (i)
agreeableness refers to a person’s ability to get along with others; (ii)
conscientiousness refers to the order and precision a person imposes on
activities; (iii) emotional stability deals with the inclination to maintain a
balanced emotional state (e.g. calm, secure); (iv) extraversion indicates
the person’s comfort level with relationships; and (v) openness measures
a person’s rigidity of beliefs and range of interests. All these traits could
influence the students’ conceptions and perceptions of a problem task.
4. How is students’ commitment to PBL related to the conceptions of
important task elements?
Commitment refers to the way a person has internalized certain values
(Robbins, 2000). When students enter the University of Maastricht, they
seldom have any experience with PBL as a learning and teaching
approach. In the research reported, the students are second year students,
having experienced PBL for more than one year. As described by Prosser
and Trigwell (1999), these prior experiences may influence students’
conceptions of the learning environment to a large extend.
5. How is students’ attitude towards information systems related to their
conceptions of important task elements?
Attitudes can be described as a summation of emotions and feelings
experienced over time. They are quite stable with moderate intensity. In
contradiction with beliefs that have a high cognitive loading, attitudes

have an important affective component (Gal & Garfield, 1997). There has
been a vast amount of research in traditional educational settings on the
nature and correlates of students’ attitudes towards science (Helgeson,
1993). Also some research has been done in mathematics (McLeod,
1992). They all indicate a relation between students’ attitudes towards the
subject matter studied and student perceptions of the learning
environment.
6. How is students’ prior knowledge about information systems related to
their conceptions of important task elements?
It is known from ample research that prior knowledge plays a significant
role in students learning. On the basis of a meta-analysis on research on
prior knowledge, Dochy, Segers and Buehl (1999) conclude that
generally prior knowledge has a positive effect on students’ performance.
Additionally, they argue that further research is necessary to understand
the facilitating effect of prior knowledge in educational situations.
2.2 Students’ Perceptions
Perceptions refer to the way a student actually experiences his/her own

situation. Own situation in this case refers to the way the tasks have been
written. Similar to conceptions, the perception of the environment is
influenced by individual characteristics, such as personality traits, prior
knowledge, attitudes and learning style. Linking these aspects to the
conception of tasks leads to a set of six research questions. They have a
format similar to questions 1 to 6, so the same theoretical framework could
apply.
7. How do students perceive the problem tasks as implemented in a course
on information systems?
8. How is students’ learning style related to the perceptions of problem tasks
as implemented in a course on information systems?
9. How are personality characteristics related to the students’ perceptions of
problem tasks as implemented in a course on information systems?
10.How is commitment to PBL related to the students’ perceptions of
problem tasks as implemented in a course on information systems?
11.How is the attitude to information systems related to the students’
perceptions of the way problem tasks are implemented in a course on
information systems?
12.How is prior knowledge related to the students’ perceptions of the way
problem tasks are implemented in a course on information systems?
3. THE MAASTRICHT CASE: THE INFORMATION

SYSTEMS COURSE
The course on information systems is an obligatory course in the second

year of the International Business program at the Faculty of Economics and
Business Administration at the University of Maastricht (FDEWB). The
block deals with the role of information systems in an organization.
Specifically, the way in which the introduction of information systems
influences the structure and functioning of an organization. The technical
aspects of information systems will also be dealt with. Also, the possibilities
of new developments for organizations, such as enterprise resource planning
systems, the Internet and electronic commerce, will be discussed. After the
course students should have acquired insight in the possibilities modern
information systems can offer organizations as well as in the difficulties and
caveats inherent to the design, development and use of information systems.
When developing a course at FDEWB the following logistical constraints
are relevant. A course lasts for seven weeks. Groups of about 13 students
meet two times per week in sessions of two hours under the supervision of a
tutor and work on the tasks in the course book. Students are supposed to
spend 16 hours on self-study per week. During a course students also
attended a parallel course absorbing 20 hours per week. The seven weeks of
instruction is followed by one week reserved for testing. The course on
information systems contained about 24 problem tasks covering all the
relevant topics. In figure 3 an example of a task is given.
During each session two tasks are analyzed, resulting in learning goals
that are a guide for self-study. After the session students should study the
relevant literature at home. Regarding the literature the students had to use
either Turban et al. (1999), Information Technology for Management or
Laudon and Laudon (2000), Management Information Systems. This is in
line with PBL as it promotes using different sources of information.
Furthermore, there was a compulsory reader and sufficient copies of other
interesting books in the library. In the following session students report on
their findings and apply the literature to the task. The course coordinator
provided the tutors with a tutor guide with possible problem statements, a
short analysis of the task and references to the relevant literature. The tutor
guide is not given to the students. Assessment was done on the basis of a
mark for the test with 80 true/?/false questions, a mark for participation
during the meetings and a mark for a group paper about applications of
information systems.
Setting: Woodmac is a company that produces chain saws for woodcutting. At this moment the
company’s production facility and other departments are located in Aachen. The products are
normally sold to German customers.
Going international (problem task)
During a management team meeting of Woodmac the following conversation could be observed.
Marketing manager: “We found out that our products are also used in the Netherlands and in
Belgium. I think we should use this opportunity. We could use the global strategy ‘Domestic
exporter’ for the Netherlands. In Belgium we will open some sales locations.” The production
manager added: “To fulfill demand in Belgium we should build a production facility in Dinant.”
The general manager concluded: “ Seeing this viewpoints I think we should decentralize as much
as possible.” The information manager follows the discussion with interest. He states: “I expect
that our current information system is suitable to handle the export of products to the
Netherlands. Regarding the locations in Belgium I think we should process the information in
Aachen.”
Figure 3: Example of a Task.
4. THE DATA
To gather the necessary data for answering the research questions

different approaches were used. On the basis of a literature study a
questionnaire with 28 questions was developed to measure the relevant
aspects of task. To measure learning styles, traits, commitment to PBL and
attitude to information systems, frequently used and validated questionnaires
were used. Prior knowledge was measured by just one question in the course
evaluation. An overview of all the questionnaires used is given in table 1.
Except for the learning style questionnaire all questionnaires used a
Likert scale. Twelve out of 32 groups were selected, resulting in a
population of 149 students. At the beginning of the course students were
asked about personality, learning style, commitment to PBL and attitude to
information systems. In the last meeting the conception and perception of
important task elements were measured. Because of students’ absence in the
first or the last session, missing values and the stratified approach to reduce
the number of questions for the students, the actual number of cases for
correlation analysis is lower. To analyze the data the following techniques
were used: ANOVA, factor analysis and, for correlation calculation,
Spearman and Pearson.
Table 1: Scales and their Measurement.

Scale Instrument Description
Characteristics of Task Elements A questionnaire based on a literature study.
problems
Learning style Kolb (1976) A questionnaire with nine questions. Each
question has four possible answers that have to
be ranked. Based on the answers one of the four
learning styles is determined.
Personality Goldberg (2001) This is a 50 item questionnaire with the
subscales: extraversion, agreeableness,
conscientiousness, emotional stability,
openness
Commitment to PBL Mowday, Steers, A questionnaire with 9 selected items. The
Porter (1979) original questionnaire was used to measure
commitment to an organization. For the present
research the word PBL replaced organization.
Attitude to information Gal, Ginsburg, This is a 28 item questionnaire with the
systems Schau (1997) subscales: affection, cognitive competence,
value and difficulty.
Prior knowledge about Prior knowledge is measured by one question in
information systems the course evaluation.
We first studied which task elements are valued by the students as

important. Only four items scored between 2.5 and 3, the others scored
above 3. Indicating that students perceived almost all task aspects as
important. All items were distributed normally.
Table 2: Descriptives of the Sample.

Scale n of Average Standard Cronbach’s n
items deviation alpha
Personality characteristics
Agreeableness 10 36.2 6.55 0.83 129
Extraversion 10 39.5 4.62 0.77 131
Conscientiousness 10 35.0 5.95 0.77 130
Emotional stability 10 34.7 6.62 0.80 131
Openness 10 36.9 5.34 0.75 130
Commitment to PBL 9 36.1 8.82 0.84 51
Attitude to business informatics

Affect 6 28.1 6.09 0.82 50
Cognitive competence 6 28.3 5.79 0.78 50
Value 9 48.0 8.03 0.87 50
Difficulty 7 25.3 5.21 0.76 50
Prior knowledge 1 2.7 1.18 n.a. 105

The analysis of the students’ learning styles is based on 94 cases,

distributed as follows: diverging (51), assimilating (10), converging (15) and
accommodating (19). The other descriptives of the students’ characteristics
are shown in table 2.
5. THE RELATION BETWEEN STUDENTS’

INDIVIDUAL DIFFERENCES AND THEIR
CONCEPTIONS AND PERCEPTIONS OF
PROBLEM TASKS
The explorative study presented indicated students’ conceptions as well

as perceptions of the value of different problem tasks elements and the
relation between a set of individual characteristics and their conceptions and
perceptions.
5.1 Students’ Conceptions of Problem Tasks Elements
To come to a set of characteristics of tasks (research question 1) a factor

analysis was appropriate. The Kaiser Meyer-Olkin measure is 0.744. This
indicated that the data are between middling and meritorious for doing a
factor analysis (Kaiser & Rice, 1974, in Sharma, 1996, p. 116). The
Principal Component Analysis was used to extract factors, the varimax with
Kaiser Normalization to rotate factors, and the eigen value greater than one
criterion was used to determine the number of factors. This resulted in seven
factors see table 3.
Hair et al. (1998) provide guidelines for identifying significant factor
loadings based on sample size. If the sample size is 120 then a factor loading
of .50 is still significant. Because of explorative nature of the research
smaller loadings were also accepted. However, no factor loading was lower
than .43. We identified seven factors: motivation, guidance, solution
orientation, framing, literature evaluation, uncertainty avoidance and self-
direction. The reliability of the scales, in terms of the Cronbach alpha (Į) is
acceptable except for the last two factors, probably because of the low
number of items.
The results of the factor analysis as well as the mean score of the items
on scale (factor) level (sum of the item scores/number of items per scale)
indicate that the students conceive the motivating aspect and the aspect of
guidance as the two main task elements. Least important are the solution
orientation and uncertainty avoidance.
Table 3: Task elements resulting from the Factor Analysis.

Motivating (21% explained, Į .76)
Enough time to study the relevant literature. .74 .18 -.11 .14
Based on real life situations. .69 .32
Reflects problems of the professional practice. .67 .20 .23
Highly motivating. .61 .20 -.13 .12 .19 .21
Stimulates sufficiently self-study. .65 .32 .10 .12 .47
Guidance (12% explained, Į .76)
Clear readability of language used in the task .78 -.10 .24 .11
Enough time to report in the tutorial group. .49 .59 .10 -.15
Invites search for problem-statements. .28 .56 -.11 .15
Enough cues to formulate learning objectives. .52 .54 .14 .14 .26
Clear guidance to literature. .15 .50 -.36 .19
Literature can be used to discuss the problems .29 .43 .15 .18 -.29 .28
in the task.
Solution orientation (6% explained, Į .77)
The situation is familiar to you. .79 -.11 .13
Presence of concrete questions in the task. .78 .14 -.12 .13
Much guidance by the title of the task. -.10 .20 .66 .35 .25
Has one single solution. .13 .53 .47 .12 -.29
Strict prescription what to do. .43 .46 .12 -.17
Framing (6% explained, Į .62)
Clear linkage with other tasks. .12 .19 .66 .25 -.10
Embedded in a case-description. .13 .43 .18 .56 .12
Correct application of the seven-jump. .27 .13 .53 -.24 .28
Presence of the relevant terms in the task. .50 -.25 .52 .17 .25
Stimulates a good brainstorming. .17 .45 -.34 .48 .23
Literature evaluation (5% explained, Į .60)
Provokes extended search behaviour. .17 .70 -.10
Served to focus and summarize knowledge in .21 -.14 .14 .69 .19
the post discussion.
Stimulates a critical evaluation of the .21 .49 .17 .50 .18
literature.
Uncertainty avoidance (4% explained, Į .33)
A lot of detailed information. .15 -.12 .73
Emphasis on a narrow range of specific .12 .12 .16 .26 .65
subject objectives.
Self-directing (4% explained, Į .29)
Allows studying in different areas. .23 -.18 -.10 .80
Manageable by the students themselves. -.10 .49 .18 .59
5.2 The Relation Between Individual Differences and

Students’ Conceptions of Problem Task Elements
Five individual characteristics were taken into account: the students’

learning style, personality traits, commitment to PBL, attitude towards the
subject matter studied and prior knowledge.
The first relation refers to learning style and conception (research
question 2). Table 4 shows the scores of the different learning styles on the
task elements.
Table 4: Conception of Important Task Elements per Learning Style (n=93).

Task element diverging assimilating convergence accommodating Total
Motivating 4.22 (.55) 4.16 (.60) 3.90 (.65) 3.98 (.68) 4.13 (.60)
Guidance 4.23 (.46) 4.17 (.41) 4.25 (.69) 4.07 (.64) 4.19 (.51)
Solution orientation 3.01 (.77) 3.14 (.99) 2.74 (.71) 2.72 (.78) 2.94 (.80)
Framing 3.68 (.55) 3.73 (.72) 3.52 (.54) 3.46 (67) 3.63 (.60)
Literature 3.71 (.70) 3.74 (.58) 4.00 (.57) 3.54 (.81) 3.71 (.69)
evaluation
Uncertainty 3.09 (.80) 3.21 (.67) 3.25 (.68) 3.42 (.69) 3.19 (.75)
avoidance
Self directing 3.74 (.65) 3.77 (.62) 4.00 (.62) 3.84 (.82) 3.79 (.67)
Concerning students’ learning style a one-way ANOVA analysis

indicated that there were no significant differences in conceptions between
the different learning styles. This implies the way students conceive task
elements, such as motivating is not related to their learning style.
The influence of the other four characteristics is analyzed using Pearson
and Spearman correlations. The results are shown in table 5, in which some
significant correlations can be found.
Task Personality characteristics Commit Attitude to Information systems Prior
element ment to know-
PBL ledge
Extra Agree Conscien Emotional Open- Affection Cognitive Value Difficulty

version ableness tiousness stability ness competence
Motivating 0.19 -.08 0.19 0.01 -.06 0.01 0.33 0.18 0.10 0.22 -.11
Guidance 0.19 0.01 0.21* 0.07 0.04 0.36* 0.19 0.10 0.06 -.07 -.16
Solution 0.13 -.17 0.10 0.07 -.26** 0.16 -.13 -.05 -.06 -.21 -.18
orientation
Table 5: Conception of Important Task Elements.
Framing 0.03 -.14 0.17 0.03 -.02 0.40* 0.17 0.19 0.29 -.06 -.08
Literature -.03 0.07 0.17 0.16 -.00 0.06 0.07 0.23 0.17 0.04 -.08
evaluation
Learning about Teaching Information Systems in PBL
Uncertainty -.08 -.02 0.11 0.14 0.03 0.21 0.57** 0.61** 0.69** 0.43* 0.12
avoidance
Self directing 0.14 0.23* 0.03 0.27** 0.03 -.08 0.28 0.28 0.31 0.16 -.03
** correlation is significant at the 0.01 level

* correlation is significant at the 0.05 level
243
Regarding students’ personality traits (research question 3) there are four

significant relations. The first relation is between agreeableness and self-
direction. The more a student is able to get along with others, the more a task
should give opportunities to be managed by the students themselves. Second,
there is a significant correlation between conscientiousness and guidance.
The more a student values order and precision, the more he values guidance
as a task element. Thirdly, there is a relation between emotional stability and
self-direction. The more emotionally stable a student is, the more a task
should give opportunities to manage the learning process himself. The fourth
relation is between openness and solution orientation. This correlation is
negative, implying the more open students are concerning beliefs and
interests, the less important they conceive the solution orientation of problem
tasks. To conclude, although the correlations between personality and task
elements are rather low, there are some indications that some personality
characteristics do make a difference in the students’ conceptions of task
elements.
Beside personality traits, we explored the relation between students’
attitude and their conceptions. We distinguished between attitude towards
PBL as an instructional approach (research question 4) and attitude towards
the subject mater studies (research question 5). Concerning commitment to
PBL, as indicated by the correlation coefficients in table 5, the correlation
with task aspects is low to moderate. There are two significant correlations
regarding the commitment to PBL and tasks elements. First, the more PBL
orientated a student is the more important guidance is. Students who have a
positive attitude towards PBL indicate the necessity of guidance. Second, the
more PBL oriented the student is, the more framing he indicates to be
important.
Concerning the attitude towards the subject matter studied, the
correlation between the scales of the attitude to information systems varies
between low and high. Four correlations are rather high and significant.
These deal with the four subscales: affection, cognitive competence, value
and difficulty and the task element: uncertainty avoidance. This indicates
that the more students like, have a positive opinion about, prefer and expect
difficulties with information systems, the more certainty a task should give.
Finally, research question 6 dealt with the extent to which the students’
prior knowledge of the subject matter studied, as estimated by themselves, is
related to their conceptions of task elements. The Spearman coefficient
indicated that the students’ prior knowledge, as indicated by themselves,
does not relate to the task elements. From this it can be concluded that
different levels of prior knowledge do not result in different perceptions of
task elements.
5.3 Students’ Perceptions of the Problem Task Elements
Beside the students’ conceptions of the value of the task elements, we

asked them, after studying the course, how they had experienced the seven
task elements (research question 7). Using the items of problem task
conceptions questionnaire, students were asked to indicate how they had
experienced the various task elements in the course on information systems
they had studied. Subsequently, based on the factor analysis of the previous
section the scores per factor were calculated. Table 6 shows the results.
Table 6: Descriptives of the Perception Scales (n = 114).

Task element Average Standard deviation Cronbach’s alpha
Motivating 3.08 .57 .60
Guidance 3.42 .56 .65
Solution orientation 3.05 .54 .62
Framing 3.16 .62 .68
Literature evaluation 2.96 .74 .63
Uncertainty avoidance 2.98 .70 .36
Self-directing 2.98 .66 .09
The reliability of most factors is acceptable, except for uncertainty

avoidance and self-directing. The limited number of items per factor, namely
2, could be a reason for this. The scores on the scales are about 3, indicating
an average score quality of the tasks. The guidance offered by the problem
tasks is seen as the best task element.
5.4 The Relation Between Individual Differences and

Students’ Perceptions
Concerning students’ learning style and perceptions of tasks (research

question 8), we compared the averages for each task element per learning
style. These averages can be found in table 7.
Table 7: Perception of important task elements per learning style (n=93)

Task element diverging assimilating converging accommodating
Motivating 3.16 (.53) 2.97 (.66) 3.06 (.51) 2.95 (.62)
Guidance 3.44 (.53) 3.42 (.60) 3.28 (.69) 3.46 (.60)
Solution orientation 3.04 (.55) 3.12 (.67) 3.02 (.56) 3.02 (.42)
Framing 3.13 (.60) 3.06 (.73) 3.33 (.44) 3.27 (.68)
Literature evaluation 2.96 (.73) 3.10 (.71) 3.13 (.74) 2.78 (.84)
Uncertainty avoidance 2.88 (.66) 2.87 (.64) 3.20 (.82) 3.21 (.75)
Self-directing 3.05 (.64) 2.93 (.53) 3.20 (.54) 2.74 (.81)
246
Task Personality characteristics Commit Attitude to Information systems Prior

element ment to know
PBL ledge
Extra Agree Conscien Emotional Open- Affection Cognitive Value Difficulty

version ableness tiousness stability ness competence
Motivating 0.06 -.10 0.06 -.11 -.08 -.04 -.48** -.52** -.49 ** -.54** 0.05
Guidance 0.07 -.00 0.08 0.03 -.11 0.01 -.12 -.13 -.11 -.11 0.02
Solution 0.10 -.09 0.02 -.03 0.07 -.14 .-.04 0.01 -.03 -.08 0.26**
orientation
Table 8: Perception of Important Task Elements.
Framing 0.10 0.05 0.04 -.02 -.08 0.10 -.24 -.17 -.23 -.27 0.08
Literature -.02 0.01 0.04 0.01 -.25* -.07 -.33 -.32 -.30 -.33 0.21*
evaluation
Uncertainty -.01 -.04 -.09 0.12 -.05 -.13 -.07 0.04 -.07 -.09 0.13
avoidance
Self directing 0.12 -.00 0.12 -.08 -.09 0.04 -.38* -.40* -.44** -.33 0.11
** correlation is significant at the 0.01 level

* correlation is significant at the 0.05 level
Jan Nijhuis, Mien Segers & Wim Gijselaers
The results of the ANOVA analysis indicate that there are no differences
in the scores per learning style per task element. This implies that students’
perceptions of task elements, such as motivating and guidance are not related
to their learning style.
For personality traits (research question 9), table 8 shows that the
correlation with the students’ perceptions of task elements is very low to
low. There is one significant relation, this is between openness and literature
evaluation. The more open a student is the lower a task scores on literature
evaluation. So, students who are more open to different beliefs and a variety
of interests, experience the problems tasks of the course as less inviting for
evaluation of the literature than those who score lower on the openness
personality scale.
Exploring the relation between students’ perceptions and their attitude
towards PBL (research question 10) and towards the subject matter studied
(research question 11), the results do not indicate clear evidence.
Commitment to PBL is very low to low correlated to the perceptions of
problem tasks. None of the relations is significant. The commitment to PBL
seems to have no influence on perception of problem tasks. For the relation
between attitude towards the discipline of information systems and the
students’ perceptions, almost all correlations are negative. Two task
elements have significant correlations. The task element, motivating, is
negatively correlated to all the attitude scales. This indicates that the more
affection, the more cognitive, the more value and the more difficult
information systems is, the less motivating the tasks were. The task element,
self-directing, is negatively correlated with three attitude scales. Thus, if
students who are explicitly positive towards information systems have higher
expectations they may be more disappointed afterwards.
Finally, we explored the relation between students’ estimated prior
knowledge of the subject matter studied and their perceptions of the task
elements (research question 12). The correlation coefficients indicate that
prior knowledge is very low to low correlated to the perception of the seven
task elements. Two task elements are correlating significantly with prior
knowledge. The first element is solution orientation, indicating that students
with more prior knowledge perceive the tasks as more solution oriented. The
second element is literature evaluation, indicating that students with more
prior knowledge perceive the tasks as more suitable for evaluation of the
literature.
6. IMPLICATIONS FOR INSTRUCTION AND

FURTHER RESEARCH
Seven important task elements could be distinguished which are in line

with the literature. According to the students, motivating and guidance are
the two most important elements. When writing tasks staff should take into
account study time; real life situations; professional practice; language used;
and the relation between literature and the problems. Regarding the
characteristics of the students, it can be concluded that learning style and
personality seem to have no relation to both conception and perception of
important task elements. This could indicate that PBL does not favor a
special learning style or personality of a student. Students’ attitude to
information systems is related to the conception of the task element,
uncertainty avoidance. Although students have a positive attitude to the
subject they still want to have less uncertainty. However, uncertainty
avoidance is not related to personality. Students’ attitude to information
systems is negatively related to the perception of the task elements,
motivating and self-directing. Students who have a negative attitude to
information systems see the tasks as more motivating and more self-
directing. This could be one of the goals of tasks, namely helping students
who need extrinsic motivation. Students with a positive attitude already have
an intrinsic motivation and see the tasks not as motivating.
The research was limited to uni-variate analyses; more advanced analyses
could give more insight in the perception of tasks (e.g. using the conceptions
of tasks to explain the perceptions of tasks). Tasks were considered as one
group in this research. However, tasks differ individually. An analysis at the
individual task level would be useful for more insight into task construction.
The correlations gave indication that the content of a course matters in
problem construction. Repeating this research in courses with another
content could give more insight into the relevance of the course content. In
this way more insight is obtained into the construction of task. Furthermore,
other factors in PBL like the functioning of the tutor and the productivity of
the group work could be related to students’ characteristics.
REFERENCES
Albanese, M.A., & Mitchell, S. (1993). Problem-based learning: A review of literature on its
outcomes and implementation issues. Academic Medicine, 1, 52-80.
Biggs, J.B. (1978). Individual and group differences in study processes, British Journal of
Educational Psychology, 48, 266-279.
Biggs, J.B. (1985). Student approaches to learning and studying. Hawthorn, Victoria:
Australian Council for Educational Research.
Birenbaum, M. (2000). New insights into learning and teaching and the implications for
assessment. The future is now – Are we ready to meet the challenges. Key-note Address at
the Assessment 2000 Conference of the EARLI SIG on Assessment and Evaluation, the
Netherlands, Sept.
Birenbaum, M., & Dochy, F. (1996). Alternatives in assessment of achievements, learning
processes and prior knowledge. Boston: Kluwer Academic.
Boekaerts, M. (1997). Self-regulated learning: A new concept embraced by researchers,
policy makers, educators, teachers and students. Learning and Instruction, 7 (2), 161-186.
Bowden, J., & Marton, F. (1998). The university of learning. London: Kogan Page.
Costa, P.T. Jr., & McCrae, R.R. (1985). Concurrent validation after 20 years: The
implications of personality stability for assessment. In J.N. Butcher, & C.D. Spielberger
(Eds.), Advances in personality assessment (pp. 31-54). Hillsdale, NJ: Erlbaum.
Coulson, R.L., & Osbourne, C.E. (1984). Insuring curricular content in a student-directed
problem-based learning program. In H.G. Schmidt & M.L. de Volder (Eds.) Tutorial in
problem-based learning. A new direction in teaching the health professions (pp. 225-229).
Assen: Van Gorcum.
Davey, B. & Tatnall, A. (1994). Problem solving and Management Information Systems. In
M. Ostwald & A. Kingsland (Eds.) Research and development in problem based learning.
Newcastle: Australian Problem Based Learning Network.
De Corte, E. (1990). A state-of-the-art of research on learning and teaching. Keynote lecture
presented at the 1st European Conference on the First Year Experience in Higher Educa-
tion, Aalborg University, Denmark, April.
Dochy, F., Segers, M., & Buehl, M (1999). The relation between assessment practices and
outcomes of studies: The case of research on prior knowledge. Review of Educational
Research, 69 (2), 145-186.
Dolmans, D.H.J.M., Gijselaers, W.H., & Schmidt, H.G. (1993). Course content coverage as a
measure of instructional quality, Evaluation and Health Professions, 16 (4), 448-471.
Dolmans, D.H.J.M., Snellen-Balendong, H., Wolfhagen, I.H.A.P., & van der Vleuten, C.P.M.,
(1997). Seven principles of effective case design for a problem-based curriculum, Medical
Teacher, 19 (3), 185-189.
Gal, I., & Garfield, J. (1997). The assessment challenge in statistics education. Amsterdam:
IOS Press.
Gal, I., Ginsburg, L., Schau, C., (1997). Monitoring attitudes and beliefs in statistics
education. In I. Gal & J.B. Garfield, The assessment challenge in statistics education (pp.
37-51). Amsterdam: IOS Press.
Gentry, J.A., & Helgesen, M.G., (1999). Using learning style information to improve the core
financial management course, Financial Practise & Education, 9 (1), 59-70.
Gijselaers, W.H., Schmidt, H.G., (1990). The development and evaluation of a causal model
of problem-based learning. In Z. Nooman, H.G. Schmidt & E. Ezzat (Eds.) Innovation in
medical education: An evaluation of its present status (pp. 95-113). New York: Springer.
Goldberg, L.R., (2001), Possible questionnaire format for administering the 50 big-five factor
markers. [WWW document]. [On-line] (Accessed 14/08/02), URL:
http://ipip.ori.org/ipip/qform50.htm
Hair, J.F., Anderson, R.E., Tatham, R.L., & Black, W.C. (1998). Multivariate data analysis
(5th ed). Upper Saddle River: Prentice Hall.
Helgeson, S.L. (1993). Assessment of science teaching and learning outcomes. Columbus,
Oh: National Center for Science Teaching and Learning.
Jurema, A.C., Costa-Lima, M.E., Dalmau, M.C., & Jurema, M. (1995). Towards a pedagogy
of informatics. Paper presented at the 1st International Conference on Computer Support
for Collaborative Learning, Bloomington, Indiana, October.
Kaiser, H.F., & Rice, J. (1974). Little jiffy mark IV, Educational and psychological
measurement, 34, 111-117.
Kolb, D.A. (1976). Learning style inventory. Boston: McBer & Co.
Koschmann, T.D., Myers, A.C., Feltovich, P.J, & Barrows, H.S. (1994). Using technology to
assist in realizing effective learning and instruction: A principled approach to the use of
computers in collaborative learning, The Journal of Learning Sciences, 3 (3), 227-264.
Laudon, K.C., & Laudon, J.P., (2000). Management information systems (6th ed.). London:
Prentice Hall.
Ma, I. (1994). Problem based learning and students’ approaches in information systems. In M.
Ostwald & A. Kingsland, Research and development in problem based learning (pp. 101-
110). Newcastle: Australian Problem Based Learning Network.
Marton, F., & Booth, S. (1997). Learning and awareness. Mahwah, NJ: Lawrence Erlbaum.
Matthews, D.B. (1991). The effects of learning style on grades of first-year college students,
Research in Higher Education, 32 (3), 253-268.
McLeod, D.B. (1992). Research on affect in mathematics education: A reconceptualization. In
D.A. Grouws (Ed.) Handbook of research on mathematics teaching and learning (pp. 575-
596). New York: Macmillan
Mergendoller, J.R., Bellisimo, Y., & Maxwell, N.L. (1999). Content knowledge and student
characteristics: A comparative study of the impact of problem-based learning and
traditional instructional strategies in high school economics with different types of
students. Paper presented at the 1999 American Educational Research Association Annual
Meeting, Montreal, Canada, April.
Moust, J.H.C., Bouhuijs, P.A.J., & Schmidt, H.G. (1989). Probleemgestuurd onderwijs: Een
wegwijzer voor studenten. Groningen: Wolters-Noordhoff.
Mowday, R., Steers, R., Porter, L., (1979). The measurement of organizational commitment,
Journal of Vocational Behavior, 14, 224-247.
Norman, G.R., & Schmidt, H.G. (2000). Effectiveness of problem-based learning curricula:
Theory, practice and paper darts. Medical Education, 34, 721-728.
Osland, J.S., Kolb, D.A., & Rubin, I.M., (2001). Organizational behavior (7th ed.). Upper
Saddle River: Prentice Hall.
Penjam, J. (1997). On conception of informatics education. [WWW Document] [On-line]
(Accessed 14/08/02) URL: http:// greta.cs.ioc.ee/~jaan/papers/riia3.html
Piedmont R.L. (1998). The revised neo personality inventory. New York: Plenum Press.
Poikela, E., & Poikela, S. (1997). Conceptions of learning and knowledge: Impacts on the
implementation of problem-based learning. Zeitschrift fur Hochschuldidactik, 21 (1), 8–
21.
Prosser, M., Trigwell, K., Hazel, E., & Gallagher, P. (1994). Students’ experiences of
teaching and learning at the topic level, Research and Development in Higher Education,
16, 305-310.
Prosser, M. & Trigwell, K. (1999). Understanding learning and teaching: The experience in
higher education. Buckingham: The Society for Research into Higher Education & Open
University Press.
Regehr, G., & Norman, G.R. (1996). Issues in cognitive psychology: Implications for
professional education. Academic Medicine, 71 (9), 988-1001.
Robbins, S.P. (2000). Essentials of organizational behaviour. Upper Saddle River: Prentice
Hall.
Savery, J.R., & Duffy, T.M. (1995). Problem based learning: An instructional model and its
constructivist framework. Educational Technology, Sept - Oct, 31 - 38.
Schahabudi, S.H. (1987). Content coverage in problem-based learning, Medical Education,
21, 231-313.
Schmidt, H.G. (1993). Foundations of problem-based learning: Some explanatory notes.
Medical Education, 27, 422-432.
Schmidt, H.G., Patel, V.L., Gijselaers, W.H., Nuy, H., & de Grave, W.S. (1985). Dimensions
of problems for problem-based learning. Unpublished paper, Universiteit Maastricht.
Sharma, S. (1996). Applied multivariate techniques. New York: Wiley & Sons.
Stoy, D. (1999). The link between entrepreneurial success and advanced skills in organisation
development. In J. Hommes, P. Keizer, M. Pettigrew and J. Troy (Eds.). Educational
Innovation in Economics and Business IV (pp. 229-238). Dordrecht: Kluwer Academic
Publishers.
Todd, R. (1999). Reading the future: Knowledge management 4: Towards a community of
learning. Scan, 18 (3), 34-37.
Turban, E. Mclean E., & Wetherbe, J., (1999). Information Technology for Management (2nd
Ed.), New York: John Wiley & Sons.
PART V
DISTANCE AND ON LINE LEARNING

The Use of a Virtual Learning Environment to
Support Learners on Work-Based Learning Programs
Len Bird
Work-based Learning Unit, Coventry Business School, Coventry University, UK
1. INTRODUCTION
Historically, work-based learning programs were not delivered with on-

line support in mind. Boud, Solomon and Symes (2001) give a potted history
of work-based learning that starts with work placements and sandwich
courses in the 1960s; leads on to independent study programs and negotiated
learning experiences of the 1970s; and culminates in the enterprise
movement of the 1990s, with its focus on generic skills and the
competencies of “graduateness”. None of these antecedents used technology
in any meaningful way.
Previous research (Laurillard, 1993 & 1995; Conole & Oliver, 1998;
McConnell, 2000) on integration of technology into the curriculum indicates
that it requires a complete rethink from first principles in course design and
delivery. The author has been working on the pedagogical design of work-
based learning courses at Coventry University for the last two years and the
broad focus of this paper is on the use of on-line support for students on a
Postgraduate Diploma in Management (PgD) by work-based learning run at
Coventry Business School, for which the author is the course tutor.
1.1 The Postgraduate Diploma in Management
The PgD has the classical features of a work-based program. It has a

negotiated curriculum, autonomous learning, a focus on experiential and
255
256 Len Bird
reflective processes and is designed to develop transferable cognitive and

lifelong learning skills. The course can be represented by figure 1.
Consultancy Module
Independent Study
Module
Action Learning Sets
Figure 1: The Postgraduate Diploma in Management – Course Structure.
At the heart of the course is a double module of Independent Study.

Alongside this double module is a compulsory Consultancy module in which
students carry out a group assignment in a real business organization. These
three modules are encapsulated within a framework of action learning in
which students meet monthly in action learning sets to support each other in
their projects and assignments. The attendance pattern and student support
involves:
• two 2-day Residentials;
• monthly action learning set meetings (ALSs); and
• on-line support via WebCT
The monthly ALSs provide an opportunity for emotional support, a
forum for social interaction with peers and an arena for public reflection.
The power of action learning is well documented (Pedler, 1991; McGill &
Beaty, 1995). In the context of the PgD, action learning provides the
backbone of the learning and the function and process of the set is at the
heart of the on-line WebCT support described later.
As O’Hara et al. (1997) put it, action learning provides a way of
integrating the personal learning of the individual with learning about an
aspect of the world that is perceived by them as important. At the ALSs
members address these real issues, take part in public reflection, are exposed
to new insights and perspectives through questioning and discussion and
then plan new actions as a consequence. The process results in the social
construction of knowledge as students modify their cognitive maps
surrounding the issues involved (Vygotsky, 1978). Students present their
situations in a personal way and then the ALS participants’ question and
challenge the presenter, so opening up new understandings that are
accommodated by the presenter into a new realization of the situation. The
Virtual Learning Environment 257
process is facilitated by a tutor in the face-to-face sessions and continued by

an on-line e-moderator (Salmon, 2000) in the weeks between meetings.
Central to the delivery of the course is the use of the integrated virtual
learning environment (VLE) WebCT. This tool, which supports all 2000+
modules at Coventry University, provides a platform on which the on-line
support for learners is facilitated (Deepwell & Syson, 1999). WebCT
provides all the normal features of a VLE, such as e-mail, bulletin boards,
chat rooms, contents areas, web portals, electronic library, on-line
assessments and student marks. However, arguably the most valuable feature
of WebCT, which is central to the support given to the participants on the
PgD, is the asynchronous bulletin board, which facilitates computer-
mediated conferencing (CMC). The asynchronous conferencing allows the
action learning process, in operation at the monthly face-to-face ALSs, to be
continued in the weeks between meetings.
The continuation of the face-to-face process within the on-line
environment requires careful management by the e-moderator. Part of the
focus of this paper is an analysis of the way students engage within the
conferencing facility and how the student participation in the on-line action
learning process can be improved.
2. METHODOLOGY
The work has been carried out within an action research framework (Carr
& Kemmis, 1986; McNiff, 1988; Elliot, 1996). The course tutor has taken a
stance as an “insider researcher” and endeavored to collaborate with the
learners on the PgD in an effort to improve the student experience. The work
has centered on a change intervention, the introduction of an on-line learning
environment for work-based learning, which has been observed and
examined for its impact. Several cyclical iterations of the research have been
planned. This paper presents the results of one of them.
Firstly, the content of the conference messages on the bulletin board were
analyzed. The WebCT bulletin board stores every message posted and
allows a printed compilation of every piece of information entered into the
system. The analysis concentrated on understanding the nature of the on-line
communication and the purpose the sender intended for the message. Having
reviewed the vast array of methodologies available for analyzing
communication patterns (Holsti, 1968), it was decided that, at this stage, a
simple content analysis was appropriate. The contributions to the PgD CMC
were printed out and then “contribution clusters” were used to group the
messages. A contribution cluster was defined as a theme within the messages
that signified a meaning and purpose intended for the contribution.
258 Len Bird
To triangulate the data focus groups and in-depth interviews were carried
out to improve understanding of the participants’ experiences and intentions.
The student cohort consisted of 15 students who all had a first degree and at
least five years of managerial experience. Two focus group sessions were
held involving five or six students and four preliminary interviews were
conducted.
As Morgan (1988) points out, focus groups can provide a rich source of
qualitative data. For analysis purposes I made a TV video recording of the
focus groups and further recorded participant’s contributions by using rich
pictures. These involved the focus group participants making a pen and
paper drawing on flip chart paper that represented their on-line experience
on the PgD. Group members then spoke to these drawings, elaborating on
their intended messages.
The focus groups and in-depth interviews allowed further interrogation of
the issues that emerged from the content analysis of the CMC postings.
Gradually a picture emerged of why and how participants contributed to the
bulletin board and what learning process was in action.
3. DISCUSSION OF RESULTS
3.1 The 3-S Model
A rather simple framework is proposed within which to consider a wide

range of possible contributor types to asynchronous on-line conferences. The
purpose is to allow course designers and conference moderators the
opportunity to categorize their students and consider ways of improving the
quality of contributions and hence facilitate deep learning. Of course models
containing categories are criticized by some educationalists who
immediately cite instances which do not fit into the model. Nevertheless, a
debate concerning the merits and contradictions of competing interpretations
and models broadens understanding and helps build academic theory. It is
useful therefore to begin to understand the CMC process from the
perspective investigated on the PgD if only to allow others to consider the
proposal and redefine the model in their own terms. The results can be
summarized by the 3-S Model given in table 1.
The model identifies three types of CMC contributor associated with the
PgD asynchronous conferences: strong members, supportive members and
strategic visitors. Strong members love the on-line environment and are very
comfortable in the text-based environment. They tend to be thoughtful and
reflective with a good command of the underpinning course content. They
are confident without being dominant. Their ability to express themselves in

writing is clearly demonstrated and forms a key skill that adds to their
enjoyment of the conferencing. Supportive members, on the other hand, are
more conservative in their contributions but have a strong motivation to
support the group. They make fewer contributions containing firm opinions,
but tend to support the community with engagement about social issues and
matters of course procedure. The third group, strategic visitors, are
instrumental, concerned only to pass the assessment. As this requires some
contribution to the on-line conferences, they do just enough to pass.
However, they do have a conscience and they need to be accepted by their
peers as bona fide participants on the course and not be regarded as “free
riders”. In summary, the 3-S model is intended to map out a generic
classification of contributors that provides a framework for devising
strategies to convert students to the “strong” category. Examples of data
from each category are given below.
Table 1: The 3-S Model.

Type of CMC Member Characteristics
Strong Member Contributions are frequent, influential and authoritative. They
regularly start debates and respond to further contributions
quickly. When joining debates started by others they make strong
contributions and stimulate further replies.
The enjoyment of the medium is a key motivator. They have
good language writing skills.
Supportive Member Contributions are less frequent and tend to be more concerned
with social issues and focussed on supporting and encouraging
others. They want to be involved but feel less confident about
making positive statements, which may expose their own self-
doubt.
Being part of the on-line community is important to these
members.
Strategic Visitor Contributions are rare and infrequent but sufficient to stay in
touch with the course and maintain credibility as a course
participant. Assessment matters are the main drivers and
motivators for these students. However the need to belong to the
community and not to be rejected by their fellow learners is also
important.
3.1.1 Strong Members
The type of contribution made by strong members is typified by the

following extract.
260 Len Bird
“Dear all,
Glad to see we are all talking at last. Quite a relief really. I’m afraid I was
late to the action learning set meeting on Friday. Ben took me through the
substance of the discussion and I agree with the outcome.
Unfortunately, I have a planning committee on the 8th and will not be

able to attend that meeting. I will, however, be available for the 15th etc.
To give an idea of what I’ve been up to I attach the first part of the
questionnaire, identifying the preferences of those who completed the
survey form. I am in the process of devising a database to link some of
the more interesting information to an analysis of both age and ethnic
origin, where it can be identified. I will post when it is ready.
Chris, I’ll have to pick out Tony’s e-mail address and send it onto you
direct. I’ll do that in a couple of minutes.
Laurie, I’ve asked Alison to give me a date for us to meet. Will chase.
Have you had any joy with West Midlands Arts?
Bill”
This posting was made after a short lapse in communication. The sender
acknowledges the situation and then goes on to encourage a resumption of
activities. He leads by example giving details of work he is doing and
finishes the message by prompting others into action. Bill has attached work
for scrutiny in a confident way, inviting others to comment. He has no
problem with disclosing his work for scrutiny. A strong and confident
approach to the conference is portrayed in an open manner.
A feel for the authoritative nature of the postings by strong members can
be gained from the following extract.
“James,
I have come up with a structure for the report and will post it tomorrow. I
will put each person’s name associated with the different sections of the
report as I see it so far. I am a little concerned that only Paul has
produced any text of any kind so far so I am up in the air as to what
Laurie, Kim and Bill are doing at the moment.
I am happy to collate all sections completed and put them together as an

overall report.
We need a final brainstorming session for everyone to come up with

some ideas for the centre. I would like to see each person in the group
come up with a list of suggestions for improvements in light of the
evidence collated and submit it on the web. This is the real meat of the
report and is what consultancy is all about.
CU Jim”
Jim is attempting to act as the chair of the group for the purposes of
compiling the consultancy report for the PgD. He is showing signs of
frustration with the lack of effort from the other members, nevertheless he is
attempting to lead the group by volunteering to edit the final document. He
shows authority and the ability to stir others into action. By mapping out a
way forward (i.e. brainstorming and posting suggestions for improvements)
he is setting out an agenda in an authoritative way and also probing the
understanding of the consultancy process of other group members.
Although the text-based medium of the conferencing facility lacks the
physical and social cues that are inherent in face-to-face communications
(e.g. eye contact, gestures, body language, tone of voice, see Harris, 1999)
the attitude and intentions of Bill and Jim are clear from the transcripts
above. Their frequent contributions of an influential and authoritative nature
stimulate others and move debates and activities along. Their obvious
excitement and enjoyment of the medium is apparent, as is the power that
they extract from the status they enjoy.
3.1.2 Supportive Members
Being part of the on-line community is important to supportive members.

They find ways to stimulate discussion but in a more gentle, less
authoritative way.
“I like your consultancy model Steve. Perhaps we could discuss it further
CU soon.”
“You have produced an excellent draft project brief. I am hoping to be

able to post some comments soon – hope others can do the same.”
“I would like to meet and discuss with Alison. Have we decided on a date
and venue? Any night is Ok 4 me, I could make daytime if that helps.”
“Has anyone heard from James? Are you still out there?”
262 Len Bird
“I think it is important that we have regular contact otherwise we are not

going to meet deadlines and our cred with the NAC will die. Has anyone
got a venue for the next meet?”
Supportive members tend to “lurk” (i.e. read messages without replying
or posting new ones) more than strong members and their contributions are
less frequent, but they do provide an important “lubricant” for the
conference by easing people back into the discussion. They want to be
involved but do not take the lead by posting examples of work done or texts
for discussion. Their terminology is direct and their invitations to participate
explicit.
The interviews revealed that supportive members hold back from posting
messages mostly due to the fear of exposing their ignorance or lack of
comprehension of the issues. The permanent nature of the contributions on
the bulletin board is seen as a threat, as errors and mistakes are framed and
recorded for all to see and refer to.
3.1.3 Strategic Visitors
Strategic visitors post messages infrequently and rarely contribute

meaningfully to debates. Their contributions include many examples of
excuses for non-participation.
“Ben, I am now fully able to access the Coventry web site and use it as
intended. I have spent a lot of time catching up on E-mails and looking
through attachments etc. I have also found the resources section and have
begun downloading and digesting materials. I hope, therefore, that this will
enable me to reach full speed by the end of Jan. and get into some serious
study. See you this p.m. Thanks James.”
“To all in my set. I apologize (again) for not contributing over the last
couple of weeks. I have however done the work that I should have by the
22nd of Feb – i.e. Get together and collate the academic work on the subject
of consultancy. I am still in and have been working.”
“Steve, sorry I have not been in touch but I have been busy at work – I
got your message re last week’s meet. I have a meet with our CEO today and
have to pick my daughter up from school as my wife is ill – I will therefore
not be able to attend this week’s meet but will be in touch on the web. I have
made progress on the staff issues but have not spoken to the arts centre at all
yet. CU soon”.
The excuses could all be genuine and low participation could be a matter
of bad luck and circumstances, but interview data from students suggests
that this is not the case.
On the PgD the ALS members meet face-to-face and virtually so that the
group dynamic has a mixture of characteristics of both formats. The flesh
and blood human identity mixes with the virtual identity created on the web.
Strategic visitors feel they owe a level of contribution to the group and need
to “pull their weight” to be able to establish and maintain a status and
identity in the face-to-face meetings. They cannot hide totally behind their
on-line persona. They need to “belong” to the community and do not wish to
be rejected by their fellow learners. This mixture of face-to-face and virtual
interactions is used as a key component of the strategy for improving
conference participation developed below.
It would be wrong to say that strategic visitors are the only category of
contributor clearly focused on assessment requirements. Assessments, above
all motivators, determine the nature of the learning context and drive the
performance of student activities (Biggs, 1999). As Savin-Baden (2000)
argues in her treatment of the disjunction of learning, the design of
assessments can pressure students to adopt learning approaches that give
high grades, rather than those that are in their own best interests as
individuals or as group members. Strong and supportive members seem to
resist this pressure and see the “added value” to be gained from
conferencing. Strategic visitors do not. They seem to feel that they can skip
in and out of the CMC process with no detriment to their personal learning
or their final course grading.
4. STRATEGY FOR IMPROVING PARTICIPATION
Salmon (2000), McConnell (2000) and Beaudin (1999) have all

published recipes for improving the general quality of CMC contributions.
Salmon focused on the five stages of learner activity through which
contributors pass in their route to becoming sophisticated on-line learners
and gives excellent advice and guidance on the training needs of the e-
moderators controlling the conference. McConnell takes a more holistic
approach to the overall design of computer supported co-operative learning
and produces a checklist of what a tutor should do to create a supportive
learning environment. Beaudin identified the four key mechanisms on-line
tutors used to keep asynchronous discussions on topic. However, none of
these address the identity or character that contributors generate through a
mixture of face-to-face and on-line communication. On the PgD the
combination of on-line interactions and face-to-face ALSs, in a work-based
learning context, gives a new dimension to the on-line mediation task. Co-
presence is part of the scenario. Learner relationships are a mixture of
normal social interactions and virtual exchanges. This has implications for
264 Len Bird
the nature of the learning community created and the power of the facilitator
to make interventions both virtually and face-to-face.
The 3-S model can be used to categorize students and then target them
appropriately both virtually and in the ALSs. Strategies could involve:
• taking the fear out of the assessment process for strategic visitors;
• fostering a co-operative atmosphere for all members, but particularly for
supportive members, in which helping others is the norm and being
wrong is OK;
• empowering learners to self-manage their own learning environment and
thus confront strategic visitors;
• converting strong members to e-moderators;
• giving confidence to supportive members through positive feedback;
• rewarding supportive members who take risks through praise and public
acknowledgement;
• explaining the value of on-line conferencing to strategic visitors; and,
most importantly,
• prompting activities that transport the supportive nature of the face-to-
face ALS process to the on-line conference.
The set facilitators have the opportunity to implement the strategic
actions above in a subtle way. The facilitator’s role is “custodian of the
process” rather than the initiator of the content of the discussions. However,
through careful questioning and challenging of each set member’s position
and attitude the facilitator can open up and explore the issues above as each
presenter takes centre stage. Preparation and classification of the set
members within the 3-S model prior to the meeting can give the facilitator a
guide agenda for each participant. This process can give structure and
purpose to the facilitator’s on-line and face-to-face interactions with the
group. In summary, the strategy is no more than a guide, but it does provide
set facilitators/e-moderators with a framework for addressing the obvious
difference in contribution levels of participants which is observable and
present in all on-line conferences.
5. DISCUSSION OF THE LEARNING PROCESS
Vygotsky’s (1978) social constructivist approach to learning has already

been mentioned as the key learning process in the ALSs. But how do
students learn from the overall experience of the course? The theory of
“situated learning” and the concept of “communities of practice” may offer
an answer.
Lave and Wenger (1991) introduce the concepts of situated learning and
communities of practice. In a radical critique of the conception of learning,
they argue that most accounts of learning ignore the basic social nature of
the learning process. They propose that learning is a process of participation
in communities of practice, with the participation at first legitimately
peripheral but gradually increasing in engagement and complexity. Their
analysis has wide implications for course design.
Lave and Wenger (1999) provide an analysis of situated learning in five
different settings: Yucatuc midwives, native tailors, navy quartermasters,
meat cutters and alcoholics. In all cases there was a gradual acquisition of
knowledge and skills as novices learned from experts in the context of
everyday activities. There was little observable teaching. The more basic
process was learning by engagement with the community. From their
research they define a community of practice as a group of practitioners who
jointly hold a socially constructed view of the meaning of their subject
knowledge and what it takes to be an expert in the field. The term
“community” does not necessarily imply co-presence but suggests a group
that participates in an activity system about which members share
understandings concerning what they are doing and how it should be done.
This model of learning emphasizes the inherently socially negotiated quality
of meaning and claims that learning and knowing are embedded in the
relations and interactions of people engaged in activities within their socially
and culturally structured world. Learning becomes a social practice in a
world of jointly constructed meaning and knowledge.
The model begins with newcomers to the community being granted
legitimate peripheral participation as a means of both absorbing and being
absorbed in the culture of the practice. From this peripheral perspective
learners gradually construct a general idea of what constitutes the practice of
the community. Legitimate peripheral participation is the core of the learning
that takes place. This peripheral activity gives rise to a learning curriculum
that is a series of learning experiences available to the peripheral learner
within the community of practice. The learning curriculum is thus “situated”,
in that it cannot be considered in isolation or analyzed apart from the social
relations that shape legitimate peripheral participation in the community.
“Newcomers become old timers through a social process of increasingly
centripetal participation which depends on legitimate access to ongoing
community practice. Newcomers develop a changing understanding of
practice over time from improvised opportunities to participate peripherally
in ongoing activities of the community. Knowledgeable skill is encompassed
in the process of assuming an identity as a practitioner, of becoming a full
participant, an old timer.” (Lave, 1999, p. 68)
The model sees learning as a transformation of newcomer to old timer
rather than a process of transmission of skills from tutor to pupil. In addition,
it alters the normal Piagean conceptualization of learning from something
266 Len Bird
that takes place in the individual mind, as an accommodation of new mental

models and structures, to something that takes place in a participation
framework. Learning is, within this model, distributed among co-
participants, not a one-person act. Social constructivism (Vygotsky, 1978),
however, is the key process at work.
Social constructivism broadens Piaget’s view of individuals constructing
knowledge and poses the question: “Is the ‘mind’ located in the head or in
social action?” (Cobb, 1994, p. 13). If the mind resides in our heads then we
can think of learning as “acquiring” knowledge; if the mind is influenced
and shaped by society, social action and social responses, as in a community
of practice, then we can think of learning as participating in knowledge held
jointly by the community.
Lave and Wenger make the point that if novices do not learn directly
from the expert but from peers and the rest of the community then
knowledge can be considered not to reside in an individual but to reside
within the community as a whole. To quote: “Mastery resides not with the
master but in the organization of the community of practice of which the
master is a part.” (1999, p. 23) This decentering of the learning process
moves the focus of the learning away from pedagogy and the skills of the
master to the efficacy of the community’s learning resources. Thus the
challenges for the university tutor are not those of pedagogy, how best to
teach, but the challenge of how the community of practice can be replicated
within the confines of the university’s learning environment
Thus, in accordance with the theory of situated learning, becoming a full
member of a community of practice requires social interaction with, and
general access to:
• a wide range of ongoing authentic activities;
• old-timers (i.e. full members of the community);
• peers and other members of the community;
• relevant information;
• learning resources; and
• opportunities for participation.
“How much of this is provided by the course?” Clearly the workplace
provides authenticity and the PgD activities allow participation. The social
interaction and collaboration are provided by both the ALS activity and the
collaborative assignment required in the consultancy module. The relevant
information and the access to learning resources are provided by the WebCT
study web. “But, is the recipe complete?” “What community of practice is
being joined and how is it replicated in the course’s overall learning
environment?”
In trying to answer this question this research suggests firstly, that the
community of practice does not reside in the student’s workplace.
Respondents made it clear that, although some expert practitioners were

present in their place of work, in general, they were not surrounded by expert
colleagues. In some instances, for example in the case of a company
accountant, students were the only practitioner in their discipline. Students
did have company mentors who were a source of support and advice, but, in
general, the respondents agreed that the community of practice resided
beyond the workplace environment.
Secondly, the tutors on the course made no explicit attempt to artificially
construct a community of practice. This would have limited the social
construction of the community to the tutor’s interpretation, clearly a bounded
scenario. The students on the course were from very diverse backgrounds
and had very different learning objectives. It would not have been possible
for the tutors to construct multiple communities appropriate for each
student’s needs and would not have achieved an authentic learning situation.
Having ruled out the two obvious explanations of the nature of the
learning community, the picture that emerged from the data was one in
which the community of practice existed from a combination of the
following:
• the social interaction and collaboration between the learners on the
course;
• social interaction with mentors and some expert colleagues at work;
• contact with tutors and university academics;
• engagement with academic literature/resources; and
• exposure to how other managerial experts operate.
This is presented diagrammatically in figure 2.
268 Len Bird
Peers
Experts
Learning
Resources
Newcomer Co-Learners
Mentor Subject to social
& cultural
pressures Activities
www
Resource
Tutors Working
Methods
Academics
Literature
Figure 2: The Virtual Community of Practice on the PgD
The research supported this picture of the community in several ways.

Respondents talked repeatedly about the value and richness of the support
received from their peers and co-learners. For example:
“The action learning set was the most supportive part of the course”;
“It was useful seeing how other members of the set challenged my ideas
and made me think differently”; and
“Without the support of my fellow students I think I would have given up

the course”.
The emotional and academic support afforded by ALSs is well
documented, but clearly, within the learning context provided by the course,
this feature was regarded very highly by participants.
The various assignments on the PgD created numerous opportunities for
legitimate peripheral participation in the workplace. Students reported that
being on the course gave them a license to explore and engage with areas of
their business that stretched their expertise. Mentors and expert colleagues
gave them time and advice in a fashion similar to “master and apprentice”,
but in a more socially engaging routine manner, without any hint of a power
differential.
The CMC facility allowed constant interaction with university academics
and co-learners from a much broader group of managerial practitioners.
Asynchronous conferencing within the WebCT study web and other “mail
list-type” external discussions brought a dimension to the students’

interactions that could be described as “academically and socially
stimulating”. For example:
“The bulletin board allowed me to keep in touch and stay focused”;
“Electronic resources posted by others gave me heaps of ideas for my

project”; and
“I checked the web every day and found it motivated me to contact others
on the course”.
From the analysis of the CMC transcripts it was clear that the
collaborative nature of the consultancy assignment created the highest
degree of legitimate participation and reflected the whole ethos of the
situated learning process. The teamwork involved and the need to engage
with a real, live business problem allowed a level of authentic participation
that, through cultural and social interaction with the players concerned,
provided a rich social experience. Students learned by doing, but more than
this, they collectively built a body of knowledge about the organization and
its problems that was held by the group, not by one individual. No one single
member held all the answers; they were creating a microcosm of the
community of practice they were aspiring to join. The learning taking place
at the micro level on this one assignment mirrored the situated learning ethos
of the whole course.
6. CONCLUDING REMARKS
Having reviewed the theory and the results associated with the PgD what
can be concluded about CMC learning in virtual communities of practice on
the web? The 3-S model, although rudimentary, can provide set
facilitators/e-moderators with a framework for addressing the obvious
difference in contribution levels of CMC participants. The combination of
on-line interactions and face-to-face ALSs in a work-based learning context
gives a new dimension to the on-line mediation task. Co-presence is part of
the scenario. Learner relationships are a mixture of normal social
interactions and virtual exchanges. Exchanges virtually have face-to-face
repercussions so that identities become a mixture of the real and the virtual
persona. This has implications for the nature of the learning community
created and the power of the facilitator to make interventions both virtually
and face-to-face.
270 Len Bird
This new form of social existence brings with it a new opportunity for
learning. Some authors (Boden & Molotch, 1994) have argued that co-
presence is so important to communication and social relationships that
when it cannot be achieved anything else is second best. They state that
when people cannot actually secure a state of co-presence, they strive to
approximate it as best they can by using the telephone or the Internet.
However, this study has shown that virtual communities of practice can
compete. The dispersed nature of both the community and the knowledge
held atomistically by its members makes WebCT an ideal tool to trawl the
various depositories of knowledge in a sequence of legitimate activities that
brings an ever increasing confidence and expertise for the student.
REFERENCES
Beaudin, B. (1999). Keeping online asynchronous discussions on topic. Journal of
Asynchronous Learning Networks, 3 (2) [On-line]. (Accessed 19/07/02) URL:
http://www.aln.org/alnweb/journal/Vol3_issue2/beaudin.html
Biggs, J.B. (1999). Teaching for quality learning at university. Buckingham: SRHE/OU
Press.
Boden, D., & Molotch, H. (1994). The compulsion of proximity. In R. Friedland, & D.
Boden, (Eds.), Nowhere: space, time and modernity (pp. 257-286). Berkeley: University of
California Press.
Boud, D., Solomon, N., & Symes, C. (2001). New practices for new times. In D. Boud & N.
Solomon, Work-based learning: A new higher education (pp 3-17). Buckingham:
SRHE/OU Press.
Carr, W., & Kemmis, S. (1986). Becoming critical: Education, knowledge and action
research. London: Falmer Press.
Conole, G., & Oliver, M. (1998). A pedagogical framework for embedding C&IT into the
curriculum. ALT-Journal, 6 (2), 4-16.
Cobb, P. (1994). Where is the mind? Constructivist and sociocultural perspectives on
mathematical development. Educational Researcher, 23 (7), 13-19.
Deepwell, F., & Syson, A. (1999). Online learning at Coventry University: You can lead a
horse to water…. Educational Technology & Society, 2 (4) [On-line]. (Accessed 19/07/02)
URL: http://ifets.gmd.de/periodical/ vol_4_99/deepwell.html
Elliot, J. (1996). Action research for education. Buckingham: Open University Press.
Harris, R. (1999). Computer-conferencing issues in higher education. Innovations in
Education and Training International, 36 (1), 80-91.
Holsti, O. (1968). Content analysis. In G. Lindzey, & E. Aronson, (Eds.), The handbook of
social psychology: Research methods (pp. 596-692). Reading, MA: Addison-Wesley.
Lave, J. (1999). Situated learning and communities of practice. In L.B. Resnick, J. Levine, &
S. Teasley, (Eds.), Perspectives on socially shared cognition. Pittsburgh: Learning
Research and Development Centre.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation.
Cambridge UK: Cambridge University Press.
Lave, J., & Wenger, E. (1999). Learning and pedagogy in communities of practice. In J.
Leach, & B. Moon, (Eds.), Learners and pedagogy (pp. 66-68). London: Chapman.
Laurillard, D. (1993). Rethinking university teaching. London: Routledge.

Laurillard, D. (1995). Multimedia and the changing experience of the learner. British Journal
of Educational Technology, 26 (3), 179-189.
McConnell, D. (2000). Implementing computer supported co-operative learning. London:
Kogan Page.
McGill, I., & Beaty, L. (1995). Action Learning: A guide for professional, management and
educational development. London: Kogan Page.
McNiff, J. (1988). Action research: Practice and principles. London: Routledge.
Morgan, D. (1988). Focus groups as qualitative research . Beverley Hills, CA: Sage.
O'Hara, S., Beaty, L., Lawson, J., & Bourner, T. (1997). Action learning comes of age - part
2: action learning for whom? Education + Training, 39 (3), 91-95.
Pedler, M. (Ed.) (1991). Action learning in practice. Aldershot: Gower.
Salmon, G. (2000). E-Moderating: The key to teaching and learning online. London: Kogan
Page.
Savin-Baden, M. (2000). Problem-based learning in HE: Untold stories. Buckingham: OU
Press.
Vygotsky, L. (1978). Mind in society: The development of higher psychological processes.
London: Harvard University Press.
Distance Learning: The Experience of Accounting at
the University of Natal (Durban), South Africa
Anthony B. Lumby & Adrian D. Saville

University of Natal, South Africa
“Tell me it’s difficult, tell me it’s time consuming, but don’t tell me you
can’t do it.” Lockwood (in Marland, 1997, p. viii)
1. BACKGROUND
The South African university system consists of 21 universities. These

universities serve approximately 400 000 students spanning a wide spectrum
of qualifications and educational platforms. However, under the banner of
apartheid, South Africa’s past policy of racial segregation led to universities
establishing themselves along race-determined lines during the second-half
of the 20th Century. Additionally, arbitrary government regulation produced
further artificial divides within the education system. By way of example,
prior to 1994 South Africa’s tertiary education sector was divided into two
distinct parts. These were residential universities, which adopted direct
teaching methods and accounted for the bulk of student enrolments, and
“open” universities that adopted distance learning as their educational
platform. Within this sharply divided educational context, the University of
South Africa (UNISA) and Technikon RSA dominated distance-learning
programs. The two institutions collectively accounted for the bulk of tertiary
education enrolments by distance learners and were responsible for just
under 30% of all higher education enrolments in the first-half of the 1990s.
The country’s historically White universities, on the other hand, dominated
residence-based teaching, accounting for almost two-thirds of student
enrolments.
273
274 Anthony B. Lumby & Adrian D. Saville
Since 1994, however, South Africa’s new political dispensation has

allowed for greater fluidity and flexibility in the education system, with
policy efforts directed at dismantling apartheid structures, as well as
removing other artificial divides. One such enhancement has been to allow
residence-based universities to provide niche programs by way of distance
learning. In the case of accountancy, historically only UNISA was accredited
to provide distance-learning courses for the education and training of
professional accountants. In effect, then, the embargo on the provision of
distance-learning courses by residential universities was lifted.
Consequently, and in lock step with trends elsewhere in the world (Marland,
1997; Race, 1997), the past seven years has seen a number of residence-
based universities stepping into the rapidly expanding distance-learning
market. On a pedagogic note, distance learning and open learning are often
considered to be synonymous. However, in line with Race (1997), distance
learning is considered to be a subset of open learning, where open learning is
defined as any educational platform that gives learners some degree of
choice and control in the education process. Thus, open learning or open
education are seen as an instructional approach and, as such, may or may not
be embraced within the framework of a given distance-learning program.
Set against this backdrop, in 1996 a decision was taken by the executive
of the University of Natal (Durban campus) to engage in distance learning.
This decision entailed shifting the education platform from one of direct
contact into a mixed-mode offering, which involved coupling direct (or
contact) education with a distance-learning program. The course that was
selected, as the pilot program, was the Postgraduate Diploma in Accounting,
redesignated as the Bachelor of Commerce Honours (Accounting) in 1999.
This serves as the springboard for the professional qualification of Chartered
Accountant (CA) in South Africa. The University’s decision was driven by
four main factors.
First, over the course of the past two decades, the University has
established itself as a centre of excellence in the provision of education for
prospective CAs. As evidence of this, the University has enjoyed substantial
growth in student enrolments over the period (the evidence is reviewed at a
later stage in this paper). Somewhat more importantly, however, is the fact
that the results achieved by students of the University’s School of
Accounting and Finance in the qualifying examinations, set by the South
African Institute of Chartered Accountants (SAICA) as an independent
litmus test, have generally been significantly ahead of the national average.
With this in mind, distance learning allowed the University to extend this
service to a wider market.
Second, and not unrelated to the first point, by going the distance
learning route, the University hoped to encourage an increase in the number
Distance Learning 275
of trainees entering the accounting program. In turn, this effort would serve
to swell the pool of qualified accounting professionals in South Africa. At
this juncture, it is worthwhile noting two allied points. South Africa suffers
from an acute shortage of trained professionals (Republic of South Africa,
2001). Professional accountants are no exception, with the available
evidence pointing to an industry shortage of the order of 15.0 to 20.0% of
the extant accounting workforce (Klein, 1997; Department of Labour, 2001).
Further, South Africa currently exports more CAs than it trains in any one
year and it is estimated that in excess of one-quarter of South African trained
CAs currently live outside of the country. A particularly disturbing feature of
this trend is that many accountants who leave the country are those with
longer service (eight to ten years). This compares to figures of four to seven
years in 1995-1997 and two to four years in 1992-1995 (Bennett, 2000).
There are, however, a variety of reasons for this outcome, a phenomenon
commonly referred to as “brain drain”. Here there are two broad sets of
factors at work: negative push factors and positive pull factors. Push factors
include an array of socio-economic ills. However, by far the most widely-
cited reasons amongst professionals for leaving South Africa are high rates
of unemployment and low rates of job growth; low levels of literacy and
education; unacceptably high levels of crime (particularly violent crime);
punitive tax structures; an absence of accessible and appropriate education,
training and skills promotion programs; and, arguably most significantly, an
absence of a clear government strategy designed to address and redress the
foregoing socio-economic ailments (Kaplan, 1998; SATIS, 2000). A range
of positive pull factors has exacerbated South Africa’s on-going loss of
skilled workers. By far the most significant of these pull factors are the
perception of improved career prospects abroad and meaningfully improved
access to public goods (e.g. health, education, social security benefits)
(Brown & Van Staden, 1998).
Third, by embracing distance learning, the University would be able to
make training in accountancy available to members of previously
disadvantaged communities (PDCs). On this score, members of PDCs
remain seriously under-represented in the accounting profession. As
evidence of this, as at November 2000 SAICA boasted a membership of
19,106 individuals. In contrast to this number, there are currently fewer than
400 qualified African accountants in the profession in South Africa. To put
this figure into perspective, whereas Africans make up more than 75% of the
country’s population, they currently account for a little over 2% of South
Africa’s population of professional accountants. As an aside, in addition to
the obvious adverse macro-effects of the above disparity, the shortage of
skilled African CAs also has undesirable micro-effects, such as promoting
job-hopping (Palmer, 2001).
Fourth, the University’s distance-learning program was designed as an

income generating initiative. As such, the program would allow staff
members to augment their incomes. In turn, it was argued that this would
help ameliorate the wide and growing gap that existed between the
remuneration packages of academic and private sector professionals. In this
regard, evidence of the extent of this income gap is provided by a 1997
survey conducted by Deloitte and Touche. At the time the survey was
conducted, a newly qualified CA entering employment in the private sector
could expect a remuneration package that was typically one-and-a-half to
two times the value of packages offered to CAs entering academia (Klein,
1997). Furthermore, the gap was found to widen with experience. As
corroboration of this, the survey revealed that by the time professional
accountants were in their early thirties, the discrepancy between private
sector remuneration packages and those offered by the accounting
departments of South Africa’s largest universities had grown to a factor of
two to four times. Furthermore, it is arguable that the gaps reported above
have increased with time. The past five years has seen the remuneration
packages of accountants in the private sector growing by between 10% and
20% per annum, whereas those of university employed professionals have
typically expanded at a pace more in line with the rate of domestic consumer
price inflation of 7.5% to 10% per annum (Statistics South Africa, 2001).
2. THEORETICAL ROOTS
Given the above backdrop, the University’s School of Accounting and

Finance introduced the postgraduate course in accounting by distance-
learning mode at the beginning of 1997. In order to ensure effective
implementation of the program and to promote the program’s potential
success, it was seen as essential that accepted principles and practice of
distance learning were effectively applied in the design and construction of
the course. Moreover, considering South Africa’s socio-economic backdrop,
it was important that the course meshed closely with the extant education
policy framework. Accordingly, in this section we provide a brief review of
the learning principles and education policy framework that were considered
to be instrumental in influencing the construction and shape of the Bachelor
of Commerce Honours (Accounting) course that was introduced for the
University’s distance-learning students at the beginning of 1997.
Mainstream educational principles suggest that a range of well-defined
variables is instrumental in determining the success of distance-learning
programs. However, the overarching criterion, according to Race (1997), is
to ensure that programs are designed to facilitate and reinforce the processes
involved in completion of the “wanting-doing-feedback-digesting” learning

cycle.
Viewed against this backdrop, Race identifies six main factors that are
argued to promote the learning cycle. One of the most valuable energizers of
the learning cycle is the process of “learning by doing” (1997, pp. 18-21 &
27-29). To this end, emphasis is placed on the importance of the practical
aspects of distance learning (e.g. self-assessment questions, tutor marked
assignments and computer marked assignments). It is also acknowledged
that feedback to students plays a key role in ensuring that the learning cycle
is strengthened and perpetuated.
By virtue of the nature of the market that distance-learning programs
cater for, the success of any program is typically dependant on its flexibility
(Race, 1997). Here, by flexibility it is meant that distance-learning programs
should be able to provide for learners with a range of different needs and
constraints. This might include catering for, inter alia, the unequal pace at
which learners proceed through programs; the varying amount of time each
is able to devote to learning; geographical diversity; and differences in the
educational backgrounds of learners.
It is desirable that the material presented in distance-learning programs is
readily accessible to learners and that learners find the material easy to use.
It is also desirable that learners understand the relevance of the material to be
explored. Thus, and by way of example, the “tone” and “style” of distance-
learning material is considered to be important in buttressing the program
material (Race, 1997).
The importance of distance learners being aware of the aims and
objectives of the material, as well as being alert to the competence
requirements of the program also needs highlighting (Race, 1997). Whilst
the above features are plainly relevant in any learning environment, they are
of particular relevance in distance-learning environments where learners are
often stripped of many sources of help and do not always have the
opportunity to establish, as in a face-to-face environment: “What is
required?” or “What is expected of me?” (Bloom et al., 1971; Bloom 1972).
Self-assessment is viewed as pivotal in the process of distance learning
(Race, 1997). In this regard, it is seen as critical that intended learning
outcomes are translated into “learning-by-doing” opportunities. In particular,
structured feedback that enables self-assessment is regarded as a vital
ingredient in the successful learning cycle (Rowntree, 1989; Walkin, 1991;
Race, 1997). A variety of question tools are available to course designers to
facilitate and promote self-assessment. Such tools include multiple choice,
sequencing, true-false, fault-finding and open-ended questions (Brown &
Knight, 1994; Race, 1997).
Practical elements must also be considered in course design, with

attention being paid to factors, such as total cost and cost effectiveness
(Race, 1997), as well related issues, such as the appropriateness and
flexibility of technology employed in the distance-learning program (Bates,
1997). The latter point is of particular relevance in societies, such as South
Africa, where high degrees of inequality exist in terms of learners’ access to
resources.
Whilst educational principles provide guiding lights in the design of
distance-learning courses, it must be recognized that, to be effective, course
design must also embrace the education policy framework. To this end, we
provide below a policy summary drawn from the Department of Labour’s
key policy document, The National Skills Development Strategy: April
2001–March 2005 (Department of Labour, 2001). Whilst the views cited
below obviously post-date the period over which the University’s distance-
learning course was designed and introduced, it is submitted that they
provide a clear analysis of the principles that have directed education policy
as it has evolved in South Africa over the past five years.
3. THE SOUTH AFRICAN POLICY FRAMEWORK
The Department of Labour’s key policy document incorporates a number

of guidelines. The most important are summarized below.
3.1 Policy Guidelines
3.1.1 Demand-Led Education
Historically, South Africa’s education policy framework has focused on

education and skills development in the absence of a realistic assessment of
how the skills are to be employed. In contrast to this, the country’s policy
makers now recognize that appropriate education and skills development
requires emphasis to be placed on the promotion of skills and competencies
required to support factor productivity, international competitiveness, the
mobility of workers, self-employment and meeting well-defined and clearly-
expressed community needs.
3.1.2 Flexibility and Decentralization
As alluded to above, over the course of the past century, South Africa’s
education system evolved in a fashion that was both inflexible and
centralized. Whilst these features were designed to serve the ends of South
Africa’s past policies, it is evident that the principles applied fly in the face
of well established and well accepted education and skills promotions
principles adopted elsewhere in the world. Thus, post-1994, and with the
country moving into a new democratic era, the educational framework has
shifted sharply to allow for greater flexibility, fluidity and decentralization in
the promotion of learning and skills development. In the same breath, it is
interesting to note that government has conceded that it is public employers,
private employers and workers who are best placed to make judgments about
educational priorities and to determine the most effective providers to meet
those needs. Within this new paradigm, the tasks of government become
more refined to those of providing the framework, direction and coordination
for the country’s education and skills development strategies and of
monitoring the implementation and success of these strategies.
3.1.3 Partnership and Co-operation
The South African government has traditionally played the central role in
the provision of education, training and skills promotion. In contrast to this,
under the new political dispensation, it is explicitly recognized that at
national, sector, provincial, community and workplace levels, the definition
and implementation of learning and skills development strategies should be
based on partnerships between and amongst social constituencies.
3.1.4 Efficiency and Effectiveness
Finally, it is recognized by policy makers that the delivery of education

and skills development programs and initiatives must be characterized by
cost efficiency and that the programs should lead to positive outcomes for all
who invest in them.
3.2 Policy Objectives
In order to fulfill the mission mapped out above, the South African
government has identified five objectives to drive their skills and education
development strategy (Department of Labour, 2001). The objectives can be
summarized as follows:
to develop a culture of high quality lifelong learning;
to foster productivity and employability in the formal sector via
appropriate education and skills development programs;
to stimulate and support skills development in small businesses;
to promote employability and sustainable livelihoods through social

development initiatives; and
to assist new entrants into employment.
With these guiding principles and policy objectives in mind, the paper
turns to consider the educational structure adopted in the University’s
distance-learning program for accountants.
4. DISTANCE LEARNING: THE UNIVERSITY’S

APPROACH TO THE PROGRAM
The School of Accounting and Finance’s distance-learning program has

two offerings. The first is the Distance Learning One-Year Program, which
replicates the full-time Bachelor of Commerce Honours (Accounting)
contact program, in terms of syllabus and examinations. The course is
designed for those candidates whose undergraduate degree was completed at
a university registered with SAICA for the training of CAs and whose final-
year undergraduate results are regarded as satisfactory. The second offering
is the Distance Learning Preparatory Program. The program is designed for
those candidates whose undergraduate degree was completed at a university
that is not registered with SAICA, or whose final-year undergraduate results
are regarded as too weak to be able to succeed on the Distance Learning
One-Year Program, or whose undergraduate degree does not contain all the
modules required for entry to postgraduate studies in accounting. In this
vein, the Distance Learning Preparatory Program is specifically designed to
strengthen candidates’ abilities in the four core accounting disciplines:
Financial Accounting, Managerial Accounting & Finance, Auditing and
Taxation. In this way, the School of Accounting and Finance believes that,
notwithstanding the fact that distance-learning candidates are effectively
engaged in part-time studies, the completion of the Preparatory Program
should allow candidates to avoid the low pass rates associated with other
programs, such as the University’s discontinued part-time Bachelor of
Commerce Honours (Accounting) contact program, which is reviewed
below.
Before moving on, it is important to note that, in the view of the
University, there exists a fundamental difference between the School of
Accounting and Finance’s distance-learning program and that of its major
competitors. More to the point, the School of Accounting and Finance’s
program is deliberately structured along the lines of supported distance
learning. In this regard, the School of Accounting and Finance drew heavily
on the experience gained by the UK’s Open University in the provision of
distance-learning programs. Accordingly, a collection of forms of support is

provided to candidates enrolled on the distance-learning program.
Learners are provided with a series of carefully organized distance-
learning materials that provide both the theoretical principles, backed by
case-study examples, and numerous exercises, which are covered, section-
by-section, in each of the weekend and Saturday morning study workshops.
Arrangements are made for four two-day study workshops during the
academic year, which are held in each of the major centers (Johannesburg,
Cape Town, Port Elizabeth, East London and Durban). At the first three
workshops, intensive lectures are provided on the study material and the
exercises for each section are explained and reviewed. The fourth workshop
comprises a “practice” examination designed to prepare students for the
formal examinations to follow. In association with the National School of
Accounting, which is the educational wing of the accounting profession in
South Africa), candidates are provided with access to four Saturday morning
workshops per month during the academic year. One workshop per month is
devoted to each of the advanced disciplines, namely Advanced Financial
Accounting, Advanced Taxation, Advanced Managerial Accounting and
Finance and Advanced Auditing.
Video material on the key principles involved in a selection of the four
accounting disciplines is also available. This material elaborates on the
principles presented in the course textbook and course notes, and emphasizes
application.
The School of Accounting and Finance also makes available a phone-in
service to answer learner queries through the offices of the directors for each
of the four advanced disciplines. This service is manageable because in
practice students tend to bring queries to Saturday morning workshops.
In addition to the preparatory examinations referred to above, the
students are exposed to at least two formal test sessions per module. The
tests are written under examination conditions, which further helps learners
prepare for the final examinations.
Over and above the aforementioned services, students are also able to
identify one another at registration time. This allows students to establish
learner networks for peer support. Moreover, many of the students enrolled
in the distance-learning program are employed in accounting firms. This
means that many students have contact and support bases within their places
of employment.
Importantly, the above facilities are not seen to be substitutes for one
another. Rather, the facilities are considered to be complementary in that
they are designed to provide an enriched learning experience and an
extensive support base for students enrolled on the program.
In summary, the course and material design of the School of Accounting

and Finance’s distance-learning program places emphasis on embracing the
established and accepted principles of distance learning identified above.
These include program flexibility, accessibility and core educational criteria,
such as learning by doing and self-assessment, as well as practical
considerations. In addition, it is submitted that the program goes some way
to acknowledging South Africa’s socio-economic context by building upon
government’s policy guidelines, namely demand-led education with an
emphasis on principles of flexibility and decentralization, partnership,
efficiency and cost effectiveness. In the final analysis, however, the success
of any distance-learning program must be measured by its outputs. To this
end, we review the evidence in the section below by comparing the output
performance of the School of Accounting and Finance’s distance-learning
program in the context of its historical track record, as well as relative to the
performance of its major competitor.
5. EVIDENCE AND EXPERIENCES: THE FIRST

FOUR YEARS OF DISTANCE LEARNING FOR
ACCOUNTANTS
By way of background to a discussion of the relative success of the

distance-learning program, enrolments for the Bachelor of Commerce
Honours (Accounting) contact program (including both full-time and part-
time enrolments) fluctuated around an average of 132 students per annum
over the period 1991-1993 (see table 1). During that time, the pass rate
oscillated around an average of 57%, which was considered by the
University to be unduly low, and to represent an inefficient use of resources.
Thus, at the end of 1993, the School of Accounting and Finance launched an
investigation into the unsatisfactory pass rate. The review drew three broad
conclusions.
The pass rate amongst full-time candidates, ranging from 65% to 75%,
was considered reasonably satisfactory and suggested that the School of
Accounting and Finance’s undergraduate program prepared candidates
satisfactorily for entry to postgraduate studies. The overall pass rate was
being dragged down by the extremely low pass rate, ranging from 20% to
25%, amongst part-time candidates. Part-time candidates faced inordinately
heavy demands because of the prevailing rule, stipulated by SAICA, that
Part I (Advanced Financial Accounting, Advanced Managerial Accounting
& Finance and Advanced Taxation) and Part II (Advanced Auditing) had to
be passed in one examination sitting (including supplementary and aegrotat

examinations).
For the above reasons, at the beginning of 1994, the part-time program
was discontinued and this explains the decline in enrolments for the contact
program post-1994 (see table 1 below). During 1994-1997, enrolments for
the full-time contact program fluctuated around an average of 77 per annum.
At the same time, the pass rate rose significantly, averaging 75% for the
period 1994-1997. As an aside, it should be noted that, against the
background of a growing shortage of CAs in South Africa, the profession
appeared to offer strong employment opportunities, and, concurrently, the
introduction of the distance-learning program in 1997 raised the profile of
the School of Accounting and Finance as a major center for accounting
education in South Africa. As a result, enrolments for the full-time contact
program rose from 71 in 1997 to 138 in 2000.
Evidence of the success of the distance-learning program is summarized
in table 1. In this regard, enrolments for the Distance Learning One-Year
Program have risen by 69% from 465 in 1997 to 784 in 2000. The pass rate
for these candidates has fluctuated around an average of 40% for the period
1997-2000. Whilst this is significantly below the pass rate of the full-time
contact program, it should be noted that the pass rate on the Distance
Learning One-Year Program has exceeded that of the former part-time
contact program by a considerable margin. The University sees this as a
manifestation of the appropriateness of the selection process, as well as a
reflection of the value of the Distance Learning Preparatory Program.
Furthermore, the pass rate on the Distance Learning One-Year Program is
almost double that of the only competing institution, UNISA, which engages
solely in distance learning for all of its programs.
The reasons for the success of the distance-learning program are argued
to be essentially threefold. First, the quality of the learning materials is
acknowledged to be at the cutting-edge of developments in the discipline and
therefore candidates on the program are better prepared to meet the
challenges posed by the professional examinations. Second, with advice
from the Open University in the UK and the South African Institute for
Distance Education (SAIDE), a good deal of work has been put into making
these materials more user-friendly. Third, and perhaps most importantly, the
program is not a correspondence program, but is specifically designed to
provide various forms of support in order to assist distance learners.
Returning to table 1, the data show that enrolments for the Distance
Learning Preparatory Program have risen even more rapidly (by 96% from
242 in 1997 to 475 in 2000). The pass rate has averaged 45% per annum for
the period 1997-2000. Given the fact that the overwhelming majority of
candidates enrolled for this program have already completed majors in
accounting disciplines (albeit with low pass marks or at an institution not

registered with SAICA for the training of CAs), the relatively low pass rate
suggests that the concerns relating to the preparedness of these candidates
for direct entry to the Distance Learning One-Year Program are valid.
Table 1: Postgraduate Diploma in Accounting/Batchelor of Commerce Honours (Accounting)

1991-2000.
Year 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000
Contact Program
No. Registered 122 115 159 87 65 86 71 103 121 138
No.Wrote 117 113 156 81 63 80 67 98 119 131
Examinations
No. Passed 69 70 79 61 45 60 53 75 82 94
Pass Rate 59% 62% 51% 75% 71% 75% 79% 77% 69% 72%
Distance Learning
(One Year
Program)
No. Registered 508 796 873 900
No. Wrote 465 715 760 784
Examinations
No. Passed 178 295 290 784
Pass Rate 38% 41% 38% 39%
Distance Learning
(Preparatory
Program)
No. Registered 243 361 371 475
No. Wrote 212 296 307 369
Examinations
No. Passed 94 142 164 142
Pass Rate 44% 48% 53% 39%
Notes
As noted above, the Postgraduate Diploma in Accounting was re-designated as Batchelor of
Commerce Honours (Accounting) in 1999. The renaming of the course came into effect in
2000.
During 1991-1993, the contact program included both full-time and part-time enrolments.
The Distance Learning One-Year and the Distance Learning Preparatory Programs were first
introduced in 1997.
The difference between ‘No. Registered’ and ‘No. Wrote Examinations’ excludes de-
registrations and represents the number of absentees from the examinations who had not
applied for an aegrotat examination.
‘No. Wrote Examinations’ includes supplementary and aegrotat examinations.
‘No. Passed’ includes supplementary and aegrotat examination results.
In summary, the above data point to some early successes in the School
of Accounting and Finance’s distance-learning program. At the same time,
however, it is generally accepted that candidates enrolled for the Distance
Learning One-Year Program will not achieve the same pass rates as those
enrolled for the full-time contact program. After all, distance-learning
candidates are, with very few exceptions, engaged in full-time employment
and therefore face all the problems associated with part-time studies, as well
as SAICA’s ruling that Parts I and II must be passed at the same examination
sitting. Thus, a key task of the School of Accounting and Finance becomes
that of providing learners with the resources and tools to overcome these
barriers. Therefore, and notwithstanding the fact that the School of
Accounting and Finance’s distance-learning program is relatively new and
that all staff members are engaged in a steep learning curve, the question
which arises is: “What can be done to improve both the learning experience
and the pass rate on the Distance Learning One-Year Program?”
6. FINDING THE WAY FORWARD
Given the relatively short history of the School of Accounting and

Finance’s distance-learning program, it is difficult to provide a
comprehensive answer to the above question. Nevertheless, based on the
experiences of the past few years, as well as the evidence gathered from
various literature surveys and lessons learned from other institutions, it is
possible to provide a number of near-term objectives that are capable of
improving the learning experience and pass rates of students enrolled in the
School’s distance-learning program. We offer five main suggestions.
In line with the principles of distance learning presented in Section 3 of
this paper, it is argued that the School’s distance-learning program should
place a greater emphasis on feedback to students. In particular, based on
arguments presented by Race (1997) and Marland (1997), it is argued that, in
addition to the distance-learning material and facilities already provided,
tutor-marked assignments could play a key role in enhancing the “wanting-
doing-feedback-digesting” learning cycle identified by Race (1997).
Second, and in a similar vein, it is acknowledged that tutors could play a
more active, and expanded, role in engendering an interactive learning
environment and in mentoring distance learners (Race, 1997). In both
instances, however, the difficulty is overcoming the hurdles of distance and
financial expense. Yet, as the reach of technology increases, these obstacles
are likely to become easier to clear, particularly if government is successful
in achieving the goals set out by its telecommunications policies. Indeed,
some early successes have already been enjoyed by South African-based
distance-learning programs in using new technologies to extend the reach
and penetration of programs (De Kock, 1999). Within this context then, the
primary challenge becomes that of selecting appropriate distance-learning
technologies and exploiting the potential advantages of web-based teaching

and learning, rather than dealing with more traditional impediments, such as
distance and funding (Bates, 1997).
Whilst members of the School of Accounting and Finance have played a
central role in preparing distance-learning materials, it is arguable that the
task of producing effective materials is never complete. Indeed, in line with
Race’s (1997) comments with regard to “tone” and “style”, it is accepted that
the School should strive to continually improve presentation and palatability
of distance-learning materials.
Finally, information should not, nor does it, flow in one direction. By this
it is meant that the School of Accounting and Finance is just as capable of
learning from its distance learners as its distance learners are capable of
learning from the School. To this end, it is suggested that the School should
place greater emphasis on gathering information about the program’s
features, such as cost effectiveness, flexibility, accessibility and students’
achievements of end objectives, in order to enhance the reach and
effectiveness of its distance-learning program.
It is readily acknowledged that the above initiatives are by no means
comprehensive or definitive in mapping a way forward for the School of
Accounting and Finance’s distance-learning program. Nevertheless, these
initiatives are seen as providing important extensions and additions to the
program. As such, they should enable the School to maintain its position as
market leader in the provision of distance-learning accounting education and
to propagate the education- and skills-promotion objectives of the country’s
policy makers.
REFERENCES
Bates, A.W. (1997). Technology, distance learning and open education. London: Routledge.
Bennett, J. (2000). Working on the skills shortage. Business Times, Nov 12th.
Bloom, B.S. (Ed.) (1972). Taxonomy of educational objective. Book 1: Cognitive domain.
London: Longman.
Bloom, B.S., Krathwohl, D.R., & Masia, J.F. (1971) Taxonomy of educational objectives.
Book 2: Affective domain. London: Longman.
Brown, M., & Van Staden, C. (1998). Response to the brain drain phenomenon. The South
African Network of Skills Abroad. Working Paper, July.
Brown, S., & Knight, P. (1994). Assessing learners in higher education. London: Kogan
Page.
De Kock, J.H. (1999). Critical issues facing education, training and development in a
changing South Africa. Potchefstroom: Human Science Research Council.
Department of Labour (2001). The national skills development strategy: Skills for productive
citizenship for all (April 2001-March 2005). Pretoria: Department of Labour, Republic of
South Africa.
Kaplan, D.E. (1998). Migration of the professional, semi-professional and technical

occupations in South Africa: Past patterns, current trends and policy. In J. Charum & J-B.
Meyer (Eds.), International scientific migrations today (pp. 1-20). Proceedings of the
International Symposium held in Bogotá, June.
Klein, M. (1997). Financial professionals hit the big time with golden halos. Business Times,
Mar 23rd.
Marland, P. (1997). Towards more effective open and distance teaching. London: Kogan
Page.
Palmer, J. (2001). Career paths need careful strategies. Business Day, May 9th.
Race, P. (1997). The open learning handbook. London: Kogan Page.
Republic of South Africa (2000). A nation at work for a better life: A summary of President
Mbeki's state of the nation address to parliament on 4 February. Pretoria: Government
Printer.
Rowntree, D. (1989). Assessing students: How shall we know them? London: Kogan Page.
SATIS (2000). South African ICT sector development framework. Pretoria: Department of
Trade and Industry.
Statistics South Africa, (2001). Demography: mid-year estimates. Pretoria: Department of
Statistics
Walkin, L. (1991). The assessment of performance and competence: a handbook for teachers
and trainers. Cheltenham: Stanley Thornes.
Creating and Improving a “Virtual Object” Through
Web-Mediated Discourse
Gordon Wells
University of California, Santa Cruz, USA
1. LEARNING AND TEACHING AS DIALOGIC

INQUIRY
Learning has been given a variety of definitions, some of them more

illuminating than others. As it is currently defined, operationally, in many
educational institutions, learning is what is assumed to have taken place
between teacher input and student performance on a subsequent occasion of
testing. There are two problems with this version. First, the process of
learning remains mysterious and unavailable to observation; and second, it is
treated as a relatively passive response to curriculum delivery, while the
active role in the process is attributed to the teaching, whether through
teacher lectures, textbook readings or the more novel electronic means of
providing input. Furthermore, the learning and teaching is “encapsulated”, to
use Engeström’s (1991) term, for little consideration is given to the
relationship between the knowledge to be acquired and the students’ past
experience, or to the relevance of the knowledge for their current concerns
and future intentions.
An alternative definition, that is favored by many researchers, sees
learning as the continuing process of constructing and extending meaning
that occurs in and through the learners’ engagement in situated joint
activities (Resnick, 1987; Rogoff, 1990; Halliday & Matthiessen, 1999).
Through attempting, with the help of others, to solve the problems to which
these activities give rise, learners appropriate the resources of the culture,
289
290 Gordon Wells
constructing their own version of them on the basis of their existing

understanding, and contribute to the enhancement of understanding by the
group as a whole. As Lave and Wenger (1991) emphasize, learning on this
view is not a separate and independent activity, but an integral aspect of the
formation of personal identity that occurs through increasingly full
participation in the productive activities of a “community of practice.”
This “apprenticeship” model is most effective in characterizing the sorts
of informal learning that have occurred in all societies over the many
millennia of human history and that still play an important role in modern
societies, for example in the home and in sporting, religious and political
organizations. However, there are two features that are lacking in this model
as far as modern, literate societies are concerned. First, there are specialized
knowledgeable skills for which systematic preparatory learning is required,
for example in the physical and social sciences or in medicine. Although
participation in the collaborative solving of real-life problems can play a
significant role in inducting novices into the activities of these communities,
apprenticeship of this kind is not in itself sufficient (Lemke, 2002). And
second, the disposition to adopt a reflective, metacognitive stance to the
activities in which one is engaged is not one that is particularly fostered
through apprenticeship, yet it is crucial to the achievement of really high
levels of performance in many contemporary fields of endeavor. A further
problem, that apprenticeship shares with the first definition of learning
sketched above, is that learning itself remains unexplained. Yet, if we are to
provide optimal learning conditions for students undertaking formal
education at any stage in their lives, it is important to have a theory of
learning that is amenable to empirical investigation. The question that
remains to be answered, then, is whether the conception of learning as
apprenticeship can be reconciled with the very different one that underpins
the formal education that schools and universities are intended to provide
and, if so, how.
The solution to this problem that I have been investigating is suggested
by work in the tradition of cultural historical activity theory (CHAT) that has
grown from the insights put forward by Vygotsky (1978, 1981, 1987) in the
early part of the 20th Century. The key tenets of this approach can be
summarized as follows:
• Purposeful collaborative activity is both the setting and the motivator for
the interactions through which learning and development occur, both on
the time-scale of cultural history and that of individual life trajectories.
• Such activities are always situated in space and time and are mediated by
the particular cultural resources available, both the material and semiotic
artifacts that are to hand and the practices in which they are deployed.
They are also mediated by the knowledgeable skills of the human
Web-Mediated Discourse 291
participants, of which the skills of meaning making are of paramount

importance. Discourse, both spoken and written, thus plays a central
role.
• Knowledge in different modes, practical, aesthetic and theoretical, is one
of the outcomes of such activities, embodied in the artifacts produced
and in the enhanced skill in using these artifacts by the participants
involved. Seen in this light, knowledge is to be valued not as a personal
holding of context-free cultural capital, but as a resource to mediate the
solving of problems in future situated activities.
• From this perspective, learning can fruitfully be conceptualized as
appropriating and personally transforming the knowledge and skills
enacted in such activities, and developing the dispositions to use these
resources responsibly and effectively to contribute to further projects of
personal and social significance.
Because new learning always builds on personal prior experience,
individuals construct different meanings from the same event. Starting from
different cultural niches, learning trajectories are therefore both diverse and
unique, as are the identities that are formed over time. This diversity
constitutes a rich resource within society, both for creatively meeting new
demands and for challenging and transforming the status quo.
Given this conception of learning as the active appropriation and personal
transformation of the knowledge, skills and practices encountered in situated
activities, the role of the teacher can then be conceptualized in terms of a
twofold responsibility. The first is to select and organize activities that
provide challenging opportunities for learners to develop their dispositions,
skills and knowledge. The second is providing assistance that is both
contingently responsive to individual learners’ difficulties with the current
task and also aimed at enabling them to “go beyond themselves” in their
capacity for autonomous action (Wells, 1999).
2. THE ROLE OF DISCOURSE IN LEARNING AND

TEACHING
I have already remarked that in both the “delivery” and the

“apprenticeship” models, individual learning remains unexplained. However,
this lacuna is remedied in the CHAT model by the central role given to
discourse. For Vygotsky (1978, 1981, 1987), learning is the process through
which actions undertaken in joint activity become “internalized” to
constitute the means whereby individuals are able to carry out the
comparable actions “in the head” and autonomously. For example, by
participating in the forms of “social speech” in which genres of reasoning
292 Gordon Wells
are used to solve problems faced by the group, the individual constructs the
related genres in “inner speech”, which enables him or her to solve similar
problems through a form of dialogue with self. Halliday, a social
semiotician, puts forward a similar explanation when he proposes that:
“Language is the essential condition of knowing, the process by which
experience becomes knowledge” (1993, p. 94).
If we want to observe learning in action, therefore, it is to discourse that
we need to turn. Discourse comes in various modes, of course. Most typical
is the spoken inter-personal discourse that either organizes and comments on
the non-verbal activity it accompanies (e.g. in carrying out a surgical
operation or assembling an artifact from its parts) or in itself constitutes the
activity (e.g. a committee meeting or a gossip session). But discourse can
also occur in the written mode (e.g. by letter or e-mail or as a contribution to
a learned journal). It can also occur in non-verbal mode, as in the case of co-
ordinated action. By comparison, inner discourse is more difficult to
observe, but it can be tracked to some degree through “think-aloud”
protocols or glimpsed by means of introspection.
EXPERIENCE INFORMATION A
C
T
I
O
UNDERSTANDING KNOWLEDGE N
BUILDING
Figure 1: The Spiral of Knowing. Adapted from Wells (1999).
However, since “the individual develops into what he/she is through what
he/she produces for others,” as Vygotsky puts it (1981, p. 162), it is
primarily to social discourse that we should look to see learning in action,
for it is in the effort to formulate one’s meaning in a contribution that both
responds to preceding contributions and advances the understanding of the

group that one increases and refines one’s own understanding (Wells, 1999).
This can occur, of course, in any of the modes of meaning making discussed
above.
The central role of discourse in learning can thus be thought of as the key
phase in a cycle that starts with the initial understanding of a situation or
topic one has, based on previous experience, and leads to an increment in
one’s understanding as new information is worked on through the
collaborative discourse of knowledge building, in which the attempt is
jointly undertaken to make sense of all the available information in order to
arrive at shared understandings “that are new to the local participants and
that the participants recognize as superior to their previous understanding”
(Bereiter, 1994, p. 9). I have tried to capture this cycle in figure 1.
3. PLANNING AND TEACHING A COURSE
In the light of the foregoing account of my theoretical orientation, it will

come as no surprise that I devote a considerable part of my classes to
discussion, in both small group and whole class formats. In the M.Ed.
courses, there are typically two focuses for these discussions. The first is the
readings assigned for the weekly classes and the second the practical
investigations that I ask each course member to carry out in his or her place
of work (most if not all of the students are also full-time teachers). In the
doctoral seminar, the weekly readings provide the main basis for discussion,
but I encourage course members to draw on their previous experience to
ground the theoretical constructs encountered in the readings and to carry out
a small practical investigation in which they explore some aspect of the
theory under discussion.
In both types of class, the role I undertake as teacher is threefold:
• to put together a cumulatively coherent selection of readings, together
with questions to initiate discussion;
• to propose relevant additional activities and provide support and advice
when needed; and
• to chair the whole-class discussions.
I also circulate during group discussion time to gain a sense of how the
course members have made sense of the reading or other activities and to
contribute, as appropriate, to their discussions. In this way, I prepare myself
to contribute, in as relevant and as effective a manner as possible, to the
reflective whole-class discussions that provide the culmination of each class
meeting.
294 Gordon Wells
Although discourse plays the central role in my classes, it does not have
to take place exclusively in the face-to-face mode. In fact, I believe that there
is much to be gained from engaging in written discourse as well, for what
this mode lacks in immediacy it gains in the permanence of the utterance
produced, which allows both the writer and the reader to adopt a more
reflective stance on the topic. I have long seen the final assignment in the
course as providing an opportunity for the writer to engage in a dialogue
with his or her own text in the process of composing and revising it. With
the advent of e-mail, I also began to experiment with the possibilities that a
“class listserv” could provide for a more interpersonal dialogue, in which I
was only one of the reader/writer participants. In those early days, not all
students had access to the Internet and among those who did not all found it
congenial. A particular problem was that, after being read, messages tended
either to get inadvertently deleted or lost as they were buried by new
incoming mail. So, when the computer supported intentional learning
environment (CSILE), the collaborative database program developed by my
colleagues, Bereiter and Scardamalia (Scardamalia, Bereiter, & Lamon,
1994), was produced in a web-based version, I switched to the web
knowledge forum (Web KF) for our written class dialogue, in the hope that it
would provide a better medium for the sort of knowledge building discourse
that I was seeking to promote (more information about Web KF can be
found at: http://csile.oise.utoronto.ca/kfdemo/kfdemo.html: Accessed
15/07/02). With this new tool and the increasing proportion of teachers with
Internet access, there has definitely been an increase in both quantity and
quality of written discourse in the last two years.
This, then, was the way in which I approached the courses that are the
subject of this paper. One of them was a regular doctoral seminar entitled:
Sociocultural Theory and its Applications to Educational Practice and
Research. For most of its duration, the class consisted of six students, but
unfortunately one had to withdraw for personal reasons about two-thirds of
the way through. The second was an M.Ed. Course entitled: Language,
Literacy and Learning from a Social Constructivist Perspective. Unusually, I
taught two sections of this course simultaneously, one in the regular face-to-
face mode, and the other in a city some 60 miles from Toronto, where I was
teaching at the time. For this second section, I negotiated a mixed format,
with some face-to-face meetings and the rest of the classes in distance mode.
Although it was with the latter group that the experiment I am about to
describe originated, I immediately made the same change in the doctoral
seminar and it is the results of the experiment with the seminar group that I
shall present in the remainder of this paper.
4. SOCIOCULTURAL THEORY AND ITS

APPLICATIONS TO EDUCATIONAL PRACTICE
AND RESEARCH
For most if not all of the students who take this course, cultural historical
activity theory is a completely unknown field. While most have heard of
Vygotsky, their acquaintance with his ideas is usually second-hand, gleaned
from passing references to his work in courses on other subjects. The early
meetings of the class are therefore demanding and often somewhat
frustrating. There are many new ideas to be grappled with and, as is often the
case with a large and comprehensive theory, each aspect only makes sense in
relation to some understanding of the theory as a whole. Yet this is precisely
what students do not have in the early weeks. In this context, the immediacy
of face-to-face discussion is an advantage, as it allows problems to be
addressed as soon as they arise; misunderstandings to be resolved; and
conjectural interpretations explored and modified where necessary. On this
occasion, my impression was that we established preliminary agreement
about the key ideas and their inter-relationships quite quickly. So, in the
third session, I judged that the introduction of written dialogue between our
weekly meetings would be a useful addition.
I had already set up a Web KF at the beginning of the course and, after a
brief introduction; we had exchanged some messages in the “hello” view. On
Web KF, a “view” is the entry to a discussion thread. New views can only be
added by the teacher of the course, but students can ask for views to be
added as they see fit. When a new Web KF starts up a “hello” view is
already in place. At the end of the class, I announced that I would be adding
a new view on the topic of the papers we had just discussed and I invited all
class members to continue our discussion on the web.
At this point, it may be useful to give a brief description of the Web KF
and to explain some of its salient features. Class members choose their user
name, which may be a pseudonym, and they control their password. On
logging in they are presented with a list of views currently active and, down
the left side, the commands that are available. By clicking on one of the
views displayed they call up the list of notes so far posted to that view with
the menu of available commands arranged across the top of the page (see
figure 2). The default arrangement shows the notes in their temporal order of
posting but with an indication of any links between notes that have been
created by the use of the “build-on” feature. This feature allows a new note
to build on an existing note, the link being shown in a footnote at the end of
the note to which the built-on note is attached and also at the end of the
built-on note. Where notes are linked in this way, the display shows them in
a cascade, with each build-on indented one-tab space to the right of the
296 Gordon Wells
parent. It is also possible to select alternative displays in which notes are

arranged by author or by time and date of posting.
Navigation Note Administration Resources Security

All Views New Edit Accounts Send E-mail Announcements Preferences
Note
Selected Build- E-mail Personal Help Change
View On Addresses Directory Password
Search Edit Edit Views Shared Who Has
Note Directory Access?
Delete Add Web Logout
Note Announcements Resources
Figure 2: Display of Commands Available in the Web Knowledge Forum.
From the beginning, the web-based discussion was lively and all class
members participated. Each week, I added a new view for the topic under
consideration. However, this did not mean that all discussion of previous
topics dried up, new messages were added as long as people had new
thoughts to contribute. Nevertheless, most of the discussion each week took
place within the view that had just been added.
As the weeks went by, I began to feel a little disappointed. The notes
were all very interesting and they certainly addressed issues arising from the
readings and the oral discussion in class. There was also evidence that class
members were reading and building on each others’ notes. Notes that were
linked by the use of the build-on command produced a “thread” of linked
notes. However, the discussion was not as progressive (Bereiter, 1994) as I
had hoped. This was even more the case with the Web KF in which my
M.Ed. Class was participating. Something needed to be added that would
give a stronger focus to the web-based discourse.
5. THE ROLE OF THE OBJECT IN ACTIVITY
Not entirely coincidentally, a possible solution occurred to me as we

were discussing the chosen readings on activity theory, which included
articles by Leont’ev (1981) and Engeström (1991). It was, in fact, the
attempt to clarify the meaning of “object” in Engeström’s diagrammatic
representation of an activity system (see figure 3) that acted as the trigger.
Figure 3: Activity System Triangle (Engeström, 1991).
In activity theory, the object has two related interpretations. On the one
hand, it is the motive or goal of the activity (Leont’ev, 1981). On the other
hand, it is the entity that is acted upon by the subject, using such appropriate
cultural artifacts as are available in the situation as tools to mediate the
achievement of his or her action. The upper portion of the triangle thus
represents the key understanding that humans’ actions are performed by
agents-acting-with-mediational-means (Wertsch, 1998). The remainder of
the diagram sets the action within the larger context of an activity system,
giving recognition to the fact that actions are performed jointly with other
members of the community in order to achieve the object (i.e. motive) of the
activity as a whole. It also recognizes the different roles that individuals play
in the activity (i.e. division of labor) and the cultural rules and conventions
that both facilitate and constrain the manner in which the activity can be
realized in the specific social and historical context in which it occurs
(Engeström, 1990).
As we were sorting out the relationships between the constituent parts of
this diagrammatic representation of activity, I recalled another seminal use
of the term “object”. In one of their expositions of the role played by CSILE
in supporting students’ knowledge-building activities, Scardamalia, Bereiter
and Lamon (1994) make the distinction between accumulating knowledge
and working on knowledge in order to improve it. For classrooms to function
as knowledge building communities comparable in their goals to cutting
edge research and development communities in universities and industry,
298 Gordon Wells
they argued, students need to have an object that they are attempting to
improve, for example a functioning model that they are constructing or an
explanation of some phenomenon they are observing. In other words,
knowledge building requires that there be an “improvable object.”
(Scardamalia et al., 1994)
This seemed to be exactly what our class needed for our web discourse to
take on greater focus and for our attempts at knowledge building to lead to
real progress in developing a richer and more coherent understanding of the
relevance of CHAT for education. At the end of the session, I proposed that
we start a new view in which we would attempt to create and improve such a
“virtual object” (VO). The object would be (to build) an ideal learning
environment (ILE).
For the remainder of the course work on this object proceeded with
enthusiasm. When I subsequently asked class members to answer a brief
questionnaire about the role of the discourse on Web KF in the course (see
Appendix), they were in general very positive about its value. Although it
made extra demands on members’ time to read and contribute on an almost
daily basis, which some found to be more than they had bargained for, the
time was considered to be well spent. However, what I most wanted to know
was whether they had found working on the “improvable object” to be
different from discussion on the topics in other views. My own impression
was that the discussion had indeed been more focused and that there had
been a more sustained attempt to develop some of the ideas that were
proposed. In order to attempt a more objective evaluation of these hunches, I
turned to the data constituted by the notes that had been posted to the Web
KF throughout the course.
6. ANALYSES AND RESULTS
Over the course as a whole, discussion took place in relation to six views
(excluding the “hello” view). Class members varied in the extent of their
participation, both absolutely and in relation to the different views. To a
large degree, this was because of circumstances beyond their control, such as
computer breakdown or unavailability for interaction because of other
responsibilities. As already mentioned, one member of the class had to
withdraw for reasons that had also limited their participation in the preceding
weeks. When these factors are taken into account, there was no substantial
difference in class members’ commitment to participating in the Web KF
discourse. As far as my own participation was concerned, I tended to
contribute to threads already under way in the same way and to the same
extent as other members. In general, however, I did not initiate new threads,
except to post references to other relevant information on the Internet and, in

some cases, to quote passages from related material I had been reading. In
sum, each view included notes posted by six members, including myself.
The Thanksgiving holiday, which in Canada occurs early in October,
meant that there was a gap of two weeks between the introduction of the
second and third topics. As a result, a “scientific concepts” view was the
focus of discussion for about twice as long as for any of the other topics
except the VO. The latter was introduced at the point when all but one of the
main themes of the course had been introduced. Thereafter, our weekly
meetings addressed topics that student members selected, making
connections between the theory addressed in the earlier part of the course
and applications to particular research or teaching concerns that were
important to them. These later topics were not each given a new “view”, but
they certainly added to the breadth of the discussion around the creation of
the ideal learning environment.
The first, and most obvious, way of comparing the discourse that
occurred in the VO view with that in the views that proceeded was to
generate a number of quantitative measures. These are presented in table 1.
Table 1: Descriptive Statistics on Notes in View.
Mean %
# of Mean %Links
St. # of Length Links
Topic Note Length to Other
Dev. Threads of to
s of Notes Notes
Threads Texts
Semiotic mediation 15 222.1 179.6 4 3.8 40 13.3
Scientific concepts 45 291.4 194.1 7 6.4 73 6.7
Learning in the ZPD 28 278.6 142.6 5 5.6 93 32.1
Genetic approach 19 260.2 224.7 6 3.2 53 31.6
Activity theory 9 224.9 102.4 3 3.0 56 22.2
Virtual object 37 412.0 293.5 4 9.3 95 24.3
The first difference to note is in the average length of contributions. This

is substantially greater in the VO view than in any of the others. In all cases
the standard deviation of length is high. However this variability is to be
expected in a conversation-like medium, since some notes are no more than
brief rejoinders to questions asked or observations made in preceding notes.
The second interesting finding is the mean length of threads, calculated
as the number of notes in the view divided by the number of threads. On this
measure, the VO view was far ahead of the others, having a mean length
almost 50% greater than its closest rival, scientific concepts. It is possible, of
course, that in the early stages, course members did not always remember to
use the “build-on” command, even when it would have been appropriate.
300 Gordon Wells
However, that explanation does not seem plausible, as the second highest
mean length of thread was in the second view, while the two lowest values
were for the two penultimate views. Further evidence of the tighter linking
of messages and of the greater cohesiveness of the threads is to be seen in
the average number of links and references to preceding notes per note. Here
again, the VO view came out ahead by a comfortable margin. Interestingly
the other measure of linking, mean proportion of links to texts beyond those
that were required reading, was not as high as that found in two of the other
views. Nevertheless, when all these measures are considered together, it does
seem that working on the improvable VO led to more focused and cohesive
discourse than did the discussion of any of the other topics.
By their nature, however, the results of these quantitative analyses can
provide very little information about the quality of the discourse. To
investigate this, I adopted a different approach. Reading through a printout
of the notes sequentially, I identified each new major theme that was
introduced (i.e. a sub-topic within the overall topic of the view) and, using
different colored markers, I marked each occurrence of a substantive
contribution to each of these themes. Then I selected one of the earlier views
to discover the way in which the contribution to themes was distributed over
the messages that were posted to that view. I selected the view “learning and
teaching in the zone of proximal development” (ZPD), as it occurred about
half way through the course and, by comparison with other views, was
intermediate in number of contributions and mean length of threads. In table
2, the vertical axis indicates to which of the identified themes each note in
the latter view contributed, while the horizontal axis shows the temporal
sequence of the notes’ occurrence (shown by numbers in row one and by the
initial letter of the contributor’s name in row two).
As can be seen from table 2, the distribution had two salient features.
First, there were more themes than notes. Although there were only 28 notes
in total, 38 different themes could be identified. Second, with a few
exceptions, the themes were continued over only two or three notes, which
can be seen from the distinct diagonal pattern in table 2. This pattern was
typical of the early views, in that participants tended to “edge-match” their
notes, starting by connecting to a theme mentioned in the preceding note and
then continuing by introducing one or more new themes of their own.
Table 2a: Distribution of Themes by Notes in the ZPD view.

Themes 1 2 3 4 5 6 7 8 9 10
Contributors D U M G U D M M G C
1 Cooperative learning X
2 Conflict/dissonance X
3 Learning thru conflict X
4 Challenge
5 Cognitive devt.
6 Identity X X
7 Holistic learning X X X
8 Emotions and values X X
9 Motivation
10 Non-verbal modalities X X X X
11 Logocentrism X
12 Computerized learning X X X
13 Video conferencing X
14 Emotions ‘on-line’ X
15 Being v. doing X
16 Compartmentalization X
17 Contexts X
18 Interactive relationship X
19 Individual differences
20 Cultural differences
21 Anxiety
22 T’s awareness of affect
23 Contingent response
24 ZPD a ‘delicate zone’
25 Order from chaos
26 Resolving dissonance
27 Scientific concepts
28 Revolutionary activity
29 ZPD classroom
30 Textbooks in ZPD
31 Assessment in ZPD
32 Feedback
33 Diagnostic activity
34 Summative assessment
35 Teacher’s ZPD
36 Delivery v. facilitation
37 Current educ. System
38 Political reality
39 References X X X X
302 Gordon Wells
Table 2b: Distribution of Themes by Notes in the ZPD view.

Themes 11 12 13 14 15 16 17 18 19 20
Contributors M D U D E D G C M C
1 Cooperative learning
2 Conflict/dissonance X X X
3 Learning thru conflict
4 Challenge X
5 Cognitive devt. X
6 Identity
7 Holistic learning X
8 Emotions and values X X X
9 Motivation X
10 Non-verbal modalities
11 Logocentrism
12 Computerized learning
13 Video conferencing
14 Emotions ‘on-line’
15 Being v. doing
16 Compartmentalization X
17 Contexts
19 Individual differences X X
20 Cultural differences X X
21 Anxiety X
22 T’s awareness of affect X
23 Contingent response X
24 ZPD a ‘delicate zone’ X X X
25 Order from chaos X
26 Resolving dissonance X X
27 Scientific concepts X
28 Revolutionary activity
29 ZPD classroom
30 Textbooks in ZPD
31 Assessment in ZPD
32 Feedback
33 Diagnostic activity
34 Summative assessment
35 Teacher’s ZPD
36 Delivery v. facilitation
37 Current educ. System
38 Political reality
39 References X X X X X
Table 2c: Distribution of Themes by Notes in the ZPD view.

Themes 21 22 23 24 25 26 27
Contributors U G D C E D E
1 Cooperative learning
2 Conflict/dissonance
3 Learning thru conflict
4 Challenge
5 Cognitive devt.
6 Identity
7 Holistic learning X
8 Emotions and values X
9 Motivation X
10 Non-verbal modalities
11 Logocentrism
12 Computerized learning
13 Video conferencing
14 Emotions ‘on-line’
15 Being v. doing
16 Compartmentalization
17 Contexts
19 Individual differences
20 Cultural differences
21 Anxiety
22 T’s awareness of affect
23 Contingent response X
24 ZPD a ‘delicate zone’
25 Order from chaos
26 Resolving dissonance
27 Scientific concepts
28 Revolutionary activity X
29 ZPD classroom X X
30 Textbooks in ZPD X X
31 Assessment in ZPD X X X X
32 Feedback X
33 Diagnostic activity X
34 Summative assessment X
35 Teacher’s ZPD X
36 Delivery v. facilitation X
37 Current educ. System X X
38 Political reality X
39 References X X
304 Gordon Wells
In order to discover whether the distributional pattern of contribution to

themes in the VO view was different from that in the preceding views, I
carried out the same procedure on the 37 notes that were contributed to the
VO view. The results were very interesting. Not only were proportionally
less themes brought into play in the VO view, but those that were introduced
were returned to frequently throughout the discussion. In addition, most
notes contributed to several themes. These features of the VO discussion can
be clearly seen in table 3.
Table 3a: Distribution of Themes by Notes in the Virtual Object View.

Themes 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Contributors E A M D E G A C M M C C D A
1 Building the X X
ILE
2 Feast X X X X
metaphor
3 Co- X X
construction
4 T & S as X X X X X
learners
5 Community X X
of learners
6 Teachers X X
reflecting
7 Teachers X X
characteris-
tics
8 Encapsula- X X X
tion
9 Zone of X X X X X X
proximal
devt.
10 Buddies / X X X X X X
Mentors
11 Lifelong X X
learning
12 CMC X
13 Physical X X X
space
14 Assessment X
15 Diversity of
learners
16 Practicality X X X
17 Organization X
18 Virtual X X
reality
19 References X X
Table 3b: Distribution of Themes by Notes in the Virtual Object View.

Themes 15 16 17 18 19 20 21 22 23 24 25 26
Contributors C D E M E D M C A M G D
1 Building the
ILE
2 Feast X X X X X
metaphor
3 Co- X
construction
4 T & S as
learners
5 Community
of learners
6 Teachers
reflecting
7 Teachers
characteris-
tics
8 Encapsula- X X X
tion
9 Zone of X
proximal
devt.
10 Buddies / X X X X X X X X X
Mentors
11 Lifelong
learning
12 CMC X
13 Physical X X
space
14 Assessment
15 Diversity of
learners
16 Practicality X X X X X X
17 Organization X X X X
18 Virtual X X
reality
19 References X
306 Gordon Wells
Table 3c: Distribution of Themes by Notes in the Virtual Object View.

Themes 27 28 29 30 31 32 33 34 35 36 37
Contributors E C A E M E G G E E G
1 Building the X X X X
ILE
2 Feast
metaphor
3 Co- X
construction
4 T & S as
learners
5 Community
of learners
6 Teachers
reflecting
7 Teachers
characteris-
tics
8 Encapsula- X
tion
9 Zone of X X X X
proximal
devt.
10 Buddies / X
Mentors
11 Lifelong
learning
12 CMC
13 Physical X
space
14 Assessment
15 Diversity of X X
learners
16 Practicality
17 Organization X X X X X
18 Virtual X X X
reality
19 References X X X
Several further features brought out by this display are worth

commenting on. Theme 1, the encouragement to build, was introduced at the
very beginning and then again, in some desperation, near the end, when the
same contributor tried to refocus colleagues by proposing that the object to
be built should be a particular type of high school, as much of the preceding
discussion had been concerned with education in general. Interestingly, it
was this same contributor who tended to stress the need for practicality
(theme 16) and who most frequently suggested that the group should
consider matters of organization, such as timetable and freedom of

movement (theme 17).
The second theme was also interesting. Introduced in note 2, this theme
metaphorically described the ILE as a feast and this metaphor was sustained
over several notes. In note 18, with a slight deflection of the original
metaphor, it was suggested that meeting in a bar was conducive to
knowledge building and, in subsequent notes, this theme was appropriated
by several contributors who imagined themselves (or actually were) drinking
excellent wine as they wrote their notes. Clearly, for this group of teachers,
education should ideally be a feast at which to regale oneself, as they seemed
to find themselves doing during this knowledge building activity.
Themes 3, 4, 5, 9 and 10 are also of interest. A key feature of CHAT is
that learning occurs in the context of collaboratively undertaken activity, in
which the learner receives assistance in what Vygotsky (1978, 1987) called
“the zone of proximal development”. Introduced in the first note, this theme
(9, linked to 3, 4 and 5) persisted throughout the discussion and was
particularized, in theme 10, in the proposal to include the practices of
“buddying” and “mentoring”. Several notes explored the possibility of
learners receiving assistance from buddies or mentors beyond the walls of
the school, such as retired persons, family members and experts in relevant
fields. It was in this context that the ideas of lifelong learning (theme 11) and
using computer-mediated communication (theme 12) were introduced and
immediately accepted. In contrast, theme 8 drew attention to the
“encapsulated” nature of contemporary schooling (Engeström, 1991) and to
the ways in which many categories of actual and potential participants were
excluded from the feast. Conversely, the need to recognize and value
diversity (theme 15) was emphasized at the end of the discussion.
One final theme is worth commenting on. At various points throughout
the discussion, group members questioned the limits on building a virtual
ILE (theme18). How far should the current realities of schooling constrain
the construction of an ILE? The general feeling of the group is captured in
this extract from Daphne’s note 20:
“I don’t think Andrew was proposing to ‘abandon’ the conventional form
of education, but rather he was indicating that the conventional school
could be expanded by … for example, utilizing the modern technology
that transcends borders and that allows us to tap into the wealth of
resources (knowledge, skills, expertise) possessed by others, through an
association with those who are willing to share them.”
Daphne then goes on to point to a number of practical matters that would
need to be considered in acting on Andrew’s suggestion. Obviously, as a
whole, the group favored evolution rather than revolution. However, this is
308 Gordon Wells
perhaps not surprising when it is recalled that all of them were professional
educators.
For the final analysis, I sorted contributions to the different themes by
author. I was interested to discover whether the persistence of particular
themes resulted from individual members having their own axes to grind or
whether it represented a shared concern. The results, which can also be
gleaned from table 3, were conclusive. Apart from the theme of assessment
(14), which was only mentioned once (in note 1), and that of teacher
characteristics (theme 7), which occurred twice, in conjunction with the need
for both teachers and students to be learners (theme 4), most other themes
involved at least three contributors. And the more frequently a theme was
mentioned, the greater the proportion of the group who contributed to its
discussion.
7. CONCLUSION
Taken in conjunction with the course members’ responses to my

questionnaire, the findings reported above allow me to draw two tentative
conclusions. First, the inclusion of written discourse on the Web KF
significantly enhanced the knowledge building opportunities provided by the
course. And second, the introduction of an improvable object in the form of
the building of an ILE to focus discussion was relatively successful. But both
these conclusions pose further questions about the role of the teacher in
ensuring that these objectives are achieved. I shall address each of these
points in turn.
7.1 Written Discourse on the Web Knowledge Forum
In preparing to carry out the analyses reported above, I conducted a

limited Internet search to find out whether others had engaged in similar
inquiries. One article I found (Turoff, 1999) was of particular interest, as it
reported success in incorporating computer mediated communication (CMC)
in regular face-to-face classes over a period of years. Since the classes were
relatively large, one of the most obvious benefits Turoff noted was that, by
extending discussion outside class meetings, all students could be actively
involved. He then continued:
“More important, students could take time to reflect on the ongoing
discussion and provide thoughtful contributions. This generated more
comprehensive discussion than what occurred in other face-to-face
classes. Furthermore, students for whom English was a second language
became equal participants and had the ability (as we observed) to reread
the discussion as much as needed before replying.” (1999)
This same benefit was mentioned by several class members in answering
my post-course questions.
“The Web Forum allowed us to extend our discussions beyond the
classroom without having to wait a week until we met again. This was a
great benefit since in many cases there were thoughts that didn’t
materialize until after our classroom discussions had ended or time ran
out. Certainly we could have simply jotted them down on paper and
brought them to the next class, but it’s much more satisfying to continue
the discussion electronically while everyone’s contributions are still fresh
in mind.” (Maurice)
“If dialogue does facilitate meaning, as I believe it does, then our

discussions on the web forum allowed us to extend our dialogue beyond
the constraints of class time. In that way, we could reflect on what was
discussed in class, and then bring our additional questions to class
members via web kf for further reflection and dialogue.” (Carol)
However, one member added another advantage that I consider
particularly important:
“Often, we don’t get to share our experiences with the material in class
(for various reasons: e.g. fear of embarrassment, fear of disapproval,
inappropriate time/place, etc.). I think it’s valuable for students to share
their direct relations to the material and the classroom discussions in a
way that is non-threatening. I believe that this encourages community-
building.” (Martha)
New theory has to be connected to personal experience if it is to be
understood, in the sense of affecting and interpreting future action. As
Martha points out, the Web KF allows this to happen in a dialogue with
colleagues who have similar, or at least related, experiences.
7.2 Working on an Improvable Virtual Object
In his article, Turoff also made reference to “allowing a group of students

to develop a collective model of a complex situation.” (1999) This
suggestion was made, it is true, in his discussion of methods of evaluating
student learning. But this is not as far removed from the improvable object
as might appear, since we would both agree, I believe, that such
310 Gordon Wells
collaborative knowledge building allows students both to use/improve and to

make evident to others what they have come to understand.
In retrospect, I realize that working on the VO could have been still more
effective if some guidelines had been provided in advance on how to
proceed. However, on this occasion, I was interested to see how the course
members would tackle these problems themselves. The following comments
both identify some of the difficulties they encountered and suggest some
strategies for future building operations.
“It is evident that both the ‘thing’ we were trying to create and the
characteristics we were assigning were difficult for us to agree upon. I
don’t remember stating up front that we were to agree (perhaps my
memory is failing me!) but there seemed to be that understanding. To
move forward in creating a virtual object we needed to start somewhere.
After struggling on web kf for sometime, tossing ideas back and forth, we
did agree in class one day that our virtual object would be a secondary
school. From there we brought together some of the ideas that had
previously been floating in cyberspace. Combining the web kf
discussions with our in-class discussions facilitated the process of
building our virtual object.” (Carol)
“Students should develop a set of guidelines in face-to-face conversation

before engaging in such creative activity on-line, otherwise much time is
‘wasted’ in frivolous exchanges developing the ‘rules’ and ‘objectives’ of
the project. ... It would be a good idea to set aside a small portion of time
during class to have students verbally discuss their contributions and the
development of the object under construction. This would help bring the
discussion back to earth ... nothing seems to be quite as sobering as a
good face-to-face talk.” (Maurice)
Although not considered here, feedback from the students who took the
concurrent M.Ed. course in the combined distance and face-to-face mode
supported the same conclusions. In fact, in some ways they were more
successful in moving to the stage of actually specifying the design features
of their ILE, because they did develop their own guidelines in a face-to-face
meeting.
7.3 The Role of the Teacher
One issue that concerns most educators who include web-based

discussion as part of the courses that they teach is how they themselves
should participate. From studies reported to date, it seems clear that simply
setting up the facility is not sufficient, if the intention is for students to
engage in genuine dialogue that advances their understanding of the course

content. However, if the teacher responds to every student contribution, this
sets up a many-to-one dialogic mode in which the teacher becomes the focus
of discussion, as is typical in face-to-face classrooms. My own practice has
been to try to be a contributor rather than a leader, but I have also felt the
need to act sometimes as an “agent provocateur”, by introducing alternative
perspectives, raising problems or nudging contributors to explore the
implications of their positions. In acting in this way, I am trying to put into
practice what I understand by “working with the students in their zones of
proximal development”, which is one of the key tenets of CHAT and a
central theme of the courses I teach. But I do not consider that I have yet
been completely successful and this will continue to be an ongoing object of
my action research.
In the present inquiry, my focus was on the effect of introducing an
improvable object as both motivator and goal of discussion. My interest was
in observing how the course members would take up the challenge without
further intervention on my part. The results were both interesting and
encouraging. However, in the light of the evidence, I believe I could have
usefully played a more active role in the construction process by initially
defining the parameters more precisely and by prompting the group to select
a particular type of institution to concentrate on. Next time I propose
working on a VO, I shall be alert to ways of helping the group to define their
object more speedily.
But for me perhaps the most significant outcome of this practice-based
investigation was the evidence it provided of the dialectic relation between
theory and practice. It was the need to find a way of encouraging course
members to make the discussion more progressive that set the project in
motion, but it was the theory-oriented readings that suggested the direction
to take. At the same time, by treating the ensuing discussion as data for
analysis, I both created an object for further improvement and enhanced my
own understanding of the central role of the object in joint activity, whether
the motive of the activity be practical or theoretical. Indeed, I have come to
understand more fully that this is a false dichotomy, since all activity is
simultaneously practical and theoretical, as are the possibilities for
improvement of the artifacts created or modified in the process. Finally,
given the focus of the course, I was particularly encouraged by Carol’s
recognition that “using the web kf helped us to enact some of the deeper
meanings of socio-cultural theory.”
312 Gordon Wells
ACKNOWLEDGEMENTS
I should like to thank the students involved in the course that provided
the data for this paper, both for our interesting discussions and for their
permission to draw on them in carrying out this investigation. Individual
students are referred to by self-chosen pseudonyms. I should also like to
thank Merrill Swain for her helpful comments on an earlier draft.
APPENDIX: QUESTIONS TO INVITE

REFLECTION ON WEB FORUM
My proposal is to ask you to send me your responses by e-mail; then, when all responses
have been received, I will put them on the web forum so that we can, if we wish, continue the
discussion. What do you think?
1. Did the use of the Web Forum contribute to the course? Please explain.
2. Did you act and feel differently when participating in the creation of the Virtual Object as
compared with participating in the other views? Please explain.
3. Please evaluate the use of the Web Forum as a component of the course. Should it be
included in future courses and, if so, are there any ways in which its use could be improved?
4. Any other comments you would like to make.
REFERENCES
Bereiter, C. (1994). Implications of postmodernism for science, or, science as progressive
discourse. Educational Psychologist, 29 (1), 3-12.
Engeström, Y. (1990). Learning, working and imagining: Twelve studies in activity theory.
Helsinki: Orienta-Konsultit.
Engeström, Y. (1991). Non scolae sed vitae discimus: Toward overcoming the encapsulation
of school. Learning and Instruction, 1, 243-259.
Halliday, M.A.K. (1993). Towards a language-based theory of learning. Linguistics and
Education, 5, 93-116.
Halliday, M.A.K., & Matthiessen, C.M. (1999). Construing experience through meaning: A
language-based approach to cognition. London: Cassell.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New
York: Cambridge University Press.
Lemke, J.L. (2002). Becoming the village: Education across lives. In G. Wells & G. Claxton
(Eds.), Learning for life in the C21st: Sociocultural perspectives on the future of education
(pp. 34-45). Oxford: Blackwell.
Leont’ev, A.N. (1981). The problem of activity in psychology. In J.V.Wertsch (Ed.), The
concept of activity in Soviet Psychology (pp. 37-71). Armonk, NY: Sharpe.
Resnick, L. (1987). Learning in school and out. Educational Researcher, 16(9), 13-20.
Rogoff, B. (1990). Apprenticeship in thinking: Cognitive development in social context. New
York: Oxford University Press.
Scardamalia, M., Bereiter, C., & Lamon, M. (1994). The CSILE project: Trying to bring the
classroom into World 3. In K. McGilley (Ed.), Classroom lessons: Integrating cognitive
theory and classroom practice (pp. 201-228). Cambridge, MA: MIT Press.
Turoff, M. (1999). An end to student segregation: No more separation between distance
learning and regular courses. Invited plenary for the Telelearning 99 meeting, Montreal,
Canada [On-line]. (Accessed 14/07/02). URL: http://eies.
njit.edu/~turoff/Papers/canadapresent/segregation.htm
Vygotsky, L.S. (1978). Mind in society: The development of higher psychological processes.
Cambridge, MA: Harvard University Press.
Vygotsky, L.S. (1981). The genesis of higher mental functions. In J.V. Wertsch (Ed.), The
concept of activity in Soviet Psychology (pp. 144-188). Armonk, NY: Sharpe.
Vygotsky, L.S. (1987). Thinking and speech. In R.W. Rieber & A.S. Carton (Eds.), The
collected works of L.S. Vygotsky, Volume 1: Problems of general psychology (pp. 39-285).
New York: Plenum.
Wells, G. (1999). Dialogic inquiry: Towards a sociocultural practice and theory of education .
Cambridge: Cambridge University Press.
Wertsch, J.V. (1998). Mind as action. Oxford: Oxford University Press.
Testing Social Information Processing Theories in
Distance Education
Wm. Benjamin Martz, Jr. & Morgan M. Shepherd

Information Systems Department, College of Business, University of Colorado at Colorado
Springs, USA
1. INTRODUCTION
“Technology serves best when tailored in support of human values,

meaning and purpose.” Peter and Trudy Johnson-Lenz in Groupware for
the 21st Century
A combination of stakeholder expectations and an evolving learning
environment drive the interest in distance education. Technology is the key
enabler of distance education by removing the geographical and temporal
constraints imposed by traditional classroom environments. Today, distance
education is struggling to manage and to meet the expectations of all of its
stakeholders. For example, state organizations want to use taxpayer dollars
efficiently; instructors want to present course topics for effective learning;
and students want to use their learning to enhance their careers. It is safe to
say that ultimately all parties want the best outcome. The biggest problem is
that no one is sure of all the parts or of how those parts fit together.
For the purpose of this discussion, distance education is defined as
creating a learning environment that facilitates structured learning without
the traditional practice of face-to-face interaction on-campus. Strictly, the
practice of rural teachers traveling between remote settlements to educate
children in their homes meets the spirit of the definition. However, in
today’s world, distance education usually implies some sort of technological
support through the Internet, e-mail, and videoconferencing.
315
316 Wm. Benjamin Martz, Jr. & Morgan M. Shepherd
As distance education becomes more viable, the practice of traditional

education is changing also. One pedagogical model receiving significant
attention is called “learning centered education”. This “places learning and
learners at the core of the educational process” (Bilimoria & Wheeler, 1995,
p. 327). Essentially, a learning partnership is created whereby the teacher
identifies what needs to be learned and the student helps identify the means
by which their own learning occurs.
In summary, distance education is not a new concept. However, the
unprecedented growth in technology and the pedagogy changes are powering
its growth and new emphasis. Internet, e-mail, videoconferencing, etc. are
enabling distance education to take a firm hold technically.
2. SOCIAL COMPONENT CONCERNS OF

LEARNING
As we scrutinize the learning process, one of the fundamental concerns is

when does learning actually occur. John Locke (Chaplin & Krawiec, 1960)
believed that humans start with a blank slate, a tabula rasa, and that human
beings write their experiences on that slate throughout life. Early researchers
in the field of psychology (Ebbinghaus, 1913; Gulliksen, 1934; Hull, 1943)
picked up on this notion and tried to measure learning quantitatively. “How
much was being added to the tabula rasa?” More recently, other researchers
argue that real learning requires a social context. Papert (1980) makes his
case in the book, Mindstorms, as he relates teaching computer programming
logic to children. Social learning theory (Bandura, 1977) emphasizes the
interaction of people both with other people and with the environment. Hills
and Francis (1999) conducted their research with respect to computer-based
training (CBT). Somewhat paradoxically, their findings showed that CBT
environments were more successful when they included more social context.
Much like CBT, which is generally “a solitary experience which takes
place away from the real job” (Hogan & Kwiatkowski, 1998, p. 1404), the
distance environment will be required to take into account the social aspects
of learning. Hogan and Kwiatkowski (1998) report on the social aspects of
teaching in large groups in the UK. Their argument includes the premise that
technology can handle the activity of teaching to large groups, but that the
emotional aspects of this teaching method have been ignored. Similar
concerns are raised in Australia, where technology has been supporting
distance teaching for many years. Hearn and Scott (1998) argue that before
adopting technology for distance teaching, technology must acknowledge the
social context of learning.
Social Information Processing Theories in Distance Education 317
For example, Haythornthwaite and colleagues (2000) coined the term

“fade back” to describe students that do not participate in the distance class.
They point out that fading back is accomplished more easily in distance
learning environments where the number of social cues is reduced. These
cues include text without voice; voice without body language; class
attendance without seating arrangements; and students signing in without
attending (Haythornthwaite et al., 2000). This implies that in similar classes,
with similar levels of student interest, the likelihood of students fading back
is greater in distance learning classes than in face to face classes.
The issues around implementing distance education are not just
conceptual. Students want to obtain skills for better jobs and employers want
to obtain graduates with better skills. In a research study comparing recruiter
and student perception of key skills necessary for employment and career
advancement, several of the top skills required social interaction (Martz &
Landof, 2000). Teamwork, project management and leadership skills ranked
in the top five characteristics desired by recruiters. These skills require
significant interaction with others to be successful and are therefore usually
considered to be “socially driven.”
In summary, distance education may create significant problems on two
quality-oriented fronts. These are learning itself and learning the specific
skills desired by employers.
3. HOW TECHNOLOGY AND PEOPLE INTERACT
As is the goal of all education, the goal of distance education is to help

people learn. However, more technology is required for distance education
to occur. Technology enables distance education. In turn, this leads one to
consider how people, while in a learning environment, interact with
technology. Looking at theories from each of these areas, learning and
technology-people interaction can help guide our investigation.
First there are theories that have developed around how individuals are
motivated to learn. These are labeled foundation theories because the
motivation to learn must exist before distance learning can be effective.
Second, there is a set of theories concerning how individuals and groups
interact with technology. These theories are labeled technology interaction
theories because they predict how individuals and groups interact with
technology. Commonalities across these two sets of theories (see figure 1)
can help identify issues faced by distance education.
Technology Interaction
Media Richness Theory
Social Information Processing Theory
Adaptive Structuration Theory
Foundation Theories
Field Theory
Social Exchange Theory
Needs Theory
Figure 1: Commonalities across Two Sets of Theories.
3.1 Foundation Theories
Theories such as field theory, social exchange theory and needs theory
represent foundation theories and try to explain the various behaviors
exhibited by individuals in groups. Briefly, field theory (Lewin, 1951) looks
at the group dynamics caused by how individuals behave to overcome
barriers as they try to achieve a goal. Groups with their large number of
possible interactions complicate this behavior. Social exchange theory
describes behavior in terms of their rewards and costs. “Rewards are
pleasurable outcomes associated with particular behaviors; costs include
such things as mental effort, anxiety or even embarrassment.” (Beebe &
Masterson, 1986, p. 30). Viewed in this economic way, individuals will seek
to establish relationships that accrue a net reward. In addition, individuals
will enact behaviors to continue those relationships with a positive value.
In 1943, Maslow presented a first, rough draft of his need hierarchy
theory. Essentially, the theory, along with derivatives, such as Aldefer’s
(1972) existence-relatedness-growth, proposes that human beings have a set
of “staircased” needs that interact and “combine with [other] biological,
cultural and situational factors to determine behavior.” (Miner, 1980, p. 19).
Of interest to us in this discussion is the need in which Maslow includes the
desire for a sense of affiliation and general belongingness, broadly termed
the “love needs” (Miner, 1980). The “need for affiliation” has been found to
be a key ingredient in helping with achievement motivation (McClelland,
1961; Shipley & Veroff, 1952) and has been related to job satisfaction
(Miner, 1980). Complementing this idea is that of a reference group whereby
an individual expresses his or her need for affiliation. Kelley (1951)
identifies two such groups. The first was termed a peer group, or
comparative group, to which an individual compares and contrasts his or her
stature to other members of the group. The second group was termed a
reference group with which an individual identifies with the same general
feelings of the other members of the group. For example, a student
comparing a test score to other students in the class is using a comparative
group, while a senior student expressing their excitement with this year’s
graduation would have graduating seniors as a reference group.
3.2 Technology Interaction Theories
Another important reference area for distance education is one called

group support systems (GSSs). GSSs add a technology component to group
work, much like distance education is adding a technology component to
learning (DeSanctis & Gallupe, 1985). As such, GSS research has been a
major contributor to the body of knowledge concerning group interaction
with technology and learning. In GSSs, manual methods for meetings and
group work (Osborn, 1953; Delbecq & Van de Ven, 1971; Linstone &
Turoff, 1975; Saaty, 1980) have been automated and the interaction of
technology with people studied (Briggs et al., 1998; Fjermested & Hiltz,
1999).
As GSS researchers studied the impacts of automating these processes,
theories evolved. Essentially, face-to-face meeting environments are said to
have many mediums to communicate information. These include verbal (i.e.
tone of voice, volume) and non-verbal (i.e. body language, facial
expressions). As technology is added to the process, it produces a lens that
impacts the media by either restricting the flow of information or missing
some of the information completely. This is illustrated in figure 2, which
serves as the basis for further discussion.
b.
C U E S
a. L
e
n Group Process
s
Figure 2: Technology Lens.
Social information processing theory (SIP) (Chidambaram, 1996;

Walther, 1992) and adaptive structuration theory (AST) (Poole & DeSanctis,
1990) suggest that the technology, the lens, is a limiting factor or constraint
for communication. In its own way, each theory discusses how the sender of
the cues adapts to the lens. SIP suggests that, over time, the sender will be
able to get all the cues. It simply takes more time. The sender chooses to use
the limited channel to process the task-oriented communications first and
then, if time permits, the theory predicts that the more social
communications would follow. AST proposes a somewhat different tactic
but with the same ends. The sender figures out alternative ways to send the
cues based upon what is “allowed” through the lens. The term used for this
action is “appropriation” and “refers to the manner in which structures are
adapted by a group for its own use …” (Gopal et al., 1993, p. 47). A good
example here is how a sender of e-mail may use emoticons (e.g. :-> –
sarcastic smile, ;) – wink, :o – exclamation of surprise) to communicate
more about their emotion concerning the subject to the receiver. These
theories predict that the channel works and that social and task messages
ultimately make it through the lens over a period of time.
In contrast, media richness theory (Daft & Lengel, 1986) suggests that
the inherent characteristics of the technology filter out cues and ultimately
the cues do not make it through the lens to provide information to the
receiver or, in the case of GSSs, to the group process. The richness of the
media (i.e. the size of the lens) directly impacts on the effectiveness of the
technology in the situation. The capacity of the channel to provide
communication defines the richness of the channel. A face-to-face meeting
offers a richer medium than does a posted letter. Similarly, social presence
theory (Short, Williams & Christy, 1976) argues that since the lens limits the
amount of cues that make it through, most cues will be action or task-
oriented. In other words, the lens limits the social cues the receiver is able to
observe.
3.3 Current Environment
The debate still rages as to the effectiveness of distance education. The

complex environment contains issues around how students learn; what
motivates students to learn; how to make the learning environment better;
how to make the environment worthwhile for all the stakeholders; and how
technology impacts the learning environment. Several researchers and
educators are studying how distance education classes create a sense of
community (Dede, 1996; Haythornthwaite et al., 2000). A review of over
400 studies compared distance to traditional classroom instruction with
complex and conflicting results (Russell, 2001). On the positive side, faculty
at eCollege.com reported that their students learned equally effectively on-
line as they did on campus (TeleEducation NB, 1999). GSS studies are
identifying and reconfirming the important characteristics in developing a

sense of community, such as satisfaction (Chidambaram & Bostrom, 1993);
trust (Aranda, Aranda & Conlon, 1998); cohesiveness (Chidambaram, 1996);
and participation (Nunamaker et al., 1991).
However, distancelearning environments may be creating some
additional concerns such as higher rates of student anxiety and frustration
(Hora & Kling, 2001) and these environments may inherit the GSS “process
loss” characteristic of less consensus in decision making (Daft & Lengel,
1986; Briggs et al., 1998). The study of these complex issues and
interactions falls into the broad field of social informatics (Kling, 2000),
which is defined as the study of the interaction of information technologies
with an institutional and cultural perspective.
In summary, the foundation theories show a concern for the affiliation of
people in a learning environment. Technology has changed the environments
in which these theories were first developed. It is important to understand
how well these theories hold up in the new technology enhanced, distance
education environments
4. THE RESEARCH STUDY
With figure 2 in mind again, one would assume that the technology lens
in a distance class would be less media rich than that found in the classroom.
In addition, the need for affiliation between the individuals in two classes
should be the same. Combining these two assumptions, one would expect to
see more behavior geared toward creating or attaining social cues in the
distance learning class than in the face-to-face class.
This paper describes one study designed to look at this issue in more
detail. Using SIP theory as a basis, one can view the need for affiliation as a
social need that helps with learning. This alignment is best seen when
Chidambaram describes one underlying premise of the SIP theory this way:
“Implicit in the SIP perspective is the idea that users of computer media are
driven by these needs [affiliation motive – the need to like and to be liked by
others] just as much as those in non-computer settings.” (1996, p.146)
This study was designed to look for differences in the socialization or
need for affiliation between students in a distance MBA-level, introductory,
information systems course and students taking the same course in an on-
campus environment. The central conjectures are around the influence of
technology on learning or what Tu (2000) refers to as the “technological
social presence” of distance learning. The central premise is that students in
distance classes (DIST) would show more need for affiliation than would
students in on-campus classes (ONC). The following outcomes were

expected:
H1: DIST will be more inhibited by technology than ONC

H2: DIST will have higher levels of group cohesiveness than ONC
H3: DIST will have more participation in groups than ONC
H4: DIST will exhibit a greater need for affiliation
H5: DIST will report higher levels of satisfaction with groups and group
work than ONC.
A questionnaire with 42 items was given to both groups. The questions
asked about demographics (Q1-Q8); group experience in the course (Q10-
Q17); perception of technology (Q18-Q21); and perception of group work
(Q23-Q42). Short answer and seven-point Likert questions were used (1 =
Strongly Agree ... 7 = Strongly Disagree). An example of the questionnaire
is provided in the Appendix. The questionnaire was provided to both MBA
classes. As previously mentioned one class was conducted totally by
distance while the other was on-campus. All the students in the two classes,
25 from the ONC and 17 from the DIST, completed the questionnaire.
The analysis of the data is exploratory in nature and as such is subject to
several limitations. First, the questionnaire was developed to measure a
broad set of characteristics that interact in many ways. A factor analysis will
help decompose these characteristics for revising the questionnaire and
producing better questions. Second, the two classes were separated in time;
they were taught in different semesters and while the instructor and the
overall learning goals were the same, the exposure to topics varied
somewhat. Finally, the low number of responses demands a more
conservative statistical analysis that may not find subtle differences.
Those items discussed further, including those with statistically
significant results, are provided in table 2 below.
Consistent with the prior research in this area, the results from the study
are mixed. H1 was not supported. None of the differences between the
technology-specific questions (Q18-21, Q26, Q40) proved significant. The
technology did not hinder the DIST any more or any less than the ONC
group as H1 predicted. H2 was not supported. Differences in the responses
to Q41 and Q42, dealing with cohesiveness, did not prove significant.
H3 was supported weakly. Q30 dealing explicitly with level of
interaction did not have significant differences as would be predicted, both
groups disagreeing slightly with the proposition of low participation. The
ONC groups reported meeting more times per week than the DIST groups
(Q12), and more of the ONC time in meetings was spent on socialization
than task (Q13). Additionally, a significant difference was found concerning
fair share of group work. ONC groups reported more agreement with this
question than DIST implying that the level of participation created a fairer
workload distribution. This could be a confirmation of the fade back finding
because DIST students found it easier to opt out of group work. These
findings are contrary to the expectations that DIST groups have a higher
need for affiliation. However, support for H3 is found with the DIST groups
reporting having spent more time in meetings (Q11) and the DIST groups
having a greater percentage of that time on task than ONC groups (Q14). So
while DIST groups held fewer meetings than the ONC groups, they spent
more time in each meeting and more time on task in each meeting than the
ONC groups. In addition, whatever the level of participation created by the
DIST groups, that level was felt adequate for those group members to feel
fairly treated in regard to workload.
Table 1: Distance vs. On-campus characteristics.

Item Description (referenced by question number) On-campus Distance Sign.(1)
Average Average
06. How many hours do you spend each week on your 10.10 14.25 .005
MBA work?
08. What is your current GPA? 3.54 3.72 .000
11. On average, how many hours did you spend with 1.82 3.05 .008
your group each week?
12. On average, how many meetings did your group 1.35 0.75 .026
hold each week?
13. On average, what percentage of your meetings was 20.60 4.61 .001
spent on socializing?
14. On average, what percentage of your meetings was 79.00 84.00 .028
spent on the project task?
24. There was too much socialization in my group 5.92 6.61 .005
27. I was satisfied with this course 2.72 3.56 .074
29. I was disappointed in how the course worked out 5.44 4.61 .094
31. Everyone in my group did their fair share of work 2.50 3.66 .061
Not Statistically Significant
30.The level of interaction in my group was low 4.72 4.83 .699
32. I participated in this group more than I usually do in 4.56 4.39 .639
my other groups
41.I felt that I was really part of my group 2.00 2.61 .406
42.I would be willing to work in the same group again 2.60 3.0 .422
1. Mann-Whitney test – SPSS Ver. 10.4
The results for H4 were mixed. One would expect students with a need
for affiliation to exhibit that need by establishing more relationships to other
class members and these relationships would not be based upon the group
task in the class. Q5 tried to uncover just such relationships. Only three
subjects in each sub-group, ONC and DIST, responded that he or she had
looked to establish groups with other students in the class outside their
assigned work group. This level of response did not prove significant.
However, the differences in responses to Q24, “there was too much
socialization in my group,” did prove significant. The DIST group disagreed
more strongly that there was too much socializing. So, one interpretation is
that while the members of the DIST group did not seek out more
relationships, they may have wanted more socialization to occur in their
groups.
H5, higher satisfaction with the group or group work by distance classes,
is not supported. No group work characteristic measured by Q33-Q39
proved significant as predicted. With regard to the course itself, the ONC
group members seemed more satisfied than the DIST group members. First,
the ONC group disagreed more strongly that they were disappointed,
although they were less disappointed than DIST, with how the course
worked out. Second, the ONC agreed more strongly, they were more
satisfied than DIST, with the course (Q27). Overall, the DIST group was less
satisfied and more disappointed with the course than the ONC group.
One possible explanation for these mixed results is with expectations.
Students in these two groups may start their respective courses, distance and
on campus, with drastically different expectations in regard to many factors,
such as participation, fair share and group work. For example, an ONC
student may expect there to be 15 minutes of socializing at the beginning of
a meeting that takes an hour; that level is acceptable. Whereas, a DIST
student may find one socially-oriented comment out of every four comments
in a forum discussion unacceptable. There may be a need to calibrate or
standardize expectations between groups.
Another explanation revolves around possible inherent differences in
establishing a presence or a “perceived participation” in the two
environments. For example, if you attend a face-to-face meeting, you
establish your presence simply by being there. If you attend a distance
meeting, you must establish your presence more proactively such as logging
in. In addition, your continued presence is more obvious in a face-to-face
environment than the distance. Now the question becomes: “Would most
people equate logging in once and fading back equivalent to showing up to a
meeting and not contributing or does one ‘action’ create more perceived
participation than the other?” If the second case is true, then the two
environments require different levels of action or activity to create the “same
perceived participation.”
Finally, the two environments may be self-fulfilling. By this we mean
that DIST are perceived as task oriented with little or no socialization, while
ONC have the socialization factored in. Distance students may satisfy their
need for affiliation through other channels. Returning to Kelley (1951) as a
reference point, students in ONC may use classmates as a comparative group
on several social and task levels while students split their comparatives
groups between task (the distance course students) and social (their work or
social peer groups).
5. SUMMARY
In summary, as technology moves quickly to enable distance education

and learning environments, one concern that has to be dealt with is how
students best interact with the technology. In understanding this, one must
first understand the process of learning and then the process by which people
interact with technology. This paper has provided a brief review of some of
the theories in these areas. Socialization and its impact on learning were
prevalent themes throughout the reviews. This ultimately led to the proposal
to study differences in socialization between an ONC and DIST.
The amount of socialization between distance and on campus
respondents showed significance in two areas. First, the on-campus
respondents reported more socialization in their group meetings than did the
distance education respondents. Second, the distance education respondents
strongly disagreed that there was too much socialization, indicating a
willingness for more socialization. While both groups were on the “agree”
side of the scale when asked if everyone in their group contributed equally, it
needs to be noted that the DIST groups agreed significantly less. However,
more direct measures for differences in participation did not prove
significant.
This study provides the basis for future research in several ways. First,
there is some support from this preliminary data for the view that
socialization differences do exist between ONC and DIST students. Based
on this preliminary analysis, the questionnaire can be refined so that it can
better detect differences. A factor analysis of this questionnaire identifies six
factors within the data. In four cases, one of the statistically significant
characteristics identified in this study provides the anchor. Finally, the
understanding of the socialization needs associated with distance education
will help designers to provide a better learning environment.
APPENDIX : RESEARCH QUESTIONNAIRE

We are looking at ways to improve the MBA program and this survey will help us do that.
Please take a few minutes and answer this survey as it pertains to IS619. Some of the
questions may seem redundant. The reason for this is to ensure the accuracy of your answers
and to make sure this is a valid survey. As such, please read each question carefully and pay
particular attention to the scale you are using to answer the question. All data will be kept
confidential.
Demographics:
1. What is your gender? (circle one) male female
2. In which age group do you fall? (circle one) 21-25 26-30 31-35 36-40 41-
45 46-50 50+
3. How many semesters (including this one) have you been in the UCCS MBA
program? ______
4. How many MBA courses have you taken (including this semester)? ________
courses
5. If you have a job, how many hours do you work each week? _______
hours/week
6. How many hours do you spend each week on your MBA work? ______
hours/week
7. If you have one, what is your area of emphasis? _______________
8. What is your current GPA? _________
9. Some classes require a major group project. How many of these major group
project teams have you
been on (all MBA classes, including this semester)? __________ teams
You may have had a group project in IS619. If so please answer questions 10 – 14
relating to your experience with your group. If you did not have a required group
project, please continue with question 16.
10. Did the instructor assign groups or did the students pick their own group?
(circle one) instructor assigned student assigned
11. On average, how many hours did you spend with your group each week?
_______ hours/week
12. On average, how many meetings did your group hold each week? _________
meetings/week
For the purpose of this survey, we break down the time spent in meetings as time
spent on socializing, and time spent working on the assigned task. For questions 13
and 14 please indicate the percentage of time spent on socializing and the time spent
on task. The two percentages should total 100%.
13. On average, what percentage of your meetings was spent on socializing?
________ %
14. On average, what percentage of your meetings was spent on the project task?
________ %
Total 100%
15. Not including the groups that were assigned to complete the group project
requirements, did you form any group(s) with other students in the class? (indicate
by placing an X in the appropriate line)
______ Yes, it had _______ members
______ No, I did not form any other groups
It can sometimes take some time before a student feels comfortable with the
complexity encountered in a particular class. The following are some of the things
included in our definition of feeling comfortable: believing you can meet the
expectations outlined for the course; feeling you can ask a fellow student or the
instructor for help; following the lectures and participating in the discussions.
16. At what point during the course did you feel comfortable? (please circle one)
first day ¼ of the way through ½ way through ¾ of the way through
never felt comfortable
17. At what point during the course did you first ask non-trivial questions of fellow
students or the instructor (e.g. clarification or administrative questions about the
syllabus would be considered trivial whereas content-oriented questions for fellow
students or the instructor would be considered non-trivial)?
first day ¼ of the way through ½ way through ¾ of the way through
never asked questions
You can accomplish group work in a same-place environment (face-to-face) or at a
distance (group members do not meet face-to-face). Questions 17 – 21 refer to the
technology (i.e. email, telephone) you used to do group work at a distance. Referring
to the scale (1 = strongly agree, 4 = neutral, 7 = strongly disagree), circle the
appropriate response. If you always met face-to-face please skip ahead to question
22.
Strongly Strongly
Agree Neutral Disagree
1 2 3 4 5 6 7
18. The technology we used 1 2 3 4 5 6 7
for our group work was
easy to use.
19. The technology we used 1 2 3 4 5 6 7
for our group work helped
us get our work done.
20. It did not take a lot of 1 2 3 4 5 6 7
extra work to install the
technology we used for
our group work.
21. I will use this technology 1 2 3 4 5 6 7
again for my next group
project.
22. What was the name of the technology(ies) that you used? (please list all
technologies)
____________________________________________________________________
____________________________________________________________________
Using your group experience in this course, please indicate your level of agreement
with questions 22 – 39 below. Please refer to the scale ( 1 = strongly agree, 4 =
neutral, 7 = strongly disagree ) and circle the appropriate response.
Strongly Strongly
Agree Neutral Disagree
1 2 3 4 5 6 7
23. I do not think my work 1 2 3 4 5 6 7
group experience was

valuable for my education
24. There was too much 1 2 3 4 5 6 7
socialization in my group
25. The grading of the group 1 2 3 4 5 6 7
project was fair
26. The technology used in 1 2 3 4 5 6 7
the class to support
groups was too complex
27. I was satisfied with this 1 2 3 4 5 6 7
course
28. I think this course served 1 2 3 4 5 6 7
my needs
29. I was disappointed in how 1 2 3 4 5 6 7
the course worked out
30. The level of interaction in 1 2 3 4 5 6 7
my group was low
31. Everyone in my group did 1 2 3 4 5 6 7
their fair share of work
32. I participated in this 1 2 3 4 5 6 7
group more than I usually
do in my other groups
33. Group work is just busy 1 2 3 4 5 6 7
work in classes
34. Groups do not stay on 1 2 3 4 5 6 7
task and waste time
35. Group work is not 1 2 3 4 5 6 7
important to my career
36. I like working in groups 1 2 3 4 5 6 7
in this program
37. The basic idea of working 1 2 3 4 5 6 7
in groups is good
38. The importance of group 1 2 3 4 5 6 7
work is overemphasized
39. The use of group work in 1 2 3 4 5 6 7
the program is an
advantage
40. Technology was helpful 1 2 3 4 5 6 7
for our group work
41. I felt that I was really part 1 2 3 4 5 6 7
of my group
42. I would be willing to 1 2 3 4 5 6 7
work in the same group
again
REFERENCES
Alderfer, C.P. (1972). Existence, relatedness and growth: Human needs in organizational
settings. New York: Free Press.
Aranda, E.K., Aranda L., & Conlon, K. (1998). Teams: Structure, process, culture and
politics, Englewood Cliffs, NJ: Prentice-Hall.
Bandura, A. (1977). Social learning theory. Englewood Cliffs, NJ: Prentice-Hall.
Beebe, S.A., & Masterson, J.T. (1986). Communicating in small groups: Principals and
practices (2nd ed.). Glenview, Illinois: Scott, Foresman and Company.
Bilimoria, D., & Wheeler, J.V. (1995). Learning Centered Education: A Guide to Resources
and Implementation. Journal of Management Education, 19 (3), 326-341.
Briggs, R.O., Nunamaker, Jr., Jay .F., & Sprague, Jr., R.H. (1998). 1001 unanswered research
questions in GSS. Journal of Management Information Systems, 14 (3), 3-21.
Chaplin., J.P., & Krawiec, T.S. (1960). Systems and theories in psychology. New York: Holt,
Rinehart and Winston.
Chidambaram, L. (1996). Relational development in computer-supported groups. MIS
Quarterly, 20 (2), 143-165.
Chidambaram, L., & Bostrom, R.B (1993). Evolution of group performance over time: A
repeated measures study of GDSS effects. Journal of Organizational Computing, 3 (4),
443-469.
Daft, R.L., & Lengel, R.H. (1986). Organizational information requirements, media richness
and structural design. Management Science, 32 (5), 554-571.
Dede, C. (1996). The evolution of distance education: Emerging technologies and distributed
learning. American Journal of Distance Education, 10 (2), 4-36.
DelBecq, A.L., & Van de Ven, A. (1971). A group process model for problem identification
and program planning. The Journal of Applied Behavioral Science, 7 (4), 466-492.
DeSanctis, G., & Gallupe, B. (1985). Group Decision Support Systems: A New Frontier. Data
Base, 16(1), 3-10.
Ebbinhaus, H. (1913). Memory. A contribution to experimental psychology, Translated by
Ruger and Bussenius, New York: Columbia University.
Fjermestad, J., & Hiltz, S.R. (1999). An assessment of group support systems experimental
research: Methodology and results. Journal of Management Information Systems, 15 (3),
7-149.
Gopal, A., Bostrom, R. B., & Chin, W. (1993). Applying adaptive structuration theory to
investigate the process of group support systems. Journal of Management Information
Systems, 9 (3), 45-69.
Gulliksen, H. (1934). A rational equation of the learning curve based upon Thorndike’s law of
effect. Journal of General Psychology, 11, 395-434.
Haythornthwaite, C., Kazmer, M.M., Robins, J., & Showmaker, S. (2000). Community
development among distance learners. Journal of Computer-Mediated Communication, 6
(1) [On-line]. (Accessed 21/07/02) URL: http://www.
ascusc.org/jcmc/vol6/issue1/haythornthwaite.html
Hills, H., & Francis, P. (1999). Interaction learning. People Management, 5 (14), 48-49.
Hearn, G., & Scott, D. (1998). Students staying home. Futures, 30 (7), 731-737.
Hogan, D., & Kwiatkowksi, R. (1998). Emotional aspects of large group teaching. Human
Relations, 51 (11), 1403-1417.
Hora, N., & Kling, R. (2001). Information, Communications and Society [WWW document].
[On-line]. (Accessed 12/01/01) URL: http://www.slis.indiana.edu/CSI/wp00-01.html
Hull, C. (1943). Principles of behavior. New York: Appleton-Century-Crofts.
Kelley, H.H. (1951). Communication in experimentally created hierarchies. Human Relations,

4, 39-56.
Kling, R. (2000). Learning about information technologies and social change: The
contribution of social informatics. Information Society, 16 (3), 217-232.
Lewin, K. (1951). Field theory in social science: Selected theoretical papers. Dorwin
Cartwright (ed.). New York: Harper and Brothers.
Liedner, D.E., &. Jarvenpaa, S.L. (1995). The use of information technology to enhance
management school education: A theoretical view. MIS Quarterly, 19 (3), 265- 291.
Linstone, H.A. & Turoff, M. (1975). The Delphi method: Techniques and applications.
Reading, Mass: Addison-Wesley.
Martz, Jr., W. B., & Landof, G. (2000). Information systems careers: A comparison of
expectations. Journal of Computer Information Systems, 40 (2), 41-46.
Maslow, A. (1943). A theory of human motivation. Psychological Review, 50, 370-396.
Miner, J.B. (1980). Theories of organizational behavior. Hinsdale, Ill: Dryden Press.
McClelland, D. (1961). The achieving society. New York: Van Nostrand Reinhold.
Nunamaker, Jr., J.F., Dennis, A.R., Valacich, J.S., Vogel, D.R., & George, J.F. (1991).
Electronic meeting systems to support group work. Communications of the ACM, 34, 42-
58.
Osborn, A.F. (1953). Applied imagination. New York: Charles Scribner’s and Sons.
Papert, S. (1980). Mindstorms. New York: Basic Books.
Poole, M.S., & DeSanctis, G. (1990). Understanding the use of group decision support
systems: The theory of adaptive structuration. In J. Fulk & C. Steinfeld (Eds.),
Organizations and communication technology (pp.173-193). New Bury Park, Ca: Sage
Publications.
Russell, T. (2001). No significant difference. [On-line]. (Accessed 12/01/01) URL:
http://nova.teleeducation.nb.ca/ nosignificantdifference/
Saaty, T.L. (1980). The analytical hierarchy approach. New York: McGraw-Hill.
Short, J., Williams, E., & Christie, B. (1976). The social psychology of telecommunications.
New York: John Wiley.
Shipley, T.E. & Veroff, J. (1952). A Projective Measure of Need Affiliation. Journal of
Experimental Psychology, 43, 349-356.
TeleEducation, NB. (2001). College.com study shows positive results for online education
[On-line]. (Accessed 20/07/02) URL: http://teleeducation.nb.ca/anygood/
Tu, Chih-Hsiung. (2000). Online learning migration: From social learning theory to social
presence theory in the CMC environment, Journal of Network and Computer Applications,
23, 27-37.
Walther, J.B. (1992). Interpersonal effects in computer-mediated interaction: A relational
perspective, Communications Research, 19 (1), 52-90.
PART VI
SPECIFIC APPLICATIONS OF LEARNING

TECHNOLOGY
Electronic Tutorial for Moral Reasoning in Business
Education: A Technological Learning Tool to
Facilitate Students’ Moral Reasoning
Margarita Alemán Vargas

The Monterrey Institute of Technology Campus Guadalajara, Mexico
1. INTRODUCTION
Many educational institutions are now willing to develop moral

reasoning through pedagogical strategies in courses which will introduce
business students to issues in which legality, profits and ethics seem
irreconcilable. The purpose of this paper is to describe an electronic tutorial
as a tool to help students to recognize three elements in ethical problems:
non-moral facts, moral facts and moral judgments. The paper begins with
some background and scene setting, then the content and method of the
electronic tutorial are described and discussed, and finally some conclusions
are drawn.
2. BACKGROUND
When educators explicitly assume the goal of teaching moral reflection,

moral reasoning and ethical behavior, then social, as well as pedagogical
factors, must be considered because the students’ socio-economic status
affects their ethical-moral values, standards and pervades the students’
personality.
The teacher’s first responsibility is to define the purposes of ethical-
moral education. These may include socialization; clarification of values;
333
334 Margarita Alemán Vargas
fostering the individual’s moral maturity and virtues (Wilson, 2000; Puig
Rovira, 1995); facilitation of students’ adaptation into the current dynamics
of the business world; and helping students to understand the need to balance
individual economic profits and society’s ethical-moral gains.
To accomplish this, the teacher must:
• understand the ethical dilemmas;
• have good teaching skills;
• manage educational technology; and
• understand the students’ attitudes in order to avoid the risk of
indoctrinating them, creating a double code of morality or increasing the
gap between classroom knowledge and workplace praxis.
In business education, teaching and learning ethics involves developing
moral decision making, moral reasoning and the identification of ethical and
unethical behaviors embedded in business cases. Traditional tools, such as
case studies, narratives, simulations and models, have been subject to two
main criticisms. First, the application of theory to the resolution of moral
dilemmas has not paid sufficient attention to the context of the real business
world. Second, it is common to find teachers who use case studies without
an extensive theoretical background (Buchholz & Rosenthal, 2001).
At present, textbooks on business and management include topics
concerning ethical issues because educators are trying to prepare business
education students for a world that is rife with conflicts of interests, external
pressures and internal dilemmas. The purpose of topics in textbooks is to
show the complexity in deciding between right and wrong; legality; profits
and loss; and the necessity of creating win-to-win negotiations and fair play
among the firm, its clients and society. Owing to this kind of reasoning, a
great amount of literature has been addressed to defend the ethical dimension
as an inherent feature of the decision making process and of management
programs and also to provide procedures for thinking critically in ethical-
moral reasoning or judgments (Ivancevich & Matteson, 1993; Locker, 1995).
It is common to find articles that seek to explain real life in the business
world; to describe the efforts of a few businesses to develop an ethical sense
in their employees; to promote innovation with ethics ombudsmen; to
support codes of ethics; and to find ways to reward ethical behaviors
(Ivancevich et al., 1994). The ethical-moral development models most
commonly used at business schools are mainly designed with cognitive
perspectives in mind, but this does not go without criticism. The heart of the
critique resides in the fact that increasing ethical-moral reasoning is a
necessary requisite for evolving from a lower level of moral development to
a higher level, but not a sufficient condition to guarantee the concordance
between moral reasoning or moral development and moral behavior
(Noddings, 1998). To help students’ critical reflection on ethical-moral
Electronic Tutorial for Moral Reasoning in Business Education 335
issues, it is common to find causal models of ethical behavior-decision

making; computer simulation games to practice the analysis of ethical issues
and the legal dimension in business dilemmas; and the use of tests derived
from research in cognitive-moral development. However, in such an
approach several problems can be detected. First, the models have a general
level of reflection or philosophical criteria (Gautschi III & Jones, 1998).
Second, the programmed simulations generally do not offer flexibility to
make changes in instructional material (Wolfe & Fritzsche, 1998). Finally,
tests and dilemmas, such as the Rest questionnaire or Kohlberg dilemmas,
have problems related to the validity of research (Feldman, 1995).
Despite the aforementioned problems, in moral education the general
trends are to develop critical thinking in ethical-moral issues; to avoid
indoctrination; to base practice on theoretical frameworks; to foster the
students’ congruence in diminishing the distance between ethical-moral
reasoning and moral behavior; and to take care to train not only students’
reasoning but also their emotions, habits of virtue and autonomous moral
judgment. In short, moral education endeavors to educate and construct the
total ethical moral personality: reasoning, emotion, behavior, values and
character (Berkowitz, 1995; Puig Rovira, 1995).
In business education it is hoped that educators will gain awareness of the
limitations of their teaching-learning techniques and that they will keep in
mind that moral reasoning, moral decision-making and ethical behaviors are
not isolated cognitive procedures since they are part of the students’ personal
process. Therefore the teaching-learning strategies used in school should
attach importance to the contextual and individual elements that affect those
kinds of processes. Examples of contextual or individual elements are widely
mentioned by Singhapakdi, Vitell and Frank (1999). They explain that these
elements include a personal moral philosophy, such as relativism or
idealism; moral sentiments and intentions, such as courage (Park, 1998); the
situation and characteristics of the person, such as self-esteem, ego and
probability of being caught or watched, (Buckley, Wiese & Harvey, 1998;
Matsuba & Walker, 1998); the change of attitude and action when people
know that ethical-moral cases, dilemmas and decisions are hypothetical or
real (Armon, 1998; Tirri, 1998); and the diversity of moral standards related
to cultural, moral pluralism and the differences and socioeconomic
circumstances in a country (Sexty, 1998).
Moral reasoning is a concept designed to explain the level of maturity
expressed through a person’s ethical-moral judgments and it can be referred
to as a level of moral development or critical thinking addressed to ethical-
moral issues. In schools, teachers frequently use printed questionnaires and
dilemmas related to Lawrence Kohlberg’s pre-conventional, conventional
and post-conventional stages of moral reasoning (Woolfolk, 1990; Biehler &
Snowman, 1990) to help students move from a lower to a higher stage. In an

extended meaning, moral reasoning defines the set of cognitive processes
focused on observing, comparing, analyzing, evaluating, discerning or
reflecting on those aspects, behaviors, affections, sentiments and emotions
belonging to ethical-moral issues. In a word, critical thinking is directed
towards developing what Paul (1993) has called ethical-moral abilities.
These are the ability to distinguish facts from moral principles in
information; to clarify ethical-moral values; to make moral decisions or
ethical judgments; to evaluate and to make moral arguments; to develop
autonomous morality; to self-evaluate ethical behaviors and so on.
Without lessening the traditional concept of moral reasoning, this paper
upholds the second definition of moral reasoning, that is, a cognitive process
focused on ethical moral issues in order to develop ethical-moral abilities.
Besides, in keeping with philosophical rigor, it was decided to differentiate
between the adjectives ethical and moral because ethics and morality are not
the same. Certainly, there are many definitions of ethics and morality but in
this paper ethics is understood as the study of the concepts of right and
wrong used as criteria to evaluate human conducts; to analyze a variety of
codes of morality; and to investigate ways of behaving, such as hedonism,
utilitarianism, cynicism, virtue, moral duties and the like. Morality is
understood as standards, norms and rules of conduct, that are created by
individuals or by different socio-cultural groups (Fieser & Dowen, 2001;
Abbagnano, 1978). Moreover, the increasing awareness of cultural diversity
and the obvious presence of a variety of codes of morality in the world have
led teachers and business people to avoid moral ethnocentrism and to
consider moral pluralism.
3. THE ELECTRONIC TUTORIAL, ITS CONTENT,

AND METHODOLOGY
Electronic tutorial is a term widely used in computer-assisted instruction

and it refers to pre-designed computer programs which the teacher uses to
present information, carry out simulations, give tests to students, foster
higher levels of comprehension and help students to master the academic
material in specific teaching-learning domains (Bates, 1995). While using
the electronic tutorial students read instructions or questions and type or
select an answer. Next, the tutorial provides feedback to let the user know
whether the answer is correct or not and then facilitates access to more
advanced stages of the task.
Depending on the software’s technological features, some tutorials are
flexible in inputting new information while others do not allow editing to
adapt data to students’ cultural needs, social values and legal mechanisms in
business and management cases. Whatever the situation may be, the point is
that educational technology software is an effective teaching-learning tool
for moral reasoning in business education if it is developed or selected under
two main criteria:
• The teacher or teaching team must be trained in business courses as well
as in ethical-moral knowledge and strategies.
• The teachers or specialists in charge of designing or selecting
educational technologies base the design on Bates’s ACTIONS model
(1995): access, costs, teaching and learning objectives, interactivity,
organizational issues, novelty and speed.
The electronic tutorial for moral reasoning in business education has been
created as a complementary didactic tool for teachers whose educational
purpose is to develop ethical-moral reasoning and who want to incorporate
this into their academic course. The electronic tutorial’s instructional
objective is to develop the cognitive-moral ability to recognize the three
moral issues involved in perceiving ethical-moral problems: non-moral facts,
moral facts and moral judgments.
According to theorists Gutiérrez (1999) and Rodríguez, Perdomo and
Albelo (1996), a non-moral fact is a past or current event of nature (e.g. the
rain made the river overflow; the wind blows) or it is an involuntary human
act perpetrated without freedom or knowledge (e.g. Smith, under torture,
communicated confidential management information) or biological human
acts (e.g. Lucy breathes). A moral fact is a voluntary human act that has
occurred or is happening and involves the respect for, or violation of, moral
standards and ethical values (e.g. Smith denounced the illegal commercial
trade). A moral judgment is a point of view about the goodness or
wickedness of a moral fact. It is a particular opinion that is not necessarily
true or false (e.g. it is good to be an honest employee).
Designed for students in the early stages of a business degree course, the
electronic tutorial begins with a welcome message. Then the student reads
about the instructional objective of developing moral reasoning by means of
practicing the cognitive-moral ability of distinguishing non-moral facts,
moral facts and ethical-moral judgments embodied in written information
derived from daily life and business settings. Secondly, the student reads the
theoretical section describing a non-moral fact, a moral fact and ethical-
moral judgments and providing examples and philosophical criterion to
differentiate a human act (i.e. able to be judged as a moral fact) from acts by
man (i.e. unable to be judged as a moral fact). Figure 1 gives an example of
the electronic tutorial content. The first level of the electronic tutorial
contains ten short sentences which include an isolated event (from daily life
and business) to be classified by clicking on one of the three choices

mentioned: non-moral fact, moral fact and ethical-moral judgment.
Figure 1. A Tutorial Example.
After making their choice, students can compare their own answer with
the correct option marked by the electronic tutorial and they can display the
explanation of why the other choices are not the right ones. Students can
answer at their own pace, go back or forward between the tutorial’s
sentences.
At the second level, the students are asked to read two brief texts with
some underlined sentences and once again, students have to select a single
choice for each sentence, compare the answers and review the explanations.
However, the degree of difficulty is higher than in the first level because
some of the underlined sentences express two events that are related to each
other (i.e. cause-effect relation). Therefore, if the student does not observe
the contextual information then he or she could confuse a moral fact with a
non-moral fact.
This kind of situation must be posed in the electronic tutorial’s texts
because in real life, one event is often related to another one. For instance, it
is not the same to read, “Lucy walked away from the office”, as, “after being
paid for favoring a supplier in the contest, Lucy walked away from the
office”. It is necessary to be on the alert to avoid the wrong interpretation

that could cause the student to mistake the classification of what could be a
non-moral fact, a moral fact or an ethical-moral judgment, because a mistake
like this could affect the comprehension, evaluation and decision making
process involved in an ethical-moral situation.
4. THE ELECTRONIC TUTORIAL AND

SUPPORTING METHODOLOGY
The supporting methodology for the electronic tutorial is in four parts.

The first part introduces a controversial text or debate with the intention of
exploring students’ beliefs and ethical-moral values, points of view and the
extent to which they can identify ethical-moral issues. At the end of session,
the teacher comments on the relevance of developing the above mentioned
cognitive-moral ability to increase the possibilities of making a better
ethical-moral judgment and diminishing the chance of being manipulated by
others or falling into an embarrassing personal or situation at work.
The second part posts the first level of the electronic tutorial on the
selected technological platform, such as e-mail, ICQ (chat or computer-
mediated communication), web page or power point, and the teacher may
schedule the time of exposure and can also monitor students’ answers.
The third part could be another debate, a problem-based learning case or
a real case that demands, at the same time, a solution and arguments. The
focus of the teacher’s attention would be on observing, listening and
addressing questions. His/her aims would be to analyze students’ responses
in distinguishing the ethical-moral issues, concrete non-moral facts, moral
facts and judgments, and to assess the ways they have been to improve moral
reasoning. Furthermore, the teacher attempts to pay attention to the
underlying ethical values. Obviously, if the teacher wants to reach this last
kind of analysis, he or she needs to be well trained. Otherwise, the teacher
could focus solely on providing feedback on the students’ cognitive-moral
ability performance.
The fourth part is to post the second level of the electronic tutorial to
monitor and provide feedback on the students’ answers.
The electronic tutorial presents advantages for instructional planning
because of its self-study characteristic and of being able to deliver the
content in different technologies. This electronic tutorial is the second
version of an original set of five electronic tutorials that were designed by
Alemán (1998) for a course of civic ethics set in Learning Space-Lotus
Notes. The electronic tutorial was designed for use by teenagers and young
adults (14-18 years old) who, at their own pace, answered each tutorial in
order to be ready for debates in class; to take exams; and to analyze and
comprehend the legal, political, historical and moral dimensions of their own
human acts and those of others. Each tutorial, including this one in business
education, is supported by Learning Space-Lotus Notes technology and is
highly flexible because it allows the teacher to insert and edit texts according
to the group’s needs and current events. This technology sends a message to
teacher once the student is finished and it displays a score record in the
student’s electronic portfolio. Moreover, Alemán (1998) created the content
of each electronic tutorial and guidelines for the transference from one
technology to another and for using the electronic tutorials as a self- study
tool.
In order to prove the effectiveness of the original electronic tutorial
version, last year (2000) a pre-experimental, pre-test/post-test investigation
was conducted with a single group. The study took place in the Tec de
Monterrey and the participants were 23 high school teenage students
enrolled in the second semester of the civic ethics course. The instructor was
trained in the aforementioned cognitive-moral ability and electronic tutorial
methodology.
At the beginning of the semester, the researcher used a previously
defined and revised pre-test to evaluate the student’s ability to recognize and
distinguish between non-moral facts, moral facts and moral judgments. After
this, the researcher asked for the students’ participation and total discretion
to avoid the risk of communicating personal answers related to the content of
the electronic tutorial questions. The students agreed and they were exposed
to the electronic tutorial methodology over a period of three months. One
hour was allocated to each debate and dilemma directed by the teacher and
the students spent the rest of the time on answering the electronic tutorials at
their own pace. At the end of the semester, the researcher applied the post-
test. Although the results cannot be totally reliable because of the pre-
experimental design, they showed that 17 students out of 23 improved their
ability to identify ethical-moral issues and to differentiate non-moral facts
from moral facts and ethical-moral judgments. Next year, at Tec de
Monterrey Campus Guadalajara, a second research project will begin with a
carefully thought out experimental pre-test/post-test and experimental and
control groups in order to obtain better conclusions on the effectiveness of
an electronic tutorial.
5. DISCUSSION
Based on a cognitive-constructivist approach, ethics of minimum values

and a non-religious perspective, the electronic tutorial has a self-study
function. This is valuable when an academic course has a restricted schedule

for teaching business ethics, but the teacher is interested in using cases,
narratives, dilemmas or debates to foster moral reasoning or critical thinking
on ethical-moral behavior in business and management. Without denial of a
real world, cognitive-constructivist learning theories defend the existence of
different ways of learning and interpreting reality. Therefore, instructional
designers may take this theory into account in class planning, the preparation
of teaching-learning materials and the selection and use of technologies
(Bednar et al., 1992) to enrich teaching and accomplish learning objectives.
In response to this learning theory, the content of the electronic tutorial on
moral reasoning in business education can develop the cognitive-moral
ability through the suggested instructional material. However, the tutorial
does not restrict teaching decisions to the inputting of new information
according to the students’ characteristics, it also incorporates structural
criteria to facilitate the pedagogical task of actualizing instructional material.
The reason for being based on ethics of minimum values and a non-
religious perspective of ethical-moral values is twofold. On the one hand,
ethics of minimum values is a recent term widely used to refer to a group of
ethical values and moral standards (e.g. freedom, equality, justice, tolerance,
democracy, honesty, responsibility, respect for individual rights, universal
human rights) which are found in any society and which cannot be refused
without denigrating human dignity (Cortina & Conill, 1994). Thus, ethics of
minimum values matches the educational goal of educating people as
citizens with social rights and duties. On the other hand, Cortina and
Martínez (1996) notice that, for historical reasons, religious doctrines have
been in charge of codes of morality to unite people. However, it could not be
inferred that the entire conception of morality is exclusive to religion
because there are many other codes of morality without a confessional
origin. Obviously, it has been clear enough that a non-religious perspective
in education is defined as an ideological position that is autonomous from
any religion since it is not opposed to or in favor of religion. Moreover, it is
important to know that this kind of a non-religious position is commonly
called a “laic position” and should be distinguished from “laicism” which is
a belligerent position that excludes and opposes any religion (Abbagnano,
1974; Cortina & Martínez, 1996). To educate under the ethics of minimum
values approach is one way of diminishing moral relativism and
transcending towards moral pluralism and the shaping of individuals’
integral ethical-moral personality. To be circumscribed to a non-religious
perspective in a scholastic context is a necessity because in that way the
respect of students’ diverse religious beliefs can be ensured.
The rationale of the electronic tutorial is that moral reasoning should be
effectively enhanced in both daily life and business education and this kind
of effort demands that we redirect teaching-learning situations. In this way,

instead of staying at superficial levels of understanding or reflection on
ethical-moral aspects, the teaching process has to start with the basic element
of moral reasoning, such as identifying non-moral facts, moral facts and
judgments, as well as their respective philosophical characteristics. This
effort is important because in a classroom it is common to see how much
subtle, but relevant, moral factual information could go unperceived in
written or verbal discourses, dilemmas, debates. It is also the case that
students could discuss or experience uncomfortable feelings when they are
trying to convince others about their supposedly truthful ethical-moral
judgment, when it is known that these kinds of judgment cannot be true or
false but are relative to a specific ethical-moral scale. Consequently, the
students have to be open to understanding; to tolerating different points of
view; and to learning to argue their own opinions by supporting and
reviewing ethical-moral standards underlying their position or decisions
(Rodríguez, Perdomo & Albelo, 1996; Alemán, 1997). This cognitive-moral
ability to distinguish between non-moral fact, moral fact and ethical-moral
issues has to be practiced in daily life as well as business settings because
behaving as a person, son, daughter, parent, student and citizen or as an
employee, entrepreneur, client and supplier is not a set of different hats that
individuals can put on depending on the context and circumstances. On the
contrary, if we want to heighten students’ ethical-moral awareness, then
teachers have to let the students know that they must have, or construct,
unique ethical-moral standards as a basis for their ethical-moral personality
to be coherent with self values and to be congruent, upright and worthy of
other people’s trust. It would be a profound error to misdirect all the
educational enterprise of educating the ethical-moral personality and to end
up fostering a double code of ethics: one for personal life, another for
professional-working life and another according to convenience or personal
interests, whatever they might be.
The effectiveness of the electronic tutorial in developing moral reasoning
in business education finds its limits in the terrain of thinking and does not
attempt to educate moral emotions or sentiments which affect moral
reasoning, ethical-moral judgment and the decision making process. That is
the reason why the electronic tutorial must be combined with other didactic
teaching-learning strategies focused on cultivating so-called ethical
intelligence. This attaches importance to emotions as well as to reflecting on
ethical-moral situations (see Marina, 1999; Marina, 2000). It also embraces
real learning contexts that allow students to acquire an autonomous moral
consciousness; to be independent from the teacher’s point of view on ethical-
moral situations; to cope with others’ unethical pressures; and to become
more congruent between the ethical-moral reasoning level and ethical-moral

behavior.
6. CONCLUSIONS
At the university level, it is not wise to attempt to change the students’

already formed ethical-moral values and standards. What is important is to
promote critical thinking on ethical-moral issues and then moral reasoning
can be developed through curricular context whenever a theoretical-
pedagogical background supports it. The focus is not on discussing whether
it is necessary to have an ethics course or not in business education curricula.
What matters is to pay attention to enhancing moral reasoning and to leaving
behind the superficial levels of use of cases, debates, narratives and
dilemmas that go towards the complete development of an ethical-moral
personality. Although increasing reasoning in ethical-moral issues is not
sufficient to make someone behave in an ethical-moral way, it is at least a
necessary condition for developing morally. This was the main motivation
behind the creation of the electronic tutorial. Therefore, the methodology of
the electronic tutorial is offered as a complementary didactic tool to respond
to the educational demand of contributing to facilitating young students’
evolution to higher stages of moral reasoning that should be exhibited in
those future business leaders who will have the social responsibility of
improving business and economic conditions which may have an influence
on other spheres of human interaction.
Certainly, a teacher of business education should have a basic knowledge
of moral philosophy and applied ethics; a clear understanding of, and the
skills to identify, ethical-moral issues; and the ability to differentiate
between a non-moral fact, a moral fact, and an ethical-moral judgment in
order to be able to facilitate students’ awareness in judging any ethical-moral
situation in any context. Teachers should be able to improve students’
performance in self-evaluation of behavior, in written and oral debates,
arguments and comments in class. They should be able to guide students
them on their particular path towards independence of thinking and
autonomous consciousness without indoctrinating them. Additionally,
teachers have to invest time in training in other complementary cognitive
abilities. These include the ability to recognize what kind of attitude and
emotions could influence the students’ moral reasoning and the formulation
of ethical-moral judgments whenever they are dealing with ethical-moral
situations; to observe and provide feedback on ethical-moral reasoning and
behaviors; to create an adequate classroom climate for fostering habits of
thinking and behaving ethically; and to establish the congruence between

thinking and acting.
To develop moral reasoning cannot be restricted to a specific academic
content because real life is not made up of self-contained events and does it
function in that way. Moreover, it is an underlying assumption of the
electronic tutorial that it is necessary to develop the cognitive ability to
differentiate between a non-moral fact, a moral fact and ethical-moral
judgment in daily life, at the same time as it occurs in a business setting,
because of the importance of inculcating in students the idea of constructing
and holding a unique ethical-moral standard as a basis for their ethical-moral
personality. In doing so, the chances of reinforcing double codes of morality
significantly decreases. It is important that educators use a variety of
pedagogical techniques and computer technologies in their academic
courses. They should also make sure that students have sufficient time to
develop abilities and attitudes related to the ethical-moral dimension. To
close, I would like to reiterate that behind this effort there is a simple but
powerful reason. Business education students are the ones who represent the
future generation of business leaders and they are the ones who will make
ethical decisions, the consequences of which will affect us all.
REFERENCES
Abbagnano, N. (1974). Diccionario de filosofía (2nd ed.). México, D.F.: Fondo de Cultura
Económica.
Abbagnano, N. (1978). Historia de la filosofía, Vol. 1, (3rd ed.). Barcelona, España: Montaner
y Simón.
Alemán Vargas, M. (1997). Moral judgment. In Model Course Civic Ethics Electronic
Database. Tec de Monterrey Campus Guadalajara. [Delivered on Learning Space/Lotus
Notes]. Retrieved on 20/08/00 from database lspace\va\curmod\pd95202\ mcenter.nsf.
(Limited access).
Alemán Vargas, M. (1998). General teaching-learning strategy for civic ethics subject. In
Model Course Civic Ethics Electronic
Database. Tec de Monterrey Campus Guadalajara. [Delivered on Learning Space/Lotus
Notes]. Retrieved on 20/08/00 from database lspace\va\curmod\pd95202\mcenter.nsf.
(Limited access).
Armon, C. (1998). Adult moral development, experience and education. Journal of Moral
Education, 27 (3), 345-370.
Bates, A.W. (1995). Technology, Open Learning and Distance Education . New York, NY:
Routledge.
Bednar, A.K., Cunningham, D., Duffy, T.M., & Perry, J.D. (1992). Theory into practice: How
do we link. In T.M. Duffy y D.H. Jonassen (Eds.), Constructivism and the Technology of
Instruction (pp. 18-34). Hillsdale, NJ: Lawrence Erlbaum & Associates.
Berkowitz, M.W. (1995). Educar la persona moral en su totalidad. Revista interamericana de

educación, No. 8. Organización de Estados Iberoamericanos. May/Aug. [On-line].
(Accessed 20/10/00). URL: http://www.campus-oei.org/oeivirt/rie08a03.htm
Biehler, R.F., & Snowman, J. (1990). Psychology applied to teaching (6th ed.). Boston, MA:
Houghton Mifflin.
Buchholz, R.A., & Rosenthal, S.B. (2001). A philosophical framework for case studies.
Journal of Business Ethics, 20 (1/2), 25-31.
Buckley, M.R., Wiese, D.S., & Harvey, M.G. (1998). An investigation into the dimensions of
unethical behavior. Journal of Education for Business, 73 (5), 284-290.
Cortina, A., & Conill, J. (1994). Ética de la empresa. Madrid, España: Trotta.
Cortina, A., & Martínez, E. (1996). Ética. Madrid, España: Akal.
Feldman, R.S. (1995). Psicología: con aplicaciones para iberoamérica. México, D. F.:
McGraw Hill.
Fieser, J., & Dowden, B. (Eds.). (1998). Ethics . The internet encyclopedia of philosophy.
[On-line]. (Accessed 07/06/01) URL: http://www.utm.edu/research/iep/e/ethics.htm
Gautshi III, F.H., & Jones, T.M. (1998). Enhancing the ability of business students to
recognize ethical issues: An empirical assessment of the effectiveness of a course in
business ethics. Journal of Business Ethics, 17 (2), 205-216.
Gutiérrez Sáenz, R. (1999). Introducción a la ética. México, D. F.: Esfinge.
Ivancevich, J.M., Lorenzi, P., Skinner, S.J., & Crosby, P.B. (1994). Management: Quality and
competitiveness. Boston, USA: Irwing.
Ivancevich, J.M., & Matteson, M.T. (1993). Organizational behavior and management (3rd
ed.). Chicago: Irwing.
Locker, K.U. (1995). Business and administrative communication (3rd ed.). Ohio: Irwing.
Marina, J.A. (1999). El laberinto sentimental. Barcelona, España: Anagrama.
Marina, J.A. (2000). Teoría de la inteligencia creadora. Barcelona, España: Anagrama.
Matsuba, M.K., & Walker, L.J. (1998). Moral reasoning in the context of ego functioning.
Merrill-Palmer Quarterly, 44 (4), 464-484.
Noddings, N. (1998). Thoughts on John Dewey’s “ethical principles underlying education”.
The Elementary School Journal, 98 (5), 479-488.
Park, H.J. (1998). Can business ethics be taught? A new model of business ethics education.
Journal of Business Ethics, 17 (9/10), 965-977.
Paul, R.W. (1993). Critical Thinking: What every person needs to survive in a rapidly
changing world (3rd ed.). Santa Rosa, CA: Foundation for Critical Thinking.
Puig Rovira, J.M. (1995). Construcción dialógica de la personalidad moral. Revista
iberoamericana de educación, No. 8.
Organización de los Estados Iberoamericanos. May/August. [On-line]. (Accessed 22/09/00).
URL: http://www.oei.org.co/oeivirt/ rie08a04.htm
Rodríguez Lozano, V., Perdomo, F., & Albelo, M.L. (1996). Ética (5rd ed.). México, D.F:
Addison Wesley-Longman.
Sexty, R.W. (1998). Teaching business ethics in transitional economies: Avoiding ethical
missionary. Journal of Business Ethics, 17 (12), 1311-1317.
Singhapakdi, A., Vitell, S.J., & Franke, G.R. (1999). Antecedents, consequences, and
mediating effects of perceived moral intensity and personal moral philosophies. Academy
of Marketing Science Journal, 27 (1), 19-36.
Tirri, K. (1999). Teachers’ perception of moral dilemmas at school. Journal of Moral
Education, 28 (1), 31-47.
Wilson, J. (2000). Methodology and moral education. Oxford Review of Education, 26 (2),
255-262.
Wolfe, J., & Fritzsche, D.J. (1998). Teaching business ethics with management and marketing
games. Simulation & Gaming, 29 (1), 44-59.
Woolfolk, A.E. (1990). Psicología educativa (3rd ed.). México, D.F.: Prentice Hall.
The Five Key Benefits of On-line Final Examinations
(with Three Free Bonus Benefits)
Jeremy B. Williams
Brisbane Graduate School of Business, Queensland University of Technology, Australia
1. INTRODUCTION
The increasing market orientation of higher education has brought

sweeping changes within universities throughout Australia and elsewhere in
the world. Among other things, changes to government funding have forced
universities to become more innovative in their resourcing arrangements.
Partnerships with professional bodies and the private sector are becoming
more widespread, universities have expanded geographically to tap into non-
traditional markets. For course developments to proceed, tangible evidence
of student demand and cost efficiency is of the utmost importance. The
product of these changes, and others, is that universities have developed a
much stronger “customer focus”. In short, the hard reality of life in the
higher education sector, as we move into the new millennium, is that failure
to recognize students as “clients” is to run the risk of anonymity in the
marketplace or, worse still, notoriety, which can only lead to reduced
funding, cuts to courses and staffing levels and even closure.
One way in which some institutions have sought to enhance their
international competitiveness is through the flexible delivery of their
programs. Flexible delivery is, by definition, a client-oriented approach
because it is a commitment, on the part of the education provider, to tailor
courses to meet the various individual needs of its students. Furthermore, it
is tacit recognition of the “massification” of higher education (Carrier,
1990), whereupon the student profile has changed quite dramatically -
347
348 Jeremy B. Williams
socially, culturally, economically - and that, pedagogically, there is a need to

cater for this increasingly diverse student body.
As this paper will highlight, flexible delivery provides students with a
number of different options for study. It is not prescriptive in the sense that
one approach to study is identified as being superior to another. A student
can chart a route through a degree that is most compatible with their budget;
their social, family and working lives; and their preferred learning style. In
short, flexible delivery is non-discriminatory, catering equally for an
international student, a single parent working part-time, a business executive
traveling regularly overseas and interstate, or a school leaver.
The main aim of this paper will be to focus on one important and
frequently overlooked aspect of flexible delivery, the assessment component.
It will demonstrate, by reference to the case of some flexibly delivered
course units within the MBA program at the Brisbane Graduate School of
Business (BGSB) at Queensland University of Technology (QUT), that
flexible assessment systems are effective, easy to administer and very
popular with the student body. The BGSB is one of six schools in the
Faculty of Business at QUT and currently has around 1000 students in the
MBA and associated programs. In 2000 it was ranked third in Asia for the
quality of the distance delivery of its programs by Asia week magazine.
Ironically, the BGSB does not consider itself to be a bona fide distance
education provider, but it received recognition as such because of the
flexible delivery of its programs off campus for corporate clients.
The discussion will concentrate, first of all, on the defining
characteristics of a flexibly delivered program. This is followed by a brief
discussion of the flexible assessment model (FAM) that is being used in a
number of MBA course units and how greater flexibility in the format of the
final examination is a natural extension of this model. The results of some
preliminary research into student views regarding an on-line, take-home
examination are then evaluated. The final section draws together the strands
of the discussion and puts forward some tentative conclusions.
2. WHAT IS FLEXIBLE DELIVERY?
There has been a commitment to the principle of flexible delivery at

QUT since the mid-1990s when it became a central feature of the
University’s teaching and learning strategic plan. Care has been taken not to
define flexible delivery too rigidly so as not to erect any obstacles to
innovative ways of delivering programs. However, to assist staff in
developing some notion of flexible delivery, the following internal QUT
memorandum was circulated in April 1999:
On-line Final Examinations 349
“A flexibly delivered unit is one in which the options for delivery include
alternatives to the traditional ways of on-campus in-classroom lectures,
seminars, tutorials and practical classes. Such a unit will be designed with
the aim of meeting students’ diverse learning needs by incorporating one or
more aspects of flexibility in time, place and/or technology, such as:
• delivery in the workplace or remote from the campuses of the university;
• delivery in block mode, other intensive mode, or other non-standard
delivery time format either on or off campus;
• delivery with non-standard beginning and completion dates for the units
(insofar as the Student Information System can cope with this kind of
flexibility); and
• the use of technology and resources for learning support to provide
options for any student to access and use materials at his or her own time
and place (e.g. web-based teaching materials and exercises), or to be
assessed without having to attend examinations at a specific place and
time.
These aspects of flexibility will form the majority of the delivery
mechanisms used in any flexibly delivered unit.”
Within a relatively short period of time flexible delivery has become a fairly
prominent feature within courses offered by the Faculty of Business and has
been adopted, to varying degrees, in individual course units, specialisations
(groupings of course units) and whole courses.
The BGSB, in particular, has embraced the notion of flexible delivery
with some alacrity. It offers a range of customised certificate courses in
management for the defence forces, government departments, professional
institutes and individual companies in the private sector. Integral to this
strategy has been the production of study guides for all course units, thus
facilitating intensive block mode, on or off campus, to meet the special
needs of these groups of students.
A further initiative has been the development of on-line teaching (OLT)
sites (see http://olt.qut.edu.au/studentEntry.cfm - Accessed 25/07/02). There
are currently 457 OLT sites University-wide, with the Faculty of Business
housing the largest number. Commencing with two sites in 1997, there are
currently 157 OLT sites in Business, 43 of which are located in the BGSB.
The framework for OLT sites varies from course unit to course unit but,
typically, there is a download facility where students can access PowerPoint
lecture slides, tutorial solutions, past examination papers and the like,
discussion forums (electronic bulletin boards), chat space, (internet relay
chat or IRC), and discipline-relevant links to the web. Until recently,
however, little attention has been devoted to assessment and how this might
be integrated with the OLT system.
3. THE CASE FOR FLEXIBLE ASSESSMENT
Continuous assessment has, in the past, been perceived as being

“flexible” in that a student, by undertaking it, is not forced to rely wholly on
their ability to perform well in final examinations. More innovative
continuous assessment regimes have gone so far as to permit student choice
with respect to the assignment topic and maybe the submission date.
However, while it is true that this does provide students with an element of
flexibility, there is not sufficient flexibility that a person could elect not to
undertake a piece of continuous assessment if it was incompatible with their
preferred learning style or their work patterns. The fact of the matter is that,
while there may be some latitude in terms of the essay title or handing-in
date, continuous assessment is typically summative in nature and, for this
reason, it is not a truly flexible assessment system.
Consider the example of the conscientious student who submits an
assignment early in the semester but scores below average marks. It would
be very rare for permission to be granted for this person to be given a
“second bite at the cherry” by resubmitting or attempting an alternative
assignment with a later handing-in date. Increasingly large enrolments tend
to preclude this option. In short, when a term-time essay is submitted there is
no going back. For this reason, standard assessment systems are unforgiving
because a student is not permitted to learn from their mistakes.
To summarise, if there is a publicly stated commitment to flexible
delivery, it is fatuous to persist with the kind of assessment systems that
have come to characterise the university sector. In the BGSB steps were
taken during 1999 to address this problem with the introduction of the FAM
into a number of MBA course units. This was developed by Stanford, and
adapted by Layton and Williams, for use in first year economics. In essence,
this system gives students the choice of completing all, or some combination
of a series of optional assessment items, or indeed, no optional assessment at
all, such that it is possible to have anywhere between 50% and 100% riding
on the final exam, the only compulsory assessment item. Importantly, a
computer spreadsheet identifies which combination of assessment items
maximises a student’s mark. That is, it is not up to students to nominate their
preferred assessment combination in advance. The spreadsheet performs this
task for them. Any assessment item that yields a lower final mark is simply
disregarded and a summative assessment item is treated as formative
assessment. In this way, a student can take advantage of the feedback they
receive and avoid making the same mistake in the final examination. By
electing not to complete the continuous assessment, a student foregoes this
opportunity, but this is their choice based on their preferred learning style,
their work patterns or family commitments (see Williams, 1998, for a more
detailed description and analysis).
Student endorsement of the FAM within the MBA program has been
quite resounding, a reflection of the fact that a high proportion of the student
body study part-time and they often struggle to complete items of continuous
assessment given the large number of competing interests on their time. It is
important to note, however, that the FAM does not discriminate in favour of
one type of student over another. The numerous assessment options it
provides accommodate student diversity and cater for different learning
styles. Indeed, the full-time MBA students are equally as supportive of the
system as their part-time counterparts. All students find it comforting
knowing that, unlike conventional assessment systems, they can complete a
continuous assessment item under the FAM and not worry that a poor
performance will have an irreversible effect on their final grade.
The commendations received by the FAM notwithstanding, much work
remains to be done. At this point, only the assessment system has been
determined. Within this framework, attention must be given, not just to
outcomes, but also to the student experiences that lead to these outcomes
(Boud 1991). To this end, academics are being encouraged to focus on these
experiences with a view to improving the whole of student learning. This has
to be a collaborative activity, involving people from across the University
community, including students themselves, and from outside the University,
specifically employers. This has been a high priority in the Faculty of
Business at QUT following the release of the Faculty Education
Committee’s report, “Review of Assessment Policy and Practices”, in
August 1999. Subsequently, in the BGSB there has been a determination to
investigate the effectiveness of all the assessment regimes currently in
operation and the usefulness of the assessment instruments within them, as
part of its five-year rolling review of course units.
3.1 On-Line Assessment and the Authentication

Problem
The FAM, together with the course unit study guides and the OLT sites,
genuinely allows for the possibility of self-paced learning. This is something
that has been further facilitated, since September 2000, with the introduction,
in a number of course units, of on-line multiple-choice tests. These tests
were previously conducted in class as part of formal assessment and
invigilated in the normal way.
The tests, accessible via the OLT system, are marked by computer
instantaneously and are primarily designed to give students continuous
feedback on their progress. Importantly, a student receives a mark for
participation (5% for the completion of 5 tests) rather than a mark for
performance. The decision to go along this path arose for a variety of
reasons including the view that, while multiple-choice is a useful mechanism
for testing content knowledge and the understanding of simple concepts, it is
an instrument that does not lend itself to the kind of higher order skills
required of MBA graduates (e.g. critical thinking skills). Another related
reason is that by getting multiple-choice tests out of the classroom more time
is available to concentrate efforts on the development of these higher order
generic skills. Of all the reasons, however, perhaps the most compelling was
the fact that students have the opportunity to cheat if the tests are
unsupervised.
With on-line teaching becoming increasingly common, not to mention
the burgeoning number of colleges and universities around the world
offering whole courses over the Internet, the problem of ‘cybercheats’ has
come to occupy the mind. Indeed, a whole industry has grown up around
trying to devise ways of offering educators sufficient security that they can
feel comfortable about offering on-line examinations. A recent study by
Fröhlich (2000) offers a comprehensive evaluation of the various
technologies that could facilitate what he calls “authenticated secure on-line
computer assisted assessment”. These include fingerprint recognition, smart
cards, hand geometry, retinal scans, iris recognition, facial recognition, voice
recognition and remote invigilation. Aside from the prohibitive cost of some
of these devices, Fröhlich acknowledges (through personal communication)
that, while their reliability is increasing as time goes by, none are 100%
cybercheat-proof.
This obviously presents something of a quandary. If there is no reliable
mechanism for preventing students from cheating in examinations and tests,
has the considerable amount of time, effort and money invested in the
development of on-line teaching resources been misdirected? What is the
point of an institution committing itself to the flexible delivery of a course if,
come exam time, it has to resort to the highly inflexible practice of dragging
students into classrooms and examination halls to be watched over by a team
of invigilators? Clearly, this is not what these institutions have in mind. To
quote from the official QUT memorandum detailed earlier in this paper,
flexible delivery is about, among other things, “the use of technology and
resources for learning support to provide options for any student to access
and use materials at his or her own time and place … or to be assessed
without having to attend examinations at a specific place and time [emphasis
added]”.
A simple solution would be to abandon the idea of examinations and opt
for assessment by assignment work only. Many institutions do this already,
of course, on the basis that timed, closed-book examinations constitute an
ineffective means of assessing a person’s ability. It might test the ability to

cram material into one’s head the night before an exam and reproduce it the
following day in a reasonably coherent form, but it does not necessarily
assess whether any deep learning has taken place. This said, courses assessed
by assignment are not without their problems either. Not least of these is the
problem of authentication. “How can we be sure that the work submitted is
the work of the student concerned?” This is an age-old question and the fact
there is no readily available answer is one reason why the BGSB at QUT has
a policy of compulsory examinations in all course units, with the exception
of research units.
Table 1: Type of examination and stress levels.

Question 1: How do on-line (open book) take-home examinations compare to Score %
regular (closed book) in-class examinations in terms of the stress and anxiety
they cause?
A. Take-home examinations are far less stressful. 13 36
B. Take-home examinations are a little less stressful. 13 36
C. Take-home examinations generate the same level of stress as in-class 5 14
examinations.
D. Take-home examinations are a little more stressful. 5 14
E. Take-home examinations are far more stressful. 0 0
Table 2: Time allowed for take-home examinations.

Question 2: If an examination question (requiring a 2000 word answer) is set at Score %
12pm on a Friday, and due in at 12pm the following Monday, is this an
appropriate amount of time to complete the task?
A. This is way too much time. 0 0
B. This is a little more time than is needed. 4 11
C. This is just about the right amount of time. 28 78
D. A little more time would have been better. 4 11
E. A lot more time is needed. 0 0
Table 3: Type of examination and deeper learning.

Question 3: How do on-line (open book) take-home examinations compare to Score %
regular (closed book) in-class examinations in terms of the opportunity they
provide for mature reflection and deeper learning?
A. Take-home examinations provide far greater opportunity. 26 72
B. Take-home examinations provide a little more opportunity. 7 19
C. Take-home examinations provide the same opportunity as in-class 2 6
examinations.
D. Take-home examinations provide a little less opportunity. 1 3
E. Take-home examinations provide far less opportunity. 0 0
Table 4: Type of examination and cheating.

Question 4: In your opinion, how do the opportunities for cheating in an on- Score %
line (open book) take-home examination compare to a regular (closed book) in-
class examination?
A. Take-home examinations provide many more opportunities. 8 22
B. Take-home examinations provide a few more opportunities. 10 28
C. Take-home examinations provide the same opportunities as in-class 12 33
examinations.
D. Take-home examinations provide fewer opportunities. 3 8.5
E. Take-home examinations provide very few opportunities. 3 8.5
Table 5: Preferred type of examination overall.

Question 5: Do you think that, after considering their advantages and Score %
disadvantages, an on-line (open book) take-home examination is a superior
alternative to a regular (closed book) in-class examination?
A. A take-home examination is the far superior alternative. 16 44
B. A take-home examination is a slightly better alternative. 11 31
C. A take-home examination is no better or worse than an in-class 6 16
examination.
D. A take-home examination is a slightly worse alternative. 2 6
E. A take-home examination is a very inferior alternative. 1 3
These raw figures provide some interesting insights. 72% of respondents

were of the view that take-home examinations were less stressful than the
conventional in-class exam. 78% of respondents thought that the 3-day
period for the exam was just about right. While 91% were of the view that
take-home exams were better or no worse than in-class exams. The questions
that produced particularly interesting results, however, were those relating to
deeper learning and cheating.
A resounding 91% of respondents were of the opinion that take-home
exams provided a better opportunity for deeper learning than in-class exams.
The comments that accompanied these responses are particularly instructive:
“A take-home exam allows students time to reflect and develop their
thoughts. In an in-class exam students are usually pressed for time and so
just regurgitate what they have memorised. Much prefer take-home
exams at the post-graduate level.” (serial no. 239)
“In class exams tend to be rote learnt, and in my experience a lot of

valuable info is lost as a ‘data dump’ occurs after the exam. Take home
allows additional reading and consideration, which results in information
staying in the brain longer!” (serial no. 247)
“They allow the student to really show their understanding of the subject
through drawing on appropriate references. They permit a fairly complete
answer to be made without the rush of trying to get it all down in an
exam room, perhaps at the expense of structure and logic.” (serial no.
294)
“Take-home exam provides far greater opportunity, especially to

overseas students, this way gives them plenty time to think and organize
the words and ideas, and to explain it clearly and correctly.” (serial no.
326)
These comments are among many in response to Question 3 that provide
considerable evidence to suggest that open book, take-home examinations
are pedagogically superior to the closed book, in-class alternative. This is
extremely encouraging for those educators committed to improving the
quality of assessment.
The raw data generated in response to Question 4 is less resounding. 50%
of respondents felt that the opportunity for cheating was the same or less for
a take-home exam than it was for an in-class exam, while 50% felt there
were greater opportunities for cheating. Interestingly, the written comments
present a somewhat different picture. A selection of students answering A or
B made the following comments:
“This is purely dependent on the way the ‘exam’ is structured. In the
work environment you have access to reference material thus at the post
grad level the ‘test’ should be aimed at the students understanding of
concepts not data recall ability.” (serial no. 223)
“The chances obviously increase because there is no supervision. But,

you still need to be prepared, if you want to do well, and copying from a
text should be fairly easy to spot. The only way to really successfully
cheat would be to get somebody who is an expert in the field, or who has
previously completed the course, to do the entire exam for you.” (serial
no. 294)
“Never thought about cheating, I am here is to learn something, not just

to pass the exam. But I won’t say everyone has the same goal.” (serial no.
326)
“There are 2 sides to this. One is that there is the opportunity for some
students to free-ride off other students knowledge. It will be up to the
lecturer to spot this in the question response. The positive is that students
are able to discuss the core issues behind the question and this
encourages some students to go into further research who otherwise
would not in an examination situation.” (serial no. 393)
Looking at these responses from students believing that take-home exams

presented greater opportunities for cheating quite significant qualifications
are made regarding scope. Indeed, viewing all the responses to Questions 3
and 4 alongside one another, an interesting picture emerges. Take-home
examinations certainly present the opportunity for deeper learning, but
students recognising this suspect that not all of their peers will take
advantage of it. The key question is whether this latent doubt over the
integrity of their peers would diminish if students were sufficiently
convinced that mechanisms were in place to prevent cheats from prospering.
To test this hypothesis, a second survey was conducted the following
semester at the completion of another course unit using the same assessment
system. This time, the class consisted entirely of on-campus students, and
the number of respondents to the survey was slightly larger (46). On this
occasion, much greater emphasis was given beforehand in class, in the
course unit outline and on the OLT site, as to the kind of answer that would
attract high marks in the examination and how attempts to cheat could be
more easily monitored because of electronic submission. The same five
questions were asked of this second group of students and the responses
were broadly similar to those collected in the first survey with one notable
exception; viz. the question on the opportunities for cheating. The proportion
of respondents believing that the opportunity for cheating was the same or
less for a take-home exam than it was for an in-class exam remained at 50%,
while the proportion who felt that there were greater opportunities for
cheating fell to 24%. The remaining 26% opted to give no answer, the
written statements accompanying these responses suggesting that the
assessment system would have to develop a track record before they would
become convinced one way or the other.
4. SUMMARY AND CONCLUSIONS
While much could be made of the fact that on-line examinations are
ineffective because of the authentication problem, it is worth bearing in mind
that if a student is of a mind to cheat this is precisely what they will do. In
short, it is irrational to shy away from on-line take-home examinations on
the grounds that it is not possible for an invigilator to check the back of
calculators for crib notes. The point to observe is that neither system is
perfect. A student’s sister with a PhD from Harvard can just as easily sit an
exam for them in an examination hall as they can by sitting at the computer
in their home office. Moreover, they are equally as likely to get caught.
There are, however, a number of benefits to arise from setting on-line
take-home examinations that do not apply to the regular in-class exams.
First, and most importantly, the quality of the student learning experience is
superior as indicated by the data in this study. Second, they are more
authentic in that, as one student pointed out above (serial no. 223), in the
work environment you have access to reference material, so why create the
unrealistic scenario of the invigilated examination hall? For more discussion
on this see Nelson (1998). Third, students find the experience far less
stressful suggesting they are likely to be more productive. Fourth, on-line
take-home examinations are more equitable from the point of view of
students whose first language is not English. These students might need a
similar amount of time to native speakers to compute their ideas, but slightly
longer to put these ideas on paper. An electronically submitted open-book
examination can get around this problem. Last, and by no means least,
cybercheats who cut and paste from web sites are unlikely to prosper
because increasingly, it is critical analysis, not content knowledge, which is
the key to success.
On the subject of cheating, an unexpected bonus benefit to emerge is that
suspect examination responses are far more likely to be detected if they are
in an electronic form because of the web search engines that allow
examiners to search for word strings. Furthermore, examiners are likely to be
more inclined to pursue the unethical among the student body if they have
the time to do so. This is where two other bonus benefits come into play.
Typed printouts rather than handwritten, sometimes illegible, scripts can be
processed more quickly and answers stored electronically, rather than hard
copy, can be accessed and searched more easily. For evidence of how
electronic submission makes it is easier to police plagiarism see, for
example, the work of Bloomfield at Virginia University
(http://www.plagiarism.phys.Virginia.edu - Accessed 25/07/02).
To summarise, this paper has argued that the flexible delivery of a course
is contingent upon the existence of mechanisms that permit flexible
assessment. Assessment systems that are believed to be flexible are, in many
cases, not terribly flexible at all in that they do not cater for the increasingly
diverse student profile. If an assessment system is to be truly flexible, then
students must be presented with an element of choice and they must be able
to be assessed without having to attend examinations at a specific place and
time.
There is some way to go yet before anyone can claim to have the perfect
solution. Certainly, the success of the mode of assessment described in this
paper is, to a large extent, contingent upon the nature of the discipline or at
least dependent on the type of question(s) one asks. It works well in a subject
like economics (“the inexact science”), but uninvigilated assessment along
the lines suggested is only suitable for language-intensive examinations that
permit the electronic detection of forgeries.
This important caveat aside, it is suggested here that on-line, open book,
take-home examinations constitute a step in the right direction, so long as
three conditions are satisfied:
• examinations are as course unit specific as possible by insisting that
answers make direct reference to course materials thereby making it
difficult for outsiders to sit the exam in the time allowed;
• the time period for the exam is sufficiently tight, to prevent classmates
from submitting two significantly different answers; and
• the examiner makes it clear, as a stated objective of the course unit, that
they are looking to reward evidence of depth of learning and sound
critical analysis rather than recall of content knowledge.
If these conditions are met, and the student body is sufficiently convinced
that cheats will not prosper, putting examinations on-line for postgraduate
and undergraduate programs alike should become a whole lot easier. The
BGSB publishes a list of offences on a web site each semester and the
penalties that offenders received (minus their names). The penalties are
heavy, and it has become an integral part of the School’s institutional culture
not to tolerate any form of plagiarism.
REFERENCES
Boud, D. (1991). Three principles for good assessment practices. The New Academic,
Autumn, 4-5.
Carrier, D. (1990). Legislation as a stimulus to innovation. Higher Education Management, 2
(1), 88-98.
Faculty Education Committee (1999). Review of assessment policy and practices. Faculty of
Business, Queensland University of Technology, August.
Fröhlich, R. (2000). Keeping the wolves from the doors … Wolves in sheep’s clothing, that is.
Proceedings of the Fourth International Computer Assisted Assessment Conference, (pp.
39-46). Loughborough University, England.
Nelson, G.E. (1998). On-line evaluation: Multiple choice, discussion questions, essay, and
authentic projects. Paper presented at the Teaching in the Community Colleges, Kapiolani
Community College, April. [On-line]. (Accessed 25/07/02) URL:
http://leahi.kcc.hawaii.edu/org/tcon98/paper/nelson.html
Williams, J.B. (1998). Flexible Assessment as an Integral Part of Flexible Delivery. Paper
presented at the Fifth Educational Innovation in Economics and Business Conference,
Cleveland, USA, June.
Assertion-Reason Assessment in Formative and
Summative Tests: Results from Two Graduate Case
Studies
Luke B. Connelly
Brisbane Graduate School of Business, Queensland University of Technology, Australia
1. INTRODUCTION
This paper examines several dimensions of assertion-reason (ARN)

testing as administered to Master of Business Administration (MBA)
students of economics. The paper is principally concerned with the use of
ARN tests to serve two, quite distinct, ends.
First, it is concerned with examining an application of ARN testing for
summative purposes. More specifically, this paper provides some case study
evidence that is concerned with addressing the question: “How well does the
ARN format serve to test learning outcomes, over and above simple recall?”
This question is of practical importance. If ARN testing discriminates finely
between students with different learning outcomes, this objective test format
may be considered a good substitute to other test-types, under certain
circumstances. In particular, if ARN tests are good substitutes for these
formats, their applications might be labor-saving and/or quality-improving in
some applications. The paper presents some preliminary results that shed
some light on this particular question, albeit with some important
qualifications.
Second, this paper is concerned with the results of using ARN tests for,
primarily, formative purposes. In particular, the question of whether or not
the performance of students in a compulsory final, essay-only exam is
predicted by participation in, and performance on, six optional on-line ARN
359
360 Luke B. Connelly
tests, ceteris paribus, is of direct interest. This, second component of the

analysis comprises the central focus of the paper.
The structure of the paper is as follows. Section two provides a brief
overview of the nature of ARN tests, along with some of their benefits and
disadvantages. Section three provides a description of the data and methods
used to analyze data on a summative application of ARN test items and the
results of the analysis. Section four presents the data, methods and results for
a formative application and section five presents some conclusions.
Attention is now directed to a brief overview of the ARN format and several
of its previous applications.
2. ASSERTION-REASON QUESTIONS: A BRIEF

OVERVIEW
ARN questions belong to the genre of “objective tests”. The central

property of an objective test question (or “item”) is that its answer is pre-set
so that, at the time of marking, no further judgment by the examiner
regarding the correctness of the respondent’s answer is required. Common
examples of objective tests are those of conventional multiple-choice and
true-false items. The term “conventional” here relates to the Type A
multiple-choice questions described by Hubbard and Clemans (1961).
Like conventional multiple-choice questions (MCQs), ARN questions
invoke a multiple-choice component. However, by contrast with
conventional MCQs, ARN items also include a true/false element (CAA
Centre, 2000). Specifically, each item consists of two statements that are
linked by the word “because” and followed by a multiple-choice legend (see
figure 1).
Assertion-Reason Assessment in Formative and Summative Tests 361
Assertion Reason
Provided an economy produces an BECAUSE The production possibilities
output combination that is “inside” frontier describes the maximum
its production possibilities frontier, combinations of two outputs that
it may be said to be technically (or can be produced by an economy
productively) efficient. that maximises its output, given
the fixed stock of inputs
available.
(a) True; True; Correct reason

(b) True; True; Incorrect reason
(c) True; False
(d) False; True
(e) False; False
The correct answer is d.
Figure 1: An Assertion -Reason Test Item.
Solving the answer to such a question involves at least two, but up to

three, steps. First, the respondent must determine whether the “assertion” is
“true” or “false” and whether the “reason” is “true” or “false”. If one, or
both, of the statements is deemed “false”, the answer has been determined, in
that one of the responses (c), (d) or (e) applies. A third step is required only
if both statements are deemed to be true and involves determining whether
the second statement provides an accurate explanation for the first. When
this third step is required, the respondent must choose between responses (a)
and (b). If the reason statement is deemed to provide an accurate explanation
of why the assertion statement is true, response (a) is chosen. If, on the other
hand, the reason statement is true but does not constitute a reasonable
explanation of the (true) assertion statement, response (b) is chosen. Such
questions are sometimes referred to, in the medical education literature, as
“relationship analysis” or “Type E” questions, following the classification of
Hubbard and Clemans (1961) (Fox, 1983).
There is nothing particularly new about ARN tests. According to
Heywood (1999), ARN questions first appeared in the UK in the 1960s in A-
level secondary school examinations. However, their introduction in US
medical exams appears to predate their application in the UK (Moore, 1954).
Notwithstanding the vintage of ARN items and the voluminous literature
on MCQs more generally, the literature on ARN items per se is apparently
miniscule. The empirical literature on ARNs, in particular, is meager. Case
studies by Newble, Baxter and Elmslie (1979), Skakun and colleagues
(1979), Fox (1983) and Scouller and Prosser (1994) were the only empirical
examples located by this author after an extensive literature search, including
on-line full text databases, such as ERIC and academic Search Elite.
Furthermore, all of these studies come from the field of medical education.
The apparently limited practical application of this test format, especially
outside medicine, together with the small body of literature dedicated to
ARN tests, raises a number of interesting questions regarding the veracity
and general usefulness of the item-type itself and/or the difficulties that may
be related to the construction of ARN questions.
One of the attractive attributes of the ARN item is that its structure
facilitates the construction of questions that test student learning beyond
recall. In particular, higher-level thinking and application of key concepts
may sometimes be more easily constructed using this format, than by using a
conventional multiple-choice approach alone. (For an excellent discussion of
the various conceptions of higher level thinking and its assessment using
“conventional” multiple choice items, see Haladayna, 1999.) An interesting
example of the use of ARNs was produced by Moore (1954) and is reprinted
in Hubbard and Clemans (1961). Moore’s (1954) purpose was to show how
a bank of MCQ questions, which includes ARN items, could be used to
supplant an essay question on a National Board of Medical Examiners’
pathology exam.
There are, of course, difficulties associated with using ARN questions as,
indeed, there are problems with any test item-type. One of the difficulties
relates to the fact that the ARN test format involves the incorporation of a
true/false component:
“[i]t is extremely difficult to set good true/false items, mainly because it
is not easy to find many statements which are true or false beyond a
peradventure. It is also, rather paradoxically, difficult to set true/false
items which are not either too easy or too difficult. The easy type of
true/false item is of the type ‘Chicago is the capital city of the United
States. True/false’. A difficult type of true/false item is one based, for
example, on a philosophical argument or on a complicated description of
a firm’s balance sheet or a country’s economic position.” (Macintosh &
Morrison, 1969, p. 22).
Hudson also refers to the difficulty of setting questions that are
unequivocally true or false in the context of ARN test setting, but he also
adds the following point:
“[i]tems with keys [(a) or (b)] are particularly difficult to write since,
when both the statements are true, there must be no doubt as to whether
the second is or is not a legitimate reason for the first. The distinction
between keys [(a) and (b)] needs very careful consideration...[to] ensure
that the problem does not become too subtle for the pupils for whom the
test is intended...” (1973, p. 135).
The applications of ARN testing that are analyzed in this paper relate to
two economics core subjects in an MBA course. The experiences of this
author and a colleague have been that writing ARN test items for these two
courses has proven neither a particularly difficult, nor onerous, task.
Nonetheless, this author’s initial experiences with the ARN format gave rise
to very low mean scores (e.g. 4-5 out of 10) in this student group. The
cautionary note provided by Hudson (1973), regarding the need to attenuate
the subtleties of ARN items is of considerable practical importance in this
author’s experience.
A related point is that the ARN format is considerably more time-
intensive for students than is conventional MCQ. Qualitative evidence from
our first applications of the format for the MBA group at the Queensland
University of Technology (QUT) indicated that an allowance of one to one
and a half minutes per question was generally considered too meager by the
student body. Our most recent applications of this test format have allowed
approximately two minutes per question, including perusal time.
Finally, it is worthwhile to point out that ARN items represent a novel
MCQ format for most students. Indeed, anecdotal evidence from the MBA
group indicates that none of our students had experienced this test format,
before enrolling on our MBA. Thus, we have found it important to provide
all students with the opportunity to review, and ask questions about, the
ARN format well in advance of our in-class ARN tests. This decreases the
risk that our ARN items will act as de facto tests of general intelligence
(Hudson, 1973), rather than tests of learning. It also takes account of the
results of the empirical study by Skakun and colleagues (1979), which may
have been downward-biased by the fact that their candidates first
encountered ARN items under examination.
In addition to the above observations about ARN, it is also worthwhile
noting that ARN items are subject to many of the pros and cons that are
commonly associated with conventional multiple-choice test items. A
common sentiment is expressed by Wood:
“Multiple choice does best when it checks out factual knowledge quickly.
Claims are made that it can do more and item types have been
constructed which appear to elicit higher order skills ... yet there is still
uneasiness about the supporting evidence.” (1991, p. 35).
There are probably several important reasons for the “uneasiness” of
(some) educators about the effectiveness of MCQs as tests of deep learning.
One reason is conceptual in nature. Exactly what is meant by “higher
learning” or “deeper learning” is, itself, a matter that is subject to debate

(Haladyna, 1999) and, indeed, it has been argued that “attempts to describe
and classify higher order skills have amounted to little” (Wood, 1991, p. 48).
Another reason relates to the state of the empirical evidence and the nature
of the required research. Wood summarizes the empirical evidence on MCQ
tests as follows:
“Areas [for investigation] have been pinpointed where research has been
thin when it might have been substantial. As usual, the reason is that it is
tricky research to do ... The soft research has to do with the ‘nuts and
bolts’ of the MCQ and there has been no lack of that ... Would that the
same time and effort had been put into studying essay tests” (1991, p.
37).
Consider, for example, the very high levels of scoring error reported by
Hubbard and Clemans (1961) in their review of the early empirical literature
on scoring errors in MCQ and essay assessment items. In one case study,
correlations on the scores of essay responses were as low as 0.55, despite the
dissemination of detailed marking criteria! However, a detailed
consideration of this particular dimension of assessment validity is outside
the scope of this paper. Attention is now directed to conducting some of the
“tricky research” to which Wood (1991) has referred.
3. EMPIRICAL ANALYSIS: A SUMMATIVE

APPLICATION OF ASSERTION-REASON ITEMS
This component of the empirical analysis is motivated by a concern with

the substitutability, or otherwise, of ARN questions, MCQs and/or short-
answer (SAN) questions (sometimes referred to as “free-response”
questions) as summative assessment tools. A discussion of the data and
methods precedes the presentation and discussion of the results.
3.1 Data
The data employed in this section of the paper pertain to an economics

subject that is offered by the BGSB. It is a core (i.e. compulsory) unit in the
MBA program. The sample is a class of 86 Economics of Strategy I students
that undertook the subject in second semester of 2000. The class comprised
Graduate Certificate of Management (GCM), Graduate Diploma of Business
Administration (GDBA), Master of Business (MBus) and MBA students.
Specifically, the composition of the student group was as follows: GCM

(n=1), GDBA (n=3), MBus (n=5), and MBA (n=77).
The data analyzed here are primarily by-products of the subject’s
assessment scheme. The assessment for the subject comprises one,
compulsory, invigilated final examination and two optional assessment
items, viz. an individual essay assignment and an invigilated, mid-semester,
ARN test. The final examination for the unit is composed of three sections:
an ARN sub-test, an MCQ sub-test and an SAN sub-test. Given the
compulsory nature of the final examination, data are available for 86
observations on each of the three final exam sub-tests. The number of
observations on the optional items of assessment is determined by the
number of (voluntary) submissions. For the optional essay assignment, the
number of submissions was 35, while the number of optional mid-semester
test observations is 78. It is serendipitous that this combination of optional
and compulsory items is useful for the empirical analyses conducted here.
These useful implications are discussed in Section 3.2. The high
participation rate in the optional mid-semester test is explained by the
marking scheme employed in this unit. Marks on optional assessment items
are only counted towards the student’s final grade if they exceed, in
percentage terms, the mark achieved on the final examination. Thus, while
participation in the mid-semester examination is optional, there is really no
“downside” for participants who sit the examination unprepared. The
assessment structure and marking schemes are motivated by a desire to
provide useful formative assessment items for students enrolled in this
subject.
3.2 Methods
The empirical analyses are primarily quantitative in nature and involve (i)
statistical comparisons, by correlation analysis, of student performance on
banks of each final examination test item-type; (ii) statistical comparisons,
by correlation analysis, of student performance on the optional essay, with
performance on the final examination sub-tests; and (iii) multiple regression
analyses of the impact on the final examination ARN sub-test score, of
student participation in, and performance on, the optional mid-semester
ARN test and essay items.
These methods were chosen for the following reasons. First, correlation
analysis of the final exam sub-scores will reveal the extent to which student
performance on an ARN test are more or less correlated with standard MCQ
or SAN scores. Second, correlation analyses of all final exam sub-tests with
the optional essay scores may provide a useful validating method.
Specifically, properly constructed and assessed essays can be useful devices
for examining higher-level learning (Haladyna, 1999), especially in the

social sciences (Brown, Bull & Pendlebury, 1997). Thus, correlations of
student performance on essays and an ARN test may provide some insight
into the extent to which our ARN sub-tests measure deeper learning. Third, a
multiple regression technique has been chosen to examine the impact of
student participation and scores achieved in the mid-semester ARN test to
account for several factors that may be expected, a priori, to confound
simple test score correlations. This final component of the analysis deserves
some further explanation.
To reduce noise that might “contaminate” simple test-score correlations,
a multiple regression model of the following form was specified, and
estimated, in levels:
FARSTi = β 0 + β1 MIDSEM i + β 2 ESSAYi + β 3 DVGCM i + β 4 DVGDBAi +

β 5 DVMBAi + β 6 DVESSAYi + β 7 DVMIDSEM i + ε i
In this equation FARSTi is the ith student’s percentage score on the final
ARN sub-test; MIDSEMi is the ith student’s percentage score on the optional
mid-semester ARN test; ESSAYi is the ith student’s percentage score on the
optional essay assignment; DVGCMi is a dummy (or binary) variable =1 if
the ith student is enrolled in the GCM program; DVGDBAi is a dummy
variable =1 if the ith student is enrolled in the GDBA program; DVMBAi is a
dummy variable =1 if the ith student is enrolled in the MBA program;
DVESSAYi is a dummy variable =1 if the ith student submitted the optional
essay item; DVMIDSEMi is a dummy variable =1 if the ith student sat the
optional mid-semester ARN test; εi is a well-behaved stochastic error term;
and βi are the parameters to be estimated. A priori, the expected sign of all
coefficients, with the exception of the program dummy parameters (β3, β4,
and β5), is positive. There is no a priori expectation of a particular sign on
the course dummies.
Equation (1)’s specification simultaneously addresses several
methodological issues. First, it distinguishes between student participation
in, and results achieved on, both optional items of assessment. This is
advantageous because, if the scholarly attributes of students who opt to do
the essay are significantly different to those who opt not to, the mean ARN
test score of the groups might also be significantly different. The dummy
variables for participation (DVESSAY and DVMID) help to control for the
possibility of sample bias. Similarly, it is possible that enrollees in different
degree programs have systematically different scholarly attributes. If this
were true, failure to control for these attribute differences (e.g., by using the
dummy variables DVGCM, DVGDBA and DVMBA) might also give rise to a
biased estimate of the extent to which essay and ARN sub-test scores are
correlated for pedagogic, rather than sampling, reasons. More specifically, if
scholarly attributes differ by degree program, the extent to which the ARN
sub-test acted as a de facto test of intelligence, academic experience and
similar factors, as opposed to a test of learning outcomes, might be discerned
from parameters β3, β4 and β5.
The technique applied to estimate Equation (1) is ordinary least squares
(OLS), and a general-to-specific [or “Hendry” (Pagan, 1995)] modeling
approach is adopted. Since an initial inspection of the data revealed the
presence of heteroscedasticity, all t-statistics reported for this component of
the analysis were calculated using White’s heteroscedasticity corrected
standard errors (HCSEs). For general introductory discussions of the
statistical techniques employed in this paper see Gujarati (1995), Pindyck
and Rubinfeld (1998); Greene (2000); Judge et al. (1988) and Davidson and
Mackinnon (1993).
3.3 Results and Discussion
Table 1 presents a simple correlation matrix for student scores on final

examination sub-tests and the optional assignment.
Table 1: Correlation Coefficients (ρS) for Student Results on Final Examination Sub-tests.
FARST FMCQ FSAN ESSAY DVAS
FARST 1.00 0.46 0.40 -0.10 -0.17

FMCQ 0.46 1.00 0.51 -0.05 -0.07
FSAN 0.40 0.51 1.00 -0.15 -0.19
ESSAY -0.10 -0.05 -0.15 1.00 0.98
DVAS -0.17 -0.07 -0.19 0.98 1.00
Notes
(i) FARST is the percentage score on the final (examination) ARN sub-test.
(ii) FMCQ is the percentage score on the final examination multiple-choice question sub-
test.
(iii) FSAN is the percentage score on the final examination short-answer sub-test.
(iv) ESSAY is the percentage score on the optional essay assignment.
Initially, consider the data in the first three rows and columns of table 1.
These data are correlates of student scores on the ARN sub-test (FARST), the
multiple-choice sub-test (FMCQ) and the short-answer sub-test (FSAN)
components of the final examination. It is noteworthy that all of the off-
diagonal correlation coefficients in this region are positive, but that none is
particularly large. The positive sign on these coefficients is reassuring.
Generally students who performed better than their peers on one section of
the exam, performed better on all other sections of the exam. The most
strongly correlated final examination scores are those on the MCQ and SAN
sections of the examination (ρ=0.51), perhaps suggesting that these banks of
questions were closer substitutes, in terms of the type(s) of learning tested,
than the other exam sub-sections. The remaining correlation coefficients on
FARST also indicate that scores on the ARN sub-test were more highly
correlated with those on the MCQ sub-test, than those on the SAN sub-test.
Now consider the correlation coefficients for ESSAY and DVESSAY, with
respect to each of the final sub-test scores. Each of these coefficients is
negative. The negative coefficients on DVESSAY suggest that students who
chose to submit optional essays fared more poorly, on average, in each final
examination sub-test than their counterparts. More alarming, though, are the
negative correlation coefficients on ESSAY and the exam sub-tests. Although
very small in magnitude, these suggest that students with higher essay marks
fared worse on every sub-test on the examination.
Are the results of these simple correlations useful? Well, these results
may tell part of an important story, but they are univariate in nature.
Additionally, simple correlations of this kind provide limited insights for the
reasons demonstrated by Choppin (1974), and neatly summarized by Wood
(1991). For these, and the reasons outlined in Section 3.2, it is therefore wise
to direct attention to the results of multiple regression analyses of the data.
Table 2 provides the estimated coefficients and the output of diagnostic
tests for both the general and specific forms of Equation (1). Note, however,
that the binary variable, DVMIDSEM, was not included, even in the general
form of the model. The reason is that Equation (1) is inestimable when both
DVMIDSEM and MIDSEMESTER are included, since these variables are
almost completely collinear. The near-perfect collinearity of these variables
is indicated by their correlation coefficient (r=0.98), in table 1 (for a
discussion of the econometric consequences of multicollinearity see Greene,
2000).
Attention is directed to the parsimonious model 2
reported in column (iii)
of table 2. The “goodness-of-fit” (i.e. the R and F-) statistics for this
equation are reasonable, given the cross-section nature of the sample, and the
residuals are normally distributed, as evidenced by the Jarque-Bera statistic.
The model also passes the Regression Specification Error Test (RESET)
(Ramsey, 1969).
Table 2: Ordinary Least Squares Results of Estimating Equation (1): Dependent Variable is
Variable (i) GENERAL MODEL SPECIFIC MODEL
Coefficients (ii) Coefficients (iii)
Intercept 55.20* 51.18*
(6.97) (9.65)
ESSAY 1.04* 1.02*

(2.97) (2.92)
MIDSEM 0.30* 0.28*

(3.67) (3.50)
DVESSAY -80.20* -78.67*

(-2.98) (2.93)
GCM -12.46*** -8.02**

(-1.69) (-2.06)
GDBA -9.62 -
(-1.03)
MBA -4.63 -
(-0.68)
Goodness of Fit and Diagnostic tests

R2 2 0.25 0.25
R 0.19 0.20
Equation F-Statistic 4.07* 6.00*
Ramset RESET (F-Statistic) 1.91 1.86
Jarque-Bera Normality Test (of Residuals) 1.05 1.01
FARST
Notes
(i) * indicates statistical significance at the 1% level.

(ii) ** indicates statistical significance at the 5% level.
(iii) *** indicates statistical significance at the 10% level.
(iv) Data in parentheses are t-statistics.
(v) All t-statistics have been calculated on heteroscedasticity-consistent standard errors
(HCSEs) (see, e.g. Hall et al. (1995)).
(vi) RESET is the Regression Specification Error Test (Ramsey, 1969).
(vii) See section 3.2 for definitions of the variables referred to in column (i).
Now consider the estimated coefficients. The coefficient on ESSAY is of

primary interest: it is positive and statistically significant at the 1% level.
Evaluated at the mean, this coefficient indicates that, ceteris paribus, a 1%
increase in a student’s essay mark was correlated with a 1.02% increase on
the ARN sub-test score. This is an interesting result, because it suggests a
near-perfect correlation between learning outcomes, as assessed on an essay
item, and learning outcomes as assessed on an ARN test. Note also, that the
dummy variable DVESSAY has a negative sign and is also statistically
significant. This suggests that, ceteris paribus, the average score of students
who submitted an optional essay was lower than the average for students
who chose not to submit an essay. Thus, it turns out that the dummy variable
for assignment submission was an important inclusion in this model. Indeed,
when only the assignment score is included in the regression, it is a
statistically insignificant predictor of performance in the final 2ARN sub-test
and has a negative sign. Furthermore, the resulting R for such a
specification is only 0.09.
The positive coefficient on MIDSEM indicates that, ceteris paribus,
students who performed well on the mid-semester ARN test also performed
well on the final ARN sub-test. Finally, the negative, and statistically
significant coefficient on GCM indicates that students enrolled in the GCM
typically did not perform as well on this final examination sub-test as
students enrolled in other graduate programs. This too, is an interesting
result that, incidentally, also arises if the final grade in the unit is regressed
on the course dummies. Such a difference in performance could be
associated with a difference in student characteristics per se, but it might
also be a function of the stage at which students enroll in the unit.
Specifically, GCM students are likely to enroll in this unit earlier in their
studies, than are students in the GDBA and MBA. This proposition is based
on the fact that the GCM is a 48 credit point course, while the GDBA and
MBA programs are 96 and 144 credit points, respectively. In future analyses,
it would be useful to control for the student’s semester of enrolment, in
addition to his/her program of enrolment.
An important question to answer in relation to these findings is: “Are
these results peculiar to the ARN sub-test scores or, do similar results arise
when the MCQ and SAN scores are regressed on the same independent
variables?” The answer is given by considering table 3. This table presents
the results of regressing FMCQ and FSAN on the independent variables for
the “Specific” model in table 2.
Table 3: Ordinary Least Squares Results on Dependent Variables FMCQ and FSAN
Variable FMCQ FSAN
(i) SPECIFIC MODEL SPECIFIC MODEL
Coefficients Coefficients
(ii) (iii)
Intercept 51.41* 62,93*

(10.58) (25.05)
ESSAY 0.25 0.21

(0.90) (0.78)
MIDSEM 0.26* 0.37*

(3.14) (4.04)
DVESSAY -19.86 -27.95

(-0.97) (-1.18)
GCM
-21.54 -23.58
(-1.32) (-1.24)
Goodness of Fit and Diagnostic Tests

R2 0.15 0.22
R2 0.11 0.18
Equation F-Statistic 3.37** 5.33*
Ramset RESET (F-Statistic) 0.23 0.30
Jarque-Bera Normality Test (of Residuals) 1.07 8.54**
Notes

(iv) Data in parentheses are t-statistics.
(v) All t-statistics have been calculated on heteroscedasticity-consistent standard errors
(HCSEs) (see, e.g. Hall et al. (1995)).
(vi) RESET is the Regression Specification Error Test (Ramsey, 1969).
(vii) See section 3.2 for definitions of the variables referred to in column (i).
Notably, there is no statistically significant correlation between ESSAY

and the sub-test score on either remaining section of the final examination.
This result may suggest that our ARN questions were more discriminating
tests of deeper learning than were our MCQ and SAN items.
In this section, empirical evidence has been produced to suggest that
ARN items may be effective tests of deeper learning. Furthermore, the
empirical results from this case study also suggest that ARN items may be
close substitutes for essay items in this subject area. This finding contrasts
with the results produced by Newble, Baxter and Elmslie (1979), who found
that SAN tests were more discriminating measures of interns’ clinical
abilities than objective tests that employed true/false questions and
conventional MCQs. The comparability of these results is, however, limited
not only because the subject areas are disparate, but also because these
authors did not employ ARN items in their test.
There are numerous caveats on the results reported in this section. The
sample size is modest; the case study is subject-specific; and the style of
question-writing in each section could influence the results to an important
degree. Notwithstanding these caveats, the results do shed some light on a
subject that demands illumination.
Attention will next be focused on an empirical analysis of a formative
application of ARN items.
4. EMPIRICAL ANALYSIS: A FORMATIVE

APPLICATION OF ASSERTION-REASON ITEMS
The empirical analysis presented in this section is concerned with the

correlation, if any, between student participation in, and performance on, six
formative ARN tests, and performance on a final, essay examination. Fox
(1983) has reported positive and statistically significant correlations of
student participation in MCQ tests (which included ARN items) and final
grades at the undergraduate level. Here, the concern is somewhat more
specific, viz. do formative ARN tests appear to influence student
performance on an essay exam? Beneath this more general question is also
the question of whether or not engaging students in questions designed to
test deep learning, foster a deep learning approach and/or outcome by the
student. Scouller and Prosser (1994) provide an analysis of the relationships
between “surface” and “deeper” learning approaches and, inter alia, student
performance on a final MCQ exam. Their paper provides empirical support
for the notion that deep learning approaches to study are positively
correlated with quantitative measures of student achievement.
4.1 Data
The empirical analyses conducted in this section employ assessment data

from another economics core subject in the MBA, viz. Business Conditions
Analysis I. This unit has one, compulsory final examination and three
optional items of assessment. The optional items of assessment are (i) an in-
class group presentation; (ii) an essay assignment; and (iii) six on-line ARN
tests of ten questions each.
For the purposes of this unit, optional item (iii) was considered primarily
a formative assessment tool. However, to encourage participation, students
were offered 5% of the unit weight if they (a) completed each of the six on-
line tests; and (b) scored at least seven out of 10 on their final attempt at
each test. Students were also encouraged to undertake each test until a
perfect score (10/10) was achieved. Students were made aware that no
penalty would be applied for repeating any on-line test.
The sample class was taken from second semester 2000 and comprised
84 students. Of these, four were enrolled in the GCM or the Graduate
Certificate in Business Administration (GCBA), seven were enrolled in the
GDBA, three were enrolled in the MBus, and the remainder were enrolled in
the MBA program.
To take advantage of the data from each of the six formative ARN tests,
cross-section and time-series data were pooled for this analysis. More
specifically, the data were stacked “by time” (i.e. by student results for tests
1 through 6), with the (unique) final examination results stacked against each
these. The following equation was then estimated:
FINALEXAM i = β 0 + β 1 NATTij + β 2 FIRSTAij + β 3 HIGHESTij + β 4 DVESSAYi

+ β 5 DVPRES i + β 6 DVONLINEi + β 7 DVGCM + β 8 DVGDBA
+ β 9 DVMBA + φit
In this equation FINALEXAMi is the ith student’s percentage score on the

final (essay) exam; NATTit is the number of attempts by the ith student at the
jth on-line ARN test; FIRSTAij is the ith student’s score on the first attempt
at the jth on-line ARN test; HIGHESTij is the ith student’s highest score on
the jth test; DVESSAYi is a dummy variable =1 if the ith student submitted an
optional essay; DVPRESi is a dummy variable =1 if the ith student
participated in an optional presentation; and DVONLINEi =1 if the ith
student participated in at least one of the j on-line tests, φ it is a well-behaved
error term; and βi are the parameters to be estimated. A priori, the expected
sign of NATT, DVGCM, DVGDBA and DVMBA are indeterminate, while the
expected signs on all other variables is positive. The sign on NATT, in
particular, is indeterminate because the number of attempts may
simultaneously be positively correlated with student effort and inversely
correlated with student aptitude. Furthermore, a large number of attempts
may even signal uninformed guessing by students, whose specific objective
is to qualify for the 5% weight for participation without engaging the test
items.
Some apparent alternatives to this approach are: (i) to estimate cross-

section equations for each individual student; (ii) to estimate a cross-section
equation for each formative test (i.e. to run separate regressions for all j=1 to
6); (iii) to suppress the time-series dimension of the data set by substituting
the means of NATTit, FIRSTAit, and HIGHESTit for those variables; or (iv) to
adopt a procedure using some combination of (i) and (iii) or (ii) and (iii).
The difficulty with the first of these alternatives is that only one final
examination observation is available for each student, so (i) is not a tractable
alternative. It is feasible to proceed using method (ii), but the conceptual
appeal of relating a student’s final exam score to a single, 10-question,
formative test is limited, at best. Method (iii) is the most attractive of these
alternatives, but it entails the suppression of detailed data that are available,
by test, on each student. For these reasons, Equation (2) has been estimated
only on the stacked data described above.
The chosen method of estimation for Equation (2) is Feasible
Generalized Least Squares (FGLS) (Greene, 2000). An application of this
method assumes the presence of cross-sectional heteroscedasticity and uses
the cross-section residual variances, as weights, to correct for its presence
(see Hall, Lilien, Sueyoshi, et al., 1995). Table 4 presents the estimated
coefficients and goodness-of-fit data for Equation (2). It also presents the
FGLS estimates of the parameters in Equation2
(2), along with the goodness-
of-fit statistics for the regression. The R and Equation F-statistics indicate
that this model is an extremely good fit to the data. All coefficients are
statistically significant at the 1% or 5% level, and all parameters have the
expected signs.
Interestingly, the program dummy variables (i.e., DVGCM, DVGDBA
and DVMBA) again indicate systematic differences in the final exam
performances by students enrolled in different programs. In this regression,
all three course dummies have statistically significant coefficients and the
magnitudes of the (negative) coefficient on DVGCM, and the (positive)
coefficient on DVMBA, in particular, are non-trivial.
Now consider the coefficients on the variables of primary interest in this
study, that is those on NATT, FIRSTA, HIGHEST and ONLINEDV. Note that
the first of these is negative: ceteris paribus, a higher number of attempts is
correlated with a lower final exam score (although the percentage impact i.e.
-0.06%, is trivial). Conversely, higher first-attempt (FIRSTA) and final
attempt scores (HIGHEST) are both positively correlated with final
examination performance. Indeed, the magnitude of the latter coefficient is
close to unity, which suggests that the performance on the last attempt is
almost perfectly positively correlated with performance on the final essay
exam. Finally, note the large magnitude of the coefficient on the binary
variable for ONLINEDV. Recall that this variable distinguishes between
participants and non-participants in the on-line ARN tests. The literal

interpretation of its coefficient is that students who participated in the on-line
tests did, on average, approximately 12% better on the final exam than did
non-participants.
Table 4: FGLS Regression Results on Equation (2): Dependent Variable is FinalExam.

Variable Coefficient
Intercept 34.31*
(34.30)
NATT -0.06*
(-2.63)
FIRSTA 0.30*
(11.19)
HIGHEST 0.97*
(24.30)
DVESSAY 5.38*
(23.75)
DVPRES 5.72*
(4.66)
DVONLINE 12.36*
(16.93)
DVGCM -15.01*
(-86.59)
DVGDBA -1.53*
(-10.30)
DVMBA 8.42*
(34.56)
Goodness of Fit (Weighted Statistics)

R2 2 0.99
R 0.99
Equation F-Statistic 11574.73*
Notes
(iv) see section 4.2 for a description of the variables referred to in this table.
How much, if any, of the effect registered by DVONLINE is actually due

to (i.e. caused by) participation in on-line testing? This question cannot be
answered, using the present data set. The result on ONLINEDV could simply
reflect differences in the underlying scholarly attributes of participants and
non-participants. However, to the extent that this variable is a proxy for
student attributes, it provides reassurance that the estimated coefficients on
NATT, FIRSTA and HIGHEST measure the phenomena of central interest
here. Similarly, the coefficients on DVPRES and DVESSAY may indicate: (i)
causal effects of these items of formative assessment; (ii) student sorting by
participation; or, most likely, both.
The empirical results of this regression analysis appear to provide
evidence of the usefulness of formative on-line ARN tests. In particular,
participation in and performance on the on-line tests appear to be good
predictors of performance in the summative final essay exam.
5. CONCLUSIONS
This paper has examined two applications of ARN tests in graduate

business education. In particular, the analyses were concerned with the
success with which ARN tests can be applied to assess deeper learning, in a
summative manner and foster deeper learning, in a formative manner.
The empirical results on summative assessment indicate that ARN items
may be useful tests of deeper learning. Indeed, for the cases studied here, our
ARN items appear to be marginally superior tests of deeper learning than our
MCQ and SAN items.
However, the more compelling results are those on a formative
application of ARN tests. The results suggest strong, positive relationships
between formative ARN test participation and outcomes and results on a
final essay-only examination. This is an encouraging result, which suggests
that ARN items may be useful tools for the development of deeper learning
outcomes.
These case study results add an empirical dimension to a reasonably
small contemporary discussion about the educational merits of ARN tests.
Although both case studies employ samples of an acceptable size, further
empirical tests of the effectiveness of this test type in business education are
necessary if ARN items are to be employed with confidence. In particular,
research on other applications of this test type (e.g. in undergraduate
curricula) and in business disciplines, other than economics, may be fruitful.
In addition, the quantitative techniques employed in this paper might be
supplemented with qualitative methods of analysis and research might be
conducted on the relative performance of students with different learning

styles and from different language backgrounds.
Another topic for future research is whether or not ARN items, used in
formative applications, affect students’ approaches to learning. For example,
one may ask whether this item type, which appears to encourage critical
analysis, is also an effective way of encouraging students who adopt
“surface” (e.g. rote) learning strategies to adopt “deep” learning strategies.
ACKNOWLEDGEMENTS
I gratefully acknowledge the financial assistance of the Queensland

University of Technology under the Small Teaching and Learning Grant
scheme. I also wish to acknowledge the collaborative contributions of Dr
Jeremy Williams on the construction of ARN items and the assistance of Liz
Heathcote and others in the Teaching and Learning Support Services
division at QUT, with the implementation of the on-line tests referred to in
this paper. I also wish to thank Dr Jason Fitzsimmons for his capable
research assistance; Dr Carol Dalglish for her useful comments on some
qualitative dimensions of this research; and Dr Laurent Falque for his helpful
comments on an earlier draft of this paper. Needless to say, the usual caveat
applies.
REFERENCES
Brown, G., Bull, J., & Pendlebury, M. (1997). Assessing student learning in higher education.
Routledge: London.
CAA Centre (2000). Designing and using objective tests. Paper presented at workshop for
colleagues at Queensland University of Technology. CAA Centre: Luton.
Choppin, B.H. (1974). The correction for guessing on objective tests. IEA Monograph
Studies, No.4, Stockholm: IEA.
Davidson, R., & MacKinnon, J.G. (1993). Estimation and inference in econometrics. Oxford:
Oxford University Press.
Fox, J.S. (1983). The multiple choice tutorial: Its use in the reinforcement of fundamentals in
medical education. Medical Education, 17, 90-94.
Greene, W.H. (2000). Econometric analysis (4th ed.). Upper Saddle River: Prentice-Hall.
Gujarati, D.N. (1995). Basic econometrics (3rd ed.). New York: McGraw-Hill.
Judge, G.G., Hill, R.C., Griffiths, W.E., Lütkepohl, H., & Lee, T-C. (1988). Introduction to
the theory and practice of econometrics. (2nd ed.). New York: Wiley.
Haladyna, T.M. (1999). Developing and validating multiple-choice test items (2nd ed.).
London: Lawrence Erlbaum Associates.
Hall, R.E., Lilien, D.M., Sueyoshi, G. et al. (1995). EViews user guide version 2. Irvine:
Quantitative Micro Software.
Heywood, J. (1999). Review: assessing student learning in higher education. Studies in

Higher Education, 24 (1), 133-34.
Hubbard, J.P. & Clemans, W.V. (1961). Multiple choice questions in medicine: a guide for
examiner and examinee. Philadelphia: Lea and Febiger.
Hudson, B. (1973). Assessment techniques: An introduction, London: Methuen.
Macintosh, H.G., & Morrison, R.B. (1969). Objective testing, London: University of London
Press.
Moore, R.A. (1954). Methods of examining students in medicine, Journal of Medical
Education, 29 (1), 23-27.
Newble, D.I., Baxter, A., & Elmslie, R.G. (1979). A comparison of multiple-choice and free-
response tests in examinations of clinical competence. Medical Education, 13, 263-68.
Pagan, A. (1995). Three econometric methodologies: A critical appraisal. In L. Oxley & C.J.
Roberts (Eds.), Surveys in Econometrics (pp. 9-29). Oxford: Blackwell.
Pindyck, R.S., & Rubinfeld, D.L. (1998). Econometric models and economic forecasts (4th
ed.). Boston, Mass: McGraw-Hill.
Ramsey, J.B. (1969). Tests for specification errors in classical linear least-squares regression
analysis. Journal of the Royal Statistical Society, Series B, 31, 350-71.
Scouller, K.M., & Prosser, M. (1994). Students’ experiences in studying for multiple choice
question examinations, Studies in Higher Education, 19, 267-79.
Skakun, E.N., Nanson, E.M., Kling, S., & Taylor, W.C. (1979). A preliminary investigation
of three types of multiple choice questions. Medical Education, 13, 91-96.
Wood, R. (1991). Assessment and testing: a survey of research commissioned by the
University of Cambridge Local Examinations Syndicate. Cambridge: Cambridge
University Press.
Exploring the Implications of Videoconferencing for
Management Learning
Vivien Hodgson & Mireia Asensio

Department of Management Learning, The School of Management, Lancaster University, UK
1. INTRODUCTION
Videoconferencing has been used for some time in Higher Education

(HE). As with other educational technologies different reasons for using
videoconferencing for education purposes are often suggested, these range
from it being seen as a substitute, to an alternative, to an enhancement to
current provision. In recent years it has been increasingly used in the context
of management education programs and it is for this reason we would like to
share some of our research and findings on the different ways it is used and
how this is experienced by students.
In a recent field study on the students’ learning experience of using a
variety of networked learning technologies it was found that the least
favored, most unreliable and least understood learning environment by
students was videoconferencing. In that study the media was generally
perceived as a “second best” learning experience, but appropriate for the
provision of access to remote students (see JISC/CALT, 2000). In the three
case studies presented in this paper the reasons for its application in each
instance varied. They included the provision of access to remote students;
shortage of teaching staff; cost saving; and providing the means for
collaborative work amongst geographically dispersed learners in Europe.
Though we believe the reasons behind the application of technology can
have an influence on the way the students perceive its use, it is more relevant
for us to examine how videoconferencing is being used pedagogically and
what are the different design assumptions that are made in each case. It is
379
380 Vivien Hodgson & Mireia Asensio
equally important to examine more closely how the students experience the
different pedagogical approaches and designs that are applied when using
videoconferencing. In the following section we consequently describe the
three case studies in terms of the reasons for the application of
videoconferencing and the distinctive learning design used in each case.
Essentially the case studies used videoconferencing to support either lecture
delivery, tutorial sessions or collaborative project work amongst distributed
students. The contrasting differences of experience that emerge from the
case studies have pointed to two questions.
• “Did the technology impact on the way the students experienced the
learning event?” This question covers connection problems, problems
with sound image, etc.
• “Did the different design models have a qualitative impact on the way
the students experienced videoconferencing?”
Through the exploration of these questions, as described in the next
section, we identified two further areas of enquiry: the significance of
presence and the significance of interaction. Examination of these two
aspects led us to look closer at the nature of engagement in learning and,
building on the work of Kearsley and Schneiderman (1999) on engagement
theory, we suggest that this is potentially of greater influence on the
students’ experience than the media as such. We suggest that the nature of
engagement should thus be considered in the learning design of
videoconferencing sessions.
2. THE THREE CASE STUDIES
These three contrasting case studies took place in different university

institutions in the UK and in Europe. Separate evaluations of each case study
were carried out drawing on the phenomenographic tradition and
interviewing method described by Marton, who defines phenomenography
as:
“the empirical study of the different ways in which people experience,
perceive, apprehend, understand, or conceptualize various phenomena in,
and aspects of, the world around them” (1994, p. 4424)
This type of research approach is helpful when the aim is to describe
qualitatively different ways of experiencing specific phenomena. The use of
non-structured interviewing was particularly valuable to obtain rich and
illuminative accounts of the students’ experiences. The use of observation in
the natural setting provided a wider perspective to describe these accounts
and complemented the interview method.
Implications of Videoconferencing for Management Learning 381
2.1 Case Study 1: Lecture Delivery
This case study took place at one of the two campuses of a UK

University. We observed and interviewed students taking part in a Micro
Economics module. The students were conventional students in their late
teens and early 20s and most of them were studying on the first year of a
degree course. The lectures were broadcast live from one campus of the
University to the other “remote” campus some 80 kilometers away. The
students from the two campuses had not met each other physically and did
not know each other’s names, although they could see and potentially hear
each other. It is important to note that their classes were all concentrated into
two full days a week, thus these days were very intensive and the students
extremely busy. The videoconferencing room was a fully equipped lecture
theatre with three microphones hanging from the ceiling and two big TV
monitors one next to the other. One had a picture of the students from the
remote campus and the other a picture of the lecturer. There were two
technicians, one at each campus looking after the videoconferencing
connection throughout the lectures.
The Micro Economics module had two-hour videoconference lectures
with a break. As well as attending the lecture, the students met regularly with
the lecturer for one-hour tutorials to work through problems individually or
in groups. There was also a web site facility for the module in which
students could participate in an on-line discussion forum that was available
for questions and answers and, in addition, could access and download the
lecturer notes and the Power Point slides for each lecture. In the class there
were about 25 students at the home campus (i.e. the one from which the
lecturer did the videoconference broadcast) and about 10 at the remote or
receiving campus. During the videoconference sessions the lecturer sat at the
desk with his laptop and the camera, which prohibited him from standing up
throughout the session. He used Power Point slides as a delivery tool for his
lecturing, these were projected on a wide screen and also broadcast to the
remote campus. The majority of the students from both campuses took notes
or jotted things down occasionally. However, some of them took no notes at
all since they were able to download the lecture notes from the web.
Students often complained about technical breakdowns. Connections
with the remote campus were frequently difficult and took time, sometimes
about 30 minutes to sort out. At times failure to broadcast to the remote
campus meant that some of the lectures had to be cancelled or postponed to
another day.
A striking finding from the interviews with students was the number of
times students referred to feelings of boredom to describe either their own
experiences or those of others. Being bored was often linked to the
difficulties with maintaining concentration for a long period of time.

Students found that looking at the projection screen or taking notes copied
out from the TV monitor was hard and tiring. In addition the fact that the
lecturer was limited in his or her movements did not help to hold of the
students’ concentration throughout the whole lecture.
Students at the home campus had the impression that the students at the
remote campus got bored easily and would fall asleep every week.
“It’s quite funny because you can see the boredom or whatever, like
today some of them were half dead.” (Sue, home campus student)
“I think one’s concentration is going to go when you are looking at a

television. Like those two chaps falling asleep I mean they weren’t just
falling asleep they were actually asleep from what I gather when I look
up because I was trying to listen and my colleagues say ‘look they are
asleep’.” (Ben, home campus student)
Observations in situ indicated that the same phenomenon also appeared
to happen with the home campus students. One of the students interviewed
said that he often fell asleep even when the lecturer was in the room, because
he felt sleepy when looking at the PowerPoint screen and staring at the TV
monitor all the time. Another problem posed was with respect to questions,
the students shared a common concern regarding asking the lecturer
questions if they needed to. They said that, in general, students did not ask
questions during the lecture, as the lecturer did not like to be interrupted
when teaching through videoconferencing. Some students regarded the on-
line discussion and the tutorials as the space to ask questions. However,
other students commented on the management of questions during the
lecture:
“Basically we don’t ask a lot of questions really. He says ‘do you all
understand this or do you understand that’ and most people will nod or
shake their head and if the majority is shaking their heads then he will go
through it again. It gives us a chance to understand it and then he’ll do it
again if we don’t, but that’s the only thing. Because the students are in X
(the other/remote campus), he doesn’t look at them and they don’t
respond anyway so he doesn’t actually see if any of them understand it or
not from what I can tell from it anyway.” (Ben, home campus student)
Some students felt fortunate to have the lecturer present in the room as
they could ask questions during the break, whereas the remote campus
students were less likely to ask questions.
“I do feel like they are missing out because I would personally prefer the
teacher to be in the room, just in case you want to stop them and ask a
question and say ‘hang on a sec, I don’t understand’. I think it’s quite
hard for them to do that and I think it’s a bit intimidating having a screen
on your face, because you’re less likely to say, ‘oh hang on what does
this mean?’” (Paula, home campus student)
One student claimed not feeling confident enough to ask questions when
the tutor was not in the room. Because of this and because of the medium,
some students felt that they were not getting enough attention and they felt
sorry for their peers at the remote campus.
2.2 Case Study 2: Tutorial Session
Although this case study is not directly related to management and

business studies, it provides a clear example of a videoconferencing
supported tutorial session, which is an approach not uncommon in
management and business studies. The accounts of the students’ experiences
described here focus specifically on the technology itself and the learning
design and were, therefore, felt to be relevant to other subject areas. The
location of the case study is the Centre for Continuing Education (CCE) of a
University in a geographically isolated part of the UK. All the programs in
the CCE lead to a university certificate, diploma or degree and consist of
courses that are free standing and which can be taken independently. All the
students are mature, with ages varying from 30 to late 60s.
The use of technology was intended to open access to students located in
rural/isolated areas who attended what were frequently very geographically
remote study centers. The videoconferencing system was ISDN 2 and was
centralized and managed from the study centre at the University, which set
up the connection with the other study centers involved. The technician,
based at the University was responsible for starting and ending the
videoconferencing connection on time and was available throughout the
tutorial to ensure the technology functioned properly.
Students were interviewed and two videoconferencing sessions on Cell
Biology and Field Archaeology courses were observed in situ at the
University study centre. Both courses ran twice a month for a full hour and a
half. For both courses the tutors were in study centers that were located away
from the University itself. Thus, in this case the home University students
were the remote participants of the videoconferencing session. The Cell
Biology course had approximately six students with four students in the
University study centre and the tutor on her own at another study centre. The
Field Archaeology course had a larger number of participants.
Approximately 30 students were enrolled with the majority being at two

study centers one of which included the University and the other the study
centre where the tutor was located. It was difficult to determine the actual
number of students attending the tutorials. In the more remote study centers
there was often only one student present. This differed from the University
and other main study centers where, by comparison, a higher number of
students attended. The videoconferencing room at the University consisted
of a rectangular table where students sat facing each other. At one end of the
room there was a TV monitor and there was a microphone system on the
table. Students could see the tutor at all times. They could also see other
students from the other study centers when the microphone was on and a
student spoke, thus activating the monitor. Potentially, and depending
whether the microphone was activated or not, they could see small pictures
of all the study centers including themselves around the main picture, plus
the tutor.
At the beginning of the course the students were given a printed resource
of all the course material. Students brought this folder to each session as the
tutors often referred to the material during the session. Tutors advised the
students to read the material accordingly before each session in order to
prepare for the tutorial and possible questions. The structure of the session
varied from tutor to tutor but frequently involved the tutor presenting the
material followed by questions and answers at the end of the session.
In this case, a key issue related to the dependence on the technology. As
one of the students observed:
“You are relying on the technology to work otherwise you are in the
situation which we were in where, we couldn’t talk to anyone and if we
did switch our microphones on so that we could be heard, everyone just
heard white noise and were deafened by the white noise, so you are
relying on the technology to work.” (Valerie, University/home based
student)
This was a common sentiment expressed by all the students. They had all
experienced technology barriers and breakdowns. The ISDN2 system
displays a short delay between the voice and the image and the picture often
becomes diffuse and jerky. Students often experienced problems with
hearing and seeing properly and regularly made complaints.
In addition, most students experienced not feeling part of a whole group,
but more of a class comprised of different individuals from different study
centers. The Cell Biology students, as there were just a few, seemed to know
their remote class peers better as they had more chances to interact with each
other. Conversely the Field Archaeology students in the University centre
had different kinds of perceptions of their remote peers, some of whom they
could not see or hear at times. They thought of them as being shy,
embarrassed and reluctant to speak because they were on their own. They
thought that they seemed isolated and generally felt sorry for them. A
student described the experience of interaction as lacking “social content”:
“Yes, yes the tutor speaks to them, everyone does sort of say hello to
each other when they appear. It’s hello in X (name of town) and everyone
in X goes ‘hello’ and the other people will say ‘hello X’ type thing but
other than that there was no student conversation um. For example there
was Duncan in Y and I don’t think I talked to him at all during the whole
thing apart from saying ‘hello’ you know so there was no social content
in the time period at all.” (Jackie, University/home based student)
Tutoring styles and approaches to teaching through videoconferencing
were regarded as important to facilitate learning and understanding. Most of
the students’ interviewed had had experiences of working with different
tutors and were very explicit about what they believed makes a good tutor in
this medium. In particular the ability to involve students from the different
study centers was seen as an important skill. These skills were described as,
for example, the ability to encourage students to participate more in the
questions and answers at the end of the session and to make them feel part of
a whole group. Characteristics, such as awareness of the medium and
consciousness of what the students see, avoiding, for example, distracting
poster backgrounds that were not relevant to the subject matter were
considered important. Other characteristics identified as very important skills
to have were a clear voice, a sense of humor, liveliness, good presentation
skills and an ability to use the OHP.
2.3 Case Study 3: Collaborative Project Work Amongst

Distributed Learners
The final case study involved students from Universities in Scotland,

Belgium and France. The students at the Universities were either
technical/engineering students or business/ entrepreneurial strategy students.
The students at the three Universities varied in age but were all either final
year degree students or postgraduate diploma students. Some of the students
were studying full-time and some part-time, something that affected when it
was possible to have the videoconferencing sessions. In this case study
videoconferencing was used to support interdisciplinary-based group
projects. The students worked together in transnational and cross-
disciplinary groups and were each expected to use their respective skills and
knowledge to collaboratively produce a feasibility study for a new product of
their own choosing.
The pedagogical approach adopted in this case study was intended to

encourage discovery learning kinds of experience. Whilst the project groups
used a range of communications technology to communicate with each
other, a key feature was the use of regular PC based videoconferencing
sessions between the students. These took place on either a weekly or
fortnightly basis for a period of 20 to 30 minutes between students based in
the home Scottish University and the remote students based in either the
French or Belgian Universities. The quality of the videoconference
transmission between sites varied according to whether the partner
institution had installed an ISDN link or depended on simple IP supported
connection.
The students in Scotland used multiple microphones whilst at the other
sites it was more normal to use just the one microphone. This led to a
number of differences in the way students worked at the sites. In Belgium,
for example, one student tended to do the talking on behalf of the whole
group although that student would consult regularly with the rest of his or
her group. However, as not all the Belgian students expected to talk directly
to their partner group students in Scotland it was not unusual for these
students to sit in a way that they were not all visible on screen. By contrast,
the Scottish students tended to sit in a way that meant they all appeared on
screen and there was much more likelihood of two or three of them talking
during a session. This said, one student would do most of the talking on
behalf of his or her colleagues. Apparently, on occasions, there had been
problems with distortion due to more than one of the Scottish students
speaking at the same time.
As in the second case study, students commented on the difficulties of
hearing and seeing properly, a problem that was compounded in the opinion
of both the Belgian and French students by the Scottish accents of their
partner students particularly when experienced across a video link.
The most frequently mentioned aspect of their experience, however, was
the problem of not understanding or knowing what each other could or
wanted to do. A Belgian student commented:
“The biggest problem that we have is not knowing what each of us can
do”.
In a similar way the French students explained that they had expected to
do more on design and drawing but had been asked to do market research,
which they claimed was not their field. It seemed from talking to the
different groups of students that the perceptions of what each other could do
were frequently not the same as students’ own perceptions of what they
could or would prefer to do. In addition, the different groups of students
frequently had very distinct views on their counterparts’ performance, which
were very often the result of lack of knowledge about what they were each
working on in order to complete the joint project. There was a tendency, for
example, for the Scottish students to see the Belgian students as failing to
meet deadlines that in practice were more meaningful to the Scottish
students than the Belgian students. There was it seemed often a lack of
understanding about each other’s views of what were the important aspects
to focus on.
The students who only had videoconferences every two weeks said that
this was not enough to resolve these differences in understanding. Equally it
was mentioned that the videoconference sessions were quite short and
particularly at the beginning did not allow enough time for everyone to
speak and to understand each other.
3. SUMMARY OF THE CASE STUDIES
The analysis of these case studies indicates that clearly technological

barriers were a drawback in the students’ perception and acceptance of
videoconferencing. However this did not appear to be the only influence on
the way students experienced using videoconferences. What this study
indicates is that students from case study 1 felt bored in class and in some
cases felt sleepy. They regarded videoconferencing as a “second best”
learning experience and feared that they were missing out in their education.
Students from case study 2 claimed to appreciate tutors who were skilful
enough to involve and engage students from the different study centers and
make them feel part of the class. Though they also had a preference for face-
to-face teaching. However, they felt videoconferencing provided a clear
advantage to the more remote students. The experience of the students
involved in the collaborative group work with other European students in
case study 3 identifies the problem of establishing what each other could or
wanted to do whilst working together as a group. Nonetheless,
communicating and working in this way did offer some clear benefits to
learning.
We need, however, to examine further why videoconferencing is
sometimes perceived as a second best learning experience by students and
ask whether the perceived problems have more to do with the way learning
is designed and managed rather than shortcomings in the media itself. It
appears that in these three case studies students’ experiences and perceptions
of videoconferencing highlighted issues around the nature and significance
of presence (i.e. the value of physical presence, the perception of the other)
and interaction (i.e. the value of interacting with each other and the tutor in
the learning context). Both presence and interaction appeared to be important
critical factors for the students when articulating their experience of

videoconferencing in all three of the case studies, as we discuss further
below.
3.1 The Significance of Presence and Interaction
The students perceived, in some cases, a lack of presence and interaction

and most of them believed that this was due to the technology and the lack of
skills on the part of the tutor. Thus technology and its pedagogic
management were perceived as a barrier to what the students believed to be
effective presence and interaction.
The students often mentioned the significance and value of physical
presence. In case study 1 some students believed that students got bored,
particularly at the remote campus, because they lacked the personal touch
and could not interact with either the lecturer or the students during the class.
“No not at all in general most of them are sleeping, we caught five today
we were watching them on the screen, not at the same time they’ve only
got six in the class. No they don’t seem to pay much attention but then
they don’t get much attention paid to them either so.” (Pat, home campus
student)
These students’ accounts highlighted the issue around the importance of
physical presence. Students believed that people paid more attention if the
tutor was physically present in class. One student described how, when
doing another videoconferencing module where the tutor had been at the
other campus, he had left a class and had not felt guilty about it because “she
(i.e. the tutor) was not present”. Interestingly, however, another student said
that often the tutor did not realize what was happening because he had to
keep an eye on two places at the same time (i.e. both the home and remote
campus) while making sure that the PowerPoint presentation was working
properly. Thus though the lecturer was physically present, he was perceived
by the students as being distant, particularly because they felt they could not
interrupt to ask questions if they needed to.
In case study 2 students also felt that the experience of attending a
videoconferencing tutorial without the tutor in the room could make people
feel isolated and students often drop out from the course. For this case study,
as well as case study 3, the students often mentioned issues around the
perception of the other. Generally they felt sorry for the students studying on
their own in the more remote centers and they talked about them as being
shy, bored or as one student said when referring to them “the wee souls”.
Some students in case study 2 seemed to think that to bring the students
together was the tutor’s responsibility, whereas others said that working with
too many study centers at the same time, inhibited people and did not allow
them to build relationship and belong to a group. Opportunities for
continuing dialogue and lack of peer interaction made it difficult to form a
group with particularly the remote students.
Students from case study 3 who only had videoconferences every two
weeks said that this was not enough to resolve the kinds of differences in
understanding when negotiating what each one could do or wanted to do in
their group project. Equally it was mentioned that the videoconference
sessions were quite short and particularly at the beginning did not allow
enough time for everyone to speak and to understand each other.
In all cases then it seems that whilst physical presence was considered
important, arguably more important was the students’ experience of social
presence and interaction. Garrison (1997), along with a number of other
authors, has addressed the issue and significance of social presence in
computer mediated communication (CMC) and collaborative learning
environments (see also Spears & Lea, 1992; Yates, 1997; Hodgson, 2001).
Garrison’s basic point is that collaborative learning approaches are rooted in
a social constructionist paradigm and that within such a paradigm social
presence and/or social identity is an integral aspect in the social construction
of knowledge and thus in collaborative learning. He uses the term social
presence to describe the degree individuals project themselves through the
medium. From a social constructionist perspective it is possible to argue that
all learning emerges from relational dialogue with and/or through others. In
which case social presence has to be considered significant whatever the
learning situation or medium used.
3.2 Engagement and Engagement Theory
Kearsley and Shneiderman take an essentially constructivist perspective

to develop a framework for technology-based teaching and learning that
focuses on the concept of engagement and on the assumption that students
should be “meaningfully engaged in learning activities through interaction
with others and worthwhile tasks” (1999).
It seemed to us that students’ reference to issues related to physical
presence and interaction was apparently built on the belief that face-to-face
is a more effective way of teaching and learning than videoconferencing.
This could, however, be less to do with the characteristics of the media itself
than with the way the learning events had been designed and/or managed by
the academic staff. It may be that beyond the significance of face-to-face
presence and interaction that the nature of engagement was influential to the
students’ experience as much as the constraints of the media per se. In other
words, it may be that the problem is the lack of attention given in the design
for learning to the issue of engagement, rather than the lack of physical
presence and interaction. Arguably, the quality of interaction and
engagement between tutor and students, together with the processes that
contribute to people feeling socially “present” in the learning context, are
arguably far more important to the learning experience than physical
presence. Here, we are using the term “engagement” to describe a learning
situation where student(s) feel engaged, in tune with the subject matter, in a
way that is meaningful to his/her experience and conception of the world.
Kearsley and Shneiderman (1999) emphasize collaboration among peers
and a community of learners and thus suggest that engagement theory can be
aligned with situated learning theories. They claim that technology can
facilitate engagement in ways that are otherwise difficult to achieve.
Kearsley and Shneiderman are thus using engagement in a slightly different
way to how we describe it above. They emphasize learning designs that
support the active participation of students during a learning event. We
would argue that engagement described only in this way does not recognize
that people can be engaged when passive during a learning event. That is the
student can experience engagement vicariously without always having to be
“actively” participating during the learning event. In a study of learners’
experience of lectures (Hodgson, 1997) it was found that, for example, their
experience of relevance of the content could be either intrinsic or extrinsic
or, alternatively, it could be a vicarious experience. Learners who
experienced the relevance intrinsically or vicariously were more likely to be
engaged with the lecture content and also ultimately more likely to do well
on the course. Vicarious experience of relevance in this study was directly
associated with the lecturer and the stories and metaphors he or she used or
the energy and enthusiasm for the topic/subject that she portrayed or
projected during the lecture.
That the tutor can be equally important for a vicarious experience of
relevance/engagement during videoconferencing sessions was reflected in
the comments made by students in the case studies described here. It
appeared that the tutor was similarly able, during videoconferencing
sessions, to engage the students through the use of story telling, use of
metaphor and visualization, thereby offering a sense of immediacy and
connection with the world.
The case studies discussed would suggest that the nature of engagement
during the learning event is more important than being physically present. It
is arguably this that makes a difference to the learning experience. We would
like to suggest that the notion of engagement is a critical concept when
considering not only videoconferencing but also learning technologies,
networked learning and e-learning more generally. Understanding the nature
of engagement when using learning technologies could challenge the way
some of the technologies are being used and why they are being perceived
negatively or as second best by students.
4. CONCLUSION
We have argued that, in the light of the case studies, the perceived
problems with videoconferencing have more to do with the way the learning
event has been designed and managed than with the shortcomings of the
media itself. Social presence and more specifically lack of engagement are
seen as more important than physical presence and interaction per se. Both
feelings of social presence and engagement appeared to have an impact on
the way the students experienced a learning event supported or mediated by
the use of videoconferencing. The use of videoconferencing might be seen as
limited and arguably of reduced educational value, less because of the
apparently constraining characteristics of the technology itself but more
because of how the design for learning is being managed and attention given
to these two important and inter-related aspects.
ACKNOWLEDGEMENTS
This work was part-funded by a grant from the Committee on Awareness,

Liaison and Training (CALT) of the Joint Information Systems Committee
of the UK Higher Education Funding Councils (JISC). The views expressed
here are not necessarily those of JISC or CALT. Further information about
the project can be obtained at http://csalt.lancs.ac.uk/jisc/ - Accessed
31/07/02. We would like to thank all those students who took part in the
interviews and the lecturers and tutors that gave permission to observe the
learning events.
REFERENCES
Garrison, D.R. (1997). Computer conferencing: The post-industrial age of distance education.
Open Learning, 12 (2), 3-11.
Hodgson, V.E. (1997). Learning from lectures. In F. Marton, N. Entwistle, & D. Hounsell
(Eds.) (2nd ed.), The experience of learning (pp. 159-171). Edinburgh: Scottish Academic
Press.
Hodgson, V.E. (2001). Issues for democracy and social identity in computer mediated
communication and networked learning. In C. Steeples & C. Jones (Eds.) Networked
learning: Perspectives and Issues (pp. 229-243). London: Springer Verlag.
Marton, F. (1994). Phenomenography. In T. Husen & T.N. Postlethwaite (Eds.), The

International Encyclopedia of Education (2nd ed.) (pp. 4424 – 4429). Oxford: Pergamon.
JISC/CALT (2000) Networked Learning in Higher Education 1998-2000 [On-line].
(Accessed 31/07/02). URL: http://www.lancs.ac.uk/users/edres/
research/csalt/networklearn/default.htm)
Kearsley, G. & Shneiderman, B. (1999). Engagement theory: A framework for technology-
based teaching and learning. [On-line]. (Accessed 31/07/02) URL:
http://home.sprynet.com/~gkearsley/engage.htm
Spears & Lea (1992) Social influence in CMC. In M. Lea (Ed.) Contexts of computer-
mediated communication (pp. 30-65). London: Harvester Wheatsheaf.
Yates, S. (1997). Gender, identity and CMC. Journal of Computer Assisted Learning, 13 (4),
281-290.
Bates, A.W. 278, 286, 336, 337, 344
Baxter, A. 361, 372, 378
Beam, H.H. 147, 159
Beatty, J.R. 158, 159
Beaty, L. 256, 271
Beaudin, B. 263, 270
Becker, W.E. 109
Bednar, A.K. 341, 344
Index Beebe, S.A. 318, 329
Benbasat, I.W. 117, 118, 126
Bennett, J. 275, 286
Bereiter, C. 293, 294, 296, 297, 312, 313
Berkowitz, M.W. 335, 345
Bernard, E. 226
Biehler, R.F. 335, 345
Biggs, J.B. 263, 270
Bilimoria, D. 316, 329
Bird, L. 255
Blake, R.L. 224, 227
Block, J.H. 175, 176, 177, 186, 213
Blommaert, J. 226
Abbagnano, N. 336, 341, 344 Bloom, B.S. 25, 28, 40, 277, 286
Ackoff, R.L. 117, 126 Boden, D. 270
Aiken, G. 147, 159 Boekaerts, M. 224, 225
Alavi, M. 65, 76, 77, 89, 90 Borghans, L. 58
Albanese, M.A. 206, 224, 225 Bostrom, R.B. 321, 329
Albelo, M.L. 337, 342, 345 Boud, D. 226, 255, 270, 351, 358
Alderfer, C.P. 329 Bourner, T. 271
Alemán Vargas, M. 333, 344 Boyatiz, R. 147, 159
Alexander, P.A. 211, 225 Bradshaw, D. 51, 58
Alie, R.E. 147, 159 Briggs, R.O. 319, 321, 329
Amin, A. 5, 6, 10, 13, 17 Brown, G. 377
Anderson, J.R. 211, 225 Brown, J.S. 6, 7, 17, 366
Anderson, L. 48, 58 Brown, M. 275, 286
Anderson, P. 163, 171 Brown, S. 277, 286
Anghern, A. 86, 90 Browne, M.N. 109
Antepohl, W. 222, 225 Browne, N. 93, 94, 109
Aranda, E.K. 321, 329 Bruner, J. 78, 90
Armon, C. 335, 344 Buchholz, R.A. 334, 345
Armstrong, E.G. 224, 226 Buckley, M.R. 335, 345
Aronson, E. 270 Bull, J. 366, 377
Arts, A.R. 227 Bulte, J.A. 227
Arts, J.A. 206, 213, 225, 260
Asensio, M. 379 Calvey, D. 3
Camp, M.G. 151, 226
Bandura, A. 316, 329 Campbell, D.T. 210, 225
Banks, M. 3, 13, 17 Carchidi, D.M. 19
Bannister, D. 94, 109 Carey, T.A. 147, 159
Bariff, M.L. 117, 126 Cariaga-Lo, L. 226
393
394 Index
Carr, W. 257, 270 DelBecq, A.L. 329

Carrier, D. 347, 358 Delhoofen, P. 190, 204
Carton, A.S. 313 Dellana, S.A. 119, 126
Chang, A. 109 Dellinger, S. 134, 146
Chaplin, J.P. 316, 329 Dennis, J.F. 330
Charum, J. 287 DeSanctis, G. 319, 329, 330
Checkland, P. 81, 90 Dexter, A.S. 118, 126
Chidambaram, L. 319, 321, 329 Dick, S. 81, 90
Chin, W. 329 Dickson, G.W. 118, 126
Choppin, B.H. 368, 377 Distlehorst, L.H. 222, 225
Chou, D.C. 119, 126 Dochy, F. 205, 206, 207, 208, 211, 212,
Christ, L.F. 129 225, 226, 227
Christ, M.Y. 129, 135, 146, 147, 159 Donnellon, A. 161, 171
Christian-Carter, J. 8, 17 Donnelly, J.H. 114, 126
Christie, B. 330 Doucet, M.D. 222, 225
Churchman, C.W. 117, 126 Dousma, T. 213, 225
Clarke, J. 96, 103, 110 Dowden, B. 345
Claxton, G. 312 Dreyfus, H. 77, 86, 91
Cleland, D. 118, 126 Drucker, P. 78, 79, 87, 90
Clemans, W.V. 360, 361, 362, 364, 378 Duffy, T.M. 96, 109, 344
Cobb, P. 266, 270 Duguid, P. 6, 7, 17
Cohen, A.J. 96, 100, 101, 102, 104, 105,
109 Ebbinhaus, H. 329
Collis, B.A. 204 Elden, M. 147, 159
Confessare, G. 147, 159 Elliot, J. 257, 270
Confessare, S. 147, 159 Elmslie, R.G. 361, 372, 378
Conill, J. 341, 345 Engel, C.E. 205, 226
Conlon, K. 321, 329 Engeström, Y. 289, 296, 297, 307, 312
Connelly, L.B. 359 Entwistle, N. 391
Conole, G. 255, 270 Erskine, J.A. 213, 226
Coombs, S.J. 93, 94, 95, 96, 105, 109
Cooper, H.M. 214, 225 Farrow, S. 118, 126
Coppola, N. 76 Feiner, S. 97, 109
Cortina, A. 341, 345 Feldman, R.S. 335, 345
Couch, P. 147, 160 Fieser, J. 336, 345
Crosby, P.B. 345 Fjermestad, J. 329
Cunningham, D. 344 Flores, F. 77, 86, 87, 91
Flower, L.S. 96, 97, 109
Daft, R.L. 191, 203, 320, 321, 329 Flowers, S.H. 43, 45, 46, 47, 58
Davidson, L.S. 102, 109 Fox, J.S. 361, 372, 377
Davidson, R. 367, 377 Francis, P. 316, 329
Davis, G.B. 117, 126 Franke, G.R. 345
De Block, A 225 Fransman, M. 5, 17
De Corte, E. 211, 212, 226 Freeman, K. 94, 109
De Kock, J.H. 285, 286 Freiberg, J. 149, 159
De Volder, M.L. 185, 186 Freiberg, K. 149, 159
Dearden, J. 117, 126 Friedland, R. 270
Dede, C. 320, 329 Fritzsche, D.J. 335, 346
Deepwell, F. 257, 270 Fröhlich, R. 352, 358
Index 395
Heene, J. 213, 225

Gagné, E.D. 208, 226 Heijke, H. 93, 109
Gallupe, B. 319, 329 Hertz –Lazarowitz, R. 187
Gautshi III, F.H. 345 Herzig, S. 222, 225
Gemino, A. 61, 70, 76 Heywood, J. 361, 378
Gensler, P.J. 119, 126 Hill, C.W. 51, 110, 161, 171
George, J.F. 159, 330 Hill, R.C. 377
Gibson, J.L. 114, 126 Hills, H. 271, 316, 329
Giddens, A. 13, 17 Hiltz, S.R. 319, 329
Gijbels, D. 205, 206, 208, 225, 227 Hoag, J.H. 93, 109
Gijselaers, W.H. 58, 205, 206, 207, 225, Hodges, P. 116, 126
226, 229 Hodgson, V.E. 379, 389, 390, 391
Gill, J. 81, 90 Hogan, D. 316, 329
Ginzberg, M.J. 117, 126 Holdrinet, R.S. 227
Glaser, R. 212, 226 Holsti, O. 257, 270
Goh 93, 109 Holwell, S. 81, 90
Gommer, E.M. 204 Hommes, J. 109, 110
Gopal, A. 320, 329 Hora, N. 321, 329
Graham, S. 129 Hornett, A. 164, 165, 171
Greene, W.H. 367, 368, 374, 377 Horsten, A. 213, 225
Griffiths, W.E. 377 Hounsell, D. 391
Gruber, H. 223, 226 Howard, B. 163, 171
Gujarati, D.N. 367, 377 Hubbard, J.P. 360, 361, 362, 364, 378
Gulliksen, H. 316, 329 Hudson, B. 362, 363, 378
Gumesson, E. 81, 90 Hughes, B. 163, 171
Gumnior, E.C. 102, 109 Hull, C. 316, 329
Gunnarsson, E. 226 Hulshof, M.J. 204
Gutiérrez Sáenz, R. 345 Husen, T. 392
Haas, R.W. 158, 159 Ivancevich, J.M. 114, 126, 334, 345
Hacker, K.L. 77, 90 Ives, B. 76, 77, 90, 117, 126
Haladyna, T.M. 364, 366, 377
Hall, B. 20, 40 Jacobs, N. 161, 171
Hall, R.E. 369, 371, 374, 377 Jacques, D. 184, 186
Halliday M.A. 312 Jarvenpaa, S.L. 77, 89, 90, 330
Halliday, M.A. 289, 292, 312 Johnson, D. 147, 159
Halpern, D. 100, 109 Johnson, P. 81, 90, 315
Hamilton, S. 117, 126, 161 Johnson, R. 147, 159
Hansen, W.L. 100, 101, 103, 109 Johnson, R.A. 115, 126
Harden, M.B. 119, 126 Jonassen, D.H. 96, 109, 344
Hare, P. 187 Jones, C. 391
Harel, I. 78, 90 Jones, T.M. 345
Harri-Augstein, E. 94, 96, 97, 105, 109 Jones, V. 204, 335
Harri-Augstein, S. 94, 110 Judge, G.G. 367, 377
Harris, R. 261, 270
Hartland, S. 53, 58 Kaplan, D.E. 275, 287
Harvey, M.G. 335, 345 Kaplan, J.E. 129, 133, 136, 146
Haythornthwaite, C. 317, 320, 329 Kasper, H. 226
Hearn, G. 316, 329 Kast, F.E. 126
396 Index
Kaufman, D.M. 225 Liedner, D.E. 330

Kazmer, M.M. 329 Lilien, D.M. 374, 377
Kearsley, G. 380, 389, 390, 392 Lindzey, G. 270
Keen, P.G. 126 Linstone, H.A. 319, 330
Keizer, P.K. 93, 109, 110, 226 Lipnack, J. 163, 171
Kelley, H.H. 318, 325, 330 Littlejohn, A.H. 96, 103, 110
Kelly, G. 94, 96, 97, 110 Locker, K.U. 334, 345
Kelly, J. 51, 58 Lorenzi, P. 345
Kemmis, S. 257, 270 Lovatt, A. 17
Kennevan, W.J. 115, 126 Lukasiewicz, J.M. 116, 127
Killingsworth, B.L. 119, 126 Luke, T. 56, 58
King, W. 118, 126 Lumby, A.B. 273
King, W.A. 118, 126 Lütkepohl, H. 377
Kirs, P.J. 119, 126
Kish, C.K. 104, 110 Macintosh, H.G. 362, 378
Klein, M. 275, 276, 287 MacKinnon, J.G. 377
Kling, R. 321, 329, 330 Malik, S. 163, 171, 172
Kling, S. 378 Malinger, M. 147, 159
Knight, P. 277, 286 Mandl, H. 223, 226
Knoll, K. 77, 89, 90 Marina, J.A. 342, 345
Knowles, M. 147, 159 Marland, P. 273, 274, 285, 287
Kolb, D.A. 96, 110 Martínez, E. 341, 345
Krathwohl, D.R. 286 Marton, F. 380, 391, 392
Krawiec, T.S. 316, 329 Martz, Jr., Wm. B. 315, 317, 330
Kwiatkowksi, R. 329 Masia, J.F. 286
Maslow, A. 318, 330
Laagland, E.F. 204 Masterson, J.T. 318, 329
Lamon, M. 294, 297, 313 Matsuba, M.K. 335, 345
Landof, G. 317, 330 Matteson, M.T. 334, 345
Langille, D.B. 225 Matthiessen, C.M. 289, 312
Langrehr, F.W. 147, 159 Mayfield, K. 52, 58
Laurillard, D. 255, 271 McCarthy, D.J. 19
Lave, J. 5, 17, 264, 265, 266, 270, 290, McClelland, D. 318, 330
312 McClung, M.S. 213, 226
Lawson, J. 271 McConnell, D. 255, 263, 271
Lawton, L. 163, 171 McCuddy, M.K. 129, 135, 146, 147, 159,
Lee Looi Chng, V. 93 160
Lee, T-C 377 McEwen, B.C. 93, 94, 110
Leenders, M.R. 213, 226 McFarlan, F. 54, 58
Leidner, D.E. 65, 76, 77, 90 McFarlane, M. 226
Lejk, M. 118, 126 McGill, I. 51, 256, 271
Lemke, J.L. 290, 312 McGilley, K. 313
Lengel, R.H. 320, 321, 329 McKenney, J. 54, 58
Leont’ev, A.N. 296, 297, 312 McKenny, J.L. 118, 126
Levine, J.M. 96, 110, 270 McNiff, J. 257, 271
Lewin, K. 318, 330 Merchant, S. 113
Lewis, K.E. 222, 226 Meyer, J-B 287
Lewis, R. 90 Michaelson, L. 129, 146, 147, 160
Lieberman, S.A. 224, 226 Millar, V.E. 62, 76
Index 397
Miller, J.A. 147, 160 Perry, J.D. 344

Milter, R. 58 Perry, W. 97, 98, 110
Miner, J.B. 318, 330 Petr, J.L. 100, 101, 104, 110
Mintzberg, H. 77, 79, 86, 90 Pettigrew, M. 109, 110
Mitchell, S. 206, 224, 225 Philp, J. 226
Mitroff, I.I. 118, 126 Piaget, J. 78, 90, 266
Molotch, H. 270 Piccoli, G. 65, 76
Monk, D. 45, 58 Pindyck, R.S. 367, 378
Moore, G.A. 52, 53, 54, 58 Pirie, W.L. 129, 135, 146, 147, 159, 160
Moore, R.A. 361, 362, 378 Platt, R. 163, 172
Morgan, D. 98, 258, 271 Poikela, E. 205, 226
Morocco, C.C. 105, 110 Poikela, S. 205, 226
Morrison, R.B. 362, 378 Poole, M.S. 319, 330
Morse, K. 161, 163, 167, 171, 172 Porter, M.E. 62, 76
Postlethwaite, T.N. 392
Nabeth, T. 77, 86, 90 Powell, J. 3, 17
Nanson, E.M. 378 Price, M. 49, 50, 58, 129, 146, 160
Nelson, G.E. 98, 357, 358 Prosser, M. 361, 372, 378
Newble, D.L. 361, 372, 378 Puig Rovira, J.M. 334, 335, 345
Newby, H. 43, 58 Punnett, B.J. 162, 172
Nijhuis, J. 229 Purdy, R.A. 225
Noble, D. 43, 46, 58, 62, 76
Noddings, N. 334, 345 Race, P. 274, 276, 277, 278, 285, 286,
Norman, G.R. 223, 226 287
Nunamaker, Jr., J.F. 321, 329, 330 Raffo, C. 6, 17
Ramaekers, G. 93, 109
O’Connor, J. 17 Rami, A. 76
Ober, P. 226 Ramsey, J.B. 368, 369, 371, 378
Oeseburg, B. 227 Ramsey, V. 147, 160
O'Hara, S. 256, 271 Reason, P. 109
Oliver, M. 270 Reeve, S.D. 43, 45, 46, 47, 58
Oliver, R. 204, 255 Renkl, A. 223, 226
Osborn, A.F. 319, 330 Resnick, L.B. 96, 110, 270, 289, 312
Oxley, L. 378 Rich, M. 90
Richards, B.F. 222, 226
Pagan, A. 367, 378 Ricks, D.A. 162, 172
Page, D. 172 Rieber, R.W. 313
Paivio, A. 70, 76 Robbs, R.S. 222, 225
Palmer, J. 275, 287 Roberts, B. 97, 109
Papert, S. 77, 78, 84, 90, 316, 330 Roberts, C.J. 378
Park, H.J. 335, 345 Robins, J. 329
Parker, D. 61, 70, 76 Rodríguez Lozano, V. 345
Parker, D.C. 69, 76 Rogers, E.M. 52, 53, 58
Paul, R.W. 260, 336, 345 Rogoff, B. 289, 312
Pedler, M. 256, 271 Rosenthal, S.B. 334, 345
Peelm, J.L. 226 Rosenzweig, J.E. 115, 126
Pendlebury, M. 366, 377 Rossner-Merrill, V. 69, 76
Percac, S. 224, 226 Rotter, N. 76
Perdomo, F. 337, 342, 345 Rowan, J. 109
398 Index
Rowntree, D. 277, 287 Spinosa, C. 77, 86, 91

Rubinfeld, D.L. 367, 378 Sprague, R.H. 117, 127, 329
Ruijter, C.T. 185, 187 Stamps, J. 163, 171
Rupp, R. 226 Stanley, J.C. 210, 225
Russell, D. 3 Starr-Hiltz, R. 70, 76
Russell, T. 320, 330 Steeples, C. 391
Stefani, L.A. 96, 103, 110
Saaty, T.L. 319, 330 Stinson, J. 58
Salmon, G. 257, 263, 271 Strasser, S.E. 159
Saunders, P. 110 Strassmann, P. 58
Saville, A.D. 273 Stroup-Beham, C.A. 226
Savin-Baden, M. 263, 271 Sueyoshi, G. 374, 377
Scardamalia, M. 294, 297, 298, 313 Swanson, E.B. 118, 127
Schade-Hoogeveen, B.E. 226 Swanson, G. 4, 17
Scherpbier, A.J. 227 Symes, C. 255, 270
Schmidt, H.G. 185, 186, 223, 226 Syson, A. 257, 270
Schön, D. 77, 79, 89, 90, 91, 176, 187
Schroeder, R. 117, 126, 159 Tamblyn, R.M. 222, 226
Sciglimpaglia, D. 158, 159 Tans, R.W. 222, 226
Scott, D. 316, 329 Taylor, W.C. 378
Scouller, K.M. 361, 372, 378 Teasley, S.D. 96, 110, 270
Segers, M. 205, 209, 211, 212, 225, 226, Tempelaar, D. 212, 213, 226, 227
229 Terlouw, C. 186
Sexty, R.W. 335, 345 Thoma, G.A. 97, 99, 100, 110
Shackelford, J. 99, 110 Thomas, L.F. 94, 96, 97, 105, 109, 110
Sharan, S. 176, 187 Thompson, D. 163, 172
Sheehan, J.K. 104, 110 Thompson, M. 46, 58
Shepherd, M.M. 315 Tirri, K. 335, 345
Shipley, T.E. 318, 330 Todd, M. 76
Shneiderman, B. 389, 390, 392 Todd, R. 205, 227
Short, J. 320, 322, 330 Treadwell, T. 90
Showmaker, S. 329 Troy. J. 109, 110
Silvestri, G.T. 116, 127 Tschudi, S. 100, 110
Singhapakdi, A. 335, 345 Turk, J. 76
Skakun, E.N. 361, 363, 378 Turoff, M. 308, 309, 313, 319, 330
Skinner, S.J. 345
Slabbekoorn, H. 204 Valacich, A.R. 330
Smit, N.J. 175, 185, 187, 189, 190, 192, Van Dam, G.T. 175, 186
204 Van de Kamp, I.N. 204
Smith, I. 94, 95, 96, 109, 337 Van de Ven, A. 319, 329
Smith, K. 147, 159 Van de Wiel, M. 206, 227
Snowman, J. 336, 345 Van den Bossche, P. 205, 206, 207, 208,
Snyder, W.M. 6, 17 222, 224, 225, 227
Solomon, M.Z. 105, 110 Van der Heijden, M.P. 227
Solomon, N. 255, 270 Van der Veen, J.T. 189, 194, 196, 200,
Son, B. 207, 222, 225, 226 204
Speake, T. 3, 17 Van der Vleuten, C.P. 227
Spencer, J. 96, 100, 101, 102, 104, 105, Van Hout, J.F. 186
109 Van Riemsdijk, M. 189, 192, 204
Index 399
Van Sickle, R.L. 207, 222, 225, 226 Zmud, R.W. 118, 127
Van Staden, C. 275, 286
Van Vilsteren, P.P. 227
Van Woerden, W.M. 175, 185, 187
Vasquez Bronfman, S. 77, 90, 91
Verhoeven, B.H. 222, 227
Vermetten, Y.J. 206, 227
Vernon, D.T. 224, 227
Veroff, J. 318, 330
Verwijnen, G.M. 227
Vitell, S.J. 335, 345
Vogel, D.R. 77, 89, 90, 330
Vos, P. 184, 187
Vygotsky, L.S. 78, 91, 256, 264, 266,
271, 290, 291, 292, 295, 307, 313
Walker, L.K. 335, 345

Walkin, L. 277, 287
Walstad, W.B. 110
Walther, J.B. 319, 330
Watson, D. 52, 58
Watts, M. 109
Webb, C.D. 187
Welker, M.G. 129, 133, 136, 146
Wells, G. 289, 291, 293, 312, 313
Wenger, E. 5, 17, 264, 265, 266, 270,
290, 312
Wenger, E.C. 6, 7, 17
Wertsch, J.V. 297, 312, 313
Wheeler, J.V. 316, 329
Wiedersheim-Paul, F. 226
Wiese, D.S. 335, 345
Wignall, D. 90
Willems, J.M. 186, 204
Williams, E. 320, 330, 377
Williams, J.B. 347, 350, 351, 358
Wilson, J. 334, 345
Wise, P. 4, 17
Wolfe, J. 335, 346
Woltjer, G. 205, 206, 226
Wood, R. 363, 364, 368, 378
Woolfolk, A.E. 335, 346
Wyvill, M. 126
Yates, S. 389, 392

Yoo, Y. 77, 89, 90
Zaccaro, D.J. 226

Zani, W.M. 117, 127
Educational Innovation in Economics and Business
1. W.H. Gijselaers, D.T. Tempelaar, P.K. Keizer, J.M. Blommaert, E.M. Bernard and H.
Kasper (eds.): Educational Innovation in Economics and Business Administration.
The Case of Problem Based Learning. 1995 ISBN 0-7923-3272-5
2. D.T. Tempelaar, F. Wiedersheim-Paul and E. Gunnarsson (eds.): Educational Innov-
ation in Economics and Business II. In Search of Quality. 1998
ISBN 0-7923-4901-6
3. R.G. Milter, J.E. Stinson and W.H. Gijselaers (eds.): Educational Innovation in
Economics and Business III. Innovative Practices in Business Education. 1998
ISBN 0-7923-5001-4
4. J. Hommes, P.K. Keizer, M. Pettigrew and J. Troy (eds.): Educational Innovation in
Economics and Business IV. Learning in a Changing Environment. 1999
ISBN 0-7923-5855-4
5. L. Borghans, W.H. Gijselaers, R.G. Milter and J.E. Stinson (eds.): Educational Innov-
ation in Economics and Business V. Business Education for the Changing Workplace.
2000 ISBN 0-7923-6550-X
6. T.A. Johannessen, A. Pedersen and K. Petersen (eds): Educational Innovation in
Economics and Business VI. Teaching Today the Knowledge of Tomorrow. 2002
ISBN 1-4020-0478-8
7. A. Bentzen-Bilkvist, W.H. Gijselaers and R.G. Milter (eds.): Educational Innova-
tion in Economics and Business VII. Educating Knowledge Workers for Corporate
Leadership: Learning into the Future. 2002 ISBN 1-4020-1064-8
8. R. Ottewill, L. Borredon, L. Falque, B. Macfarlane and A. Wall (eds.): Educational
Innovation in Economics and Business VIII. Pedagogy, Technology and Innovation.
2003 ISBN 1-4020-1787-1
KLUWER ACADEMIC PUBLISHERS – DORDRECHT / BOSTON / LONDON

Educational Innovation in Economics and Business Pedagogy, Technology and Innovation

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Educational Innovation in Economics and Business Pedagogy, Technology and Innovation

Uploaded by

Copyright:

Available Formats

Educational Innovation in Economics and Business VIII

Educational Innovation in Economics and Business

SPRINGER-SCIENCE+BUSINESS MEDIA, B.V.

ISBN 978-90-481-6505-6 ISBN 978-94-017-1386-3 (eBook)

Printed on acid-free paper

All Rights Reserved

Part I: The Business Context 1

How E-Learning Businesses Meet Client and End User Needs:

Business Process, Experience and Memory: Educational Approaches

Part II: Challenges for the Academy 41

From High Level Clarity to Ground Level Confusion: Exactly Where

Moving a University Toward On-line Learning: Opportunities,

Linking Pedagogical Innovation and Information Technology to

The Experience of Self-Organized Learning Through the Use of

Part III: Team and Collaborative Learning 111

Using Student Consulting Team Assignments as a Vehicle to Teach a

Using Team Learning in the Classroom: Experiences and Lessons 129

Using Teams in the Classroom: Meeting the Challenge of Evaluating

International Management: Early Experience in Multicultural Virtual

Part IV : Problem Based Learning 173

PRAXIS: A Practice-Based Instructional System in the First Year of

Using the Web for Problem-Based Learning 189

Effects of Problem-Based Learning in Business Education: A

Learning about Teaching Information Systems in a Problem-Based

Part V : Distance and On Line Learning 253

The Use of a Virtual Learning Environment to Support Learners

Distance Learning: The Experience of Accounting at the University

Creating and Improving a “Virtual Object” Through Web-Mediated

Testing Social Information Processing Theories in Distance

PART VI : Specific Applications of Learning Technology 331

Electronic Tutorial for Moral Reasoning in Business Education:

The Five Key Benefits of On-line Final Examinations

Assertion-Reason Assessment in Formative and Summative Tests:

Exploring the Implications of Videoconferencing for Management

Margarita Aleman Vargas, Faculty Member of Bilingual High

Mireia Asensio Department of Management Learning, The School

Mark Banks, Manchester Institute for Popular Culture,

Len Bird, Work-based Learning Unit, Coventry Business School,

David Calvey, Centre for Employment Research, Manchester

Daniel Carchidi, Knowledge Systems GE Capital, Performance

Leroy Christ, College of Business Administration, Valparaiso

Mary Christ, College of Business Administration, Valparaiso

Luke Connelly, Brisbane Graduate School of Business,

Steven Coombs, Department of Curriculum Studies & Secondary

Filip Dochy. Educational Innovation and Information Technology

Stephen Flowers, University of Brighton Business School, UK.

Andrew Gemino, Faculty of Business Administration, Simon

David Gijbels, Educational Innovation and Information

Wim Gijselaers, Department of Educational Development and

Steven Graham, Purdue University North Central, Indiana, USA.

Vivien Hodgson, Department of Management Learning, The

Vivien Lee Looi Chng, Temasek Polytechnic 21 Temasek Avenue 1

Anthony Lumby, Faculty of Management Studies, University of

Wm Benjamin Martz, Information Systems Department, College of

Sylnovie Merchant Department of Management, California State

David McCarthy, Run Fast Inc, USA. djmrunfast@aol.com

Michael McCuddy, College of Business Administration,

Kenneth Morse, Department of Marketing & International

Jan Nijhuis, Department of Management Science, Faculty of

Drew Parker, Faculty of Business Administration, Simon Fraser

Wendy Pirie, College of Business Administration, Valparaiso

Stephen Reeve, University of Brighton Business School, UK.

Maarten van Riemsdijk, Faculty of Technology & Management,

David Russell, Centre for Employment Research, Manchester