This article was downloaded by: [Gazi University]

On: 02 May 2015, At: 10:32

Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered
office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Quality in Higher Education

Publication details, including instructions for authors and
subscription information:
http://www.tandfonline.com/loi/cqhe20

Evaluating Teaching Quality

a a
Jacqueline Douglas & Alex Douglas
a
Liverpool John Moores University , UK
Published online: 16 Aug 2006.

To cite this article: Jacqueline Douglas & Alex Douglas (2006) Evaluating Teaching Quality, Quality
in Higher Education, 12:1, 3-13, DOI: 10.1080/13538320600685024

To link to this article: http://dx.doi.org/10.1080/13538320600685024

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the
“Content”) contained in the publications on our platform. However, Taylor & Francis,
our agents, and our licensors make no representations or warranties whatsoever as to
the accuracy, completeness, or suitability for any purpose of the Content. Any opinions
and views expressed in this publication are the opinions and views of the authors,
and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content
should not be relied upon and should be independently verified with primary sources
of information. Taylor and Francis shall not be liable for any losses, actions, claims,
proceedings, demands, costs, expenses, damages, and other liabilities whatsoever
or howsoever caused arising directly or indirectly in connection with, in relation to or
arising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Any
substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,
systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Conditions of access and use can be found at http://www.tandfonline.com/page/terms-
and-conditions
 Quality in Higher Education, Vol. 12, No. 1, April 2006

Evaluating Teaching Quality

JACQUELINE DOUGLAS* & ALEX DOUGLAS
Liverpool John Moores University, UK
J.A.Douglas@ljmu.ac.uk
Quality
10.1080/13538320600685024
TEDP_A_168472.sgm
1353-8322
Original
Taylor
102006
12
Mrs
00000April
JacquelineDouglas
and
&inArticle
Francis
Higher
(print)/1470-1081
Francis
2006Education
Ltd (online)

ABSTRACT A comparison is made of student feedback questionnaires, peer review and the potential
use of mystery students as a means of evaluating the quality of teaching and learning and providing
a vehicle for continuous improvement. The available literature and primary data obtained via semi-
structured interviews conducted with staff within a UK business school, as well as secondary
Downloaded by [Gazi University] at 10:32 02 May 2015

analysis of the results of an interview exercise with the same staff concerning student feedback ques-
tionnaires, were used in the analysis. The findings show that staff have very little faith in student
feedback questionnaires, whether module or institutional, participate (sometimes reluctantly) in the
school’s peer review scheme and are divided on the potential use of mystery students.

Introduction
Business schools face intense competition to attract the best home-based undergraduates
and graduates on to their programmes of study. This competition has now extended to
attracting international students paying premium rates. To differentiate themselves from
their competitors, engender loyalty in their students and to prove that they are offering a
high quality educational experience many business schools have turned to some form of
external accreditation.
In the UK and Europe the Association of Master of Business Administration (MBAs) has
accredited higher education institutions’ MBA programmes. EQUIS, the quality assurance
scheme run by the European Foundation for Management Development (EFMD), has
accredited 50 higher education institutions in management and business administration
world wide. In the USA the Association to Advance Collegiate Schools of Business (AACSB)
has accredited nearly 400 business schools that have satisfied their standards. Some
business schools in the UK, Europe, South America, Australia and Asia are turning to the
AACSB in the search for further external approval of their business and management
programmes. Indeed, 25 business schools world wide have achieved all three accreditations
(London School of Business and Finance web site, www.lsbf.org.uk/), including RSM
Erasmus University in The Netherlands, Grenoble Graduate Business School in France,
Brisbane Graduate School of Business in Australia and Aston Business School in the UK.
To gain such approval, business schools have to provide evidence that they provide high
quality teaching and learning and that it is continually improving. They must demonstrate
how the teaching delivered by their staff leads to learning by their students. This issue is
one that higher education providers have already had to address to satisfy their own
government controlled quality assurance agencies in order to receive government funding

*Corresponding author. Faculty of Business and Law, Liverpool John Moores University, 98 Mount Pleasant,
Liverpool L3 5UZ, UK. Email: j.a.douglas@ljmu.ac.uk

ISSN 1353-8322 print; 1470-1081 online/06/010003-11 © 2006 Taylor & Francis

DOI: 10.1080/13538320600685024
 4 J. Douglas & A. Douglas

to provide their educational services. However, for many business schools world wide it is
an issue that they are tackling for the first time when going for AACSB accreditation.
This paper aims to evaluate and compare student feedback, peer review and mystery
students as means of appraising teaching performance such that the management of a
business school will have evidence to support their contention that they deliver high quality
teaching and learning and are continually improving and, hence, satisfy the criteria of
external accreditation agencies. This evaluation draws on an institutional case study of
‘Riverbank Business School’.
Riverbank Business School is part of a post-1992 university located in the UK and has
approximately 4000 full-time equivalent students and 200 full-time equivalent teaching
staff, supported by 50 administrative staff and 12 technical staff. The business school has
been conducting student feedback surveys for many years. A peer review scheme has
operated since the mid 1990s, with several different approaches being adopted by each
department (management, finance, marketing and human resources management). Most
observations appear to have taken place in relation to the observation of mainstream teach-
Downloaded by [Gazi University] at 10:32 02 May 2015

ing and learning activities, for example group teaching, lectures and practical sessions.
There are other learning activities undertaken at Riverbank, such as distance learning,
online learning, project work and problem-based learning. However, at the time of writing,
peer review guidelines focused on the mainstream teaching activities mentioned above.

Methodology
Yin (2003, p. 23) described a case study as: ‘An empirical inquiry that investigates a contem-
porary phenomenon within its real life context, especially when the boundaries between
phenomenon and context are not clearly evident, and in which multiple sources of evidence
are used’. Primary data was gathered via a series of 40 semi-structured interviews with vari-
ous levels of teaching staff and management within the case study business school. The
interviewees consisted of a sample of programme managers, module leaders, a director of
teaching and learning and teaching and learning coordinators. The aim of the interviews
was to determine:

• what the staff and managers perceived was the purpose of peer review;
• what role, if any, they had played in the process;
• whether or not they had undergone any training beforehand;
• whether or not they felt it enhanced their own teaching;
• their views on student feedback questionnaires
• how they would react to the concept of mystery students;
• whether or not they could recommend an alternative method.

Data capture was via note taking. Transcripts were then produced and analysed. The
primary data was supported by secondary data from an internal consultation exercise on
student feedback questionnaires conducted by the quality assurance unit of the business
school’s host university. A desktop analysis of this published report was conducted.

Higher Education and Consumerism

Potter (1988, p. 149) defined consumerism as: ‘Attempts to redress the imbalance of power
that exists between those who produce goods and services and those for whom they are
 Evaluating Teaching Quality 5

provided’. The concept is based on the belief that by increasing the power of consumers the
quality of services will increase. Quality is equated with the ‘voice’ and ‘choice’ of the
customers (Pollitt, 1991). Bennington and Taylor (1992) argued that the various citizens’
charters are concerned not with the rights and concept of citizenship, but with consumer
protection.
Consumerism emphasizes five principles: access; choice; information; redress; represen-
tation (Potter, 1988; Sanderson, 1992). These are: first, that people should have access to the
benefits of a particular product or service; second, that people should have as wide a choice
as possible; third, that people should have as much access to information about the product
or service as possible; fourth, that people should have a means of complaining about the
product or service if things go wrong and receive some form of compensation or redress;
fifth, people should have the right of representation of their views to those making deci-
sions that affect them.
These consumerist principles were first developed for the private sector, but were soon to
be adopted in the public sector. Indeed, part of the Conservative Government strategy in
Downloaded by [Gazi University] at 10:32 02 May 2015

the early 1990s was the empowerment of the public, who were to view themselves as
customers so that public management could be based on private sector models (Ranson &
Stewart, 1994).
It was around that time that the concept of the ‘student as customer’ was first mooted in
the UK by Crawford (1991). Fifteen years on, the language of higher education reflects the
commercial sector in its use, by some academics and managers, of terms such as ‘custom-
ers’, ‘competitors’ and ‘markets’. The concept of the student as customer has been given
further credence by the Higher Education Funding Council for England (HEFCE), which
has introduced a national student survey aimed at final year students. The survey seeks
views on several aspects associated with the teaching, learning and assessment provided by
their university and its courses. More recently, due to concerns about possible legal action
by students accusing the university of failing to educate them properly, Oxford University
has proposed requiring new students to sign contracts obliging them to attend classes and
complete assignments. Such contracts are aimed at ‘a new breed of fee-paying “student
consumers” who are demanding ever higher standards from universities’ (Boone, 2006). To
satisfy the needs of their customers (students) universities will have to provide quality
services with regards to teaching, learning and assessment. Indeed, students are suggesting
that universities could use the extra funds generated through top-up fees to ensure high
quality teaching (Nicholds, 2006).
Teaching provided in higher education is a service that is produced, delivered and
consumed simultaneously. One of the aspects of services that distinguish them from a prod-
uct include, among others, perishibility, i.e. an inability to be stored (Dale, 2003). It is this
aspect that makes it important to capture and measure a service at its point of delivery,
rather than after the event, to ensure reliability of data. To ensure quality teaching, most
higher education institutions will have to have in place systems to monitor and measure
teaching performance. It is this that is discussed in the next section.

Monitoring Teaching Quality

Harvey and Green (1993) stated that there are a variety of stakeholders within higher educa-
tion, including government, professional bodies, employers, parents, students, staff, both
teaching and non-teaching, and auditors. Tam (2001) found that each stakeholder held their
own view of what quality in education means to them. For example, those steeped in the
 6 J. Douglas & A. Douglas

world of academia may perceive quality education as the production of a constant stream of
highly intelligent graduates able to advance and spread their knowledge. An employer, on
the other hand, may expect the product of a quality education to be a graduate who can ‘hit
the ground running’, capable of adapting and adopting new skills to cope with any changes
within the workplace. It would seem essential that ‘equal expression of the legitimate
voices’ should be facilitated in the evaluation of quality, even if they are not saying the same
thing. Therefore, a triangulation of information must be sought, both to monitor and
manage the quality of education. Given that the direct receivers of the delivery of the teach-
ing service are students, their experience and its improvement should be at the forefront of
any monitoring of higher education quality. It is therefore proposed that use of the three
main ways to monitor service quality, namely feedback surveys, inspection (via peer obser-
vation) and mystery customers (students), could be used within higher education.
However, only two methods are currently used, namely inspection in its various guises and
student feedback surveys. All the methods, both used and proposed, will be examined in
the following sections.
Downloaded by [Gazi University] at 10:32 02 May 2015

Student Feedback Surveys and Focus Groups

Student feedback surveys and focus groups emphasize the measurement of outcomes.
This can best be illustrated by some examples. Typical questions include rating agreement
on a Likert scale to statements such as ‘the teaching accommodation was suitable’ and
‘overall, formal lectures were satisfactory’. Both measure outcomes. A mystery customer
checklist would measure such things as the lighting, the temperature, seating comfort,
decoration, layout of chairs and tables, cleanliness (based on amount of litter) of the
accommodation and the ability of the lecturer to be heard from the back of the room, the
clarity of the overhead slides (if used), the length of the lecture and the lecturer’s voice
modulations. All of these are associated with the delivery process. Academic life involves
courses, subject modules and a complete range of student activities across the whole insti-
tution. Students’ opinions are sought on all of these elements. Module questionnaires are
usually fairly short, with questions on various aspects of a particular subject module.
Conversely, institution wide surveys tend to cover all aspects of student life. It is gener-
ally detailed, lengthy and time consuming for students to complete. When administered
electronically such surveys have a variable response rate. Hamilton (2003) found that
online surveys for industry and government, mainly in the USA and Canada, had an
average response rate of 32.5%, with most surveys receiving at least a 26% response rate.
An online institutione-wide student survey conducted by Riverbank University in 2004
received a response rate of 17%. However, Geall (2000), in an online survey of first year
students at the City University of Hong Kong, received a response rate of 52.2% after the
survey had been online for 12 days. This was increased to 67.5% following the sending of
Email and mail. The UK National Student Survey, conducted on behalf of HEFCE,
received an overall response rate of just over 60%, although there were large variations
between institutions, but only after intensive and expensive follow-ups by telephone (LTS
News, 2005).
When designing the research instrument the approach usually follows the same format,
with focus groups of staff and students used to identify which subjects are covered in the
questionnaire, followed by opinions on both satisfaction and importance being sought. The
resultant information is intended to provide local and central management with informa-
tion on which to act (Harvey, 1995).
 Evaluating Teaching Quality 7

The literature reports a number of reasons for staff not supporting the use of student
feedback questionnaires, including problems with students’ competencies in filling in such
questionnaires (King et al., 1999), doubts over students’ abilities to evaluate classes and the
influence of a number of variables, such as course, tutor and student characteristics
(Rowley, 2003).
A recent consultation exercise by Riverbank Business School’s host university found little
support from staff for module feedback questionnaires in particular. The arguments for this
lack of support included: many of the questions were subjective and therefore responses
would be influenced by personal prejudice; given that the purpose of the feedback
questionnaire was the measurement of student satisfaction with any given module many of
the questions were not relevant to the delivery of the module and also covered items
outside the control of the module leader, for example, textbook availability in the library;
the questionnaires were open to abuse by respondents, who would deliberately give low
ratings for all questions as an act of ‘revenge’ for, for example, low assignment or examina-
tion marks; finally, the validity and reliability of such surveys was questioned. As an exam-
Downloaded by [Gazi University] at 10:32 02 May 2015

ple, in many cases where modules contained no coursework element in their assessment,
when asked whether or not coursework had been returned in reasonable time a large
proportion of students would offer answers covering the full range of responses on a 1–5
Likert scale, thus throwing open to question their ability to offer useful feedback.
Many of these arguments may be indicative of poor questionnaire design or indifferent
respondents, but this did not deter the business school management from using the results
for performance appraisal purposes.

Inspection or Peer Review

Inspection entails one individual monitoring the behaviour of another by watching and
listening to how he/she performs, i.e. performance appraisal by inspection. Peer review in
higher education falls within this category.
The process of peer review has undergone several iterations to make it sound less threat-
ening and has been variously described in the extant literature as ‘peer review’ (Cox &
Ingleby, 1997; Madden, 2000), ‘peer evaluation’ (Greguras et al., 2001) and ‘peer support
review’ (Bingham & Ottewill, 2001), the latter used by one higher education institution as a
means of stressing that the process was one of ‘mutual assistance’.
Cole (2003) viewed peer review as an essential process for reviewing ideas and ‘catching
mistakes’ and for it to be successful there needs to be a ‘culture of criticism’, particularly if a
quality improvement culture is to thrive.
Roberts (2002) argued that if an appraisal is to be used by management as part of an eval-
uation of an individual’s career progression then peer review was essential to ensure the
process was an ethical evaluation of performance.
Pagani (2002) described peer review as a tool for change. It can be used to assess perfor-
mance to help individuals improve their performance, as well as to ensure that standards
are being met. This also allows for the identification of best practice, which can be shared
with others.
Peer review in the USA has traditionally been used to provide feedback to teaching staff
on the strengths and weaknesses of their own teaching so that they can make improvements
(Osburne & Purkey, 1995). It is also used, together with student evaluations of teaching, by
management when deciding about such issues as promotion and tenure (Rowley, 2003).
Indeed, Bingham and Ottewill (2001) argued that professional judgement of teaching staff
 8 J. Douglas & A. Douglas

should be used to supplement the views of other stakeholders (for example, students and
managers) to obtain a ‘balanced picture’ of the evaluation of programmes of study. This
concurs with Ramsden’s (2000) belief that over-reliance on statistics, from student evalua-
tions of teaching, to improve teaching practice would be a mistake. The peer review process
could enhance the whole evaluation process.
The Collins English Dictionary (HarperCollins, 2000) defines ‘peer’ as ‘a person who is an
equal in social standing, rank, age, etc.’, thus a manager or an external expert would not be
included within the scope of this definition. Furthermore, this dictionary defines ‘review’ as
‘to inspect, especially formally or officially’. It also defines ‘observation’ as ‘the act of
observing or the state of being observed’.
Therefore, for the purposes of this paper, the process will be referred to as ‘peer review’
and will be defined as the ‘evaluation of teaching performance by peer observation’.
Peer review differs from the student feedback surveys discussed earlier in that it is not,
strictly speaking, a method of measuring quality performance. It has more to do with qual-
ity improvement, in that its primary aim is to bring about changes in teaching practice,
Downloaded by [Gazi University] at 10:32 02 May 2015

where required. This may ultimately lead to measurable improvements in student satisfac-
tion, but this is not its primary function and it is this that differentiates it from mystery
customers/students and student feedback surveys, which are both direct measures of
quality performance, whether the performance of teaching staff, ancillary service staff, the
university facilities or the university systems as a whole. The link between performance and
satisfaction has long been established in the extant quality and satisfaction literature
(Cronin & Taylor, 1992; Spreng & MacKoy, 1996; Dion et al., 1998; Lee et al., 2000; Hoising-
ton & Naumann, 2003).

Advantages and Disadvantages of Peer Review

The purpose of appraisal differs across the higher education sector; it is either used as a
management tool for target and objective setting (Shelley, 1999) or a self-evaluation tool for
the individual. Piggot-Irvine (2003) advocated that appraisal per se should be an educative
process, one that is based on trust and openness between both appraiser and appraisee.
Therefore, correct training is essential for success. If openness and trust are not ‘factored in’,
then problem solving cannot take place.
There are a number of disadvantages identified in the literature. For example, Cox and
Ingleby (1997) found that peer review via observation of teaching can be a perpetuation of
conformity of teaching, the behaviour of the observed is affected by the observer and no one
else is qualified to comment on someone else’s teaching. Bingham and Ottewill (2001)
recognized that the assessment of peers might be too self-congratulatory (a mutual back-
slapping exercise) and, therefore, inaccurate. There needs to be checks and balances in place
to prevent this from occurring. Other weaknesses identified in the literature include the fear
of being critical, and yet, according to Cole (2003), a culture of criticism is essential for a
successful peer appraisal scheme. Cox and Ingleby (1997) argued that peer reviewers do not
have to be from the same subject disciplines and that good practice can be shared amongst
the various subject specialists.
Those members of the teaching staff who had undertaken a professional teaching qualifi-
cation were more open to sharing best practice according to Chadwick (1995). Prior to the
introduction of such professional qualifications in 1992 it was expected that effective
teaching came from practising in the classroom (Elton & Partington, 1991), learning by
doing or on the job training.
 Evaluating Teaching Quality 9

Enjoying or Enduring the Process?

All 40 staff and managers interviewed in Riverbank Business School had experienced the
peer review scheme, as both observer and observed. In all cases members of their own
subject area made the observations.
Of the academic staff that had been in higher education for over 10 years (n = 14), the
majority responded that they only ‘endured the process’, with one person admitting that
‘you do act differently when you know you are being observed and feel that the class
doesn’t go as well’, a form of Hawthorne effect. Another interviewee indicated that she did
not like being watched and was already very self-critical ansd so tended to only focus on the
less positive feedback. However, she did say that if it was something that was carried out
more often then she would get used to it. A common phrase used by interviewees was that
it did not feel ‘natural’ when the observations took place and, therefore, did not reflect
reality (only 4 of the 14 long serving respondents enjoyed the experience). This finding
conflicts with the view of Murphy and Cleveland (1995), that peers can observe one another
in ‘a real setting’.
Downloaded by [Gazi University] at 10:32 02 May 2015

The majority of staff who had worked in higher education for 3–10 years (n = 20) also
endured the experience and made particular reference to the amount of paperwork attached
to the process, which was seen as a negative aspect. Some felt that it could be a useful
process and some enjoyed the opportunity to reflect on their own teaching.
The newer staff (1–3 years in higher education) (n = 6) seemed to enjoy the process more,
particularly as they felt they were still on a ‘learning curve’ and so developing their own
teaching style.

Does the Process Lead to Any Improvement?

Only two interviewees felt that the process had lead to improvement: one said that this
improvement was the development of a shared understanding, the other had a general
value of reflection in that it allowed her to reflect on her own practice and what worked and
what did not work in the teaching arena. Both of these respondents were involved in a
coordinating teaching and learning role. Other interviewees commented that people gener-
ally did not like to provide negative criticism as they did not want to undermine colleagues’
confidence. This reflects the view of Ramsden (2000), who stated that better teaching did not
necessarily come from appraisal.

What is the Purpose of Peer Review Within Teaching?

Respondents identified three main purposes for peer review. First, peer review provides
feedback on teaching performance (identified by 6 respondents). Peer review allowed staff
to reflect on their own standards of teaching and have feedback; it was a form of reassur-
ance that staff were doing a reasonable job and could identify areas for improvement. It also
gave staff confidence to know that what they were doing was no worse than anyone else.
Second, peer review aids continuous improvement (n = 28). Peer review was deemed
useful when observing others, where there was an opportunity to be exposed to different
methods and techniques. It was a mechanism to share good practice and the respondents
thought that they learnt quite a lot from observing, although not much from being
observed, as their peers were too kind. It provided an opportunity to observe other people
and get good ideas from them. Peer review was seen as a vehicle for integrating part-time
 10 J. Douglas & A. Douglas

staff into the school by pairing them with a full-time member of staff. Peer review offered
the opportunity for identifying areas for improvement, it was seen as an opportunity to
help each other and it offered a different view on how you could improve. It was also
viewed as part of a continuous professional development process.
Another purpose of peer review was compliance (n = 8). Peer review was perceived to be
a paper exercise providing evidence on compliance with external agency standards, a
means of keeping staff ‘on their toes’ and providing a paper trail for any audits.

Are There Any Alternatives?

Approximately half the interviewees suggested alternative methods of evaluation, such as
more use of team teaching, although it was recognized that this would be a costly process.
Another suggestion made by several people was the use of VCRs to video best practice.
This seems to make sense, as a lot of people indicated that they had benefited more from
observing others than from being observed. Another thought that the method itself was
Downloaded by [Gazi University] at 10:32 02 May 2015

not in question, but rather that what was done with the information afterwards should
change; more supportive critical feedback was needed, in line with the view of Cole
(2003), who argued that a culture of criticism needed to be fostered, while at the moment
it was more like a culture of mutual admiration or, possibly, fear (of reprisal). The use of a
mentor for new teaching staff was mentioned by a few interviewees, although this would
not necessarily require class observations, but could involve discussions about one’s own
practice.
The majority of staff were happy for the peer review process to continue, although some
found the experience fraught with anxiety, given that it was their own peers that were pass-
ing judgement. Many agreed that a non-subject specialist observing them might be less
intimidating, as they would not be able to focus on content. Most interviewees admitted
that whilst they could not recall any particular improvements that came out of the last
round of observations, which in some cases was over 18 months ago, they did concede that
observing another’s practice gave them ideas on how to improve or incorporate different
elements into their own practice.

Mystery Customers
Mystery customers have been used by organizations since the 1970s (Erstad, 1998) to
measure and manage service quality via a form of participant observation. They are
currently used in hospitals (mystery patients), in hotels (mystery guests), on trains and
airlines (mystery passengers), in retail outlets (mystery shoppers) and in bars and restau-
rants (mystery drinkers and eaters). The researcher poses as a customer, or potential
customer, (Wilson, 1998) and rates the service encounter using a pre-prepared evaluation
sheet to record performance against pre-set standards, for example were you greeted upon
arrival, was the service person courteous, was she/he wearing a name badge, was she/he
wearing the correct uniform, was the wait in the queue brief? Typically mystery customers
measure the process rather than the outcome (Erstad, 1998), although more recently some
organizations have also used them to judge the outcomes of the service encounter. For
example, in fast food restaurants the mystery customer now rates the product received by
the customer, for example the pizza or burger purchased.
There are both advantages and disadvantages to using mystery customers. One advan-
tage is that mystery customers emphasize the evaluation of service delivery as it unfolds,
 Evaluating Teaching Quality 11

rather than outcomes (Wilson, 1998). However, this can also be viewed as a disadvantage.
The service consists of both a ‘functional’ component (the process or the ‘how’) and a ‘tech-
nical’ component (the outcome or the ‘what’) (Gronroos, 1984). Failure to evaluate both
components may lead to situations where the delivery process was a success but the
customer was dissatisfied with the outcome. For example, the chef followed the recipe to
the letter but the dish was inedible. Mystery customer evaluation allows an objective view
to be registered by an independent assessor at the time and place the evaluation takes place,
thus overcoming one of the main weaknesses of customer surveys, that they take place
hours, days or even weeks after the service was experienced and customers typically only
remember their overall impressions of the service rather than the details (Wilson, 1998).
Mystery customers collect facts, not beliefs or opinions. For example, a customer may
believe that they waited for a long time in a queue without actually timing their wait and
their opinion of reality may vary from the actuality. A mystery customer will always
measure the length of a wait and compare this to the organization’s service standard for that
service component. The use of mystery customers may have a positive impact on employees
Downloaded by [Gazi University] at 10:32 02 May 2015

performance, a form of Hawthorne effect (Erstad, 1998). Finally, provided management is

open to criticism, the mystery customer evaluation can aid decision-making by, for exam-
ple, identifying training needs or areas for more resources to be allocated.
The disadvantages of using mystery customers include: employees can view their use as
threatening, for example company spies, particularly where negative evaluations are linked
to disciplinary actions rather than to development activities, such as training; the novelty of
being ‘mystery shopped’ will wear off and staff apathy may set in, thereby minimizing the
advantages; the use of mystery customers is a costly exercise to undertake if an organization
fails to gain any benefits from implementing such a programme of evaluation.
For higher education institutions, the question remains to be seen if they could success-
fully utilise the concept of mystery students to evaluate what is happening in the class-
room. A number of issues including recruitment and the lack of service delivery standards
would have to be addressed before they could be utilised successfully (Douglas &
Douglas, 2006).

Mystery Students and Academic Freedom

Questions concerning the concept of using mystery students were the most emotive and
thought provoking for some interviewees, with one person expressing shock and disgust
that it could even be considered, with another saying that it would be an attack on academic
freedom. However, other teaching staff, and most new teachers, were either apathetic to the
concept of using mystery students or were happy for them to be used.
One of the disadvantages of using such a scheme is that it could be a costly process with
no recognized benefits. This matched comments from the interviews with staff, that the peer
review scheme was adequate to evaluate those teaching staff that ‘did a reasonable job’ but
was inadequate in dealing with those whose performance was poor. It was generally felt
that these people were never ‘dealt with’ appropriately.
During the interviews one member of staff, when asked her opinion of the concept of a
mystery student was horrified and explained that when peer observation of teaching was
first mooted staff thought that it was a way of measuring performance by management and
that it would in some way lead to penalties. She felt that mystery students would be threat-
ening and remove academic freedom and control from the staff member, something she
thought had already been eroded in so many other ways. Conversely, other members of
 12 J. Douglas & A. Douglas

staff felt in no way threatened by the concept of mystery students and thought it a good
idea, providing the standards against which they would be judged were clearly defined and
communicated.

Conclusion
Universities and business schools world wide are having to provide evidence of the quality
of student teaching and the learning experience to satisfy the requirements of funding agen-
cies, governments, accreditation bodies and more demanding students. The usual methods
of doing this are through feedback questionnaires or peer observation of teaching, both of
which offer advantages and disadvantages with regard to ownership by teaching staff and
the reliability and validity of the methodologies. The concept of the ‘mystery student’ offers
an alternative method of information gathering that negates some of the disadvantages of
the other two methods. However, there are issues regarding the practicalities of using this
method (Douglas & Douglas, 2006).
In the interests of validity and reliability this paper, concurring with Bingham and
Downloaded by [Gazi University] at 10:32 02 May 2015

Ottewill (2001) and Ramsden (2000), suggests that a triangulation of methods may be the
best way forward if robust information on teaching and learning is to be gathered. The
paper further suggests that these methods should be feedback questionnaires, peer observa-
tion and mystery student reports. The key to this happening will be the creation of a culture
of criticism in which staff are aware that any data collected is for the purpose of improving
the quality of teaching and learning and not about the assessment of teachers.

References
BENINGTON, J. & TAYLOR, M., 1992, ‘The renewal of quality in the political process’, in SANDERSON, I. (Ed.)
(1992) Management of Quality in Local Government (Harlow, UK, Longman), pp. 164–86.
BINGHAM, R. & OTTEWILL, R., 2001, ‘Whatever happened to peer review? Revitalising the contribution of
tutors to course evaluation’, Quality Assurance in Education, 9(1), pp. 22–39.
BOONE, J., 2006, ‘Oxford draws up student study contracts’, The Financial Times, 1 February, http://
www.FT.com, accessed 31/12/06.
CHADWICK, P., 1995, ‘Academic quality in TQM: issues in teaching and learning’, Quality Assurance in Educa-
tion, 3(2), pp. 19–23.
COLE, R.E., 2003, ‘New organizational designs for sustainable quality improvement’, keynote address
presented at the 6th International Conference on Quality Management and Organisational Development
(QMOD), Paris, France, 2–3 October.
COX, B. & INGLEBY, A., 1997, Practical Pointers for Quality Assessment, (London, Kogan Page)
CRAWFORD, F., 1991, ‘Total Quality Management’, Committee of Vice-Chancellors and Principals Occasional
Paper, London, December. Cited in Hill, F. M. (1995) ‘Managing service quality in higher education; the
role of the studen as primary consumer’, Quality Assurance in Education, 3(3), pp. 10–21.
CRONIN, J.J., JR & TAYLOR, S.A., 1992, ‘Measuring service quality: a re-examination and extension’, Journal of
Marketing, 56(July), pp. 55–68.
DALE, B.G., 2003, Managing Quality, 4th edn (Oxford, UK, Blackwell).
DION, P.A., JAVALGI, R. & DILORENZO-AISS, J., 1998, ‘An empirical essessment of the Zeithaml, Berry and
Parasuraman Service Expectations Model’, The Service Industries Journal, 18(4), pp. 66–86.
DOUGLAS, A. & DOUGLAS, J., 2006, ‘Campus spies? Using mystery students to evaluate university perfor-
mance’, Educational Research, 48(1), pp. 117–25.
ELTON, L. & PARTINGTON, P., 1991, Teaching Standards and Excellence in Higher Education: developing a culture for
quality (London, Commitee of Vice-Chancellors and Principals).
ERSTAD, M., 1998, ‘Mystery shopping programmes and human resource management’, International Journal of
Contemporary Hospitality Management, 10(1), pp. 34–8.
GEALL, V., 2000, ‘The expectations and experience of first-year students at City University of Hong Kong’,
Quality in Higher Education, 6(1), pp. 77–89.
 Evaluating Teaching Quality 13

GREGURAS, G.J., ROBIE, C. & BORN, M.P., 2001, ‘Applying the social relations model to self and peer evalua-
tions’, Journal of Management Development, 20(6), pp. 508–25.
GRONROOS, C., 1984, ‘A service quality model and its marketing implications’, European Journal of Marketing,
18(4), pp. 36–44.
HAMILTON, M.B., 2003, Online Survey Response Rates and Times: background and guidance for industry
(Cambridge, MA, Tercent Inc.), http://www.supersurvey.com/whitepapers.htm, accessed 31/12/05.
HARPERCOLLINS, 2000, Collins English Dictionary, 5th edn (Glasgow, HarperCollins).
HARVEY, L., 1995, ‘Student satisfaction’, The New Review of Academic Librarianship, 1, pp. 161–73.
HARVEY, L. & GREEN, D., 1993, ‘Defining quality’, Assessment and Evaluation in Higher Education, 18(1), pp. 9–34.
HOISINGTON, S. & NAUMANN, E., 2003, ‘The loyalty elephant’, Quality Progress, 36(2), pp. 33–41.
KING, M., MORISON, I., REED, G. & STACHOW, G., 1999, ‘Student feedback systems in the business school: a
departmental model’, Quality Assurance in Education, 7(2), pp. 90–100.
LEE, H., LEE, Y. & YOO, D., 2000, ‘The determinants of perceived service quality and its relationship with
satisfaction’, Journal of Services Marketing, 14(3), pp. 217–31.
Lts NEWS, 2005, ‘Teaching quality information and the National Student Survey’, LTS News, 1(October), p. 2.
MADDEN, A.D., 2000, ‘When did peer review become anonymous?’, Aslib Proceedings, 52(8), pp. 273–76.
MURPHY, K.R. & CLEVELAND, J.N., 1995, Understanding Performance Appraisal: social organizational and goal-
based perspectives (Thousand Oaks, CA, Sage).
Downloaded by [Gazi University] at 10:32 02 May 2015

NICHOLDS, J., 2006, ‘The implications of student contracts’, The Oxford Student, 3 March, pp. 33–41.
OSBURNE, W.L. & PURKEY, W.W., 1995, ‘A model faculty peer review process for counseller education
programme’, Journal of Counselling and Development, 73, pp. 654–58.
PAGANI, F., 2002 ‘Peer review: a tool for global co-operation and change’, Observer, 235(December), p. 1.
PIGGOT-IRVINE, E., 2003, ‘Appraisal training focussed on what really matters’, The International Journal of
Educational Management, 17(6), pp. 254–61.
POLLIT, C., 1991, The Politics of Quality: managers, professionals and consumers in the public services (London,
Centre for Political Studies).
POTTER, J., 1988, ‘Consumerism and the public sector: how well does the coat fit?’, Public Administration,
66(Summer), pp. 149–64.
RAMSDEN, P., 2000, Learning to Teach in Higher Education (London, Routledge).
RANSON, S. & STEWART, J.D., 1994, Management for the Public Domain—Enabling the Learning Society (London,
Macmillan).
ROBERTS, G.E., 2002, ‘Employee performance appraisal system participation: a technique that works’, Public
Personnel Management, 31(3), pp. 333–42.
ROWLEY, J., 2003, ‘Designing student feedback questionnaires’, Quality Assurance in Education, 11(3), pp. 142–49.
SANDERSON, I. (Ed.), 1992, Management of Quality in Local Government (Harlow, UK, Longman).
SHELLEY, S., 1999, ‘Diversity of appraisal and performance related pay practices in higher education’, Person-
nel Review, 28(5/6), pp. 439–54.
SPRENG, R.A. & MACKOY, R.D., 1996 ‘An empirical examination of a model of perceived service quality and
satisfaction’, Journal of Retailing, 72(2), pp. 201–14.
TAM, M., 2001, ‘Measuring quality and performance in higher education’, Quality in Education, 7(1), pp. 47–54.
WILSON, A.M., 1998, ‘The role of mystery shopping in the measurement of service performance’, Managing
Service Quality, 8(6), pp. 414–20.
YIN, R.K., 2003, Case Study Research: design and methods, 3rd edn (London, Sage).