You are on page 1of 5

Background

Evaluating the The strong move towards evidence-based


health-promoting health education and promotion in recent
school years, the focus on quality standards, and the
need to justify the use of scarce resources,
have made monitoring and evaluation an
Alysoun Moon
essential part of any health-related
programme in schools. Questions about the
impact of a programme, its effectiveness, and
resultant changes in knowledge, attitudes and
behaviour have to be answered. There are a
number of difficulties, however, particularly
when programmes encompass a whole school
approach and are not focused and behaviour
specific (Moon et al., 1999a).
The author
Alysoun Moon is a Senior Research Fellow, Health Care
Research Unit, University of Southampton, Southampton, The components of an effective
UK. intervention

In recent years, the concept of health has been


Keywords
broadened, from a focus on specific individual
Schools, Health, Curricula behaviours in a personal and social vacuum,
to a holistic one which incorporates lifestyle
Abstract and examines the influences of home, school,
Examines some of the constraints on the effective the media, the community and the workplace.
evaluation of the health-promoting school, particularly This has meant that the process of evaluation
where a whole-school approach is used, and outcomes has become complex. A healthy school
are not behaviour-specific. Considers the components of a approach includes and involves teachers,
successful school health intervention, and identifies some support staff, parents, visitors to the school
of the difficulties in achieving comprehensive evaluation. and the wider community, as well as the
Explores the advantages of randomisation and controlled pupils. There is a need for multiple
studies and identifies some limitations of these interventions and approaches when trying to
approaches, when the main aims are to examine process bring about behavioural, environmental or
and non-specific behavioural outcomes. Identifies some social health-related change.
common errors that have been made in school-based Allensworth (1993) carried out an
evaluations in the past and argues that the use of assessment of the state of health education
multiple methods of evaluation are necessary for a multi- and promotion in the USA in the early 1990s.
factorial whole-school approach to teaching and learning Following this review she noted the increasing
about health. Discusses the recommendations for future complexity of health education and its many
evaluation by the WHO European Working Group on elements that need to be addressed
Health Promotion Evaluation. (Allensworth, 1994), and identified a number
of components of an effective school-based
Electronic access intervention. These include:
The current issue and full text archive of this journal is
. the use of multiple theories and models
available at when planning interventions;
http://www.emerald-library.com
. a focus on priority health behaviours;
. an expanded curriculum, to include, for
example, food services, and the
community;
. the use of multiple strategies in
addressing ``problem'' behaviour;
Health Education
Volume 100 . Number 6 . 2000 . pp. 237241 Received February 2000
# MCB University Press . ISSN 0965-4283 Revised version accepted June 2000
237
Evaluating the health-promoting school Health Education
Alysoun Moon Volume 100 . Number 6 . 2000 . 237241

. co-ordination of school and community Randomised controlled trials are still


health promotion activities; thought by many to be the best research
. co-ordination of whole school design, because they avoid selection bias, and
programme through all subjects; distribute confounders by chance. There have
. promotion of active pupil participation in been a number of school-based evaluation
lessons, and the use of active learning studies in which different schools have been
methods; randomised to different conditions. The
. a focus on the development of life skills; studies have been critically appraised
. a wider view of all aspects of school life, subsequently and judged to be sound
such as developing a caring, nurturing methodologically. Nutbeam and Smith (1991)
environment; and suggest that ideally, youth health evaluations
. close co-operation with parents, carers should follow a basic randomised control,
and, where possible, whole families. experimental design. This will involve:
It is clear that any extensive evaluative studies . pre-test studies to establish baseline
of school-based health promotion, measurements;
particularly of the health-promoting school . the use of a representative sample of the
and its effectiveness in changing school target population;
management, policy and practice and pupil . the random assignment of subjects to
behaviours, are likely to include a range of intervention and control groups;
methods and approaches (Elder, 1991; . the use of a clearly designed intervention;
Bremberg 1991; Moon, 1999b). and
. the use post-test studies to identify
change from baseline measurements.
Some difficulties in evaluating However, as Nutbeam and Smith (1991)
health-promoting schools acknowledge, it is impractical, and possibly
unethical, to allocate children to control and
The need to monitor and assess each
intervention groups and recognise that school-
component highlights the importance of
based research can be very difficult. There are
employing a variety of research methods and
so many other influences on the health of
approaches. Each intervention approach and
young people that they cannot always be
its implementation will need to be dealt with
accounted for. Young people may spend a
separately and variables excluded where
possible. These variables may include considerable part of their time in school, but
extraneous influences that will be hard to they spend much more within their own home
identify and measure but which can skew the background with family, friends and others.
results. The media, for example, may be Even carefully matched schools that are close
responsible for having a powerful impact on together can be contaminated, for example,
attitudes through a programme screened at where groups from each school mix with each
the time of the intervention. These attitudes other socially and share what is happening in
may then be reinforced by the intervention the intervention.
but cannot be said to be a specific outcome of The complex nature of schools means that
the intervention. randomised controlled trials are not always
Funding bodies generally focus on appropriate or manageable, logistically and
behavioural change as a measure of ethically, in school settings. Random allocation
effectiveness, and usually over a period which is not always possible, particularly in a study
is short. This demonstrates a lack of where schools volunteer to take part in the
understanding about what is involved in intervention. The fact that intervention schools
school-based health promotion and of the are volunteers introduces a bias, and may be
difficulties of changing behaviour through one, indicative of an immediate difference between
or even many, interventions. Short-term them and any controls before the study starts.
follow-up is inadequate to detect medium to Where randomisation is not possible, an
long-term, sustained behaviour change. In any experimental, non-randomised controlled
case, the diverse nature of many health- design is likely to be the best alternative. But
promoting school projects cannot always be the use of controls can also be problematic if
evaluated by looking only at outcomes, as some they too are volunteers, again introducing a
projects will not have measurable outcomes. form of selection bias. Furthermore, the
238
Evaluating the health-promoting school Health Education
Alysoun Moon Volume 100 . Number 6 . 2000 . 237241

feasibility of the randomisation process can be prevention in some way and may result in
a major problem: it requires large samples and changes in their drug related behaviour.
this may be difficult to achieve.
The danger of such errors occurring
When it is not possible to allocate individual
emphasises the need to involve a statistician
students to intervention or control groups, for
early at the planning stage of an evaluation to
example where there is a whole school
ensure that the risks of such errors are minimal.
intervention, the unit of analysis becomes the
Tones (1996) has drawn attention to the
school. This can cause problems because of
dangers of Type III errors. A Type III error
lack of power to detect change unless there is
occurs when a health promotion intervention is
a large number of schools in the sample.
judged to be ineffective when it is actually the
intervention itself which is grossly deficient.
This emphasises the need to pay attention to
Type I, II and III errors
the components of an effective intervention.
Some of these difficulties have resulted in a
number of statistical errors that commonly
occur in school-based evaluations. They Beyond outcome measures
include Type I and Type II errors (Tones and
In the 1970s, Kreuter and Green (1978)
Tilford, 1990; Hansen, 1992). A Type I error
recommended that evaluation should be
happens when a study falsely finds differences
carried out at three levels, namely process
in results or outcomes statistically that do not
evaluation, precursor evaluation and then,
in fact exist. A Type II error occurs when a
where appropriate, outcome evaluation. This
study fails to find differences statistically that
are actually there. In his review of abuse multi-level model is particularly useful in the
prevention programmes, Hansen (1992) context of the health-promoting school, the
highlights the fact that while most studies now holistic nature of which touches on every
seek to eliminate Type I errors, not enough aspect of school life, including ethos and
attention has been given to Type II errors. He environment, and thus demands that
suggests that the conditions that predispose to evaluation should be multi-faceted.
Type II errors include: Process or formative evaluation, which
. Insufficient statistical power to detect measures the activities of the programme, its
differences that may exist (because of quality and who it is reaching, is vital because,
insufficient numbers of subjects by design without it, it is impossible to identify which
or attrition or from insufficiently strong parts of a programme contributed towards
programme impact). An example of this any successful outcomes. Precursor
might be using the school as the unit of evaluation refers to the evaluation of criteria
analysis, where a large number of schools which theoretically, or by previous empirical
would be required to ensure validity. study, have a high probability of affecting
. Control groups that for some reason do health outcomes. Precursor evaluations might
not exhibit sufficient change in behaviour include cost effectiveness studies and
to make any differences detectable. evaluation of the immediate impact of the
. Pre-test non-equivalence that biases intervention on specific knowledge and skills.
against the intervention group. One Kreuter and Green caution that, unless steps
example might be using frequent school are taken to clarify the specific immediate
absentees as a control group, a factor that function of school health education,
makes them different from the rest of the programmes might be judged on outcome
school population from the start and measures that are inappropriate and
highly likely to demonstrate different unrealistic.
health-related behaviour than others. Tones (1996) states that the health-
. Contamination of experimental promoting school ``. . . should be judged
manipulations through the adoption of primarily by its contribution to such health
alternative programme strategies by promotion goals as equity and empowerment
control groups. Pupils in the drug or by the success of initiatives contributing
education control group, for example, to such long term goals''. He then goes on to
may be given alternative work to the argue that, rather than focusing on outcome
intervention that is still related to drug measures to assess output following an
239
Evaluating the health-promoting school Health Education
Alysoun Moon Volume 100 . Number 6 . 2000 . 237241

intervention, intermediate measures of What of the future?


effectiveness and efficiency are needed. These
might include the acquisition of knowledge A major concern for the future is the funding
and understanding, evidence of successful of health promotion evaluations. The growing
values-clarification, the efficient performance awareness amongst teachers and researchers
of health skills, the enhancement of self- of the components of an effective intervention
esteem and changes in self-efficacy beliefs. and of the need for a range of research
Nutbeam and Smith (1991) concluded that methods and approaches, particularly when
only outcomes-related, empirical, evaluating issues surrounding the health-
experimental research is perceived as having promoting school, is not always shared by
high status, and that the value of process- funding bodies, which are still concerned
related research, so called ``soft'' research, is primarily with behaviour change. Process
devalued. ``The paucity of literature on evaluation takes time, even when being
process evaluation in health education carried out by teachers within a school, and
research is testimony to the fact that it is not will need to be costed. Without adequate
yet taken seriously by researchers''. Since funding, practitioners will remain uncertain as
these words were written, the recognition of to what works or does not, and how best to
process evaluation, as well as outcomes move forward. Inadequate and inappropriate
measures, as particularly suited to school interventions, unsupported by research, will
settings, has grown (Baric, 1991; Denman, continue to be used and there will be many
1994; Parsons et al., 1996). Where there are more lost opportunities to influence positively
many process measures of effectiveness, it is the health and well being of all members of
vital to assess which parts worked and why. It the school community, including those in its
is very hard, nonetheless, to isolate input, environs.
process change and outcome as three different A WHO European Working Group on
areas for examination when there are multiple Health Promotion Evaluation that was
``ins'' and multiple ``outs''. In 1996, Tones composed of distinguished, international
(1996) still felt that there remained a lack of health promotion specialists, met in April,
formative or process evaluation, which is 1998. They identified four core features of
necessary to determine which aspects of an approaches appropriate for the evaluation of
intervention actually worked and which had health promotion initiatives:
no impact and were a waste of time and (1) the active participation of all key players;
money. As Tones comments, the simple (2) the use of multiple methods;
application of a ``who'' did ``what'' with (3) evaluations which should enhance the
``whom''; ``where'' did it happen and ``why''; capacity of individuals, communities,
``how did it work?'' checklist may be all that is organisations and governments to address
required. health promotion concerns; and
In practice, outcome evaluation and process (4) evaluations that accommodate the
evaluation are not mutually exclusive. There complex nature of health promotion
are a number of recorded studies in which the interventions and their long-term impact.
acceptability of the intervention to the study
group has been assessed because of the effect They drew up the following recommendations
this can have on the outcome of the for policymakers, applicable to schools, stating
intervention (DiClemente and Wingwood, that they should:
1995; Ploem and Byers, 1997). In a recent . encourage the adoption of participatory
evaluation of a healthy school award scheme approaches to evaluation that provide
in the South of England (Moon, 1999b), the meaningful opportunities for involvement
findings of process evaluation carried out by all of those with a direct interest in
through focus group interviews with pupils health promotion initiatives;
were triangulated with the results of a . require that a minimum of 10 per cent of
quantitative pupil health-related behaviour the total financial resources for a health
questionnaire to identify similarities between promotion initiative be allocated to
the two. The use of both process and outcome evaluation;
measures in evaluating a health education . ensure that a mixture of process and
intervention can enhance the study and give outcome information is used to evaluate
greater insight and validity to the results. all health promotion initiatives;
240
Evaluating the health-promoting school Health Education
Alysoun Moon Volume 100 . Number 6 . 2000 . 237241

. support the use of multiple methods to DiClemente, R.J. and Wingwood, G.M. (1995), ``A
evaluate health promotion initiatives; randomised controlled trial of an HIV sexual
. support further research into the risk-reduction intervention for young
African-American women'', Journal of the
development of appropriate approaches American Medical Association, Vol. 274 No. 16.
to evaluating health promotion initiatives; pp. 1271-6.
. support the establishment of a training Elder, J.P. (1991), ``From experimentation to
and education infrastructure to develop dissemination: strategies for maximising the impact
expertise in the evaluation of health and spread of school health education'', in
Nutbeam, D., Hagland, B., Farley, P. and Tillgren, P.
promotion initiatives; and (Eds), Youth Health Promotion: From Theory to
. create and support opportunities for Practice in School and Community, Forbes
sharing information on evaluation Publications.
methods used in health promotion Flay, B.R. (1985), ``A review of 27 school-based studies of
through conferences, workshops, psychological approaches to smoking prevention'',
Health Psychology, Vol. 4 No. 5, pp. 449-88.
networks and other means. Goldstein, H., Rasbash, J., Plewis, I., Draper, D.,
These recommendations are particularly Browne, W., Yang, M., Woodhouse, G. and
Healy, M. (1998), A User's Guide to MlwiN, Institute
relevant to the health-promoting school in
of Education, University of London, London.
which the use of participatory approaches and Hansen, W.B. (1992), ``School-based substance abuse
multiple methods are necessary to ensure that prevention: a review of the state of the art in
all aspects of the concept are assessed and curriculum, 1980-1990'', Health Education Research,
evaluated thoroughly. There is indeed a need Vol. 7 No. 3, pp. 403-30.
Kreuter, M.W. and Green, L.W. (1978), ``Evaluation of
for further research into the development of
school health education: identifying purpose,
appropriate approaches to evaluating health keeping perspective'', Journal of School Health,
promotion initiatives. The calls for further April, pp. 228-35.
training in evaluation techniques and Moon, A.M., Mullee, M.A., Rogers, L., Thompson, R.L.,
opportunities to share expertise, if acted Speller, V. and Roderick, P. (1999a), ``Helping
schools to become health-promoting environments
upon, can only benefit future evaluation
an evaluation of the Wessex Healthy Schools
studies of the health-promoting school and Award'', Health Promotion International, Vol. 14
make the whole process easier for those No. 2, pp. 111-22.
involved. Finally, the need for dedicated Moon, A.M., Mullee, M.A., Thompson, R.L., Speller, V.
funding for evaluation, to be a built-in part of and Roderick, P. (1999b), ``Health-related research
and evaluation in schools'', Health Education,
every grant for a new intervention, is not only
Vol. 1, pp. 27-34.
essential but will help to ensure that evidence- Nutbeam, D. and Smith, C. (1991), ``Evaluating youth
based health education and promotion are the health education: how can we do it and what
foundation of a health-promoting school. should we measure?'', in Nutbeam, D., Hagland, B.,
Farley, P. and Tillgren, P. (Eds), Youth Health
Promotion: From Theory to Practice in School and
Community, Forbes Publications.
References and further reading Parsons, C., Stears, D. and Thomas, C. (1996), ``The
health-promoting school in Europe: conceptualising
Allensworth, D.D. (1993), ``Health education: state of and evaluating the change'', Health Education
the art'', Journal of School Health, Vol. 63 No. 1, Journal, Vol. 55, pp. 311-21.
pp. 14-20. Ploem, C. and Byers, E.S. (1997), ``The effects of
Allensworth, D.D. (1994), ``The research base for two AIDS risk reduction interventions on
innovative practice in school health education at heterosexual college women's AIDS-related
secondary level'', Journal of School Health, Vol. 64 knowledge, attitudes and condom use'', Journal
No. 5, pp. 180-7. of Psychology and Human Sexuality, Vol. 9 No. 1,
Baric, L. (1991), ``Health-promoting schools: evaluation pp. 1-24.
and auditing'', Journal of the Institute of Health Tones, K. (1996), ``Editorial: the health-promoting school:
Education, Vol. 29 No. 4, pp. 114-20. some reflections on evaluation'', Health Education
Bremberg, S. (1991), ``Does school health education affect Research, Theory and Practice, Vol. 4 No. 4, pp. 1-8.
the health of students? A literature review'', in Tones, K. and Tilford, S. (1990), Health Education:
Nutbeam, D., Hagland, B., Farley, P. and Tillgren, P. Effectiveness, Efficiency and Equity, 2nd ed.,
(Eds), Youth Health Promotion: From Theory to Chapman & Hall, London.
Practice in School and Community, Forbes World Health Organisation (WHO) (1998), Health
Publications. Promotion Evaluation: Recommendations to
Denman, S. (1994), ``Do schools provide an opportunity Policymakers, WHO European Working Group on
for meeting the Health of the Nation targets?'', Health Promotion Evaluation, World Health
Journal of Public Health Medicine, Vol. 16 No. 2, Organisation Regional Office for Europe,
pp. 219-24. Copenhagen.

241