T-KIt N

o
10
Educational Evaluation in Youth Work
Youth Partnership
1-kit on
|ducational |valuation
in youth work
“1asting the soup”
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Welcome to the T-Kit Series
Some of you may have wondered: what does T-Kit mean? We can offer at least two
answers. The frst is as simple as the full Version in English: “Training Kit”. The second
has more to do with the sound of the word that may easily recall “Ticket”, one of the
travelling documents we usually need to go on a journey. In our Imagination, this T-Kit
is a tool that each of us can use in our work.
More specifcally, we would like to address youth workers and trainers and offer them
theoretical and practical tools to work with and use when training young people. The
T-Kit series is the result of a collective effort involving people from different cultural,
professional and organisational backgrounds. Youth trainers, youth leaders in NGOs and
professional writers have worked together in order to create high quality publications,
which would address the needs of the target group while recognising the diversity of
approaches across Europe to each subject.
The T-Kits are a product of the Partnership in the feld of Youth between the Council of
Europe and the European Commission. Besides the T-Kits, the Partnership has resulted
in other areas of cooperation such as training courses, the magazine “Coyote”, research
and youth policy activities and an Internet site hosting also the European Knowledge
Centre for Youth Policy.
To fnd out more about developments in the Partnership (new publications, training
course and seminar announcements, etc.) or to download the electronic version of the
T-Kits, visit the Partnership website: www.youth-partnership.net
Council of Europe Publishing – F-67075 Strasbourg Cedex
ISBN 92-871-6023-2
© Council of Europe and European Commission April 2007
This document does not necessarily express the offcial view of the European Commission
or the Council of Europe, their member states or the organisations co-operating with the
institutions.
Reproduction of material from this publication is authorised for non-commercial educational
purposes only, provided the source is quoted.
2
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
3
Actors
Editorial Team
Authors
Paul Kloosterman is a free-lance trainer and consultant from the Netherlands, presently
living in Italy. He specialises in training of trainers, intercultural learning, working with
disadvantaged youngsters and international teamwork.
Kerstin Giebel works as a “co-ordinator for quality development in international youth
work” at IJAB e.V. (International Youth Exchange and Visitor’s Service) a non-governmental
organisation based in Germany. Her tasks include consulting, training and publishing
on quality related questions to national organisations and European networks. She is
qualifed as an “European excellence assessor” (EFQM) and “Quality manager and auditor
– social institutions”.
Ozgehan Senyuva is from Turkey and has been involved in youth work since 1994. Since
2005, he has been the Vice-President of the Youth Express Network, an international
non-governmental youth organisation specialising in projects against the exclusion of
young people. He works as a trainer and resource person on European youth policies,
Intercultural Learning, social exclusion and integration, anti-racism and xenophobia.
Currently he is completing his doctoral studies in Political Sciences. He is a member of
European Youth Forum’s Pool of Trainers.
Language editor
Yael Ohana works as a free-lance capacity building consultant and editor in the youth and
development felds and is currently based in Bratislava, Slovakia (www.frankly-speaking.
org; www.nonformality.org). From 2000 to 2005 she was an educational advisor at the
Council of Europe’s European Youth Centre in Strasbourg.
Editor in chief
Miguel Ángel García López is an experienced free-lance trainer from Spain, living in Germany.
He also works in the University of Osnabrück, Germany as a resource person for European
Studies. He has been the external evaluator of several youth projects and has completed
post-graduate studies on “Educational Evaluation” in the University of Deusto, Spain.
Acknowledgments
Silvia Volpi for her precious contribution to the development of this T-Kit.
Ian Redpath of the British Council for his organisational and administrative support.
Partnership Secretariat
Hanjo Schild (Partnership Coordinator)
Marta Me˛dlin´ska (Assistant)
Dianna Osayande (Administration)
4
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Partnership in the Youth feld between the Council of Europe
and the European Commission
Council of Europe
NGBEN
1, Quai Jacoutot
F-67075 Strasbourg Cedex
Tel: +33-3-90-21-50-57
Fax: +33-3-88-41-27-77
Web: http://www.youth-knowledge.net
Council of Europe
European Youth Centre Strasbourg
30 Rue Pierre de Coubertin – F-67000 Strasbourg, France
Tel: +33-3-88 41 23 00 – Fax: +33-3-88 41 27 77
European Youth Centre Budapest
Zivatar ucta 1-3 – H-1024 Budapest, Hungary
Tel: +36-1-212-4078 – Fax: +36-1-212-4076
European Commission – DG Education and Culture
Youth Policy Unit
European Commission
Directorate General for Education and Culture
Directorate D, Unit 1
B-1049 Brussels
Tel: +32-2-299-1318
Fax: +32-2-299-4038
Web: http://ec.europa.eu/youth/index_en.html
T-Kit Coordinators
László Földi co-ordinated the T-Kit planning process and composed the editorial team in
co-operation with SALTO UK for the Partnership Secretariat. He currently works for the
Hungarian National Agency of the YOUTH Programme. He is also working as trainer and
consultant in youth work and youth policy issues with special focus on European Citizenship
and Human Rights Education.
Susie Green is the SALTO YOUTH Co-ordinator and works for the SALTO Resource Centre
Network. Evaluation is one of the horizontal priorities of the network. She was involved
in producing the SALTO Evaluation CD Rom for youth projects in 2004.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
5
Table of contents
Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
The European policy context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1. The Ingredients of Educational Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.1 What is Educational Evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.2 Why evaluate? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.3 What is evaluation for? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.4 What to evaluate? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.5 Who and for whom? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
1.6 When to evaluate? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
1.7 Typologies of evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
1.8 Educational evaluation in a broader framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
1.9 Educational evaluation and quality in youth work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
1.10 Develop your own evaluation! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2. “Pots, pans and spices”
Practicing Educational Evaluation! Educational Evaluation in practice! . . . . . . . 49
2.1 How educational evaluation is built into a project? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.2 A project within a project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.3 How to evaluate? Evaluation methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Personal methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Interpersonal methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Group interactive methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
2.4 Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
2.5 A critical look at evaluation methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
2.6 Collecting and organising information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
2.7 Establishing evaluation criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
2.8 Reporting the outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
2.9 Evaluation practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
“Take5” – 40 youngsters evaluating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
EVS and Leonardo multidimensional evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Madzinga – External evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Evaluation in an E-learning pilot course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Overall evaluation of a Pilot Long Term Training Course . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
The “good” fnal evaluation questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
2.10 We still have to cook our meal! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
3. Further exploration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Evaluation standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
A tool for developing your own evaluation plan – The SALTO checklist . . . . 105
Bibliography and webography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
I
Foreword
Evaluation is not something that people get immediately excited about. Often it is asso-
ciatwed with “writing long reports”, “justifying your work to someone else”, “measuring
the immeasurable”, “flling out forms and questionnaires”, etc. Boring, tiring, diffcult:
not exciting at all.
In this T-Kit we want to show you that evaluation can be thrilling, electrifying, stimulating,
fun, exciting… or at least attractive and inspiring enough to integrate it into your educa-
tional work. Evaluation becomes Educational Evaluation.
Evaluation is something natural
Evaluation is something that is natural for human beings. We do it all the time. We collect
information, we process it, we give it a meaning and a value and we act or react according
to it. We do it every day when we cross the street, when we buy something, when we
talk, when we work, when we move, when we cook and we do it without reports and
questionnaires.
So you might ask yourself why should I read a T-Kit on something I already do every
day? The answer is to make it conscious, explicit, refective and organised and to share
it with others. Our “everyday evaluation” we do all by ourselves and in our own way.
But, from the moment we work together with other people we have to agree on how,
when and what we want to evaluate.
Evaluation is like cooking!
Let’s get cooking!
Imagine you are having friends over for dinner. You want to prepare them
a really nice meal, one that takes you through a nice evening. You
decide on the menu you want to serve: four courses starting with a
wonderful soup full of vegetables, then pasta with gorgonzola sauce
followed by the main dish of chicken in curry sauce accompanied by rice
and carrots. For desert you intend to surprise them with your wonderful
home made Tiramisu. All this is accompanied by a nice red wine. You
go the market in the morning and buy all the ingredients and when you
get home you immediately start to prepare. In short: you have a wonderful evening,
great discussions, there is a good atmosphere and your friends really liked the food.
Still, looking back there are a few things that could be improved. After the pasta
most of your guests were already full and were a bit shocked when you served the
chicken curry. And then there was still the Tiramisu waiting for them. Maybe both
pasta and soup before the main course is a bit too much? One of your friends turned
out to be vegetarian, so for her the main course ended up being only rice and car-
rots. When again inviting friends for dinner, you resolve to frst ask them about any
dietary or other special needs they might have. Finally, another friend does not
8
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
drink alcohol and the only thing you could offer him was water. Next time, you
resolve to buy some non-alcoholic drinks to offer to your guests along with wine.
You did your own evaluation and next time dinner will certainly be better. Probably other
things will pop up and you will evaluate those too. You will take them into account
for future dinners with friends. You did this ‘evaluation’ without having a meeting,
without questionnaires and you did not write a report you just did it, naturally.
Try to imagine another situation: you like cooking a lot and so does one of your
friends. The idea is to start a restaurant. Not immediately but, say, in two years.
Until then you want to try things out. Together with your friend you decide to offer
“dinner-evenings” for groups of between 6 and 12 people twice a week, on
Wednesdays and Sundays.
By doing so you can practice cooking, experiment which with ideas
for menus that could eventually be served in the restaurant, fnd out
how the co-operation between the two of you works out, see if there
is any way to earn enough for a reasonable living for two people, and
so on. Your friend’s house has a room, which seems to be perfect for
dining and her kitchen is quite good. So, off you go!
And what is one of the frst things you do? You start planning the evaluation! That is
because you want to “try things out” and to see what the results of your experiment are.
You end up in long discussions with your friend about all kinds of things.
What do you want to evaluate?
the quality of the food
the satisfaction of the customers
the variety of the food
the price and cost of the services offered
the assortment of drinks you offer
the atmosphere during the evenings
the co-operation between you and your friend
….
And how do we evaluate these things?
How do we know the customers like the food? By the tip they leave? Do we give
them a questionnaire? Do we count how many people come back?
When is the atmosphere good? When people leave singing? When everybody is rolling
about laughing?
Do we count costs and benefts every night or should we go for a more long-term
approach?
These refections bring on other questions, like:
What do we (the two of us) mean by quality food? What is our standard?
What kind of atmosphere would we like to create?
When it comes to questions of cost and proft, what would we consider a satisfactory
or a good outcome?
How do we know that people are honest when we ask them if they are satisfed?
Which aspects of the evaluation results can be used when we go to the bank to
ask for assistance in fnancing our plan to establish a restaurant?
And many more …
That is what this T-Kit is all about: planning evaluation, deciding about what you want to
get achieve, exploring its educational value, when to do it, with whom, for what and for
whom.

















www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
9
lntroduction
Target group, objectives and structure of the T-Kit
This T-Kit is primarily addressed to youth workers closely involved in learning processes
and educational activities with young people: trainers, facilitators, mentors, group lead-
ers… Nevertheless this publication should be of interest to other actors involved in youth
projects like organisers, administrators and decision makers.
The ultimate aim of this T-Kit is to support and contribute to the improvement of youth
projects in Europe, by developing the competence of youth workers in the theory and
practice of Educational Evaluation.
In the frst chapter, “The Ingredients of Educational Evaluation” (the “theoretical” part of
the T-Kit) we will explore the basic ideas and concepts around evaluation that are relevant
for educational activities with young people. In the second chapter “Pots, pans and
spices” (the “practical” part) we will map out, share and critically look at existing prac-
tices of evaluation (strategies, methods…). We will then invite you to defne your own
evaluation strategies, methods and criteria so that you can improve your projects (in
terms of achievement of its objectives, products, structure and process).
In the foreword, we have introduced the metaphor of
cooking. It is used in other parts of the T-Kit, (you can
always identify them because they are sign-posted with
our cooks)…
Nevertheless, this T-Kit is not a recipe book on evaluation methods that can be auto-
matically transferred into your activities. Rather, it is an educational and methodological
tool to help you to naturally develop, integrate and improve educational evaluation in
your youth work. Concrete experiences and existing practices of evaluation as developed
in practical youth work in Europe will serve us as reference material.
Foci of the T-Kit: educational activities
and the scope of educational evaluation
Educational evaluation is a wide topic with a lot of implications. In this T-Kit, we will
deal with a number of them, but we want to stay focused:
The frst focus is an educational one. This T-Kit is about educational evaluation.
This means that we look at evaluation as an inherent part of educational processes
in youth work and we think about how to put evaluation at the service of educa-
tional achievements.
There are other approaches to evaluation: as part of decision-making processes, from
the point of view of project management, income-outcome oriented evaluations, etc. Our
approach does not necessarily contradict any other approaches to evaluation. It is simply
different. We refer to other approaches on several occasions. Nevertheless, the focus will
remain on (non-formal) education.

IU
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
With this particular focus, this T-Kit complements and in some cases completes the ideas
on educational evaluation already present in the T-Kit series (for example in the T-Kits
on Training Essentials, Project Management, Organisational Management and European
Citizenship). In the same way, the CD-Rom on Evaluation produced by SALTO
16
is comple-
mentary to this T-Kit and has been used as a source in its production.
The other focus refers to the scope of youth activities that we will consider in the
“practical” part of the T-Kit. These are educational feld activities involving learning
processes with / involving young people, such as: youth exchanges, seminars, training
activities, volunteering. Among these, we will concentrate on the specifcity and
practice of youth projects with a European dimension.
Again, this focus is not intended to be exclusive. As relevant, we have also made reference
to non-feld activities and other youth services such as information, publications or
campaigns.
16 SALTO-YOUTH.net is a network of 8 Resource Centres working on European priority areas within the youth
feld. It provides youth work and training resources and organises training and contact-making activities
to support organisations and National Agencies within the frame of the European Commission’s YOUTH
programme and beyond. More information at http://www.salto-youth.net In 2004, SALTO produced a CD
ROM on evaluating training in European Youth Work. Theory is combined with examples of proven meth-
ods from extensive experience of evaluating training activities. It includes documents to download and
adapt and some suggestions for going further. More information at: http://www.salto-youth.net/evaluation-
intro/

www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
II
The European Youth Policy Context
For both the Council of Europe and the European Commission evaluation plays a very
important role in verifying the effects of their policies and programmes related to young
people and for planning further strategies and priorities.
In the past decade in the youth feld, both institutions have put also signifcant emphasis
on developing educational activities in which evaluation and its methodology is a primary
tool for improving and refning the quality of youth work.
Since adoption of the White Paper: A new impetus for European Youth
17
, the European
Commission promotes the elaboration of evidence-based youth policies in the member
states and is committed to planning its policies through a consultative process
involving several evaluation methods. It is called the open method of co-ordination.
The European Youth Pact
18
, which highlights youth issues in core areas of the Lisbon
strategy for growth and jobs, invites Member States to ensure effective implemen-
tation and follow-up by setting measurable objectives and developing a structured
dialogue. Both Member States and the European Commission are invited to evaluate
the framework for European co-operation in the youth feld in 2009.
The Youth in Action programme for the period 2007 – 2013 was developed on the
basis of a comprehensive evaluation process: the results of a public consultation,
the results of national and European level mid-term evaluations established to refect
on the YOUTH programme
19
2000 – 2006 and leading to an interim evaluation report
and an ex-ante evaluation of the future instrument. The interim evaluation of the YOUTH
programme also included a number of recommendations based on the expectations
and requests for simplifcation expressed by stakeholders and young people.
In the European Commission’s YOUTH Programme refective evaluation is considered
to be an essential criteria for implementing quality youth projects. The programme
expects youth workers and project promoters to use evaluation techniques in order
to make sure that the objectives they set are realistic, achievable and serve the
needs of the young people involved, keeping in mind the community context in
which the project takes place.
17 The White Paper on Youth was launched in November 2001, after a one and half year long consultation
process involving young people, experts in the youth-feld, national authorities and NGOs. It is a docu-
ment containing proposals for Community action in the youth feld. It presents a detailed and well-argued
policy for discussion and for decision. The White Paper is available at:
http://europa.eu.int/comm/youth/whitepaper/index_en.html
18 The European Youth Pact was adopted by the Spring European Council in March 2005 as one of the instru-
ments contributing to the achievements of the Lisbon objectives of growth and jobs. The Pact focuses on
three areas: employment, integration and social advancement; education training and mobility; reconciliation
of working life and family life. The European Youth Pact is available at PRESIDENCY CONCLUSIONS - Annex
I: European Youth Pact at http://ue.eu.int/ueDocs/cms_Data/docs/pressData/en/ec/84335.pdf
19 The Youth in Action programme is the EU’s mobility and non-formal education programme targeting young
people. The programme is open to youth in 31 European countries. The programme offers possibilities to
young people in the form of group exchanges and individual voluntary work, as well as support activities.
More information at http://eacea.ec.europa.eu/youth/index_en.htm

I2
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
In order to foster recognition and visibility of youth work activities, particularly in
the framework of the YOUTH programme, the European Commission is going to
establish a YOUTH-PASS, a tool for assessing non-formal learning progresses.
The Council of Europe and its Directorate of Youth and Sport have been pioneers
in initiating the discussion on evidence-based youth policies as well as the quality
of youth work in Europe.
The European Ministers responsible for Youth of the 46 Member States of the Council
of Europe identifed four key items as priorities of the Council of Europe’s youth
sector for 2006-2008
20
: developing and promoting standards for youth policies,
fostering the recognition of youth work and non-formal education competences,
developing and sharing knowledge on the situation of young people and supporting
the quality and sustainability of European youth work training and policy.
Accordingly the Directorate of Youth and Sport is running a long-term project for
the evaluation of national youth policies in a range of the Member States. On the
basis of an extensive national evaluation report and study visits in the countries
concerned, an expert team drafts an international evaluation report (“review”), which
is presented to the Minister responsible for youth issues at a public hearing. The
compilation of youth policy indicators provides Member States with a frame of
reference and guidelines for undertaking a thorough evaluation of their youth
policies, in terms of consistency, co-ordination between relevant domains and
practical implementation.
With regard to quality standards in education and training activities the Directorate
for Youth and Sport has elaborated reference criteria for all those involved and
concerned by these kind of activities of the organisation, including staff, trainers,
consultants, participants and partner organisations. The criteria serve as a benchmark
for the evaluation of education and training activities and are understood as
minimum standards. A thorough and open process of evaluation is defned as one
of the relevant quality criteria in order to secure, among others, stock-taking of the
results, the evaluation of the quality of the learning process and the follow-up to
be given.
In the same framework the Council of Europe promotes the recognition of non-formal
education and youth work by investing in the description of what constitutes quality
youth work. The Directorate of Youth and Sport is now going to complete the long
process of producing a new quality assessment tool: the European Portfolio for youth
leaders and youth workers
21
, which after a public testing phase will be published
in early 2007.
Also in the Partnership Programme in the Youth feld between the Council of Europe
and the European Commission
22
, evaluation plays a crucial role in all of its strands:
training, research and Euro-Med. All activities are developed with the support of
appropriate evaluation approaches. The Training strand of the Partnership itself was
as well subject to an external evaluation assessing the relevance, the impact, the
effciency and effectiveness of activities. As one of the results the three separate
pillars of the Partnership that existed earlier were merged to one single partnership
agreement under a joint management.
20 See http://www.coe.int/t/e/cultural_co-operation/youth/2._Priorities/policy.asp
21 The European Portfolio for youth leaders and youth workers can be accessed at
www.coe.int/youthportfolio
22 More information on the Partnership Programme: www.youth-partnership.net


www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
I3
1. The ingredients of educational
evaluation
1.1 What is Educational Evaluation? Defnitions
Some possible defnitions of educational evaluation are:
“Educational evaluation is the systematic investigation, observation and interpretation
of information.”
23
“Educational evaluation is a method (procedure) and to prove if the expectations and
aims of an educational process refect reality (results of the process).”
24
“Educational evaluation is the process of obtaining information and using it to come to
some conclusions which will be used to take decisions.”
Without wishing to present only one defnition or to summarise all existing ones, we can
conclude that different processes of educational evaluation have some common components.
The following defnition embraces them.
“Educational evaluation is a systematic and ongoing process which includes:
Researching and collecting information, from different sources, about the learning
process, the content, the methods, the context, the outcomes of an educational
activity
The organisation and analysis of that information
The establishment of certain criteria (evaluation criteria)
The discernment and judgement of the analysed information (according to the
set evaluation criteria and at the light of the educational objectives).
Drawing conclusions and recommendations which allow the re-orientation and
eventual improvement of the educational activity”
25
.
It is necessary to distinguish educational evaluation from the process of collecting and
obtaining certain kinds of information. The collection of information is something punctual,
and it is done in the evaluation process at certain moments. But educational evaluation
is an ongoing process. It implies judgement (good, bad, acceptable, ok, advantageous,
disadvantageous, of high quality, of low quality, etc.) Educational evaluation implies
measurement. But, educational evaluation goes beyond the mere measurement: it provides
explanations and conclusions.
23 Tenbrink, T. & Cooper, J. M. (2003). Educator’s Guide. Page 64.
24 Nydia Elola, Lilia V. Toranzos 2000 Evaluación educativa: una aproximación conceptual. Page 2.
25 Giovanni Iafrancesco, 2001. Hacia el mejoramiento de los procesos evaluativos en relación con el aprendizaje.
Page 6.





I4
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Educational evaluation should not be a justifcation of shortcomings. As a result of the
evaluation we might fnd out why certain shortcomings appeared, but the actors involved
should not conceive and initiate the evaluation process looking for “excuses” or with a
defensive attitude.
Educational evaluation should not be a money making strategy. As we will see later, one
operational objective of educational evaluation is to refect whether the educational
activity met the expectations of partners and funding institutions. Those institutions often
use evaluation as an instrument of control and sometimes even as a criteria to decide
which projects and organisations to support. This has fnancial implications for those
concerned. But, the starting point, the nature and primary aim of educational evaluation,
is not to make money. If we “fnancially” misuse evaluation and fall into a lack of honesty
and transparency, we might in the best case gain some short-term fnancial beneft.
Experience, however, shows us that apart from weakening the legitimacy of educational
evaluation, fnancially motivated evaluation damages trust between funders and those
they support.
Educational evaluation should not become an exercise in power politics. In educational
evaluation, different actors with different levels and areas of responsibility are involved.
Often and naturally their conclusions differ. This, together with the fact that conclusions
can to changes that can affect different actors in more or less positive ways, might imply
that educational evaluation degenerates into a kind of power game, with different actors
trying to assert their power over others using the evaluation as an instrument of their
objectives. There is no magic rule for resisting this. But, actors who hold positions of
responsibility should be those most open to criticism and generous in assuming respon-
sibility, even if the results of the evaluation are not to their liking. These attitudes are
an important contribution to making sure that the aims and nature of the evaluation are
properly respected.
Educational evaluation should not be a public relations exercise. Of course, the outcomes
of educational evaluation and its achievements should be shared witwh concerned target
groups, organisations and institutions. But, this should be an integral consequence of
the process rather than an objective of the evaluation in itself. Conceiving and planning
educational evaluation as a public relations exercise will damage the process and in long
term becomes counter productive co-operation and partnership with others.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
I5
1.2 Why evaluate?
The aims and nature
of educational evaluation
To learn
The frst and most basic aim of educational evaluation is to learn. The aim is the learning of
all actors involved: their access to additional knowledge and to a new learning opportunity.
The educational purpose is the inherent characteristic of what makes educational evaluation
different from other kinds of evaluation.
While evaluating, the actors involved learn to understand, to give a value and to draw
conclusions on their own learning experiences. Through educational evaluation we learn
from experience. The changes and actions resulting from educational evaluation become
critical action and refective praxis
26
.
All the actors involved in educational evaluation learn to express their knowledge: knowl-
edge not of “topics” but of the relevance of their educational experience to their own
lives. A certain educational activity might seem to be very good from a lot of points of
view but in reality it might be disconnected with the life of participants, and vice versa.
This relevance, and connection between youth work and the lives of young people, is
probably the most important “knowledge” in youth work. Very often it is learned during
the evaluation process.
Participants also learn while sharing and confronting their judgements with those of their
colleagues. During the evaluation process, different interpretations, meanings and inter-
relations are raised and debated. Very often actors involved in educational evaluation
ask themselves: What does it mean? How should I interpret this or that result? What are
the implications? The doubt implied by the diversity of answers that one might get to
these questions can be considered as a matter or competence. The different actors need
to have a certain “tolerance of ambiguity”, the ability to live with several different pos-
sible outcomes, not all of which ft easily together or with one’s personal, professional
or organisational values, to be able to accept that the outcomes of the evaluation may
not be to their liking. Where this competence is present, educational evaluation can be
a motor of curiosity, a source of learning and an impulse to continue learning. When
evaluation and learning take place at the same time, the actors involved create, discern,
imagine, analyse, contrast, elaborate answers, formulate questions, come up with doubts,
search for other sources. In other words, they truly evaluate.
In our opinion, and without wishing to ignore other purposes or diminish their importance,
educational evaluation should frst and foremost be put at the service of learning.
26 Praxis: the translation of an idea into practice. Source Word Reference Dictionary http://www.wordreference.com.
I6
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
To motivate
The evaluation process should lead to improvements and change. Change, improvement,
evolution and further development are factors of motivation for all the actors involved in
the educational process. That is the reason why a constructively carried out educational
evaluation contributes to maintaining a challenge and to fostering motivation within a
project.
An evaluation whose results or process de-motivates becomes limited and incomplete
because it cannot maintain the participation of all actors. Some participants might have
negative and discouraging perceptions of evaluation. This can be a result of the fact that
at times in formal education, evaluation (or more precisely put student assessment) is used
as a mechanism to “select” or “exclude”.
However, achieving the objective of motivating while evaluating does not only depend
on “recognising the achievements as well as shortcomings” of what is being evaluated.
It also depends a lot on the attitude adopted by those involved, the atmosphere in which
the evaluation takes place and on the imagination of the actors about what will happen
after the evaluation results have been made public.
To participate
Educational evaluation is an opportunity both to promote the values of participation and
to practice it. Obvious as it may seem, all the actors involved in the educational process
should therefore also be involved in its evaluation. This participative dimension goes
beyond the “democratic legitimacy” of changes to the educational process. It also has
an educational dimension. It would not be coherent or consequent to aim for the pro-
motion of participation in an educational activity but to evaluate the fulflment of that
aim in a non-participatory manner. This aim of promoting participation while evaluating
has methodological consequences: in educational evaluation participatory methods are
very important.
To change and improve
As we have seen in its defnition, change and improvement are integral to the process
of educational evaluation.
This idea of change is generally assumed in an “operational” way: change of tools, formats,
methods, places, targets. Change as a consequence of the accelerated changes taking place
in our societies and in the reality of young people.
In educational evaluation the changes also happen at the personal level: change of attitudes,
of values, of ways of understanding. This “personal” dimension of change is often less
visible than the “operational” one. But, both are equally important: educational evaluation
requires openness to changing our ways of doing things as well as our way of thinking.
Resistance to evaluation is often rooted in resistance to “internal” and “external” changes
that might be required of an individual or of a group as a result of the outcomes of the
evaluation process.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
II
1.3 What is evaluation for?
The Operational Objectives of Educational
Evaluation
It is possible to identify some “operational objectives” or uses of educational evaluation,
in addition to the general aims of educational evaluation.
The following objectives are linked to the implementation and practice of educational
evaluation and, to the uses of its outcomes at different moments of the process. The
following list is certainly not exhaustive. Educational evaluation has a lot of potential
and uses, but among others, the following:
To plan better
Educational evaluation can help to change things and to plan “different things”, but it
can also help us to plan things better, in order to prevent negative consequences and
to compensate for possible shortcomings.
To take stock of achievements
It is important to recognise, name and give value to the achievements of the educational
process so that they do not get lost or not suffciently used.
To consolidate results
Identifed results can be consolidated by making them explicit at the end of the evalu-
ation process. The description, sharing and further use of results are natural follow-up
steps of educational evaluation.
To check if we met the interests of the funding institutions
When funding institutions support a certain educational project, they do it according to
certain criteria: the nature of the project, its objectives, their priorities. Funding institu-
tions usually ask to receive a descriptive and evaluative report at the end of the project.
Even so, educational evaluation plans and criteria should not be limited to the expecta-
tions of the organisations that fund the project. But, it is important to consider and
include them. Usually this is not a diffcult exercise: in most cases the questions they
would like to answer through evaluation would be part of our evaluation anyway.
I8
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
To reinforce co-operation with partners
If partners are involved in the educational project, they will be involved in its evaluation.
A constructive and participative evaluation will naturally reinforce co-operation.
But, even if your partners are not directly involved, the results of the evaluation can be of
interest to them. You might share new ideas for common projects, other felds of common
interest and ways of co-operating, new partners and networks with them.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
I9
1.4 What to evaluate?
Fields and “models” of evaluation -
a critical view and a proposal
Together with the question “Why evaluate?”, “What to evaluate?” is another key to under-
standing the different options and approaches to educational evaluation available to us
in our youth work.
In the following sections, we will briefy present some models, which propose a certain
structure of “What to evaluate?”. We will look at the models we present critically, not
individually, but rather refecting on some approaches to evaluation present in all of
them. Based on this exploration, we will make a proposal: “Educational evaluation as a
total experience”.
Evaluation felds and “models”
In the T-Kit on “Training Essentials” (Page 77) we can fnd the description of four differ-
ent models of educational evaluation. To those we can add one favoured by the USA
Department of Education as mentioned in the T-Kit on European Citizenship (Page 57).
Each of these models focuses on different felds, when it comes to “What to evaluate?”
The following table describes briefy all those models and their associated felds of eval-
uation.
The Kirkpatrick model:
Four felds:
Reaction: personal refection from participants, i.e. on satisfaction, effect and util-
ity of the training programme
Learning: Growth of knowledge, learning achievements
Behaviour: Changes in behaviour, transfer of competencies into concrete actions/
situations
Results: long term lasting transfer, also in organisational and institutional terms.
An example of educational evaluation carried out following this model is described
in the Chapter of this T-Kit entitled “Evaluation Practices”. See “Overall evaluation
of a Pilot Long Term Training Course” (Pages 94-96) for more in-depth information.




2U
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
CIPP model:
Four felds:
Context evaluation – Are the chosen goals the right ones for this activity?
Input evaluation – Is the programme well planned? Are there enough resources
to implement the activity?
Process evaluation – How was the fow of the activity? What feedback was received
from the participants?
Product (outcome) evaluation – Were the objectives reached?
The Brinkerhoff model:
Six felds:
Goal setting – What are the needs? Are these needs real?
Programme design – What is required to meet these needs? Is this design going
to meet the needs?
Programme implementation – How do we evaluate the programme in practice?
Immediate outcomes – Did the participants learn? What did they learn?
Intermediate or usage outcomes – Are the participants implementing their learning?
Impacts and worth – Did it make a worthwhile difference to the participants’
organisations and their personal development?
Systems approach (Bushnell):
Four felds:
Input – What goes into the training effort? (Trainee qualifcations, trainer competence,
resources, etc).
Process – How adequate are the planning, design, development and implemen-
tation of the activity?
Output – What are the participants’ reactions? Have they gained knowledge or
skills?, did they refect on their behaviour? Did their attitudes change?
Outcomes – What are the effects on the participants’ organisations?”
27
United States Department of Education:
Three felds:
Outcome – The immediate, direct consequences of an educational activity on its
participants.
Impact – Long-term outcomes of a programme as well as unanticipated effects.
Process – Focuses on the procedures, methods and their implementation.”
28

27 “T-Kit No. 6 – Training Essentials”. Council of Europe and European Commission (October 2002) Page 77.
Original Sources J.J Jackson, Training and Evaluation and R.L. Simone and D.M. Harris, Human Resource
Development.
28 “T-Kit No. 7 - Under construction...Citizenship, Youth and Europe”. Council of Europe and European
Commission (May 2003) Page 57.

















www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
2I
A critical view on evaluation felds and “models”
As we have seen, each of those models put the emphasis on different evaluation felds.
Instead of analysing in detail, one by one, the potentialities and limitations of each
model, we would like to critically refect on the different main focuses that an evaluation
can take horizontally: by objectives, by competencies, by achievements. This horizontal
perspective will provide us with elements to critically understand the previously described
models or others and (what is more important) will inspire our own approach to educa-
tional evaluation.
Evaluation by objectives
This is the frst and the most natural orientation of an educational evaluation: to evaluate
a project according to its objectives. The purpose of the objectives in a project is to
guide the educational process and to give an orientation to its evaluation.
The objectives should be clear, relevant, be organised in a hierarchy of priorities, be
adapted to the needs and profle of participants. They should also be organised in a
time frame for their achievement. Evaluation by objectives is associated with “keeping
track of those objectives” and re-orienting the evaluated educational process if it is not
going in the right direction.
The advantages and disadvantages of this kind of evaluation are:
Advantages Disadvantages
There is a clear direction in the eval-
uation process: looking at the fulfl-
ment of the objectives
The objectives of an educational
activity are the common reference for
the different elements and actors. For
this reason, evaluating by objectives
facilitates the interaction between
them


An evaluation that focuses exclusive-
ly on the set objectives cannot alone
guarantee the fulfilment of the
desired learning objectives
There is the risk of paying little
attention to the process and other
aspects of the activity
Sticking only to an evaluation of the
objectives limits the creativity and
originality of the educational process
and does not refect the diversity or
multidimensionality of relevant infor-
mation that can come out of an edu-
cational activity



“Woolf (1999) state that evaluation can only be efecive if SMART obecive
are se for a projec. SMART obecive are those that are Speifc, Measurable,
Achievable, Realisic, and achievable within a Timecale. Tis aproach can be very apealing
for its neatne and simplicity, and indeed evaluation by SMART obecive doe have its place
and value. However, we feel that an evaluation proce defned purely around measurable
obecive canno always do justice to the richne o out-come, and the long-term impact
o educational procee. Is SMART so smart, afer all?”
29
29 “T-Kit No. 7 – Under construction...Citizenship, Youth and Europe”. Council of Europe and European
Commission (May 2003) Page 57.
22
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Evaluation by competencies
This approach evaluates the competencies (knowledge, skills, attitudes, abilities and
values) gained, developed or achieved during the educational activity.
In non-formal education, competence is understood as “knowing how to do something
in a certain context”. Consequently, evaluation by competencies analyses the pertinence,
relevance and value of the educational process within its social context. This kind of
evaluation tries to answer the question: In the end, what is the value of this educational
activity in our social context? For example: evaluation by competencies could look at the
value of an activity in which participants develop their intercultural competencies in the
context of our multicultural societies.
The advantages and disadvantages of this kind of evaluation are:
Advantages Disadvantages
Evaluation by competencies makes
more explicit the link between non-
formal education and its social con-
text. And this relation / interaction
can be a strong source of learning
for participants
The combination of individualised
qualitative and quantitative informa-
tion that can be gathered in this kind
of evaluation can be very useful for
supporting the personal develop-
ment of participants. For example,
individual information can be the
basis for designing personal devel-
opment plans and tools for self-
evaluation / assessment
The interpretations, arguments and
propositions explored in this kind of
evaluation can become a catalyst for
the further development of partici-
pants’ competencies



It is not always straight-forward to
evaluate competencies in non-formal
education and to be able to relate
their value to a wider context
For example: a certain participant
develops in an educational activity
the competence of “team work”.
Although “team work” is a very
important competence in a wider
social context, the extent to which
(s)he be able to put it into practice
outside of the educational activity is
diffcult to asses
The concept of competence is asso-
ciated with a large number of catego-
ries and indicators, which can
complicate the organisation and
implementation of the evaluation. For
example, the team work competence
is associated with: communication,
planning, management, negotiation,
mediation among others.
If the context (particularly the fnan-
cial context) has too much infuence
in the identifcation and defnition of
the competencies to be developed
in educational activities, there is a
risk that non-formal education will
become subordinated to the needs
of the market.
In recent years, an important focus
of evaluation in the feld of non-for-
mal education has been on the ways
in which the “employability”
15
of par-
ticipants can be improved through
non-formal education.



15 Employability can be defned as “the capability to move self-suffciently within the labour market to realise
potential through sustainable employment” Source: The Wiki Encyclopaedia.
http://en.wikipedia.org/wiki/Employability
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
23
Evaluation by achievements
Achievement, as such, is an incomplete concept: Achievement of what? In educational
evaluation the notion of achievement is related to the “achievement of objectives” or
the “achievement of a certain level of competence”. As such, one would be forgiven for
thinking that evaluation by achievements is just another way of evaluating the fulflment
of the objectives or evaluating the competence developed through any given activity.
But, evaluating by achievements does provide some new perspectives to refect on.
Associated with the notion of achievement is the concept of an “indicator”.
“An indicator is a signal, indication, characteristic, data of perceivable information
that after being confronted and interpreted according to what was expected,
can provide us with evidence of the level or evolution of a certain aspect of the
educational process.”
30

For example, if one of the objectives of an activity is to promote participation, an indi-
cator of the achievement of that objective could be the number of people taking the
foor in plenary to express their opinion. If another objective is to encourage team-work,
an indicator for the achievement of this objective could be the number and composition
of teams voluntarily set up by participants. If another objective is to promote co-operation
among participants, an indicator could be the number and characteristics of common
initiatives and projects.
The advantages and disadvantages of this kind of evaluation are:
Advantages Disadvantages
Achievements and indicators of
achievement are more concrete com-
pared with the ambiguity of evaluating
the fulflment of the objectives and the
development of competence in a more
general way
Evaluation by achievements and using
indicators can generate further ques-
tions and debates.
If in the middle of an activity the train-
er says “Come on, so far just one
quarter of the participants have so
far taken the foor to speak in ple-
nary”, then, participants might start
to refect why, or even to discuss if,
taking the foor in plenary is an indi-
cator of “participation”.
Some indicators could be common
to different educational activities cre-
ating some comparability between
evaluation results.



There is little conceptual clarity
around the notion of achievement.
Achievement of what? For what?
Under whi ch ci rcumstances?
Achievements are always related to
other aspects of evaluation
A narrow and limited use of indica-
tors can be counter-productive and
confusing.
If the only indicator to evaluate the
achievement of participation is talk-
ing in plenary then the notion of par-
ticipation might become exclusively
equated with speaking in plenary
(e.g. participation=talking in plenary).
If the expected level of achievement
and/or the indicators are set by an
external person or institution, they
can become instrumental objectives
of the activity.
If trainers know that the funder, the
organiser or the external evaluator will
“take note of the number of people
who talk in plenary”, then “talking
in plenary” (and not “participating”)
might become an instrumental and
misleading objective of the activity.



30 La Evaluación como Experiencia, Cerda, G. Hugo. Coop. Editorial Magisterio, Bogotá, 2000
24
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Evaluation by performance
Performance is a concept that is commonly found in the private sector, and is often it
related to the assessment of human resources and is often used as a criterion in staff
selection.
Evaluation by performance focuses on the appreciation of how individuals behave “pro-
fessionally”, taking into account their potential for further development. For example,
evaluation by performance of a youth leader would analyse how (s)he organises an activity,
interacts with participants, facilitates a group discussion or develops an exercise.
In non-formal education, evaluation by performance is commonly found in the context of
“training for trainers” or in long courses in which participants are expected to implement
workshops or projects autonomously as part of the learning experience. Evaluation by
performance is strongly linked to the further training and lifelong learning needs of expe-
rienced actors in non-formal education.
The advantages and disadvantages of this kind of evaluation are:
Advantages Disadvantages
This kind of evaluation is good for
assessing human potential and
putting it at the service of a wider
group (other participants, colleagues
- trainers, youth leaders and youth
workers).
This kind of evaluation can be very
helpful to set up strategies for update
and further training.
It can be used to promote the recog-
nition of individual professionalism
and of non-formal education, in gen-
eral, in other contexts.



This kind of evaluation is only rel-
evant for a limited group of actors
in non-formal education, those who
already have a certain level of expe-
rience in non-formal education.
Evaluation by performance implies
the management of a large number
of variables: individuals performing
in different circumstances, doing dif-
ferent things, in different contexts.
This personalised evaluation requires
time, specialised staff and a lot of
resources in order to be consistent.
If the evaluation environment is not
safe or professional enough it can
imply lack of transparency and pro-
voke mistrust between those engaged
in the evaluation and those who per-
formance is being evaluated.
In this kind of evaluation personal
styles and preferences can play an
important role. If it is not carried out
with the necessary distance from
social roles, there is the risk of the
evaluation suffering from subjectivity.




A good illustration of these advantages, disadvantages, potentialities and limitations is
one kind of evaluation, often used in training for trainers in non-formal education and
youth work. This usually takes the form of participant-trainers
31
having to plan and imple-
ment a workshop in teams. After it the participant-trainers and the trainers’ team evalu-
ate the whole process. The evaluation includes a critical analysis of their performance
as trainers in the preparation and running of their workshops. This evaluation by perfor-
mance leads to a refection on the competencies required for being a trainer and how
to improve them.
31 Participant-trainer refers to the double role of participants in training for trainers. They are trainers and
in the frame of the course, at the same time, participants.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
25
Evaluation by outcomes
The concept of outcome refers to different “consequences” of the educational process.
In terms of educational evaluation, “outcomes” are mostly associated with fulflling the
objectives, learning achievements, organisational implications and the impact of the
activity for a wider social context.
Evaluation by outcomes gathers all these aspects. We have already described the advan-
tages and disadvantages of focusing the evaluation on each of them and we shall not
repeat those arguments now. In principle, considering all of them under the umbrella
notion of “outcome” allows for a greater richness of evaluation results and for a wider
perspective on the educational process evaluated.
But there are risks associated with educational evaluation that uses this, in principle,
integrative perspective. Due to the infuence of the commercial sector, there is a growing
tendency to evaluate educational programmes (and not only their management) by com-
paring incomes and outcomes using the purist logic of production.
This perspective might be a valid one when it comes to the management of non-formal
educational processes or to ensuring transparency in the use of public resources. But,
its application without mediation to the educational processes that constitute non-formal
education affects its most basic character. Using a simplistic income-outcome evaluation,
non-formal education can be reduced to a “social technology” or an instrument for “social
intervention” functioning like a machine: you put something in and you automatically
get something predetermined out.
One of the clearest visualisations of this danger was shown in the video “The Wall”
32
.
To the music of “Another brick in the wall”
33
, anonymous pupils enter a factory, forming
an enormous row on a conveyor belt. They fall into a grinder whose product is minced-
meat. This symbolises the homogenisation of individuals. The lyrics of the song are “We
don’t need no education, We don’t need no thought control…”
This is, of course, a caricature of formal education, but the “machine logic” can also be
found in non- formal education. The reductive misuse of “evaluation by outcomes” can
contribute to the reinforcement of this logic.
“Perhaps due to the preure upon projec organisers to prove the value o their work
to eernal bodie (particularly those fom wich the seek funding) there is a tendency
for disproportionate atention to be aforded to outcome evaluation…
…Also, it is likely that unprediced outcome, and certainly unprediced procee, will arise
over the course o a projec. Te fact that some outcome canno easily be evaluated, and
certainly cannot be quantified, should not diminish their importance. We consider it
imperative that educational aims are no reduced to those that can be measured, for the
sake o being able to prove wat has been achieved. An evaluation based only on preonceived
noions o outcome, thus, is unlikely to do full justice to any projec.”
34

32 Pink Floyd The Wall (1982) Directed by Alan Parker.
33 “Another brick in the wall”. Rock opera/concept album “The Wall”. Pink Floyd 1979.
34 “T-Kit No. 7 – Under construction...Citizenship, Youth and Europe”. Council of Europe and European
Commission (May 2003) Pages 57-58
Incomes
Non-formal
education!?
Outcomes
26
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Evaluation by process
Historically, this approach emerged a critical alternative to evaluation by objectives and
by achievements. Evaluation by process tries to overcome their limitations and it is based
on qualitative evaluation
35
.
In general terms, in evaluation by process, the starting point is the needs of the differ-
ent actors. From those needs expected achievements are formulated. The objectives are
fxed to “conduct” the educational process starting from the needs and moving towards
the expected achievements. In this case, the “results” would be the sum of most of the
elements of the educational process.
Evaluation by process is not just a matter of checking “how things are going”. This is
very common and at times intentioned simplifcation. Evaluation by process analyses the
relationship between the needs of participants, the expected outcomes, the objectives
and the results of a youth project. For example, evaluation by process would mean to
analyse the relation, connection, coherence and correspondence between:
the need of having leisure activities
the expected outcome of creating a cultural association
the objective of promoting co-operation and participation during leisure time
the result of having sport competitions and a small cyber centre
The advantages and disadvantages of this kind of evaluation are:
Advantages Disadvantages
Evaluation by process has a particu-
lar educational value because it does
not just look at the different stages
of an educational activity: it goes
through them.
Since this kind of evaluation takes
has its starting point the needs of
the different actors, it covers par-
ticularly well the functions of diag-
nosis, orientation and motivation.
It can help participants, trainers and
organisers to clarify their own chal-
lenges and to get to know the most
important obstacles in the educa-
tional process. Consequently, it helps
to develop a more solid and consis-
tent educational process.



There is little agreement and
conceptual clarity around the notion
of “process”.
It is diffcult to “capture” the complexity
and dynamism of the educational
process.
In reality it is difficult to make
compatible the “theory” and the
“praxis” of this kind of evaluation
due to its complexity
Being basically qualitative, there is
a bigger risk of the evaluation
becoming too partial




35 Qualitative evaluation looks at “qualities”: characteristics of the educational experience. It also looks at
their meaning for different actors. It raises questions like How? Why?, rather than how many or how much?
Quantitative evaluation aims to measure and count. It focuses on the “quantity” of the educational expe-
rience raising the questions like How many? How much? How often?...
More complete defnitions of qualitative and quantitative evaluation can be found in pages 33-34 in the
section called “Typologies of evaluation”.




www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
2I
The diffculty of evaluation by process is very obvious in training courses. The following
questions commonly arise:
How can the diverse needs of participants be related meaningfully with the objec-
tives of the course and with the programme at the beginning of the course?
How can the needs be checked in the middle of the course in light of the objec-
tives and considering the development of the group?
How can the needs be referred back to at the end of the course to see the level of
their achievement?
On many occasions evaluation by process is unfortunately reduced to picking out some
elements of the above here and there without embracing the full complexity of the dif-
ferent processes that participants experience during a course.
Our proposal: Educational Evaluation as a Total Experience
Given the complexity of the educational process the frst and most spontaneous answer
to the question “What to evaluate?” would be EVERYTHING. In this line, the editorial
team of this T-Kit understands educational evaluation as a total experience. This is our
proposal and our invitation.
Educational evaluation as a total experience:
is characterised by its global vision, its integrity and by the articulation of different
approaches, methods, theory and praxis
combines quantitative and qualitative approaches to evaluation
looks at the whole educational process (integrally and from different points of view)
prioritises the most valuable information, without falling into “doing everything”
The methodological proposal of this approach to educational evaluation is the combina-
tion of diverse spaces, actors, times, methods, sources and techniques of evaluation.
By doing so, the “conceptual” (learning, motivating…) and the “instrumental” (judging,
changing, deciding…) visions of evaluation become complementary rather than contra-
dictory.
The ethical and educational foundations of this kind of evaluation are:
Dialectic
36
unity
Dialectic unity describes when an idea, a conclusion, an explanation or a thesis and
its opposite are not exclusive: both are part of a whole. This happens very often.
For example, for some participants a session was too long and for others too short.
Educational evaluation as a total experience would not pretend to discern who is
right or wrong about that, but to understand, value, put in a certain perspective an
eventually challenge both conclusions.
Unity in diversity
The presence of diversity in educational activities is a fact. Very often it is one of
the most important sources of learning for the group. For the purpose of analysing
them, evaluation often puts the emphasis on separating and breaking down back-
grounds, opinions, experiences, conclusions, contexts, contents, organisational
frameworks and methods. However, and necessarily, this diversity should be respect-
ed and all of the aspects of the educational process have to be considered. This
does not contradict the idea of considering all of them part of a whole or of a
unity, the educational process itself.
36 Broadly defned, a dialectic is an exchange of propositions (theses) and counter-propositions (antitheses)
resulting in a synthesis of the opposing assertions, or at least a qualitative transformation in the direction
of the dialogue. Source: Wikipedia 2006. http://en.wikipedia.org









28
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Educational evaluation as a total experience aims not just to look at diversity and
to cover the different dimensions of the educational process, it tries to analyse the
connections, mutual infuences, coherence and correspondence between them. It
raises questions like: How does the organisational framework infuence the con-
tents? Does the profle of participants correspond to the activity format? Can any
trends be observed in the different experiences participants went through?
This approach constitutes a signifcant challenge. In non-formal education, the approach
is clearly still under development. It is diffcult to translate this approach into practice
because it requires effort to deal with complexity. And in practice, given limited time and
resources, it is necessary to prioritise among all the possible evaluation felds, among
different ways of collecting and processing information.
Our invitation to you is to discover and develop educational evaluation as a total expe-
rience. A total experience as it is integral to the non-formal educational process, which
the evaluation should serve.
All the previous “models” of evaluation and any other you may use or know of evaluat-
ing have their place and their role. We believe it is good to know them, to look at them
critically, and to create a basis for the development of our own evaluation approaches
and strategies.
“We propose that evaluation, as an element o practice, should be a reponsive
proce, implemented in a manner suited to the particular projec, and according
to the particular skill, taste and understanding o the actors involved.
Evaluation must also be founded in ehical practice and should refec the aims being sought
in its implementation. Perhaps even more important than the need for evaluation to addre
immediate and long-term outcome, is the need for ongoing evaluation o procee and
outcome to inform reponsive educational practice.
Te fact that unanticipated reults might be signifcant and that outcome are comple and
no always predicable or measurable, doe no save us fom the need to se goals and think
about wat we want to achieve fom the earliet stage o planning. Equally, the fact that
step by step procee canno guarantee succe doe no remove the need to consider wat
aproach you might adopt.
In particular it will be important to work out your value and ideas – the ehos or
philosophy o your aproach, if you like.”
37

Ingredients, recipes, previous experiences, tips… are, for a chef, just the basis for elabo-
rating, time and again, a new and suitable menu. We believe the same can be said for
effective educational evaluation.
37 “T-Kit No. 7 - Under construction...Citizenship, Youth and Europe”. Council of Europe and European
Commission (May 2003) Page 58
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
29
1.5 Who and for whom?
The actors of educational evaluation
Having reviewed the possible defnitions, aims, operational objectives and felds, it is
not diffcult to identify the actors of educational evaluation. The questions “who?” and
“for whom?” are deliberately formulated together because everybody involved in the
educational process should participate in its evaluation and should be informed about
its results, as follows:
The participants: as learners and the target group of the activity
The facilitators, leaders or team members: as responsible persons for animating
the activity
The organisers and partners: as promoters of the activity
The funders: as supporters of the activity
The decision-makers: as those “responsible” for considering the results of the evaluation
in further decision making processes.
These different actors have different responsibilities and tasks in the evaluation. They
should participate at different levels and in different phases of the evaluation. Nevertheless,
it is important to involve everybody. The evaluation should be democratic, transparent
and simultaneously accessible for all actors concerned.
Sometimes evaluation is conducted by an “external evaluator”. But, this does not play
against the idea of involving all the actors. Moreover, one of the tasks of the external
evaluator is to involve the different actors and facilitate their interaction during the eval-
uation process. A common assumption is that the external evaluator will be in a position
to do that with more easer and with fewer compromises than so-called “insiders”.





3U
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
1.6 When to evaluate?
The timing of educational evaluation
Evaluation is an ongoing and continuous process! However, “when” we evaluate is also
an important decision to make in the planning of the evaluation process. It is important
because it will give us an idea about how things are going at different and distinct points
and will allow us to use the information collected to make necessary changes to improve
during the educational process. Three main types of evaluation can be identifed according
to when they take place. These are initial, mid-term and fnal evaluation.
Imagine you are cooking! Before you start cooking, you place all the
different things you need for cooking on the work surface and you check
if you have everything you need, and more importantly, if you have all
the right ingredients. It may not be a good idea to notice half way
through cooking that you do not have the corn, or even worse, that
you bought fsh instead of chicken!!!
After you start cooking, you occasionally open the lid on the
pot to check the colour and the aroma, to make sure things are going
according to plan. Of course, you also occasionally taste the dish you
are cooking to check if you have added the right amount of salt or if
the pasta is cooked to perfection. And fnally, when you have proudly
served the meal you prepared to your friends or family, you naturally
ask them if they like it.
You have probably got the idea already: The above is simply a sequence of initial, mid-
term and fnal evaluations. Evaluating the process and outcomes at different stages gives
you more control over the big picture, and ensures that you do not end up with under
salted and under cooked chicken. Without even noticing it, you have used initial, mid-
term and fnal evaluations all the way through.
Initial evaluation is the evaluation done at the very beginning. Before we start our proj-
ect, whether it is a youth exchange or a training course, it is a good idea to take some
time out to check our aims, objectives and methodologies. Some questions you might
want to ask yourself at this stage could be:
Are the objectives of our project in line with our overall aim?
Do the methods chosen serve the fulflment of this aim?
Does the programme cover everything we want to address? Is it realistic?
Does our team have the necessary expertise and capacity to run this programme
or do we need to call in some support?
Your turn! Take fve minutes to write down a few questions you may want to ask yourself
during your initial evaluation and before you start the project.




www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
3I
Information about the participants is another important part of initial evaluation, and is
commonly called “profle information”. It can be understood as “an initial snapshot of
participants”. This information is vital for checking how much effect your programme will
have on the lives of young people. In this information there are data, which do not
change, for example the age of the participants. This type of information is useful for
checking, who on our programme is most likely to beneft. Profle information is crucial
for designing your programme in a way that fts the needs of participants.
The mid-term evaluation is our chance to check how things are going. As the name
implies, it takes place during your project. It may be an “ongoing evaluation”, such as
daily reviews at the end of each day, or a whole set of methods that take place at the
mid-point of the programme.
Mid-term evaluation may prove to be extremely useful for project teams, as it gives them
the possibility of identifying potential problems or shortcomings as the programme pro-
gresses. Team members may decide to make necessary changes in the programme, to
tackle these challenges, or meet additional needs that the participants may raise.
Once we complete our programme, it is time for the fnal evaluation. Using a variety of
methods, the participants and the team evaluate the whole project in light of the out-
comes (i.e. fulflment of objectives, learning achievements, organisational implications
and impact in a wider social context) with a certain distance and perspective.
32
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
1.7 Typologies of Evaluation
We have seen in the previous section different types of evaluation according to their timing
(initial, mid-term, and fnal). In this section, we will introduce other types of evaluation,
this time categorised according to their actors (personal, interpersonal, group), to their
functionality (formative, summative) and to their nature (quantitative, qualitative).
Personal/interpersonal/group
Personal evaluation is the kind of evaluation in which each individual or actor involved
in the educational process makes their own judgements and draws their own conclusions
about the experience they have had.
Interpersonal evaluation happens when more than one individual actor involved in the
educational process shares and discusses their judgements and conclusions. Often this
kind of evaluation takes place in a small group setting. While individual judgments may
change as a result interpersonal evaluation, arriving to a consensus is not the aim. The
purpose is simply to share and discuss those individual evaluations.
Group evaluation has an added dimension. It is not simply an interpersonal evaluation
in which the number of actors involved is bigger. Since the group as such is part of the
context and very often an important source of learning in non-formal education, group
evaluation specifcally at aspects and dimensions of the learning process that can be
observed and judged from a group point of view, including for example the atmosphere,
the co-operation among participants, the contribution of the group to the learning and
the group process. It should be noted however that this is not its exclusive function.
Formative/summative
“When the cook tastes the soup while cooking, that’s
formative; when the guests taste the already fnished
soup, that’s summative.”
38

Formative evaluation accompanies the learning process and can contribute to it. It consists
of continuous appreciation, ongoing analysis and drawing conclusions.
Summative evaluation looks at to the overall and fnal outcomes (e.g. the fulflment of
the objectives, learning achievements, organisational implications and impact in a wider
social context). In other words, it consists of the verifcation of the expected results and
drawing conclusions at the end of the process.
38 Robert Stakes quoted in “Evaluation and Education: A Quarter Century”. Chicago: University of Chicago
Press, 1991: p. 169. Robert Stakes is professor of education and director of CIRCE at the University of Illinois.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
33
Quantitative/qualitative
Quantitative evaluation focuses on the “quantity” of the experience. It aims to count or
measure different phenomena (literally). The major questions that quantitative evalua-
tion raises include “How many?”, “How much?” and “How often?”. For instance:
How many young people participated in the youth exchange?
How many countries were represented?
How often did they stay in touch with each other after the exchange?
Qualitative evaluation, on the other hand, relates to the quality of the programme and
of the experience. So, qualitative evaluation looks at the meaning of the experience for
different actors. This can be on an individual level or on a group level. For example, the
experiences of a given youth exchange for Arda from Turkey is considered along side the
experience of a group of youth workers involved in a complex training in the city where
Arda lives. The questions that qualitative evaluation usually raises are “How” and “Why”.
For instance:
Why did the participants of the contact-making seminar not create any follow up
projects?
How were the methods of work presented in the training course used by the partici-
pants once they went home?
Before we get into further detail of quantitative and qualitative evaluation, it is important
to note that solid and professional educational evaluation usually involves a combination
of both quantitative and qualitative evaluation methods. These provide different types
of information that enables you to have a larger and sharper image of the performance
of your project.
Exercise: Quantitative or Qualitative?
Read the following questions and try to fnd out if they are quantitative or qualita-
tive. Afterwards, discuss why.
1. How many young people have participated in the activity?
2. In 2005, how many volunteers participated in voluntary work camps organised
by your organisation?
3. We had 200 volunteers from Istanbul in our project! Why do we have much
fewer volunteers from the Eastern cities in Turkey?
4. We have had 200 volunteers in our project! How many of them went to Eastern
European countries?
5. Why are students and especially university students more likely to apply to
become volunteers?
6. How did our campaign on volunteering help to raise awareness?
7. Where is the best place to recruit volunteers?
8. How often do volunteers participate in voluntary work camps after their frst
experience?
9. Why do we work with volunteers exclusively?
10. Why do we have more female volunteers than male ones?
Answers: 1: Quantitative – 2: Quantitative – 3: Qualitative – 4: Quantitative –
5: Qualitative – 6: Both – 7: Both – 8: Quantitative – 9: Qualitative –
10: Qualitative.





34
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
The differences between quantitative and qualitative evaluation are not limited to the
type of questions. There are also other major differences:
Quantitative and Qualitative Evaluation: Basic differences:
39
QUANTITATIVE QUALITATIVE
1. Measurement 1. Explanation
2. Conclusions are based on
the analysis of data
2. Conclusions from observations
are dependent on interpretation
3. Replication of the process
is possible
3. The process is diffcult to replicate
4. Structured 4. Unstructured
The frst difference is that quantitative evaluation measures phenomena that appear in
our programme. Quantitative statements contain measurements such as “55% of par-
ticipants were from Western European countries”. Qualitative evaluation, on the other
hand, seeks to explain why the majority of the participants were from Western European
countries. One explanation might be that that participants from outside the European
Union did not have enough time to take apply for and receive their visa.
The second difference arises from issues of transparency. In quantitative evaluation, how
we reached the result on the number of participants that took part is quite clear – we
counted the number of participants. However, when using qualitative methods, we move
from observations to conclusions using interpretation something, which is considered
more subjective than objective.
The third distinction relates to replication. Quantitative methods can be replicated exactly,
so that each actor will have the same evaluation experience. On the other hand, qualitative
data depends much more on the context. Even something as simple as the time of day
the evaluation takes place can have an impact on the results.
Finally, the fourth distinction: Qualitative evaluation is said to be unstructured. The focus
of the evaluation is on the importance of the different actors in the process having the
opportunity to express their opinions on the activity. However, quantitative evaluation is far
more dependent on a set structure, with set information sought from each respondent
having been decided in advance by the evaluators.
Another level of difference between quantitative and qualitative evaluation is therefore
the methods we use, although, as with the exercise we did identifying qualitative and
quantitative evaluation questions, the line between the two can be blurred.
The following table shows different methods of qualitative and quantitative evaluations:
QUALITATIVE QUANTITATIVE
UNSTRUCTURED SEMI STRUCTURED STRUCTURED
Observation of participants Structured interviews Surveys
Letter to oneself Observation with guidelines Questionnaires
Focus groups Spot checks Checklist observations
Diaries
Film/Video
In-depth interview
39 Kneale, Dylan (2004), Un-published presentation on Evaluation, Youth Express Network seminar on
Evaluation, Avanos: Turkey.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
35
1.8 Educational evaluation
in a broader framework
Yeast and a pizza in Italy!
If as a foreigner in Italy you would like to prepare a pizza, you might
run into a few problems. As soon as you arrive in the “supermercato”
to buy yeast to prepare the dough you could easily take the wrong one
and end up with a pizza with a sweet cake-like base. That is because in
Italy, there are several kinds of yeast. But, as a foreigner you probably
would not know that. Anyway, baking a pizza could bring
you into deep discussions about different perceptions of pizza with
your Italian friends. That’s because the question of pizza is not only
about understanding what “yeast” is, it’s about what should be on it,
about when to eat it, about regional differences, about the history of
the pizza, about what is good tomato-sauce, what is the right cheese,
which herbs to use… it is about the whole framework.
At the time of writing, the term “evaluation” is frequently raised in discussions of non-
formal education, together with terms such as “assessment”, “validation”, “accreditation”,
and “qualifcation”. All this is part of the ongoing discussion about the recognition of
non-formal education. There seems to be an agreement that people learn a lot in non-
formal education. But what does learn a lot actually mean? What exactly do people learn?
How can we measure that? Does it have any value on the labour market?
What can the average 17-year old girl do with her three years of experience of working
as a volunteer in the local youth association? Would it not be useful if her experiences
were noted on paper, so that when she has an interview for a job she could show her
competencies? Of course, that would be great. But, what does it prove? Did she really
learn from those experiences and how good is she at the things she claims she is able
to do? How can non-formal education be more precise in describing what people learn
and the extent to which they learn it?
One of the things that makes non-formal education different from formal education is that
in non-formal education assessment of participants does not take place. In non-formal
education we do not organise exams and we do not give marks or grades. We evaluate
the programme, the process, the outcomes, but we do not assess the individual level of
participants. We rather ask participants what they think they have learned.
Still, it could help the 17 year-old girl to gain recognition as a competent person for that
job if she had a precise description of what she learned and on what level. This would
also mean that her youth association is accredited as an organisation that meets certain
standards, allowing it to give her the paper that describes her competencies and ensuring
that the paper is recognised as having a certain value.
36
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
A large debate exists as to whether the role of non-formal education should also be to
provide certifcation and diplomas for the learning that takes place during its educational
processes. It is a controversial issue, as actors concerned disagree about whether this
is the way non-formal education should evolve.
Supporters of “recognition” say that lots of young people would beneft from the certi-
fcation of non-formal education. This would broaden their options in their transitions
to the labour market and to an autonomous life.
Opponents, on the other hand say th at non-formal education would lose its specifc
nature and character if it would provide certifcation. The inherent role, social importance
and civic mission of non-formal education could be threatened by the movement towards
certifcation. It would completely change the relationships between youth leaders, youth
workers and youth trainers and to their target groups. Their role as “facilitators” would
be conditioned by a power logic, as they would have to assess participants in their pro-
grammes.
Discussions around assessment, certifcation and accreditation of non-formal education
are provoking strong educational and ethical debates at very different levels. One of the
key criticisms of the movement towards certifcation is that such approaches can be used
as tools for selection and exclusion, risking the restriction of equal access to knowledge
and having negative consequences for the nature, curriculum and practice of non-formal
education.
If we understand non-formal education as a democratic process of access to processes
of self- development, any selective misuse of certifcation would be more than question-
able, would raise ethical issues of the most fundamental nature and would put into
question the legitimacy of non-formal education and its actors.
One signifcant emerging “outcome” of those debates and an alternative to the so far
“black and white” nature of the debate is the concept of self-assessment. The idea behind
this form of assessment is to provide participants with adequate tools and support for
assessing their own learning and for documenting its results. For example the portfolio:
a kind of folder bringing together all materials that prove a person’s efforts in non-formal
education and their learning achievements.
In recent years some self-assessment tools for non-formal education have been devel-
oped on the European level. The Council of Europe has developed a self-evaluation
matrix, which helps you to assess your language competencies
40
. In the youth sector of
the Council of Europe, a group of experts has developed the European Portfolio for Youth
Leaders and Youth Workers
41
.
40 This self-evaluation matrix is part of the “European Framework of Reference for Languages: Learning,
Teaching, and Assessment”. This framework is a tool for setting standards in an internationally comparable
manner. It facilitates a clear defnition of teaching and learning objectives and methods and provides the
necessary tools for assessment of profciency. It can be accessed at http://www.coe.int/T/E/Cultural_
Cooperation/education/Languages/Language_Policy/Common_Framework_of_Reference
41 The European Portfolio for youth leaders and youth workers is a tool which has been designed to enable
youth leaders and youth workers in Europe to assess and describe their competences, on the basis of a
core set of European quality standards. It can be accessed at http://www.coe.int/youthportfolio
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
3I
The words explained
“Evaluation: In English, evaluation only means to make a reasoned judgement
about or to give a plausible account of something. It does not imply any specifc
purpose (such as grading individual performance), nor does it imply any par-
ticular method of evaluation (such as a written test), and nor does its outcomes
automatically suggest that something is of greater value or importance than
something else (such as Council of Europe activities in comparison with SALTO
activities).
Assessment takes place when evaluation has a comparative dimension that
involves setting individuals, activities or institutions into a ranking order of per-
formance or achievement. The ranking may be set in relation to criteria that are
specifc to the context, process or outcomes that are being assessed (such as:
who swam the river fastest, or which EVS
42
agency has the highest success rate
in attracting socially disadvantaged young people into the programme).
Alternatively, relative performance may be assesses against an external standard
(such as in the case of the PISA
43
attainment tests for 15-year-olds in different
countries).
Certifcation refers to a standardised process of formally validating knowledge,
know-how, skills and/or competencies acquired by an individual or represented
through a learning/service provider.
Certifcates or diplomas are the “pieces of paper” which record the outcome of the
certifcation process. It most frequently has the status of an offcial document,
but this is not an absolute prerequisite.
Accreditation: formally or socially recognised authorities or instances (university,
department of education…) accredit courses, activities and their outcomes. This
means they testify that organisations and individuals meet standards to which
all have agreed to conform. They vouch for the credibility of the certifcates and
diplomas that are issued, and hence for the reliability and validity of the moni-
toring, evaluation and assessment of the individuals and the organisations whose
judgements are given the stamp of approval.”
44
“Feedback: giving feedback is to pass on to a person the effect of his/her
behaviour for their use and learning. Feedback is supposed to be helpful to
the person receiving it.”
45

42 The European Voluntary Service (EVS) is a part of the Youth in Action programme of the European Union,
that promotes the mobility and voluntary involvement of young people.
More information at http://eacea.ec.europa.eu/youth/programme/index_en.htm#structure
43 The Programme for International Student Assessment (PISA) is an internationally standardised assessment
that was jointly developed by participating countries and administered to15-year-olds in schools. The sur-
vey was implemented in 43 countries in the frst assessment in 2000, in 41 countries in the second assess-
ment in 2003 and at least 58 countries will participate in the third assessment in 2006. More information
at http://www.pisa.oecd.org
44 Taken from the “Terminology Cheat Sheet” (by Lynne Chisholm Pages 44-46), part of the report of ‘Bridges
for Recognition 2005’. This report is accessible at http://www.salto-youth.net/bridgesReport/
45 Defnition adapted from the report of the Training Module II: Responding Effectively to Cultural Insensitivity
of the Multicultural Leadership Institute, 2004.
38
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
1.9 Educational evaluation
and quality in youth work
Why talk about quality?
Educational evaluation and quality are very closely related. Evaluation can be considered
part of quality. And, vice versa: quality can be part of educational evaluation. What does
this mean?
Educational evaluation as part of quality: Educational evaluation can be considered as
a tool or mechanism for achieving quality. In this case quality would be the aim and
evaluation the instrument.
Quality
Educational Evaluation
Quality as part of educational evaluation. Educational evaluation involves making a value
judgement on what is evaluated. This “giving a value” would be equivalent to “attaching
a certain quality” to what is evaluated. In this case quality would be a layer, a reference
at the service of the evaluation process.
Educational Evaluation
Quality
As we can already see, the intimate relationship between educational evaluation and
quality is dialectical and uneasy, if one tries not to conceive one of them as a mere
instrument of the other.
In the debate on Educational Evaluation and Quality there are two poles:
On the one hand, to reduce educational evaluation to a mere tool at the service of
achieving quality would, we believe, imply not to respect the educational nature,
aims and potentials of evaluation. Even if “improving” is one of its objectives, edu-
cational evaluation is much more than a tool for improvement;
On the other hand, to consider “quality” as just another “element” of the educa-
tional evaluation process would, we believe, imply not facing the quality challenge.
Indeed, nowadays, the social demand for quality is growing in a lot of felds (indus-
trial products, social services, public services, etc) including non-formal education.
Funding bodies, partners, organisers, trainers and participants work for and expect
a growing level of quality in educational activities.


www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
39
With these “poles” in mind we can affrm that in the relationship between quality and
educational evaluation there is a plenty of potential for challenging pre-defned boundaries
and, as a result, for mutual enrichment.
Educational
Evaluation
Quality
“Understandings” of quality
What is quality? There are many different conceptions of quality:
One coming from the private sector would be: “Quality is: if the client comes back
to you”, or “quality is: satisfying the needs of the customer”. Even if we do have
a generous and fexible interpretation of this commercial understanding of quality
(identifying customers as participants, providers as organisers…) we can conclude
that this understanding is rather limited for the educational feld. The educational
phenomenon and the relations between its actors (participants, trainers, organisers…)
go beyond this “commercial transaction logic”.
This approach certainly has its place when it comes to the management of pro-
grammes or to the use of public resources. But, reducing the educational fact, or
experience to being evaluated, to a client-provider relationship is, in the best case,
a simplifcation.
The European Committee for Standardisation
46
agreed on the following defnition
in the non-governmental and business felds, published as a European standard EN
ISO 9000
47
: “Quality is a degree to which a set of inherent characteristics fulfls
requirements. The term quality can be used with adjectives such as poor, good or
excellent”
48
.
In this defnition quality is related to inherent characteristics, to requirements (set
or expected from outside) and to a subjective value judgement expressed in terms
of adjectives. This “triangulation” is very relevant to explore the notion of quality.
The quality of a coffee would be related at frst to the fact
of being coffee: to all the inherent characteristics of every
coffee (taste, aroma…). Then it would be related with the
expected requirements of coffee (strong, weak…). And fnally,
according to those and other “factors” (i.e. comparisons
or other experiences) we would conclude using some adjec-
tives: it is a coffee of good quality, of poor quality… The same
as we have said for a coffee we could say, in our feld, for
an exchange or for a training activity or for a workshop…
46 CEN, the European Committee for Standardization, was founded in 1961 by the European national standards
bodies. Now CEN is contributing to the objectives of the European Union with voluntary technical standards.
More information at http://www.cenorm.be/cenorm/
47 ISO (International Organization for Standardization) is the world’s largest developer of standards useful
to industrial and business organisations of all types, to governments, to suppliers and customers of products
and services in both public and private sectors… More information at http://www.iso.org
48 DIN-Taschenbuch, 2001, Normen zum Qualitätsmanagement, Beuth-Verlag, Berlin Wien Zürich, Page 451


4U
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Despite the large number of references and the growing body of literature about
quality in education
49
(quality control, auditing, valorisation, policies and public
funding provisions...), the concept of quality is still not very clearly articulated.
With this background, Harvey and Green (1993), have identifed and analysed four different
dimensions of quality, which are, in our opinion, a substantial contribution for the
understanding of quality in non formal education.
The dimensions of quality identifed by those authors are:
Quality as “fdelity” to the aims.
Quality as coherence - ethos.
Quality as transformation (qualitative change).
Quality related to something new.
They see quality in non-formal education as a mix of those four dimensions.
We do not consider this “defnition” as the “right one”. But, from our point of view, these
four dimensions in the understanding of quality are particularly pertinent, inspiring and
challenging in our educational feld.
The Quality Management Cycle and youth work
Dr. William Edwards Deming
50
(1900 – 1993), known as the father of quality management
developed the so-called P-D-C-A-Cycle
51
.
The quality management cycle
Every project is like a circle with four parts or phases:
Planning (P)
Acting (A) Doing (D)
Checking/Studying (C)
Planning: First we need an idea for the project. We think about aims and poten-
tialities of it. Then we work out a concrete concept and work plan.
Doing: We start to transfer our plan to reality. We follow our plan as closely
as possible.
Checking/Studying: While working we observe smaller or bigger gaps between our plan
and the results. We fgure out what is going fne and what needs to
be changed.
Acting: We take action on our critical review: changing, correcting, adapting…
There is a chronological order to the cycle. Each part builds upon the last. It is not possible
to change the direction of the cycle or to turn the system around.
49 Two interesting books for further exploration are: “Total Quality Management in Education” Edward Sallis
Publisher: Routledge Falmer 2002 , “Quality in Education: An Implementation Handbook” Jerome S. Arcaro,
Jerry Arcaro Publisher: St. Lucie Press, 1995
50 American statistician and quality-control expert, b. Sioux City, Iowa. Deming used statistics to examine
industrial production processes for faws and believed that improving product quality depended on
increased management-labour cooperation as well as improved design and production processes.
51 Source: http://www.valuebasedmanagement.net/methods_demingcycle.html. Value Based Management is
a portal with specifc information on value creation and managing for value and valuation.





www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
4I
The cycle theory in youth work
Demings’ theory was a starting point for developing different models of quality manage-
ment system for different felds (industry, public administration, the primary sector…)
In later adaptations, the circle developed into a spiral to underline the ongoing charac-
ter of a project. As we know every project has its own dynamic and there is never a real
beginning and never a complete end, because the concrete end is often already the frst
step into a “new” project.
Considering that, youth work is mainly structured in projects. Deming’s theory has had
an important impact on this feld, too. This understanding of youth work as project cycles
has led institutions and organisations in the feld to try to improve the quality of the
projects they support all through their different stages. In order to that they identify three
key concepts for quality management
52
:
Quality criteria: characteristics chosen to defne quality in coherence with the objec-
tives of the project. The assumption is that if a criterion is fulflled, the quality is
good and if it is not fulflled not so good or bad. Quality criteria are relevant and
valid.
An example of quality criteria in youth work could be:
Involvement of participants
– “Involvement of participants” is a characteristic of an educational project
– It is coherent with the objectives of the project. Youth projects in general have as
objectives the promotion of active participation, engagement, co-responsibility
and the use of active methodologies
– The assumption is that if there is involvement of participants it is good and if
there is no involvement it is bad
– “Involvement of participants” is a relevant and valid criterion
Quality standards: These are conditions set to criteria. Standards are often expressed
as the highest or lowest value or allowed range. They defne the expected amount,
the intensity and the manner of the criteria
An example of a quality standard in youth work for the quality criteria “Involvement
of participants” could be:
Participants self-manage and organise their projects and the free time
– “The self management and organisation of their projects and free time” is a
condition, an amount, a manner of looking at the criteria “involvement of par-
ticipants”
– There are other criteria to evaluate the involvement of participants but this stan-
dard (about their project and the free time) is what is expected.
Quality indicators: Most of the time we are not able to directly observe a criterion.
Therefore, if we wish to operationalise the criteria, we need indicators. Indicators
are measures that indicate if expected quality described in the standard is achieved.
Indicators can be qualitative or quantitative and should be reliable and measurable.
52 Sources: BMFSFJ: QS-Compendium, Brochure No. 24, p.75 ff, Bonn, 2001 and “A resource book for social
managers” by Paul Nies and Philip C Berman 2004, European Management Association.



42
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Following our example, some indicators (quantitative and qualitative) for the standard
“Participants self-manage and organise their projects and the free time” could be:
Number and kind of projects
Project team composition
Number and kind of self-organised activities during the free time
Social committee’s way of working
Sharing responsibilities

Example of quality criteria, standards and indicators
In addition to the example given while defning quality criteria, standards and indi-
cators, let us imagine a youth exchange project with the objective of promoting
mutual understanding and co-operation between participants…
A quality criterion could be:
Communication between the participants
A quality standard could be:
Communication in the project happens:
– Between all participants of the project
– All through the different phases of the project
– In the different activities and moments of the day
Some quality indicators could be:
– Interaction between participants during the free and informal time
– Direct communication/interaction of participants in the “working spaces”
– Initiatives to overcome communication barriers (i.e. language, etc)
– Level of understanding / misunderstanding in case of disagreement or confict
– The use of non verbal communication










Quality
indicators
Quality
criteria
Quality
standards
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
43
Here we have developed just one example. For each objective of our project it is necessary
to develop specifc quality criteria, standards and indicators. All those together constitute
the “quality catalogue” of our project.
Every project is unique so we rather create our quality catalogue with the team, with the
group and eventually maybe with the support of an external evaluator, for every new
project.
This catalogue of criteria, standards and indicators is useful in two ways. First, it defnes
what quality is and means in our project and second it is a reference for our educational
evaluation. A further example of such a process is the “Tool for developing your own
evaluation plan - SALTO Check list” (Pages 105-112).
Ideas about quality management enrich strategies of educational evaluation bringing
new perspectives to evaluation by achievements and by objectives. As we saw in the
section called “What to evaluate?” (Pages 19-28), evaluation by achievements and evalu-
ation by objectives are central in educational evaluation. But, without the complementary
information provided by other approaches (and, especially without qualitative information
about the process) they cannot embrace the complexity of an educational experience.
The global approach to educational evaluation as a total experience, as we propose in
this T-Kit, benefts from, but is not limited to, quality management.
Quality in youth work:
steps and debates in the European institutions
Quality in youth work at European level
What do we consider “good” or “bad” in our work in educational youth projects?
Establishing evaluation criteria is an essential and sensitive part of the evaluation pro-
cess of an educational activity. When touching on this and other related questions we
enter into the complex debate on the quality of non-formal education in Europe. Further,
in setting up criteria for evaluating educational activities we inevitably enter the feld of
the quality of non-formal education and youth work.
The quality of non-formal education and youth work is an ongoing discussion among
researchers, trainers and youth workers in Europe, which has been stimulated by the
Council of Europe and by the European Commission. For both institutions, it is becom-
ing a crucial question especially in the efforts of these institutions to develop the social
and political validation and recognition of non-formal education, thus of youth work as
well.
To better position non-formal education and learning in a wider socio, political and eco-
nomic context, it is indispensable to introduce specifc quality criteria. Such criteria apply
to organisers, youth workers and trainers and their performance, locations chosen, dis-
semination of the learning offer made and success, preparation of participants, cost
effectiveness, coherence, evaluation and the links to other experiences in education and
learning with regard to personal development, social inclusion, public and civic life or
the labour market.
Quality is also about relevance – with regard to life skills, cognitive learning and under-
standing, living in groups and communities. At European level it includes intercultural
competencies such as communicating in foreign languages, respecting difference, dis-
covering universal values, living diversity and developing tolerance of ambiguity. Quality
is indeed about ensuring reliability and validity.
44
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
European level initiatives on quality in the youth feld
Both, the European Commission and the Council of Europe put a strong emphasis on
quality assurance and quality development in their policies and activities. The issue has
been addressed in many training for trainers courses, for instance in the “Advanced
Training for Trainers in Europe” with very valuable approaches and results. Important work
on the topic has also been done within the frameworks of the Youth Programme of the
European Commission, the network of National Agencies and the SALTO Resource Centres.
Especially the Partnership between the two institutions (European Commission and the
Council of Europe) plays an important role in the feld of quality development in youth
work and non-formal education. It has organised several activities (expert meetings,
seminars, trainings) to stimulate the widest possible consultation on this issue, amongst
the establishment of an expert group on “Quality Standards, Evaluation and Validation”.
While this debate has not yet been concluded it has produced several fndings and ques-
tions for further debate. Clearly this issue is controversial; it opens up our very deep
concerns about the values and impact of youth work, our role as youth workers and the
mission in the profession. These issues are seen slightly differently in different countries,
from different cultural perspectives, across regions and by a variety of individuals.
There is a growing body of working papers, reports and educational materials focusing
on different aspects of quality, which can help us understand the nature, the importance
and beauty of this subject. The following documents are particularly interesting because
they deal with quality and the related topics of validation, recognition, self assessment,
quality standards in the educational youth work.
The long term training course Advanced Training for Trainers in Europe (ATTE) has
been developed to meet the increasing need for qualifed trainers in this feld and
to enlarge and further develop the European networks of trainers, who have the
competence and the motivation to develop and implement European level training
activities in the youth feld. Through its new and innovative approach it was an
important step towards ensuring the quality of training of youth workers and youth
leaders at European level and towards the recognition and certifcation of training
for trainers in the feld of non-formal education.
ATTE was seen as a pilot course, therefore an appropriate evaluation of its effec-
tiveness with respect to individual learning, group learning and institutional invest-
ment was seen as necessary from the very beginning. The evaluation involved
various partners, including a limited number of experts, the curriculum developers
of the course, the European Youth Forum
53
, the National Agencies for the YOUTH
Programme as well as the Course Team and participant trainers. The Evaluation
Report of the Advanced Training for Trainers in Europe
54
is therefore of remarkable
relevance.
“ATTE did not make great progress in establishing quality criteria, but it has suc-
ceeded in raising participants’ awareness of the need for quality criteria and quality
monitoring. Whether to assess the quality of learning outcomes and then how to
do it, proved to be ATTE’s thorniest challenge. Participants were prone to think that
implicit assessment of their qualities as trainers was taking place continuously. At
the outset, tutors generally held the view that assessment of any kind has no place
in non-formal learning, but they modifed their views as the course proceeded.
Transparency is the key to resolving such problems, which are by no means exclusive
to the non-formal youth training sector.
ATTE ultimately decided to use self-assessment procedures, complemented and
challenged by feedback from peers, the course team and external experts. Practice 2
53 The European Youth Forum is made up of more than 90 National Youth Councils and International Non-
Governmental Youth Organisations, which are federations of youth organisations in themselves. It brings
together tens of millions of young people from all over Europe, organised in order to represent their common
interests. More information at http://www.youthforum.org
54 Accessible at http://www.training-youth.net/INTEGRATION/TY/TCourses/olc_atte/atte_evaluation.html

www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
45
projects, a training quality product (TQP), a portfolio and participation in the ATTE
seminars themselves would furnish the evidence. Half the participants met the core
expectation to produce a TQP by the fnal seminar, and many who did not regret-
ted that they had not done so, given the value of the feedback sessions.”
55
With regard to quality standards in education and training activities of the Directorate
of Youth and Sport of the Council of Europe the development of quality criteria
56

aims at contributing to the deepening and widening of transparency, accountability,
reproducibility, sustainability, quality and innovation of activities so that the Council
of Europe can remain a trend-setter and high quality non-formal education provider
in the youth feld.
According to the key document on this subject quality of the education and training
activities is assured by and verifable through a range of criteria: a relevant needs
assessment, clear, concrete and assessable objectives, the defnition of compe-
tences addressed and learning outcomes for the participants, the relevance to the
Council of Europe programme and youth policy objectives, an adequate and timely
preparation process, a competent team of trainers, an integrated approach to inter-
cultural learning, recruitment and selection of participants, a consistent practice of
non-formal education principles and approaches, adequate, accessible and timely
documentation, a thorough and open process of evaluation, structurally optimal
working conditions and environment, adequate institutional support and an inte-
grated follow-up within the DYS programme and its partner organisations, visibility,
innovation and research.
In February 2004 the Partnership has produced a working paper on “Pathways
towards validation and recognition of education, training and learning in the youth
feld”
57
It summarizes the state of affairs and highlights a strong need for social
and formal recognition of non-formal and informal learning in youth activities. The
paper was a milestone in the discussion on non-formal learning and led a number
of political initiatives.
As a result of a research seminar on non-formal learning which was organised in
the framework of the Partnership Programme in April 2004 a more precise picture
of the impact of youth work and of the skills and competences of those active in
the youth feld was achieved
58
.
The Bridges for Recognition conference in January 2005 in Leuven, Belgium, was
aimed at increasing the visibility of the value of youth work and discussed ways to
a better recognition, thus laying the foundations for further action.
As a follow-up the SALTO Resource Centre for Training in Germany got the mandate
to develop a specifc “YOUTH-PASS”, a European level validation instrument spe-
cifcally for activities in the framework of the YOUTH Programme.
Based on a decision of the political bodies at the Council of Europe the organisation
convened an expert group to produce the European Portfolio for youth leaders and
youth workers. The Portfolio has been completed and will be available – after a
testing period – in early 2007. The test version is available at the following website:
www.coe.int/youthportfolio. This tool includes methods of self-assessment enriched
through external feedback, functional analysis, competency tables and levels of assess-
ment, suggested ways to demonstrate evidence of experience and a glossary.
The political debates in the European Union coordinated by the European Commission
led to the Council Resolution on the Recognition of the value of non-formal and
55 Advanced Training for Trainers in Europe. Volume 2 - External evaluation. (2006) Authors: Lynne Chisholm
with Bryony Hoskins, Marianne Søgaard Sorensen, Lejf Moos, Ib Jensen. Page 12.
Available at http://www.training-youth.net/INTEGRATION/TY/TCourses/olc_atte/atte_course_pub_vol2.html
56 Quality standards in education and training activities of the Directorate of Youth and Sports of the Council
of Europe, March 2005, DJS/ET (2005)1
57 Accessible at http://documents.youth-knowledge.net/documents/39.pdf
58 Trading up – potential and performances in non-formal learning, edited by Lynne Chisholm, Bryony Hoskins
with Christian Glahn, Council of Europe, August 2005







46
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
informal learning within the European youth feld, adopted under the Austrian
Presidency in May 2006.
There are several other documents, which can help us in understanding the different
dimensions of quality. For this we would like to recommend a visit to the website
of the Partnership at http://www.training-youth.net/INTEGRATION/EKC/Intro/index.
html where downloads of a wide range of information are available.
Conclusion:
Quality in European youth training and non-formal education has increasingly become a
concern for the stakeholders and actors involved:
for participants in training and non-formal education activities, who want a quality
learning offer;
for trainers, organisers and organisations, who want recognition of the quality of
their work in the feld of non-formal education and training;
for sponsors and public authorities, who have an interest in an effective use of the
funds and the support they provide in this feld.
Linked to the interest in the quality of non-formal education and training there is the
interest in recognition of the non-formal education sector as a whole and in particular
of the offers in this feld and of those who offer it – trainers, organisers, etc.
Therefore, sustaining and further developing quality in European youth training and non-
formal education is one of the objectives of the Partnership Programme on European
Youth Worker Training run by the Council of Europe and the European Commission.
Evaluation is an indispensable element of the process.




www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
4I
1.10 Develop your own evaluation!
Experienced evaluators agree that there are no “golden rules” for the development of a
perfect educational evaluation. We believe that the defnitions, models and questions
explored in this “theoretical” part of the T-Kit (“Ingredients of Educational Evaluation”)
can help a lot but they cannot guarantee an evaluation process free of resistance and
mishaps. The most important is to remember that, if the educational evaluation is not
to undermine the objectives of the project, then, it must actively support them.
Be creative and courageous! Just give it a try! In the next more “practical” part of the T-Kit
(“Pots, Pans and Spices”) we will present you with some tools, methods and instruments
that will support the development of your own educational evaluations and for integrating
educational evaluation into your projects.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
49
2. “Pots, pans and spices”
Practicing Educational Evaluation! – Educational Evaluation in Practice!
2.1 How educational evaluation
is built into a project? …
While it might already be clear, starting to prepare the evaluation
the night before the last day of your project is too late.
Evaluation needs to be planned before you start your project.
In other words, planning the evaluation is an integral part of planning
the project as a whole.
The main questions to ask ourselves when we begin are:
Why evaluate? (what are the reasons for doing evaluation in our project?)
What for? (what are the objectives of evaluation in our project?)
What should be evaluated? (What are the evaluation felds and elements?)
How and when should the evaluation be done? (What are the methods, sources for
the evaluation?)
Who will do it and for whom will it be done? (Who are the actors of the evaluation?)
As we have seen, the aims and objectives of your project are an important dimension
of what to evaluate. That means that these aims and objectives should be formulated
in such a way that they can be evaluated. There should be agreement about what is
meant by an objective.
“Improving the intercultural competence of the participants” can be an objective. But
what do we mean by that? Can we describe what kind of behaviour we see as improved
intercultural competence? What are the indicators that demonstrate that participants are
more competent in intercultural situations than they were before the project started?
To set up a proper evaluation, it is crucial to be clear about our aims and objectives.
Still, there might be other elements of the project that we want to evaluate, which are
not mentioned in the aims and objectives, for example the process. We planned the
desired process into the programme of our activity, but how did it work out in reality?
Or if we use our cookery metaphor: the meal turned out fne, but the struggle that took
place in the kitchen to get to the result was awful and needs to be improved.
The way we work in our team will probably not be mentioned in the aims and objectives.
But, since the team carries the educational responsibility, it is certainly important for all
concerned to evaluate its work. The same could be said for the co-operation with other
partners and stake-holders as well as for the way the different resources such as accom-
modation, environment, transport and materials were dealt with. The way in which they
contributed to the outcomes of the project can be evaluated.
We need to be clear before starting the project about what we want to evaluate. When
we have a common understanding about that, it is possible to decide how we will do
the evaluation and when.





5U
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
A lot of questions are implied by this: What kind of information do we need to be able
to do our evaluation? If we return to the example of “improving intercultural competence”,
this might mean that we should fnd out how inter-culturally competent the participants
were at the moment they started to participate in the project. This means we have to
start collecting information before the start of the project. And, this means we have to
make up our minds about how to collect that information. Will we use questionnaires or
interviews? Will we collect information from individuals or in small groups?
Another question is whether we want a moment during the project to check if we are on
the right track and how the project is developing? What information do we need for that?
How should we collect that information? Who will do that?
Have we achieved an adequate balance among the different kinds of methods we plan
to use for the evaluation? Are the diverse preferences and needs of participants met by
the methods we have planned?
What kind of reporting do we need to produce at the end of our project? For whom is this
reporting intended? What are the implication of the target audience for the content, the
style and the image of the report? Will it be a written report? Will we produce a CD-Rom?
So as you can see, even before the project begins, there are plenty of things to consider,
especially when planning the evaluation.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
5I
2.2 … A project within a project
How can educational evaluation be built into our project? It helps us to understand how
by thinking of evaluation as a project within the project. We will use the following sections
to explore this idea.
The three “traditional” steps of a project were defned as: planning, implementing and
evaluating
59
. However, this way of conceptualising youth projects can limit educational
evaluation “mostly” to the fnal stages of a project.
If we limited it in this way, the potential of educational evaluation be can reduced and
its nature can be subverted. This can even become counter-productive. An evaluation
only at the end quite often implies a reduction of the educational value to the minimum
and it can result in problematic conclusions.
The idea of educational evaluation presented in this T-Kit implies a continuous and ongoing
process, a kind of responsive practice, maintaining fexibility and responding to individuals,
to the context and to new situations as they arise.
When practice and evaluation go hand in hand, it is diffcult to draw a clear distinction
between educational evaluation and educational practice. The ongoing refection that
accompanies the practice brings continual adjustment and modifcation. This responsive
working practice becomes itself part of the evaluation process.
Educational evaluation should not be regarded as something to be tacked on to the end
of the project. Educational evaluation should happen all through the different stages of
a project, from the preparation, until the fnal implementation.
Consequently we should follow the stages (preparation, design and implementation)
so that we can build an evaluation plan or strategy: our evaluation project within our
project.
In the following diagram we can see a simplifed description of the different stages of
a project
60
and vertically, through all of them, ongoing educational evaluation. The
different stages of educational evaluation will be disclosed, through guiding questions,
in the following sections.
59 See Pages 43-44 T-Kit on Project Management accessible for download at
http://www.training-youth.net/INTEGRATION/TY/Publications/T_Kits.html
60 For more details on project planning and its stages see T-Kit on Training Essentials Pages 65-74 and T-Kit
on Project Management Page 43. Accessible for downloads at
http://www.training-youth.net/INTEGRATION/TY/Publications/T_Kits.html
52
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
The Poject
Educational evaluation within a project
Educational Evaluation
Preparing the Project:
Needs analysis
Personal motivations
Aims of the organisation
Defning the aims of the project




Designing the Project:
Concrete objectives
Profle of participants
Resources
Contents
Methodology
Learning outcomes
Detailed programme
Methods








Implementing the Project:
Following the plans
Adapting
Taking care of the fow
Involving participants
Using the space, environment
and resources
Material, documentation, experts






Preparing
the evaluation
Designing
the evaluation
Collecting and organising
the information
Interpreting
the fndings
Drawing
conclusions
Reporting
the outcomes
Implementing
the results
Ongoing
Educational
Evaluation
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
53
Together with all the theoretical background already explored, the following questions
and considerations can be of a great help for designing, integrating and implementing
educational evaluation in our projects.
As a refection of the approach proposed in this T-Kit, the following questions and refections
at different stages try to be a guide for “educational evaluation as a total experience”
within a project. Based on our experiences, we propose to combine different evaluation
approaches (by objectives, by achievements, by outcomes, etc) and to diversify spaces,
actors, times, methods, sources and techniques of evaluation. At the same time we try
to fx some priorities so that we do not fall into the unrealistic pretension of trying to
deal with everything at once.
We encourage you to adapt and modify the following guide to the stages of educational
evaluation to account for the specifcities of your project.
Educational evaluation in stages
Stages of educational evaluation
Preparing the evaluation
In this frst stage we defne the aims, nature and general approach of our evaluation
Why evaluate and what for?
– Defning the aims and concrete objectives of our evaluation.
– Adapting and making concrete the general objectives of evaluation (to learn,
to participate, to improv e, and to motivate).
Who will evaluate?
– Who should evaluate and who should be involved / participate? (Young people,
trainers, organisers, stakeholders, partners)?
– In principle we could think that all actors involved in the project but in which
combination and at what different levels of the evaluation should they be
involved?
– Who is competent to conduct our evaluation in terms of the required skills
and knowledge?
– Who has the (political) mandate to conduct the evaluation?


54
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work

Collecting and organising the information
Choose and combine evaluation methods which allow the collection and organi-
sation of the necessary information.
How to evaluate?
Which evaluation methods will you use?
In which sequence and combination will the methods be organised / used?
Which different kinds of information? Quantitative or qualitative? Written, verbal,
non verbal?
From which sources will you collect the information?
How are you going to collect and organise the information obtained?
Can you identify any clues for “understanding”?
Tip: It is recommended to use a variety of methods (quantitative and/ or qualitative)
given the complementary nature, advantage and disadvantages of each of them
for the collection and analysis of that information.







Page
80
Pages
57-74
Designing the evaluation plan
Articulate your evaluation plan with clear objectives, felds, timing and indicators.
What to evaluate?
– Which felds should be evaluated?
– How will we group the different elements that are to be evaluated? Will we
use existing models of evaluation or will be develop our own?
– What should be prioritised?
Tip: It can be interesting to evaluate aspects such as context, learning process,
content, methods and outcomes group process, technicalities, organisational frame,
participants’ profle, “products”.
Choosing indicators (qualitative, quantitative?)
– Indicators: Signals, data, information, which help us to recognise the achieve-
ments (achievements of the objectives, learning achievements, etc.)
When?
– Educational evaluation is a continuous and ongoing process. At the same
time, some moments are especially relevant for the evaluation (i.e. the end
of the preparation, the frst day of team work, the free day in the middle of
an activity, etc). We can foresee and plan those moments when we will pay
special attention to the evaluation.
– In which time frame and sequence do you wish to organise the evaluation
process?



www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
55
Interpreting the fndings – What do they mean?
Interpretation or “judgement” gives meaning to that which is evaluated according
to the aim, considering the information collected and using the criteria and
indicators previously identifed.
Remember the aim of each element evaluated – what is it for?
Establish criteria.
– The criteria are elements, which will allow us to compare reality with the
objective or the expected outcome of the educational activity. This is a central
part of every evaluation.
Identify what happened: Was it a consequence or an accident? Is there any
cause-effect relationship to be observed? Was it foreseeable or not?
Take into account that not everything is measurable!
Think about “Objectivity” and “Subjectivity”.
– Interpretation is always subjective. Be aware of that and make that fact
explicit. To “limit” the degree of subjectivity, you can try to diversify sources,
cross-check information collected, involve other people, check alternative
interpretations and consider previous understandings of similar facts.





Page
81
Drawing conclusions
This is the moment for “extracting lessons” from the evaluation. It is important to
be honest.
Recall the objectives of the evaluation.
State the results achieved (facts), distinguish facts from opinions.
Look for patterns, common characteristics or repeated mechanisms.
What lies was behind these results? What do they imply?
What can we conclude and learn from them?





Reporting the outcomes
Share the information, fndings and conclusions resulting from the evaluation with
the different actors of the project and with other interested audiences.
Which different target groups will your reporting address?
Which emphasis will it have?
What kind of evaluation report will you produce?



Pages
82-84
56
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work

We have marked with a magnifying glass some important, and at times especially sensitive,
stages of the educational evaluation process:
How to evaluate? Evaluation methods
Collecting and organising information
Establishing criteria
Reporting the outcomes
We will look at them in more detail in the following sections.
In the section introducing the T-Kit you refer to the cook symbol, partly in reference to how
to use the T-Kit. I suggest that you add a reference to this symbol as well at the same
point for the sake of coherence.




Implementing the results
The implementation of the results of the evaluation is the last step and can be
the starting point of another project.
What should be improved?
What should be maintained?
Which changes, measures, adaptations, future strategies are necessary or implied
by the results?
Some advice at this point:
Be constructive!
Be realistic!
Be aware of resistance to change!
Foresee adequate time!
Start to implement the results when the conclusions are fresh and in everyone’s
minds, when the energy is there and without waiting too long!








www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
5I
2.3 How to evaluate? Evaluation methods
In this part and when describing different methods, we have chosen to focus on methods
that are used for evaluating with the participants of youth exchanges, training courses,
EVS projects and other educational activities. This does not mean that these methods
cannot be used in other circumstances. Many of the methods described can, with some
adaptation, serve for evaluation purposes within teams or with other stakeholders of our
project as well.
One way to classify evaluation methods is to distinguish between personal, interpersonal
and group methods. We will explore the uses of several evaluation methods within each
of these categories.
Personal methods
Personal methods are activities in which, the actors involved in the evaluation make their
own judgements and draw their own conclusions concerning the progress of the activity,
the learning process, the outcomes, their personal feelings and involvement, etc.
Some examples of personal evaluation methods are:
Surveys and Questionnaires
Put simply, surveys are a method for collecting qualitative and quantitative data. Comparable
information is gathered by using standardised methods, such as questionnaires.
Questionnaires are unfortunately the frst thing that comes to mind when we speak about
evaluation. Unfortunately, because in some projects the team simply distributes a question-
naire on the last day, and feels that they have done the job of evaluating. It is true that
a well-designed and well administrated questionnaire can provide a lot of useful infor-
mation. However, as we have argued before, questionnaires alone can only provide part
of the overall picture.
You may administrate questionnaires in three basic ways:
Self Administrated questionnaires may be completed by the respondents
61
them-
selves;
In structured interviews interviewers may administer questionnaires in face to face
encounters
Reading the items to respondents and recording the answers; or interviewers may
conduct telephone surveys.
61 When specifcally applied to research, respondent means one who is inspired to participate in a focus group,
interview or survey. Source Encyclopaedia Britannica Online http://www.britannica.com/



58
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
When you are designing a questionnaire, you should pay attention to three very important issues:
Content
Question type
Question structure
Content is related to the overall purpose of the questionnaire. You should make sure that
the subject can indeed be measured through a questionnaire. To see if the participants
were satisfed with the logistical aspects of the organisation of the activity such as the
food may be easy to measure through a questionnaire. However, if you want to measure
the degree of intercultural learning achieved during the activity, due to its complexity
ant to its group dimension, you may need other methods, in addition to a questionnaire.



Question types
In a questionnaire, the questions may be classifed in two categories: closed or
open-ended. Closed-ended questions have a limited number of answers while
open-ended questions have an infnite number of answers. By their very nature,
the answers to closed-ended questions are much easier to analyse. This is, of
course, an important consideration. However, an advantage of using open-ended
questions is that you may learn new and unexpected things. The results of the
evaluation can be very rich.
A closed-ended question offers respondents a number of response choices. The
respondents are asked to mark their response using a tick, cross, or circle etc.
Their response may be a simple Yes/No, Male/Female; or may involve a range of
different choices.
Example:
Q. Which energiser(s) you have enjoyed most? (Please circle two)
1. Shark in the pool 2. Kiss the bunny
3. Chinese frecrackers 4. Moje Sutra
5. Camel wrestling 6. Group juggling
Closed-ended questions may have three different types of responses:
Nominal: The responses are categories that differ in name; Gender (Male, Female),
Position (Volunteer, trainer, youth worker)
Ordinal: The responses are categories that can be ordered, Agreement (Strongly
disagree, disagree, agree, strongly agree).
Interval: The responses in which the distance between categories is meaningful,
Age (measured in years), Number of volunteers.
Open-ended questions provide space to the respondent to answer in his/her own
words and own logic. They can be extremely useful, as you usually cannot guess
all the possible responses to a question.
Example:
Q. What do you think about the T Kit on Educational Evaluation?
A. I think it is a cool book. I always read it, especially before going to sleep…
Sometimes a combination of open-ended and closed-ended questions is the best.
You may list all possible responses and an additional category (other) so that if the
list does not cover the respondents desired answer, they can add it themselves.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
59
When designing a questionnaire, you should also check if you need more than one ques-
tion to evaluate the aspect that you are investigating. The respondents should be able
to remember the context of each question and they should also have the necessary
information to be able to reply. That is why it is a good idea to review the programme
with the participants before distributing the questionnaire, if you wish to evaluate the
programme on the last day of a project.
Regarding the question type, you have to be clear if what you need to measure is better
measured using open-ended or closed-ended questions. If a question is closed-ended,
you have to make sure that the responses you provide cover all possibilities. You should
not overly limit the options of the respondent. For example, if evaluating the food, you
can ask: How satisfed are you with the food? But, to provide the respondent with the
options to answer only I love it or I hate it is too limiting. Maybe some people think it
was good, but did not fall in love with it. Often when flling out questionnaires, partici-
pants tend to focus on scale or score questions, as they are easier to answer and take
less thinking effort. Open-ended questions are often replied to without a lot of detail.
Question structure is another important dimension of designing a good questionnaire.
Designing clear and well-written questionnaires is a real art. Unfortunately, there are no
magic rules for how to do this. However, there are some simple things that can help you
to improve your questionnaires.
When writing your questionnaire:
DO: DO NOT:
Use clear and short questions Use double negatives
Put your questions in a logical order Use technical jargon and abbreviations
Try to be positive and motivate a reply Use emotionally loaded words (i.e. frus-
trating, exciting, annoying, fantastic)
Give clear instructions (i.e. tick, circle a
number, check the box, etc.)
Use leading questions (i.e. “Name the
innovative elements of this course”)
Be consistent: clear, direct, precise. Use long complex questions
Give the questionnaire a title (i.e. par-
ticipant evaluation questionnaire of
activity X?)
Give a short introduction to the ques-
tionnaire explaining what you will use
the information for (i.e.: the information
collected will be used to compile the
evaluation section of the fnal report to
the funding institutions)
Use idioms or culture specifc expressions
(i.e.: “it’s a piece of cake”, “green light”,
“fire brigade”, “cup of tea”; burning
issues”)
Present it in a user-friendly manner and
give it an attractive layout and appear-
ance
Use biased questions (i.e. “How did the
group process contribute to your learn-
ing?”)
Provide an adequate variety of options
for answering closed-ended questions
Present two questions in one
Consider the language, literacy and writ-
ten expression capacities of those who
will answer the questionnaire
Put important questions at the end of the
questionnaire
6U
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
The Bad Questionnaire exercise
62
Have a look at the Bad Questionnaire and try to identify the problems with it.
Discuss them with your evaluation colleagues.
The problems are exaggerated on purpose but the logic underlying them is present
in many questionnaires and other evaluation methods. Discuss how these problems
can be avoided?
62 Kneale, Dylan. “Evaluation seminar” Avanos, Turkey 4th -10th October 2004 Organised by Youth Express
Network and supported by the European Youth Foundation of Council of Europe.
The Bad Questionnaire
Please complete the following questionnaire – thanks!
How old are you?
1-20 years old n 22-24 years old n over 26 years old n
What is your ethnicity?
Yes No Not sure
Where did you hear about the programme and why did you decide to come on the
programme?
My Mum told me about it and told me to come on it
In the newspaper and I decided it would be a good idea
My teacher said it was something to do after I had fnished school and I agreed
Other:
What did you learn on the programme?
Everything n Nothing n
Do you like to play football
Yes l No l Often l
What are you doing now?
In employment n In education or training
Both n
What is your current occupation? (for example: I am a farmer)
I am a ________________________________________________________________________________________________________
Where do you live?
What is your name?
Yes n No n
Will you tell all of your friends how good this programme is?
Yes l No l Not sure l
Have you got a wife yet?
Yes l No l Not sure l
How has your opinion about life and the universe altered now that you have most
successfully terminated, completed and concluded our programme?
THANK YOU FOR ANSWERING OUR QUESTIONNAIRE!!!!!!!!!




www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
6I
Without wishing to present a model (rather, an example!) we provide a possible structure
for a “good” fnal evaluation questionnaire (Pages 97-101).
A letter to oneself (also known as “letter to yourself ”)
The “letter to yourself ” is one of the simplest and most commonly used personal eval-
uation methods. It is also one of the most effcient. The “letter to yourself ” can achieve
some very interesting and in depth results about the impact of a programme. As it is a
personal method, it can also assist the participant to become aware of certain progress
made as a result of his/her participation, which s/he would not realise otherwise.
If you are planning to use this method, you should keep in mind that the “letter to your-
self ” is a very personal method. The facilitator does not have any possibility to intervene,
infuence or interact in the process as s/he does not read the letter. After all, it is a let-
ter to “yourself ”, the participant may write in their own language and what they write is
totally between the participant and, well, themselves.
You ask the participants to write themselves a letter. When you ask them to prepare the
letter depends on the programme of your project. This may be on the very frst day of
the programme, in the middle or on the last day, before departure. The decision will be
made in function of what you want to achieve with the letter. In all cases, it is important
to provide enough time and space for writing this letter and its importance should be
made quite clear to all participants.
If the participants are writing the letter on the frst day of your activity, you may ask
them to write about a large variety of issues, for example, their expectations, fears, why
they have come to the activity (their motivations), their frst impressions of the team,
other participants or the country (if they have travelled abroad to the activity).
If the exercise is taking place on the last day, the letter may include similar issues, but
focus on different aspects, for example, are they satisfed with the activity, what have
they learned, what are they taking back home with them, which part of the programme
have they enjoyed most/least, what is their plan of action for implementing the things
they have learned once they get home?
Once they have fnished writing the letter, they seal it in an envelope on which they write
their full name and postal and address. The team sends the letters to the participants
after a period of time after the project has ended, for example, 2 months after. This
allows the participants to make a comparison of their initial perspectives and feelings
with the ones they have 2 months after the project.
Diary
Another way of giving participants the opportunity to refect on the process they are
going through is to ask them to write a diary. The team provides the participants with a
notebook (the diary), in which they can note down the experiences of each day during
the activity.
This is, of course, very suitable to projects that last more than just a few days. This can
be a good method in EVS-projects to help the volunteers to refect on the large number
of new impressions that they have to deal with while living and working in another coun-
try. In residential seminars, the experiences accumulated outside of the “offcial pro-
gramme” are also very rich. A diary can be also useful for documenting and evaluating
these activities.
62
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Of course a diary is something personal and therefore is not to be read by youth
leaders, mentors or facilitators! But, aspects of what the participants might note
down in their diaries can be discussed in particular group settings such as for example
“refection groups”. We provide a description of refection groups in a later section
(see page 67).
One approach to this activity is to provide an empty notebook for participants to use at
their discretion, but another and one which is often appreciated by participants as they
receive some guidance, is to give some guiding questions.
For example:
Describe what happened today in your own words
Were there any remarkable moments?
Are there things/questions still buzzing around in your head?
Is there anything ‘new’ you learned today?
Is there anything you want to pay further attention to?
Remember that participants may need to be motivated to use their diary on a daily basis.
A diary that looks nice or that is presented in a creative way can motivate.
Interpersonal methods
In interpersonal evaluation methods more than one individual actor share and discusses
their judgements and conclusions, often in a pair or small group. The value and purpose
of those methods is to share, to confront opinions and to learn. This takes place because
participants have the opportunity to evaluate from more than one point of view. Just two
examples of interpersonal methods of evaluation are interviews and focus groups.
Interviews
There are two types of interviews, in depth interviews and structured interviews. The
main difference between these types is the amount of fexibility allowed to the interviewer
and to the respondent.
• Structured interviews
In a structured interview both interviewer and respondent have very little fexibility
because there is a questionnaire that has to be followed. The interviewer has a
questionnaire and reads out each question to the respondent. The majority of the
questions will be open-ended, for example: “How did you feel during the simula-
tion exercise?” Even though the answer of each respondent will be different, the
question remains the same.
• In depth interviews
In an in-depth interview there is no pre-set questionnaire, just a list of important
issues that the interviewer must cover with each respondent. The interviewer has
much more autonomy and fexibility in the development of a conversation with the
respondent.
However, in both types of interviews, what is important is that the interviewer acts in a
neutral manner and tries not to direct or infuence the responses of the person being
interviewed. The participant should feel comfortable with and have trust in the interviewer.





www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
63
Before starting an interview we should always be clear about what we want to evaluate.
For example, if our aim is to evaluate the recruitment of participants, it is possible that
we will want to collect information about where participants heard about the programme
and what motivated them to get involved.
As we see, conducting interviews requires some experience, knowledge and competences.
We should ask ourselves if we are the right person to conduct the interview or if it is
appropriate for us to do so at all.
For an ideal interview, the participant should feel comfortable with and should trust the
interviewer. As in other interpersonal situations, in an interview, the “race”, socio-econom-
ic background, gender and even accent of the interviewer can infuence a respondent’s
answers.
It is important to record the answers of the respondent during the interview. This is a
practical issue. Trying to write everything down at the same time as conducting the inter-
view is very complicated and may even be impossible. It is not appropriate to ask a
respondent to slow down, as this would interrupt their thought fow.
It can be useful to record the interview with a hand held tape or digital recorder. It might
also be relevant to take notes about the non-verbal communication that takes place dur-
ing the interview. Recording the interview can liberate us to note down such behavioural
observations on paper.
Interviews are very work intensive and time consuming. A lot of resources are required
to do and analyse them. But, they have the advantage that they can give a better insight
into why a participant thinks that, for example, the energisers should get 4 points out
of 5.
Preparing and making in depth interviews
It is a good idea to prepare some questions or prompts before starting any interview. If
we are conducting a structured interview, we have already prepared a questionnaire. But
for an in-depth interview it is also very important to prepare in advance. Even if you do
not formulate the questions, make sure to list the items you want to discuss with the
respondents.
When preparing and making in depth interviews, it is important to take following points
into consideration:
Try to avoid closed or dichotomous (YES/NO) questions. Imagine asking: “Did you
enjoy the programme?” The participant will say “Yes”, and that will be it.
The questions should trigger the respondents to think. We are aiming to get as much
information as possible. For example, you can ask: “In your opinion, which aspects
of the programme were most useful?”, instead of “Did you enjoy the programme?”.
Before asking a question, we should consider if the respondent will answer the
question truthfully. We recommend not to get too personal and not to ask ques-
tions that would put the respondent in the position of lying out of politeness or
for legal reasons. For instance, the trainer asking participants in an activity: “Do
you think I am a good trainer?” is not very appropriate. Participants may feel under
pressure to say even yes if they do not think the trainer is good.
Interviewers should be neutral: their presence should not have any effect on the
responses given to questions. In case the responses given to an open-ended ques-
tion are unclear or not suffcient in the opinion of the interviewer, a probe (a neu-
tral, non-directive question) can be used. Some examples of probes include, “Anything
else?”, “What do you mean?”, “Can you explain in more detail?”, “In which ways?”




64
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Focus Groups
In very simple terms, a focus group is “a group of people sharing similar characteristics
who talk about a subject they have in common”.
63
Focus groups are one of the most
effcient qualitative and interpersonal evaluation methods. Their small size and relaxed
atmosphere allow for deep and frank evaluation.
In a youth project, the young people themselves are the “group of people” that forms
the focus group and their common subject are aspects of the project that need to be
evaluated.
Even though it sounds quite similar, focus groups differ extensively from interviews. The
main difference between focus groups and interviews is that an interview is a one on
one activity that takes place between an interviewer and an interviewee. Focus groups,
however, are a group activity, as the name implies.
A focus group is made up of a group of young people and a facilitator or moderator. In
a focus group, interaction between the participants, leads them to think out loud and
form opinions during and as a result of the process.
Focus groups have some important advantages and disadvantages. The frst and most
important advantage of using focus groups as a method for evaluation is that they allow
you to explore the way in which young people form their opinions as well as fnding out
what their opinions are. In a focus group, the reason why a participant is thinking in a
particular way is as evident as the opinion they hold. The participant reaches an opinion
as a result of the discussions with other participants in the focus group.
Another major advantage of focus groups is that they encourage participants to speak
in public. Many participants feel more secure expressing themselves in a small group of
their peers than sitting alone in a corner with an interviewer. As a result they tend to
express their opinions more openly and more bravely in a focus group situation. But,
this does not hold true in all cases.
Group pressure may also negatively affect the sense of security of participants to hon-
estly express themselves. Thus, extreme care should be paid to group dynamics and the
amount of peer pressure that exists in any group when deciding how to approach the
evaluation or when deciding whether to use focus groups.
But, forming focus groups also has certain diffculties. It is very diffcult to make sure
that all participants will participate with the same level of concentration and attention.
If we are going to form a focus group during the programme of a given activity, we should
help participants to stay focused and be ready to allocate enough time for them to
express their opinions.
It should also be noted that during a focus group, it is very diffcult to record the infor-
mation and to keep track of what is being said at the same time. To overcome this is
good to collect all together, at the end, the main conclusions or simply a mechanism of
rotation for taking notes.
One fnal but nevertheless important disadvantage of the focus group can be the eventual
generalisations that can take place in the discussion. We should check if the opinions
raised are representative of all the participants in the group, as it is not rare to have one
or two outspoken members who speak in the name of others in an activity.
For example, the frst day, several participants who have already been in similar activi-
ties might express their “impatience”. But, the silent majority of the group might have
another opinion.
63 ibid.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
65
Here we have collected some general recommendations for facilitating focus group:
64
Keep the group small (ideally 4 to 6 people)
Take into consideration the language skills of the international participants as it
affects the time they need to communicate their opinion
Keep your questions neutral and avoid manipulating the discussion with biased
questions. This can create a negative reaction from the participants
Be very patient, as it takes time for the discussion to build up and for all members
to feel invited and comfortable enough to join in. It is counter productive to inter-
fere in the discussion too much and to constantly offer your own opinion. It can
make the participants feel pushed
Make clear at the beginning what you want to achieve by holding the focus group
discussion. Tell participants what the aims and expectations are. If the facilitator is
clear with the participants, it is very likely that the participants will be clear too
Try through everything you want to discuss in just one session. Remain focused,
and limit the number of topics and issues for discussion to the most important and
relevant ones. Otherwise you risk that the participants lose interest and you may
end up with very little information.
Group interactive methods
Group interactive methods of evaluation can be used to check what the feelings or views
of the group are at a particular stage of the programme or to evaluate a certain session.
Of course we can do that in an informal way by observing the group and the activities
and by asking questions. These observations would be valuable but at the same time
subjective to the team or the observers.
Personal or interpersonal evaluation methods like questionnaires or interviews with par-
ticipants might be time-consuming, depending on the programme. Shorter interactive
group methods, which include elements in addition to verbal and written expression, are
a good option. These methods not only inform us about the views and feelings of the
group, but also create the possibility for participants to refect and to learn about how
others experience the activity.
Group evaluation methods have an additional dimension. Since the group in non-formal
education is source of learning, group evaluation methods can specifcally deal with
group dimensions of the learning process including for example, group life, the atmosphere
in the group, the co-operation among participants…
Checking expectations
What are the expectations of participants towards the programme? What do they want
to take home as a result of their participation? Are there things that they absolutely do
not want to do? For different reasons it is good to ask participants to take some time to
refect on what their expectations are at the beginning of the activity. To formulate what
you want helps you to remain focused on your needs during the programme.
For participants it is interesting to fnd out about the needs and wishes of the others.
This is essential for the team. The team has every interest in understanding how par-
ticipants view their purpose in participating in the activity. It helps the team to refocus
the programme or clarify to the participants what they can and cannot achieve in the
context of the activity.
64 ibid.






66
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Clothes line
An effective method for collecting the expectations of participants is called the
‘Clothes Line’. All participants receive papers on which items of clothing are drawn.
Each item of clothing stands for a different kind of expectation: for example, Hopes,
Fears or Offers to the programme that participants may wish to make. Time is given
for participants to refect on their expectations using the categories of clothing as
a prompt. They write their different ideas related to each category on the clothing
shapes and stick them on a clothes line made of string (hung in an appropriate
space in the working room) or drawn on the wall. Everybody has a chance to look
at the “laundry” of the others. It is important that the group has the time for it.
Clothes line
Three posters
A similar method involves placing three big blank posters on the wall. Each one is
given a title, for example: ‘What do I want to take home?’, ‘What do I want to
avoid?’ and ‘What can I contribute?’. On sticky pieces of paper participants are asked
to write down their answers to each of the questions/poster-titles and to stick them
on the appropriate poster. Remember to give the group suffcient time to review
individually the answers of all the participants. A team member or facilitator can
also make a review and read out the most common answers given to provide the
group with a global picture of the expectations in the group.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
6I
The legitimacy of these methods depends on the conscious use of the information collected
by the team and by the group. The expectations should be carefully considered to discern
which ones can be fulflled in the activity and they should to the extent possible be inte-
grated into the different programme elements.
At the end of the activity, one can return to the initial expectations of participants and
to refect on the extent to which participants have got what they wanted out of their
participation. A very simple and visual method to do this is to invite participants to go
to the “clothes line” or to the “posters” and to remove their fulflled expectations. They
should leave behind the ones which have not been fulflled. Then participants should
explain the reasons why some expectations were fulflled and others not. Another way
to return to the initial expectations is to group them and discuss their fulflment or not
in small discussion groups. Often unexpected outcomes arise at this point, something
which is interesting for both participants and team to know about.
End of the day reviews
In a programme that lasts more than a few days, it can be advisable to make sure there
is a regular moment to look back on the experience of the participants in the activity
part of the programme. This regular moment for evaluating can take place everyday (end
of the day review) or at regular intervals during the programme, and is often done in
small groups of 5 to 7 persons. For the participants this can be a valuable moment to
share experiences about the programme of the day or of several days, to express their
feelings and to hear how others have experienced that same part of the programme. For
the facilitators, it provides an effective means for staying in touch with the way the group
develops and to take up any problems and challenges that may be communicated by
the group. It is important to plan appropriate time for these reviews. This is not always
easy at the end of the day. But, rushing through such a group evaluation just before
dinner begins does not allow you or the group to refect properly and runs the risk that
participants feel they are not being taken seriously.
End of the day reviews with the whole group have the advantage that participants learn
about the impressions of everyone in the group and can create a sense of common respon-
sibility for the group life and the programme in the group as a whole. But, large groups can
have the disadvantage that only some few participants take the foor. Therefore, if under-
taking an end of day review in a large group, it is important to propose methods that make it
possible for all the members of the group to express themselves. We propose below some
of those methods: “The Three Word Review”, “The Telegram” and “The Living Dartboard”.
Refection Groups
Refection groups are small groups of between 4 and 6 people. They usually take
place daily at the end of the programme and last between 30 minutes and one
hour. Their composition is normally fxed for the whole duration of the activity. This
allows the members of the refection group to create their own atmosphere during
the several sessions over the duration of the activity.
It is interesting to have in each group participants from different backgrounds, with
different personalities and profles, so that within each group there is wide range
of points of view. This is particularly important for the evaluation. In addition, small
groups offer a more secure space for expression in public than a large group. .
The Refection Group gives the possibility participants:
to exchange experiences with fellow participants
to evaluate the day
to let off steam
to come up with suggestions and ideas to improve the programme
For facilitators, refection groups are a way to “keep in touch” with what is going on
in the group.




68
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Other methods, as described below, can be used as ‘warm-up’ activities in refection groups
and as we said also to evaluate interactively with a large group. The group leader does not
take an active leading role but stimulates the active participation of all group members.
The Three Word Review
A simple method at the end of the day, for the whole group, is the “Three word review”
where participants are asked to write down for themselves three words that describe
their feelings about the day. Then participants are invited to call out their words which
are noted down on a fip-chart. After that, a discussion about the meaning and rel-
evance of the words can be initiated. This commonly leads to a lively discussion.
The “Three word review” involves all the participants from the start which makes it often
easier for those who are usually not the frst to speak, to contribute to the discussion.
The Telegram
A variation on this is the “Telegram” in which participants give a general impression
about a day or session. This activity is very appropriate after a tiring day or session
when there is no energy left for a thorough evaluation. Participants are invited to
think for a few minutes about the day or session and then to share their opinion
with the group in three words: a positive word – a negative word – a concluding word.
For example: Intensive – Headache – Exhausted
Make sure to write down on a fipchart what participants say and it can be useful to make
a short summary of what the general group feeling is at the end. When time is limited,
the exercise may be concluded by asking participants to read each others’ telegrams.
Another way of sharing outcomes is to invite participants to write the telegrams on
sticky pieces of paper and to put them on the wall for everybody to read.
The Living Dartboard
An energetic way to evaluate the day is the “Living Dartboard”, in which participants
are invited to move around the room according to their position towards statements
that are read out about the programme, or other aspects being evaluated.
You need an empty, rather large room for this activity. In the middle of the room is
an object (e.g. chair, paper, fower, etc.). This object represents the centre of the
dartboard (also known as the “bull’s eye”).
The facilitator reads out statements about the activity and participants choose a spot
in the room according to their opinion about that statement. The more you agree with
the statements that the facilitator reads out, the closer you move to the middle-
point. The more you disagree, the farther away from that point you should move.
After everyone has chosen their position participants can be asked to explain their
reasons for standing where they are.
Towards the end, participants may be invited to come up with their own statement/s
about the activity. This way they have a chance to check how the rest of the group
feels about it.
Some examples of statements are:
The rhythm and timing of the programme was well planned
The methods in this activity were monotonous
Being in an international group helped me a lot to understand what Intercultural
Learning is about
I missed theoretical inputs
The food was fabulous







www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
69
In these three methods it is important for the facilitator to make notes of what has been
said so that the outcomes of the evaluation do not get lost and can be used by the
group and the team.
Visual group evaluation methods
Visual group evaluation methods give a clear overview about how other participants
value a certain part of the activity. They do so, not so much by using words, but by taking
advantage of the potential that non-verbal communication can offer. Many people feel
much more comfortable when they can express themselves without using words, espe-
cially if the activity is taking place in a foreign language. Many such methods exist.
The “River” is a method which stimulates participants to use their creativity when
evaluating. It is a method that focuses on the process of an activity rather than on its
contents or other aspects commonly evaluated. The metaphor of a river can help people
to refect on their experience during the activity.
Another more process-oriented method is called “Boats on the Sea”. This activity can
help participants to take stock of what has been done and what still lies ahead of them
in the activity. Participants are invited to use their creativity by evaluating through the
use metaphoric symbols: the sea, boats, sea animals, islands, other ships, etc.
A useful metaphor for looking towards the future is that of the “Train with different
wagons”. Each wagon represents a different element of the programme that participants
should refect on. Participants are asked to write down ideas related to each of these
elements for follow-up to the activity in the appropriate wagon.
And of course you can develop and adapt all these methods using the idea of a trip or
a journey to represent a process.
When using this kind of method, it is useful to create a nice and relaxing atmosphere.
This can help to make people feel comfortable and to evaluate more effectively. This can
be done using some background music.
Another visual way of helping participants to fnd out about how they feel at a given
moment is called “Puppets in a Tree”. Images help participants to think about how they
feel. Pictures of little fgures or puppets are shown in different positions in a tree.
Participants think about how they feel and decide which of the puppets in the tree best
symbolises their mood or state of mind at that given moment.
The River
The river is a very versatile evaluation method. It can be used on the last day of
the programme, for mid-term evaluation and for daily evaluation. It offers partici-
pants the possibility to express, in a creative way, how they see their development
during the programme.
Participants are divided into small groups (5 to 7 participants). Each group is pro-
vided with the picture of a River drawn on a large piece of paper. The beginning
and the end of the river should be marked on the paper. Each group receives paper
(different colours), scissors, glue, pencils and markers. Participants are asked to
work individually on giving creative expression, using the material provided, to their
personal development during the programme and to place the results of their cre-
ative efforts onto whichever part of the river they consider appropriate.
When everybody has fnished participants are asked to explain their creations.
IU
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Boats on the Sea
This exercise can be done in small groups as well as individually.
Prepare a large drawing of a sea with two harbours (one on the top and one on
the bottom) in advance. The sea between the two harbours stands for the period
of time between two moments of the programme. For example, this could be the
beginning and the end of the project, but any other programme part could also be
chosen.
Ask participants to design their own boats and to put these boats somewhere in
between the two harbours. Within the sea, islands or rocks or other symbols can
be drawn. Participants are free to add anything they feel helps them to clarify the
position of their boat in this metaphor.
This method can be used in many different ways. You can give participants different
kinds of boats representing different elements you want to evaluate (for example,
the boat of “my learning” or of “the group process” or of “my participation in the
programme”, etc.).
Train to the Future
This process oriented evaluation method is useful for looking towards the future
and can be used in the evaluation to think about possible follow-up to a project.
It is a group visual method that can be used in large groups.
But, the starting point for working on it, can be personal (by including this drawing
of the metaphor in a questionnaire) or in small groups (using a fipchart). The results
can be then shared on a big poster so that the whole group can visualise and
decide upon the future co-operation after a certain project.
The different wagons allow a certain grouping of the co-operation possibilities (i.
e. communication, networking, follow-up activities, common projects, publications…).
Within the metaphor of the train, other symbols (i.e. the machinist, the passengers,
the power…) can be used to discuss and decide about the different roles and respon-
sibilities in the future co-operation.
FUTURE TRAIN(ING)
COOPERATION .................. in future? ..........
Documentation
Question Question Question Question Question
Thank
you
Project
1
2 3 4
5
6
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
II
Puppets in a Tree
Puppets in a Tree
This method can be used for the end of day review as well as for a fnal review in
smaller or bigger groups. The advantage of this method is that you can easily adapt
it to different situations or needs.
All members of the group are invited to express their current feelings and / or their
satisfaction with the programme or on other aspects such as their learning, the
group dynamic and so on.
To do this participants are asked to choose one of the fgures (or puppets) on the
picture that they consider most representative of their present mood or state of
mind in relation to the aspect being evaluated.
When everybody has chosen is their puppet, participants explain to each other the
reasons of their choice.
In all these visual group methods of evaluation, it is important to take notes of the com-
ments and discussions. The written notes are a necessary complement to the visual
information: they help to explain, understand and conceptualise it. Altogether the visual
information and the notes can be organised and used by the team and the whole group
to draw conclusions and identify possible changes in the programme. On a practical note,
they can be used in the evaluation report.
I2
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Snap-shots
Snapshots are useful when you do not have a lot of time left but you still want to get a
picture of the general feeling in the group. There are some short visual methods that can
be used for this. These methods give you an impression: a snap-shot. They are certainly
not as thorough as other some of the methods already described, but they can be helpful
for facilitators and participants to see how the others feel.
A simple method for checking the atmosphere in the group is to use the “Thermometer”
for “taking the temperature” of the group. This metaphor allows you to how people are
in the group. With a little creativity you can easily adapt it to create other metaphors to
fulfl the same purpose, for example, a sunny beach, under the umbrella on a rainy day,
freezing in the snow, etc.
In the same logic, you can ask participants to evaluate how they feel about the pace of
the programme by using the “Speedometer”. Here you can also replace the speedometer
with animals or means of transport going at different speed to diversify the metaphor.
A quick and easy way to test how people feel about different elements of the programme
is to use “Applause”. It can also be used as an energizer to warm people up to evaluating.
And, again you can be creative and make your own version using other sounds instead
of applause.
The combination of several of these methods can be used to carry out a lively mid-term
evaluation half way through the programme.
The Thermometer
A simple method for “taking the temperature in the group” is the “Thermometer”.
The thermometer can be drawn on a fip chart and participants are invited to put a
personalised mark (their name or initials) on the temperature they feel best represents
how they view a particular part of the programme or how they feel the group is.
Warm
Friendly
Atmosphere in the group Businesslike
Cool
Unsupportive
In this example, the atmosphere in the group is evaluated. But, other aspects can
be evaluated with the thermometer, for example the informal moments, group
life.
Participants can also be asked to share their evaluations and to discuss how things
can be improved.
The fip chart depicting the group evaluation can be posted on the wall and at a
later stage of the activity can be used again, to see how things have developed.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
I3
The Speedometer
How do participants feel about the pace of the programme? The “Speedometer”
offers participants to express their feeling about the rhythm of the activity and can
be used as an ongoing (daily) evaluation method.
Draw the speedometer on a fip chart and ask participants to put their personal
mark according to their opinion about the pace of the programme.
Further instructions needed to understand how this can be used, for example, let
all the participants take a look at the other participants’ opinions and discuss as
appropriate.
Remember to mention what the team will do with the information.
Applause
This is a very simple method to evaluate different elements of the activity at the
end of or during the programme.
Ask participants to form a circle. The facilitator reads out different elements of the
programme and according to their satisfaction with that element participants clap
their hands.
The higher the level of satisfaction, the louder and longer the applause should be.
As this activity is only intended to measure the satisfaction of participants with a
particular programme element (in other words, whether they liked or disliked it), it
can be used as a warm up activity before moving to the business of more serious
evaluation activities.
SPEED
slow
too slow
OK
fast
too fast
I4
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Last round
The last round in an activity offers participants the possibility to express the things that
they consider as the most important elements or those issues “which have not yet been
mentioned” before the end and before everyone leaves. This is sometimes an emotional
moment before the programme really ends.
Last round
This is as simple as giving every participant, one by one, the possibility to speak
and to say anything they feel is appropriate. Just remember that this can easily end
up in a long and tiring session. To avoid this and to help participants to focus on
the most essential, you can limit each intervention, for example, to one sen-
tence.
A very dynamic variation on the last round is known as “The Matchbox”. Its dyna-
mism comes from tension and speed. You need one or two boxes of matches
(one match per person is enough) and a plate or a bucket. One by one, partici-
pants are asked to burn a single match and to speak only for the time that the
match is burning. When it burns out you have to end your point immediately!
Another way to deal with the last round is to take an object, preferably one that
somehow represents the project, and to send it around the circle of participants.
The one who has the object in their hand speaks and when fnished hands it over
to another person in the group who is then invited to express themselves.
A variation on this is to use a ball of string. As participants express themselves
and pass the ball from person to person, a spiders’ web of links emerges between
the members of the group.
The ideas expressed in the last round are, sometimes, not very well developed
or may seem not to be very relevant. But, these feelings and impressions expressed
are usually very revealing and can help us to understand the outcomes of other
evaluation methods. For this reason, it is important to note down the main ideas
expressed in the last round.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
I5
2.4 Feedback
Feedback can be part of interpersonal and group evaluation. In some of the methods
outlined previously the evaluation process implies giving feedback to somebody.
Feedback is “the transmission of evaluative or corrective information to the original source
about an action, event, or process.”
65
One very important element should be emphasised:
FEEDBACK SHOULD BE HELPFUL TO THE PERSON RECEIVING IT
To be helpful, feedback to somebody must be such that the person:
understands the information
is able to accept the information
is able to do something with the information.
Feedback can be used in one to one situations as well in small groups/teams.
When using feedback in a group it is helpful to begin by reminding the group briefy of the
meaning of feedback and to set a time frame so that everybody has the chance to both
give and receive feedback.
Some things that we consider to be feedback serve only the needs of the person giving
it and not the needs of the person receiving it. In other words, if you are fed up with
somebody and fnally come to the point telling them, you might be relieved in doing so,
but this is does not constitute giving feedback. This serves only your need to tell them
how you feel. It does not necessarily provide them with constructive information that
they can act on for improved relations.
Feedback is somewhat more complicated than one might imagine. To avoid misunder-
standings and problems in the group, it is important to be sensitive about the group
dynamic when using feedback. When working in the context of human relations, feedback
should be guided by the principle of mutual respect for each other as learning subjects.
Take your time. Make sure you have enough space (in terms of the working room and
time frame). Think about an adequate environment in which to do feedback. Think about
the atmosphere in the group and the implications of undertaking feedback at any given
moment. These simple things can provide participants with the security they need in
order to undertake honest feedback and serious evaluation.
Below you will fnd some suggestions for how to give and receive feedback.
65 Source: Merrian Webster Online Dictionary. http://www.m-w.com



I6
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
How to give and receive feedback?
Giving feedback
Speak for yourself by using “I” instead of “You” or “We” when you give your feedback
to the group. If speaking about your personal opinions and feelings, they should
not be related as being those of others even if you feel they are representative
Begin your feedback with a positive review rather than with negative statements
When you give feedback to one particular person in the group address that person
directly rather than to the group as a whole. This involves saying who the feedback
is for
Do not judge other people. Rather you should describe your personal perception
of a situation or your feelings
Give feedback in relation to the “here” and “now” situation rather than to past events
If something confictual arises, try to be constructive rather than destructive. Be
supportive to those who propose ways to deal with diffculties and conficts. It can
help to remind oneself of the human potential for learning and for change
It is alright to praise somebody! Express what you liked most in somebody’s behav-
iour, what made the most important impression on you or what you suggest for
someone to do in future
Receiving feedback
Think of the feedback you receive in terms of constructive criticism. You have a
chance to receive information about how others view your performance and to
understand what they think about you
If you feel you are being criticised, try to avoid immediately taking a defensive atti-
tude. You do not have to justify your behaviour or to explain why something is like
it is
Try not to interrupt the person who is giving you or the group feedback. If neces-
sary and agreed in advance of the feedback session by all involved, responses can
be given at the end










www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
II
2.5 A critical look at evaluation methods
Sorbet for breakfast?: When to use what method?
Serving sorbet for breakfast or fsh for dessert might not
be the best idea in some countries. Of course, one must
be careful with this kind of statements, especially in an
intercultural environment! This example should alert us
to the idea that spending some time thinking about ‘what
to serve when’ makes a lot of sense, both in cooking
and in evaluation.
When choosing an evaluation method, the frst question we have to ask ourselves is:
what do we want to get out of the evaluation? What do we want to fnd out? Do we want
to know more about the learning process of participants? Do we need information about
how people feel in the group? Are we curious about how people feel about the method-
ology used in the programme? Or do we want frst of all to give participants the possi-
bility to refect on their learning and to share that with the others?
Methods like “Applause” or the “Living Dart-board” will not help you to fnd out more
about the learning process of the individual participants. This kind of method can help
you to get an impression of the atmosphere in the group, for example. It is useful not
only for the content of the answers that the participants provide, but also for the infor-
mation provided by the way the group deals with the questions. When during the “Living
Dartboard” all participants demonstrate complete satisfaction with every item evaluated
by running immedia tely to the middle point, one justifable conclusion would be that
“having an opinion which is different from that of the others” is not (yet) a shared cul-
ture in the group.
Questionnaires and interviews will provide you with information about the individual
learning process of participants. But, refection groups can serve this purpose as well.
When a refection group meets regularly it can provide a safe environment where people
can share their feelings and the challenges they consider have arisen. This kind of
exchange helps participants to give words to their experiences and to compare them to
and learn from the experiences of the others.
Methods which rely on the use of creativity (drawing, non-verbal expression, theatre, for
example) can help participants to give expression to issues arising as a result of the
process they are part of in the activity and which may raise mixed emotions. By using
objects as symbols, drawing or body language you offer participants other ways of
expressing their concerns and ideas. Verbalising these ideas afterwards for the sake of
clarity and mutual understanding can then be used in the second step.
People are different and so are participants in educational projects. One method will ft
better with one participant than with another. Some people love to fll in questionnaires
and will provide you with long answers to the questions. Others have problems in writing
I8
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
down their experiences and feelings, but are excellent in expressing themselves in more
creative ways. It is important to offer a variety of methods that provide participants with
different communication styles with equal opportunities to express their views.
Evaluation methods should also be thoroughly introduced. It is important to explain to
participants before beginning the evaluation activity what its aim is, why you have chosen
it, what the purpose is for those conducting the evaluation and especially what will be
done with the results.
Outcomes
What do the outcomes of all these evaluation methods tell us? First of all, they can pro-
vide a rather positive impression of how participants value the activity. But even when
participants have expressed satisfaction with an activity, it remains important to take
critical look at the outcomes. Sometimes the level of satisfaction can be caused by the
lack of challenging activities or a good group atmosphere. Whether they are positive or
negative, the outcomes of the evaluation should be carefully considered.
The way in which the participants answer to the questions posed in different evaluation
methods is infuenced by many things including, for example, the specifc moment at
which the questions are asked, the atmosphere in the group and how others answer the
questions.
Especially in visual methods like the “Thermometer” there is the risk that participants
will adjust their opinion to what others have already noted down. Most people want to
feel like they belong to the group and this can lead to not wanting to obviously differ
in opinion from that of others. But, this kind of “group conformity” is also a matter of
which stage of development the group is in. The important thing is to remain aware that
the risk exists and to consider it when interpreting the outcomes of a given visual meth-
od of evaluation.
It is also good to take a critical approach to the outcomes of evaluation questionnaires.
Questionnaires are flled in at a specifc moment in the activity. The answers to the eval-
uation questionnaire on the last day are infuenced by the fact that people are soon
leaving the activity. Participants may have mixed or confused feelings on the last day.
Often it is diffcult for participants to answer questions about what they have learned
on the last day of an activity. Participants might answer quite differently two weeks after
they have gone home and had a chance to refect with some distance.
This does not mean that such evaluation methods are worthless. It simply means that
it is relevant to consider the variety of possible infuences in the interpretation of their
results.
Resistance to evaluation
Often when we undertake an evaluation we can count on the support and backing of a
team or an organisation. But, sometimes people do not like the idea of evaluating. We
should, therefore, be aware of the fact that there can be resistance to evaluation.
Why is that? Where might such resistance come from?
Evaluation needs resources (time, human and fnancial resources) whether it is an
external, internal or (self-) evaluation. If we invite external evaluators to conduct
our evaluation, we will most defnitely have to spend more money on it, but we
will get an independent and professional evaluation conducted by an outsider,
whose role it is to be independent. Some members of our organisation might not
consider this worthy of signifcant fnancial investment or they might be worried

www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
I9
that the resources they need for their work will be used to fnance the evaluation.
Therefore, it is important to work out with the team or the organisation backing
the activity which amount of resources are we willing (and able) to use for the
evaluation in advance.
In the social and civic sectors, people with different roles and responsibilities (policy
makers, educators and trainers) might feel threatened by evaluation because it may
include judgements on the quality of their programmes and products. In other words,
they might feel negatively towards the evaluation if the evaluation may be critical
towards the quality of their work. For example, sometimes critical evaluation results
can have the effect that the continuity of a programme is put into question. Maybe
funding will not be repeated next year? Maybe the evaluation will recommend a
review of the competence of those working on the programme? Some trainers might
perceive evaluation as interference in the educational process or as waste of time
and resources. Decision makers might feel their actual space for decision making
is limited.
Transparency and professionalism are very important in overcoming such resistance to
evaluation, as is remembering that educational evaluation should always be done “in
service” of the project.

8U
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
2.6 Collecting and organising the information
The information collected during an evaluation can be both qualitative and quantitative
in nature. Qualitative information is normally processed and summarised focusing on
the most important and recurrent facts. Quantitative information can be analysed and
presented in the form of statistics, charts and graphs.
Regarding quantitative data, we would like to encourage the use of statistics in the pre-
sentation of evaluation outcomes. It is considered “professional” when an evaluation report
demonstrates that the evaluators were able to manage statistics and to calculate and
interpret averages and standard deviations
66
. For this, software tools which can help us
to analyse and process quantitative data (i.e. excel tables, access applications or any
other specifc statistical software) are useful and (self ) trainings exist in which one can
become profcient in basic statistical applications.
On the other hand, we would like to stress that quantitative data has limits. Quantitative
information (without reference to some qualitative explanations) is far too limited to
explain the educational process. If, for example, we have a quantitative data that says
that 25% of participants found the contribution of the group to their learning excellent,
50% ok and 25 % bad. But, having only this data, we still do not know about the reasons
for each score being given: was it because of too many strong personalities? Or because
of sub-groups among participants? Or because it was their frst international activity? Or
was it because of an overly “professional” atmosphere without inter-personal exchange?
It is quite common that in evaluation questionnaires, both quantitative and qualitative
information is asked of participants, but that in the reporting only the statistics are pre-
sented. This may be because statistical tables look attractive and they offer a veneer of
objectivity or credibility. But, even from a purely statistical point of view the amount of
data that we usually manage to collect in educational activities is far too small to be of
a high statistical signifcance. For example, in a group of 30 participants a very bad mark
given by just one participant can cause a signifcant decrease in the average score. But,
this one very bad mark can be an “accident”, caused by reasons which have nothing to
do with the educational process. The “accidental marks” in a group of 300 or of 3000
have a much smaller infuence on the average evaluation score and the statistical results
are, therefore, much more reliable.
This collection and organisation of the information still takes place at an early stage of
the evaluation process. It is not yet the moment of arriving to conclusions or of reporting
the outcomes of the educational evaluation. Nevertheless, it is good to have in mind
already at this stage that some quantitative and qualitative data might later be part of
“reporting the outcomes” and, therefore, of the evaluation report. You can fnd some tips
for the presentation of that information in “Reporting the outcomes” (Pages 82-84).
66 Standard deviation is a statistic used as a measure of the dispersion or variation in a distribution. It is
equal to the square root of the arithmetic mean of the squares of the deviations from the arithmetic mean.
Source: www.answers.com/topic/standard-deviation.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
8I
2.7 Establishing criteria
As we have seen before criteria are elements, which will allow us to compare reality with
the objective or the expected outcomes of the educational activity.
Establishing criteria is one of the most diffcult and often confictual steps of the evalu-
ation process. On the one hand, there is the risk of reducing the whole evaluation to a
normative imperative (it should be so!). On the other hand, there is the risk of getting
caught up in the defnition of criteria, because it can be diffcult to embrace the com-
plexity of educational evaluation.
For example, if we want to evaluate the objective of a given project. The objective of the
project is “to experience the diversity of youth work”. There are several criteria that we
could use to assess the fulflment of this objective, as follows:
the spaces provided in the programme for exchanging experience among participants
the diversity within the group: in terms of nationality, cultural background, youth work
experience
the contribution of “externals” and other resource persons
A quantitative indicator for the criteria “diversity within the group” could be the number
of different nationalities represented in the activity (i.e. the nationality distribution of
participants). However, diversity cannot be reduced just to difference in nationality. There
are other factors that make up diversity, including age, gender, living in a rural or an
urban environment or coming from different regions within a country among others.
In order to be able to evaluate the objective (to experience the diversity of youth work),
we should decide upon the most relevant criteria and indicators, even if those criteria
and indicators cannot embrace the entire complexity of the objective being evaluated.
But, the number of criteria and indicators cannot be unlimited. We need to select the
most signifcant ones. If the project deals with environmental issues maybe the “living
in the city / living in rural areas” indicator is as relevant as the nationality of participants
for evaluating the objective of “experiencing the diversity of youth work”.



82
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
2.8 Reporting the outcomes
Reporting the outcomes of the evaluation is very important. It is a tool for sharing infor-
mation, fndings and conclusions with the different actors and the frst step to involving
them in the follow-up and future projects. Reporting the outcomes does not only consist
of summarising the fndings and putting them in a report for the funding authorities.
Reporting requires a certain amount of planning, structure, focus and adaptation according
to the target groups that we want to reach.
Good use of the outcomes of our evaluation can be useful at a very practical level. For
example, we can decide on the basis of our evaluation to continue to work with the
same youth hostel for the next youth exchange we organise. The same is true for the
educational level. For example, our evaluation could lead us to conclude that we should
use a certain methodology again but the next time it should be adapted in this or that
way. Therefore, it is essential to give the same amount of attention to the processing
and reporting of the outcomes, so that they can be well used by us and others in the
future, as we put into the evaluation process itself. This last stage of the evaluation
process can be the cherry on top of the delicious cake, coming back to our cookery
metaphor.
The evaluation report
The evaluation report is not the only way of reporting the outcomes: oral reports, articles,
photos or other informal ways can also be excellent channels.
Nevertheless, the evaluation report is usually a key tool for reporting the outcomes of
the evaluation. The evaluation report offers quite a comprehensive vision of the project,
covering many different aspects, from different perspectives and with a certain distance.
This makes it a very useful learning tool for participants and organisers, both present
and future. It can be used to improve the quality of your next projects, by focusing on
the strengths and by trying to tackle the weaknesses revealed by the evaluation. It can
also help others who may want to undertake and implement similar projects.
Thinking about the evaluation report when planning educational evaluation may prove
to be very useful. By doing so:
We can decide which information we need to analyse
We can decide which methods are likely to be most effective for getting that information
We can foresee which stakeholders would we like to target
We can think in advance what we would like to emphasise to whom
The emphasis of reporting shifts according to who it is intended for. If the report will be
read by experts in Brussels, it will have to have a different emphasis and probably also
style than if it is going to be read by the secretary of the Mayor of our town or by other
youth workers. Nevertheless, this does not mean that we should tell different stories to
different people. In the report, the order of presentation and the emphasis may shift,
not the signifcance of the outcomes of the evaluation.




www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
83
General considerations
In principle there are three specifc types of reader that will use our report: the
participants of the activity, the stakeholders in the activity and outsiders who may
potentially beneft from the fndings presented in our evaluation.
Another important potential audience to keep in mind when writing the evaluation
report is those who may in the future beneft from a similar project. This is a wide
group of stakeholders and can include actors ranging from the funding authorities
to volunteers in the organisation. Therefore, the report should be clear and com-
prehensible.
All aspects of the evaluation should be described in suffcient detail to permit an
outsider to reach more or less the same conclusions as someone who attended the
activity. This is especially true if we have used questionnaires or interviews. If we
present an interpretation, we should make sure to support that interpretation with
direct quotes from interviews or with relevant and comprehensible statistics.
Transparency is a very important aspect of reporting evaluation. All participants of
the project should receive a copy of the evaluation report once it is ready. Participants
might not fnd their own words in the report, but they should feel that it is repre-
sentative of their experience. The report should, of course, cover both the negative
and positive aspects of the evaluation outcomes. By respecting transparency, we
contribute to the sense of collective ownership of the project.
If we use quantitative data and statistics in the report, we should present them in
a very clear and comprehensible way for the readers. It is a good idea and reader-
friendlier to use simple visual representations of statistical data such as pie charts
and colour coded graphs, rather than just numbers, numbers and more numbers.
Tables, charts, and fgures, if any, should be integrated into the text of the report,
appearing near that portion of the text discussing them. Sometimes people describe
their analyses in the body of the report and place all the tables in an appendix at
the end. This method can be quite problematic for the reader. As a general rule, it
is best to (a) describe the purpose for presenting the table, (b) present it, and (c)
review and interpret it.
Some key parts of the evaluation report
Every evaluation report is and should be different. We can, nevertheless, identify some
common parts which should be covered by any well presented and structured evaluation
report.
Introduction
We should give enough information to an outside reader about the process and what
we are evaluating. The introduction should include:
All the relevant background information about the project
Who conducted the evaluation and with which competencies
The criteria and the perspective of the evaluation
The description of the design and execution of the evaluation
Presentation of fndings
After this important introduction, we should present our fndings. The presentation of
fndings (statistics from questionnaire, qualitative analysis of interviews), and our inter-
pretations should be integrated into a logical whole. It can be very frustrating for the
reader to go through a list of seemingly unrelated analyses and fndings with a promise










84
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
that all the loose ends will be tied together later in the report. Every step in the analysis
should make sense to the reader. We should present our rationale for a particular analysis,
present the data relevant to it, interpret the results, and then indicate where that result
leads next. At times, the inclusion of some personal refections (i.e. quotations of partici-
pants) can help to illustrate more clearly some ideas. However, we should always make
the distinction between personal refections and results clear at all times.
Summary
Having provided the general overview and the results, it is essential to summarise the
most important points. We should avoid reviewing every specifc point in the evaluation,
but we should review all of the signifcant ones, pointing once more to their general
signifcance.
The focus of each evaluation is different and, therefore, it is diffcult to indicate what
information would be the most signifcant in an evaluation report. The conclusions concern-
ing the “format” of the project that may eventually lead to changes are very relevant (for
example, change of target group, of duration, of educational approach). Suggestions for
follow-up (i.e. further projects, support measures for participants, systematization of
results in a publication…) are also important.
Conclusions
The report can conclude with a statement of:
what we have discovered about the educational progress, the direction and degree
of change of the participants in the evaluation
the extent to which the objectives set in the beginning were met
some ideas about where future projects might be directed
However, ending with the conclusion that “more projects are needed” is of little value
unless we can offer some concrete suggestions about the nature of future projects and
unless we can provide arguments for why they are needed. We should review the par-
ticular strengths and weaknesses of our own project as demonstrated by the evaluation
outcomes and suggest ways in which those shortcomings might be avoided in future
projects. Any comments and suggestions offered should be supported by fndings. This
section is not an appropriate space for making declarations.
Appendices
The outline of the full programme outline and the list of participants can be among the
most common appendices to evaluation reports. In addition to these, it is relevant to
include some photos or other materials which can transmit the “taste” of the activity to
the readers. It is also a good idea to append any bibliographic references consulted in
the preparation of the evaluation report. Some evaluation reports also include the raw
data from the evaluation conducted in the appendices.



www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
85
2.9 Evaluation practices
In this section we will present some “practices” of educational evaluation or projects in which
evaluation played a relevant and specifc role. Following the inviting and non-normative
approach of this T-Kit we have tried to avoid calling this section “good” practices.
Instead of presenting “examples of good practise”, the purpose of this section is to share
experiences and refections, successes and frustrations, potentialities and limitations
coming from practices on educational evaluation. These do not correspond to a systematic
mapping of youth work projects. There are certainly many others at least as interesting as
those presented here and many aspects relevant to understanding educational evaluation
will not be covered by them. Nevertheless, we have tried to cover different types of activities
(from a youth exchange to an e-learning project), different approaches (from a systematic
evaluation to permanent and ongoing adaptation), different target groups (from young-
sters to experienced trainers) and different methods (from questionnaires to non verbal
methods). We hope that these practices can contribute to inspire the development of
your own.
“Take5” – 40 youngsters evaluating
The “Take5” project
Supported by the YOUTH programme and the ‘European Year against Racism’
67
the Take5
project took place in 1997. The project involved 40 disadvantaged youngsters from fve
countries (Poland, Germany, Finland, Sweden and the Netherlands). Take5 was a long
term project that took up the challenging idea of combining youth work and theatre and
worked with a stable group of participants over a period of 6 months in total.
The idea for the project came into being during an international training course youth
workers. The fve people involved decided to co-operate on setting up a music and the-
atre project centred on the theme of social inclusion. The basic idea of the project was
to approach the participating youngsters from the perspective of their capacities not
from the perspective of their problems. This does not mean that the youngsters did not
originate from “problem situations” but these situations were not taken as a starting
point.
67 1997 was the European Year against Racism. It was the frst time that Member States of the European
Union and the European Institutions came together to take joint action against Racism. A lot of
projects, seminars and campaigns took place at national and at European level. More information at
http://europa.eu.int/comm/employment_social/fundamental_rights/public/arcreyar_en.htm
86
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
The starting point was formed by the youngsters’ abilities and the challenges offered by
an international theatre project. Subsequently, the young people continuously ended up
in situations which were new to them and which demanded new solutions. In the course
of the project they were constantly confronted with problems and challenges which could
not be dealt with by means of their “traditional solutions”. This forced them to look for
other ways of solving things. In short one may say that ‘broadening the options’ was
the essential objective for the project.
The participating organisations were: Verdandi Södertälje (Sweden), Kinder und Jugendzentrum
Freiberg (Germany), Youth Club Europe (Poland), Legion Theatre (Finland), Tandem Nijmegen
(the Netherlands) and Stichting Spectrum (the Netherlands).
The project took place in fve phases:
a. In the frst phase, groups worked within their local environment with their youth
leaders on preparing the frst international encounter which took place in Germany.
b. In Freiberg, Germany eight days were spent on working with the youngsters on
intercultural communication and theatre skills. The frst ideas for the fnal perfor-
mance were shaped here. The starting point was to make a performance centred
on the personal experiences of the young people with regard to social exclusion.
In addition to the fve original team members, the group was accompanied by fve
co-workers and a German theatre director.
c. After Freiberg, the participants returned to their own countries to work on enhance-
ment of their theatre skills and they made up scenes of which some were incor-
porated in the performance. In this phase, the director visited all groups and spent
two days working with each of them.
d. Five months later all participants met again in Zielona Gora in Poland. In this phase,
all scenes developed in the various countries, were presented to the others. Subsequently,
elements from the different scenes were used to put together a play. The group
worked in the theatre under great tension and time pressure. Despite, but perhaps
also thanks to the tension and time-pressure, a magnifcent play by the name of
“The Raft” was created and performed on the opening night, in which the young-
sters tell their stories with great conviction.
e. In the last phase of the project the group toured the fve participating countries
performing a theatre show made out of their experiences and directed by a pro-
fessional theatre director.
For most of the young people participating in this project, this was a challenging experi-
ence that brought them into many new demanding situations, often under a lot of pres-
sure. That is the reason why evaluation played a very signifcant role. Since the project
was extremely challenging appropriate time was needed for the youngsters to refect, to
look back and to digest on what happened.
The development of evaluation through the project
In the international encounters, national groups came together every day to refect
(refection groups) from the very beginning. The team recognised a need for par-
ticipants to be able to refect in their own language, since it was not easy for many
of them to express themselves in English (the working language of the international
parts of the project). Although, as the project went on the language capacities of
the participants improved a lot, the national refection groups went on meeting.
These groups offered the possibility to participants to express some of their more
personal concerns more openly and in a relaxed atmosphere.

www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
8I
After a confict in the group on the 4th day of the frst international meeting in
Germany, the youngsters proposed a daily review with the whole group. Although
in the beginning the team was quite sceptical, it developed into one of the most
important elements of the project. In the beginning the group just sat together in
a circle and discussed the day, problems were taken up and plans for improvement
were exchanged. Since it was their own idea to start this daily refection, participants
felt responsible for its success and acted accordingly. One of the team members
was experienced in relaxation and meditation exercises. In the beginning most of
the youngsters were not that enthusiastic about these methods, but that changed
during the process and at a certain moment participants asked her to work with
these exercises at the beginning of the daily review. It helped them to relax and to
refect on the day on a personal level before going into the group discussion.
During the last three weeks of the project in Poland and throughout the theatre
tour a common diary was introduced. The diary was always somewhere around and
participants had the opportunity to write down their impressions. The diary was
not used by all youngsters but a quite a few wrote in it regularly. All of them read
the diary. It became a space for personal and interpersonal evaluation and for shar-
ing experiences, impressions, frustrations, conclusions, learning achievements and
complaints. The team followed it closely and used its outcomes for the planning
and adaptation of the programme.
After the frst international meeting and at the end of the whole project participants
were asked to fll in evaluation questionnaires. Participants answered the questions
in their own language. This involved a lot of work for the team members, as the
questionnaires had to be translated into English in order for the whole team to be
able to be involved in the assessment of the forms.
A follow-up evaluation
Although on the last day of the project in Finland some of the youngsters expressed their need
to meet again after some time to see “how everybody is doing”, a follow-up evaluation
was never planned. But then, when the team had their evaluation one month after the
project, it was decided that it was worthwhile to put efforts into organising an evaluation
meeting with all the participants to see what the impact of the project would be.
Participants met in the Netherlands for fve days to evaluate approximately 6 months
later. Although a lot of talks took place, both in groups as on individual level, the meeting
lacked a planned evaluation and therefore missed quite some opportunities. It showed
that an unplanned evaluation can be a wonderful event, but equally that it has limited
value when it comes to the real evaluation of a project.
The brochure written about the project was fnished as planned in the original project
plan and published during that evaluation meeting with all participants present. A very
nice happening, but the long term outcomes were not described.
Nevertheless, there was one very signifcant outcome of the evaluation meeting. It was
very obvious that participants profted a lot from all the evaluation moments during the
project. The majority of these youngsters were now quite capable to describe the mean-
ing of the project for them, to refect on changes they feel they experienced and to point
out the impact of this project on their future plans. The capacity to refect and to give
words to their experiences and feelings turned out to be one of the main competences
that the youngsters developed in the “Take5” project.
The evaluation was developed and adapted according to the challenges that arose in the
project and to the needs of participants. This experience demonstrates that consistent
planning and openness for adaptation and further development are both very important
when working with evaluation within a project.



88
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
EVS
68
and Leonardo
69
- Multidimensional evaluation
EVS and Leonardo in Padova
XENA
70
is a non-proft, cultural association situated in Padova in Italy. Amongst many
other things XENA hosts young people in the framework of the “European Voluntary
Service” (EVS) and the “Leonardo da Vinci” programmes every year.
European Voluntary Service is part of the YOUTH programme of the European Union and
offers young people the opportunity to stay for a longer period of time, up to one year,
in another country working as volunteers. XENA has been involved in EVS since 1997
and hosts volunteers as well as sending them abroad.
Leonardo da Vinci is the European Union action programme for implementing the European
Union’s vocational training policy. The aim is to use trans-national co-operation to enhance
quality, promote innovation and support the European dimension of vocational training
systems and practices. Within the Leonardo da Vinci programme, XENA works with the
primary aim to promote the development of innovative employment sectors taking into
account the impact employment creation can have on the environment and society. XENA’s
key areas of work are environmental protection, social and cultural services, animation,
alternative tourism, not for proft activities and unemployment.
Each year, six EVS volunteers are hosted in Padua for six months. Around 100 participants
take part in Leonardo projects in Padua per year. They stay for three months. The EVS
participants work within XENA, while the Leonardo participants are employed in different
organisations and companies all over Padua. XENA has almost daily contact with the EVS
volunteers. Both sets of participants are offered the opportunity to take part in an inten-
sive Italian language course in the frst month of their stay in Padua.
Multidimensional evaluation with participants
A long term mobility project can be a learning experience in many different ways. It
includes language learning, professional experience, intercultural learning, autonomy and
independence, social life, meeting new people and organisational aspects.
For those who organise a mobility project these aspects are clearly distinguished but for
participants this is one big, often new and confusing experience. Often the personal
situation of participants strongly affects the perception of other aspects of their experi-
ences of the mobility project.
Evaluation meetings can be used to make distinctions, separate the different aspects
and avoid overly general judgements. They are a way to refect on how participants can
change things or deciding together if and what the organisation can do to improve the
situation.
68 See footnote 28 – Page 37 for further information about EVS
69 Leonardo da Vinci was a vocational training programme of the European Commission. The programme promoted
trans-national projects based on co-operation between the various players in vocational training (training
bodies, vocational schools, universities, businesses, chambers of commerce, etc.) in an effort to increase
mobility, to foster innovation and to improve the quality of training. The Leonardo da Vinci is a pillar of the
Lifelong Learning programme of the European Commission. It aims at helping people improve their skills
throughout their lives.
More information is available at http://europa.eu.int/comm/education/programmes/leonardo/leonardo_en.html
70 XENA is a non-proft, cultural association which was set up in the autumn of 1994. Its name derives from
ancient Greek and means foreign things. Its main objective is to increase and improve the level of contact
and interaction amongst different cultures. It promotes projects mainly within the framework of European
Programmes, such as “Leonardo da Vinci”, “Youth” (i.e. “Youth for Europe”, “European Voluntary Service”)
and others. XENA works in co-operation with other bodies (European institutions, associations, municipalities,
enterprises, schools, trade unions, informal groups, etc.) in Italy and abroad. XENA is an active member
of the Eurodesk network, managing a local information centre. More information at http://www.xena.it
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
89
There is a lot to evaluate and many people have to be involved: the language course,
the place the volunteers live, the working environment, the group life, the support offered
by XENA, life in Italy.
The mechanisms for such a multidimensional evaluation with participants are:
In the frst month there is an evaluation meeting every week. The language course
is of course an important topic. But, the conditions where people have been accom-
modated and the dimension of “getting to know Italian culture” also need to be
discussed. The weekly evaluation meeting is a group evaluation. In the frst meeting
the “Clothes Line” (Page 66), is used to collect the expectations of the participants.
Many different exercises including the “Dartboard”, the “Puppets in the tree” (Pages
68, 71) and different forms of statement exercises
71
are done to evaluate the different
elements.
The Leonardo participants are regularly asked to fll in evaluation questionnaires.
In the beginning most of the questions ask for quantitative information and scales
for scoring are provided. At this point, the language capacities of the participants
in Italian are still limited, so scoring is easier for them. Later more open-ended ques-
tions are added to the questionnaires. The questionnaires focus on the language
course, the working place and the living conditions.
The EVS’ers have their own questionnaires focussing on their specifc situation. In
those questionnaires the focus is on the hosting organisation, the mentor, their tasks,
as compared to the job-description they received, and the living conditions.
The EVS participants have a mentor with whom they have regular consultation
meetings. Often these are not offcially scheduled meetings because the participants
meet their mentor almost daily and unplanned (evaluation) conversations take place.
During their whole stay, participants have the possibility to put ideas or complaints
into a suggestion box which is installed in a central spot. This is a possibility to
express yourself outside the group and anonymously, if needed.
Evaluation as feedback for the organisation
For XENA evaluation is an important instrument for receiving feedback from participants
and for improving the way projects are organised.
This feedback is often about a specifc aspect of the experience of participants (accom-
modation, work placement) and can lead to the decision to make some changes in the
organisation of the project, for example, not to use a certain accommodation provider
any longer, to place trainees in that company again, or that placement is possible, but
only if certain conditions are met. In other cases the feedback can be about the way the
project is organised.
During the frst period of working with long term mobility projects XENA realised that
some participants felt they did not receive enough support from XENA, although a lot of
time and energy was spent on organising the support already offered. Considering the
result of evaluations, XENA became aware that an overly informal approach to this sup-
port can have negative effects. Some participants saw their mentor more as a friend than
as a tutor and as a result, felt that the organisation was not offering much professional
support. This feedback was taken seriously and, although maintaining an informal atti-
tude, more “formal” evaluation meetings, written questionnaires and a clarifcation of
the role of the mentor were foreseen in subsequent editions of the hosting project. These
measures helped participants to feel more secure and to get more out of their time in
Padua.
71 Described as “Where do you stand?” in “T-Kit No. 4 – Intercultural Learning”, Council of Europe and European
Commission (November 2000), Page 51.





9U
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
XENA also realised that although many people are needed for the successful organisa-
tion of a long term mobility project, for participants it is confusing and problematic to
have to communicate with several contact persons during the organisation of their time
abroad and during their stay. Participants need to refer to one or two people whose tasks
and responsibilities are clearly explained to them. Once again, the staff of XENA had to
consider not only what they were doing but also how it was perceived: many people
were working hard to guarantee a good result, but the effect was that the participants
did not know to whom they had to refer in case of a problem. As a result, the decision
was made that only a few people work directly with the participants.
XENA developed a system for evaluation:
Long term plans concerning the general aims of the project are developed in
co-operation with the whole staff (10-15 people)
Each co-ordinator in charge of a department elaborates a plan on a yearly basis,
periodically checked with the manager of the association and formally evaluated
after 6 and 12 months
The manager is constantly monitoring and coaching the staff, using an informal
approach
The staff meets at the beginning of each week. In this meeting the staff can share
information, evaluate the team work and co-ordinate for the tasks of the following
days. It is as well an occasion to learn from each other thanks to the reports of
those involved in external training.
Internal evaluations are organised within smaller groups of staff after the realisation
of specifc activities.
Many different aspects of the project are evaluated in the different analyses and phases:
the impact on groups hosted, the impact on society, fulflment of objectives, outcomes,
contents developed, individual well being and satisfaction of the people collaborating,
interpersonal dynamics, management, logistics, economical/fnancial issues/sustainability,
follow-up.
As you can see, XENA takes evaluation seriously!
Madzinga – External evaluation
The Madzinga project
Madzinga was a Long Term Training Course
72
organised as a cooperative project by
Outward Bound Belgium, Kitokie Projektai, Lithuania, Hitt Husid, Iceland and Outward
Bound Slovakia in 2002 to 2003. In addition to resources provided by the organisers
and fees paid by participants, the project was fnancially supported by the YOUTH
Programme of the European Union, the European Youth Foundation of the Council of
Europe and the Soros Foundation.
72 A Long Term Training Course consists on several residential seminars with project or practice phases in
between. Its total duration is between one and two years. This training course format was for the frst
time developed by the Directorate of Youth and Sport of the Council of Europe. For more information
please consult the following website: www.coe.int/youth





www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
9I
Madzinga focused on the intercultural learning dimension through outdoor experiential
learning and comprised three phases.
The frst phase was a residential seminar in Lustin, Belgium, where experiential training
methods were explored and the concepts linked to them were deepened. Twenty four
participants from thirteen countries attended the seminar. In the second phase partici-
pants developed projects at home and stayed in contact through so called coaching
groups. The third and last phase took place in Samukas, Lithuania, where participants
developed activities for Lithuanian youngsters and further projects.
The people who developed this training course had worked together in previous years
and wanted to develop a new training concept with this long term course. That was one
of the reasons why they wanted to have an external evaluator / observer who followed
the whole project, being present from the frst preparation meeting, throughout the training
and during all the team meetings.
External evaluation: somebody is watching you!
Imagine you are baking a cake and someone is in your kitchen watching you. This person
is not only watching you, they are taking note of everything you do: How much sugar
you put in, how many eggs you use, counting the minutes you are spending on mixing
the cake mix. It might be quite disturbing. Later, that person makes a report, in which
you can fnd a description of your activities in the kitchen along with some critical remarks
about you as a bake.
As we already described, one specifc approach to evaluation is to invite an external
evaluator to conduct your evaluation. This is someone who is not involved in the project
as such, but who follows the process as an observer, records what is happening and
reports about it. Not being involved in the project as a team member or as a participant
has the important advantage that the external evaluator can look at the project more
objectively or at least from the perspective of an outsider. Another beneft is that the
only task of the external evaluator is to observe and report on the activity, and therefore,
can dedicate all their energy to the job. The external evaluator collects data and records
it all so that they will have enough information to develop the report the outcomes of
the project.
Being “an outsider”, the external evaluator is more capable of asking the “right” critical
questions about the project. Being involved in a project and in all the work that it implies,
risks that we take things for granted and simply do not take (or have) the time to take
a step back to look at what is happening. The “outsider” is in the position to question.
Especially when it comes to diffcult aspects of the evaluation, such as measuring the
learning outcomes of a project for participants, the external evaluator can be much more
precise in collecting data and describing developments in the project. But, being the
external evaluator is a challenging and diffcult job that requires specifc competencies.
The result of the external evaluation in Madzinga: the report
73

The result of the external evaluation in the Madzinga project is a 160 page publication
that reads like a story. A story about non-formal learning, a story that tries to show what
people learn in less formal learning situations such as this training course and makes
an attempt at measuring those learning outcomes.
73 The Madzinga report can be downloaded at: www.outwardbound.be/madzinga/MADZINGA.pdf
92
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
The Madzinga report is descriptive, but it also strongly refective. All activities are described.
But, more even important are the descriptions of the way participants act and react in
different situations. Team meetings are reported and essential issues and discussions
are raised.
Apart from this refective dimension, this report is interesting from the point of view of
evaluation because attention is given to the role of the external evaluator and to the
problems and challenges of the tasks involved. Spending time, a lot of time with a group
of people deeply involved in a process and being in a position which is actually ”outside”
of that process is not very easy. It implies a lot of questions: How much can you get
involved? How invisible should you be? To what extent do you take part? When should
you speak? Should you speak at all? How should you deal with the personal and profes-
sional dimensions of the work? Imagine what happens to a team and to a group of par-
ticipants when somebody is around them all the time taking notes. How does the person
taking notes to deal with the way the group deals with them?
The report does not give an answer to all these questions, but it does make an effort to
explore the challenges faced by an external evaluator in the feld of non-formal education.
This makes it a worthwhile read for people who plan to work with or work as external
evaluators.
Evaluation in an E-learning pilot course
E-learning is booming at the moment. Lots of students learn while sitting behind their
computers. In the feld of non-formal education the frst tentative steps are being taken
to explore the opportunities that web-based learning can offer.
Especially in the context of long term projects, the educational possibilities of the inter-
net are being explored and discovered. The combination of residential meetings and
follow-up or “in between” virtual encounters offers interesting educational potential.
But, what about situations in which the participants do not know each other and will
never have the opportunity to meet? That has an enormous impact on the educational
process. And what does it mean for evaluation?
Human Rights Education: Compass Contact Point
The Polish organisation Association for Children and Young People “CHANCE” took up
the challenge to start a pilot training project on ‘Human Rights Education with Young
People’ by e-learning. “COMPASS – a manual on Human Rights Education with Young
People”
74
published by the Council of Europe was the starting point for the training
course. Twenty people participated in the course that took place from June until November
2005. The course was designed in a modular format. The modules took the participants
through the different subjects of Human Rights Education. Material was provided on the
website to study and practice. Several tasks were given to participants in the process of
the course. On the website it was possible to communicate with other participants, by
chat as well as by forum discussions.
The software used was Moodle
75
, specially developed for educational purposes and free
available on the internet.
74 Compass portal with downloadable versions at http://www.eycb.coe.int/compass/
75 Moodle is a course management system (CMS), a free Open Source software package designed using sound
pedagogical principles, to help educators create effective online learning communities. More information
at http://moodle.org/
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
93
On-line evaluation
Evaluation forms after the sessions
After every session an on-line form was flled in by the participants. Moodle offers eval-
uation forms which can be adapted to the needs of the users. The software processes
the forms into statistics which can be visualised on the website. This way, participants
are informed in a clear way about the opinions of their colleagues.
The topics evaluated were:
the relevance of what was learned
the way refective thinking was stimulated
the level of interactivity
the support of the tutors and of peers
if participants could make sense of the messages posted on the platform
The evaluation was largely conducted using multiple choice questions. Nevertheless,
participants had the opportunity to mention any other comments they may have had to
make.
Ongoing evaluation
In the discussion forum, the opportunity for “ongoing evaluation” was provided. Participants
were invited to comment on the course and to discuss opinions among themselves.
Individual interviews
Individual interviews were conducted using chat. The moderator scheduled meetings with
each of the participants to discuss their learning process and problems and challenges
they met.
Final evaluation questionnaire
At the end of the course, the participants were asked to fll in a more “traditional” eval-
uation questionnaire designed by the moderator. It contained many more open-ended
questions.
Changes as a result of the evaluation
As a result of the evaluation changes were made in the programme of the course.
One major problem encountered, as is the case of many educational platforms, was the level
of “interactivity”. In the regular evaluation forms after every session almost all topics
scored positively. The exception was the level of interactivity of the learning process. The
evaluation of the discussion forum showed that participants had problems to communicate
with each other. As a result, the moderators decided to stimulate communication between
participants by giving them tasks in which they had to work together in small groups.
This was successful. The communication increased, not only on the website, but also by
email and telephone.
Another problem that came up was the pace of the course. For many participants things
just went to fast. In the beginning, topics were taken off the website after the subject
was closed. But, not all participants followed the rhythm proposed by the moderators.
The moderators realised that they did not take enough into account that participants
had their own schedules defned by working and family life. So, the speed of the course
was reduced and all the material was left on the website even after the subject was
closed.
Evaluation played a big role in this course because it was a pilot training and because
of all the new challenges that e-learning presented to the moderators.









94
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Overall Evaluation of a Pilot Long Term
Training Course
The “Coaching
76
in international youth work” Course
In 2000 a Steering Group of representative from fve institutions (two international youth
organisations, an European educational centre, a working group of trainers and a
“Fachhochschule” (University of Applied Sciences) started a further qualifcation course
called “Coaching in international youth work”. The course was organised by the International
Youth Exchange and Visitor’s Service of the Federal German Republic, also known as IJAB
e.V. and transfer e.V. The participants had more than three years of work experience on
either a voluntary or professional basis (as managers or multipliers) in the feld of inter-
national youth work.
24 persons from fve countries (Belgium, Russia, Netherlands, Switzerland and Germany)
took part in the 5 module training course. The seminar was led by two feld experienced
facilitators. From module to module additional trainers were engaged for specifc inputs
and training elements.
The concept of the course was to provide a clear understanding about the following things:
What is a coach?
What should be the approach of a coach to international groups?
Where are potential felds of work as a coach?
Evaluation strategy
The project was planned as a pilot supported by Federal funds. As a result, the Steering
Group decided to make an evaluation of the entire training course. Therefore, a special
evaluator worked with the whole group. During 2000 – 2002, all the actors of the course
participated in the evaluation process.
The evaluation itself was based on the theory of Kirckpatrick. His evaluation model is
based on four levels of learning and behaviour
77
:
Reaction: Personal refection by participants on the satisfaction of participants, the
effect and utility of the training programme, for example
Learning: Growth of knowledge, learning achievements
Behaviour: Changes in behaviour, transfer of competencies into concrete actions/
situations
Results: Long term transfer (also in organisational and institutional terms)
This model by Kirkpatrick worked out very well for this training course. All levels were
implemented. Aspects, belonging to the level of reaction and learning, like the atmo-
sphere, teamwork, contents, methods, etc. were already included in the training course.
However, the long term results could not be known immediately. As a result, participants
were contacted again after one year. Each participant was contacted by telephone for an
interview.
76 Coaching is an ongoing professional support and orientation process. Through the process of coaching,
coaches deepen their learning and improve their performance. Source The international coach federation
http:www.coachfederation.org
77 Kirkpatrick, Donald L.: Evaluating Training Programs. The four levels. San Francisco 1998







www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
95
The following table demonstrates the different stages of the evaluation and the timetable
for its implementation. The evaluation with the whole group was completed one year after
the project.
Elements of evaluation,
Time frame
Issue/objectives
of Evaluation
Instruments
of observation
Participants
Mid-term evaluation and fnal
evaluation
Every module
Self assessment of participants
At the beginning, during and at
the end of the seminar
External Assessment by
facilitators, participants, tutors,
holders of project
During and after fnal seminar
Long term evaluation -
Interview. 1 year after the
qualifcation
Satisfaction
Quality of modules
Effectiveness of learning tools
Level of knowledge
Expectations for next modules
Understanding of the concept
Coach’s role
=> reaction, learning
Personal competencies
Possible transfer
Understanding about a Coach’s
role
=> learning, sometimes
behaviour, results
Personal competencies
Overall satisfaction
Quality of modules
Effectiveness of learning tools
Level of knowledge
Coach’s role
Increased competence
Changes in Behaviour
Transfer
=> reaction, learning,
behaviour, results
Questionnaires
Practice target
Complaints box
Group visual evaluation
methods: i.e. “the tree of
perception”, “barometer of
mood”, “water level”, “fever
curve”
Verbal evaluation methods: i.e.
“brainstorm”, “the reporter”,
“hanging-lamp-feedback”
Video interviews at beginning
of seminar chain
Profle of competencies
Turn of feedback rounds
Small talks with tutors
Turn of feedback rounds
Small talks with tutors
Questionnaires to the project’s
stake-holders at the end of the
project
Partially structured interviews
Facilitators / Tutors
Mid-term and fnal
evaluation
Timing is dependent on the
availability of the tutors
personal view on professional
competencies of participants
Feedback
Individual consultations with
tutors
Participants’ sending organisations
Starting interview
Between the “Kick-Off ”
and the 2nd module
Final interview
One year after the fnish of
training course
Importance of trained compe-
tencies and knowledge,
Agreement with total concept
Possible transfer
Working felds for Coaches
Long term development of
competence among participants
Transfer of knowledge into the
working feld (including to
institutions), etc.
=> learning, behaviour, results
Questionnaires
Questionnaires
96
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Results: a second course and next steps
All parties were involved in the evaluation. This gave the organisers of the project very
detailed information about what they would have to change in future if they wish to
repeat the course.
Some of the main conclusions of this evaluation were
Clarifcation of the defnition of a coach
78
has to take place:
In the next training course the team should agree on one common defnition and
stick to it during the whole course.
Elements and themes of the training course
Most participants agreed that there is a need for more detailed attention to be paid
to learning about communication and coaching during any future course. The model
of oriented coaching has to be given more space in any future course.
Evaluation of the evaluation
Participants gave feedback that the evaluation itself took too much time and space
in the training course. That happened because participants were not always clear
about why and what certain kind on evaluation was needed for. The next course
team should avoid this.
The following year, a second training course was designed on the same subject, but
following very closely the recommendations made after the frst cycle.
The “good” fnal evaluation questionnaire
At the end of this evaluation practices section, we would like to present a possible struc-
ture for a fnal evaluation questionnaire. It is not the result of a concrete project, but
rather of the accumulated practices of several experts in different projects concerning
fnal evaluation questionnaires. This “good” fnal evaluation questionnaire is comple-
mentary to the explanations of working with questionnaires already outlined in this T-Kit
(see pages 57-61). In this case, the adjective “good” has no normative meaning. It is
used in opposition to the “bad evaluation questionnaire” exercise on page 60. We hope
this fnal evaluation questionnaire will be a source of inspiration for developing your own.
This questionnaire is presented in two parts, each one corresponding to a column. On
the left you can fnd an example of questions that can be asked in a questionnaire and
on the right you fnd an exploration of “what lies behind”. This is simply an example.
But, we have tried to propose examples that can illustrate some important considerations
for the construction of evaluation questions. We would like to encourage you to refect
on the options, the priorities, the advantages and disadvantages of each question. We
hope that this refection will help you to design your own questionnaire, making informed
choices and formulations for the questions you ask.
We should not be afraid of “fnal evaluation questionnaires”. At frst glance, it may be
intimidating to have to fll four empty pages up with evaluation questions. But, experi-
ence does demonstrate that participants of educational activities take flling in the eval-
uation questionnaire very seriously and often even enjoy it. In fact it is quite often one
of the highest moments of refection, learning and participation.
The fnal evaluation questionnaire is often just one part of the fnal evaluation. Even if
it tries to review a lot of issues, it should be conceived one of several evaluation tools,
including for example, group interactive methods or group discussions.
78 The person in charge of the coaching process is called a coach. Source The international coach federation
http:www.coachfederation.org



www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
9I
Without pretending to cover every aspect, we have tried to refect our approach to evalua-
tion (evaluation as a total experience) in this evaluation questionnaire. The questionnaire
contains there is a combination of evaluation “by objectives”, “by process” and “by per-
formance”. We recommend that you consider refecting on this questionnaire while consult-
ing the ideas outlined in the sections of this T-Kit entitled “why to evaluate” (Pages 15-16)
and “what to evaluate” (Pages 19-28). We have tried to embrace the complexity of an edu-
cational activity.
This example has been designed for an educational activity in general. It can and it should
be adapted to the kind of activity evaluated (an exchange, a training course, a workshop…)
and to each particular context. It is just a tool of inspiration. The most important is to
create something new out of it.
Final evaluation
questionnaire
Comments/ Logic /
What is behind the question?
Title of the Educational Activity
Place and Date
Evaluation questionnaire
Do not forget to clearly identify your
questionnaire! Too many
are not specifc!
Name (optional): On the one hand, it is necessary to respect
the anonymity, if desired, of the par-
ticipant. On the hand, knowing the iden-
tity of the participant will help to better
understand and contextualise the indi-
vidually flled out questionnaires.
General Impression
What is your general level of satisfaction with the
activity?
nnnn
0% 50% 100%
Please explain the score you have given:
Other comments:
It is good to start by asking for a gen-
eral impression of the activity, a kind of
“taste”, combining qualitative and quan-
titative evaluations. It is a simple but
very revealing starting point. It provides
a kind of “overall general average level
of satisfaction” for each participant.
Objectives
In your opinion, to which extent were
the objectives of the activity reached?
++ + o -- -
Objective 1: To… Very Good Ok Bad Very
good bad
Please explain
your score?
Objective 2: To… ++ + o -- -
Please explain
your score?
Evaluation by objectives (Page 21) is a
relevant component of a global evalua-
tion. It is appropriate to combine qualita-
tive and quantitative evaluation.
Remember to remind the participants of
the objectives of the activity that you
want them to evaluate.
This part provides more specifc informa-
tion than a general impression. Even if
at times the answers will mix both these
impressions, it is important to differen-
tiate them.
This part will also help you to put into
perspective the following more specifc
parts of the questionnaire that come
later.
The quantitative scale (++, +, o, -, --)
should be explained at the beginning.
Numeric scales are sometimes confus-
ing: in some countries the scales are
from 1-10, in others from 1-5, in some
countries the 1 is the best, in others 5
is the best… That is the reason why this
symbol scale is an interesting alterna-
tive.
98
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Expectations
What were your initial implications?
Were your expectations fulflled?
++ + o -- -
Professional
Personal
Which of your expectations have been fulflled
the most? Please explain your answers.
Which of your expectations have been fulflled
the least? Please explain your answers.
Learning outcomes
What are the most important things you have learned
in at this activity?
Methodology
Please comment on the appropriateness
of the methods used during the activity?
Evaluating the extent to which expecta-
tions have been fulflled will allow you
to see the extent to which the activity
was relevant to the target group.
As we have already seen (Pages 26-27)
dealing with the expectations (together
with the objectives and the programme)
is part of evaluating “by process”: seeing
the process and the relationships.
Again, we use a mixture of quantitative
and qualitative evaluation. It is interest-
ing to differentiate between profession-
al and personal expectations because
in international youth work they can
differ a lot.
The two questions about the most and
the least fulflled expectations give us
some additional information. The main
reason for including them in the ques-
tionnaire is to “oblige” participants to
compare their original expectations to
what happened. Otherwise, at the end
of the activity, they might be “lost”,
“mixed with memories” or out of focus.
From “evaluation by outcomes” (Page 25)
the most relevant for educational eval-
uation are the learning outcomes for
participants. This question helps us to
fnd out, in general terms, what par-
ticipants learned during the activity.
Asking in general about the methods
will help us to evaluate the overall
methodology of the activity (variety,
combination, links between methods…)
Experience tells us that if we include
the word “methodology” in the formula-
tion of the question it can cause some
confusion, so it is often easier for par-
ticipants to answer a question about
the methods.
More specifc comments about each
method can be asked in the part of the
questionnaire devoted to the pro-
gramme elements
Final evaluation
questionnaire
Comments/ Logic /
What is behind the question?
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
99
Final evaluation
questionnaire
Comments/ Logic /
What is behind the question?
Programme elements
What do you think of the general fow of the
programme?
This part deals with the detailed
and analytical evaluation of each
programme element. It is useful to
order the programme elements
chronologically as they have
appeared in the programme and to
use titles which help participants to
easily remember them.
For each programme element there
is a line for quantitative evaluation
of the learning and the enjoyment.
This distinction has advantages and
disadvantages but the contrast
between them can be interesting and
noteworthy. A line for comments
allows for qualitative evaluation of
each pr ogr amme el ement .
Participants can qualify their quan-
titative evaluation and add other
refections they consider important.
Having evaluated the whole pro-
gramme chronologically, partici-
pants have an overview of its fow.
The links and connections between
the different programme elements
are very important for the learning
process. This is the reason why the
programme fow is an important
and relevant aspect to evaluate.
The “actors”
How do you evaluate the role of the group in your
learning?
How do you evaluate the dynamic in the group?
How do you evaluate the work of the team?
How do you evaluate your own contribution to
the activity?
The group and the team in non-formal
education are not only a context but
also a source of individual learning.
For this reason it is very important
to evaluate them in relation to the
level of participation of members of
the group.
By doing so, we provoke a “trian-
gulation” of actors, exploring their
roles, interaction and dynamics.
Those are at times key factors of
success or of failure; a bad relation
between the participants and the
team, for example, can provoke the
failure of a, in principle good pro-
gramme.
These aspects are closely linked to
“evaluation by performance” (Page
24)
Learning How did you
achievements enjoy it?
Programme
++ + o - - -

Element
Welcome
session
Comments:


IUU
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Follow-up
How are you planning to use this experience in
your future work with young people?
At the individual level
(personally / professionally)?
In your organisation:
With other partners:
Please describe your plans for any future projects
you may now have?
This part evaluates the “outcomes”
of the activity beyond individual
learning. See “Evaluation by
outcomes” (Page 25). It might be
interesting to distinguish among
different levels of outcomes
(individual, within the organisation,
in co-operation with others) or to
ask specifcally about future projects
if this was an expected outcome.
For most participants, the real
outcomes will only become clear
after some time has passed. The
answers to this question will,
therefore, refer mostly to plans or
intentions. But, even if from the
“factual” point of view the answers
will not always be valid, it is still
important to include a question
about this in the fnal evaluation
questionnaire. Refecting on this will
help participants to start to transfer
what they have learned to their
home context. This is one of the
clearest examples of the learning
and motivating nature of educational
evaluation.
Preparation
In your opinion, was the preparation of this activity
adequate?
In terms of contents:
In terms of technical information:
In term of other practicalities:
At the end, you can come back to
the beginning. In other question-
naires, we can fnd the evaluation
of the preparation of the activity at
the beg inning. We fnd it interest-
ing to ask participants about it at
the end. After going through all the
steps of the process participants
can imagine “How things would
have been if the preparation would
have been done differently?”. This
is, for us, the key question.
Logistics and technicalities
How do you assess the quality of the logistical
organisation of the activity in terms of:
Accommodation:
Food:
Working rooms:
Facilities (internet access, free time possibilities…):
Equipment:
Technical information before arrival:
The evaluation of logistics and
technicalities allows for the
improvement of future activities.
Those aspects also have a relevant
infuence in the learning process,
because they infuence the comfort,
concentration, sense of security,
etc.
Final evaluation
questionnaire
Comments/ Logic /
What is behind the question?
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
IUI
Other
Please share any other comment you would like
to make?
Do you have any suggestions to improve the
quality of this activity you would like to share
with the organisers of future activities?
Leave enough space for answers to
this question and if you do not have
enough space, invite to use any other
extra space available.
If it is clear that a similar activity will
take place (i.e. next year) you can ask
for suggestions.
Thank you for your co-operation! Filling in an evaluation questionnaire
is an intensive exercise. It is fair, not
just polite to recognise it and thank
participants for their attention and
active participation.
Final evaluation
questionnaire
Comments/ Logic /
What is behind the question?
IU2
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
2.10 We still have to cook our meal!
We have gone through the different stages of evaluation going, through most of them
in some depth, and through different projects in which educational evaluation played an
important role.
Educational evaluation is considered one of the most complex and challenging aspects
of a project. It involves a lot of aspects and dimensions!
We would like to close this practical chapter of the T-Kit with an invitation: Be ambitious
but realistic!
We can plan a wonderful evaluation, where all elements are covered
and many different methods are used, but evaluation takes time and
resources. Developing good questionnaires reading them and making
conclusions out of them takes time. Doing interviews can be a very
good method but is time-consuming. There are loads of excellent
group exercises for evaluation, but they only make sense when you
note down the outcomes and work with the results.
In other words, we should not overdo it. After considering the complexity
of evaluation, we should remain realistic. We should reserve the
appropriate time and human resources to do it paying attention to how the evaluation
relates to the rest of the project.
Evaluation techniques and methods are not so diffcult to learn. The diffculty comes in
their application. A tool is as good as the person using it. Dealing with evaluation requires
refection, competence and experience.
The theoretical and practical ideas in this T-Kit offer a starting point. The most important is
to remember that the educational evaluation, to be of a high quality, should be implemented
with maximum respect to the nature of each project.
At this stage, we hope not to disappoint you but we still have to cook our meal!
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
IU3
Evaluation standards
During the 1990’s a number of evaluation societies were founded in Europe. The German
Evaluation Society (DeGEval)
79
and the Swiss Society of Evaluation (SEVAL)
80
, have published
27 evaluation standards
81
organised in four categories:
• Utility:The utility standards are intended to ensure that an evaluation is guided by both
the clarifed purposes of the evaluation and the information needs of its intended users.
– Stakeholder identifcation
– Clarifcation of the purposes of the evaluation
– Evaluator credibility and competence
– Information scope and selection
– Transparency of values
– Report comprehensiveness and clarity
– Evaluation timeliness
– Evaluation utilisation and use
• Feasibility: The feasibility standards are intended to ensure that an evaluation is planned
and conducted in a realistic, thoughtful, diplomatic, and cost-effective manner.
– Appropriate procedures
– Diplomatic conduct
– Evaluation effciency
• Propriety: The propriety standards are intended to ensure that in the course of the
evaluation all stakeholders are treated with respect and fairness.
– Formal agreement
– Protection of individual rights
– Complete and fair investigation
– Unbiased conduct and reporting
– Disclosure of fndings
• Accuracy:The accuracy standards are intended to ensure that an evaluation produces and
discloses valid and useful information and fndings pertaining to the evaluation questions.
– Description of the evaluation
– Context analysis
– Described purposes and procedures
– Disclosure of information sources
– Valid and reliable information
– Systematic data review
– Analysis of qualitative and quantitative information
– Justifed conclusions
– Meta-evaluation
79 The German Evaluation Society (DeGEval e.V.) was created 1997. It is an association of persons and insti-
tutions, who are active in the feld of evaluation. Its aims are to promote professional evaluation practices,
the sharing of different perspectives on evaluation as well as the information and exchange about it. More
information at http://www.degeval.de.
80 The Swiss Society of Evaluation (SEVAL) understands itself as forum for the exchange over evaluations
between politics, administration, universities, NGO’s and consultants. It is a multidiscipline organization.
It promotes the quality and spreading of evaluation. More information at http://www.seval.ch/de/index.cfm.
81 Available at http://www.degeval.de/index.php?class=Calimero_Webpage&id=9023
3. Further exploration
IU4
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Application of the standards
These standards can help the evaluation team ensure the utility, feasibility and ethical
grounding of the evaluation being conducted. They can be used in the planning of the
evaluation, as a check list, or during the evaluation process for adapting or readjusting it.
In the application of these evaluation standards we should consider that:
The DeGEval standards are designed to uphold and raise the quality of evaluations.
They formulate key points which evaluators should respect and goals they should
pursue. They are intended to provide a frame of reference for conducting and assessing
evaluations.
How they are implemented is a deciding factor. It cannot take place schematically.
These evaluation standards are not intended to devalue evaluations which do not
meet a particular standard in a certain way.
Some standards will not be applicable to certain evaluations. Brief grounds for not
applying a standard should be provided in such a case.
The evaluation will often need to weigh up the advantages and disadvantages of
various alternatives. It is not always possible to give all standards equal consideration.
From time to time the standards might contradict each other. The evaluation team
and all participants have the job of fnding an appropriate solution which takes
account of the purposes and context of the evaluation in hand.
The standards apply to evaluations in general, but not to the evaluation of individuals,
for example in performance judgement processes or employee assessment (staff
evaluation is excluded).
Exercise on Evaluation Standards
You are planning a 5 day seminar with 20 youngsters from Russia, Hungary and Algeria.
One of your chosen evaluation methods is a personal interview with each partici-
pant. You have planned 5 minutes per person for the interviews. You have one staff
member responsible for conducting the evaluation interviews. The seminar pro-
gramme allows you a maximum of 15 minutes per day every morning for evaluation.
Check this evaluation plan against the four categories of evaluation standards: Does
it meet them all? How and why?
Based on this, think about which circumstances are necessary for a “good” evaluation?
Solution:
Utility: Yes, interviews give us the most personal contact with the participants and team.
Feasibility: With a group of 20 and 5 days, you would have to conduct interviews with
4 persons per day. Check the time frame: you have only 15 minutes per day, so you
can conclude that individual interviews cannot be done properly. Another solution
could be to do mini-group interviews. But, then you need very clear questions.
Propriety: You are only one person evaluating. You can not take notes and ask
questions in the 15 minutes available. You can ask the interviewees if they agree
to use a Dictaphone. Doing interviews means to take note of the statements of
participants. You need this information to document interviews properly.
Accuracy: In your group, there are participants with very different social and religious
backgrounds. Or the group is quite homogenous. Think about the questions you want
to ask before going into the interviews. If you would like to fnd out something
related to personal, ethical or religious aspects, please proceed with the necessary
sensitivity and think about how the context of a mini-group interview changes the
way participants may react to the questions.






www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
IU5
A tool for developing your own evaluation plan
– The SALTO checklist
This document is a map or framework on which you can base an evaluation of a project.
It will help you from the moment that you start thinking about aims and objectives until
the presentation of your evaluation report. It is only a suggestion; please feel free to adapt
it according to the nature of your project.
It has been split into 5 sections:
Pre stages
Development and Strategy
Selection
Activities
Outcomes
Each section has basic questions to ask in this area, methods for collection of information
and some hints on analysis. There is a box to tick when you have completed each
one.
You can use Section 1 before you start planning your project to help you think through
the processes needed. All other Sections should be followed during and after your
activity. There is a timetable which you can refer to when to use which Section and its sub
sections.
SECTION 1: Pre Stages
A set of questions that should be answered when you are creating your plan or activity.
SECTION 2: Development and Strategy
The areas to assess when you are evaluating the development and strategy of a project.
This is the important ground work to a report and recommendations for future changes.
SECTION 3: Selection
The areas to evaluate in the selection procedure of those involved, i.e. participants and
trainers. This makes transparent the process of involvement of people within the exercise.
SECTION 4: Activities
Explores key areas within the activities to evaluate.
SECTION 5: Outcomes
Explores evaluation of outcomes, demonstration of the impact of the project on the target
group and the wider community.
SECTION 6: Time Frame for Evaluation
A map of when each part of the evaluation should take place.





IU6
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
TIME WHEN THINGS SHOULD HAPPEN Which Evaluation Section to Complete
Following the Numbers from the Guide
STRATEGY DEVELOPMENT n 1 Pre Stages Check list
n 1 Decision on evaluation programme,
choice of methods and development
of tools.
n 2.2 Expectations of different stakeholders
ACTIVITY START TIME n 2.2 Expectations of participants
n 5.2 Learning needs of participants
DURING ACTIVITY n 3.1 Are participants your target group?
END OF ACTIVITY n 2.2 Expectations of participants met
n 4.2 Infrastructure and support by
participants
n 4.3 Content and methods by participants
n 5.2 Learning outcomes of participants
TEAM EVALUATION AFTER ACTIVITY n 2.2 Expectations of team met
n 2.3 Co-operation and communication
by team
n 4.2 Infrastructure and support by team
n 4.3 Content and methods by team
n 5.1 Evaluation of outcomes by team
n 5.2 Learning of participants by team
SHORTLY AFTER n 2.3 Co-operation of partners
n 3.1 Selection of participant
n 3.2 Selection of team
By 6 MONTHS n 2.1 Strategy Analysis
n 2.2 Expectations of stakeholders met
n 4.1 Preparation, delivery, evaluation
and follow up
n 5.2 Collection of examples of good
practice and new methods
n 5.3 Individual activity Evaluation
Report complete
n 5.5 Participants follow up of activities
By 12 MONTHS n 5.3 Evaluation of whole project
complete.
n 5.5 Participants follow up of activities
n 5.6 Long term impact
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
IUI
Before starting:
Read through this document to the end and consider all the points before you begin
your whole project and the individual activities within it.
Write a schedule of what evaluation will take place, when it will happen and who
will do it.
When you are completing your evaluation, follow all sections below very carefully.
1. Pre Stages
Answer these questions when you are constructing your project plan.
Tick the boxes on completing the questions.
n – What are the aims and objectives for your project and why?
n – What are all the stakeholders’ expectations (funders, youth leaders, volunteers,
young people etc.)
n – Who is/are your target group(s) and why?
n – How will you gain access to this/these target group(s)?
n – Who will run your activities and on what criteria have you selected them?
n – What methods and resources will you use to fulfl your aims and objectives
and why?
n – What outcomes do you hope to achieve?
n – How will you assess these outcomes?
2. Development and Strategy
2.1. Strategy Analysis
n complete
QUESTIONS
– What were the aims and objectives of the activity/project?
– Why were these aims and objectives chosen?
– Were the activities/strategies based on analysis of needs of young people?
– Were the activities/strategies based on the needs of youth workers?
– Were the activities/strategies based on the priorities of the funders (if any)?
– Were the activities/strategies based on the priorities of your organisation?
– What was the process of deciding on the strategy?
– Who was the target group and why was this group chosen?
– Was the activity/strategy manageable?
METHODS
– interview or questionnaire of decision makers
– reading of reports from relevant meetings
– reading of needs analysis research completed
ANALYSIS
– Develop an understanding of the process of construction of the aims and objectives!











IU8
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
2.2. Expectation of key people who have an interest in your activity
n complete
QUESTIONS
– Who were the key stakeholders (key players)?:
a) Funders e.g. European Commission/ National agencies/local authorities
b) Target group e.g. youth workers/ disadvantaged young people
c) Actors in the strategy e.g. trainers, volunteers, facilitators
d) Actors in the feld e.g. Youth Organisations
– What was the relationship between the stakeholders?
– Were their expectations met?
NOTE:
– Each stakeholder needs to be involved in the evaluation to provide a balanced
picture. All evaluations in every feld of work are political and you must be aware
of the context in which you complete your work.
METHODS
– Questionnaires, focus groups or interviews to all stakeholders before the activities
take place.
– Documentation and minutes from meetings.
ANALYSIS
– Decide what the expectations of these differing sets of people towards your project
are and towards each individual activity within it. Look at tensions or differences.
2.3. Co-operation with partners
n complete
QUESTIONS
– Which partners were you cooperating with on your project?
– What roles did the different partners play in the
a) development
b) process
c) evaluation of the different activities?
– Was there equal co-operation in the development of the activity between all partners?
– Was the communication between partners adequate?
– Was the communication between all actors (i.e. funders, partners, trainers, participants)
involved adequate?
– Were there misunderstandings through different use of words (defnitions) in the
aims, objectives or setting the target groups? If yes explain what happened.
METHODS
– Interview, focus groups or questionnaires of the actors involved.
ANALYSIS
– Use the data to decide if there was a democratic process in the development of
the project and to draw a picture of the involvement of the different partners in
the activities.
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
IU9
3. Selection
3.1. Selection of participants
n complete
QUESTIONS
– Who were the intended participants for the activities?
– Was there a shared understanding of the target group?
– Who was the announcement of the activity sent to?
– Who applied?
– What were the selection criteria?
– Were the participants who turned up to the activity your specifed target group?
METHODS
– Interview, focus groups or questionnaire of actors involved in selection
– Interview a number of participants and trainers to fnd out about the participants
(this could be done by telephone)
ANALYSIS
– Use the data collected to decide if you reached your target group in your activities
and if not where the problems arose!
3.2. Selection of actors to carry out the activities
n complete
QUESTIONS
– How were actors (such as trainers, volunteers) selected and on what criteria were
they selected?
– Did the actors implement the activity you suggested?
– Was there a gender balance?
– Was there a regional balance?
– Was there a balance in the areas of experience and expertise?
– Did the selected actors become a team?
– Was the team effective at organisation and time management?
– Were there conficts and were they resolved?
– Would you recommend the same actors for future activities?
METHODS
– Interview, focus groups or questionnaire of actors involved in selection
– Questionnaires of participants after the training course
– Questionnaires to the team after the training course.
ANALYSIS
– Use the data to decide if appropriate actors were selected for the tasks and, if a
team was created, whether it worked successfully together!
IIU
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
4. The activity
4.1. Preparation, delivery, evaluation and follow up
n complete
QUESTIONS
– Were adequate resources of time, energy and money given to the:
a) preparation of each activity?
b) delivery of each activity?
c) evaluation of each activity?
d) follow up of each activity?
METHODS
– Interview or questionnaire of actors involved
ANALYSIS
– Decide if enough commitment and total resources were spent on each element
within the production of the activity!
4.2. Infrastructure and Support
n complete
QUESTIONS
– Did the location for the activity have enough or appropriate space?
– Was the appropriate equipment available?
– Was there adequate support staff for the activity?
METHODS
– Questionnaire after training course for participants
– Interview or questionnaires for actors who ran the event.
ANALYSIS
– Examine the data to decide if the infrastructure and support were satisfactory for
the activity!
4.3. Contents and Methods
n complete
QUESTIONS
– Which priorities was the activity orientated towards?
– Were the methods chosen participant centred? If yes, give an example.
– Were the contents of the activities based in the realities of the participants? If yes,
give an example.
– Was there an effective group learning process?
– Were the participants actively involved in the activities?
– Did the activities use resources from the feld?
– Did the content and theme of the/each activity ft the aims and objectives set?
– Did your project have an impact on more people than the participants involved?
METHODS
– Observations: have an observer sit and watch the process and make notes (external
evaluator).
– Ask questions on the process in a post questionnaire or interview.
– Ask those people creating the activity to provide detailed documentation (daily
reports by the participants, detailed programme, etc.).
ANALYSIS
– Use the data to decide if the appropriate contents and methods were used!
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
III
5. Outcomes
To explore outcomes each individual activity needs to be examined separately and then
brought together in a synthesis.
5.1. Evaluation
n complete
QUESTIONS
– What were your expected outcomes from the individual activity?
– Were your expectations of outcomes met?
– What evaluation was completed for the individual activity?
– What were the results of the evaluation?
– How do the results of the evaluation and outcomes ft into the feld of youth work
and the priorities of the project, your organisation, funders, etc.?
METHODS
– Interview or questionnaire for the coordinator of each activity.
5.2. Learning Outcomes
n complete
QUESTIONS
– What skills and competencies have the participants gained, e.g. Self Awareness,
Intercultural Learning, Communication Skills, Team Work, etc.?
METHODS
– Assessment of learning of the participants by: themselves (self assessment), their
peers, the team or by external assessors. This assessment should be comparative
between when the participants arrive and where they have reached by the end,
i.e. a pre-questionnaire and a post-questionnaire or another method of recording
and analyzing the achievements of participants. New technology methods could be
used providing evidence based portfolios of achievement on CD ROMs and videos.
5.3. Demonstrating Quality
n complete
QUESTIONS
– How can quality be demonstrated?
METHODS
– The Evaluation Report is a tangible way of demonstrating the quality of your
activities. The evaluation report will help you improve your activities next time.
Using this guide will be a start to producing an evaluation of individual activities
and your overall project. Use the different title headings for each section of your
report and then fnish with a conclusion containing recommendations for the future.
If you need help you can request the support of an external evaluator to carry out
the task. External evaluators give your report more credibility because they are
understood as being more impartial.
An evaluation meeting. This brings all the key stakeholders together to discuss the
key questions that are in this guide. A report from the evaluation meeting should
then be produced. This is not a quick fx for all the areas of evaluation and is not
a substitute for the in-depth evaluation process given above.
II2
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
5.4. Added Value to the youth feld
n complete
QUESTIONS
– How do you demonstrate added value?
METHODS
– A collection of examples of good practice such as videos, CD ROMs, websites from
the different activities.
– The creation of new methods, tools and articles on the different themes of the
activities.
5.5. Multiplying effect
n complete
QUESTIONS
– How do you demonstrate multiplication?
METHODS
– Look for an increase in the number of activities in the priority areas that you have
stated in your aims and objectives.
– Look for an increase of trainers, support people and resources in priority areas.
– To fnd this information follow up participants 6 months or longer with questionnaires
and/or interviews to fnd out what activities relating to themes have been carried
out by participants.
5.6. Long Term Impact
n complete
QUESTIONS
The long term impact is quite diffcult to measure and will be affected by many
variables other than the youth activities that you are creating.
– What is the long term impact on the participants of an activity?
– What is the long term impact on the wider community?
METHODS
– Following up participants over 2 years or more can trace the long-term impact on
their lives and the activities they have been involved in. This can be assessed by
taking part in interviews or questionnaires every 6 months, recording their life
story.
– Look for indicators of impact on young people.
– Create specifc indicators for the aims and objectives/priorities of your project, e.g.
a) Reduction of reports of racism by young people towards their peers
b) Increased participation of young disabled people in youth activities.
c) Increased participation of young people in youth organisations.
NOTE
– It is diffcult to prove the link between your activities and results!
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
II3
Bibliography
COLEMAN, J. and others (1966)
Equality of educational opportunity, U.S. Department of H.E.W., Offce of Education,
Washington, D.C.
CHISHOLM, Lynne (2001)
Bridges of recognition, Recognising non-formal & informal learning in the youth
sector, Terminology Cheat Sheet, Innsbruck
DeGeval (2001):
Summary of Evaluation Standards, German Evaluation Society Standards, Wolfgang
Beywl, Cologne, Germany and Sandy Taut, Los Angeles, USA, pp. 47-50
DE MIGUEL, M. (1989).
Modelos de investigación sobre organizaciones educativas, Revista de Investigación
Educativa, 7 (13), pp. 21-56.
DIN-Taschenbuch (2003):
Normen zum Qualitätsmanagement, 4. Aufage, Beuth Verlag Berlin, Wien, Zürich
ECIS (1987).
Evaluation and Accreditation. London: European Council of International Schools.
Eurodesk: Eurodesk Quality process, Part I: Evaluation 2004 – 2005
GARCÍA HOZ, V. (1975).
Una pauta para la evaluación de centros educativos, Revista Española de Pedagogía,
(130), pp. 117-150.
GARCÍA RAMOS, J. M. (1989).
Extensión de la evaluación. En R. Pérez Juste y J.M. García Ramos, Diagnóstico,
evaluación y toma de decisiones. Madrid: Rialp, pp. 315-385.
GOODMAN, P. y PENNINGS, J. (Eds.) (1977).
New perspectives on organisational effectiveness. San Francisco: Jossey-Bass.
GUBA, E. G. y LINCOLN, Y. S. (1981).
Effective evaluation. San Francisco: Jossey-Bass.
JOINT COMMITTEE on Standards for Educational Evaluation (1981).
Standards for evaluations of educational programs, projects, and materials. Nueva
York: McGraw-Hill.
JORESKOG, K. G. (1978).
Structural analysis of covariance and correlation matrices, Psychometrika, (43), pp. 443-477.
LAWLER, E. y otros (Eds.) (1980).
Organisational Assessment. Nueva York: Wiley Sons.
LINN, R. L. (1986).
Quantitative methods. En M.C. Wittrock (Ed.), Handbook of research on teaching,
Nueva York: Macmillan, pp. 92-118.















II4
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
LOEHLIN, J. C. (1987).
Latent variable models, LEA, Hillsdale, N.J.
LONG, J. S. (1987).
Covariance structure models. An introduction to LISREL, Beverly Hills, CA: SAGE.
U.S. Department of H.E.W., Offce of Education, Washington, D.C. (2000)
MCMAHON, A. y otros (1984).
GRIDS Handbook. Londres: Longman.
NATIONAL Council on Educational Statistics (1991).
Educations counts: An indicator system to monitor the nation’s educational health,
U.S. Department of Education, Washington, D.C.
NEVO, D. (1983).
The conceptualization of educational evaluation: An analytical review of the literature,
Review of Educational Research, 53, pp. 117-128.
NEVO, D. (1994).
Combining internal and external evaluation: A case for school-based evaluation,
Studies in Evaluation, 20, pp. 87-98.
NORTHWEST Regional Laboratory (1985).
Building Level Effectiveness Survey, N.R.E.L., Portland.
OAKES, J. (1989).
What educational indicators? Educational Evaluation and Policy Analysis, 11 (2), pp. 181-199.
PEDHAZUR, E. J. (1982).
Multiple regressions in behavioural research: Explanations and prediction. Nueva
York: Rinehart and Winston.
Quality criteria/ Standards Broschure and CD-Rom “QS-Kompendium” (2000)
Broschure No. 24, p. 75, Bundesminsterium fuer Familie, Senioren, Frauen und
Jugend (BMFSFJ)
REED, D. and others (1987).
Social control and the qualitative and quantitative. Communication in the Annual
Meeting of the American Educational Research Association, Washington, DC.
RICHARDS, C. (1988).
A typology of educational monitoring-systems. Evaluation and Policy Analysis, 10 (2),
pp. 106-116.
ROSALES, Carlos, (1990).
Evaluar es refexionar sobre la enseñanza (Madrid: Nancea)
SANCHO, J. (1993).
Evaluar, conocer, transformar y mejorar. Aula de Innovación Educativa, (6), pp. 47-51.
SANTOS, M. A. (1993).
La evaluación: un proceso de diálogo, comprensión y mejora. Archidona, MA: Aljibe.
SCRIVEN, M. (1967).
The methodology of evaluation. En R. E. Stake (Ed.), AERA Monograph Series on
Curriculum Evaluation N.° 1, Rand McNally, Chicago.
SOLER, E. (Coord.) (1993).
Fundamentos de supervisión educativa. Madrid: La Muralla, S.A.
STUFFLEBEAM, D. L. (1983).
The CIPP model for programme evaluation. En G.F. Madaus, G. F. y otros (Eds.),
Evaluation models. Boston: Kluwer-Nijhoff, pp. 117-141.
STUFFLEBEAM, D. L. (1991).
Evaluation guide for evaluations of programs, services, and organisations. Kalamazoo,
MI: CREATE.




















www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
II5
STUFFLEBEAM, D. L. and others (1971).
Educational evaluation and decision making. Peacock, Itasca, Ill.
TENBRINK, T. & Cooper, J. M. (2003).
An Educator’s Guide to Classroom Assessment Boston, Houghton Miffin.
U.S. DEPARTMENT OF EDUCATION (1991).
America 2000: An education strategy Department of Education, Washington, D.C.
WALBERG, H. J. (1976).
Psychology of learning environments: Behavioural, structural or perceptual?.
En S.L. Schulman (Ed.), Review of Research on Education, (4), Itaska, IL: Peacook.
YORKE, D. (1987).
Indicators of institutional achievement: some theoretical and empirical considerations.
16 (1), pp. 3-20.
T-Kits series
T-Kit 1:
Organisational Management
T-Kit 2:
Methodology in Language Learning
T-Kit 3:
Project Management
T-Kit 4:
Intercultural Learning
T-Kit 5:
International Voluntary Service
T-Kit 6:
Training Essentials
T-Kit 7:
Under Construction: Citizenship, Youth & Europe
T-Kit 8:
Social Inclusion
T-Kit 9:
Funding and Financial Management
http://www.training-youth.net/INTEGRATION/TY/Publications/T_Kits.html





II6
www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
Web-ography
In addition to the web – sites references of the text:
Directorate of Youth and Sport of the Council of Europe
www.coe.int/youth
YOUTH programme of the European Commission
www.europa.eu.int/comm/education/youth.html
Partnership in the Field of Youth between the Council of Europe and the European
Commission.
www.youth-partnership.net
European Youth Forum
www.youthforum.org
All Different – All Equal European campaign on Diversity, Human Rights and
Participation
http://alldifferent-allequal.info
The encyclopaedia of informal education
www.infed.org
Electronic Journal of Educational Research, Assessment and Evaluation
http://www.uv.es/RELIEVE/
European Organisation of Quality, Brussels
www.eoq.org
European Organisation of evaluation
www.Eureval.org:
International Youth- and Visitors’ Service
www.ijab.de/dija/ikl:
Centre for Programme Evaluation, Melbourne
http://www.edfac.unimelb.edu.au/EPM/CPE/
Evaluation in der Europäischen Kommission
http://europe.eu.int/comm/dg19/en/evaluation/index.htm
Evaluation of aid to non-member countries
http://www.europa.eu.int/comm/scr/evaluation/index.htm
International Evaluation Research Group
http://www.c3e.fr/Inteval/home.htm
OECD/PUMA – Performance Management (OECD)
http://www.oecd.org/document/27/0,2340,en_2649_34629_2088411_1_1_1_1,00.html















www.training-youth.net
www.training-youth.net
T-Kit on Educational
Evaluation in Youth Work
III
Notes
II9
Sales agents for publications of the Council of Europe
Agents de vente des publications du Conseil de l’Europe
BELGIUM/BELGIQUE
La Librairie Européenne -
The European Bookshop
Rue de l’Orme, 1
B-1040 BRUXELLES
Tel.: +32 (0)2 231 04 35
Fax: +32 (0)2 735 08 60
E-mail: order@libeurop.be
http://www.libeurop.be
Jean De Lannoy
Avenue du Roi 202 Koningslaan
B-1190 BRUXELLES
Tel.: +32 (0)2 538 43 08
Fax: +32 (0)2 538 08 41
E-mail: jean.de.lannoy@dl-servi.com
http://www.jean-de-lannoy.be
CANADA
Renouf Publishing Co. Ltd.
1-5369 Canotek Road
OTTAWA, Ontario K1J 9J3, Canada
Tel.: +1 613 745 2665
Fax: +1 613 745 7660
Toll-Free Tel.: (866) 767-6766
E-mail: order.dept@renoufbooks.com
http://www.renoufbooks.com
CZECH REPUBLIC/
RÉPUBLIQUE TCHÈQUE
Suweco CZ, s.r.o.
Klecakova 347
CZ-180 21 PRAHA 9
Tel.: +420 2 424 59 204
Fax: +420 2 848 21 646
E-mail: import@suweco.cz
http://www.suweco.cz
DENMARK/DANEMARK
GAD
Vimmelskaftet 32
DK-1161 KØBENHAVN K
Tel.: +45 77 66 60 00
Fax: +45 77 66 60 01
E-mail: gad@gad.dk
http://www.gad.dk
FINLAND/FINLANDE
Akateeminen Kirjakauppa
PO Box 128
Keskuskatu 1
FIN-00100 HELSINKI
Tel.: +358 (0)9 121 4430
Fax: +358 (0)9 121 4242
E-mail: akatilaus@akateeminen.com
http://www.akateeminen.com
FRANCE
La Documentation française
(diffusion/distribution France entière)
124, rue Henri Barbusse
F-93308 AUBERVILLIERS CEDEX
Tél.: +33 (0)1 40 15 70 00
Fax: +33 (0)1 40 15 68 00
E-mail: commande@ladocumentationfrancaise.fr
http://www.ladocumentationfrancaise.fr
Librairie Kléber
1 rue des Francs Bourgeois
F-67000 STRASBOURG
Tel.: +33 (0)3 88 15 78 88
Fax: +33 (0)3 88 15 78 80
E-mail: francois.wolfermann@librairie-kleber.fr
http://www.librairie-kleber.com
GERMANY/ALLEMAGNE
AUSTRIA/AUTRICHE
UNO Verlag GmbH
August-Bebel-Allee 6
D-53175 BONN
Tel.: +49 (0)228 94 90 20
Fax: +49 (0)228 94 90 222
E-mail: bestellung@uno-verlag.de
http://www.uno-verlag.de
GREECE/GRÈCE
Librairie Kauffmann s.a.
Stadiou 28
GR-105 64 ATHINAI
Tel.: +30 210 32 55 321
Fax.: +30 210 32 30 320
E-mail: ord@otenet.gr
http://www.kauffmann.gr
HUNGARY/HONGRIE
Euro Info Service kft.
1137 Bp. Szent István krt. 12.
H-1137 BUDAPEST
Tel.: +36 (06)1 329 2170
Fax: +36 (06)1 349 2053
E-mail: euroinfo@euroinfo.hu
http://www.euroinfo.hu
ITALY/ITALIE
Licosa SpA
Via Duca di Calabria, 1/1
I-50125 FIRENZE
Tel.: +39 0556 483215
Fax: +39 0556 41257
E-mail: licosa@licosa.com
http://www.licosa.com
MEXICO/MEXIQUE
Mundi-Prensa México, S.A. De C.V.
Río Pánuco, 141 Delegacíon Cuauhtémoc
06500 MÉXICO, D.F.
Tel.: +52 (01)55 55 33 56 58
Fax: +52 (01)55 55 14 67 99
E-mail: mundiprensa@mundiprensa.com.mx
http://www.mundiprensa.com.mx
NETHERLANDS/PAYS-BAS
De Lindeboom Internationale Publicaties b.v.
M.A. de Ruyterstraat 20 A
NL-7482 BZ HAAKSBERGEN
Tel.: +31 (0)53 5740004
Fax: +31 (0)53 5729296
E-mail: books@delindeboom.com
http://www.delindeboom.com
NORWAY/NORVÈGE
Akademika
Postboks 84 Blindern
N-0314 OSLO
Tel.: +47 2 218 8100
Fax: +47 2 218 8103
E-mail: support@akademika.no
http://www.akademika.no
POLAND/POLOGNE
Ars Polona JSC
25 Obroncow Street
PL-03-933 WARSZAWA
Tel.: +48 (0)22 509 86 00
Fax: +48 (0)22 509 86 10
E-mail: arspolona@arspolona.com.pl
http://www.arspolona.com.pl
PORTUGAL
Livraria Portugal
(Dias & Andrade, Lda.)
Rua do Carmo, 70
P-1200-094 LISBOA
Tel.: +351 21 347 42 82 / 85
Fax: +351 21 347 02 64
E-mail: info@livrariaportugal.pt
http://www.livrariaportugal.pt
RUSSIAN FEDERATION/
FÉDÉRATION DE RUSSIE
Ves Mir
9a, Kolpacnhyi per.
RU-101000 MOSCOW
Tel.: +7 (8)495 623 6839
Fax: +7 (8)495 625 4269
E-mail: orders@vesmirbooks.ru
http://www.vesmirbooks.ru
SPAIN/ESPAGNE
Mundi-Prensa Libros, s.a.
Castelló, 37
E-28001 MADRID
Tel.: +34 914 36 37 00
Fax: +34 915 75 39 98
E-mail: libreria@mundiprensa.es
http://www.mundiprensa.com
SWITZERLAND/SUISSE
Van Diermen Editions – ADECO
Chemin du Lacuez 41
CH-1807 BLONAY
Tel.: +41 (0)21 943 26 73
Fax: +41 (0)21 943 36 05
E-mail: info@adeco.org
http://www.adeco.org
UNITED KINGDOM/ROYAUME-UNI
The Stationery Office Ltd
PO Box 29
GB-NORWICH NR3 1GN
Tel.: +44 (0)870 600 5522
Fax: +44 (0)870 600 5533
E-mail: book.enquiries@tso.co.uk
http://www.tsoshop.co.uk
UNITED STATES and CANADA/
ÉTATS-UNIS et CANADA
Manhattan Publishing Company
468 Albany Post Road
CROTTON-ON-HUDSON, NY 10520, USA
Tel.: +1 914 271 5194
Fax: +1 914 271 5856
E-mail: Info@manhattanpublishing.com
http://www.manhattanpublishing.com
Council of Europe Publishing/Editions du Conseil de l’Europe
F-67075 Strasbourg Cedex
Tel.: +33 (0)3 88 41 25 81 – Fax: +33 (0)3 88 41 39 10 – E-mail: publishing@coe.int – Website: http://book.coe.int
CONSEIL
DE L'EUROPE
COUNCIL
OF EUROPE
In 1998, the Council of Europe and the European Commission decided to take common
action in the feld of European youth worker training. Both institutions therefore initiated
a Partnership agreement, whose aim was “to promote active European citizenship and
civil society by giving impetus to the training of youth leaders and youth workers working
within a European dimension”. This agreement has since been renewed several times
and additional agreements were initiated in the felds of “youth research” and “Euro-med
co-operation”.
Since 2005, all Partnership activities have merged into one single framework Partnership
agreement, which focuses on the following topics: European Citizenship; human rights
education and intercultural dialogue; quality and recognition of youth work and training;
better understanding and knowledge of youth; youth policy development. The Partnership
between the Council of Europe and the European Commission in the feld of youth thus
brings together the two institutions’ experience in non-formal education, youth policy,
youth research and youth work practice.
The co-operation between the two institutions covers a wide spectrum of activities and
publications, including the Training Kits. T-Kits are thematic publications written by
experienced youth trainers and constitute easy-to-use handbooks for use in educational
activities.
All Partnership activities and publications enhance the exchange of experience and good
practice between the actors involved, are embedded in a structured dialogue and contribute
to the implementation of the political objectives of both partners. Since 1998, more than
1.200 young people participated directly in these activities and many more were able
to beneft from this experience through the Partnership publications.
T-Kits are produced by the Partnership in English, French and German. Several of them
are also available in other languages following national initiatives. Please consult
the web-site www.youth-partnership.net for updated information on T-Kits and
translations.
Partnership Programme with the European Commission
Council of Europe - NGBEN
1, Quai Jacoutot, F-67075 Strasbourg Cedex
Tel.: +33 (0)3 90 21 50 57, fax: +33 (0)3 88 41 27 77
http://www.youth-knowledge.net
Educational Evaluation
in Youth Work
T-KIt N
o
10
v 12 / US$ 18
ISBN: 978-92-871-6023-2
-:HSTCSH=V[UWXW:
Eur opean Commi ssi on

T-Kit on Educational Evaluation in Youth Work

“Tasting the soup”



Welcome to the T-Kit Series
Some of you may have wondered: what does T-Kit mean? We can offer at least two answers. The first is as simple as the full Version in English: “Training Kit”. The second has more to do with the sound of the word that may easily recall “Ticket”, one of the travelling documents we usually need to go on a journey. In our Imagination, this T-Kit is a tool that each of us can use in our work. More specifically, we would like to address youth workers and trainers and offer them theoretical and practical tools to work with and use when training young people. The T-Kit series is the result of a collective effort involving people from different cultural, professional and organisational backgrounds. Youth trainers, youth leaders in NGOs and professional writers have worked together in order to create high quality publications, which would address the needs of the target group while recognising the diversity of approaches across Europe to each subject. The T-Kits are a product of the Partnership in the field of Youth between the Council of Europe and the European Commission. Besides the T-Kits, the Partnership has resulted in other areas of cooperation such as training courses, the magazine “Coyote”, research and youth policy activities and an Internet site hosting also the European Knowledge Centre for Youth Policy. To find out more about developments in the Partnership (new publications, training course and seminar announcements, etc.) or to download the electronic version of the T-Kits, visit the Partnership website: www.youth-partnership.net

Council of Europe Publishing – F-67075 Strasbourg Cedex ISBN 92-871-6023-2 © Council of Europe and European Commission April 2007 This document does not necessarily express the official view of the European Commission or the Council of Europe, their member states or the organisations co-operating with the institutions.
T-Kit on Educational Evaluation in Youth Work www.training-youth.net

Reproduction of material from this publication is authorised for non-commercial educational purposes only, provided the source is quoted.

Kerstin Giebel works as a “co-ordinator for quality development in international youth work” at IJAB e. Partnership Secretariat Hanjo Schild (Partnership Coordinator) Marta Me ´ska (Assistant) ˛dlin Dianna Osayande (Administration) T-Kit on Educational Evaluation in Youth Work www. He also works in the University of Osnabrück.org). She is qualified as an “European excellence assessor” (EFQM) and “Quality manager and auditor – social institutions”. Germany as a resource person for European Studies. anti-racism and xenophobia. He specialises in training of trainers. (International Youth Exchange and Visitor’s Service) a non-governmental organisation based in Germany. Slovakia (www. Ian Redpath of the British Council for his organisational and administrative support. social exclusion and integration. He has been the external evaluator of several youth projects and has completed post-graduate studies on “Educational Evaluation” in the University of Deusto. an international non-governmental youth organisation specialising in projects against the exclusion of young people.frankly-speaking. He works as a trainer and resource person on European youth policies. Since 2005.V. Actors Editorial Team Authors Paul Kloosterman is a free-lance trainer and consultant from the Netherlands. He is a member of European Youth Forum’s Pool of Trainers. training and publishing on quality related questions to national organisations and European networks. presently living in Italy. Intercultural Learning. Spain. living in Germany. Ozgehan Senyuva is from Turkey and has been involved in youth work since 1994. he has been the Vice-President of the Youth Express Network. intercultural learning. Currently he is completing his doctoral studies in Political Sciences. working with disadvantaged youngsters and international teamwork.training-youth. Acknowledgments Silvia Volpi for her precious contribution to the development of this T-Kit. www.net . Her tasks include consulting.nonformality. org. Editor in chief Miguel Ángel García López is an experienced free-lance trainer from Spain. From 2000 to 2005 she was an educational advisor at the Council of Europe’s European Youth Centre in Strasbourg. Language editor Yael Ohana works as a free-lance capacity building consultant and editor in the youth and development fields and is currently based in Bratislava.

training-youth.europa. France Tel: +33-3-88 41 23 00 – Fax: +33-3-88 41 27 77 European Youth Centre Budapest Zivatar ucta 1-3 – H-1024 Budapest.html T-Kit Coordinators László Földi co-ordinated the T-Kit planning process and composed the editorial team in co-operation with SALTO UK for the Partnership Secretariat. Partnership in the Youth field between the Council of Europe and the European Commission Council of Europe NGBEN 1. He currently works for the Hungarian National Agency of the YOUTH Programme.eu/youth/index_en. She was involved in producing the SALTO Evaluation CD Rom for youth projects in 2004. He is also working as trainer and consultant in youth work and youth policy issues with special focus on European Citizenship and Human Rights Education. Hungary Tel: +36-1-212-4078 – Fax: +36-1-212-4076 European Commission – DG Education and Culture Youth Policy Unit European Commission Directorate General for Education and Culture Directorate D. Quai Jacoutot F-67075 Strasbourg Cedex Tel: +33-3-90-21-50-57 Fax: +33-3-88-41-27-77 Web: http://www.youth-knowledge. Unit 1 B-1049 Brussels Tel: +32-2-299-1318 Fax: +32-2-299-4038 Web: http://ec. T-Kit on Educational Evaluation in Youth Work www.net Council of Europe European Youth Centre Strasbourg 30 Rue Pierre de Coubertin – F-67000 Strasbourg.net . Susie Green is the SALTO YOUTH Co-ordinator and works for the SALTO Resource Centre Network. Evaluation is one of the horizontal priorities of the network.

.......... The Ingredients of Educational Evaluation 1....................4 What to evaluate? ....................................................................9 Educational evaluation and quality in youth work 1............................ pans and spices” Practicing Educational Evaluation! Educational Evaluation in practice! 2..........................................net ..........................................................................................................5 A critical look at evaluation methods 2....... ...............1 What is Educational Evaluation? 1.........................................10 Develop your own evaluation! ........................................................ ...................................................................................... ..............................................................................................................5 Who and for whom? 1...................................... ......................6 When to evaluate? 29 30 32 35 38 47 ............................... ........................................................................................................................... ............ Table of contents Foreword ....4 Feedback .........8 Educational evaluation in a broader framework 1............................... 49 49 51 57 57 62 65 75 77 80 81 82 T-Kit on Educational Evaluation in Youth Work www...........7 Establishing evaluation criteria 2............................................................................................................................................................................................................................................ 1................... 1. Interpersonal methods Group interactive methods 2..... ...............training-youth................... ................................................2 A project within a project ....................................... .... .......................................................................2 Why evaluate? ..........3 What is evaluation for? 1.............. ........ 2....... “Pots.............6 Collecting and organising information 2................................................................................8 Reporting the outcomes .........1 How educational evaluation is built into a project? 2.............................................................................. .... 2............................... 7 9 11 13 13 15 17 19 Introduction The European policy context 1................................3 How to evaluate? Evaluation methods Personal methods .... ....... ................ 2........................................................................................................ ..................... 1..................................................................................7 Typologies of evaluation 1............... ................................ .....................................

...............................net ................................... 3. T-Kit on Educational Evaluation in Youth Work www........ 85 85 88 90 92 94 96 102 103 103 105 113 “Take5” – 40 youngsters evaluating EVS and Leonardo multidimensional evaluation Madzinga – External evaluation ........... A tool for developing your own evaluation plan – The SALTO checklist Bibliography and webography ................ Further exploration .............................................................................................................. ............training-youth..... .......... 2............. .......... ............................ Evaluation in an E-learning pilot course Overall evaluation of a Pilot Long Term Training Course The “good” final evaluation questionnaire 2................................... Evaluation standards ....10 We still have to cook our meal! ..................................................................................................... ............................9 Evaluation practices ........................................ ...........................................................................................................................................................................................................

“justifying your work to someone else”. In this T-Kit we want to show you that evaluation can be thrilling. when we work. reflective and organised and to share it with others. Foreword Evaluation is not something that people get immediately excited about. when we buy something. another friend does not T-Kit on Educational Evaluation in Youth Work www. In short: you have a wonderful evening. when we talk. exciting… or at least attractive and inspiring enough to integrate it into your educational work. Evaluation is like cooking! Let’s get cooking! Imagine you are having friends over for dinner. You go the market in the morning and buy all the ingredients and when you get home you immediately start to prepare. And then there was still the Tiramisu waiting for them. we process it. when we move. so for her the main course ended up being only rice and carrots. Often it is associatwed with “writing long reports”. “measuring the immeasurable”. All this is accompanied by a nice red wine. when and what we want to evaluate. Evaluation is something natural Evaluation is something that is natural for human beings. When again inviting friends for dinner. one that takes you through a nice evening. We do it all the time. You decide on the menu you want to serve: four courses starting with a wonderful soup full of vegetables. when we cook and we do it without reports and questionnaires. So you might ask yourself why should I read a T-Kit on something I already do every day? The answer is to make it conscious. We collect information. Our “everyday evaluation” we do all by ourselves and in our own way. electrifying. explicit. But. you resolve to first ask them about any dietary or other special needs they might have.net . then pasta with gorgonzola sauce followed by the main dish of chicken in curry sauce accompanied by rice and carrots. there is a good atmosphere and your friends really liked the food.training-youth. difficult: not exciting at all. looking back there are a few things that could be improved. Finally. For desert you intend to surprise them with your wonderful home made Tiramisu. Maybe both pasta and soup before the main course is a bit too much? One of your friends turned out to be vegetarian. Still. fun. Evaluation becomes Educational Evaluation. stimulating. You want to prepare them a really nice meal. We do it every day when we cross the street. etc. Boring. “filling out forms and questionnaires”. from the moment we work together with other people we have to agree on how. tiring. we give it a meaning and a value and we act or react according to it. great discussions. After the pasta most of your guests were already full and were a bit shocked when you served the chicken curry.

and so on. you resolve to buy some non-alcoholic drinks to offer to your guests along with wine. what would we consider a satisfactory or a good outcome? • How do we know that people are honest when we ask them if they are satisfied? • Which aspects of the evaluation results can be used when we go to the bank to ask for assistance in financing our plan to establish a restaurant? • And many more … T-Kit on Educational Evaluation in Youth Work www. off you go! And what is one of the first things you do? You start planning the evaluation! That is because you want to “try things out” and to see what the results of your experiment are. you want to evaluate? quality of the food satisfaction of the customers variety of the food price and cost of the services offered assortment of drinks you offer atmosphere during the evenings co-operation between you and your friend And how do we evaluate these things? • How do we know the customers like the food? By the tip they leave? Do we give them a questionnaire? Do we count how many people come back? • When is the atmosphere good? When people leave singing? When everybody is rolling about laughing? • Do we count costs and benefits every night or should we go for a more long-term approach? These reflections bring on other questions. You end up in long discussions with your friend about all kinds of things. drink alcohol and the only thing you could offer him was water. for what and for whom. experiment which with ideas for menus that could eventually be served in the restaurant. on Wednesdays and Sundays. when to do it. The idea is to start a restaurant. say. You did your own evaluation and next time dinner will certainly be better. Try to imagine another situation: you like cooking a lot and so does one of your friends. What do • the • the • the • the • the • the • the • …. You did this ‘evaluation’ without having a meeting. without questionnaires and you did not write a report you just did it. exploring its educational value. By doing so you can practice cooking. So. naturally. deciding about what you want to get achieve. Until then you want to try things out. in two years. Not immediately but. find out how the co-operation between the two of you works out. Next time. see if there is any way to earn enough for a reasonable living for two people. Your friend’s house has a room. which seems to be perfect for dining and her kitchen is quite good. like: • What do we (the two of us) mean by quality food? What is our standard? • What kind of atmosphere would we like to create? • When it comes to questions of cost and profit.training-youth. . You will take them into account for future dinners with friends. Together with your friend you decide to offer “dinner-evenings” for groups of between 6 and 12 people twice a week. with whom.net That is what this T-Kit is all about: planning evaluation. Probably other things will pop up and you will evaluate those too.

net . It is simply different. Rather. by developing the competence of youth workers in the theory and practice of Educational Evaluation. In this T-Kit. group leaders… Nevertheless this publication should be of interest to other actors involved in youth projects like organisers. Our approach does not necessarily contradict any other approaches to evaluation. This means that we look at evaluation as an inherent part of educational processes in youth work and we think about how to put evaluation at the service of educational achievements. (you can always identify them because they are sign-posted with our cooks)… Nevertheless. facilitators. In the second chapter “Pots. Foci of the T-Kit: educational activities and the scope of educational evaluation Educational evaluation is a wide topic with a lot of implications. objectives and structure of the T-Kit This T-Kit is primarily addressed to youth workers closely involved in learning processes and educational activities with young people: trainers. the focus will remain on (non-formal) education. structure and process). We will then invite you to define your own evaluation strategies. integrate and improve educational evaluation in your youth work. share and critically look at existing practices of evaluation (strategies. It is used in other parts of the T-Kit. In the first chapter. Nevertheless. methods and criteria so that you can improve your projects (in terms of achievement of its objectives. mentors. this T-Kit is not a recipe book on evaluation methods that can be automatically transferred into your activities. T-Kit on Educational Evaluation in Youth Work www. from the point of view of project management. administrators and decision makers. This T-Kit is about educational evaluation. but we want to stay focused: • The first focus is an educational one. income-outcome oriented evaluations. “The Ingredients of Educational Evaluation” (the “theoretical” part of the T-Kit) we will explore the basic ideas and concepts around evaluation that are relevant for educational activities with young people. We refer to other approaches on several occasions. we have introduced the metaphor of cooking. Concrete experiences and existing practices of evaluation as developed in practical youth work in Europe will serve us as reference material. lntroduction Target group. There are other approaches to evaluation: as part of decision-making processes. we will deal with a number of them. methods…). products.training-youth. it is an educational and methodological tool to help you to naturally develop. etc. In the foreword. pans and spices” (the “practical” part) we will map out. The ultimate aim of this T-Kit is to support and contribute to the improvement of youth projects in Europe.

As relevant. 16 T-Kit on Educational Evaluation in Youth Work www. More information at http://www. These are educational field activities involving learning processes with / involving young people. the CD-Rom on Evaluation produced by SALTO16 is complementary to this T-Kit and has been used as a source in its production. Organisational Management and European Citizenship). Among these.net/evaluationintro/ . More information at: http://www. Theory is combined with examples of proven methods from extensive experience of evaluating training activities. In the same way. publications or campaigns. we will concentrate on the specificity and practice of youth projects with a European dimension.salto-youth.net SALTO-YOUTH.net is a network of 8 Resource Centres working on European priority areas within the youth field. this focus is not intended to be exclusive. Again.10 With this particular focus.net In 2004.training-youth. SALTO produced a CD ROM on evaluating training in European Youth Work. It provides youth work and training resources and organises training and contact-making activities to support organisations and National Agencies within the frame of the European Commission’s YOUTH programme and beyond. It includes documents to download and adapt and some suggestions for going further. • The other focus refers to the scope of youth activities that we will consider in the “practical” part of the T-Kit. this T-Kit complements and in some cases completes the ideas on educational evaluation already present in the T-Kit series (for example in the T-Kits on Training Essentials. training activities. such as: youth exchanges. we have also made reference to non-field activities and other youth services such as information. volunteering.salto-youth. Project Management. seminars.

Annex I: European Youth Pact at http://ue.eu. reconciliation of working life and family life. The interim evaluation of the YOUTH programme also included a number of recommendations based on the expectations and requests for simplification expressed by stakeholders and young people. the results of national and European level mid-term evaluations established to reflect on the YOUTH programme19 2000 – 2006 and leading to an interim evaluation report and an ex-ante evaluation of the future instrument. experts in the youth-field.html 18 The European Youth Pact was adopted by the Spring European Council in March 2005 as one of the instruments contributing to the achievements of the Lisbon objectives of growth and jobs. It presents a detailed and well-argued policy for discussion and for decision. The European Youth Pact is available at PRESIDENCY CONCLUSIONS . The Pact focuses on three areas: employment.pdf 19 The Youth in Action programme is the EU’s mobility and non-formal education programme targeting young people. Both Member States and the European Commission are invited to evaluate the framework for European co-operation in the youth field in 2009. The Youth in Action programme for the period 2007 – 2013 was developed on the basis of a comprehensive evaluation process: the results of a public consultation. More information at http://eacea. The programme expects youth workers and project promoters to use evaluation techniques in order to make sure that the objectives they set are realistic. keeping in mind the community context in which the project takes place. the European Commission promotes the elaboration of evidence-based youth policies in the member states and is committed to planning its policies through a consultative process involving several evaluation methods. both institutions have put also significant emphasis on developing educational activities in which evaluation and its methodology is a primary tool for improving and refining the quality of youth work. after a one and half year long consultation process involving young people. education training and mobility. In the past decade in the youth field. • Since adoption of the White Paper: A new impetus for European Youth17.eu/youth/index_en. invites Member States to ensure effective implementation and follow-up by setting measurable objectives and developing a structured dialogue.net . national authorities and NGOs.int/ueDocs/cms_Data/docs/pressData/en/ec/84335. It is a document containing proposals for Community action in the youth field. integration and social advancement. The programme offers possibilities to young people in the form of group exchanges and individual voluntary work. which highlights youth issues in core areas of the Lisbon strategy for growth and jobs. The programme is open to youth in 31 European countries.11 The European Youth Policy Context For both the Council of Europe and the European Commission evaluation plays a very important role in verifying the effects of their policies and programmes related to young people and for planning further strategies and priorities.htm T-Kit on Educational Evaluation in Youth Work www. as well as support activities. achievable and serve the needs of the young people involved. 17 The White Paper on Youth was launched in November 2001. It is called the open method of co-ordination. In the European Commission’s YOUTH Programme reflective evaluation is considered to be an essential criteria for implementing quality youth projects. The European Youth Pact 18.eu.ec.europa. The White Paper is available at: http://europa.int/comm/youth/whitepaper/index_en.training-youth.

a tool for assessing non-formal learning progresses. the European Commission is going to establish a YOUTH-PASS.coe. A thorough and open process of evaluation is defined as one of the relevant quality criteria in order to secure. among others. the efficiency and effectiveness of activities. an expert team drafts an international evaluation report (“review”). consultants. stock-taking of the results. All activities are developed with the support of appropriate evaluation approaches.training-youth. The Training strand of the Partnership itself was as well subject to an external evaluation assessing the relevance. • Also in the Partnership Programme in the Youth field between the Council of Europe and the European Commission 22. including staff. As one of the results the three separate pillars of the Partnership that existed earlier were merged to one single partnership agreement under a joint management. evaluation plays a crucial role in all of its strands: training. With regard to quality standards in education and training activities the Directorate for Youth and Sport has elaborated reference criteria for all those involved and concerned by these kind of activities of the organisation.youth-partnership. Accordingly the Directorate of Youth and Sport is running a long-term project for the evaluation of national youth policies in a range of the Member States.1 In order to foster recognition and visibility of youth work activities. On the basis of an extensive national evaluation report and study visits in the countries concerned. In the same framework the Council of Europe promotes the recognition of non-formal education and youth work by investing in the description of what constitutes quality youth work. the evaluation of the quality of the learning process and the follow-up to be given.int/t/e/cultural_co-operation/youth/2. The compilation of youth policy indicators provides Member States with a frame of reference and guidelines for undertaking a thorough evaluation of their youth policies. The Directorate of Youth and Sport is now going to complete the long process of producing a new quality assessment tool: the European Portfolio for youth leaders and youth workers 21. research and Euro-Med. 20 21 T-Kit on Educational Evaluation in Youth Work www.asp The European Portfolio for youth leaders and youth workers can be accessed at www. co-ordination between relevant domains and practical implementation.net . which is presented to the Minister responsible for youth issues at a public hearing. particularly in the framework of the YOUTH programme.int/youthportfolio More information on the Partnership Programme: www. trainers. The criteria serve as a benchmark for the evaluation of education and training activities and are understood as minimum standards.net 22 See http://www. which after a public testing phase will be published in early 2007. in terms of consistency. • The Council of Europe and its Directorate of Youth and Sport have been pioneers in initiating the discussion on evidence-based youth policies as well as the quality of youth work in Europe. developing and sharing knowledge on the situation of young people and supporting the quality and sustainability of European youth work training and policy. fostering the recognition of youth work and non-formal education competences. participants and partner organisations.coe. the impact._Priorities/policy. The European Ministers responsible for Youth of the 46 Member States of the Council of Europe identified four key items as priorities of the Council of Europe’s youth sector for 2006-2008 20: developing and promoting standards for youth policies.

1. The ingredients of educational evaluation
1.1 What is Educational Evaluation? Definitions
Some possible definitions of educational evaluation are:

1

“Educational evaluation is the systematic investigation, observation and interpretation of information.” 23 “Educational evaluation is a method (procedure) and to prove if the expectations and aims of an educational process reflect reality (results of the process).” 24 “Educational evaluation is the process of obtaining information and using it to come to some conclusions which will be used to take decisions.”
Without wishing to present only one definition or to summarise all existing ones, we can conclude that different processes of educational evaluation have some common components. The following definition embraces them. “Educational evaluation is a systematic and ongoing process which includes: • Researching and collecting information, from different sources, about the learning process, the content, the methods, the context, the outcomes of an educational activity • The organisation and analysis of that information • The establishment of certain criteria (evaluation criteria) • The discernment and judgement of the analysed information (according to the set evaluation criteria and at the light of the educational objectives). • Drawing conclusions and recommendations which allow the re-orientation and eventual improvement of the educational activity”25.

It is necessary to distinguish educational evaluation from the process of collecting and obtaining certain kinds of information. The collection of information is something punctual, and it is done in the evaluation process at certain moments. But educational evaluation is an ongoing process. It implies judgement (good, bad, acceptable, ok, advantageous, disadvantageous, of high quality, of low quality, etc.) Educational evaluation implies measurement. But, educational evaluation goes beyond the mere measurement: it provides explanations and conclusions.

23 Tenbrink, T. & Cooper, J. M. (2003). Educator’s Guide. Page 64. 24 Nydia Elola, Lilia V. Toranzos 2000 Evaluación educativa: una aproximación conceptual. Page 2. 25 Giovanni Iafrancesco, 2001. Hacia el mejoramiento de los procesos evaluativos en relación con el aprendizaje. Page 6.

T-Kit on Educational Evaluation in Youth Work www.training-youth.net

1

Educational evaluation should not be a justification of shortcomings. As a result of the evaluation we might find out why certain shortcomings appeared, but the actors involved should not conceive and initiate the evaluation process looking for “excuses” or with a defensive attitude. Educational evaluation should not be a money making strategy. As we will see later, one operational objective of educational evaluation is to reflect whether the educational activity met the expectations of partners and funding institutions. Those institutions often use evaluation as an instrument of control and sometimes even as a criteria to decide which projects and organisations to support. This has financial implications for those concerned. But, the starting point, the nature and primary aim of educational evaluation, is not to make money. If we “financially” misuse evaluation and fall into a lack of honesty and transparency, we might in the best case gain some short-term financial benefit. Experience, however, shows us that apart from weakening the legitimacy of educational evaluation, financially motivated evaluation damages trust between funders and those they support. Educational evaluation should not become an exercise in power politics. In educational evaluation, different actors with different levels and areas of responsibility are involved. Often and naturally their conclusions differ. This, together with the fact that conclusions can to changes that can affect different actors in more or less positive ways, might imply that educational evaluation degenerates into a kind of power game, with different actors trying to assert their power over others using the evaluation as an instrument of their objectives. There is no magic rule for resisting this. But, actors who hold positions of responsibility should be those most open to criticism and generous in assuming responsibility, even if the results of the evaluation are not to their liking. These attitudes are an important contribution to making sure that the aims and nature of the evaluation are properly respected. Educational evaluation should not be a public relations exercise. Of course, the outcomes of educational evaluation and its achievements should be shared witwh concerned target groups, organisations and institutions. But, this should be an integral consequence of the process rather than an objective of the evaluation in itself. Conceiving and planning educational evaluation as a public relations exercise will damage the process and in long term becomes counter productive co-operation and partnership with others.

T-Kit on Educational Evaluation in Youth Work www.training-youth.net

1

1.2 Why evaluate? The aims and nature of educational evaluation
To learn
The first and most basic aim of educational evaluation is to learn. The aim is the learning of all actors involved: their access to additional knowledge and to a new learning opportunity. The educational purpose is the inherent characteristic of what makes educational evaluation different from other kinds of evaluation. While evaluating, the actors involved learn to understand, to give a value and to draw conclusions on their own learning experiences. Through educational evaluation we learn from experience. The changes and actions resulting from educational evaluation become critical action and reflective praxis26. All the actors involved in educational evaluation learn to express their knowledge: knowledge not of “topics” but of the relevance of their educational experience to their own lives. A certain educational activity might seem to be very good from a lot of points of view but in reality it might be disconnected with the life of participants, and vice versa. This relevance, and connection between youth work and the lives of young people, is probably the most important “knowledge” in youth work. Very often it is learned during the evaluation process. Participants also learn while sharing and confronting their judgements with those of their colleagues. During the evaluation process, different interpretations, meanings and interrelations are raised and debated. Very often actors involved in educational evaluation ask themselves: What does it mean? How should I interpret this or that result? What are the implications? The doubt implied by the diversity of answers that one might get to these questions can be considered as a matter or competence. The different actors need to have a certain “tolerance of ambiguity”, the ability to live with several different possible outcomes, not all of which fit easily together or with one’s personal, professional or organisational values, to be able to accept that the outcomes of the evaluation may not be to their liking. Where this competence is present, educational evaluation can be a motor of curiosity, a source of learning and an impulse to continue learning. When evaluation and learning take place at the same time, the actors involved create, discern, imagine, analyse, contrast, elaborate answers, formulate questions, come up with doubts, search for other sources. In other words, they truly evaluate. In our opinion, and without wishing to ignore other purposes or diminish their importance, educational evaluation should first and foremost be put at the service of learning.

26 Praxis: the translation of an idea into practice. Source Word Reference Dictionary http://www.wordreference.com.

T-Kit on Educational Evaluation in Youth Work www.training-youth.net

It would not be coherent or consequent to aim for the promotion of participation in an educational activity but to evaluate the fulfilment of that aim in a non-participatory manner. formats. This can be a result of the fact that at times in formal education. places.training-youth. It also has an educational dimension. Change. targets. This aim of promoting participation while evaluating has methodological consequences: in educational evaluation participatory methods are very important. Obvious as it may seem. That is the reason why a constructively carried out educational evaluation contributes to maintaining a challenge and to fostering motivation within a project. Resistance to evaluation is often rooted in resistance to “internal” and “external” changes that might be required of an individual or of a group as a result of the outcomes of the evaluation process. To change and improve As we have seen in its definition.net . But. evaluation (or more precisely put student assessment) is used as a mechanism to “select” or “exclude”. the atmosphere in which the evaluation takes place and on the imagination of the actors about what will happen after the evaluation results have been made public. An evaluation whose results or process de-motivates becomes limited and incomplete because it cannot maintain the participation of all actors. all the actors involved in the educational process should therefore also be involved in its evaluation.1 To motivate The evaluation process should lead to improvements and change. evolution and further development are factors of motivation for all the actors involved in the educational process. This participative dimension goes beyond the “democratic legitimacy” of changes to the educational process. However. achieving the objective of motivating while evaluating does not only depend on “recognising the achievements as well as shortcomings” of what is being evaluated. methods. It also depends a lot on the attitude adopted by those involved. of values. In educational evaluation the changes also happen at the personal level: change of attitudes. Some participants might have negative and discouraging perceptions of evaluation. This idea of change is generally assumed in an “operational” way: change of tools. improvement. T-Kit on Educational Evaluation in Youth Work www. This “personal” dimension of change is often less visible than the “operational” one. of ways of understanding. both are equally important: educational evaluation requires openness to changing our ways of doing things as well as our way of thinking. To participate Educational evaluation is an opportunity both to promote the values of participation and to practice it. Change as a consequence of the accelerated changes taking place in our societies and in the reality of young people. change and improvement are integral to the process of educational evaluation.

educational evaluation plans and criteria should not be limited to the expectations of the organisations that fund the project. Usually this is not a difficult exercise: in most cases the questions they would like to answer through evaluation would be part of our evaluation anyway. to the uses of its outcomes at different moments of the process. The following objectives are linked to the implementation and practice of educational evaluation and. To check if we met the interests of the funding institutions When funding institutions support a certain educational project.1 1. Educational evaluation has a lot of potential and uses. To consolidate results Identified results can be consolidated by making them explicit at the end of the evaluation process. To take stock of achievements It is important to recognise.3 What is evaluation for? The Operational Objectives of Educational Evaluation It is possible to identify some “operational objectives” or uses of educational evaluation. in addition to the general aims of educational evaluation.training-youth. their priorities. Even so. in order to prevent negative consequences and to compensate for possible shortcomings. the following: To plan better Educational evaluation can help to change things and to plan “different things”. T-Kit on Educational Evaluation in Youth Work www. but it can also help us to plan things better. they do it according to certain criteria: the nature of the project. But. sharing and further use of results are natural follow-up steps of educational evaluation.net . Funding institutions usually ask to receive a descriptive and evaluative report at the end of the project. it is important to consider and include them. name and give value to the achievements of the educational process so that they do not get lost or not sufficiently used. its objectives. but among others. The following list is certainly not exhaustive. The description.

net . other fields of common interest and ways of co-operating.1 To reinforce co-operation with partners If partners are involved in the educational project. T-Kit on Educational Evaluation in Youth Work www. they will be involved in its evaluation. A constructive and participative evaluation will naturally reinforce co-operation. the results of the evaluation can be of interest to them. even if your partners are not directly involved. new partners and networks with them. You might share new ideas for common projects.training-youth. But.

which propose a certain structure of “What to evaluate?”. learning achievements • Behaviour: Changes in behaviour.4 What to evaluate? Fields and “models” of evaluation a critical view and a proposal Together with the question “Why evaluate?”. transfer of competencies into concrete actions/ situations • Results: long term lasting transfer.e. we will briefly present some models. T-Kit on Educational Evaluation in Youth Work www. effect and utility of the training programme • Learning: Growth of knowledge. Each of these models focuses on different fields. also in organisational and institutional terms. Based on this exploration. See “Overall evaluation of a Pilot Long Term Training Course” (Pages 94-96) for more in-depth information. but rather reflecting on some approaches to evaluation present in all of them. not individually. “What to evaluate?” is another key to understanding the different options and approaches to educational evaluation available to us in our youth work. The Kirkpatrick model: Four fields: • Reaction: personal reflection from participants.net .1 1. To those we can add one favoured by the USA Department of Education as mentioned in the T-Kit on European Citizenship (Page 57). when it comes to “What to evaluate?” The following table describes briefly all those models and their associated fields of evaluation. Evaluation fields and “models” In the T-Kit on “Training Essentials” (Page 77) we can find the description of four different models of educational evaluation.training-youth. We will look at the models we present critically. An example of educational evaluation carried out following this model is described in the Chapter of this T-Kit entitled “Evaluation Practices”. i. we will make a proposal: “Educational evaluation as a total experience”. on satisfaction. In the following sections.

• Process – Focuses on the procedures. Council of Europe and European Commission (May 2003) Page 57..Citizenship. design. methods and their implementation.0 CIPP model: Four fields: • Context evaluation – Are the chosen goals the right ones for this activity? • Input evaluation – Is the programme well planned? Are there enough resources to implement the activity? • Process evaluation – How was the flow of the activity? What feedback was received from the participants? • Product (outcome) evaluation – Were the objectives reached? The Brinkerhoff model: Six fields: • Goal setting – What are the needs? Are these needs real? • Programme design – What is required to meet these needs? Is this design going to meet the needs? • Programme implementation – How do we evaluate the programme in practice? • Immediate outcomes – Did the participants learn? What did they learn? • Intermediate or usage outcomes – Are the participants implementing their learning? • Impacts and worth – Did it make a worthwhile difference to the participants’ organisations and their personal development? Systems approach (Bushnell): Four fields: • Input – What goes into the training effort? (Trainee qualifications. Human Resource Development.”28 27 T-Kit on Educational Evaluation in Youth Work www. 6 – Training Essentials”. Original Sources J.net 28 “T-Kit No. “T-Kit No. Training and Evaluation and R. Harris. .training-youth. Simone and D. Youth and Europe”. trainer competence.. resources. • Impact – Long-term outcomes of a programme as well as unanticipated effects. 7 .M. did they reflect on their behaviour? Did their attitudes change? • Outcomes – What are the effects on the participants’ organisations?” 27 United States Department of Education: Three fields: • Outcome – The immediate.L. • Process – How adequate are the planning. etc).J Jackson.Under construction. Council of Europe and European Commission (October 2002) Page 77. development and implementation of the activity? • Output – What are the participants’ reactions? Have they gained knowledge or skills?. direct consequences of an educational activity on its participants.

SMART objectives are those that are Specific. each of those models put the emphasis on different evaluation fields. Achievable. one by one. This horizontal perspective will provide us with elements to critically understand the previously described models or others and (what is more important) will inspire our own approach to educational evaluation. Youth and Europe”. and the long-term impact of educational processes. after all?” 29 29 “T-Kit No. The advantages and disadvantages of this kind of evaluation are: Advantages • There is a clear direction in the evaluation process: looking at the fulfilment of the objectives • The objectives of an educational activity are the common reference for the different elements and actors. and indeed evaluation by SMART objectives does have its place and value. 1 Evaluation by objectives This is the first and the most natural orientation of an educational evaluation: to evaluate a project according to its objectives. Measurable. Evaluation by objectives is associated with “keeping track of those objectives” and re-orienting the evaluated educational process if it is not going in the right direction. The purpose of the objectives in a project is to guide the educational process and to give an orientation to its evaluation. we feel that an evaluation process defined purely around measurable objectives cannot always do justice to the richness of out-comes. the potentialities and limitations of each model. evaluating by objectives facilitates the interaction between them Disadvantages • An evaluation that focuses exclusively on the set objectives cannot alone guarantee the fulfilment of the desired learning objectives • There is the risk of paying little attention to the process and other aspects of the activity • Sticking only to an evaluation of the objectives limits the creativity and originality of the educational process and does not reflect the diversity or multidimensionality of relevant information that can come out of an educational activity are set for a project. relevant. Realistic. The objectives should be clear. Council of Europe and European Commission (May 2003) Page 57.training-youth.Citizenship. 7 – Under construction.. However. and achievable within a Timescale. we would like to critically reflect on the different main focuses that an evaluation can take horizontally: by objectives. be adapted to the needs and profile of participants. “Woolf (1999) states that evaluation can only be effective if SMART objectives T-Kit on Educational Evaluation in Youth Work www. They should also be organised in a time frame for their achievement. by competencies.A critical view on evaluation fields and “models” As we have seen. For this reason. This approach can be very appealing for its neatness and simplicity. be organised in a hierarchy of priorities. Instead of analysing in detail.. Is SMART so smart. by achievements.net .

developed or achieved during the educational activity. attitudes. arguments and propositions explored in this kind of evaluation can become a catalyst for the further development of participants’ competencies Disadvantages • It is not always straight-forward to evaluate competencies in non-formal education and to be able to relate their value to a wider context For example: a certain participant develops in an educational activity the competence of “team work”. T-Kit on Educational Evaluation in Youth Work www. planning. For example. negotiation. mediation among others. In recent years. Although “team work” is a very important competence in a wider social context. For example. an important focus of evaluation in the field of non-formal education has been on the ways in which the “employability” 15 of participants can be improved through non-formal education. skills. abilities and values) gained. competence is understood as “knowing how to do something in a certain context”. Evaluation by competencies This approach evaluates the competencies (knowledge.net 15 Employability can be defined as “the capability to move self-sufficiently within the labour market to realise potential through sustainable employment” Source: The Wiki Encyclopaedia. And this relation / interaction can be a strong source of learning for participants • The combination of individualised qualitative and quantitative information that can be gathered in this kind of evaluation can be very useful for supporting the personal development of participants. evaluation by competencies analyses the pertinence.training-youth. Consequently. In non-formal education. the extent to which (s)he be able to put it into practice outside of the educational activity is difficult to asses • The concept of competence is associated with a large number of categories and indicators. http://en. which can complicate the organisation and implementation of the evaluation. This kind of evaluation tries to answer the question: In the end.org/wiki/Employability . relevance and value of the educational process within its social context. what is the value of this educational activity in our social context? For example: evaluation by competencies could look at the value of an activity in which participants develop their intercultural competencies in the context of our multicultural societies. the team work competence is associated with: communication.wikipedia. there is a risk that non-formal education will become subordinated to the needs of the market. The advantages and disadvantages of this kind of evaluation are: Advantages • Evaluation by competencies makes more explicit the link between nonformal education and its social context. individual information can be the basis for designing personal development plans and tools for selfevaluation / assessment • The interpretations. management. • If the context (particularly the financial context) has too much influence in the identification and definition of the competencies to be developed in educational activities.

taking the floor in plenary is an indicator of “participation”. an indicator for the achievement of this objective could be the number and composition of teams voluntarily set up by participants. G. Cerda. an indicator of the achievement of that objective could be the number of people taking the floor in plenary to express their opinion. But. so far just one quarter of the participants have so far taken the floor to speak in plenary”. then “talking in plenary” (and not “participating”) might become an instrumental and misleading objective of the activity.net 30 La Evaluación como Experiencia. can provide us with evidence of the level or evolution of a certain aspect of the educational process. participants might start to reflect why.  “An indicator is a signal. If another objective is to promote co-operation among participants. an indicator could be the number and characteristics of common initiatives and projects. as such. If in the middle of an activity the trainer says “Come on. Disadvantages • There is little conceptual clarity around the notion of achievement. or even to discuss if. data of perceivable information that after being confronted and interpreted according to what was expected. they can become instrumental objectives of the activity. is an incomplete concept: Achievement of what? In educational evaluation the notion of achievement is related to the “achievement of objectives” or the “achievement of a certain level of competence”. if one of the objectives of an activity is to promote participation. The advantages and disadvantages of this kind of evaluation are: Advantages • Achievements and indicators of achievement are more concrete compared with the ambiguity of evaluating the fulfilment of the objectives and the development of competence in a more general way • Evaluation by achievements and using indicators can generate further questions and debates. Hugo. T-Kit on Educational Evaluation in Youth Work www. • Some indicators could be common to different educational activities creating some comparability between evaluation results.”30 For example. Coop. participation=talking in plenary). If the only indicator to evaluate the achievement of participation is talking in plenary then the notion of participation might become exclusively equated with speaking in plenary (e. Bogotá. Achievement of what? For what? Under which circumstances? Achievements are always related to other aspects of evaluation • A narrow and limited use of indicators can be counter-productive and confusing. • If the expected level of achievement and/or the indicators are set by an external person or institution. Associated with the notion of achievement is the concept of an “indicator”.training-youth. the organiser or the external evaluator will “take note of the number of people who talk in plenary”. 2000 . characteristic. If trainers know that the funder. evaluating by achievements does provide some new perspectives to reflect on. one would be forgiven for thinking that evaluation by achievements is just another way of evaluating the fulfilment of the objectives or evaluating the competence developed through any given activity. If another objective is to encourage team-work. Editorial Magisterio.Evaluation by achievements Achievement. indication.g. As such. then.

After it the participant-trainers and the trainers’ team evaluate the whole process. potentialities and limitations is one kind of evaluation. This personalised evaluation requires time. Disadvantages • This kind of evaluation is only relevant for a limited group of actors in non-formal education. . This evaluation by performance leads to a reflection on the competencies required for being a trainer and how to improve them. • In this kind of evaluation personal styles and preferences can play an important role. at the same time. • If the evaluation environment is not safe or professional enough it can imply lack of transparency and provoke mistrust between those engaged in the evaluation and those who performance is being evaluated. • This kind of evaluation can be very helpful to set up strategies for update and further training. specialised staff and a lot of resources in order to be consistent. and is often it related to the assessment of human resources and is often used as a criterion in staff selection. A good illustration of these advantages. The advantages and disadvantages of this kind of evaluation are: Advantages • This kind of evaluation is good for assessing human potential and putting it at the service of a wider group (other participants. in other contexts. youth leaders and youth workers). Evaluation by performance focuses on the appreciation of how individuals behave “professionally”. For example. facilitates a group discussion or develops an exercise. taking into account their potential for further development. those who already have a certain level of experience in non-formal education. • It can be used to promote the recognition of individual professionalism and of non-formal education. If it is not carried out with the necessary distance from social roles. participants. interacts with participants. in different contexts. often used in training for trainers in non-formal education and youth work. doing different things. T-Kit on Educational Evaluation in Youth Work www. Evaluation by performance is strongly linked to the further training and lifelong learning needs of experienced actors in non-formal education. evaluation by performance of a youth leader would analyse how (s)he organises an activity. The evaluation includes a critical analysis of their performance as trainers in the preparation and running of their workshops. disadvantages. Evaluation by performance Performance is a concept that is commonly found in the private sector.trainers. This usually takes the form of participant-trainers31 having to plan and implement a workshop in teams. They are trainers and in the frame of the course. there is the risk of the evaluation suffering from subjectivity. evaluation by performance is commonly found in the context of “training for trainers” or in long courses in which participants are expected to implement workshops or projects autonomously as part of the learning experience. In non-formal education. in general.net 31 Participant-trainer refers to the double role of participants in training for trainers.training-youth. colleagues . • Evaluation by performance implies the management of a large number of variables: individuals performing in different circumstances.

This symbolises the homogenisation of individuals. This perspective might be a valid one when it comes to the management of non-formal educational processes or to ensuring transparency in the use of public resources.” 34 32 33 34 Pink Floyd The Wall (1982) Directed by Alan Parker. and certainly unpredicted processes. anonymous pupils enter a factory. organisational implications and the impact of the activity for a wider social context. of course. In terms of educational evaluation.formal education. “outcomes” are mostly associated with fulfilling the objectives. We have already described the advantages and disadvantages of focusing the evaluation on each of them and we shall not repeat those arguments now. They fall into a grinder whose product is mincedmeat. learning achievements. considering all of them under the umbrella notion of “outcome” allows for a greater richness of evaluation results and for a wider perspective on the educational process evaluated... Evaluation by outcomes gathers all these aspects. “T-Kit No. Using a simplistic income-outcome evaluation. Incomes  Outcomes Non-formal education!? One of the clearest visualisations of this danger was shown in the video “The Wall”32. Council of Europe and European Commission (May 2003) Pages 57-58 T-Kit on Educational Evaluation in Youth Work www. a caricature of formal education. Due to the influence of the commercial sector.Evaluation by outcomes The concept of outcome refers to different “consequences” of the educational process. Youth and Europe”. should not diminish their importance. it is likely that unpredicted outcomes. its application without mediation to the educational processes that constitute non-formal education affects its most basic character. “Perhaps due to the pressures upon project organisers to prove the value of their work to external bodies (particularly those from which they seek funding) there is a tendency for disproportionate attention to be afforded to outcome evaluation… …Also. We don’t need no thought control…” This is. for the sake of being able to prove what has been achieved. The fact that some outcomes cannot easily be evaluated. Rock opera/concept album “The Wall”. The reductive misuse of “evaluation by outcomes” can contribute to the reinforcement of this logic. 7 – Under construction. integrative perspective. forming an enormous row on a conveyor belt. To the music of “Another brick in the wall”33. The lyrics of the song are “We don’t need no education. An evaluation based only on preconceived notions of outcomes.training-youth.Citizenship. in principle. is unlikely to do full justice to any project. We consider it imperative that educational aims are not reduced to those that can be measured. thus.net . But there are risks associated with educational evaluation that uses this. there is a growing tendency to evaluate educational programmes (and not only their management) by comparing incomes and outcomes using the purist logic of production. “Another brick in the wall”. But. Pink Floyd 1979. In principle. and certainly cannot be quantified. will arise over the course of a project. non-formal education can be reduced to a “social technology” or an instrument for “social intervention” functioning like a machine: you put something in and you automatically get something predetermined out. but the “machine logic” can also be found in non.

Evaluation by process is not just a matter of checking “how things are going”.. More complete definitions of qualitative and quantitative evaluation can be found in pages 33-34 in the section called “Typologies of evaluation”. the expected outcomes. in evaluation by process. connection. • It is difficult to “capture” the complexity and dynamism of the educational process. • It can help participants. trainers and organisers to clarify their own challenges and to get to know the most important obstacles in the educational process. it helps to develop a more solid and consistent educational process. Evaluation by process analyses the relationship between the needs of participants. This is very common and at times intentioned simplification. It also looks at their meaning for different actors. T-Kit on Educational Evaluation in Youth Work www. the starting point is the needs of the different actors. It focuses on the “quantity” of the educational experience raising the questions like How many? How much? How often?.net . rather than how many or how much? Quantitative evaluation aims to measure and count. it covers particularly well the functions of diagnosis.training-youth. Disadvantages • There is little agreement and conceptual clarity around the notion of “process”. the objectives and the results of a youth project. In this case. • Since this kind of evaluation takes has its starting point the needs of the different actors. It raises questions like How? Why?. the “results” would be the sum of most of the elements of the educational process. orientation and motivation. Evaluation by process tries to overcome their limitations and it is based on qualitative evaluation35. In general terms. coherence and correspondence between: • the need of having leisure activities • the expected outcome of creating a cultural association • the objective of promoting co-operation and participation during leisure time • the result of having sport competitions and a small cyber centre The advantages and disadvantages of this kind of evaluation are: Advantages • Evaluation by process has a particular educational value because it does not just look at the different stages of an educational activity: it goes through them. The objectives are fixed to “conduct” the educational process starting from the needs and moving towards the expected achievements. there is a bigger risk of the evaluation becoming too partial 35 Qualitative evaluation looks at “qualities”: characteristics of the educational experience. • In reality it is difficult to make compatible the “theory” and the “praxis” of this kind of evaluation due to its complexity • Being basically qualitative. Consequently. For example. From those needs expected achievements are formulated. Evaluation by process Historically. evaluation by process would mean to analyse the relation. this approach emerged a critical alternative to evaluation by objectives and by achievements..

This is our proposal and our invitation. contexts. times. actors. experiences.org T-Kit on Educational Evaluation in Youth Work www. conclusions. evaluation often puts the emphasis on separating and breaking down backgrounds. Source: Wikipedia 2006. Very often it is one of the most important sources of learning for the group. For example. http://en. Educational evaluation as a total experience would not pretend to discern who is right or wrong about that. For the purpose of analysing them. theory and praxis • combines quantitative and qualitative approaches to evaluation • looks at the whole educational process (integrally and from different points of view) • prioritises the most valuable information. its integrity and by the articulation of different approaches. for some participants a session was too long and for others too short. the “conceptual” (learning. an explanation or a thesis and its opposite are not exclusive: both are part of a whole. This happens very often. In this line. opinions. contents. This does not contradict the idea of considering all of them part of a whole or of a unity. sources and techniques of evaluation.  Our proposal: Educational Evaluation as a Total Experience Given the complexity of the educational process the first and most spontaneous answer to the question “What to evaluate?” would be EVERYTHING.training-youth. and necessarily. • Unity in diversity The presence of diversity in educational activities is a fact. without falling into “doing everything” The methodological proposal of this approach to educational evaluation is the combination of diverse spaces. motivating…) and the “instrumental” (judging. organisational frameworks and methods. 36 Broadly defined. value.The difficulty of evaluation by process is very obvious in training courses. but to understand. methods. Educational evaluation as a total experience: • is characterised by its global vision. or at least a qualitative transformation in the direction of the dialogue. By doing so.wikipedia. deciding…) visions of evaluation become complementary rather than contradictory. The following questions commonly arise: • How can the diverse needs of participants be related meaningfully with the objectives of the course and with the programme at the beginning of the course? • How can the needs be checked in the middle of the course in light of the objectives and considering the development of the group? • How can the needs be referred back to at the end of the course to see the level of their achievement? On many occasions evaluation by process is unfortunately reduced to picking out some elements of the above here and there without embracing the full complexity of the different processes that participants experience during a course. The ethical and educational foundations of this kind of evaluation are: • Dialectic36 unity Dialectic unity describes when an idea. a dialectic is an exchange of propositions (theses) and counter-propositions (antitheses) resulting in a synthesis of the opposing assertions. the editorial team of this T-Kit understands educational evaluation as a total experience. However. this diversity should be respected and all of the aspects of the educational process have to be considered. the educational process itself. methods. changing.net . put in a certain perspective an eventually challenge both conclusions. a conclusion.

coherence and correspondence between them. mutual influences. In non-formal education. T-Kit on Educational Evaluation in Youth Work www. it is necessary to prioritise among all the possible evaluation fields.net 37 “T-Kit No. Educational evaluation as a total experience aims not just to look at diversity and to cover the different dimensions of the educational process. if you like. and to create a basis for the development of our own evaluation approaches and strategies. is the need for ongoing evaluation of processes and outcomes to inform responsive educational practice. Equally. just the basis for elaborating. Council of Europe and European Commission (May 2003) Page 58 . time and again. a new and suitable menu.training-youth.Under construction.” 37 Ingredients. It is difficult to translate this approach into practice because it requires effort to deal with complexity. In particular it will be important to work out your values and ideas – the ethos or philosophy of your approach.. implemented in a manner suited to the particular project. Youth and Europe”. tips… are. should be a responsive process.Citizenship. as an element of practice. Perhaps even more important than the need for evaluation to address immediate and long-term outcomes. the approach is clearly still under development. Our invitation to you is to discover and develop educational evaluation as a total experience. recipes. It raises questions like: How does the organisational framework influence the contents? Does the profile of participants correspond to the activity format? Can any trends be observed in the different experiences participants went through? This approach constitutes a significant challenge. “We propose that evaluation.. the fact that step by step processes cannot guarantee success does not remove the need to consider what approach you might adopt. given limited time and resources. previous experiences. it tries to analyse the connections. We believe the same can be said for effective educational evaluation. among different ways of collecting and processing information. All the previous “models” of evaluation and any other you may use or know of evaluating have their place and their role. The fact that unanticipated results might be significant and that outcomes are complex and not always predictable or measurable. does not save us from the need to set goals and think about what we want to achieve from the earliest stages of planning. And in practice. for a chef. taste and understanding of the actors involved. and according to the particular skill. We believe it is good to know them. which the evaluation should serve. to look at them critically. A total experience as it is integral to the non-formal educational process. Evaluation must also be founded in ethical practice and should reflect the aims being sought in its implementation. 7 .



1.5 Who and for whom? The actors of educational evaluation
Having reviewed the possible definitions, aims, operational objectives and fields, it is not difficult to identify the actors of educational evaluation. The questions “who?” and “for whom?” are deliberately formulated together because everybody involved in the educational process should participate in its evaluation and should be informed about its results, as follows: • The participants: as learners and the target group of the activity • The facilitators, leaders or team members: as responsible persons for animating the activity • The organisers and partners: as promoters of the activity • The funders: as supporters of the activity • The decision-makers: as those “responsible” for considering the results of the evaluation in further decision making processes. These different actors have different responsibilities and tasks in the evaluation. They should participate at different levels and in different phases of the evaluation. Nevertheless, it is important to involve everybody. The evaluation should be democratic, transparent and simultaneously accessible for all actors concerned. Sometimes evaluation is conducted by an “external evaluator”. But, this does not play against the idea of involving all the actors. Moreover, one of the tasks of the external evaluator is to involve the different actors and facilitate their interaction during the evaluation process. A common assumption is that the external evaluator will be in a position to do that with more easer and with fewer compromises than so-called “insiders”.

T-Kit on Educational Evaluation in Youth Work www.training-youth.net

0

1.6 When to evaluate? The timing of educational evaluation
Evaluation is an ongoing and continuous process! However, “when” we evaluate is also an important decision to make in the planning of the evaluation process. It is important because it will give us an idea about how things are going at different and distinct points and will allow us to use the information collected to make necessary changes to improve during the educational process. Three main types of evaluation can be identified according to when they take place. These are initial, mid-term and final evaluation. Imagine you are cooking! Before you start cooking, you place all the different things you need for cooking on the work surface and you check if you have everything you need, and more importantly, if you have all the right ingredients. It may not be a good idea to notice half way through cooking that you do not have the corn, or even worse, that you bought fish instead of chicken!!! After you start cooking, you occasionally open the lid on the pot to check the colour and the aroma, to make sure things are going according to plan. Of course, you also occasionally taste the dish you are cooking to check if you have added the right amount of salt or if the pasta is cooked to perfection. And finally, when you have proudly served the meal you prepared to your friends or family, you naturally ask them if they like it. You have probably got the idea already: The above is simply a sequence of initial, midterm and final evaluations. Evaluating the process and outcomes at different stages gives you more control over the big picture, and ensures that you do not end up with under salted and under cooked chicken. Without even noticing it, you have used initial, midterm and final evaluations all the way through. Initial evaluation is the evaluation done at the very beginning. Before we start our project, whether it is a youth exchange or a training course, it is a good idea to take some time out to check our aims, objectives and methodologies. Some questions you might want to ask yourself at this stage could be: • Are the objectives of our project in line with our overall aim? • Do the methods chosen serve the fulfilment of this aim? • Does the programme cover everything we want to address? Is it realistic? • Does our team have the necessary expertise and capacity to run this programme or do we need to call in some support?
T-Kit on Educational Evaluation in Youth Work www.training-youth.net

Your turn! Take five minutes to write down a few questions you may want to ask yourself during your initial evaluation and before you start the project.

Information about the participants is another important part of initial evaluation, and is commonly called “profile information”. It can be understood as “an initial snapshot of participants”. This information is vital for checking how much effect your programme will have on the lives of young people. In this information there are data, which do not change, for example the age of the participants. This type of information is useful for checking, who on our programme is most likely to benefit. Profile information is crucial for designing your programme in a way that fits the needs of participants. The mid-term evaluation is our chance to check how things are going. As the name implies, it takes place during your project. It may be an “ongoing evaluation”, such as daily reviews at the end of each day, or a whole set of methods that take place at the mid-point of the programme. Mid-term evaluation may prove to be extremely useful for project teams, as it gives them the possibility of identifying potential problems or shortcomings as the programme progresses. Team members may decide to make necessary changes in the programme, to tackle these challenges, or meet additional needs that the participants may raise. Once we complete our programme, it is time for the final evaluation. Using a variety of methods, the participants and the team evaluate the whole project in light of the outcomes (i.e. fulfilment of objectives, learning achievements, organisational implications and impact in a wider social context) with a certain distance and perspective. 

1

T-Kit on Educational Evaluation in Youth Work www.training-youth.net

we will introduce other types of evaluation.” 38 Formative evaluation accompanies the learning process and can contribute to it. ongoing analysis and drawing conclusions. T-Kit on Educational Evaluation in Youth Work www. Formative/summative “When the cook tastes the soup while cooking. this time categorised according to their actors (personal. In this section. 1991: p. Summative evaluation looks at to the overall and final outcomes (e. arriving to a consensus is not the aim. Group evaluation has an added dimension. to their functionality (formative. when the guests taste the already finished soup. it consists of the verification of the expected results and drawing conclusions at the end of the process. learning achievements. Personal/interpersonal/group Personal evaluation is the kind of evaluation in which each individual or actor involved in the educational process makes their own judgements and draws their own conclusions about the experience they have had. In other words. group). It is not simply an interpersonal evaluation in which the number of actors involved is bigger. Robert Stakes is professor of education and director of CIRCE at the University of Illinois. It consists of continuous appreciation.training-youth. Since the group as such is part of the context and very often an important source of learning in non-formal education. summative) and to their nature (quantitative.7 Typologies of Evaluation We have seen in the previous section different types of evaluation according to their timing (initial. mid-term. the contribution of the group to the learning and the group process. It should be noted however that this is not its exclusive function. interpersonal. 1. organisational implications and impact in a wider social context). . group evaluation specifically at aspects and dimensions of the learning process that can be observed and judged from a group point of view. While individual judgments may change as a result interpersonal evaluation. that’s summative. and final).net 38 Robert Stakes quoted in “Evaluation and Education: A Quarter Century”. 169. the fulfilment of the objectives. the co-operation among participants. Interpersonal evaluation happens when more than one individual actor involved in the educational process shares and discusses their judgements and conclusions. that’s formative. Often this kind of evaluation takes place in a small group setting.g. including for example the atmosphere. Chicago: University of Chicago Press. qualitative). The purpose is simply to share and discuss those individual evaluations.

“How much?” and “How often?”. For example. on the other hand. 4. how many volunteers participated in voluntary work camps organised by your organisation? We had 200 volunteers from Istanbul in our project! Why do we have much fewer volunteers from the Eastern cities in Turkey? We have had 200 volunteers in our project! How many of them went to Eastern European countries? Why are students and especially university students more likely to apply to become volunteers? How did our campaign on volunteering help to raise awareness? Where is the best place to recruit volunteers? How often do volunteers participate in voluntary work camps after their first experience? Why do we work with volunteers exclusively? Why do we have more female volunteers than male ones? Answers: 1: Quantitative – 2: Quantitative – 3: Qualitative – 4: Quantitative – 5: Qualitative – 6: Both – 7: Both – 8: Quantitative – 9: Qualitative – 10: Qualitative. 3. 5.  Exercise: Quantitative or Qualitative? Read the following questions and try to find out if they are quantitative or qualitative.training-youth. qualitative evaluation looks at the meaning of the experience for different actors. The major questions that quantitative evaluation raises include “How many?”. These provide different types of information that enables you to have a larger and sharper image of the performance of your project. So. it is important to note that solid and professional educational evaluation usually involves a combination of both quantitative and qualitative evaluation methods. 10. 1. 9. 2. relates to the quality of the programme and of the experience. 6. This can be on an individual level or on a group level. T-Kit on Educational Evaluation in Youth Work www.net . the experiences of a given youth exchange for Arda from Turkey is considered along side the experience of a group of youth workers involved in a complex training in the city where Arda lives. 8. For instance: • Why did the participants of the contact-making seminar not create any follow up projects? • How were the methods of work presented in the training course used by the participants once they went home? Before we get into further detail of quantitative and qualitative evaluation. Afterwards. It aims to count or measure different phenomena (literally). 7. discuss why. The questions that qualitative evaluation usually raises are “How” and “Why”.Quantitative/qualitative Quantitative evaluation focuses on the “quantity” of the experience. For instance: • How many young people participated in the youth exchange? • How many countries were represented? • How often did they stay in touch with each other after the exchange? Qualitative evaluation. How many young people have participated in the activity? In 2005.

which is considered more subjective than objective. Qualitative evaluation. The following table shows different methods of qualitative and quantitative evaluations: QUALITATIVE QUANTITATIVE UNSTRUCTURED Observation of participants Letter to oneself Focus groups Diaries Film/Video In-depth interview SEMI STRUCTURED Structured interviews Observation with guidelines Spot checks STRUCTURED Surveys Questionnaires Checklist observations T-Kit on Educational Evaluation in Youth Work www. seeks to explain why the majority of the participants were from Western European countries. There are also other major differences: Quantitative and Qualitative Evaluation: Basic differences:39 QUANTITATIVE 1. The third distinction relates to replication. Measurement Conclusions are based on the analysis of data Replication of the process is possible Structured 1. the line between the two can be blurred. Another level of difference between quantitative and qualitative evaluation is therefore the methods we use. with set information sought from each respondent having been decided in advance by the evaluators. 3. The second difference arises from issues of transparency. we move from observations to conclusions using interpretation something. how we reached the result on the number of participants that took part is quite clear – we counted the number of participants. Dylan (2004). The focus of the evaluation is on the importance of the different actors in the process having the opportunity to express their opinions on the activity. In quantitative evaluation. 4. However. One explanation might be that that participants from outside the European Union did not have enough time to take apply for and receive their visa. Avanos: Turkey. as with the exercise we did identifying qualitative and quantitative evaluation questions. Finally.net 39 Kneale. the fourth distinction: Qualitative evaluation is said to be unstructured. 2. 2. so that each actor will have the same evaluation experience. qualitative data depends much more on the context.training-youth. when using qualitative methods. QUALITATIVE Explanation Conclusions from observations are dependent on interpretation The process is difficult to replicate Unstructured The first difference is that quantitative evaluation measures phenomena that appear in our programme. quantitative evaluation is far more dependent on a set structure. 3. . 4. Even something as simple as the time of day the evaluation takes place can have an impact on the results. on the other hand. However. Quantitative statements contain measurements such as “55% of participants were from Western European countries”. On the other hand. Un-published presentation on Evaluation. Quantitative methods can be replicated exactly. although. The differences between quantitative and qualitative evaluation are not limited to the type of questions. Youth Express Network seminar on Evaluation.

That is because in Italy. In non-formal education we do not organise exams and we do not give marks or grades. We rather ask participants what they think they have learned.net . allowing it to give her the paper that describes her competencies and ensuring that the paper is recognised as having a certain value. As soon as you arrive in the “supermercato” to buy yeast to prepare the dough you could easily take the wrong one and end up with a pizza with a sweet cake-like base. This would also mean that her youth association is accredited as an organisation that meets certain standards. That’s because the question of pizza is not only about understanding what “yeast” is. “validation”. about regional differences. as a foreigner you probably would not know that. it’s about what should be on it. But. so that when she has an interview for a job she could show her competencies? Of course. the outcomes. “accreditation”. But. At the time of writing. baking a pizza could bring you into deep discussions about different perceptions of pizza with your Italian friends. the term “evaluation” is frequently raised in discussions of nonformal education. There seems to be an agreement that people learn a lot in nonformal education.8 Educational evaluation in a broader framework Yeast and a pizza in Italy! If as a foreigner in Italy you would like to prepare a pizza. but we do not assess the individual level of participants. that would be great. and “qualification”. T-Kit on Educational Evaluation in Youth Work www. what does it prove? Did she really learn from those experiences and how good is she at the things she claims she is able to do? How can non-formal education be more precise in describing what people learn and the extent to which they learn it? One of the things that makes non-formal education different from formal education is that in non-formal education assessment of participants does not take place. Still. 1. you might run into a few problems. We evaluate the programme. the process. together with terms such as “assessment”. All this is part of the ongoing discussion about the recognition of non-formal education.training-youth. what is the right cheese. which herbs to use… it is about the whole framework. it could help the 17 year-old girl to gain recognition as a competent person for that job if she had a precise description of what she learned and on what level. But what does learn a lot actually mean? What exactly do people learn? How can we measure that? Does it have any value on the labour market? What can the average 17-year old girl do with her three years of experience of working as a volunteer in the local youth association? Would it not be useful if her experiences were noted on paper. there are several kinds of yeast. Anyway. about what is good tomato-sauce. about when to eat it. about the history of the pizza.

For example the portfolio: a kind of folder bringing together all materials that prove a person’s efforts in non-formal education and their learning achievements. If we understand non-formal education as a democratic process of access to processes of self. It is a controversial issue. It can be accessed at http://www. The inherent role. One significant emerging “outcome” of those debates and an alternative to the so far “black and white” nature of the debate is the concept of self-assessment.training-youth.net 40 This self-evaluation matrix is part of the “European Framework of Reference for Languages: Learning. Teaching. on the other hand say th at non-formal education would lose its specific nature and character if it would provide certification. This would broaden their options in their transitions to the labour market and to an autonomous life. as actors concerned disagree about whether this is the way non-formal education should evolve.coe. certification and accreditation of non-formal education are provoking strong educational and ethical debates at very different levels. and Assessment”.int/T/E/Cultural_ Cooperation/education/Languages/Language_Policy/Common_Framework_of_Reference 41 The European Portfolio for youth leaders and youth workers is a tool which has been designed to enable youth leaders and youth workers in Europe to assess and describe their competences. The idea behind this form of assessment is to provide participants with adequate tools and support for assessing their own learning and for documenting its results. on the basis of a core set of European quality standards. would raise ethical issues of the most fundamental nature and would put into question the legitimacy of non-formal education and its actors. curriculum and practice of non-formal education. Discussions around assessment. any selective misuse of certification would be more than questionable. Supporters of “recognition” say that lots of young people would benefit from the certification of non-formal education. This framework is a tool for setting standards in an internationally comparable manner. One of the key criticisms of the movement towards certification is that such approaches can be used as tools for selection and exclusion. Their role as “facilitators” would be conditioned by a power logic. a group of experts has developed the European Portfolio for Youth Leaders and Youth Workers41. youth workers and youth trainers and to their target groups. social importance and civic mission of non-formal education could be threatened by the movement towards certification. In recent years some self-assessment tools for non-formal education have been developed on the European level. It facilitates a clear definition of teaching and learning objectives and methods and provides the necessary tools for assessment of proficiency. The Council of Europe has developed a self-evaluation matrix.coe. risking the restriction of equal access to knowledge and having negative consequences for the nature. A large debate exists as to whether the role of non-formal education should also be to provide certification and diplomas for the learning that takes place during its educational processes. Opponents. T-Kit on Educational Evaluation in Youth Work www.development. It can be accessed at http://www. It would completely change the relationships between youth leaders. as they would have to assess participants in their programmes.int/youthportfolio . In the youth sector of the Council of Europe. which helps you to assess your language competencies40.

activities and their outcomes.pisa.org 44 Taken from the “Terminology Cheat Sheet” (by Lynne Chisholm Pages 44-46).salto-youth. that promotes the mobility and voluntary involvement of young people.europa. The ranking may be set in relation to criteria that are specific to the context. process or outcomes that are being assessed (such as: who swam the river fastest. department of education…) accredit courses. Certificates or diplomas are the “pieces of paper” which record the outcome of the certification process. Certification refers to a standardised process of formally validating knowledge. Accreditation: formally or socially recognised authorities or instances (university. More information at http://eacea. evaluation and assessment of the individuals and the organisations whose judgements are given the stamp of approval.eu/youth/programme/index_en. Feedback is supposed to be helpful to the person receiving it. 2004. part of the report of ‘Bridges for Recognition 2005’. and nor does its outcomes automatically suggest that something is of greater value or importance than something else (such as Council of Europe activities in comparison with SALTO activities).”45 42 The European Voluntary Service (EVS) is a part of the Youth in Action programme of the European Union. in 41 countries in the second assessment in 2003 and at least 58 countries will participate in the third assessment in 2006. know-how. The words explained “Evaluation: In English. The survey was implemented in 43 countries in the first assessment in 2000. nor does it imply any particular method of evaluation (such as a written test). Alternatively.htm#structure 43 The Programme for International Student Assessment (PISA) is an internationally standardised assessment that was jointly developed by participating countries and administered to15-year-olds in schools.net/bridgesReport/ 45 Definition adapted from the report of the Training Module II: Responding Effectively to Cultural Insensitivity of the Multicultural Leadership Institute. Assessment takes place when evaluation has a comparative dimension that involves setting individuals. skills and/or competencies acquired by an individual or represented through a learning/service provider. They vouch for the credibility of the certificates and diplomas that are issued.” 44 “Feedback: giving feedback is to pass on to a person the effect of his/her behaviour for their use and learning. This report is accessible at http://www. It does not imply any specific purpose (such as grading individual performance). It most frequently has the status of an official document. activities or institutions into a ranking order of performance or achievement. but this is not an absolute prerequisite.oecd. This means they testify that organisations and individuals meet standards to which all have agreed to conform. evaluation only means to make a reasoned judgement about or to give a plausible account of something. T-Kit on Educational Evaluation in Youth Work www. relative performance may be assesses against an external standard (such as in the case of the PISA43 attainment tests for 15-year-olds in different countries).training-youth.ec. More information at http://www.net . or which EVS42 agency has the highest success rate in attracting socially disadvantaged young people into the programme). and hence for the reliability and validity of the monitoring.

to reduce educational evaluation to a mere tool at the service of achieving quality would. And. This “giving a value” would be equivalent to “attaching a certain quality” to what is evaluated. aims and potentials of evaluation. In the debate on Educational Evaluation and Quality there are two poles: • On the one hand. etc) including non-formal education. T-Kit on Educational Evaluation in Youth Work www. • On the other hand. we believe.training-youth. social services. vice versa: quality can be part of educational evaluation. imply not to respect the educational nature. partners. trainers and participants work for and expect a growing level of quality in educational activities. Educational Evaluation Quality As we can already see. to consider “quality” as just another “element” of the educational evaluation process would. organisers.9 Educational evaluation and quality in youth work Why talk about quality? Educational evaluation and quality are very closely related. Quality Educational Evaluation Quality as part of educational evaluation. imply not facing the quality challenge. we believe. the intimate relationship between educational evaluation and quality is dialectical and uneasy. Indeed.net . What does this mean? Educational evaluation as part of quality: Educational evaluation can be considered as a tool or mechanism for achieving quality. the social demand for quality is growing in a lot of fields (industrial products. public services. 1. Educational evaluation involves making a value judgement on what is evaluated. In this case quality would be a layer. In this case quality would be the aim and evaluation the instrument. nowadays. Evaluation can be considered part of quality. Even if “improving” is one of its objectives. a reference at the service of the evaluation process. Funding bodies. if one tries not to conceive one of them as a mere instrument of the other. educational evaluation is much more than a tool for improvement.

trainers.e.  Educational Evaluation Quality “Understandings” of quality What is quality? There are many different conceptions of quality: • One coming from the private sector would be: “Quality is: if the client comes back to you”. in our field. reducing the educational fact. for an exchange or for a training activity or for a workshop… 46 CEN. according to those and other “factors” (i.org 48 DIN-Taschenbuch. the European Committee for Standardization. providers as organisers…) we can conclude that this understanding is rather limited for the educational field. The educational phenomenon and the relations between its actors (participants. Normen zum Qualitätsmanagement. In this definition quality is related to inherent characteristics. But. Beuth-Verlag.With these “poles” in mind we can affirm that in the relationship between quality and educational evaluation there is a plenty of potential for challenging pre-defined boundaries and. Then it would be related with the expected requirements of coffee (strong. aroma…). This “triangulation” is very relevant to explore the notion of quality. comparisons or other experiences) we would conclude using some adjectives: it is a coffee of good quality. or “quality is: satisfying the needs of the customer”. to a client-provider relationship is. Even if we do have a generous and flexible interpretation of this commercial understanding of quality (identifying customers as participants. 2001. Berlin Wien Zürich. organisers…) go beyond this “commercial transaction logic”. of poor quality… The same as we have said for a coffee we could say. good or excellent” 48. published as a European standard EN ISO 900047: “Quality is a degree to which a set of inherent characteristics fulfils requirements. or experience to being evaluated. The term quality can be used with adjectives such as poor.be/cenorm/ 47 ISO (International Organization for Standardization) is the world’s largest developer of standards useful to industrial and business organisations of all types. Now CEN is contributing to the objectives of the European Union with voluntary technical standards. a simplification.net . • The European Committee for Standardisation46 agreed on the following definition in the non-governmental and business fields.iso. as a result. to requirements (set or expected from outside) and to a subjective value judgement expressed in terms of adjectives. The quality of a coffee would be related at first to the fact of being coffee: to all the inherent characteristics of every coffee (taste. to governments. And finally. More information at http://www. was founded in 1961 by the European national standards bodies. Page 451 T-Kit on Educational Evaluation in Youth Work www. for mutual enrichment. weak…).training-youth. in the best case.cenorm. to suppliers and customers of products and services in both public and private sectors… More information at http://www. This approach certainly has its place when it comes to the management of programmes or to the use of public resources.

The Quality Management Cycle and youth work Dr. Lucie Press. But. The dimensions of quality identified by those authors are: • Quality as “fidelity” to the aims.html. adapting… There is a chronological order to the cycle.net/methods_demingcycle. The quality management cycle Every project is like a circle with four parts or phases: Planning (P) Acting (A) Doing (D) Checking/Studying (C) Planning: Doing: First we need an idea for the project. 1995 50 American statistician and quality-control expert. from our point of view. the concept of quality is still not very clearly articulated. which are. • Quality as transformation (qualitative change). “Quality in Education: An Implementation Handbook” Jerome S.valuebasedmanagement. Each part builds upon the last. Checking/Studying: While working we observe smaller or bigger gaps between our plan and the results. Jerry Arcaro Publisher: St. Arcaro. Deming used statistics to examine industrial production processes for flaws and believed that improving product quality depended on increased management-labour cooperation as well as improved design and production processes. It is not possible to change the direction of the cycle or to turn the system around. We do not consider this “definition” as the “right one”. • Quality as coherence . auditing. With this background. inspiring and challenging in our educational field. 49 Two interesting books for further exploration are: “Total Quality Management in Education” Edward Sallis Publisher: Routledge Falmer 2002 .ethos.training-youth. correcting. Iowa.). these four dimensions in the understanding of quality are particularly pertinent. We follow our plan as closely as possible. T-Kit on Educational Evaluation in Youth Work www.. 51 Source: http://www. We start to transfer our plan to reality.net . have identified and analysed four different dimensions of quality. Value Based Management is a portal with specific information on value creation and managing for value and valuation. We think about aims and potentialities of it. valorisation. policies and public funding provisions. a substantial contribution for the understanding of quality in non formal education.. We figure out what is going fine and what needs to be changed.0 • Despite the large number of references and the growing body of literature about quality in education49 (quality control. They see quality in non-formal education as a mix of those four dimensions. William Edwards Deming 50 (1900 – 1993). b. Then we work out a concrete concept and work plan. Sioux City. Acting: We take action on our critical review: changing. known as the father of quality management developed the so-called P-D-C-A-Cycle51. • Quality related to something new. in our opinion. Harvey and Green (1993).

Brochure No. the quality is good and if it is not fulfilled not so good or bad.75 ff. p. because the concrete end is often already the first step into a “new” project.The cycle theory in youth work Demings’ theory was a starting point for developing different models of quality management system for different fields (industry. the primary sector…) In later adaptations. an amount. They define the expected amount. engagement. 24. Therefore. T-Kit on Educational Evaluation in Youth Work www. 2001 and “A resource book for social managers” by Paul Nies and Philip C Berman 2004. Indicators are measures that indicate if expected quality described in the standard is achieved. The assumption is that if a criterion is fulfilled. European Management Association. • Quality indicators: Most of the time we are not able to directly observe a criterion. Deming’s theory has had an important impact on this field. Standards are often expressed as the highest or lowest value or allowed range. In order to that they identify three key concepts for quality management52: • Quality criteria: characteristics chosen to define quality in coherence with the objectives of the project. Bonn. Quality criteria are relevant and valid. Youth projects in general have as objectives the promotion of active participation. co-responsibility and the use of active methodologies – The assumption is that if there is involvement of participants it is good and if there is no involvement it is bad – “Involvement of participants” is a relevant and valid criterion • Quality standards: These are conditions set to criteria. Considering that. if we wish to operationalise the criteria. youth work is mainly structured in projects. public administration. An example of quality criteria in youth work could be: Involvement of participants – “Involvement of participants” is a characteristic of an educational project – It is coherent with the objectives of the project. we need indicators. too. This understanding of youth work as project cycles has led institutions and organisations in the field to try to improve the quality of the projects they support all through their different stages. the intensity and the manner of the criteria An example of a quality standard in youth work for the quality criteria “Involvement of participants” could be: Participants self-manage and organise their projects and the free time – “The self management and organisation of their projects and free time” is a condition. Indicators can be qualitative or quantitative and should be reliable and measurable. 1 52 Sources: BMFSFJ: QS-Compendium.training-youth. As we know every project has its own dynamic and there is never a real beginning and never a complete end. a manner of looking at the criteria “involvement of participants” – There are other criteria to evaluate the involvement of participants but this standard (about their project and the free time) is what is expected. the circle developed into a spiral to underline the ongoing character of a project.net .

language. let us imagine a youth exchange project with the objective of promoting mutual understanding and co-operation between participants… • A quality criterion could be: Communication between the participants • A quality standard could be: Communication in the project happens: – Between all participants of the project – All through the different phases of the project – In the different activities and moments of the day • Some quality indicators could be: – Interaction between participants during the free and informal time – Direct communication/interaction of participants in the “working spaces” – Initiatives to overcome communication barriers (i. standards and indicators. etc) – Level of understanding / misunderstanding in case of disagreement or conflict – The use of non verbal communication T-Kit on Educational Evaluation in Youth Work www.e. Following our example. some indicators (quantitative and qualitative) for the standard “Participants self-manage and organise their projects and the free time” could be: • • • • • • Number and kind of projects Project team composition Number and kind of self-organised activities during the free time Social committee’s way of working Sharing responsibilities … Quality criteria Quality standards Quality indicators Example of quality criteria.net … . standards and indicators In addition to the example given while defining quality criteria.training-youth.

thus of youth work as well. quality management. The global approach to educational evaluation as a total experience. This catalogue of criteria. As we saw in the section called “What to evaluate?” (Pages 19-28). social inclusion. To better position non-formal education and learning in a wider socio. First. as we propose in this T-Kit. dissemination of the learning offer made and success. standards and indicators is useful in two ways.Here we have developed just one example. For both institutions. preparation of participants. Ideas about quality management enrich strategies of educational evaluation bringing new perspectives to evaluation by achievements and by objectives. but is not limited to.training-youth. cost effectiveness. living diversity and developing tolerance of ambiguity. it is indispensable to introduce specific quality criteria. A further example of such a process is the “Tool for developing your own evaluation plan . When touching on this and other related questions we enter into the complex debate on the quality of non-formal education in Europe. Quality is indeed about ensuring reliability and validity. trainers and youth workers in Europe. especially without qualitative information about the process) they cannot embrace the complexity of an educational experience. Every project is unique so we rather create our quality catalogue with the team.SALTO Check list” (Pages 105-112). evaluation and the links to other experiences in education and learning with regard to personal development. public and civic life or the labour market. standards and indicators. political and economic context. which has been stimulated by the Council of Europe and by the European Commission. The quality of non-formal education and youth work is an ongoing discussion among researchers. benefits from. it defines what quality is and means in our project and second it is a reference for our educational evaluation. cognitive learning and understanding.  Quality in youth work: steps and debates in the European institutions Quality in youth work at European level What do we consider “good” or “bad” in our work in educational youth projects? Establishing evaluation criteria is an essential and sensitive part of the evaluation process of an educational activity. discovering universal values. evaluation by achievements and evaluation by objectives are central in educational evaluation. At European level it includes intercultural competencies such as communicating in foreign languages. without the complementary information provided by other approaches (and. respecting difference. with the group and eventually maybe with the support of an external evaluator. All those together constitute the “quality catalogue” of our project. For each objective of our project it is necessary to develop specific quality criteria. Such criteria apply to organisers.net . T-Kit on Educational Evaluation in Youth Work www. living in groups and communities. But. in setting up criteria for evaluating educational activities we inevitably enter the field of the quality of non-formal education and youth work. youth workers and trainers and their performance. for every new project. it is becoming a crucial question especially in the efforts of these institutions to develop the social and political validation and recognition of non-formal education. Quality is also about relevance – with regard to life skills. Further. locations chosen. coherence.

recognition. Especially the Partnership between the two institutions (European Commission and the Council of Europe) plays an important role in the field of quality development in youth work and non-formal education. but they modified their views as the course proceeded. it opens up our very deep concerns about the values and impact of youth work. including a limited number of experts. which can help us understand the nature. the curriculum developers of the course. More information at http://www.youthforum. self assessment. our role as youth workers and the mission in the profession. It has organised several activities (expert meetings. the importance and beauty of this subject. ATTE ultimately decided to use self-assessment procedures.net . the National Agencies for the YOUTH Programme as well as the Course Team and participant trainers. which are federations of youth organisations in themselves. “ATTE did not make great progress in establishing quality criteria. therefore an appropriate evaluation of its effectiveness with respect to individual learning. quality standards in the educational youth work. the European Commission and the Council of Europe put a strong emphasis on quality assurance and quality development in their policies and activities. organised in order to represent their common interests.html T-Kit on Educational Evaluation in Youth Work www. The following documents are particularly interesting because they deal with quality and the related topics of validation. Practice 2 53 The European Youth Forum is made up of more than 90 National Youth Councils and International NonGovernmental Youth Organisations. the network of National Agencies and the SALTO Resource Centres. seminars. who have the competence and the motivation to develop and implement European level training activities in the youth field. • The long term training course Advanced Training for Trainers in Europe (ATTE) has been developed to meet the increasing need for qualified trainers in this field and to enlarge and further develop the European networks of trainers. complemented and challenged by feedback from peers. It brings together tens of millions of young people from all over Europe. These issues are seen slightly differently in different countries. amongst the establishment of an expert group on “Quality Standards. tutors generally held the view that assessment of any kind has no place in non-formal learning. Clearly this issue is controversial. Evaluation and Validation”. At the outset. trainings) to stimulate the widest possible consultation on this issue.org 54 Accessible at http://www. which are by no means exclusive to the non-formal youth training sector.training-youth. reports and educational materials focusing on different aspects of quality. from different cultural perspectives.net/INTEGRATION/TY/TCourses/olc_atte/atte_evaluation. European level initiatives on quality in the youth field Both. group learning and institutional investment was seen as necessary from the very beginning.training-youth. The Evaluation Report of the Advanced Training for Trainers in Europe54 is therefore of remarkable relevance. for instance in the “Advanced Training for Trainers in Europe” with very valuable approaches and results. but it has succeeded in raising participants’ awareness of the need for quality criteria and quality monitoring. Important work on the topic has also been done within the frameworks of the Youth Programme of the European Commission. proved to be ATTE’s thorniest challenge. Participants were prone to think that implicit assessment of their qualities as trainers was taking place continuously. The issue has been addressed in many training for trainers courses. the course team and external experts. Transparency is the key to resolving such problems. the European Youth Forum53. The evaluation involved various partners. While this debate has not yet been concluded it has produced several findings and questions for further debate. Through its new and innovative approach it was an important step towards ensuring the quality of training of youth workers and youth leaders at European level and towards the recognition and certification of training for trainers in the field of non-formal education. across regions and by a variety of individuals. ATTE was seen as a pilot course. Whether to assess the quality of learning outcomes and then how to do it. There is a growing body of working papers.

quality and innovation of activities so that the Council of Europe can remain a trend-setter and high quality non-formal education provider in the youth field. the definition of competences addressed and learning outcomes for the participants. visibility. August 2005 55  T-Kit on Educational Evaluation in Youth Work www.training-youth. thus laying the foundations for further action. March 2005. the relevance to the Council of Europe programme and youth policy objectives. (2006) Authors: Lynne Chisholm with Bryony Hoskins. concrete and assessable objectives. edited by Lynne Chisholm. DJS/ET (2005)1 57 Accessible at http://documents. The paper was a milestone in the discussion on non-formal learning and led a number of political initiatives. reproducibility. accountability. • In February 2004 the Partnership has produced a working paper on “Pathways towards validation and recognition of education. innovation and research. Volume 2 . • The political debates in the European Union coordinated by the European Commission led to the Council Resolution on the Recognition of the value of non-formal and Advanced Training for Trainers in Europe. According to the key document on this subject quality of the education and training activities is assured by and verifiable through a range of criteria: a relevant needs assessment. Bryony Hoskins with Christian Glahn. Half the participants met the core expectation to produce a TQP by the final seminar.coe. a competent team of trainers.net/documents/39. Belgium. • As a result of a research seminar on non-formal learning which was organised in the framework of the Partnership Programme in April 2004 a more precise picture of the impact of youth work and of the skills and competences of those active in the youth field was achieved58.” 55 • With regard to quality standards in education and training activities of the Directorate of Youth and Sport of the Council of Europe the development of quality criteria56 aims at contributing to the deepening and widening of transparency. a European level validation instrument specifically for activities in the framework of the YOUTH Programme. The Portfolio has been completed and will be available – after a testing period – in early 2007.pdf 58 Trading up – potential and performances in non-formal learning. • Based on a decision of the political bodies at the Council of Europe the organisation convened an expert group to produce the European Portfolio for youth leaders and youth workers. an adequate and timely preparation process. Council of Europe. suggested ways to demonstrate evidence of experience and a glossary. The test version is available at the following website: www. an integrated approach to intercultural learning. clear. was aimed at increasing the visibility of the value of youth work and discussed ways to a better recognition.youth-knowledge. adequate. a training quality product (TQP). functional analysis. and many who did not regretted that they had not done so.int/youthportfolio.External evaluation. a consistent practice of non-formal education principles and approaches. training and learning in the youth field”57 It summarizes the state of affairs and highlights a strong need for social and formal recognition of non-formal and informal learning in youth activities.projects. This tool includes methods of self-assessment enriched through external feedback. • As a follow-up the SALTO Resource Centre for Training in Germany got the mandate to develop a specific “YOUTH-PASS”. Available at http://www. accessible and timely documentation. Page 12. a thorough and open process of evaluation. a portfolio and participation in the ATTE seminars themselves would furnish the evidence. Marianne Søgaard Sorensen. sustainability.net/INTEGRATION/TY/TCourses/olc_atte/atte_course_pub_vol2.net . Ib Jensen. competency tables and levels of assessment.html 56 Quality standards in education and training activities of the Directorate of Youth and Sports of the Council of Europe. structurally optimal working conditions and environment. Lejf Moos.training-youth. • The Bridges for Recognition conference in January 2005 in Leuven. recruitment and selection of participants. given the value of the feedback sessions. adequate institutional support and an integrated follow-up within the DYS programme and its partner organisations.

organisers and organisations.net/INTEGRATION/EKC/Intro/index. organisers. sustaining and further developing quality in European youth training and nonformal education is one of the objectives of the Partnership Programme on European Youth Worker Training run by the Council of Europe and the European Commission. which can help us in understanding the different dimensions of quality. T-Kit on Educational Evaluation in Youth Work www.training-youth. • for trainers. html where downloads of a wide range of information are available. • for sponsors and public authorities. Evaluation is an indispensable element of the process. • There are several other documents. For this we would like to recommend a visit to the website of the Partnership at http://www. who want recognition of the quality of their work in the field of non-formal education and training. who want a quality learning offer. Therefore.training-youth. Linked to the interest in the quality of non-formal education and training there is the interest in recognition of the non-formal education sector as a whole and in particular of the offers in this field and of those who offer it – trainers. adopted under the Austrian Presidency in May 2006.net . etc. informal learning within the European youth field. who have an interest in an effective use of the funds and the support they provide in this field. Conclusion: Quality in European youth training and non-formal education has increasingly become a concern for the stakeholders and actors involved: • for participants in training and non-formal education activities.

10 Develop your own evaluation! Experienced evaluators agree that there are no “golden rules” for the development of a perfect educational evaluation. T-Kit on Educational Evaluation in Youth Work www. The most important is to remember that. methods and instruments that will support the development of your own educational evaluations and for integrating educational evaluation into your projects. it must actively support them. We believe that the definitions.net . Be creative and courageous! Just give it a try! In the next more “practical” part of the T-Kit (“Pots. if the educational evaluation is not to undermine the objectives of the project. Pans and Spices”) we will present you with some tools. then. models and questions explored in this “theoretical” part of the T-Kit (“Ingredients of Educational Evaluation”) can help a lot but they cannot guarantee an evaluation process free of resistance and mishaps. 1.training-youth.

.

But what do we mean by that? Can we describe what kind of behaviour we see as improved intercultural competence? What are the indicators that demonstrate that participants are more competent in intercultural situations than they were before the project started? To set up a proper evaluation. but how did it work out in reality? Or if we use our cookery metaphor: the meal turned out fine.2. but the struggle that took place in the kitchen to get to the result was awful and needs to be improved.1 How educational evaluation is built into a project? … While it might already be clear. When we have a common understanding about that. That means that these aims and objectives should be formulated in such a way that they can be evaluated. since the team carries the educational responsibility. pans and spices” Practicing Educational Evaluation! – Educational Evaluation in Practice!  2. Evaluation needs to be planned before you start your project. The same could be said for the co-operation with other partners and stake-holders as well as for the way the different resources such as accommodation. There should be agreement about what is meant by an objective. transport and materials were dealt with. “Pots. it is certainly important for all concerned to evaluate its work. Still. the aims and objectives of your project are an important dimension of what to evaluate. The way we work in our team will probably not be mentioned in the aims and objectives. starting to prepare the evaluation the night before the last day of your project is too late. planning the evaluation is an integral part of planning the project as a whole. The way in which they contributed to the outcomes of the project can be evaluated. But. The main questions to ask ourselves when we begin are: • Why evaluate? (what are the reasons for doing evaluation in our project?) • What for? (what are the objectives of evaluation in our project?) • What should be evaluated? (What are the evaluation fields and elements?) • How and when should the evaluation be done? (What are the methods. environment. T-Kit on Educational Evaluation in Youth Work www. for example the process. which are not mentioned in the aims and objectives. it is possible to decide how we will do the evaluation and when. “Improving the intercultural competence of the participants” can be an objective. it is crucial to be clear about our aims and objectives.training-youth. sources for the evaluation?) • Who will do it and for whom will it be done? (Who are the actors of the evaluation?) As we have seen. We planned the desired process into the programme of our activity. In other words. there might be other elements of the project that we want to evaluate. We need to be clear before starting the project about what we want to evaluate.net .

there are plenty of things to consider. T-Kit on Educational Evaluation in Youth Work www.training-youth. even before the project begins. This means we have to start collecting information before the start of the project. And. the style and the image of the report? Will it be a written report? Will we produce a CD-Rom? So as you can see.net . this might mean that we should find out how inter-culturally competent the participants were at the moment they started to participate in the project. this means we have to make up our minds about how to collect that information. Will we use questionnaires or interviews? Will we collect information from individuals or in small groups? Another question is whether we want a moment during the project to check if we are on the right track and how the project is developing? What information do we need for that? How should we collect that information? Who will do that? Have we achieved an adequate balance among the different kinds of methods we plan to use for the evaluation? Are the diverse preferences and needs of participants met by the methods we have planned? What kind of reporting do we need to produce at the end of our project? For whom is this reporting intended? What are the implication of the target audience for the content. especially when planning the evaluation.0 A lot of questions are implied by this: What kind of information do we need to be able to do our evaluation? If we return to the example of “improving intercultural competence”.

When practice and evaluation go hand in hand. maintaining flexibility and responding to individuals. In the following diagram we can see a simplified description of the different stages of a project60 and vertically.training-youth.training-youth.html 60 For more details on project planning and its stages see T-Kit on Training Essentials Pages 65-74 and T-Kit on Project Management Page 43. We will use the following sections to explore this idea. a kind of responsive practice. this way of conceptualising youth projects can limit educational evaluation “mostly” to the final stages of a project.2 … A project within a project How can educational evaluation be built into our project? It helps us to understand how by thinking of evaluation as a project within the project.training-youth.html T-Kit on Educational Evaluation in Youth Work www. to the context and to new situations as they arise. If we limited it in this way. through guiding questions. 59 See Pages 43-44 T-Kit on Project Management accessible for download at http://www.net/INTEGRATION/TY/Publications/T_Kits. in the following sections. The idea of educational evaluation presented in this T-Kit implies a continuous and ongoing process. until the final implementation. design and implementation) so that we can build an evaluation plan or strategy: our evaluation project within our project. the potential of educational evaluation be can reduced and its nature can be subverted. implementing and evaluating59. An evaluation only at the end quite often implies a reduction of the educational value to the minimum and it can result in problematic conclusions. ongoing educational evaluation. Educational evaluation should not be regarded as something to be tacked on to the end of the project.net/INTEGRATION/TY/Publications/T_Kits. through all of them. This can even become counter-productive. Accessible for downloads at http://www. The three “traditional” steps of a project were defined as: planning. Consequently we should follow the stages (preparation. The different stages of educational evaluation will be disclosed. This responsive working practice becomes itself part of the evaluation process. However. Educational evaluation should happen all through the different stages of a project.1 2. from the preparation.net . The ongoing reflection that accompanies the practice brings continual adjustment and modification. it is difficult to draw a clear distinction between educational evaluation and educational practice.

training-youth.net Evaluation Reporting the outcomes Implementing the results . experts T-Kit on Educational Evaluation in Youth Work www. documentation. environment and resources • Material. Educational evaluation within a project The Poject Educational Evaluation Preparing the Project: • Needs analysis • Personal motivations • Aims of the organisation • Defining the aims of the project Designing the evaluation Preparing the evaluation Designing the Project: • Concrete objectives • Profile of participants • Resources • Contents • Methodology • Learning outcomes • Detailed programme • Methods Interpreting the findings Ongoing Collecting and organising the information Educational Drawing conclusions Implementing the Project: • Following the plans • Adapting • Taking care of the flow • Involving participants • Using the space.

actors. • Who will evaluate? – Who should evaluate and who should be involved / participate? (Young people.net . As a reflection of the approach proposed in this T-Kit. organisers. Educational evaluation in stages Together with all the theoretical background already explored. nature and general approach of our evaluation • Why evaluate and what for? – Defining the aims and concrete objectives of our evaluation. the following questions and reflections at different stages try to be a guide for “educational evaluation as a total experience” within a project. and to motivate). methods. by outcomes. We encourage you to adapt and modify the following guide to the stages of educational evaluation to account for the specificities of your project. integrating and implementing educational evaluation in our projects. times. Stages of educational evaluation Preparing the evaluation In this first stage we define the aims. the following questions and considerations can be of a great help for designing. to participate. partners)? – In principle we could think that all actors involved in the project but in which combination and at what different levels of the evaluation should they be involved? – Who is competent to conduct our evaluation in terms of the required skills and knowledge? – Who has the (political) mandate to conduct the evaluation? T-Kit on Educational Evaluation in Youth Work www. by achievements. At the same time we try to fix some priorities so that we do not fall into the unrealistic pretension of trying to deal with everything at once. sources and techniques of evaluation. to improv e. we propose to combine different evaluation approaches (by objectives.training-youth. etc) and to diversify spaces. – Adapting and making concrete the general objectives of evaluation (to learn. stakeholders. trainers. Based on our experiences.

net . the free day in the middle of an activity. the end of the preparation. which help us to recognise the achievements (achievements of the objectives. fields. “products”. information. verbal.e. learning process. advantage and disadvantages of each of them for the collection and analysis of that information. T-Kit on Educational Evaluation in Youth Work www. some moments are especially relevant for the evaluation (i. etc. learning achievements. non verbal? • From which sources will you collect the information? Page 80 • How are you going to collect and organise the information obtained? • Can you identify any clues for “understanding”? Tip: It is recommended to use a variety of methods (quantitative and/ or qualitative) given the complementary nature. timing and indicators. • Choosing indicators (qualitative. – In which time frame and sequence do you wish to organise the evaluation process? Collecting and organising the information Choose and combine evaluation methods which allow the collection and organisation of the necessary information.training-youth. • What to evaluate? – Which fields should be evaluated? – How will we group the different elements that are to be evaluated? Will we use existing models of evaluation or will be develop our own? – What should be prioritised? Tip: It can be interesting to evaluate aspects such as context.) • When? – Educational evaluation is a continuous and ongoing process. methods and outcomes group process. etc). organisational frame. the first day of team work. At the same time. • How to evaluate? Pages 57-74 • Which evaluation methods will you use? • In which sequence and combination will the methods be organised / used? • Which different kinds of information? Quantitative or qualitative? Written. technicalities. participants’ profile. data. Designing the evaluation plan Articulate your evaluation plan with clear objectives. We can foresee and plan those moments when we will pay special attention to the evaluation. quantitative?) – Indicators: Signals. content.

training-youth. • Look for patterns. • Which different target groups will your reporting address? • Which emphasis will it have? • What kind of evaluation report will you produce? T-Kit on Educational Evaluation in Youth Work www.net . Page 81 – The criteria are elements. • Remember the aim of each element evaluated – what is it for? • Establish criteria. – Interpretation is always subjective. check alternative interpretations and consider previous understandings of similar facts. It is important to be honest. findings and conclusions resulting from the evaluation with the different actors of the project and with other interested audiences. This is a central part of every evaluation. involve other people. • Recall the objectives of the evaluation. common characteristics or repeated mechanisms. Interpreting the findings – What do they mean? Interpretation or “judgement” gives meaning to that which is evaluated according to the aim. distinguish facts from opinions. you can try to diversify sources. cross-check information collected. Drawing conclusions This is the moment for “extracting lessons” from the evaluation. considering the information collected and using the criteria and indicators previously identified. Be aware of that and make that fact explicit. which will allow us to compare reality with the objective or the expected outcome of the educational activity. • State the results achieved (facts). • Identify what happened: Was it a consequence or an accident? Is there any cause-effect relationship to be observed? Was it foreseeable or not? • Take into account that not everything is measurable! • Think about “Objectivity” and “Subjectivity”. To “limit” the degree of subjectivity. • What lies was behind these results? What do they imply? • What can we conclude and learn from them? Pages 82-84 Reporting the outcomes Share the information.

stages of the educational evaluation process: • How to evaluate? Evaluation methods • Collecting and organising information • Establishing criteria • Reporting the outcomes We will look at them in more detail in the following sections. Implementing the results The implementation of the results of the evaluation is the last step and can be the starting point of another project.training-youth. measures. I suggest that you add a reference to this symbol as well at the same point for the sake of coherence. and at times especially sensitive. future strategies are necessary or implied by the results? Some advice at this point: • Be constructive! • Be realistic! • Be aware of resistance to change! • Foresee adequate time! • Start to implement the results when the conclusions are fresh and in everyone’s minds. when the energy is there and without waiting too long! We have marked with a magnifying glass some important. partly in reference to how to use the T-Kit. adaptations.net . • What should be improved? • What should be maintained? • Which changes. In the section introducing the T-Kit you refer to the cook symbol. T-Kit on Educational Evaluation in Youth Work www.

interview or survey. It is true that a well-designed and well administrated questionnaire can provide a lot of useful information. respondent means one who is inspired to participate in a focus group. Questionnaires are unfortunately the first thing that comes to mind when we speak about evaluation. • In structured interviews interviewers may administer questionnaires in face to face encounters • Reading the items to respondents and recording the answers. surveys are a method for collecting qualitative and quantitative data. 61 When specifically applied to research. Some examples of personal evaluation methods are: Surveys and Questionnaires Put simply. or interviewers may conduct telephone surveys. Comparable information is gathered by using standardised methods.net . with some adaptation. 2. and feels that they have done the job of evaluating. the learning process. Many of the methods described can. serve for evaluation purposes within teams or with other stakeholders of our project as well. such as questionnaires. the actors involved in the evaluation make their own judgements and draw their own conclusions concerning the progress of the activity. Unfortunately. as we have argued before. Source Encyclopaedia Britannica Online http://www. training courses.training-youth. interpersonal and group methods. However. This does not mean that these methods cannot be used in other circumstances.com/ T-Kit on Educational Evaluation in Youth Work www. their personal feelings and involvement. One way to classify evaluation methods is to distinguish between personal. we have chosen to focus on methods that are used for evaluating with the participants of youth exchanges. We will explore the uses of several evaluation methods within each of these categories. questionnaires alone can only provide part of the overall picture. You may administrate questionnaires in three basic ways: • Self Administrated questionnaires may be completed by the respondents61 themselves. the outcomes. Personal methods Personal methods are activities in which.britannica.3 How to evaluate? Evaluation methods In this part and when describing different methods. etc. EVS projects and other educational activities. because in some projects the team simply distributes a questionnaire on the last day.

disagree. Female). What do you think about the T Kit on Educational Evaluation? A. However. you should pay attention to three very important issues: • Content • Question type • Question structure Content is related to the overall purpose of the questionnaire. Example: Q. To see if the participants were satisfied with the logistical aspects of the organisation of the activity such as the food may be easy to measure through a questionnaire. they can add it themselves. youth worker) Ordinal: Interval: The responses are categories that can be ordered. or may involve a range of different choices. However. you may need other methods. strongly agree). Example: Q. Kiss the bunny 4. Gender (Male.net . Open-ended questions provide space to the respondent to answer in his/her own words and own logic. the answers to closed-ended questions are much easier to analyse. cross. Question types In a questionnaire. Age (measured in years). You may list all possible responses and an additional category (other) so that if the list does not cover the respondents desired answer. Position (Volunteer. agree. Group juggling Closed-ended questions may have three different types of responses: Nominal: The responses are categories that differ in name. I always read it. Camel wrestling 2. Moje Sutra 6. of course. in addition to a questionnaire. if you want to measure the degree of intercultural learning achieved during the activity. an advantage of using open-ended questions is that you may learn new and unexpected things. This is. Closed-ended questions have a limited number of answers while open-ended questions have an infinite number of answers. The respondents are asked to mark their response using a tick. By their very nature. A closed-ended question offers respondents a number of response choices. Number of volunteers. T-Kit on Educational Evaluation in Youth Work www. Their response may be a simple Yes/No. due to its complexity ant to its group dimension. especially before going to sleep… Sometimes a combination of open-ended and closed-ended questions is the best. I think it is a cool book. You should make sure that the subject can indeed be measured through a questionnaire. Chinese firecrackers 5. Shark in the pool 3. or circle etc. The results of the evaluation can be very rich. When you are designing a questionnaire. Agreement (Strongly disagree. trainer. as you usually cannot guess all the possible responses to a question. the questions may be classified in two categories: closed or open-ended.training-youth. Which energiser(s) you have enjoyed most? (Please circle two) 1. They can be extremely useful. The responses in which the distance between categories is meaningful. an important consideration. Male/Female.

e. You should not overly limit the options of the respondent.When designing a questionnaire.e. circle a number. literacy and written expression capacities of those who will answer the questionnaire DO NOT: Use double negatives Use technical jargon and abbreviations Use emotionally loaded words (i. That is why it is a good idea to review the programme with the participants before distributing the questionnaire.: “it’s a piece of cake”. you can ask: How satisfied are you with the food? But. annoying.training-youth. For example. Question structure is another important dimension of designing a good questionnaire. “How did the group process contribute to your learning?”) Present two questions in one Put important questions at the end of the questionnaire T-Kit on Educational Evaluation in Youth Work www. check the box. if you wish to evaluate the programme on the last day of a project. “Name the innovative elements of this course”) Use long complex questions Use idioms or culture specific expressions (i. etc. tick. direct.) Be consistent: clear. to provide the respondent with the options to answer only I love it or I hate it is too limiting. However. Maybe some people think it was good. Designing clear and well-written questionnaires is a real art. participants tend to focus on scale or score questions. “green light”. If a question is closed-ended. but did not fall in love with it.e. Open-ended questions are often replied to without a lot of detail. there are no magic rules for how to do this. you have to make sure that the responses you provide cover all possibilities. you should also check if you need more than one question to evaluate the aspect that you are investigating.: the information collected will be used to compile the evaluation section of the final report to the funding institutions) Present it in a user-friendly manner and give it an attractive layout and appearance Provide an adequate variety of options for answering closed-ended questions Consider the language. When writing your questionnaire: DO: Use clear and short questions Put your questions in a logical order Try to be positive and motivate a reply Give clear instructions (i. The respondents should be able to remember the context of each question and they should also have the necessary information to be able to reply.e. fantastic) Use leading questions (i. precise. Give the questionnaire a title (i. you have to be clear if what you need to measure is better measured using open-ended or closed-ended questions.e. “fire brigade”. Often when filling out questionnaires. “cup of tea”.e. Unfortunately. burning issues”)  Use biased questions (i. exciting. if evaluating the food. there are some simple things that can help you to improve your questionnaires. participant evaluation questionnaire of activity X?) Give a short introduction to the questionnaire explaining what you will use the information for (i. as they are easier to answer and take less thinking effort.e. frustrating.net . Regarding the question type.

The problems are exaggerated on purpose but the logic underlying them is present in many questionnaires and other evaluation methods. Discuss them with your evaluation colleagues. . completed and concluded our programme? THANK YOU FOR ANSWERING OUR QUESTIONNAIRE!!!!!!!!! T-Kit on Educational Evaluation in Youth Work www. Turkey 4th -10th October 2004 Organised by Youth Express Network and supported by the European Youth Foundation of Council of Europe. “Evaluation seminar” Avanos.net 62 Kneale. Dylan. Discuss how these problems can be avoided? The Bad Questionnaire Please complete the following questionnaire – thanks! How old are you? 1-20 years old n 22-24 years old No n over 26 years old n What is your ethnicity? Yes Not sure Where did you hear about the programme and why did you decide to come on the programme? • My Mum told me about it and told me to come on it • In the newspaper and I decided it would be a good idea • My teacher said it was something to do after I had finished school and I agreed • Other: What did you learn on the programme? Everything n Nothing n Often Do you like to play football Yes l No l l What are you doing now? In employment n Both n In education or training What is your current occupation? (for example: I am a farmer) I am a ________________________________________________________________________________________________________ Where do you live? What is your name? Yes n No n Will you tell all of your friends how good this programme is? Yes l l No l l Not sure l l Have you got a wife yet? Yes No Not sure How has your opinion about life and the universe altered now that you have most successfully terminated.0 The Bad Questionnaire exercise 62 Have a look at the Bad Questionnaire and try to identify the problems with it.training-youth.

they seal it in an envelope on which they write their full name and postal and address. As it is a personal method. 2 months after. fears. the participant may write in their own language and what they write is totally between the participant and. the experiences accumulated outside of the “official programme” are also very rich. of course. before departure. other participants or the country (if they have travelled abroad to the activity).Without wishing to present a model (rather. what have they learned. which part of the programme have they enjoyed most/least. In residential seminars. If the exercise is taking place on the last day. After all. why they have come to the activity (their motivations). you may ask them to write about a large variety of issues. influence or interact in the process as s/he does not read the letter. in the middle or on the last day.net . The decision will be made in function of what you want to achieve with the letter. an example!) we provide a possible structure for a “good” final evaluation questionnaire (Pages 97-101). are they satisfied with the activity. it is a letter to “yourself ”. for example. it can also assist the participant to become aware of certain progress made as a result of his/her participation. If the participants are writing the letter on the first day of your activity. In all cases. their first impressions of the team. The “letter to yourself ” can achieve some very interesting and in depth results about the impact of a programme. A diary can be also useful for documenting and evaluating these activities. very suitable to projects that last more than just a few days. in which they can note down the experiences of each day during the activity. The team provides the participants with a notebook (the diary). The facilitator does not have any possibility to intervene. you should keep in mind that the “letter to yourself ” is a very personal method. When you ask them to prepare the letter depends on the programme of your project. It is also one of the most efficient. what are they taking back home with them. the letter may include similar issues. their expectations.training-youth. themselves. it is important to provide enough time and space for writing this letter and its importance should be made quite clear to all participants. This can be a good method in EVS-projects to help the volunteers to reflect on the large number of new impressions that they have to deal with while living and working in another country. for example. This may be on the very first day of the programme. If you are planning to use this method. Diary Another way of giving participants the opportunity to reflect on the process they are going through is to ask them to write a diary. but focus on different aspects. for example. This is. well. You ask the participants to write themselves a letter. 1 A letter to oneself (also known as “letter to yourself ”) The “letter to yourself ” is one of the simplest and most commonly used personal evaluation methods. The team sends the letters to the participants after a period of time after the project has ended. T-Kit on Educational Evaluation in Youth Work www. This allows the participants to make a comparison of their initial perspectives and feelings with the ones they have 2 months after the project. which s/he would not realise otherwise. what is their plan of action for implementing the things they have learned once they get home? Once they have finished writing the letter.

The main difference between these types is the amount of flexibility allowed to the interviewer and to the respondent. A diary that looks nice or that is presented in a creative way can motivate. T-Kit on Educational Evaluation in Youth Work www. for example: “How did you feel during the simulation exercise?” Even though the answer of each respondent will be different. The participant should feel comfortable with and have trust in the interviewer. The majority of the questions will be open-ended. aspects of what the participants might note down in their diaries can be discussed in particular group settings such as for example “reflection groups”. Just two examples of interpersonal methods of evaluation are interviews and focus groups. just a list of important issues that the interviewer must cover with each respondent. For example: • Describe what happened today in your own words • Were there any remarkable moments? • Are there things/questions still buzzing around in your head? • Is there anything ‘new’ you learned today? • Is there anything you want to pay further attention to? Remember that participants may need to be motivated to use their diary on a daily basis.training-youth. The interviewer has a questionnaire and reads out each question to the respondent. Interpersonal methods In interpersonal evaluation methods more than one individual actor share and discusses their judgements and conclusions.net However. mentors or facilitators! But. what is important is that the interviewer acts in a neutral manner and tries not to direct or influence the responses of the person being interviewed. Interviews There are two types of interviews. the question remains the same. to confront opinions and to learn. The value and purpose of those methods is to share. . We provide a description of reflection groups in a later section (see page 67). This takes place because participants have the opportunity to evaluate from more than one point of view. Of course a diary is something personal and therefore is not to be read by youth leaders. One approach to this activity is to provide an empty notebook for participants to use at their discretion. The interviewer has much more autonomy and flexibility in the development of a conversation with the respondent. but another and one which is often appreciated by participants as they receive some guidance. often in a pair or small group. • Structured interviews In a structured interview both interviewer and respondent have very little flexibility because there is a questionnaire that has to be followed. in both types of interviews. • In depth interviews In an in-depth interview there is no pre-set questionnaire. in depth interviews and structured interviews. is to give some guiding questions.

for example. “Can you explain in more detail?”. we should consider if the respondent will answer the question truthfully. For an ideal interview. Participants may feel under pressure to say even yes if they do not think the trainer is good. “In which ways?”  T-Kit on Educational Evaluation in Youth Work www. instead of “Did you enjoy the programme?”. the trainer asking participants in an activity: “Do you think I am a good trainer?” is not very appropriate. For example. they have the advantage that they can give a better insight into why a participant thinks that. knowledge and competences. you can ask: “In your opinion. As in other interpersonal situations. it is possible that we will want to collect information about where participants heard about the programme and what motivated them to get involved. “What do you mean?”. non-directive question) can be used. which aspects of the programme were most useful?”. As we see. It can be useful to record the interview with a hand held tape or digital recorder.Before starting an interview we should always be clear about what we want to evaluate. But for an in-depth interview it is also very important to prepare in advance. Recording the interview can liberate us to note down such behavioural observations on paper. Imagine asking: “Did you enjoy the programme?” The participant will say “Yes”. • Interviewers should be neutral: their presence should not have any effect on the responses given to questions. When preparing and making in depth interviews. A lot of resources are required to do and analyse them. We are aiming to get as much information as possible. For example. the participant should feel comfortable with and should trust the interviewer. If we are conducting a structured interview. But. It is not appropriate to ask a respondent to slow down. We should ask ourselves if we are the right person to conduct the interview or if it is appropriate for us to do so at all. make sure to list the items you want to discuss with the respondents.net . Interviews are very work intensive and time consuming. if our aim is to evaluate the recruitment of participants. In case the responses given to an open-ended question are unclear or not sufficient in the opinion of the interviewer. it is important to take following points into consideration: • Try to avoid closed or dichotomous (YES/NO) questions. • Before asking a question. the “race”. a probe (a neutral. in an interview. gender and even accent of the interviewer can influence a respondent’s answers. Preparing and making in depth interviews It is a good idea to prepare some questions or prompts before starting any interview. we have already prepared a questionnaire. • The questions should trigger the respondents to think. For instance. Some examples of probes include. Even if you do not formulate the questions. as this would interrupt their thought flow. and that will be it. conducting interviews requires some experience. It might also be relevant to take notes about the non-verbal communication that takes place during the interview. This is a practical issue. socio-economic background. the energisers should get 4 points out of 5. “Anything else?”. We recommend not to get too personal and not to ask questions that would put the respondent in the position of lying out of politeness or for legal reasons. Trying to write everything down at the same time as conducting the interview is very complicated and may even be impossible.training-youth. It is important to record the answers of the respondent during the interview.

the young people themselves are the “group of people” that forms the focus group and their common subject are aspects of the project that need to be evaluated. For example. But. Thus. extreme care should be paid to group dynamics and the amount of peer pressure that exists in any group when deciding how to approach the evaluation or when deciding whether to use focus groups. A focus group is made up of a group of young people and a facilitator or moderator. at the end. as the name implies. we should help participants to stay focused and be ready to allocate enough time for them to express their opinions. Focus groups have some important advantages and disadvantages. To overcome this is good to collect all together.training-youth. Their small size and relaxed atmosphere allow for deep and frank evaluation. But. One final but nevertheless important disadvantage of the focus group can be the eventual generalisations that can take place in the discussion. focus groups differ extensively from interviews. But. T-Kit on Educational Evaluation in Youth Work www. several participants who have already been in similar activities might express their “impatience”. as it is not rare to have one or two outspoken members who speak in the name of others in an activity. the main conclusions or simply a mechanism of rotation for taking notes. however. Even though it sounds quite similar. The participant reaches an opinion as a result of the discussions with other participants in the focus group. Focus groups. If we are going to form a focus group during the programme of a given activity. a focus group is “a group of people sharing similar characteristics who talk about a subject they have in common”. the first day. It is very difficult to make sure that all participants will participate with the same level of concentration and attention. this does not hold true in all cases. leads them to think out loud and form opinions during and as a result of the process. are a group activity. the reason why a participant is thinking in a particular way is as evident as the opinion they hold. In a focus group. Another major advantage of focus groups is that they encourage participants to speak in public. The first and most important advantage of using focus groups as a method for evaluation is that they allow you to explore the way in which young people form their opinions as well as finding out what their opinions are. interaction between the participants. In a focus group. the silent majority of the group might have another opinion. The main difference between focus groups and interviews is that an interview is a one on one activity that takes place between an interviewer and an interviewee. As a result they tend to express their opinions more openly and more bravely in a focus group situation. In a youth project. Group pressure may also negatively affect the sense of security of participants to honestly express themselves. Many participants feel more secure expressing themselves in a small group of their peers than sitting alone in a corner with an interviewer. forming focus groups also has certain difficulties. Focus Groups In very simple terms. We should check if the opinions raised are representative of all the participants in the group. It should also be noted that during a focus group.net 63 ibid. .63 Focus groups are one of the most efficient qualitative and interpersonal evaluation methods. it is very difficult to record the information and to keep track of what is being said at the same time.

net . the atmosphere in the group. group life. This can create a negative reaction from the participants • Be very patient. Remain focused. as it takes time for the discussion to build up and for all members to feel invited and comfortable enough to join in. These methods not only inform us about the views and feelings of the group. The team has every interest in understanding how participants view their purpose in participating in the activity. Personal or interpersonal evaluation methods like questionnaires or interviews with participants might be time-consuming. but also create the possibility for participants to reflect and to learn about how others experience the activity. To formulate what you want helps you to remain focused on your needs during the programme.  Group interactive methods Group interactive methods of evaluation can be used to check what the feelings or views of the group are at a particular stage of the programme or to evaluate a certain session. Shorter interactive group methods. Of course we can do that in an informal way by observing the group and the activities and by asking questions. it is very likely that the participants will be clear too • Try through everything you want to discuss in just one session. It can make the participants feel pushed • Make clear at the beginning what you want to achieve by holding the focus group discussion. T-Kit on Educational Evaluation in Youth Work www. the co-operation among participants… Checking expectations What are the expectations of participants towards the programme? What do they want to take home as a result of their participation? Are there things that they absolutely do not want to do? For different reasons it is good to ask participants to take some time to reflect on what their expectations are at the beginning of the activity. These observations would be valuable but at the same time subjective to the team or the observers. If the facilitator is clear with the participants. and limit the number of topics and issues for discussion to the most important and relevant ones. Otherwise you risk that the participants lose interest and you may end up with very little information. Since the group in non-formal education is source of learning. which include elements in addition to verbal and written expression. depending on the programme.training-youth.Here we have collected some general recommendations for facilitating focus group:64 • Keep the group small (ideally 4 to 6 people) • Take into consideration the language skills of the international participants as it affects the time they need to communicate their opinion • Keep your questions neutral and avoid manipulating the discussion with biased questions. 64 ibid. It is counter productive to interfere in the discussion too much and to constantly offer your own opinion. are a good option. This is essential for the team. Group evaluation methods have an additional dimension. group evaluation methods can specifically deal with group dimensions of the learning process including for example. Tell participants what the aims and expectations are. It helps the team to refocus the programme or clarify to the participants what they can and cannot achieve in the context of the activity. For participants it is interesting to find out about the needs and wishes of the others.

Fears or Offers to the programme that participants may wish to make. On sticky pieces of paper participants are asked to write down their answers to each of the questions/poster-titles and to stick them on the appropriate poster.training-youth. Clothes line Three posters A similar method involves placing three big blank posters on the wall. T-Kit on Educational Evaluation in Youth Work www. Clothes line An effective method for collecting the expectations of participants is called the ‘Clothes Line’. Each one is given a title. Time is given for participants to reflect on their expectations using the categories of clothing as a prompt. All participants receive papers on which items of clothing are drawn. Remember to give the group sufficient time to review individually the answers of all the participants. Hopes. It is important that the group has the time for it. ‘What do I want to avoid?’ and ‘What can I contribute?’.net . Everybody has a chance to look at the “laundry” of the others. Each item of clothing stands for a different kind of expectation: for example. They write their different ideas related to each category on the clothing shapes and stick them on a clothes line made of string (hung in an appropriate space in the working room) or drawn on the wall. A team member or facilitator can also make a review and read out the most common answers given to provide the group with a global picture of the expectations in the group. for example: ‘What do I want to take home?’.

It is important to plan appropriate time for these reviews. This allows the members of the reflection group to create their own atmosphere during the several sessions over the duration of the activity. They should leave behind the ones which have not been fulfilled. one can return to the initial expectations of participants and to reflect on the extent to which participants have got what they wanted out of their participation. They usually take place daily at the end of the programme and last between 30 minutes and one hour. But. End of the day reviews with the whole group have the advantage that participants learn about the impressions of everyone in the group and can create a sense of common responsibility for the group life and the programme in the group as a whole.  End of the day reviews In a programme that lasts more than a few days. with different personalities and profiles. Therefore. A very simple and visual method to do this is to invite participants to go to the “clothes line” or to the “posters” and to remove their fulfilled expectations. Then participants should explain the reasons why some expectations were fulfilled and others not.training-youth. This regular moment for evaluating can take place everyday (end of the day review) or at regular intervals during the programme. This is not always easy at the end of the day. Often unexpected outcomes arise at this point. The Reflection Group gives the possibility participants: • • • • to to to to exchange experiences with fellow participants evaluate the day let off steam come up with suggestions and ideas to improve the programme T-Kit on Educational Evaluation in Youth Work www. and is often done in small groups of 5 to 7 persons. small groups offer a more secure space for expression in public than a large group. it provides an effective means for staying in touch with the way the group develops and to take up any problems and challenges that may be communicated by the group. Another way to return to the initial expectations is to group them and discuss their fulfilment or not in small discussion groups. if undertaking an end of day review in a large group. We propose below some of those methods: “The Three Word Review”. it can be advisable to make sure there is a regular moment to look back on the experience of the participants in the activity part of the programme. something which is interesting for both participants and team to know about. large groups can have the disadvantage that only some few participants take the floor. In addition. But. it is important to propose methods that make it possible for all the members of the group to express themselves. rushing through such a group evaluation just before dinner begins does not allow you or the group to reflect properly and runs the risk that participants feel they are not being taken seriously. . reflection groups are a way to “keep in touch” with what is going on in the group. The expectations should be carefully considered to discern which ones can be fulfilled in the activity and they should to the extent possible be integrated into the different programme elements. This is particularly important for the evaluation.The legitimacy of these methods depends on the conscious use of the information collected by the team and by the group. Reflection Groups Reflection groups are small groups of between 4 and 6 people. . It is interesting to have in each group participants from different backgrounds. to express their feelings and to hear how others have experienced that same part of the programme. At the end of the activity. For the participants this can be a valuable moment to share experiences about the programme of the day or of several days. For the facilitators. “The Telegram” and “The Living Dartboard”.net For facilitators. Their composition is normally fixed for the whole duration of the activity. so that within each group there is wide range of points of view.

or other aspects being evaluated. participants may be invited to come up with their own statement/s about the activity. in which participants are invited to move around the room according to their position towards statements that are read out about the programme. chair. is the “Three word review” where participants are asked to write down for themselves three words that describe their feelings about the day. This activity is very appropriate after a tiring day or session when there is no energy left for a thorough evaluation. This way they have a chance to check how the rest of the group feels about it. Another way of sharing outcomes is to invite participants to write the telegrams on sticky pieces of paper and to put them on the wall for everybody to read. to contribute to the discussion. This commonly leads to a lively discussion. the exercise may be concluded by asking participants to read each others’ telegrams. can be used as ‘warm-up’ activities in reflection groups and as we said also to evaluate interactively with a large group. The more you disagree. You need an empty. for the whole group. paper. The facilitator reads out statements about the activity and participants choose a spot in the room according to their opinion about that statement. rather large room for this activity. The Three Word Review A simple method at the end of the day. After that. When time is limited. Then participants are invited to call out their words which are noted down on a flip-chart.g.). The group leader does not take an active leading role but stimulates the active participation of all group members. flower. etc. Other methods. The more you agree with the statements that the facilitator reads out. the farther away from that point you should move. After everyone has chosen their position participants can be asked to explain their reasons for standing where they are. The Living Dartboard An energetic way to evaluate the day is the “Living Dartboard”.training-youth. In the middle of the room is an object (e. Some examples of statements are: • The rhythm and timing of the programme was well planned • The methods in this activity were monotonous • Being in an international group helped me a lot to understand what Intercultural Learning is about • I missed theoretical inputs • The food was fabulous • … T-Kit on Educational Evaluation in Youth Work www. This object represents the centre of the dartboard (also known as the “bull’s eye”). The “Three word review” involves all the participants from the start which makes it often easier for those who are usually not the first to speak. The Telegram A variation on this is the “Telegram” in which participants give a general impression about a day or session. a discussion about the meaning and relevance of the words can be initiated. as described below. For example: Intensive – Headache – Exhausted Make sure to write down on a flipchart what participants say and it can be useful to make a short summary of what the general group feeling is at the end. Participants are invited to think for a few minutes about the day or session and then to share their opinion with the group in three words: a positive word – a negative word – a concluding word. Towards the end.net . the closer you move to the middlepoint.

scissors. It offers participants the possibility to express. Another more process-oriented method is called “Boats on the Sea”. Images help participants to think about how they feel. Pictures of little figures or puppets are shown in different positions in a tree. Participants are asked to work individually on giving creative expression. It can be used on the last day of the programme. Participants are divided into small groups (5 to 7 participants). especially if the activity is taking place in a foreign language. When using this kind of method. Participants think about how they feel and decide which of the puppets in the tree best symbolises their mood or state of mind at that given moment. Each wagon represents a different element of the programme that participants should reflect on. boats. using the material provided. Participants are invited to use their creativity by evaluating through the use metaphoric symbols: the sea. The “River” is a method which stimulates participants to use their creativity when evaluating.net . Many such methods exist. Each group receives paper (different colours). islands. it is useful to create a nice and relaxing atmosphere. T-Kit on Educational Evaluation in Youth Work www. This can help to make people feel comfortable and to evaluate more effectively. in a creative way. to their personal development during the programme and to place the results of their creative efforts onto whichever part of the river they consider appropriate. Participants are asked to write down ideas related to each of these elements for follow-up to the activity in the appropriate wagon. how they see their development during the programme. They do so. etc. A useful metaphor for looking towards the future is that of the “Train with different wagons”. other ships. Another visual way of helping participants to find out about how they feel at a given moment is called “Puppets in a Tree”. Many people feel much more comfortable when they can express themselves without using words. not so much by using words. glue.In these three methods it is important for the facilitator to make notes of what has been said so that the outcomes of the evaluation do not get lost and can be used by the group and the team. pencils and markers.training-youth.  Visual group evaluation methods Visual group evaluation methods give a clear overview about how other participants value a certain part of the activity. sea animals. for mid-term evaluation and for daily evaluation. The River The river is a very versatile evaluation method. When everybody has finished participants are asked to explain their creations. This activity can help participants to take stock of what has been done and what still lies ahead of them in the activity. but by taking advantage of the potential that non-verbal communication can offer. And of course you can develop and adapt all these methods using the idea of a trip or a journey to represent a process. The beginning and the end of the river should be marked on the paper. Each group is provided with the picture of a River drawn on a large piece of paper. This can be done using some background music. The metaphor of a river can help people to reflect on their experience during the activity. It is a method that focuses on the process of an activity rather than on its contents or other aspects commonly evaluated.

net .. This process oriented evaluation method is useful for looking towards the future and can be used in the evaluation to think about possible follow-up to a project... publications…).. T-Kit on Educational Evaluation in Youth Work www. The sea between the two harbours stands for the period of time between two moments of the programme.. Train to the Future FUTURE TRAIN(ING) Documentation Question Question Question Question Question Thank you Project 1 2 3 4 5 6 COOPERATION ..0 Boats on the Sea This exercise can be done in small groups as well as individually. The different wagons allow a certain grouping of the co-operation possibilities (i. islands or rocks or other symbols can be drawn. It is a group visual method that can be used in large groups. the boat of “my learning” or of “the group process” or of “my participation in the programme”. The results can be then shared on a big poster so that the whole group can visualise and decide upon the future co-operation after a certain project..). the passengers. e. common projects.e.. But.. other symbols (i.. This method can be used in many different ways.. the power…) can be used to discuss and decide about the different roles and responsibilities in the future co-operation. follow-up activities.. can be personal (by including this drawing of the metaphor in a questionnaire) or in small groups (using a flipchart).. Within the metaphor of the train.. in future? . You can give participants different kinds of boats representing different elements you want to evaluate (for example. Within the sea.. the starting point for working on it. the machinist. etc. networking. Ask participants to design their own boats and to put these boats somewhere in between the two harbours... Participants are free to add anything they feel helps them to clarify the position of their boat in this metaphor. this could be the beginning and the end of the project.. For example... Prepare a large drawing of a sea with two harbours (one on the top and one on the bottom) in advance. communication... but any other programme part could also be chosen.....training-youth.

To do this participants are asked to choose one of the figures (or puppets) on the picture that they consider most representative of their present mood or state of mind in relation to the aspect being evaluated. it is important to take notes of the comments and discussions. On a practical note. When everybody has chosen is their puppet. In all these visual group methods of evaluation.net . All members of the group are invited to express their current feelings and / or their satisfaction with the programme or on other aspects such as their learning. understand and conceptualise it.Puppets in a Tree 1 Puppets in a Tree This method can be used for the end of day review as well as for a final review in smaller or bigger groups. The advantage of this method is that you can easily adapt it to different situations or needs. The written notes are a necessary complement to the visual information: they help to explain. participants explain to each other the reasons of their choice. Altogether the visual information and the notes can be organised and used by the team and the whole group to draw conclusions and identify possible changes in the programme. T-Kit on Educational Evaluation in Youth Work www. the group dynamic and so on.training-youth. they can be used in the evaluation report.

 Snap-shots Snapshots are useful when you do not have a lot of time left but you still want to get a picture of the general feeling in the group. under the umbrella on a rainy day. for example. The combination of several of these methods can be used to carry out a lively mid-term evaluation half way through the programme. They are certainly not as thorough as other some of the methods already described. Here you can also replace the speedometer with animals or means of transport going at different speed to diversify the metaphor. With a little creativity you can easily adapt it to create other metaphors to fulfil the same purpose. the atmosphere in the group is evaluated. Warm Friendly Atmosphere in the group Businesslike Cool Unsupportive In this example. These methods give you an impression: a snap-shot. to see how things have developed. But. And. but they can be helpful for facilitators and participants to see how the others feel. for example the informal moments. again you can be creative and make your own version using other sounds instead of applause. The Thermometer A simple method for “taking the temperature in the group” is the “Thermometer”. A quick and easy way to test how people feel about different elements of the programme is to use “Applause”. The flip chart depicting the group evaluation can be posted on the wall and at a later stage of the activity can be used again. a sunny beach.net . There are some short visual methods that can be used for this.training-youth. In the same logic. etc. other aspects can be evaluated with the thermometer. A simple method for checking the atmosphere in the group is to use the “Thermometer” for “taking the temperature” of the group. group life. It can also be used as an energizer to warm people up to evaluating. you can ask participants to evaluate how they feel about the pace of the programme by using the “Speedometer”. Participants can also be asked to share their evaluations and to discuss how things can be improved. The thermometer can be drawn on a flip chart and participants are invited to put a personalised mark (their name or initials) on the temperature they feel best represents how they view a particular part of the programme or how they feel the group is. This metaphor allows you to how people are in the group. T-Kit on Educational Evaluation in Youth Work www. freezing in the snow.

it can be used as a warm up activity before moving to the business of more serious evaluation activities.net . The facilitator reads out different elements of the programme and according to their satisfaction with that element participants clap their hands. Ask participants to form a circle. The Speedometer How do participants feel about the pace of the programme? The “Speedometer” offers participants to express their feeling about the rhythm of the activity and can be used as an ongoing (daily) evaluation method. Applause This is a very simple method to evaluate different elements of the activity at the end of or during the programme. whether they liked or disliked it). for example.training-youth. the louder and longer the applause should be. OK slow fast too slow SPEED too fast Further instructions needed to understand how this can be used. The higher the level of satisfaction. let all the participants take a look at the other participants’ opinions and discuss as appropriate. Draw the speedometer on a flip chart and ask participants to put their personal mark according to their opinion about the pace of the programme. As this activity is only intended to measure the satisfaction of participants with a particular programme element (in other words. T-Kit on Educational Evaluation in Youth Work www. Remember to mention what the team will do with the information.

these feelings and impressions expressed are usually very revealing and can help us to understand the outcomes of other evaluation methods.training-youth. it is important to note down the main ideas expressed in the last round. This is sometimes an emotional moment before the programme really ends.net . Just remember that this can easily end up in a long and tiring session. The one who has the object in their hand speaks and when finished hands it over to another person in the group who is then invited to express themselves. The ideas expressed in the last round are. and to send it around the circle of participants. a spiders’ web of links emerges between the members of the group. to one sentence. preferably one that somehow represents the project. one by one. the possibility to speak and to say anything they feel is appropriate. A variation on this is to use a ball of string. not very well developed or may seem not to be very relevant. Last round This is as simple as giving every participant. participants are asked to burn a single match and to speak only for the time that the match is burning. For this reason. As participants express themselves and pass the ball from person to person. Its dynamism comes from tension and speed. T-Kit on Educational Evaluation in Youth Work www. When it burns out you have to end your point immediately! Another way to deal with the last round is to take an object. sometimes. But. To avoid this and to help participants to focus on the most essential. for example. Last round The last round in an activity offers participants the possibility to express the things that they consider as the most important elements or those issues “which have not yet been mentioned” before the end and before everyone leaves. One by one. You need one or two boxes of matches (one match per person is enough) and a plate or a bucket. A very dynamic variation on the last round is known as “The Matchbox”. you can limit each intervention.

4 Feedback Feedback can be part of interpersonal and group evaluation. In other words. It does not necessarily provide them with constructive information that they can act on for improved relations. 65 Source: Merrian Webster Online Dictionary. feedback to somebody must be such that the person: • understands the information • is able to accept the information • is able to do something with the information.”65 One very important element should be emphasised: FEEDBACK SHOULD BE HELPFUL TO THE PERSON RECEIVING IT To be helpful. Feedback can be used in one to one situations as well in small groups/teams. This serves only your need to tell them how you feel. Feedback is somewhat more complicated than one might imagine. When working in the context of human relations. if you are fed up with somebody and finally come to the point telling them. Some things that we consider to be feedback serve only the needs of the person giving it and not the needs of the person receiving it. To avoid misunderstandings and problems in the group.training-youth. it is important to be sensitive about the group dynamic when using feedback.m-w. or process. but this is does not constitute giving feedback. Below you will find some suggestions for how to give and receive feedback. you might be relieved in doing so. When using feedback in a group it is helpful to begin by reminding the group briefly of the meaning of feedback and to set a time frame so that everybody has the chance to both give and receive feedback. Take your time. feedback should be guided by the principle of mutual respect for each other as learning subjects. Feedback is “the transmission of evaluative or corrective information to the original source about an action. Make sure you have enough space (in terms of the working room and time frame). Think about an adequate environment in which to do feedback. In some of the methods outlined previously the evaluation process implies giving feedback to somebody. These simple things can provide participants with the security they need in order to undertake honest feedback and serious evaluation. Think about the atmosphere in the group and the implications of undertaking feedback at any given moment. event.com T-Kit on Educational Evaluation in Youth Work www.net . http://www. 2.

responses can be given at the end T-Kit on Educational Evaluation in Youth Work www.training-youth. they should not be related as being those of others even if you feel they are representative • Begin your feedback with a positive review rather than with negative statements • When you give feedback to one particular person in the group address that person directly rather than to the group as a whole. This involves saying who the feedback is for • Do not judge other people. If necessary and agreed in advance of the feedback session by all involved.net . If speaking about your personal opinions and feelings. How to give and receive feedback? Giving feedback • Speak for yourself by using “I” instead of “You” or “We” when you give your feedback to the group. try to be constructive rather than destructive. Be supportive to those who propose ways to deal with difficulties and conflicts. You do not have to justify your behaviour or to explain why something is like it is • Try not to interrupt the person who is giving you or the group feedback. You have a chance to receive information about how others view your performance and to understand what they think about you • If you feel you are being criticised. what made the most important impression on you or what you suggest for someone to do in future Receiving feedback • Think of the feedback you receive in terms of constructive criticism. try to avoid immediately taking a defensive attitude. Rather you should describe your personal perception of a situation or your feelings • Give feedback in relation to the “here” and “now” situation rather than to past events • If something conflictual arises. It can help to remind oneself of the human potential for learning and for change • It is alright to praise somebody! Express what you liked most in somebody’s behaviour.

for example. It is useful not only for the content of the answers that the participants provide. This kind of exchange helps participants to give words to their experiences and to compare them to and learn from the experiences of the others. both in cooking and in evaluation. Questionnaires and interviews will provide you with information about the individual learning process of participants. theatre.net . for example) can help participants to give expression to issues arising as a result of the process they are part of in the activity and which may raise mixed emotions. 2. When choosing an evaluation method. drawing or body language you offer participants other ways of expressing their concerns and ideas. but also for the information provided by the way the group deals with the questions. This kind of method can help you to get an impression of the atmosphere in the group. non-verbal expression. one must be careful with this kind of statements. reflection groups can serve this purpose as well.training-youth. One method will fit better with one participant than with another. Others have problems in writing T-Kit on Educational Evaluation in Youth Work www. But. By using objects as symbols. the first question we have to ask ourselves is: what do we want to get out of the evaluation? What do we want to find out? Do we want to know more about the learning process of participants? Do we need information about how people feel in the group? Are we curious about how people feel about the methodology used in the programme? Or do we want first of all to give participants the possibility to reflect on their learning and to share that with the others? Methods like “Applause” or the “Living Dart-board” will not help you to find out more about the learning process of the individual participants. Methods which rely on the use of creativity (drawing. Verbalising these ideas afterwards for the sake of clarity and mutual understanding can then be used in the second step. Some people love to fill in questionnaires and will provide you with long answers to the questions. People are different and so are participants in educational projects. Of course. one justifiable conclusion would be that “having an opinion which is different from that of the others” is not (yet) a shared culture in the group.5 A critical look at evaluation methods Sorbet for breakfast?: When to use what method? Serving sorbet for breakfast or fish for dessert might not be the best idea in some countries. When during the “Living Dartboard” all participants demonstrate complete satisfaction with every item evaluated by running immedia tely to the middle point. When a reflection group meets regularly it can provide a safe environment where people can share their feelings and the challenges they consider have arisen. especially in an intercultural environment! This example should alert us to the idea that spending some time thinking about ‘what to serve when’ makes a lot of sense.

for example. Especially in visual methods like the “Thermometer” there is the risk that participants will adjust their opinion to what others have already noted down. but we will get an independent and professional evaluation conducted by an outsider. It is important to explain to participants before beginning the evaluation activity what its aim is. Questionnaires are filled in at a specific moment in the activity. But. the specific moment at which the questions are asked. Some members of our organisation might not consider this worthy of significant financial investment or they might be worried T-Kit on Educational Evaluation in Youth Work www. This does not mean that such evaluation methods are worthless. Participants might answer quite differently two weeks after they have gone home and had a chance to reflect with some distance. Why is that? Where might such resistance come from? • Evaluation needs resources (time. But even when participants have expressed satisfaction with an activity. therefore. Evaluation methods should also be thoroughly introduced. the atmosphere in the group and how others answer the questions. But. what the purpose is for those conducting the evaluation and especially what will be done with the results. Most people want to feel like they belong to the group and this can lead to not wanting to obviously differ in opinion from that of others. they can provide a rather positive impression of how participants value the activity. The answers to the evaluation questionnaire on the last day are influenced by the fact that people are soon leaving the activity. why you have chosen it. Resistance to evaluation Often when we undertake an evaluation we can count on the support and backing of a team or an organisation.net . it remains important to take critical look at the outcomes. human and financial resources) whether it is an external. Sometimes the level of satisfaction can be caused by the lack of challenging activities or a good group atmosphere. Often it is difficult for participants to answer questions about what they have learned on the last day of an activity. the outcomes of the evaluation should be carefully considered. whose role it is to be independent. but are excellent in expressing themselves in more creative ways. sometimes people do not like the idea of evaluating. It is important to offer a variety of methods that provide participants with different communication styles with equal opportunities to express their views. be aware of the fact that there can be resistance to evaluation. If we invite external evaluators to conduct our evaluation. we will most definitely have to spend more money on it. internal or (self-) evaluation. We should. The way in which the participants answer to the questions posed in different evaluation methods is influenced by many things including. Outcomes What do the outcomes of all these evaluation methods tell us? First of all.training-youth. It simply means that it is relevant to consider the variety of possible influences in the interpretation of their results. this kind of “group conformity” is also a matter of which stage of development the group is in. The important thing is to remain aware that the risk exists and to consider it when interpreting the outcomes of a given visual method of evaluation. It is also good to take a critical approach to the outcomes of evaluation questionnaires. Whether they are positive or negative. down their experiences and feelings. Participants may have mixed or confused feelings on the last day.

it is important to work out with the team or the organisation backing the activity which amount of resources are we willing (and able) to use for the evaluation in advance. they might feel negatively towards the evaluation if the evaluation may be critical towards the quality of their work. Maybe funding will not be repeated next year? Maybe the evaluation will recommend a review of the competence of those working on the programme? Some trainers might perceive evaluation as interference in the educational process or as waste of time and resources. • In the social and civic sectors.training-youth. as is remembering that educational evaluation should always be done “in service” of the project. educators and trainers) might feel threatened by evaluation because it may include judgements on the quality of their programmes and products.net . sometimes critical evaluation results can have the effect that the continuity of a programme is put into question.  T-Kit on Educational Evaluation in Youth Work www. Therefore.that the resources they need for their work will be used to finance the evaluation. In other words. Transparency and professionalism are very important in overcoming such resistance to evaluation. people with different roles and responsibilities (policy makers. For example. Decision makers might feel their actual space for decision making is limited.

but that in the reporting only the statistics are presented. For example. for example.0 2. It is equal to the square root of the arithmetic mean of the squares of the deviations from the arithmetic mean. This may be because statistical tables look attractive and they offer a veneer of objectivity or credibility.6 Collecting and organising the information The information collected during an evaluation can be both qualitative and quantitative in nature. You can find some tips for the presentation of that information in “Reporting the outcomes” (Pages 82-84). T-Kit on Educational Evaluation in Youth Work www. If. excel tables. we have a quantitative data that says that 25% of participants found the contribution of the group to their learning excellent. But. Quantitative information can be analysed and presented in the form of statistics. But. It is considered “professional” when an evaluation report demonstrates that the evaluators were able to manage statistics and to calculate and interpret averages and standard deviations66. The “accidental marks” in a group of 300 or of 3000 have a much smaller influence on the average evaluation score and the statistical results are.answers. This collection and organisation of the information still takes place at an early stage of the evaluation process. Nevertheless.e. Qualitative information is normally processed and summarised focusing on the most important and recurrent facts. much more reliable. On the other hand.training-youth. both quantitative and qualitative information is asked of participants. 50% ok and 25 % bad. of the evaluation report. therefore. we still do not know about the reasons for each score being given: was it because of too many strong personalities? Or because of sub-groups among participants? Or because it was their first international activity? Or was it because of an overly “professional” atmosphere without inter-personal exchange? It is quite common that in evaluation questionnaires. software tools which can help us to analyse and process quantitative data (i. even from a purely statistical point of view the amount of data that we usually manage to collect in educational activities is far too small to be of a high statistical significance. it is good to have in mind already at this stage that some quantitative and qualitative data might later be part of “reporting the outcomes” and. charts and graphs.com/topic/standard-deviation. But. this one very bad mark can be an “accident”. . in a group of 30 participants a very bad mark given by just one participant can cause a significant decrease in the average score. having only this data. It is not yet the moment of arriving to conclusions or of reporting the outcomes of the educational evaluation. Regarding quantitative data. we would like to encourage the use of statistics in the presentation of evaluation outcomes. Source: www. caused by reasons which have nothing to do with the educational process. therefore. For this. we would like to stress that quantitative data has limits.net 66 Standard deviation is a statistic used as a measure of the dispersion or variation in a distribution. Quantitative information (without reference to some qualitative explanations) is far too limited to explain the educational process. access applications or any other specific statistical software) are useful and (self ) trainings exist in which one can become proficient in basic statistical applications.

1 2. There are several criteria that we could use to assess the fulfilment of this objective. For example. there is the risk of getting caught up in the definition of criteria. On the one hand. youth work experience • the contribution of “externals” and other resource persons A quantitative indicator for the criteria “diversity within the group” could be the number of different nationalities represented in the activity (i. which will allow us to compare reality with the objective or the expected outcomes of the educational activity. living in a rural or an urban environment or coming from different regions within a country among others.training-youth. because it can be difficult to embrace the complexity of educational evaluation. In order to be able to evaluate the objective (to experience the diversity of youth work). T-Kit on Educational Evaluation in Youth Work www. there is the risk of reducing the whole evaluation to a normative imperative (it should be so!). The objective of the project is “to experience the diversity of youth work”. even if those criteria and indicators cannot embrace the entire complexity of the objective being evaluated. Establishing criteria is one of the most difficult and often conflictual steps of the evaluation process. as follows: • the spaces provided in the programme for exchanging experience among participants • the diversity within the group: in terms of nationality. if we want to evaluate the objective of a given project. we should decide upon the most relevant criteria and indicators.net . diversity cannot be reduced just to difference in nationality.e. If the project deals with environmental issues maybe the “living in the city / living in rural areas” indicator is as relevant as the nationality of participants for evaluating the objective of “experiencing the diversity of youth work”. But. There are other factors that make up diversity. On the other hand. including age. cultural background.7 Establishing criteria As we have seen before criteria are elements. gender. However. the nationality distribution of participants). We need to select the most significant ones. the number of criteria and indicators cannot be unlimited.

not the significance of the outcomes of the evaluation. 2. we can decide on the basis of our evaluation to continue to work with the same youth hostel for the next youth exchange we organise. This last stage of the evaluation process can be the cherry on top of the delicious cake. so that they can be well used by us and others in the future. this does not mean that we should tell different stories to different people. covering many different aspects.8 Reporting the outcomes Reporting the outcomes of the evaluation is very important. Reporting the outcomes does not only consist of summarising the findings and putting them in a report for the funding authorities. Nevertheless. For example. The evaluation report The evaluation report is not the only way of reporting the outcomes: oral reports. Good use of the outcomes of our evaluation can be useful at a very practical level. findings and conclusions with the different actors and the first step to involving them in the follow-up and future projects. it will have to have a different emphasis and probably also style than if it is going to be read by the secretary of the Mayor of our town or by other youth workers. Therefore. our evaluation could lead us to conclude that we should use a certain methodology again but the next time it should be adapted in this or that way. photos or other informal ways can also be excellent channels. The same is true for the educational level. the evaluation report is usually a key tool for reporting the outcomes of the evaluation. structure. articles. If the report will be read by experts in Brussels. Reporting requires a certain amount of planning. It can be used to improve the quality of your next projects. both present and future. the order of presentation and the emphasis may shift. Thinking about the evaluation report when planning educational evaluation may prove to be very useful.training-youth. as we put into the evaluation process itself. This makes it a very useful learning tool for participants and organisers. it is essential to give the same amount of attention to the processing and reporting of the outcomes. It is a tool for sharing information. In the report. T-Kit on Educational Evaluation in Youth Work www. by focusing on the strengths and by trying to tackle the weaknesses revealed by the evaluation.net . By doing so: • We can decide which information we need to analyse • We can decide which methods are likely to be most effective for getting that information • We can foresee which stakeholders would we like to target • We can think in advance what we would like to emphasise to whom The emphasis of reporting shifts according to who it is intended for. Nevertheless. focus and adaptation according to the target groups that we want to reach. coming back to our cookery metaphor. For example. from different perspectives and with a certain distance. It can also help others who may want to undertake and implement similar projects. The evaluation report offers quite a comprehensive vision of the project.

it is best to (a) describe the purpose for presenting the table. the report should be clear and comprehensible. and figures. Introduction We should give enough information to an outside reader about the process and what we are evaluating. • Another important potential audience to keep in mind when writing the evaluation report is those who may in the future benefit from a similar project. As a general rule. This is especially true if we have used questionnaires or interviews. nevertheless. This is a wide group of stakeholders and can include actors ranging from the funding authorities to volunteers in the organisation. we contribute to the sense of collective ownership of the project. charts.General considerations • In principle there are three specific types of reader that will use our report: the participants of the activity. if any. the stakeholders in the activity and outsiders who may potentially benefit from the findings presented in our evaluation. rather than just numbers. By respecting transparency. should be integrated into the text of the report. All participants of the project should receive a copy of the evaluation report once it is ready. we should make sure to support that interpretation with direct quotes from interviews or with relevant and comprehensible statistics. • Tables. (b) present it. If we present an interpretation. we should present our findings. We can. cover both the negative and positive aspects of the evaluation outcomes. This method can be quite problematic for the reader. appearing near that portion of the text discussing them. Therefore. Participants might not find their own words in the report. • Transparency is a very important aspect of reporting evaluation. we should present them in a very clear and comprehensible way for the readers.net . and our interpretations should be integrated into a logical whole. It can be very frustrating for the reader to go through a list of seemingly unrelated analyses and findings with a promise T-Kit on Educational Evaluation in Youth Work www. identify some common parts which should be covered by any well presented and structured evaluation report. of course. The report should. It is a good idea and readerfriendlier to use simple visual representations of statistical data such as pie charts and colour coded graphs. numbers and more numbers. The presentation of findings (statistics from questionnaire.training-youth. qualitative analysis of interviews).  Some key parts of the evaluation report Every evaluation report is and should be different. The introduction should include: • All the relevant background information about the project • Who conducted the evaluation and with which competencies • The criteria and the perspective of the evaluation • The description of the design and execution of the evaluation Presentation of findings After this important introduction. • If we use quantitative data and statistics in the report. Sometimes people describe their analyses in the body of the report and place all the tables in an appendix at the end. and (c) review and interpret it. • All aspects of the evaluation should be described in sufficient detail to permit an outsider to reach more or less the same conclusions as someone who attended the activity. but they should feel that it is representative of their experience.

support measures for participants. the inclusion of some personal reflections (i. Summary Having provided the general overview and the results. Some evaluation reports also include the raw data from the evaluation conducted in the appendices. The focus of each evaluation is different and. we should always make the distinction between personal reflections and results clear at all times. and then indicate where that result leads next. present the data relevant to it. At times.net . We should review the particular strengths and weaknesses of our own project as demonstrated by the evaluation outcomes and suggest ways in which those shortcomings might be avoided in future projects.training-youth. Any comments and suggestions offered should be supported by findings. Every step in the analysis should make sense to the reader. However. the direction and degree of change of the participants in the evaluation • the extent to which the objectives set in the beginning were met • some ideas about where future projects might be directed However. It is also a good idea to append any bibliographic references consulted in the preparation of the evaluation report. of duration. of educational approach). We should avoid reviewing every specific point in the evaluation. Suggestions for follow-up (i. it is difficult to indicate what information would be the most significant in an evaluation report. systematization of results in a publication…) are also important. further projects. The conclusions concerning the “format” of the project that may eventually lead to changes are very relevant (for example. This section is not an appropriate space for making declarations.e. interpret the results. therefore. ending with the conclusion that “more projects are needed” is of little value unless we can offer some concrete suggestions about the nature of future projects and unless we can provide arguments for why they are needed. Conclusions The report can conclude with a statement of: • what we have discovered about the educational progress. it is essential to summarise the most important points. Appendices The outline of the full programme outline and the list of participants can be among the most common appendices to evaluation reports.e. it is relevant to include some photos or other materials which can transmit the “taste” of the activity to the readers. T-Kit on Educational Evaluation in Youth Work www. but we should review all of the significant ones. quotations of participants) can help to illustrate more clearly some ideas. change of target group. We should present our rationale for a particular analysis. pointing once more to their general significance. that all the loose ends will be tied together later in the report. In addition to these.

A lot of projects.net . The basic idea of the project was to approach the participating youngsters from the perspective of their capacities not from the perspective of their problems. It was the first time that Member States of the European Union and the European Institutions came together to take joint action against Racism. we have tried to cover different types of activities (from a youth exchange to an e-learning project). Germany.int/comm/employment_social/fundamental_rights/public/arcreyar_en.eu. Instead of presenting “examples of good practise”.9 Evaluation practices In this section we will present some “practices” of educational evaluation or projects in which evaluation played a relevant and specific role. This does not mean that the youngsters did not originate from “problem situations” but these situations were not taken as a starting point. the purpose of this section is to share experiences and reflections. Following the inviting and non-normative approach of this T-Kit we have tried to avoid calling this section “good” practices. There are certainly many others at least as interesting as those presented here and many aspects relevant to understanding educational evaluation will not be covered by them. successes and frustrations. These do not correspond to a systematic mapping of youth work projects.htm T-Kit on Educational Evaluation in Youth Work www. potentialities and limitations coming from practices on educational evaluation. different approaches (from a systematic evaluation to permanent and ongoing adaptation). different target groups (from youngsters to experienced trainers) and different methods (from questionnaires to non verbal methods). Finland. We hope that these practices can contribute to inspire the development of your own. Take5 was a long term project that took up the challenging idea of combining youth work and theatre and worked with a stable group of participants over a period of 6 months in total.training-youth. “Take5” – 40 youngsters evaluating The “Take5” project Supported by the YOUTH programme and the ‘European Year against Racism’67 the Take5 project took place in 1997. More information at http://europa. 67 1997 was the European Year against Racism. The five people involved decided to co-operate on setting up a music and theatre project centred on the theme of social inclusion. Sweden and the Netherlands). seminars and campaigns took place at national and at European level. The project involved 40 disadvantaged youngsters from five countries (Poland. The idea for the project came into being during an international training course youth workers. 2. Nevertheless.

Legion Theatre (Finland). The development of evaluation through the project • In the international encounters. e. In the first phase. c. Kinder und Jugendzentrum Freiberg (Germany). Germany eight days were spent on working with the youngsters on intercultural communication and theatre skills. Since the project was extremely challenging appropriate time was needed for the youngsters to reflect. After Freiberg. These groups offered the possibility to participants to express some of their more personal concerns more openly and in a relaxed atmosphere. Youth Club Europe (Poland). The project took place in five phases: a. Despite. In addition to the five original team members. as the project went on the language capacities of the participants improved a lot. in which the youngsters tell their stories with great conviction. the director visited all groups and spent two days working with each of them. In this phase. elements from the different scenes were used to put together a play. In the last phase of the project the group toured the five participating countries performing a theatre show made out of their experiences and directed by a professional theatre director. national groups came together every day to reflect (reflection groups) from the very beginning. this was a challenging experience that brought them into many new demanding situations. The starting point was formed by the youngsters’ abilities and the challenges offered by an international theatre project. In short one may say that ‘broadening the options’ was the essential objective for the project. were presented to the others. Although. T-Kit on Educational Evaluation in Youth Work www. the group was accompanied by five co-workers and a German theatre director. The team recognised a need for participants to be able to reflect in their own language. That is the reason why evaluation played a very significant role. all scenes developed in the various countries. the young people continuously ended up in situations which were new to them and which demanded new solutions. In Freiberg. the national reflection groups went on meeting. For most of the young people participating in this project. the participants returned to their own countries to work on enhancement of their theatre skills and they made up scenes of which some were incorporated in the performance. The participating organisations were: Verdandi Södertälje (Sweden). groups worked within their local environment with their youth leaders on preparing the first international encounter which took place in Germany. d. b. This forced them to look for other ways of solving things. The first ideas for the final performance were shaped here. Subsequently. but perhaps also thanks to the tension and time-pressure. The group worked in the theatre under great tension and time pressure. Tandem Nijmegen (the Netherlands) and Stichting Spectrum (the Netherlands). The starting point was to make a performance centred on the personal experiences of the young people with regard to social exclusion. In this phase. a magnificent play by the name of “The Raft” was created and performed on the opening night. Subsequently. Five months later all participants met again in Zielona Gora in Poland. to look back and to digest on what happened.net .training-youth. often under a lot of pressure. In the course of the project they were constantly confronted with problems and challenges which could not be dealt with by means of their “traditional solutions”. since it was not easy for many of them to express themselves in English (the working language of the international parts of the project).

net . The capacity to reflect and to give words to their experiences and feelings turned out to be one of the main competences that the youngsters developed in the “Take5” project. But then. Participants answered the questions in their own language. participants felt responsible for its success and acted accordingly. A very nice happening. there was one very significant outcome of the evaluation meeting. to reflect on changes they feel they experienced and to point out the impact of this project on their future plans. the youngsters proposed a daily review with the whole group. This experience demonstrates that consistent planning and openness for adaptation and further development are both very important when working with evaluation within a project. The team followed it closely and used its outcomes for the planning and adaptation of the programme. frustrations. The brochure written about the project was finished as planned in the original project plan and published during that evaluation meeting with all participants present. T-Kit on Educational Evaluation in Youth Work www. The diary was not used by all youngsters but a quite a few wrote in it regularly. as the questionnaires had to be translated into English in order for the whole team to be able to be involved in the assessment of the forms. Participants met in the Netherlands for five days to evaluate approximately 6 months later. problems were taken up and plans for improvement were exchanged. it was decided that it was worthwhile to put efforts into organising an evaluation meeting with all the participants to see what the impact of the project would be.  A follow-up evaluation Although on the last day of the project in Finland some of the youngsters expressed their need to meet again after some time to see “how everybody is doing”. Although a lot of talks took place. In the beginning the group just sat together in a circle and discussed the day. Nevertheless. It became a space for personal and interpersonal evaluation and for sharing experiences. learning achievements and complaints. It was very obvious that participants profited a lot from all the evaluation moments during the project.• After a conflict in the group on the 4th day of the first international meeting in Germany. It helped them to relax and to reflect on the day on a personal level before going into the group discussion. This involved a lot of work for the team members. a follow-up evaluation was never planned. In the beginning most of the youngsters were not that enthusiastic about these methods. but the long term outcomes were not described. The diary was always somewhere around and participants had the opportunity to write down their impressions. Since it was their own idea to start this daily reflection. One of the team members was experienced in relaxation and meditation exercises. both in groups as on individual level. All of them read the diary. Although in the beginning the team was quite sceptical. but equally that it has limited value when it comes to the real evaluation of a project. the meeting lacked a planned evaluation and therefore missed quite some opportunities. • During the last three weeks of the project in Poland and throughout the theatre tour a common diary was introduced. but that changed during the process and at a certain moment participants asked her to work with these exercises at the beginning of the daily review. conclusions. The majority of these youngsters were now quite capable to describe the meaning of the project for them. • After the first international meeting and at the end of the whole project participants were asked to fill in evaluation questionnaires. The evaluation was developed and adapted according to the challenges that arose in the project and to the needs of participants.training-youth. impressions. when the team had their evaluation one month after the project. it developed into one of the most important elements of the project. It showed that an unplanned evaluation can be a wonderful event.

 EVS68 and Leonardo69 . social life. informal groups. cultural association which was set up in the autumn of 1994. alternative tourism. autonomy and independence. social and cultural services. separate the different aspects and avoid overly general judgements. while the Leonardo participants are employed in different organisations and companies all over Padua.) in an effort to increase mobility. Often the personal situation of participants strongly affects the perception of other aspects of their experiences of the mobility project. “Youth for Europe”. Evaluation meetings can be used to make distinctions.eu. businesses. XENA has been involved in EVS since 1997 and hosts volunteers as well as sending them abroad. Amongst many other things XENA hosts young people in the framework of the “European Voluntary Service” (EVS) and the “Leonardo da Vinci” programmes every year. It promotes projects mainly within the framework of European Programmes. universities. Each year. 68 See footnote 28 – Page 37 for further information about EVS 69 Leonardo da Vinci was a vocational training programme of the European Commission.html 70 XENA is a non-profit. XENA works in co-operation with other bodies (European institutions. etc. Multidimensional evaluation with participants A long term mobility project can be a learning experience in many different ways. Its name derives from ancient Greek and means foreign things. It includes language learning. in another country working as volunteers. municipalities.training-youth. cultural association situated in Padova in Italy. meeting new people and organisational aspects.int/comm/education/programmes/leonardo/leonardo_en. The aim is to use trans-national co-operation to enhance quality. up to one year. vocational schools. Leonardo da Vinci is the European Union action programme for implementing the European Union’s vocational training policy. XENA is an active member of the Eurodesk network. often new and confusing experience. to foster innovation and to improve the quality of training. Both sets of participants are offered the opportunity to take part in an intensive Italian language course in the first month of their stay in Padua. etc.net . XENA’s key areas of work are environmental protection. XENA has almost daily contact with the EVS volunteers.e. It aims at helping people improve their skills throughout their lives. enterprises. XENA works with the primary aim to promote the development of innovative employment sectors taking into account the impact employment creation can have on the environment and society.xena. managing a local information centre. They stay for three months. Its main objective is to increase and improve the level of contact and interaction amongst different cultures.it T-Kit on Educational Evaluation in Youth Work www. associations. “European Voluntary Service”) and others. The programme promoted trans-national projects based on co-operation between the various players in vocational training (training bodies.Multidimensional evaluation EVS and Leonardo in Padova XENA70 is a non-profit. not for profit activities and unemployment. More information is available at http://europa. More information at http://www. For those who organise a mobility project these aspects are clearly distinguished but for participants this is one big.) in Italy and abroad. They are a way to reflect on how participants can change things or deciding together if and what the organisation can do to improve the situation. chambers of commerce. promote innovation and support the European dimension of vocational training systems and practices. European Voluntary Service is part of the YOUTH programme of the European Union and offers young people the opportunity to stay for a longer period of time. The EVS participants work within XENA. animation. “Youth” (i. such as “Leonardo da Vinci”. The Leonardo da Vinci is a pillar of the Lifelong Learning programme of the European Commission. schools. professional experience. Around 100 participants take part in Leonardo projects in Padua per year. intercultural learning. trade unions. Within the Leonardo da Vinci programme. six EVS volunteers are hosted in Padua for six months.

felt that the organisation was not offering much professional support. T-Kit on Educational Evaluation in Youth Work www. At this point. During the first period of working with long term mobility projects XENA realised that some participants felt they did not receive enough support from XENA. • During their whole stay. the language capacities of the participants in Italian are still limited. Page 51. not to use a certain accommodation provider any longer. if needed. In those questionnaires the focus is on the hosting organisation. work placement) and can lead to the decision to make some changes in the organisation of the project. The language course is of course an important topic. • The Leonardo participants are regularly asked to fill in evaluation questionnaires. The mechanisms for such a multidimensional evaluation with participants are: • In the first month there is an evaluation meeting every week. and the living conditions. or that placement is possible. Some participants saw their mentor more as a friend than as a tutor and as a result. although maintaining an informal attitude. although a lot of time and energy was spent on organising the support already offered. These measures helped participants to feel more secure and to get more out of their time in Padua. This is a possibility to express yourself outside the group and anonymously. In the beginning most of the questions ask for quantitative information and scales for scoring are provided. life in Italy.There is a lot to evaluate and many people have to be involved: the language course. Council of Europe and European Commission (November 2000). written questionnaires and a clarification of the role of the mentor were foreseen in subsequent editions of the hosting project. more “formal” evaluation meetings. is used to collect the expectations of the participants. 71 Described as “Where do you stand?” in “T-Kit No.training-youth. the “Puppets in the tree” (Pages 68. Many different exercises including the “Dartboard”. the place the volunteers live. This feedback was taken seriously and. Often these are not officially scheduled meetings because the participants meet their mentor almost daily and unplanned (evaluation) conversations take place. The questionnaires focus on the language course. their tasks. The weekly evaluation meeting is a group evaluation. the working environment. for example. participants have the possibility to put ideas or complaints into a suggestion box which is installed in a central spot.  Evaluation as feedback for the organisation For XENA evaluation is an important instrument for receiving feedback from participants and for improving the way projects are organised. the support offered by XENA.net . but only if certain conditions are met. to place trainees in that company again. 71) and different forms of statement exercises71 are done to evaluate the different elements. In the first meeting the “Clothes Line” (Page 66). so scoring is easier for them. • The EVS’ers have their own questionnaires focussing on their specific situation. Considering the result of evaluations. the conditions where people have been accommodated and the dimension of “getting to know Italian culture” also need to be discussed. the working place and the living conditions. XENA became aware that an overly informal approach to this support can have negative effects. the group life. This feedback is often about a specific aspect of the experience of participants (accommodation. as compared to the job-description they received. 4 – Intercultural Learning”. In other cases the feedback can be about the way the project is organised. Later more open-ended questions are added to the questionnaires. the mentor. • The EVS participants have a mentor with whom they have regular consultation meetings. But.

Iceland and Outward Bound Slovakia in 2002 to 2003. In addition to resources provided by the organisers and fees paid by participants. economical/financial issues/sustainability. but the effect was that the participants did not know to whom they had to refer in case of a problem.net 72 A Long Term Training Course consists on several residential seminars with project or practice phases in between. contents developed. periodically checked with the manager of the association and formally evaluated after 6 and 12 months • The manager is constantly monitoring and coaching the staff. T-Kit on Educational Evaluation in Youth Work www. individual well being and satisfaction of the people collaborating. Hitt Husid. Once again. the project was financially supported by the YOUTH Programme of the European Union.training-youth. Its total duration is between one and two years.int/youth . Participants need to refer to one or two people whose tasks and responsibilities are clearly explained to them. It is as well an occasion to learn from each other thanks to the reports of those involved in external training. the staff of XENA had to consider not only what they were doing but also how it was perceived: many people were working hard to guarantee a good result. evaluate the team work and co-ordinate for the tasks of the following days. • Internal evaluations are organised within smaller groups of staff after the realisation of specific activities. management. XENA developed a system for evaluation: • Long term plans concerning the general aims of the project are developed in co-operation with the whole staff (10-15 people) • Each co-ordinator in charge of a department elaborates a plan on a yearly basis. for participants it is confusing and problematic to have to communicate with several contact persons during the organisation of their time abroad and during their stay. This training course format was for the first time developed by the Directorate of Youth and Sport of the Council of Europe. Kitokie Projektai. In this meeting the staff can share information. follow-up. Lithuania. As a result. interpersonal dynamics. using an informal approach • The staff meets at the beginning of each week. logistics. As you can see. fulfilment of objectives. For more information please consult the following website: www. XENA takes evaluation seriously! Madzinga – External evaluation The Madzinga project Madzinga was a Long Term Training Course72 organised as a cooperative project by Outward Bound Belgium.coe. Many different aspects of the project are evaluated in the different analyses and phases: the impact on groups hosted.0 XENA also realised that although many people are needed for the successful organisation of a long term mobility project. the decision was made that only a few people work directly with the participants. the impact on society. outcomes. the European Youth Foundation of the Council of Europe and the Soros Foundation.

a story that tries to show what people learn in less formal learning situations such as this training course and makes an attempt at measuring those learning outcomes. Twenty four participants from thirteen countries attended the seminar. the external evaluator can be much more precise in collecting data and describing developments in the project. how many eggs you use. one specific approach to evaluation is to invite an external evaluator to conduct your evaluation. The people who developed this training course had worked together in previous years and wanted to develop a new training concept with this long term course. It might be quite disturbing.Madzinga focused on the intercultural learning dimension through outdoor experiential learning and comprised three phases. But. This person is not only watching you. throughout the training and during all the team meetings. they are taking note of everything you do: How much sugar you put in. Belgium. Another benefit is that the only task of the external evaluator is to observe and report on the activity. can dedicate all their energy to the job. Later. In the second phase participants developed projects at home and stayed in contact through so called coaching groups. being present from the first preparation meeting. The third and last phase took place in Samukas. the external evaluator is more capable of asking the “right” critical questions about the project.pdf T-Kit on Educational Evaluation in Youth Work www. but who follows the process as an observer. that person makes a report. The “outsider” is in the position to question. This is someone who is not involved in the project as such. records what is happening and reports about it. in which you can find a description of your activities in the kitchen along with some critical remarks about you as a bake.training-youth. Lithuania. A story about non-formal learning. risks that we take things for granted and simply do not take (or have) the time to take a step back to look at what is happening. As we already described. and therefore. That was one of the reasons why they wanted to have an external evaluator / observer who followed the whole project.outwardbound.be/madzinga/MADZINGA. The first phase was a residential seminar in Lustin. 73 The Madzinga report can be downloaded at: www. being the external evaluator is a challenging and difficult job that requires specific competencies. Especially when it comes to difficult aspects of the evaluation. where experiential training methods were explored and the concepts linked to them were deepened. Not being involved in the project as a team member or as a participant has the important advantage that the external evaluator can look at the project more objectively or at least from the perspective of an outsider. Being “an outsider”. The result of the external evaluation in Madzinga: the report73 The result of the external evaluation in the Madzinga project is a 160 page publication that reads like a story. such as measuring the learning outcomes of a project for participants. 1 External evaluation: somebody is watching you! Imagine you are baking a cake and someone is in your kitchen watching you. where participants developed activities for Lithuanian youngsters and further projects. counting the minutes you are spending on mixing the cake mix. Being involved in a project and in all the work that it implies. The external evaluator collects data and records it all so that they will have enough information to develop the report the outcomes of the project.net .

And what does it mean for evaluation? Human Rights Education: Compass Contact Point The Polish organisation Association for Children and Young People “CHANCE” took up the challenge to start a pilot training project on ‘Human Rights Education with Young People’ by e-learning. this report is interesting from the point of view of evaluation because attention is given to the role of the external evaluator and to the problems and challenges of the tasks involved. The combination of residential meetings and follow-up or “in between” virtual encounters offers interesting educational potential.net 74 Compass portal with downloadable versions at http://www.org/ . Several tasks were given to participants in the process of the course. T-Kit on Educational Evaluation in Youth Work www. a free Open Source software package designed using sound pedagogical principles. On the website it was possible to communicate with other participants. Apart from this reflective dimension. It implies a lot of questions: How much can you get involved? How invisible should you be? To what extent do you take part? When should you speak? Should you speak at all? How should you deal with the personal and professional dimensions of the work? Imagine what happens to a team and to a group of participants when somebody is around them all the time taking notes.coe. The course was designed in a modular format. All activities are described. the educational possibilities of the internet are being explored and discovered. Spending time. Material was provided on the website to study and practice.training-youth. Twenty people participated in the course that took place from June until November 2005. The Madzinga report is descriptive. but it also strongly reflective. In the field of non-formal education the first tentative steps are being taken to explore the opportunities that web-based learning can offer. Evaluation in an E-learning pilot course E-learning is booming at the moment. But. Lots of students learn while sitting behind their computers. More information at http://moodle. to help educators create effective online learning communities. Team meetings are reported and essential issues and discussions are raised. Especially in the context of long term projects. more even important are the descriptions of the way participants act and react in different situations. specially developed for educational purposes and free available on the internet. But. How does the person taking notes to deal with the way the group deals with them? The report does not give an answer to all these questions. a lot of time with a group of people deeply involved in a process and being in a position which is actually ”outside” of that process is not very easy. The software used was Moodle75. by chat as well as by forum discussions. This makes it a worthwhile read for people who plan to work with or work as external evaluators. but it does make an effort to explore the challenges faced by an external evaluator in the field of non-formal education. The modules took the participants through the different subjects of Human Rights Education.eycb.int/compass/ 75 Moodle is a course management system (CMS). “COMPASS – a manual on Human Rights Education with Young People”74 published by the Council of Europe was the starting point for the training course. what about situations in which the participants do not know each other and will never have the opportunity to meet? That has an enormous impact on the educational process.

As a result. The moderator scheduled meetings with each of the participants to discuss their learning process and problems and challenges they met. In the beginning. The communication increased. participants are informed in a clear way about the opinions of their colleagues. the participants were asked to fill in a more “traditional” evaluation questionnaire designed by the moderator. as is the case of many educational platforms. T-Kit on Educational Evaluation in Youth Work www. The exception was the level of interactivity of the learning process. • Final evaluation questionnaire At the end of the course. For many participants things just went to fast. The moderators realised that they did not take enough into account that participants had their own schedules defined by working and family life. The evaluation of the discussion forum showed that participants had problems to communicate with each other. • Ongoing evaluation In the discussion forum. not only on the website. One major problem encountered. topics were taken off the website after the subject was closed. This way. was the level of “interactivity”.On-line evaluation • Evaluation forms after the sessions After every session an on-line form was filled in by the participants. • Individual interviews Individual interviews were conducted using chat. Moodle offers evaluation forms which can be adapted to the needs of the users. The topics evaluated were: • the relevance of what was learned • the way reflective thinking was stimulated • the level of interactivity • the support of the tutors and of peers • if participants could make sense of the messages posted on the platform The evaluation was largely conducted using multiple choice questions. But. the moderators decided to stimulate communication between participants by giving them tasks in which they had to work together in small groups. In the regular evaluation forms after every session almost all topics scored positively. The software processes the forms into statistics which can be visualised on the website. the opportunity for “ongoing evaluation” was provided. Nevertheless. but also by email and telephone.net .training-youth. the speed of the course was reduced and all the material was left on the website even after the subject was closed. Evaluation played a big role in this course because it was a pilot training and because of all the new challenges that e-learning presented to the moderators. So. not all participants followed the rhythm proposed by the moderators.  Changes as a result of the evaluation As a result of the evaluation changes were made in the programme of the course. participants had the opportunity to mention any other comments they may have had to make. This was successful. It contained many more open-ended questions. Another problem that came up was the pace of the course. Participants were invited to comment on the course and to discuss opinions among themselves.

transfer of competencies into concrete actions/ situations • Results: Long term transfer (also in organisational and institutional terms) This model by Kirkpatrick worked out very well for this training course. The participants had more than three years of work experience on either a voluntary or professional basis (as managers or multipliers) in the field of international youth work. the long term results could not be known immediately. contents. an European educational centre.V. 24 persons from five countries (Belgium. teamwork. and transfer e. The four levels. coaches deepen their learning and improve their performance. The evaluation itself was based on the theory of Kirckpatrick.net Coaching is an ongoing professional support and orientation process.V. Aspects.: Evaluating Training Programs. Overall Evaluation of a Pilot Long Term Training Course The “Coaching76 in international youth work” Course In 2000 a Steering Group of representative from five institutions (two international youth organisations. Therefore. the effect and utility of the training programme. However. were already included in the training course. Donald L. The concept of the course was to provide a clear understanding about the following things: • What is a coach? • What should be the approach of a coach to international groups? • Where are potential fields of work as a coach? Evaluation strategy The project was planned as a pilot supported by Federal funds. Through the process of coaching. a special evaluator worked with the whole group.training-youth. Each participant was contacted by telephone for an interview. like the atmosphere. As a result. The course was organised by the International Youth Exchange and Visitor’s Service of the Federal German Republic. San Francisco 1998 . also known as IJAB e. learning achievements • Behaviour: Changes in behaviour. belonging to the level of reaction and learning. the Steering Group decided to make an evaluation of the entire training course. The seminar was led by two field experienced facilitators. a working group of trainers and a “Fachhochschule” (University of Applied Sciences) started a further qualification course called “Coaching in international youth work”. Netherlands. As a result. Source The international coach federation http:www.coachfederation. From module to module additional trainers were engaged for specific inputs and training elements. methods. all the actors of the course participated in the evaluation process. Switzerland and Germany) took part in the 5 module training course. etc. Russia.org 77 Kirkpatrick. His evaluation model is based on four levels of learning and behaviour77: • Reaction: Personal reflection by participants on the satisfaction of participants. participants were contacted again after one year. All levels were implemented. for example • Learning: Growth of knowledge. During 2000 – 2002. 76 T-Kit on Educational Evaluation in Youth Work www.

The evaluation with the whole group was completed one year after the project. sometimes behaviour. behaviour. “brainstorm”.e. “water level”. Agreement with total concept Possible transfer Working fields for Coaches Questionnaires Final interview One year after the finish of training course Long term development of Questionnaires competence among participants Transfer of knowledge into the working field (including to institutions). 1 year after the qualification Overall satisfaction Quality of modules Effectiveness of learning tools Level of knowledge Coach’s role Increased competence Changes in Behaviour Transfer => reaction.e. holders of project During and after final seminar Long term evaluation Interview. results Facilitators / Tutors Mid-term and final evaluation Timing is dependent on the availability of the tutors Starting interview Between the “Kick-Off” and the 2nd module personal view on professional competencies of participants Feedback Individual consultations with tutors Participants’ sending organisations Importance of trained competencies and knowledge. “barometer of mood”. Time frame Mid-term evaluation and final evaluation Every module Issue/objectives of Evaluation Participants Satisfaction Quality of modules Effectiveness of learning tools Level of knowledge Expectations for next modules Understanding of the concept Coach’s role => reaction. behaviour. “the reporter”.net . participants. results T-Kit on Educational Evaluation in Youth Work www. “fever curve” Verbal evaluation methods: i. etc. Elements of evaluation. tutors. results External Assessment by Personal competencies facilitators. “hanging-lamp-feedback” Video interviews at beginning of seminar chain Profile of competencies Turn of feedback rounds Small talks with tutors Turn of feedback rounds Small talks with tutors Questionnaires to the project’s stake-holders at the end of the project Partially structured interviews  Self assessment of participants Personal competencies At the beginning. learning. “the tree of perception”. learning Instruments of observation Questionnaires Practice target Complaints box Group visual evaluation methods: i.The following table demonstrates the different stages of the evaluation and the timetable for its implementation. => learning.training-youth. during and at Possible transfer Understanding about a Coach’s the end of the seminar role => learning.

group interactive methods or group discussions. In fact it is quite often one of the highest moments of reflection. We hope this final evaluation questionnaire will be a source of inspiration for developing your own. This “good” final evaluation questionnaire is complementary to the explanations of working with questionnaires already outlined in this T-Kit (see pages 57-61). the adjective “good” has no normative meaning. but rather of the accumulated practices of several experts in different projects concerning final evaluation questionnaires. On the left you can find an example of questions that can be asked in a questionnaire and on the right you find an exploration of “what lies behind”. Even if it tries to review a lot of issues. we have tried to propose examples that can illustrate some important considerations for the construction of evaluation questions. a second training course was designed on the same subject. Results: a second course and next steps All parties were involved in the evaluation. • Elements and themes of the training course Most participants agreed that there is a need for more detailed attention to be paid to learning about communication and coaching during any future course. In this case. At first glance. Some of the main conclusions of this evaluation were • Clarification of the definition of a coach78 has to take place: In the next training course the team should agree on one common definition and stick to it during the whole course.coachfederation. making informed choices and formulations for the questions you ask. it should be conceived one of several evaluation tools. we would like to present a possible structure for a final evaluation questionnaire. But. We hope that this reflection will help you to design your own questionnaire. That happened because participants were not always clear about why and what certain kind on evaluation was needed for. The final evaluation questionnaire is often just one part of the final evaluation. The model of oriented coaching has to be given more space in any future course. The “good” final evaluation questionnaire At the end of this evaluation practices section. including for example. • Evaluation of the evaluation Participants gave feedback that the evaluation itself took too much time and space in the training course. each one corresponding to a column. T-Kit on Educational Evaluation in Youth Work www.org . It is not the result of a concrete project. learning and participation. But. This gave the organisers of the project very detailed information about what they would have to change in future if they wish to repeat the course. The following year. It is used in opposition to the “bad evaluation questionnaire” exercise on page 60. the advantages and disadvantages of each question. This is simply an example. This questionnaire is presented in two parts. We would like to encourage you to reflect on the options. Source The international coach federation http:www.net 78 The person in charge of the coaching process is called a coach. it may be intimidating to have to fill four empty pages up with evaluation questions. The next course team should avoid this. We should not be afraid of “final evaluation questionnaires”.training-youth. experience does demonstrate that participants of educational activities take filling in the evaluation questionnaire very seriously and often even enjoy it. but following very closely the recommendations made after the first cycle. the priorities.

This example has been designed for an educational activity in general. we have tried to reflect our approach to evaluation (evaluation as a total experience) in this evaluation questionnaire.Without pretending to cover every aspect. -. it is necessary to respect the anonymity.training-youth. The questionnaire contains there is a combination of evaluation “by objectives”. in others from 1-5. Evaluation by objectives (Page 21) is a relevant component of a global evaluation.net . knowing the identity of the participant will help to better understand and contextualise the individually filled out questionnaires. in others 5 is the best… That is the reason why this symbol scale is an interesting alternative. This part provides more specific information than a general impression. o. combining qualitative and quantitative evaluations. of the participant. a workshop…) and to each particular context. The most important is to create something new out of it. It is just a tool of inspiration. a kind of “taste”. to which extent were the objectives of the activity reached? ++ + o Ok -100% Objective 1: To… Very Good good Please explain your score? Bad Very bad Objective 2: To… Please explain your score? ++ + o -- - T-Kit on Educational Evaluation in Youth Work www. Final evaluation questionnaire Title of the Educational Activity Place and Date Evaluation questionnaire Name (optional): Comments/ Logic / What is behind the question? Do not forget to clearly identify your questionnaire! Too many are not specific! On the one hand. It is good to start by asking for a general impression of the activity. “by process” and “by performance”. This part will also help you to put into perspective the following more specific parts of the questionnaire that come later. in some countries the 1 is the best.  General Impression What is your general level of satisfaction with the activity? n n n n 0% 50% Please explain the score you have given: Other comments: Objectives In your opinion. Numeric scales are sometimes confusing: in some countries the scales are from 1-10. it is important to differentiate them. if desired. +. We recommend that you consider reflecting on this questionnaire while consulting the ideas outlined in the sections of this T-Kit entitled “why to evaluate” (Pages 15-16) and “what to evaluate” (Pages 19-28). It is appropriate to combine qualitative and quantitative evaluation. It provides a kind of “overall general average level of satisfaction” for each participant. a training course. Even if at times the answers will mix both these impressions. Remember to remind the participants of the objectives of the activity that you want them to evaluate. We have tried to embrace the complexity of an educational activity. --) should be explained at the beginning. It is a simple but very revealing starting point. It can and it should be adapted to the kind of activity evaluated (an exchange. The quantitative scale (++. On the hand.

in general terms. This question helps us to find out. The two questions about the most and the least fulfilled expectations give us some additional information. Learning outcomes What are the most important things you have learned in at this activity? Methodology Please comment on the appropriateness of the methods used during the activity? T-Kit on Educational Evaluation in Youth Work www.training-youth. It is interesting to differentiate between professional and personal expectations because in international youth work they can differ a lot. what participants learned during the activity. Asking in general about the methods will help us to evaluate the overall methodology of the activity (variety. links between methods…) Experience tells us that if we include the word “methodology” in the formulation of the question it can cause some confusion. More specific comments about each method can be asked in the part of the questionnaire devoted to the programme elements Which of your expectations have been fulfilled the least? Please explain your answers. + o -- Again. at the end of the activity. As we have already seen (Pages 26-27) dealing with the expectations (together with the objectives and the programme) is part of evaluating “by process”: seeing the process and the relationships. From “evaluation by outcomes” (Page 25) the most relevant for educational evaluation are the learning outcomes for participants. so it is often easier for participants to answer a question about the methods. The main reason for including them in the questionnaire is to “oblige” participants to compare their original expectations to what happened. Final evaluation questionnaire Expectations What were your initial implications? Comments/ Logic / What is behind the question? Evaluating the extent to which expectations have been fulfilled will allow you to see the extent to which the activity was relevant to the target group. they might be “lost”. “mixed with memories” or out of focus. combination.net . we use a mixture of quantitative and qualitative evaluation. Were your expectations fulfilled? ++ Professional Personal Which of your expectations have been fulfilled the most? Please explain your answers. Otherwise.

These aspects are closely linked to “evaluation by performance” (Page 24) T-Kit on Educational Evaluation in Youth Work www. A line for comments allows for qualitative evaluation of each programme element. exploring their roles. interaction and dynamics. This distinction has advantages and disadvantages but the contrast between them can be interesting and noteworthy.Final evaluation questionnaire Programme elements Learning achievements Programme Element Welcome session Comments: … ++ + o -- Comments/ Logic / What is behind the question? This part deals with the detailed and analytical evaluation of each programme element. For this reason it is very important to evaluate them in relation to the level of participation of members of the group. a bad relation between the participants and the team.training-youth. can provoke the failure of a. This is the reason why the programme flow is an important and relevant aspect to evaluate. Those are at times key factors of success or of failure. Having evaluated the whole programme chronologically. For each programme element there is a line for quantitative evaluation of the learning and the enjoyment. It is useful to order the programme elements chronologically as they have appeared in the programme and to use titles which help participants to easily remember them. for example. By doing so. in principle good programme.net . participants have an overview of its flow. The links and connections between the different programme elements are very important for the learning process. Participants can qualify their quantitative evaluation and add other reflections they consider important. we provoke a “triangulation” of actors.  How did you enjoy it? … What do you think of the general flow of the programme? The “actors” How do you evaluate the role of the group in your learning? How do you evaluate the dynamic in the group? How do you evaluate the work of the team? How do you evaluate your own contribution to the activity? The group and the team in non-formal education are not only a context but also a source of individual learning.

you can come back to the beginning. for us. How are you planning to use this experience in your future work with young people? At the individual level (personally / professionally)? In your organisation: With other partners: Please describe your plans for any future projects you may now have? Preparation In your opinion. In other questionnaires. free time possibilities…): Equipment: Technical information before arrival: T-Kit on Educational Evaluation in Youth Work www. we can find the evaluation of the preparation of the activity at the beg inning. At the end. Those aspects also have a relevant influence in the learning process. within the organisation. We find it interesting to ask participants about it at the end. It might be interesting to distinguish among different levels of outcomes (individual. the real outcomes will only become clear after some time has passed.100 Follow-up Final evaluation questionnaire Comments/ Logic / What is behind the question? This part evaluates the “outcomes” of the activity beyond individual learning. was the preparation of this activity adequate? In terms of contents: In terms of technical information: In term of other practicalities: Logistics and technicalities How do you assess the quality of the logistical organisation of the activity in terms of: Accommodation: Food: Working rooms: Facilities (internet access. After going through all the steps of the process participants can imagine “How things would have been if the preparation would have been done differently?”. sense of security. This is one of the clearest examples of the learning and motivating nature of educational evaluation. But. the key question. This is. See “Evaluation by outcomes” (Page 25). therefore. etc. refer mostly to plans or intentions.net . For most participants. in co-operation with others) or to ask specifically about future projects if this was an expected outcome. Reflecting on this will help participants to start to transfer what they have learned to their home context. because they influence the comfort. The evaluation of logistics and technicalities allows for the improvement of future activities. even if from the “factual” point of view the answers will not always be valid. The answers to this question will. it is still important to include a question about this in the final evaluation questionnaire. concentration.training-youth.

e. Filling in an evaluation questionnaire is an intensive exercise. If it is clear that a similar activity will take place (i.net . invite to use any other extra space available.Final evaluation questionnaire Other Please share any other comment you would like to make? Do you have any suggestions to improve the quality of this activity you would like to share with the organisers of future activities? Thank you for your co-operation! Comments/ Logic / What is behind the question? Leave enough space for answers to this question and if you do not have enough space. not just polite to recognise it and thank participants for their attention and active participation. 101 T-Kit on Educational Evaluation in Youth Work www.training-youth. next year) you can ask for suggestions. It is fair.

Dealing with evaluation requires reflection. competence and experience. but evaluation takes time and resources. We should reserve the appropriate time and human resources to do it paying attention to how the evaluation relates to the rest of the project. and through different projects in which educational evaluation played an important role. but they only make sense when you note down the outcomes and work with the results. we should not overdo it. to be of a high quality. Educational evaluation is considered one of the most complex and challenging aspects of a project. we should remain realistic. Developing good questionnaires reading them and making conclusions out of them takes time. we hope not to disappoint you but we still have to cook our meal! T-Kit on Educational Evaluation in Youth Work www.10 2. Doing interviews can be a very good method but is time-consuming.net . Evaluation techniques and methods are not so difficult to learn. The most important is to remember that the educational evaluation. A tool is as good as the person using it. At this stage. The theoretical and practical ideas in this T-Kit offer a starting point. After considering the complexity of evaluation. In other words. There are loads of excellent group exercises for evaluation. should be implemented with maximum respect to the nature of each project. It involves a lot of aspects and dimensions! We would like to close this practical chapter of the T-Kit with an invitation: Be ambitious but realistic! We can plan a wonderful evaluation.training-youth. The difficulty comes in their application.10 We still have to cook our meal! We have gone through the different stages of evaluation going. through most of them in some depth. where all elements are covered and many different methods are used.

Further exploration Evaluation standards During the 1990’s a number of evaluation societies were founded in Europe. administration.) was created 1997. It is a multidiscipline organization. diplomatic.V. 81 Available at http://www. – Stakeholder identification – Clarification of the purposes of the evaluation – Evaluator credibility and competence – Information scope and selection – Transparency of values – Report comprehensiveness and clarity – Evaluation timeliness – Evaluation utilisation and use • Feasibility: The feasibility standards are intended to ensure that an evaluation is planned and conducted in a realistic. NGO’s and consultants.de.ch/de/index. 80 The Swiss Society of Evaluation (SEVAL) understands itself as forum for the exchange over evaluations between politics.3. More information at http://www.de/index. and cost-effective manner.seval. More information at http://www.degeval. thoughtful. Its aims are to promote professional evaluation practices. – Formal agreement – Protection of individual rights – Complete and fair investigation – Unbiased conduct and reporting – Disclosure of findings • Accuracy:The accuracy standards are intended to ensure that an evaluation produces and discloses valid and useful information and findings pertaining to the evaluation questions.cfm. – Appropriate procedures – Diplomatic conduct – Evaluation efficiency • Propriety:The propriety standards are intended to ensure that in the course of the evaluation all stakeholders are treated with respect and fairness. – Description of the evaluation – Context analysis – Described purposes and procedures – Disclosure of information sources – Valid and reliable information – Systematic data review – Analysis of qualitative and quantitative information – Justified conclusions – Meta-evaluation 79 The German Evaluation Society (DeGEval e.training-youth. It is an association of persons and institutions.degeval. have published 27 evaluation standards81 organised in four categories: • Utility:The utility standards are intended to ensure that an evaluation is guided by both the clarified purposes of the evaluation and the information needs of its intended users. universities. who are active in the field of evaluation. the sharing of different perspectives on evaluation as well as the information and exchange about it. It promotes the quality and spreading of evaluation.net . The German Evaluation Society (DeGEval)79 and the Swiss Society of Evaluation (SEVAL)80.php?class=Calimero_Webpage&id=9023 10 T-Kit on Educational Evaluation in Youth Work www.

If you would like to find out something related to personal. The seminar programme allows you a maximum of 15 minutes per day every morning for evaluation. Feasibility: With a group of 20 and 5 days. You need this information to document interviews properly. Or the group is quite homogenous. Check the time frame: you have only 15 minutes per day. you would have to conduct interviews with 4 persons per day. • Some standards will not be applicable to certain evaluations. feasibility and ethical grounding of the evaluation being conducted. They are intended to provide a frame of reference for conducting and assessing evaluations. Hungary and Algeria. there are participants with very different social and religious backgrounds. but not to the evaluation of individuals. T-Kit on Educational Evaluation in Youth Work www.10 Application of the standards These standards can help the evaluation team ensure the utility. In the application of these evaluation standards we should consider that: • The DeGEval standards are designed to uphold and raise the quality of evaluations. You can not take notes and ask questions in the 15 minutes available. These evaluation standards are not intended to devalue evaluations which do not meet a particular standard in a certain way. The evaluation team and all participants have the job of finding an appropriate solution which takes account of the purposes and context of the evaluation in hand. It is not always possible to give all standards equal consideration. please proceed with the necessary sensitivity and think about how the context of a mini-group interview changes the way participants may react to the questions. Accuracy: In your group. then you need very clear questions. You have one staff member responsible for conducting the evaluation interviews. Exercise on Evaluation Standards You are planning a 5 day seminar with 20 youngsters from Russia. But. They formulate key points which evaluators should respect and goals they should pursue. • How they are implemented is a deciding factor. Propriety: You are only one person evaluating. • From time to time the standards might contradict each other. interviews give us the most personal contact with the participants and team. for example in performance judgement processes or employee assessment (staff evaluation is excluded). • The evaluation will often need to weigh up the advantages and disadvantages of various alternatives. One of your chosen evaluation methods is a personal interview with each participant. Brief grounds for not applying a standard should be provided in such a case. Another solution could be to do mini-group interviews. They can be used in the planning of the evaluation. It cannot take place schematically. You have planned 5 minutes per person for the interviews. ethical or religious aspects.training-youth.net . • The standards apply to evaluations in general. Think about the questions you want to ask before going into the interviews. Doing interviews means to take note of the statements of participants. think about which circumstances are necessary for a “good” evaluation? Solution: Utility: Yes. or during the evaluation process for adapting or readjusting it. so you can conclude that individual interviews cannot be done properly. You can ask the interviewees if they agree to use a Dictaphone. Check this evaluation plan against the four categories of evaluation standards: Does it meet them all? How and why? Based on this. as a check list.

This makes transparent the process of involvement of people within the exercise.e. SECTION 1: Pre Stages A set of questions that should be answered when you are creating your plan or activity. SECTION 6: Time Frame for Evaluation A map of when each part of the evaluation should take place. methods for collection of information and some hints on analysis. You can use Section 1 before you start planning your project to help you think through the processes needed. SECTION 2: Development and Strategy The areas to assess when you are evaluating the development and strategy of a project. SECTION 5: Outcomes Explores evaluation of outcomes.net . 10 T-Kit on Educational Evaluation in Youth Work www. It is only a suggestion. All other Sections should be followed during and after your activity. It has been split into 5 sections: • Pre stages • Development and Strategy • Selection • Activities • Outcomes Each section has basic questions to ask in this area. It will help you from the moment that you start thinking about aims and objectives until the presentation of your evaluation report. i.A tool for developing your own evaluation plan – The SALTO checklist This document is a map or framework on which you can base an evaluation of a project. participants and trainers. please feel free to adapt it according to the nature of your project. There is a box to tick when you have completed each one. demonstration of the impact of the project on the target group and the wider community. This is the important ground work to a report and recommendations for future changes. There is a timetable which you can refer to when to use which Section and its sub sections. SECTION 4: Activities Explores key areas within the activities to evaluate.training-youth. SECTION 3: Selection The areas to evaluate in the selection procedure of those involved.

2 Learning of participants by team SHORTLY AFTER n 2.5 Participants follow up of activities T-Kit on Educational Evaluation in Youth Work www.10 TIME WHEN THINGS SHOULD HAPPEN STRATEGY DEVELOPMENT Which Evaluation Section to Complete Following the Numbers from the Guide n1 n1 Pre Stages Check list Decision on evaluation programme.2 Expectations of participants met n 4. n 2. n 5.3 Evaluation of whole project complete.1 Evaluation of outcomes by team n 5.1 Preparation.training-youth.3 Co-operation and communication by team n 4.1 Selection of participant n 3.2 Expectations of participants n 5.3 Individual activity Evaluation Report complete n 5. choice of methods and development of tools.2 Expectations of team met n 2.2 Expectations of different stakeholders ACTIVITY START TIME n 2.2 Infrastructure and support by participants n 4.2 Learning needs of participants DURING ACTIVITY END OF ACTIVITY n 3.5 Participants follow up of activities By 12 MONTHS n 5.2 Infrastructure and support by team n 4.2 Expectations of stakeholders met n 4.3 Co-operation of partners n 3. delivery.6 Long term impact .1 Strategy Analysis n 2.2 Learning outcomes of participants TEAM EVALUATION AFTER ACTIVITY n 2.3 Content and methods by team n 5.net n 5.2 Selection of team By 6 MONTHS n 2.3 Content and methods by participants n 5.1 Are participants your target group? n 2.2 Collection of examples of good practice and new methods n 5. evaluation and follow up n 5.

young people etc. • When you are completing your evaluation.Before starting: • Read through this document to the end and consider all the points before you begin your whole project and the individual activities within it. Pre Stages Answer these questions when you are constructing your project plan. Development and Strategy 2.net . � n – What are the aims and objectives for your project and why? � n – What are all the stakeholders’ expectations (funders. follow all sections below very carefully. when it will happen and who will do it. Strategy Analysis n complete QUESTIONS – What were the aims and objectives of the activity/project? – Why were these aims and objectives chosen? – Were the activities/strategies based on analysis of needs of young people? – Were the activities/strategies based on the needs of youth workers? – Were the activities/strategies based on the priorities of the funders (if any)? – Were the activities/strategies based on the priorities of your organisation? – What was the process of deciding on the strategy? – Who was the target group and why was this group chosen? – Was the activity/strategy manageable? METHODS – interview or questionnaire of decision makers – reading of reports from relevant meetings – reading of needs analysis research completed ANALYSIS – Develop an understanding of the process of construction of the aims and objectives! T-Kit on Educational Evaluation in Youth Work www.) � n – Who is/are your target group(s) and why? � n – How will you gain access to this/these target group(s)? � n – Who will run your activities and on what criteria have you selected them? � n – What methods and resources will you use to fulfil your aims and objectives and why? � n – What outcomes do you hope to achieve? � n – How will you assess these outcomes? 2.training-youth. • Write a schedule of what evaluation will take place. youth leaders.1. 10 1. volunteers. Tick the boxes on completing the questions.

Co-operation with partners n complete QUESTIONS – Which partners were you cooperating with on your project? – What roles did the different partners play in the a) development b) process c) evaluation of the different activities? – Was there equal co-operation in the development of the activity between all partners? – Was the communication between partners adequate? – Was the communication between all actors (i. All evaluations in every field of work are political and you must be aware of the context in which you complete your work. 2.10 2. funders. ANALYSIS – Decide what the expectations of these differing sets of people towards your project are and towards each individual activity within it.training-youth. ANALYSIS – Use the data to decide if there was a democratic process in the development of the project and to draw a picture of the involvement of the different partners in the activities.g. youth workers/ disadvantaged young people c) Actors in the strategy e. trainers.3. METHODS – Interview.2. European Commission/ National agencies/local authorities b) Target group e.g. METHODS – Questionnaires.net . Look at tensions or differences. trainers. volunteers. partners. facilitators d) Actors in the field e.g. Expectation of key people who have an interest in your activity n complete QUESTIONS – Who were the key stakeholders (key players)?: a) Funders e. objectives or setting the target groups? If yes explain what happened. Youth Organisations – What was the relationship between the stakeholders? – Were their expectations met? NOTE: – Each stakeholder needs to be involved in the evaluation to provide a balanced picture. focus groups or questionnaires of the actors involved.e.g. focus groups or interviews to all stakeholders before the activities take place. T-Kit on Educational Evaluation in Youth Work www. – Documentation and minutes from meetings. participants) involved adequate? – Were there misunderstandings through different use of words (definitions) in the aims.

Selection 3.1. focus groups or questionnaire of actors involved in selection – Questionnaires of participants after the training course – Questionnaires to the team after the training course. whether it worked successfully together! T-Kit on Educational Evaluation in Youth Work www. Selection of actors to carry out the activities n complete QUESTIONS – How were actors (such as trainers. if a team was created.3. ANALYSIS – Use the data to decide if appropriate actors were selected for the tasks and.net .training-youth.2. volunteers) selected and on what criteria were they selected? – Did the actors implement the activity you suggested? – Was there a gender balance? – Was there a regional balance? – Was there a balance in the areas of experience and expertise? – Did the selected actors become a team? – Was the team effective at organisation and time management? – Were there conflicts and were they resolved? – Would you recommend the same actors for future activities? METHODS – Interview. Selection of participants 10 n complete QUESTIONS – Who were the intended participants for the activities? – Was there a shared understanding of the target group? – Who was the announcement of the activity sent to? – Who applied? – What were the selection criteria? – Were the participants who turned up to the activity your specified target group? METHODS – Interview. focus groups or questionnaire of actors involved in selection – Interview a number of participants and trainers to find out about the participants (this could be done by telephone) ANALYSIS – Use the data collected to decide if you reached your target group in your activities and if not where the problems arose! 3.

net ANALYSIS – Use the data to decide if the appropriate contents and methods were used! . give an example.3. The activity 4. etc. – Ask questions on the process in a post questionnaire or interview. Preparation.110 4. energy and money given to the: a) preparation of each activity? b) delivery of each activity? c) evaluation of each activity? d) follow up of each activity? METHODS – Interview or questionnaire of actors involved ANALYSIS – Decide if enough commitment and total resources were spent on each element within the production of the activity! 4. Infrastructure and Support n complete QUESTIONS – Did the location for the activity have enough or appropriate space? – Was the appropriate equipment available? – Was there adequate support staff for the activity? METHODS – Questionnaire after training course for participants – Interview or questionnaires for actors who ran the event. T-Kit on Educational Evaluation in Youth Work www.). – Ask those people creating the activity to provide detailed documentation (daily reports by the participants. – Were the contents of the activities based in the realities of the participants? If yes.1. give an example.2. – Was there an effective group learning process? – Were the participants actively involved in the activities? – Did the activities use resources from the field? – Did the content and theme of the/each activity fit the aims and objectives set? – Did your project have an impact on more people than the participants involved? METHODS – Observations: have an observer sit and watch the process and make notes (external evaluator).training-youth. ANALYSIS – Examine the data to decide if the infrastructure and support were satisfactory for the activity! 4. evaluation and follow up n complete QUESTIONS – Were adequate resources of time. Contents and Methods n complete QUESTIONS – Which priorities was the activity orientated towards? – Were the methods chosen participant centred? If yes. delivery. detailed programme.

5. etc. Evaluation 111 n complete QUESTIONS – What were your expected outcomes from the individual activity? – Were your expectations of outcomes met? – What evaluation was completed for the individual activity? – What were the results of the evaluation? – How do the results of the evaluation and outcomes fit into the field of youth work and the priorities of the project. 5. Demonstrating Quality n complete QUESTIONS – How can quality be demonstrated? METHODS – The Evaluation Report is a tangible way of demonstrating the quality of your activities. 5. Use the different title headings for each section of your report and then finish with a conclusion containing recommendations for the future. This is not a quick fix for all the areas of evaluation and is not a substitute for the in-depth evaluation process given above. New technology methods could be used providing evidence based portfolios of achievement on CD ROMs and videos. funders. The evaluation report will help you improve your activities next time. Communication Skills. their peers.g. e. An evaluation meeting. This assessment should be comparative between when the participants arrive and where they have reached by the end. T-Kit on Educational Evaluation in Youth Work www.net . i. External evaluators give your report more credibility because they are understood as being more impartial. Learning Outcomes n complete QUESTIONS – What skills and competencies have the participants gained. a pre-questionnaire and a post-questionnaire or another method of recording and analyzing the achievements of participants. the team or by external assessors.training-youth.1.? METHODS – Interview or questionnaire for the coordinator of each activity. your organisation.5.? METHODS – Assessment of learning of the participants by: themselves (self assessment).3. Using this guide will be a start to producing an evaluation of individual activities and your overall project. Self Awareness.2. If you need help you can request the support of an external evaluator to carry out the task. Team Work. Outcomes To explore outcomes each individual activity needs to be examined separately and then brought together in a synthesis. A report from the evaluation meeting should then be produced. etc. Intercultural Learning. This brings all the key stakeholders together to discuss the key questions that are in this guide.e.

– To find this information follow up participants 6 months or longer with questionnaires and/or interviews to find out what activities relating to themes have been carried out by participants. 5. websites from the different activities. Multiplying effect n complete QUESTIONS – How do you demonstrate multiplication? METHODS – Look for an increase in the number of activities in the priority areas that you have stated in your aims and objectives. T-Kit on Educational Evaluation in Youth Work www.6. – What is the long term impact on the participants of an activity? – What is the long term impact on the wider community? METHODS – Following up participants over 2 years or more can trace the long-term impact on their lives and the activities they have been involved in. a) Reduction of reports of racism by young people towards their peers b) Increased participation of young disabled people in youth activities. – The creation of new methods.training-youth. Added Value to the youth field n complete QUESTIONS – How do you demonstrate added value? METHODS – A collection of examples of good practice such as videos. – Look for indicators of impact on young people.g. 5.net NOTE – It is difficult to prove the link between your activities and results! . c) Increased participation of young people in youth organisations. CD ROMs. – Create specific indicators for the aims and objectives/priorities of your project. Long Term Impact n complete QUESTIONS The long term impact is quite difficult to measure and will be affected by many variables other than the youth activities that you are creating. – Look for an increase of trainers.11 5. e. tools and articles on the different themes of the activities.5. support people and resources in priority areas. recording their life story. This can be assessed by taking part in interviews or questionnaires every 6 months.4.

(1981). G. New perspectives on organisational effectiveness. Recognising non-formal & informal learning in the youth sector. pp. Organisational Assessment. • CHISHOLM Lynne (2001) . Una pauta para la evaluación de centros educativos. E. pp. Revista de Investigación Educativa. Terminology Cheat Sheet. Madrid: Rialp. En R. 117-150. • LAWLER.M. 315-385. (1978). Wittrock (Ed. San Francisco: Jossey-Bass. G. Beuth Verlag Berlin. and materials. 21-56.). • GUBA. 7 (13). USA. • GOODMAN. Extensión de la evaluación. V. Cologne.11 Bibliography • COLEMAN. y otros (Eds. Wolfgang Beywl. Washington. S. Quantitative methods. García Ramos.training-youth. D. (1989). y LINCOLN.. Standards for evaluations of educational programs. Bridges of recognition. J. • JORESKOG. Department of H. • GARCÍA RAMOS. 47-50 • DE MIGUEL.S. (1989).) (1977).net .C. Handbook of research on teaching. Evaluation and Accreditation. Auflage. pp. E. J. M. pp. San Francisco: Jossey-Bass. Revista Española de Pedagogía. • JOINT COMMITTEE on Standards for Educational Evaluation (1981). y PENNINGS. (Eds. T-Kit on Educational Evaluation in Youth Work www. Diagnóstico. • LINN. L. En M. 4. Structural analysis of covariance and correlation matrices. German Evaluation Society Standards. Y. Nueva York: Macmillan. Wien. Germany and Sandy Taut. Los Angeles. Pérez Juste y J. 92-118.C. P. Office of Education. Innsbruck • DeGeval (2001): Summary of Evaluation Standards. Nueva York: Wiley Sons. pp. J. • DIN-Taschenbuch (2003): Normen zum Qualitätsmanagement. (1975). R. (43). Nueva York: McGraw-Hill. U. Modelos de investigación sobre organizaciones educativas. • Eurodesk: Eurodesk Quality process. and others (1966) Equality of educational opportunity. Psychometrika. London: European Council of International Schools. K. projects.E. (130). evaluación y toma de decisiones. Effective evaluation. pp. (1986). Zürich • ECIS (1987).W.) (1980). 443-477. M. Part I: Evaluation 2004 – 2005 • GARCÍA HOZ.

11

• LOEHLIN, J. C. (1987). Latent variable models, LEA, Hillsdale, N.J. • LONG, J. S. (1987). Covariance structure models. An introduction to LISREL, Beverly Hills, CA: SAGE. • U.S. Department of H.E.W., Office of Education, Washington, D.C. (2000) • MCMAHON, A. y otros (1984). GRIDS Handbook. Londres: Longman. • NATIONAL Council on Educational Statistics (1991). Educations counts: An indicator system to monitor the nation’s educational health, U.S. Department of Education, Washington, D.C. • NEVO, D. (1983). The conceptualization of educational evaluation: An analytical review of the literature, Review of Educational Research, 53, pp. 117-128. • NEVO, D. (1994). Combining internal and external evaluation: A case for school-based evaluation, Studies in Evaluation, 20, pp. 87-98. • NORTHWEST Regional Laboratory (1985). Building Level Effectiveness Survey, N.R.E.L., Portland. • OAKES, J. (1989). What educational indicators? Educational Evaluation and Policy Analysis, 11 (2), pp. 181-199. • PEDHAZUR, E. J. (1982). Multiple regressions in behavioural research: Explanations and prediction. Nueva York: Rinehart and Winston. • Quality criteria/ Standards Broschure and CD-Rom “QS-Kompendium” (2000) Broschure No. 24, p. 75, Bundesminsterium fuer Familie, Senioren, Frauen und Jugend (BMFSFJ) • REED, D. and others (1987). Social control and the qualitative and quantitative. Communication in the Annual Meeting of the American Educational Research Association, Washington, DC. • RICHARDS, C. (1988). A typology of educational monitoring-systems. Evaluation and Policy Analysis, 10 (2), pp. 106-116. • ROSALES, Carlos, (1990). Evaluar es reflexionar sobre la enseñanza (Madrid: Nancea) • SANCHO, J. (1993). Evaluar, conocer, transformar y mejorar. Aula de Innovación Educativa, (6), pp. 47-51. • SANTOS, M. A. (1993). La evaluación: un proceso de diálogo, comprensión y mejora. Archidona, MA: Aljibe. • SCRIVEN, M. (1967). The methodology of evaluation. En R. E. Stake (Ed.), AERA Monograph Series on Curriculum Evaluation N.° 1, Rand McNally, Chicago. • SOLER, E. (Coord.) (1993). Fundamentos de supervisión educativa. Madrid: La Muralla, S.A. • STUFFLEBEAM, D. L. (1983). The CIPP model for programme evaluation. En G.F. Madaus, G. F. y otros (Eds.), Evaluation models. Boston: Kluwer-Nijhoff, pp. 117-141.

T-Kit on Educational Evaluation in Youth Work www.training-youth.net

• STUFFLEBEAM, D. L. (1991). Evaluation guide for evaluations of programs, services, and organisations. Kalamazoo, MI: CREATE.

• STUFFLEBEAM, D. L. and others (1971). Educational evaluation and decision making. Peacock, Itasca, Ill. • TENBRINK T. & Cooper, J. M. (2003). , An Educator’s Guide to Classroom Assessment Boston, Houghton Mifflin. • U.S. DEPARTMENT OF EDUCATION (1991). America 2000: An education strategy Department of Education, Washington, D.C. • WALBERG, H. J. (1976). Psychology of learning environments: Behavioural, structural or perceptual?. En S.L. Schulman (Ed.), Review of Research on Education, (4), Itaska, IL: Peacook. • YORKE, D. (1987). Indicators of institutional achievement: some theoretical and empirical considerations. 16 (1), pp. 3-20.

11

T-Kits series
T-Kit 1: Organisational Management T-Kit 2: Methodology in Language Learning T-Kit 3: Project Management T-Kit 4: Intercultural Learning T-Kit 5: International Voluntary Service T-Kit 6: Training Essentials T-Kit 7: Under Construction: Citizenship, Youth & Europe T-Kit 8: Social Inclusion T-Kit 9: Funding and Financial Management

http://www.training-youth.net/INTEGRATION/TY/Publications/T_Kits.html

T-Kit on Educational Evaluation in Youth Work www.training-youth.net

11

Web-ography
In addition to the web – sites references of the text: • Directorate of Youth and Sport of the Council of Europe www.coe.int/youth • YOUTH programme of the European Commission www.europa.eu.int/comm/education/youth.html • Partnership in the Field of Youth between the Council of Europe and the European Commission. www.youth-partnership.net • European Youth Forum www.youthforum.org • All Different – All Equal European campaign on Diversity, Human Rights and Participation http://alldifferent-allequal.info • The encyclopaedia of informal education www.infed.org • Electronic Journal of Educational Research, Assessment and Evaluation http://www.uv.es/RELIEVE/ • European Organisation of Quality, Brussels www.eoq.org • European Organisation of evaluation www.Eureval.org: • International Youth- and Visitors’ Service www.ijab.de/dija/ikl: • Centre for Programme Evaluation, Melbourne http://www.edfac.unimelb.edu.au/EPM/CPE/ • Evaluation in der Europäischen Kommission http://europe.eu.int/comm/dg19/en/evaluation/index.htm • Evaluation of aid to non-member countries http://www.europa.eu.int/comm/scr/evaluation/index.htm • International Evaluation Research Group http://www.c3e.fr/Inteval/home.htm
T-Kit on Educational Evaluation in Youth Work www.training-youth.net

• OECD/PUMA – Performance Management (OECD) http://www.oecd.org/document/27/0,2340,en_2649_34629_2088411_1_1_1_1,00.html

training-youth.net .Notes 11 T-Kit on Educational Evaluation in Youth Work www.

.

int .uk http://www.vesmirbooks.hu http://www.co.: +36 (06)1 329 2170 Fax: +36 (06)1 349 2053 E-mail: euroinfo@euroinfo.uno-verlag.mx http://www.wolfermann@librairie-kleber. NY 10520.dk FINLAND/FINLANDE Akateeminen Kirjakauppa PO Box 128 Keskuskatu 1 FIN-00100 HELSINKI Tel.jean-de-lannoy.com Council of Europe Publishing/Editions du Conseil de l’Europe F-67075 Strasbourg Cedex Tel.gad.: +34 914 36 37 00 Fax: +34 915 75 39 98 E-mail: libreria@mundiprensa. USA Tel.ru SPAIN/ESPAGNE Mundi-Prensa Libros. S.com http://www. 1-5369 Canotek Road OTTAWA.: +33 (0)3 88 41 25 81 – Fax: +33 (0)3 88 41 39 10 – E-mail: publishing@coe.pt RUSSIAN FEDERATION/ FÉDÉRATION DE RUSSIE Ves Mir 9a.librairie-kleber.int – Website: http://book.lannoy@dl-servi.no POLAND/POLOGNE Ars Polona JSC 25 Obroncow Street PL-03-933 WARSZAWA Tel. s.o.V.com GERMANY/ALLEMAGNE AUSTRIA/AUTRICHE UNO Verlag GmbH August-Bebel-Allee 6 D-53175 BONN Tel.: +32 (0)2 231 04 35 Fax: +32 (0)2 735 08 60 E-mail: order@libeurop.) Rua do Carmo.kauffmann.de http://www.r.: +1 613 745 2665 Fax: +1 613 745 7660 Toll-Free Tel. rue Henri Barbusse F-93308 AUBERVILLIERS CEDEX Tél.: +49 (0)228 94 90 20 Fax: +49 (0)228 94 90 222 E-mail: bestellung@uno-verlag.com NORWAY/NORVÈGE Akademika Postboks 84 Blindern N-0314 OSLO Tel.coe.be http://www.com http://www.no http://www. Tel.dept@renoufbooks. 37 E-28001 MADRID Tel.A.: +33 (0)1 40 15 70 00 Fax: +33 (0)1 40 15 68 00 E-mail: commande@ladocumentationfrancaise.enquiries@tso.: +7 (8)495 623 6839 Fax: +7 (8)495 625 4269 E-mail: orders@vesmirbooks.: +31 (0)53 5740004 Fax: +31 (0)53 5729296 E-mail: books@delindeboom.hu ITALY/ITALIE Licosa SpA Via Duca di Calabria.libeurop.com SWITZERLAND/SUISSE Van Diermen Editions – ADECO Chemin du Lacuez 41 CH-1807 BLONAY Tel.tsoshop.: +32 (0)2 538 43 08 Fax: +32 (0)2 538 08 41 E-mail: jean.com http://www.de GREECE/GRÈCE Librairie Kauffmann s.cz http://www.gr HUNGARY/HONGRIE Euro Info Service kft. Canada Tel. M.manhattanpublishing.: +30 210 32 55 321 Fax.: +33 (0)3 88 15 78 88 Fax: +33 (0)3 88 15 78 80 E-mail: francois.: +47 2 218 8100 Fax: +47 2 218 8103 E-mail: support@akademika.co.pl http://www. Kolpacnhyi per.fr Librairie Kléber 1 rue des Francs Bourgeois F-67000 STRASBOURG Tel.com http://www.licosa. Stadiou 28 GR-105 64 ATHINAI Tel. 70 P-1200-094 LISBOA Tel.pt http://www.: +45 77 66 60 00 Fax: +45 77 66 60 01 E-mail: gad@gad.: (866) 767-6766 E-mail: order.com http://www.gr http://www.com. De C.11 Sales agents for publications of the Council of Europe Agents de vente des publications du Conseil de l’Europe BELGIUM/BELGIQUE La Librairie Européenne The European Bookshop Rue de l’Orme.de.ru http://www.cz DENMARK/DANEMARK GAD Vimmelskaftet 32 DK-1161 KØBENHAVN K Tel.mundiprensa.: +358 (0)9 121 4430 Fax: +358 (0)9 121 4242 E-mail: akatilaus@akateeminen.euroinfo.A.com FRANCE La Documentation française (diffusion/distribution France entière) 124. Ltd. 1/1 I-50125 FIRENZE Tel.a.com http://www.be Jean De Lannoy Avenue du Roi 202 Koningslaan B-1190 BRUXELLES Tel.: +44 (0)870 600 5522 Fax: +44 (0)870 600 5533 E-mail: book.renoufbooks.mundiprensa.es http://www.: +52 (01)55 55 33 56 58 Fax: +52 (01)55 55 14 67 99 E-mail: mundiprensa@mundiprensa. s.: +30 210 32 30 320 E-mail: ord@otenet.: +351 21 347 42 82 / 85 Fax: +351 21 347 02 64 E-mail: info@livrariaportugal.suweco.org UNITED KINGDOM/ROYAUME-UNI The Stationery Office Ltd PO Box 29 GB-NORWICH NR3 1GN Tel.: +1 914 271 5194 Fax: +1 914 271 5856 E-mail: Info@manhattanpublishing.F. 141 Delegacíon Cuauhtémoc 06500 MÉXICO.akateeminen.: +420 2 424 59 204 Fax: +420 2 848 21 646 E-mail: import@suweco. 1137 Bp. H-1137 BUDAPEST Tel. Klecakova 347 CZ-180 21 PRAHA 9 Tel. Ontario K1J 9J3.: +48 (0)22 509 86 00 Fax: +48 (0)22 509 86 10 E-mail: arspolona@arspolona.fr http://www.livrariaportugal.arspolona.a.akademika.: +41 (0)21 943 26 73 Fax: +41 (0)21 943 36 05 E-mail: info@adeco.be CANADA Renouf Publishing Co.dk http://www.com.com MEXICO/MEXIQUE Mundi-Prensa México. Szent István krt.mx NETHERLANDS/PAYS-BAS De Lindeboom Internationale Publicaties b. 12. de Ruyterstraat 20 A NL-7482 BZ HAAKSBERGEN Tel. Castelló.com.uk UNITED STATES and CANADA/ ÉTATS-UNIS et CANADA Manhattan Publishing Company 468 Albany Post Road CROTTON-ON-HUDSON.ladocumentationfrancaise. RU-101000 MOSCOW Tel. D.com CZECH REPUBLIC/ RÉPUBLIQUE TCHÈQUE Suweco CZ.delindeboom. Lda.com. 1 B-1040 BRUXELLES Tel. Río Pánuco.pl PORTUGAL Livraria Portugal (Dias & Andrade.v.fr http://www.adeco.: +39 0556 483215 Fax: +39 0556 41257 E-mail: licosa@licosa.org http://www.

.

all Partnership activities have merged into one single framework Partnership agreement.: +33 (0)3 90 21 50 57. human rights education and intercultural dialogue. better understanding and knowledge of youth. The Partnership between the Council of Europe and the European Commission in the field of youth thus brings together the two institutions’ experience in non-formal education.youth-partnership. v 12 / US$ 18 ISBN: 978-92-871-6023-2 -:HSTCSH=V[UWXW: Partnership Programme with the European Commission Council of Europe . This agreement has since been renewed several times and additional agreements were initiated in the fields of “youth research” and “Euro-med co-operation”. more than 1. youth policy. Please consult the web-site www. whose aim was “to promote active European citizenship and civil society by giving impetus to the training of youth leaders and youth workers working within a European dimension”.200 young people participated directly in these activities and many more were able to benefit from this experience through the Partnership publications. which focuses on the following topics: European Citizenship.net for updated information on T-Kits and translations. youth research and youth work practice. Quai Jacoutot. French and German. T-Kits are produced by the Partnership in English. quality and recognition of youth work and training. the Council of Europe and the European Commission decided to take common action in the field of European youth worker training. are embedded in a structured dialogue and contribute to the implementation of the political objectives of both partners. Since 1998. including the Training Kits.NGBEN 1.youth-knowledge.net . Since 2005. youth policy development. fax: +33 (0)3 88 41 27 77 http://www. Both institutions therefore initiated a Partnership agreement. All Partnership activities and publications enhance the exchange of experience and good practice between the actors involved. F-67075 Strasbourg Cedex Tel. The co-operation between the two institutions covers a wide spectrum of activities and publications. Several of them are also available in other languages following national initiatives.Educational Evaluation in Youth Work T-KIt No 10 COUNCIL OF EUROPE CONSEIL DE L'EUROPE European Commission In 1998. T-Kits are thematic publications written by experienced youth trainers and constitute easy-to-use handbooks for use in educational activities.

Sign up to vote on this title
UsefulNot useful

Master Your Semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master Your Semester with a Special Offer from Scribd & The New York Times

Cancel anytime.