TUM School of Education Technical University of Munich

Quality assurance and quality development of

STEM teaching and learning
Seminario International STEAM 2017
Lima, September 26, 2017

TUM School of Education Technical University of Munich

Accountability:
Are we doing our best?

Evaluation

1
TUM School of Education Technical University of Munich

Agenda
• Evaluation in Educational Systems
• The Experimento Program
• Effective Science Lessons I: Phenomena
• Effective Science Lessons II: Problem Orientation
• Conclusion

TUM School of Education Technical University of Munich

CONTEXT

INPUT PROCESS OUTPUT/OUTCOME

CIPP Model, Stufflebeam 2007

2
TUM School of Education Technical University of Munich

Context

? ✓ ✓
INPUT PROCESS OUTPUT/OUTCOME

CIPP Model, Stufflebeam 2007

TUM School of Education Technical University of Munich

Agenda
• Evaluation in Educational Systems
• The Experimento Program
• Effective Science Lessons I: Phenomena
• Effective Science Lessons II: Problem Orientation
• Conclusion

3
 TUM School of Education Technical University of Munich

Experimento –
Siemens Stiftung’s
international education
program
Major use of resources
Huge effort

Questions:
Are the goals we’re working toward being achieved?
How can the program be developed further?

Evaluation
8

TUM School of Education Technical University of Munich

Does the Experimento Program

achieve its goals?
INPUT PROCESS OUTPUT/OUTCOME
Promote MINT Develops the
Experimento approaches,
education
educational knowledge and
through effective
program skills of the
science lessons
students

? •

•
Includes
phenomena
Opens problems
and questions
•

•
Use concepts to
explain phenomena
Realize problems
and ask questions
9

4
 TUM School of Education Technical University of Munich

Agenda
• Evaluation in Educational Systems
• The Experimento Program
• Effective Science Lessons I: Phenomena
• Effective Science Lessons II: Problem Orientation
• Conclusion

10

TUM School of Education Technical University of Munich

Effective Science Lessons

include phenomena
Findings from science:
• Connecting working knowledge with application situations (Magnusson et al. 1999; Anders et al. 2013)
• Use of concepts to explain phenomena (National Research Council 2012)

Concept: Suitable phenomena: Non-suitable

The current flows Bulbs, hairdryer, electric phenomenon:
when the circuit is kettle Electrostatic charging
closed. of hair

11

5
 TUM School of Education Technical University of Munich

Effective Science Lessons

include phenomena
Research through evaluation
A1: Simple
(method: content analysis):
electrical circuit
• Number of phenomena per experiment
(3 experiments)
• Suitability of phenomena to the
experiment
5,0
Average phenomena per experiment 4,5
across all subject areas: 4,0
M = 2.4 Number of phenomena 3,5

3,0
Experiment Quanti What are they?
ty
2,5
How many everyday 2,0
A1.1: phenomena are
The named? 7 1,5
lamp How many of these Charge 1,0
should everyday phenomena High voltage
light up! are not suited to the 2 pylon 0,5
experimental
situation? 0,0
B3 C2 A2 B5 B6 C3 B2 A3 C1 C4 A5 B1 A1 B4 C6 A4 C5

phenomenainsgesamt
Phänomene overall suitable phenomena
geeignete Phänomene
12

TUM School of Education Technical University of Munich

Agenda
• Evaluation in Educational Systems
• The Experimento Program
• Effective Science Lessons I: Phenomena
• Effective Science Lessons II: Problem Orientation
• Conclusion

13

6
 TUM School of Education Technical University of Munich

Effective Science Lessons

open problems and questions
Findings from science:
• Problems intensify the learning process (Dochy et al. 2003)
• Problems support the development of problem-solving strategies (Hmelo-Silver 2004)
• Asking questions and having guesses are part of the scientific circle (Keys 1998; Morrison &
Ledermann 2003)

Informational text Everyday story

(A1.1 The lamp should light up!)
Many devices in our everyday life
require electric current, such as
lamps, calculators, smartphones,
televisions and refrigerators. The
current comes either from a
socket, from a battery, or perhaps
from a solar cell.
(A1.2 Switching on and off)
„The cake batter is done mixing,“ Ben calls to his
mother. He then pulls the plug from the socket while
the device is still running. „You need to turn the mixer
off before you pull the plug,“ his mother instructs him.
„Otherwise, it can be dangerous for the next person
who uses the mixer.“ „But why?“ asks Ben. „The
mixer is turned off, one way or the other.“

How can you make a lamp light Research How is a switch supposed to function
up using the available materials? Questions in a circuit?

14

TUM School of Education Technical University of Munich

Effective Science Lessons

open problems and questions

Research through evaluation More questions (teacher instruction):

(method: content analysis): • How does a switch work?
• Problem focus of introduction • Where must the switch be installed
• Number of questions in the circuit?
• Why are there different types of
switches?
Experiment • Why is a switch installed in a
A1.2: Does the introduction lead to a Yes No circuit?
Switching problem? • Where and why do we encounter
on and off Is the research question Yes No switches in everyday life?
suitable to the following
experiment?
• How many switch positions must
How many questions are Num 6 be available for a light switch?
proposed in the teacher ber
instruction?
How many questions asks the Num 10
student instruction? ber

15

7
 TUM School of Education Technical University of Munich

Agenda
• Evaluation in Educational Systems
• The Experimento Program
• Effective Science Lessons I: Phenomena
• Effective Science Lessons II: Problem Orientation
• Summary and conclusion

16

TUM School of Education Technical University of Munich

Systematic evaluation of input, processes

and output in the field of education
Summary and looking forward
• Communication and commitment about goals and standards
• Findings about strenghts and weaknesses
• Findings for further development
• Material analysis is a starting point. Additional research to:
• application and usage in lessons
• impact of the program on teacher knowledge and teaching
• impact of the program on student performance

Evaluation is a substantial component of

professional self-understanding.

17

8
 TUM School of Education Technical University of Munich

Evaluation team
Thank Chair of School Education

you for
TUM School of Education
Technical University of Munich
Prof. Dr. Eva-Maria Lankes

listening Heidi Haslbeck

Didactics of Biology
Ludwig Maximilian University, Munich
Prof. Dr. Birgit Jana Neuhaus
Dr. Lucia Kohlhauf

On behalf of
Siemens Stiftung, Munich
Dr. Natalie von Siemens
Dr. Barbara Filtzinger

TUM School of Education Technical University of Munich

References
• Anders, Y., Hardy, I., Pauen, S., Ramseger, J., Sodian, B. & Steffensky, M. (2013). Scientific studies of the
work of the “Haus der kleinen Forscher” (Little Scientists' House) foundation.
• Breaks, S. (2012). Why I Hate School But Love Education.
https://www.youtube.com/watch?v=y_ZmM7zPLyI
• Dochy, F., Segers, M., Van den Bossche, P., & Gijbels, D. (2003). Effects of problem-based learning: A
meta-analysis. Learning and instruction, 13(5), 533-568.
• Furtak, E. M., Seidel, T., Iverson, H., & Briggs, D. C. (2012). Experimental and quasi-experimental studies of
inquiry-based science teaching a meta-analysis. Review of educational research, 82(3), 300-329.
• Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn?. Educational
psychology review, 16(3), 235-266.
• Hernandez, R., Fernandez, C. & Baptista, M. (2010). Metodología de Investigación (5a ed.). McGraw-Hill.
• Magnusson, S., Krajcik, J., & Borko, H. (1999). Nature, sources, and development of pedagogical content
knowledge for science teaching. In Examining pedagogical content knowledge (pp. 95-132). Springer
Netherlands.
• Morrison, J.A. & Lederman, N. G. (2003). Science teachers’ diagnosis and understanding of students’
preconceptions. Science Education,87(6), 849-867.
• National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting
concepts, and core ideas. National Academies Press.
• Nirenberg, O., Brawerman, J. & Ruiz, V. (2003). Programación y evaluación de proyectos sociales. Editorial
Paidós Mexicana.
• Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, and applications. (Chapter 19). San
Francisco: Jossey-Bass.

19