Evaluation and Program Planning 33 (2010) 477–486

Contents lists available at ScienceDirect

Evaluation and Program Planning

journal homepage: www.elsevier.com/locate/evalprogplan

Theory underlying a national teacher evaluation program

Sandy Taut *, Verónica Santelices, Carolina Araya, Jorge Manzi
Pontificia Universidad Católica de Chile, Centro de Medición MIDE UC, Santiago, Chile

A R T I C L E I N F O A B S T R A C T

Article history: The paper describes a study conducted to explicate the multiple theories underlying Chile’s national
Received 11 May 2009 teacher evaluation program. These theories will serve as the basis for evaluating the intended
Received in revised form 2 November 2009 consequences of this evaluation system, while not losing sight of emerging unintended consequences.
Accepted 17 January 2010
We first analyzed legal and policy documents and then interviewed fourteen representatives of the four
stakeholder groups involved in the program’s design and implementation, in order to gain insight into
Keywords: their respective conceptions of the program’s functioning and intended effects. The results show that, as
Teacher evaluation
to be expected and despite the long and difficult negotiation process that preceded implementation of
Performance assessment system
Merit pay
this program, multiple political stakeholders still view the program’s intended effects differently.
Professional development However, there was substantial overlap regarding a number of intended effects, such as building the
Program theory capacity of, and triggering change in, teachers with shortcomings, and informing the selection of new
teachers and facilitating the exit of unsatisfactory teachers from the system. It was difficult to get
interviewees to talk about how exactly these intended effects are supposed to be achieved. The paper
draws conclusions regarding theory elaboration process involving multiple stakeholders in a highly
political context.
ß 2010 Elsevier Ltd. All rights reserved.

1. Introduction different levels of the educational system (local, school, individu-

Educational evaluation, assessment and accountability systems
are considered increasingly important in many countries in an
effort to increase educational achievement, quality and equality
(Greaney & Kellaghan, 2008). Teachers are attributed a key role in
these efforts and there is an on-going international debate on how
best to attract, develop and retain effective teachers (see OECD,
2005; Barber & Mourshed, 2007). The Chilean national teacher
evaluation system (NTES) has received international attention for
its explicit, standards-based nature, technical sophistication,
formative purpose and high-stakes consequences, including merit
pay (OECD, 2004, 2005; Manzi, 2009).
In the context of a larger study of the NTES’ intended and
unintended consequences, this paper describes this evaluation
system’s underlying theory, which is supposed to delineate the
assumed causal connections between program components or
activities on the one hand, and expected program effects or
consequences on the other hand. Our role was to help program
designers and implementers formulate their underlying program
theories regarding the NTES, in reaction to what was portrayed in
policy and legal documents. We explored this perceived underly-
ing theory of the NTES by focusing on its intended effects at
* Corresponding author. Tel.: +56 2 354 5302; fax: +56 2 552 2563.
E-mail address: staut@ucla.edu (S. Taut).
al), particularly highlighting two main components of the NTES: an
incentives program for high-achieving teachers (Asignación Vari-
able de Desempeño Individual, or AVDI), and a mandatory
professional development program for low-achieving teachers
(Planes de Superación Professional, or PSP). Supposed connections
between program activities and expected outcomes are known in
the program evaluation literature as the program theory, or theory
of change behind a program or policy (see Weiss, 1972; Bickman,
1987; Patton, 2008).
Since the NTES resulted from a long and difficult negotiation
process between three social groups, the study of this program’s
theory required the exploration of the conceptions held by three
stakeholder groups: (1) the Chilean Education Ministry, repre-
sented by professionals from the Centro de Perfeccionamiento,
Experimentación e Investigación Educacional [Center for In-Service
Teacher Training, Experimentation and Educational Research]
CPEIP; (2) the Colegio de Profesores [Chile’s Teacher Union]; and (3)
the Asociación de Municipalidades [Association of Local Authorities
or, Municipalities Association]. Finally, since its inception profes-
sionals from the Centro de Medición [Measurement Center] MIDE
UC of the Pontificia Universidad Católica de Chile [Catholic
University of Chile] are in charge of implementing the program.
Since their views have also had direct impact on the specific
methodology and processes used to carry out the NTES, they have
been included in this study as a (4) fourth stakeholder group. We
examined the program theory of these four stakeholder groups

0149-7189/$ – see front matter ß 2010 Elsevier Ltd. All rights reserved.
doi:10.1016/j.evalprogplan.2010.01.002
478 S. Taut et al. / Evaluation and Program Planning 33 (2010) 477–486
through interviews with a number of its current and past
representatives in leadership positions.
The work we report on in this paper is part of a larger study on
the consequences (or consequential validity) of the national
teacher evaluation system. In this larger research effort explicating
the program theory was only the first step. The second step is to
explore the program theory empirically, particularly the intended
and unintended consequences, through data collected from
evaluated teachers, school directors, and local authorities.
However, this paper only deals with the first part, the bringing
to light of the program’s underlying theory, and the second part
will be reported on elsewhere.
First, the paper provides a brief review of the literature related
to teacher accountability in the context of the introduction of the
NTES in Chile, as well as a review of program theory literature as
relevant to our study. The paper then states the research question,
describes the methodological details, and communicates the
findings of the study. Finally, we offer a discussion of these
findings and reflect on the conclusions and lessons learnt.
2. Literature review
2.1. Teacher accountability in the context of introducing the NTES in
Chile
A discussion of teacher accountability in Chile must start by
briefly outlining the history of this profession in Chile. While
during the democratic governments of the 1960s, the status of
teachers had improved, this changed dramatically during the
Military government (1973–1990). In its first five years of
existence, the Teacher Union was dismantled and a considerable
number of teachers judged to be political opponents were forced to
leave their jobs, forced in to exile, or worse, imprisoned or made to
disappear. In the course of decentralization and privatization of
education at the beginning of the 1980s, municipalities were given
the right to appoint and dismiss teachers, while teacher salaries
were not adjusted in line with other public sector professionals. In
addition, teacher education was demoted to a non-university
profession (Avalos, 2004).
The new democratic government taking power in 1990 gave
first priority to improving the working conditions and salaries of
teachers. Among its first actions was to pass a new law regulating
teachers’ contractual conditions, known as Estatuto Docente. This
law, in place to this day—although with various modifications,
allowed municipalities to hire teachers following due process but
not to fire tenured ones. In general, the law strengthened teaching
as a profession by granting the right to appropriate initial and in-
service education, adequate working conditions and improved
salaries. Between 1991 and 2001, teacher salaries improved by
141%, more than salary increases in the economy at large or in the
public sector (OECD, 2004). The law also contained a clause
regarding teacher evaluation, as a way for municipalities to remove
seriously underperforming teachers. The Teacher Union opposed
this clause, and in practice it was rarely if never enforced (Avalos,
2004).
From the mid-1990s there was growing concern about the
insufficient student achievement results of Chile’s public educa-
tion sector, as shown both by results of its national student testing
system (SIMCE) and international comparative assessments.
Different parts of society, especially the conservative opposition
and the municipalities, criticized the government for not acting
more forcefully to improve teacher performance in this context, for
example, by enforcing the teacher evaluation clause in the Estatuto
Docente. The Teacher Union decided to tackle the issue and
entered negotiations, initially only with the Ministry of Education,
to design an acceptable system. In 2000, the Municipalities
Association joined the discussions. The negotiation process
involved an international seminar on teacher evaluation present-
ing experiences in Cuba, U.K. and USA, as well as visits abroad for a
group of involved stakeholders. This led to a better understanding
of the complexities of installing a teacher evaluation system that
would satisfy different stakeholder expectations. An important
step in reaching an agreement was the publication of a set of
standards describing competent teaching, based on Danielson’s
Framework for Teaching (1996). Finally, a draft law was sent out for
consultation and was approved by nearly 80% of those teachers
responding to the consultation (which amounted to N = 65,846).
The evaluation law was passed by Parliament on August 14, 2004—
some seven years after negotations had begun (see OECD, 2004;
Avalos & Assael, 2006; Assael & Pavez, 2008).
The consensus reflects compromises from all stakeholder
groups. The system is not purely formative, as the Teacher Union
had wanted. It contains the option to remove unsatisfactory
teachers from their positions. But the system does not connect
teacher performance to student learning, something that conser-
vative groups had urged and which was strongly opposed by the
Teacher Union. The system grants incentives to high-performing
teachers but does not connect these incentives to a professional
career ladder and other existing incentives schemes, as the union
had demanded but which the Ministry resisted. The agreement
reflects a compromise between two perspectives on teacher
performance that co-exist not only in Chile but at international
level: on the one hand, there are those arguing for accountability
and control, as well as incentives, as a way to bring about
improvements in teacher performance; on the other hand, there
are those who call for trust in the professional identity and support
for the self-improvement motivation of teachers as public servants
(see OECD, 2005). In the view of Cox (2008), in the case of Chile this
tension has been resolved so far more in favor of pressure or
control than in favor of support or trust.1
2.2. Program theory
The concept of program theory has been present in the
evaluation literature since the 1970s (Weiss, 1972) but it was in
the 1980s when Chen and Rossi (1983, 1987) formally introduced
the notion of program theory evaluation. Although there is an on-
going academic debate around the definition of program theory
and program theory evaluation (Davidson, 2005), a common
definition is the one provided by Bickman (1987) who defines
program theory as ‘‘the construction of a plausible and sensible
model of how a program is supposed to work.’’
Theory-driven or theory-based evaluation comes as a response
to evaluation approaches that over-emphasized the role of
methods and aims to provide information that can be used to
improve program design and implementation (Chen, 1990). It
advocates the use of multiple methods and highlights the need for
a theory that guides the work of evaluators by helping them
prioritize evaluation questions and methodological approaches.
That theory can come from formal social science knowledge and
practitioners’ experience (Chen, 1990; Donaldson, 2007). Accord-
ing to Rossi, Freeman, and Lipsey (1999), the aim of reconstructing
program theory is to describe the ‘‘program as intended’’ and its
rationale, in particular, how the program is supposed to bring
about the intended outcomes.
1
It is interesting to note that in a recent high-level commission report on
education quality, invited by the Chilean President, these two positions come out
clearly with regard to the discussion of the Estatuto Docente and the related
performance evaluation. They are presented in the report as ‘‘Position A’’ and
‘‘Position B’’, in parallel existence and seemingly unreconcilable (Consejo Asesor
Presidencial para la Calidad de la Educación, 2006).
 S. Taut et al. / Evaluation and Program Planning 33 (2010) 477–486
Reconstructing program theory is necessary because it is
frequently implicit or unsystematic. As Weiss (1973) says:
‘‘. . . the goal of social programs are often global, diffuse, diverse
and inconsistent, vary over stakeholders and may have little to
do with program functioning. One reason for this sorry state is
that it often requires coalition support to secure adoption of a
program. Holders of diverse values and different interests have
to be won over, and therefore a host of realistic and unrealistic
promises are made in the guise of program goals (pp.180–181).’’
The relationship between evaluation and politics identified by
Weiss is especially important to the work presented in this paper.
In Weiss’ view, stakeholders differ in their assumptions and tacit
understandings, and therefore a program may have more than one
underlying theory. All these theories need to be brought to light in
order to reach a consensus on which deserve to be tested in an
evaluation (Stame, 2004). Leeuw (2003) suggests that rebuilding
program theory becomes more important the larger the assumed
impact of a program, the more money involved in the program, and
the larger the risks involved. Risks refer to both expected and
unexpected consequences.
Establishing the underlying program theory requires evalua-
tors to familiarize themselves with the program’s goals and
implementation. According to Donaldson (2007) and Rossi et al.
(1999), program theory must be developed in an iterative process
that involves evaluators on the one side and program staff and
stakeholders on the other side. Rossi et al. (1999, p. 164) call
program staff and stakeholders ‘‘the most important sources of
information’’ in this context. Theory articulation process can be
completed during the program design stage (which would be the
ideal case), but in reality often happens later in the program cycle
when the need for an evaluation arises. Ideally, there would be a
first draft developed based on program material and written
descriptions of the program. This first draft would be reviewed
and discussed with stakeholders and its plausibility checked and
agreed on. Then the evaluators would look more carefully at the
connections between the different program elements and
present a revised program theory to stakeholders in order to
get their final approval. Such interaction with the program staff
and stakeholders brings the additional benefit of increased
evaluation use, especially process use (Patton, 2008). In
summary, methods to be used in the process of reconstructing
program theory include the review of program documents,
interviews and meetings with stakeholders and program staff,
observation of program activities, and review of social science
research (Donaldson, 2007; Ehren, Leeuw, & Scheerens, 2005;
Leeuw, 2003; Rossi et al., 1999).
Some authors differentiate two components of a program
theory: (i) one that refers to the implementation of processes
aimed at providing the intended services to the target population
and (ii) another that refers to the hypothesized causal links and
impact of the intervention. Rossi et al. (1999) and Donaldson
(2007), for example, refer to these two components as the program
process theory and the program impact theory, respectively. More
specifically, Rossi et al. (1999, pp. 167–168) define program impact
theory as ‘‘the goals and objectives that describe the outcome of
the change process the program aims to bring about in social
conditions’’ as well as ‘‘any intermediate objectives that represent
steps along the pathway leading from program services, on one
end, to improved social conditions that are the program’s ultimate
goal, at the other.’’
A useful tool often mentioned for helping to reconstruct a
program theory is the logic model (Frechtling, 2007). A logic model
characterizes an intervention by defining its components and
making the connections between them explicit. The following are
479
the four basic components: (a) inputs: resources brought to a
project, (b) activities: actions undertaken by the project, (c)
outputs: immediate results of the actions, such as events,
participation, or products, and (d) outcomes: desired accomplish-
ments or change (Frechtling, 2007; McLaughlin & Jordan, 1999).
What turns a logic model into a program theory is adding detailed
change mechanisms that explain how and why the program will
achieve the desired outcomes (Patton, 2008, p. 336).
Typically, program theories are depicted as charts or graphical
displays (Rossi et al., 1999, p. 172). Examples of diagrams
illustrating program impact theories, as presented in the literature,
usually resemble simple flowcharts using boxes to contain the
program as input on the left side, mediator variables or
intermediate outcomes in the middle, and long-term intended
outcomes or impacts on the right side, with arrows connecting
them (see Rossi et al., 1999, p. 103). Donaldson (2007, p. 26) talks
about the ‘‘variable oriented approach’’ as the most widely used
approach to pictorially represent program impact theories,
including direct effects, mediator relationships, and moderator
relationships. Donaldson (2007) promotes the mediator model,
which in practice corresponds to input ! intermediate out-
comes ! long-term outcomes models, as shown in examples of
such diagrams he replicated from projects he has been involved in
(see pp. 62 and 88).
There is an ongoing debate on the importance of conducting an
assessment of the quality of program theory on the part of the
evaluator. While some authors (e.g., Rossi, Chen) argue in favor of
the normative role evaluators should play, others (e.g., Patton)
stress the participatory nature of the process and the importance of
having stakeholders come to their own conclusions about program
feasibility and coherence based on the collaborative dynamic of
rebuilding the program’s theory.
Program theory is mentioned as part of guidelines on how to
examine the consequential validity of assessment and account-
ability systems (Forte Fast & Hebbler, with ASR-CAS Joint Study
Group on Validity in Accountability Systems, 2004) and as a basis
for evaluating social programs in general (Bickman, 1987; Carvalho
& White, 2004; Chen, 1990; Donaldson, 2007). There are examples
of studies explicating theories underlying social policies like the
school inspection policy in the Netherlands (Ehren et al., 2005) and
regarding similar teacher assessment systems, for example the
accreditation program by the National Board for Professional
Teaching Standards in the United States (National Research
Council, 2008, p. 31).
3. Research objective
The research objective guiding this study is to make explicit
the expected consequences behind the NTES and the mechan-
isms by which the program would generate these expected
consequences. In Rossi et al.’s (1999) and Donaldson’s (2007)
lingo, we focused on the NTES’ program impact theory.
Specifically, we concentrated on the two main aspects of the
NTES: an incentives program for high-achieving teachers (AVDI),
and a mandatory professional development program for low-
achieving teachers (PSP).
4. Methodology
The research question stated above was explored using the
following methodological approach: First, we reviewed legal and
policy documents to develop an initial graphical representation of
the program’s underlying theory, according to these official
published sources. Then, we reconstructed empirically theory
held by the four stakeholder groups who participated in the
teacher evaluation system’s planning, design, and implementation.
480 S. Taut et al. / Evaluation and Program Planning 33 (2010) 477–486
Fig. 1. NTES’ theory as portrayed in legal and policy documents.
We analyzed these perspectives regarding their differences and
commonalities, trying to arrive at a consolidated version of the
program’s theory.
In order to reconstruct the stakeholders’ theories we con-
ducted a total of fourteen interviews, three to four per stakeholder
group. Our interviewees held leadership positions in the institu-
tions involved in the design of the NTES. For example, in the
Ministry of Education we interviewed the director of the CPEIP, as
well as the person directly responsible for coordinating much of
the process that conceived of and introduced the NTES in Chile. In
the Teacher Union, we interviewed the president as well as two
technical advisors in NTES matters. We also spoke with the
president of the Municipalities Association and his technical staff,
some of which had directly participated in the NTES negotiation
process.
We followed a semi-structured protocol that focused on
exploring the expected consequences that members representing
each stakeholder group associated with the NTES. The protocol also
investigated the mechanisms by which the program would
generate such changes. A diagram of the NTES’s underlying theory,
as described by legal and policy papers, was presented to each
interviewee in order to facilitate their analysis of the program (see
Fig. 1). We discuss this initial portrayal of the program in the
following section.
Fig. 2. NTES’ underlying theory as discussed by CPEIP (Ministry of Education).
Fig. 3. NTES’ underlying theory as discussed by Teacher Union.
Fig. 4. NTES’ underlying theory as discussed by Municipalities Association.
4.1. NTES’ underlying theory as explicated in official documents
Since 2005 the evaluation is mandatory and is the basis for
rewarding and sanctioning about 71,000 teachers working in the
Chilean public education sector. The evaluation distinguishes
between ‘‘outstanding’’, ‘‘competent’’, ‘‘basic’’, and ‘‘unsatisfacto-
ry’’ performance. Performance standards guiding the evaluation
have been defined, officially endorsed, published and widely
disseminated as the ‘‘Marco Para la Buena Enseñanza [Guidelines
for Good Teaching]’’ (Ministry of Education, 2004). Evaluation
methods include a portfolio comprising a written part and a
videotaped lesson, a supervisor questionnaire, a peer interview,
and a self-assessment questionnaire. The evaluated teachers
receive a descriptive report detailing their results for the different
portfolio dimensions, describing strengths and weaknesses, and
including their final score. The school principal and the head of the
municipal education department also receive reports providing the
final performance categories of the teachers evaluated in that
Fig. 5. NTES’ underlying theory as discussed by implementers (MIDE UC).
 S. Taut et al. / Evaluation and Program Planning 33 (2010) 477–486
school or municipality, and the average results for all those
teachers by portfolio dimension. While thus a considerable
number of key stakeholders receive the individual teacher
evaluation results, one can say that they are not public and not
used for ranking purposes. For a full description of the program see
Manzi, Preiss, González, Flotts, and Sun (2008).
Legal and policy documents (e.g., Law 19.933, 2004; Law
19.961, 2004; Decreto No. 192, 2004; Ministry of Education, n.d.-a;
Ministry of Education, n.d.-b) describe the National Teacher
Evaluation System as a program that combines monetary
incentives to motivate teachers to show good performance with
formative feedback and institutionalized support opportunities for
those in need of improvement. As the laws and regulations reflect,
the result of the evaluation has high-stakes consequences for
individual teachers: high-performing teachers are eligible for an
increase in salary, low-performing teachers are subject to
mandatory professional development, and – if repeatedly evaluat-
ed ‘‘unsatisfactory’’ – loss of employment.
The incentives program (AVDI) is accessible to teachers
evaluated as ‘‘outstanding’’ or ‘‘competent’’ and involves passing
a subject and pedagogical knowledge test following the teacher
evaluation process. The results in this test determine the percent
salary increase (0%, 5%, 15% or 25%) the teacher receives until it is
time for his or her re-evaluation (after 4 years).
Teachers evaluated as ‘‘basic’’ and ‘‘unsatisfactory’’ are required
to attend professional development courses (PSP), which are
supposed to provide them with an opportunity to learn and
improve their practices. The PSP are expected to be more effective
than traditional development courses since their designers would
take into consideration the performance weaknesses identified
through the evaluation process. Local managers of school systems
would also be able to use the information provided by NTES’
reports in hiring and firing decisions.
In the long run NTES is expected to help professionalize the
teaching career, to improve student learning and to link teachers’
salaries to their actual performance. Fig. 1 shows the full program
schematically as described in the legal and policy documents we
consulted.
4.2. Data collection
The representatives from each of the stakeholder groups were
contacted in order to schedule a personal interview. In addition to
the names originally considered by the research group, each
interviewee was asked to provide the name and position of
additional informants they considered key for the study. The goal
was to conduct a minimum of three interviews by stakeholder
group. Each interview was approximately one hour long and
conducted by a team of two researchers. The two researchers
alternated their role as moderator and observer who also took
notes. The interviews were tape-recorded for later transcription
and analysis.
4.3. Data analysis
Each interview transcript was analyzed in a data analysis
matrix especially designed for this purpose. The matrix dimensions
reflected the study’s goals and interview questions. After coding
the contents of each interview using our matrix, we then
summarized each matrix in the form of a diagram. The diagrams
served as a one-page graphical display of each interviewee’s
personal vision of the consequences and mechanisms underlying
the NTES. Once the diagrams were completed for all interviewees
from one stakeholder group, a general diagram for the group was
developed. The members of the research group worked to integrate
the different visions within one stakeholder group by building on
481
the considerable overlap that existed between them. This also
meant that we did not include in this consolidated version those
statements that were not shared among the interviewees of one
group. The last step was to look for commonalities and differences
in the views of the stakeholder groups by creating summary tables,
which are presented in the Discussion section.
5. Results
This section presents the program theories of each stakeholder
group. We developed diagrams, which are presented in the
following order: Ministry of Education’s Center for In-Service
Teacher Training, Experimentation and Educational Research
(CPEIP), Teacher Union, Municipalities Association, and imple-
menting institution.
How to read the program theory diagrams
The program theory diagrams have a common structure for all
stakeholders groups. The upper part of the diagram shows the
way interviewees expected NTES to generate change at the
educational system level. The first box of the diagram represents
the NTES as the origin of any theory regarding this program. The
second box was mentioned by three out of four of the groups and
describes a key characteristic of the NTES: it is a teacher
evaluation system based on professional standards. The central
box shows the initial expected consequences stated by the
members of the different stakeholders groups. The last two
boxes of the diagram show the expected longer-term effects of
NTES. The lower part of the diagram represents the way inter-
viewees expected the NTES to generate change at the individual
(teacher) level. The broad arrows connect both parts of the
diagram, attempting to show that the sum of the individual
changes should relate to at least some of the system level effects.
Please keep in mind that the diagram summarizes the original
intentions of the interviewees regarding NTES’ functioning, not
necessarily their opinion about the program’s real or observed
functioning after the onset of implementation. In addition, the
diagram aims to show the institutional position of each of the
four stakeholder groups. That is, they represent an integrated
version of what the three or four interviewees from one partic-
ular stakeholder group had to say, with some individual points
of view being deemphasized in this summary.
5.1. Program theory of the Ministry of Education (CPEIP)
Representatives from the CPEIP expressed the expectation that
the NTES would advance education quality and student achieve-
ment through the improvement of teacher competencies and
teacher performance. The NTES would achieve these long-term
goals by explicitly evaluating performance based on professional
teaching standards stipulated in the Marco para la Buena
Enseñanza (MBE) or ‘‘Framework for Good Teaching’’, a document
developed in collaboration with representatives of the teaching
profession and international experts. The standards-based nature
of the NTES is novel to teachers, since the previous evaluation
system focused exclusively on formal and administrative aspects
of their professsional performance.
To CPEIP’s representatives one of the NTES’ most important
features is its formative nature, that is, the opportunities and
482 S. Taut et al. / Evaluation and Program Planning 33 (2010) 477–486
support it provides to teachers in order for them to improve their
classroom practices. These opportunities are crystallized in the
NTES’ professional development program (PSP). This program,
designed and implemented at local level by municipal education
authorities, would be based on actually diagnosed local needs and
would provide teachers the tools to improve their practices given
the reality of each municipality.
The interviewees expected that the NTES would contribute to
its long-term goals by providing diagnostic information about the
strengths and weaknesses of teachers currently working in schools
today. This information and the associated resources would help
the municipal educational decision-making processes become
more efficient and effective, especially those related to teacher
professional development. Another way of using the diagnostic
would be that the best teachers would help other teachers, as well
as those students who are in most need within a municipality.
Furthermore, it is hoped that the results of the NTES are
increasingly used for hiring decisions, as well as for the non-
renewal of contracts of consistently under-performing teachers.
At the same time, those teachers who receive performance
categorizations of competent and outstanding would receive
monetary and non-monetary incentives. For example, after passing
a subject knowledge test they would obtain an increase in salary
(AVDI). These teachers would also be able to apply to become part of
a network of ‘‘teachers teaching teachers’’ which is seen as a source of
social recognition. Over time, CPEIP’s representatives expect that
these mechanisms would help keep good teachers in the classroom
and would incentivate the sharing of good practices and experiences.
Along these lines, CPEIP’s representatives expected the NTES to
improve teaching practices by fostering a dialogue between peers
focused on overcoming weaknesses and maintaining good
practices. One of the interviewees said that peer collaboration
means ‘‘a different way of doing things . . . it is about looking at
ourselves permanently and finding out what our weaknesses are.’’
At the individual level CPEIP’s representatives expected the
external feedback generated by NTES to encourage teachers to
perform regular, critical self-observations. Teachers are supposed
to engage in reflection based on the information provided in the
individual evaluation reports. This reflection would not only make
them aware of but also feel responsible for their weaknesses and
would result in a commitment to change weak practices. In
addition, just by undergoing the evaluation process teachers would
be encouraged to observe the effectiveness of their practices in the
classroom and assume their professional responsibilities toward
fostering student learning. Finally, NTES’ external feedback would
also be a source of recognition of good performance. Fig. 2
graphically depicts CPEIP’s program theory.
5.2. Teacher Union’s program theory
For the representatives of the Teacher Union the professional
teaching standards are the most central feature of NTES in terms of
bringing about the expected consequences. In their view, the
evaluation should only assess the compliance with the professional
standards and should be disconnected from student performance.
The interviewees from the Teacher Union expect NTES to improve
teachers’ practices in the short and medium term in four important
ways.
First of all, the NTES is seen as a response to the Teacher Union’s
search for a cultural change in which teachers are given a more
prominent social position and would be held in higher esteem by
being perceived as professionals bound by professional standards.
Secondly, the NTES is perceived as an opportunity for teachers
to improve their practices, but regardless of their NTES category.
The Teacher Union representatives firmly believe in the formative
character of the program and are committed to the continuing
professional development of all teachers regardless of whether
they are classified as unsatisfactory, basic, competent or outstand-
ing. The NTES would provide opportunities for teachers to improve
their practices, not only related to the official professional
development program (PSP).
Along these lines, the representatives from the Teacher Union
voiced the expectation that the NTES would generate improve-
ment through peer collaboration. The system would stimulate the
sharing of particularly successful practices, and would allow all
teachers, not only low-performing ones, to develop professionally
through peer dialogue and interactions.
Finally, the NTES was also expected to help the development of
a professional career where teachers’ experience and performance
would be associated to monetary compensation and promotion
opportunities. The Teacher Union considers the NTES as a
fundamental input for improving teachers’ job prospects through
compensation and promotion decisions that are based on teachers’
classroom practices.
At the individual level the representatives from the Teacher
Union believed the NTES would generate a self-observation that
would function throughout the entire evaluation process. This self-
regulation would begin the moment teachers become conscious of
their practices when producing the products required by the
evaluation (portfolio, video-taped lesson, self-evaluation question-
naire). Subsequently, teachers would receive external feedback on
their practices based on the results of the evaluation, elaborating on
their strengths and weaknesses. Both processes together would
motivate teachers’ continuous learning. Fig. 3 graphically displays
how the Teacher Union views the NTES’ program theory.
5.3. Program theory of the Municipalities Association
According to the representatives of the Municipalities Associa-
tion, the NTES would ultimately improve the learning and
achievement of students through the improvement of teaching
practices of public school teachers. The interviewees did not
mention the teaching standards as an important enabling factor as
the other stakeholder groups had done.
These long-term consequences would come about because, first
of all, the NTES provides a diagnostic of teachers’ practices within a
municipality, highlighting their strengths and weaknesses. This
information would allow the local authorities to make better
decisions about the allocation of resources, about the design and
implementation of teacher professional development, the planning
and implementation of educational plans, and would provide
recognition for especially proficient teachers. NTES would serve as
an external validation of the decisions that the local authority
makes regarding hiring, promotion and removal of teachers. The
information NTES provides about the current practices of teachers
is expected to confirm what the local authority already knows
regarding its teachers. Having information from an external source,
however, allows the local authority to make human resource
decisions about teachers without being viewed as arbitrary and/or
politically motivated. This is especially important when letting go
low-performing teachers.
In this context, the NTES would also serve to moderate the rigid
labor conditions that rule teachers in public schools. The
interviewees stated that the special legislation that governs the
labor situation of teachers at public schools (Estatuto Docente)
makes it very hard to remove teachers, even those for whom there
might be evidence about their lack of preparation and profession-
alism. However, none of the interviewees viewed the hiring and
removal of teachers as the only tool to improve education quality
at the municipal level.
At the individual level the interviewees from the Municipalities
Association expected that the NTES would make teachers undergo
 S. Taut et al. / Evaluation and Program Planning 33 (2010) 477–486
two types of processes: (i) the observation of and reflection about
their own practices and (ii) taking responsibility for the
consequences of their professional actions. These two processes
would result in the improvement of weak practices and would
serve as a positive reinforcement for those teaching practices
identified as effective. In Fig. 4 we show the Municipalities
Association’s program theory.
5.4. Implementers’ program theory
The Education Ministry contracted out the implementation of the
NTES to the Measurement Center MIDE UC. In the opinion of MIDE
UC representatives the NTES was expected to improve teaching
practices and student learning in the long run. These long-term
outcomes were to be achieved through the first-ever evaluation of
public school teachers using explicit professional standards of good
teaching. In the opinion of the implementers of NTES, explicitly
stating the expectations of new and current teachers by way of these
professional standards would help improve and modernize teaching
practices of public school teachers. In the words of one of the
interviewees: ‘‘just introducing the standards as part of the
evaluation gives them a very high status.’’
The interviewees suggested that NTES would work as an
information system with multiple short and medium-term
consequences. First of all, NTES would allow linking social
recognition for high-performing teachers to real and concrete
achievements, comparable at national level. The NTES would
identify good teachers and generate incentives associated to good
performance in the classroom.
In second place, the interviewees thought NTES would create an
opportunity to support and offer professional development to
teachers who show weaknesses in their practices. This is done
through the Professional Development Plans (PSP). The profes-
sional development would enable low-performing teachers to
improve and eventually to achieve competent or outstanding
performance. NTES would allow addressing teachers’ weaknesses
using solid information and leveraging on local resources.
Regarding the use of NTES’ information at local (municipal) level,
the implementers mentioned that NTES would support the capacity
of municipalities’ education staff to make informed decisions and
thereby improve education quality overall in the municipalities. For
example, it would allow them to make decisions about hiring good
teachers, and allocating them to different municipal schools based
on their quality in relation to school needs.
Lastly, the interviewees mentioned the role that the NTES
would play in reforming the teacher education system by
providing information about the weaknesses of teachers’ practices
and the areas that need most support. This information would
allow redesigning the curricula of teacher education programs and,
in the long run, improve education quality.
At the individual level the implementers thought that NTES
would trigger self-regulation processes through which teachers
Table 1
Expected short-term consequences of NTES at system level as mentioned by the four stakeholder groups.
Expected consequences
(1) offering social reinforcement of good teaching practice
(2) building capacity of teachers with shortcomings
(3) informing selection of new teachers and exit of insatisfactory teachers
(4) informing the reform of teacher education programs
(5) providing a base for peer-to-peer conversations about good practice
(6) improving teachers’ job prospects by providing incentives
(7) changing to a culture where teachers are professionals
(8) diagnosing quality of teaching practice as basis for management decisions
483
would adopt professional standards, would be able to make
significant progress in the practices identified as weak, and would
internally reinforce, and thus maintain, good practices. The
triggering of the individual self-regulation process occurs, in their
view, by participating in the evaluation process, not necessarily
only related to receiving the evaluation results. At the same time,
the interviewees mentioned that the NTES provides useful external
feedback to teachers in the form of individualized evaluation
reports. These reports indicate the results of the evaluation and
highlight individual strengths and weaknesses in teaching
practice. In the eyes of the implementers, this information would
benefit all teachers across the board, regardless of their NTES
classification. Fig. 5 graphically summarizes the implementers’
program theory.
6. Discussion
Taking as a starting point theory diagram that reflected the
existing legal documents regarding NTES, the process of recon-
structing the underlying theory of the teacher evaluation program
resulted in four different diagrams corresponding to each of the
four groups of stakeholders we interviewed: Ministry of Education
staff, Teacher Union representatives, representatives of the
Municipalities Association, and representatives of the institution
entrusted with the implementation of the evaluation system. Each
of the four diagrams contained sometimes similar, sometimes
distinct expected short-term and long-term consequences.
It is interesting to highlight a few characteristics of the different
stakeholder groups’ underlying theories before moving to an
integration of these four theories in order to guide our empirical
study of NTES’ consequences. First of all, it is striking that all groups
except the Municipalities Association highlight the importance of
the professional teaching standards. To these groups the standards
constitute a key attribute of the evaluation system so that this
system can reach its expected consequences. The standards –
mainly because they were elaborated by the Ministry in
consultation with the Teacher Union – give legitimacy to the
evaluation, and they provide a transparent image of what it means
to be a good teacher (Avalos & Assael, 2006). Similarly, it is striking
that in terms of long-term expected effects of the evaluation
system, all groups except the Teacher Union mention improved
student learning and achievement. All four groups agree that the
short-term expected consequences of the evaluation should result
in improved teaching practices, but only the Ministry, the
implementers and the Municipalities Association see as a crucial
next step the improvement of student learning. It is not surprising
that the Teacher Union does not mention this final outcome: they
have long rejected that teachers be held accountable on the basis of
student achievement (Avalos, 2004; Assael & Pavez, 2008).
We examined in more detail the wealth of immediate or
medium-term expected consequences the interviewees shared
with us. Table 1 shows a summary of the four groups’ expected
Stakeholder groups
Implementing Ministry of Teacher Municip.
institution Education Union Assoc.
  – –
   –
  – 
 – – –
–   –
–   –
– –  –
–  – 
484 S. Taut et al. / Evaluation and Program Planning 33 (2010) 477–486
Table 2
Expected short-term consequences of NTES at individual level as mentioned by the four stakeholder groups.
Expected consequences
(1) strengthening adoption of professional standards
(2) triggering change of weak practices
(3) maintaining good practices by triggering internal reinforcement of diagnosed strengths
(4) stimulating teacher learning
(5) making individuals feel accountable for diagnosed weaknesses
consequences of NTES at system level. The table is meant to allow
for an analysis of the overlap that exists among the groups.
The table shows that some stakeholder groups have more
consequences in common than others. For example, implementing
institution and Ministry of Education share three expected
consequences, while the Municipalities Association shares only
two consequences with other groups. There is no overlap at all
between the expected consequences of the Teacher Union and
those mentioned by the Municipalities Association, which is not
surprising given the diverging political viewpoints these two
entities represent (Avalos, 2004; Avalos & Assael, 2006). There are
two expected consequences that were only mentioned by one of
the stakeholder groups, respectively.
When we asked ourselves how the four stakeholder groups’
consolidated program theories would guide our empirical study, it
was clear that it was not possible to examine four different program
theories because of time and budget constraints. Therefore, we
decided to investigate those expected consequences that are shared
among at least two of the four stakeholder groups, in addition to
examining all the assumptions communicated in the legislation and
official documentation regarding NTES, and our own hypotheses as
researchers. In fact, all consequences stated in the policy and legal
documents were reiterated and agreed on by at least two
stakeholder groups. Thanks to the interviews we conducted we
could add two additional intended effects that were not explicitly
stated in the legislation but that proved important and interesting:
peer effects and the use of the information provided by NTES for
educational management decisions.
As a next step, the research team expressed the expected
consequences, as they had been identified through the document
analysis and stakeholder interviews, in terms of more detailed
underlying assumptions or change mechanisms. This would help us
further focus the data collection we were planning in the ensuing
empirical study on NTES’ consequences. The detailed underlying
assumptions we developed can be found in the Appendix.
So far we have only discussed theory the interviewees held
about NTES at the system level. We did the same exercise for the
individual (i.e., evaluated teacher) level. At this level, all four
stakeholder groups agreed that the evaluation process and the
feedback of its results should first trigger self-regulatory and self-
evaluation mechanisms. These internal mechanisms in turn were
said to yield the expected consequences at individual level that are
listed in Table 2.
Following the same approach we used for the system level, we
will empirically examine those expected consequences at individ-
ual level that were mentioned by more than one stakeholder group,
and which also include all the consequences stipulated in the
policy and legal documents. We again derived specific underlying
assumptions we could then examine in our empirical study (see
Appendix A).
7. Conclusions and lessons learnt
Reflecting back on the process of developing the program
theory we have various observations to share. First of all, we were
Stakeholder groups
Implementing Ministry of Teacher Municip.
institution Education Union Assoc.
 – – –
  – 
 – – 
– –  –
–  – –
constantly reminded that this program took shape as a compro-
mise between three political groups after seven years of
negotiation. We were reminded of how Weiss (1973) described
the political reality of social program development and adoption,
and the resulting multitude of visions regarding a given program’s
goals and functioning. Clearly, the difficult negotiation process and
fragile compromise as described in detail by Avalos and Assael
(2006) is reflected in our findings. While individual differences
within a group could often be explained by the interviewees’
professional backgrounds (e.g., teacher versus economist versus
psychologist), we also observed differences between the stake-
holder groups based on their political or institutional affiliations.
However, all the expected consequences we found to be reflected
in the legal documents regarding NTES were present and shared
between at least two groups at a time.
Second, during the interviews we observed that it was
sometimes difficult to distinguish between what the interviewee
himself or herself wished the program theory would have been like
(his or her ‘‘wishful thinking’’ about the program), had he or she
been able to influence the negotiation process more noticeably,
versus his or her description of the program theory as it really
presented itself based on the actually negotiated compromise, and
also without taking into account anecdotal evidence concerning
‘‘real’’ consequences of NTES to date.
Third, during the interviews we strove to obtain details on the
mechanisms of change, i.e., exactly how the interviewees thought
the expected consequences would be reached (e.g., intervening or
moderator variables). In most cases, especially at system level, we
were only able to talk about what were the expected conse-
quences; we were not able to move on to how these were supposed
to be reached. However, we were somewhat successful in getting
at the individual motivation theory underlying the evaluation
system, which seemed easier to think about for the interviewees
(who are mostly psychologists or teachers by training). The
difficulty of getting program designers and implementers to
explicate the precise, detailed mechanisms of change, hypotheses,
or causal links underlying their programs has already been noted in
the evaluation literature (see Weiss, 1973; Patton, 2008; Chen,
1990; Donaldson, 2007). In fact, the examples of program impact
theories published in the literature closely resemble our diagrams
in that they contain the program as input on the left hand side,
short-term or intermediate expected outcomes in the middle, and
long-term consequences or impacts on the right hand side (see
Rossi et al., 1999; Donaldson, 2007). It seems rare that, in working
with stakeholders, complex input–mediator–moderator–out-
comes models of a program can be elaborated. Also, generally
such diagrams need to be easy to grasp and communicate and this
calls for a certain level of simplicity (Donaldson, 2007).
In summary, the diagrams presenting theories underlying the
NTES, although not reflecting the complexity of all the variables at
play, are not unusual for program theories elaborated jointly with
stakeholders. Importantly, as a research team we (1) ensured that
all effects as mentioned in the policy and legal documents were
reflected in the consolidated intended effects tables guiding future
empirical research, and (2) went one step further by inferring more
 S. Taut et al. / Evaluation and Program Planning 33 (2010) 477–486
detailed underlying assumptions regarding the program’s func-
tioning, similar to the approach chosen by the Committee on
Evaluation of Teacher Certification by the National Board for
Professional Teaching Standards (National Research Council, 2008)
(see Appendix A).2
We found it useful to have consulted with stakeholders instead of
only revising existing documents. Not only did it raise additional,
valid expected consequences not explicated in the legislation, it will
also help make more relevant to stakeholders our ensuing empirical
research on intended and unintended consequences of NTES. These
stakeholder groups in general, and some of the interviewees in
particular, will be among the audience of our research findings. They
will be the ones in positions to ask for modifications of NTES in order
to make it more effectively reach its intended effects and to avoid
negative, unintended ones in the future.
Regarding theory elaboration process we chose, while we found
the stakeholder interviews beneficial in elaborating on the results
of the document analysis and further illuminating the program’s
intended consequences, we could have ended by holding a large-
group meeting where each group’s version would be presented and
commented on in terms of differences and commonalities, and
comments by each group’s representatives would be solicited. This
could have been done with NTES’ technical advisory board, which
is comprised of representatives of the four stakeholder groups
involved in our study. This may have brought additional benefits
not only for our own understanding of the program, but also the
positive side effect of triggering new insights for the stakeholders.
It is important to remember, however, that we were dealing with a
highly visible national educational policy. Besides feasibility issues
in arranging for joint meetings with all interviewees, we must
remember that this program exists as a remarkable yet fragile
compromise following difficult negotations between three social
groups. Publicly resurfacing their different views on the program
may also have had detrimental effects on the relative political
stability of the existing agreement.
In explicating NTES’ underlying theory we became aware that
no theory-based or logic modeling approach had been used in its
design—at least not explicitly or documented. Applying program
theory or logic modeling approaches when designing public
policies has its benefits and drawbacks. Adopting such an
approach makes program design more transparent and facilitates
a more effective and efficient program planning and implemen-
tation later on. On the other hand, transparency also makes more
apparent existing disagreements, and it invites public scrutiny
and accountability for intended effects. However, evaluations of
the effectiveness of social programs can overall benefit from
existing program logic models and assumptions, while hopefully
not neglecting possible unintended effects. Policy-makers will
likely need the facilitation skills and expertise of external
consultants to help them develop program theories or logic
models. Our experience and the literature coincide that it is a
difficult and time-consuming task, especially when there are no
existing social science theories that can inform this process
(Stufflebeam & Shinkfield, 2007). On the other hand, working
jointly with evaluators on this task would provide a way to link
program design with program evaluation and enhance the
evaluability of social programs and policies (Wholey, Hatry &
Newcomer, 1994).
2
A pending task to be addressed is an assessment of the quality of the
assumptions we derived, which should be based on existing literature on the topic
(see Rossi, Freeman & Lipsey, 1999). We plan on performing such a literature
review, for example, regarding the effects of monetary incentives programs for
teachers; effects of teacher professional development programs on teaching
practice and student learning; auto-regulatory and self-evaluative processes based
on performance feedback.
485
Acknowledgments
This study is part of a larger research project supported by the
Fondo Nacional de Desarrollo Cientı´fico y Tecnológico (Fondeyct, no.
1080135) of the Chilean government. We thank Patricia Thibaut for
her competent research assistance. We also thank four anonymous
reviewers and the editor for their helpful comments.
Appendix A
Examples of specific assumptions underlying NTES at system level:
(1) Offering social reinforcement of good teaching practice:
- NTES information is used by superiors, peers and the public to in fact
provide social reinforcement to good teachers.
- Good teachers perceive the social reinforcement they receive as tied to
the NTES.
- Social reinforcement is perceived as attractive and as improving
job satisfaction.
- Offering social reinforcement to some teachers has no negative
consequences,
e.g. on those teachers not rewarded, or on the work climate in general.
(2) Building capacity of teachers with shortcomings:
- NTES informs the professional development planning efforts that are
designed for teachers by the municipalities (PSP).
- The professional development courses in fact teach relevant knowledge
and skills, tackling those areas where teachers are weakest.
- The professional development courses are taken by those teachers in most
need of improvement.
- Those teachers most in need learn relevant knowledge and skills and
transfer these knowledge and skills to improve their teaching
practice in the classroom.
(3) Informing selection of new teachers and exit of unsatisfactory teachers:
- NTES scores are taken into account in teacher selection processes at
municipal level.
- Teachers with competent and outstanding performance in NTES are in
fact more likely to be hired.
- NTES provides the justification for facilitating the exit from the
classroom of consistently unsatisfactory teachers.
- Teachers with consistently (i.e., three consecutive years of)
unsatisfactory performance in NTES do in fact leave the
teaching profession.
(4) Providing a base for peer-to-peer conversations about good practice:
- NTES results and reports are discussed among peers or in teams of teachers.
- NTES results and reports provide the basis for talking among peers about
what is good practice, and how to achieve it.
(5) Improving teachers’ job prospects by providing incentives:
- Monetary incentives associated with NTES (AVDI) are perceived
by teachers as attractive and positively influence their job satisfaction.
- Monetary incentives associated with NTES in fact motivate teachers
to continue teaching in the classroom.
- Non-monetary incentives are also known, and perceived
as attractive and as improving job commitment.
(6) Using teacher performance diagnostic to target educational
management decisions:
- The reports the NTES provides to local educational decision-makers
are informative and comprehensible so that they can be taken as an
input in the educational decision-making process.
- The NTES is perceived as a credible source of information about
teacher performance.
- Local educational authorities have the know-how and experience to
transform NTES results into local educational
management and policy decisions.
Examples of specific assumptions underlying NTES at individual level:
(1) Triggering change of weak practices:
- Teachers find the information on their weak areas of practice that is
contained in the individual evaluation reports to be clear, trustworthy,
reliable and valid.
- Teachers reflect on the information and acknowledge that they do
have the weaknesses encountered by the evaluation.
- Teachers develop a strong intention to change these weaknesses.
- Teachers believe they are able to change these weaknesses (self-efficacy).
- Teachers develop concrete and useful strategies how to bring about the
necessary changes in their teaching practice.
486 S. Taut et al. / Evaluation and Program Planning 33 (2010) 477–486
Appendix A (Continued )
Examples of specific assumptions underlying NTES at individual level:
- Teachers in fact act upon their intention, successfully implement their
strategies, and finally are successful at changing their teaching practice
(changes in behavior).
- Pointing out weak areas has no substantial negative effects on teachers,
such as negative emotions (anger, depression), diminished job motivation.
(2) Maintaining good practices by triggering internal reinforcement of
diagnosed strengths:
- Teachers find the information on their strong areas of practice that is
contained in the individual evaluation reports to be clear, trustworthy,
reliable and valid.
- Teachers reflect on the information and acknowledge that they do have the
strengths encountered by the evaluation.
- This information triggers pride or a similar feeling of internal reinforcement.
- This internal reinforcement results in the teacher consciously making an
effort to maintain these good practices in the classroom.
References
Assael, J., & Pavez, J. (2008). La construcción e implementación del sistema de
evaluación del desempeño docente chileno: Principales tensiones y desafı́os
[The development and implementation of the Chilean teacher evaluation system:
Key tensions and challenges]. Revista Iberoamericana de Evaluación Educativa, 1(2),
42–55.
Avalos, B. (2004). Teacher regulatory forces and accountability policies in Chile: From
public servants to accountable professionals. Research Papers in Education, 19(1),
67–85.
Avalos, B., & Assael, J. (2006). Moving from resistance to agreement: The case of the
Chilean teacher performance evaluation. International Journal of Educational Re-
search, 45, 254–266.
Barber, M., & Mourshed, M. (2007). How the wold’s best performing school systems
come out on top. Retrieved October 30, 2009 from http://www.mckinsey.com/
locations/UK_Ireland/Publications.aspx#Reports.
Bickman, L. (1987). The functions of program theory. New Directions for Program
Evaluation, 33, 5–18.
Carvalho, S., & White, H. (2004). Theory-based evaluation: The case of social funds.
American Journal of Evaluation, 25(2), 141–160.
Chen, H. T. (1990). Theory-Driven Evaluations. Newbury Park: Sage Publications.
Chen, H. T., & Rossi, P. H. (1983). Evaluating with sense: Theory-drive approach.
Evaluation Review, 7, 283–302.
Chen, H. T., & Rossi, P. H. (1987). Theory-driven approach to validity. Evaluation and
Program Planning, 10, 95–103.
Consejo Asesor Presidencial para la Calidad de la Educación. (2006). Informe final de
consejo asesor presidencial para la calidad de la educación [Final report of the
presidential committee on educational quality]. Retrieved October 20, 2009 from
http://www.consejoeducacion.cl/articulos/Informefinal.pdf.
Cox, C. (2008). Educación en el Bicentenario: dos agendas y calidad de la polı́tica
[Education in the Bicentenario: two agendas and the quality of policies]. Retrieved
October 20, 2009 from http://mt.educarchile.cl/mt/jjbrunner/archives/2008/01/
cristian_cox_ed.html.
Danielson, C. (1996). Enhancing professional practice: A framework for teaching. Alex-
andria, VA: Association for Supervision and Curriculum Development.
Davidson, E. J. (2005). The ‘‘Baggaging’’ of theory-based evaluation. Journal of Multi-
Disciplinary Evaluation, 4, iii–xiii.
Decreto No. 192 of Education, ‘‘Reglamento sobre Evaluación Docente [Regulations
about the Teacher Evaluation]’’. (2004). Retrieved October 14, 2008, from http://
www.docentemas.cl/documentos.php.
Donaldson, S. I. (2007). Program theory-driven evaluation science. Strategies and applica-
tions. New York: Lawrence Erlbaum Associates.
Ehren, M. C. M., Leeuw, F. L., & Scheerens, J. (2005). On the impact of the Dutch
educational supervision act. Analyzing assumptions of primary education. Ameri-
can Journal of Evaluation 26(1).
Forte Fast, E. F., Hebbler, S., & ASR-CAS Joint Study Group on Validity in Accountability
Systems. (2004). A Framework for Examining Validity in State Accountability Systems.
Council of Chief State School Officer.
Frechtling, J. A. (2007). Logic Modeling Methods in Program Evaluation. San Francisco:
Jossey-Bass.
Greaney, V., & Kellaghan, T. (2008). Assessing National Achievement Levels in Education.
Washington, DC: The World Bank.
Law 19.933. (2004). Retrieved October 14, 2008, from www.rmm.cl/usuarios/equiposite/
doc/200501121332150.19933.pdf.
Law 19.961 ‘‘Evaluación Docente [Teacher Evaluation]’’. (2004). Retrieved October 14,
2008, from www.docentemas.cl/docs/Ley_19961.pdf.
Leeuw, F. L. (2003). Reconstructing program theories: Methods available and problems
to be solved. American Journal of Evaluation, 24(1), 5–20.
Manzi, J. (2009). Individual incentives and teacher evaluation: The Chilean case. Unpub-
lished manuscript prepared for the Organisation for Economic Co-operation and
Development (OECD), Paris, France.
Manzi, J., Preiss, D., González, R., Flotts, P., & Sun, Y. (2008). Design and Implementation of
a National Project of Teaching Assessment: The Chilean Experience. Paper presented at
the annual meeting of the American Educational Research Association, March 24–
28, 2008, New York City, USA.
McLaughlin, J. A., & Jordan, G. B. (1999). Logic models: A tool for telling your program’s
performance story. Evaluation and Program Planning, 22, 65–72.
Ministry of Education. (2004). Marco para la Buena Enseñanza [Framework for Good
Teaching]. Santiago, Chile: Ministry of Education of the Chilean Government.
Ministry of Education (n.d.-a). Fortalecimiento de la profesión docente [Strengthening of
the teaching profession]. Retrieved October 14, 2008, from www.rmm.cl/usuarios/
equiposite/doc/200501121331010.politica%20fortprof%20docente.doc.
Ministry of Education (n.d.-b). Polı́ticas Educacionales durante los Gobiernos demo-
cráticos [Educational Policies by the democratic governments]. Retrieved October
14, 2008, from www.rmm.cl/usuarios/equiposite/doc/200501121331270.
polt%20educ.doc.
National Research Council. (2008). Assessing accomplished teaching: Advanced-level
certification programs. Committee on Evaluation of Teacher Certification by the
National Board for Professional Teaching Standards. In: Milton D, Hakel, Judith
Anderson Koenig, Stuart W. Elliott, editors. Board on Testing and Assessment,
Center for Education, Division of Behavioral and Social Sciences and Education.
Washington, DC: The National Academic Press.
Organisation for Economic Co-operation and Development. (2004). Reviews of national
policies for education: Chile. Paris, France: Organisation for Economic Co-operation
and Development, Centre for Co-operation with Non-members.
Organisation for Economic Co-operation and Development. (2005). Teachers Matter:
Attracting, Developing and Retaining Effective Teachers. Paris, France: Organisation
for Economic Co-operation and Development.
Patton, M. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.
Rossi, P., Freeman, H., & Lipsey, M. (1999). Evaluation: A systematic approach. Thousand
Oaks: Sage.
Stame, N. (2004). Theory-based evaluation and types of complexity. Evaluation 10(1).
Stufflebeam, D., & Shinkfield, A. (2007). Evaluation theory, models & applications. San
Francisco, CA: Jossey-Bass.
Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.). Assessing the feasibility and likely
usefulness of evaluation. In: Handbook of Practical Program Evaluation (pp. 15–39).
San Francisco: Josey-Bass.
Weiss, C. H. (1972). Evaluation research: Methods for assessing program effectiveness.
Englewood Cliffs, NJ: Prentice Hall.
Weiss, C. H. (1973). The politics of impact measurement. Policy Studies Journal, 1, 179–
183.
Sandy Taut earned her Ph.D. from the Graduate School of Education at the University of
California Los Angeles (UCLA), with an emphasis on program evaluation. She currently
leads the research unit at the Measurement and Evaluation Center (MIDE UC) of the
Pontificia Universidad Católica de Chile.
Verónica Santelices received her Ph.D. from the Graduate School of Education at the
University of California Berkeley. She currently works as an educational researcher at
the Measurement and Evaluation Center (MIDE UC) of the Pontificia Universidad
Católica de Chile.
Carolina Araya holds a Masters degree in psychology from the Pontificia Universidad
Católica de Chile and works as an educational researcher at the Measurement and
Evaluation Center (MIDE UC) of the Pontificia Universidad Católica de Chile.
Jorge Manzi holds a Ph.D. in psychology from the University of California Los Angeles
(UCLA). He is a full professor at the School of Psychology at Pontificia Universidad
Católica de Chile and the director of the Measurement and Evaluation Center (MIDE
UC) of the Pontificia Universidad Católica de Chile.