R&D
Measuring What Matters
Thinkstock/Comstock
AARON M. PALLAS
Although the current
system of teacher
evaluation is an easy
target, designing a
better system is
more complicated
than it appears.
R&D appears in each issue of
Kappan with the assistance of
the Deans’ Alliance, which is
composed of the deans of the
education schools/colleges at
the following universities:
Harvard University, Michigan
State University, Northwestern
University, Stanford University,
Teachers College Columbia
University, University of
California Berkeley, University of
California Los Angeles,
University of Michigan,
University of Pennsylvania, and
University of Wisconsin.
68 Kappan December 2010/January 2011
Few practices in education are subject to as
much ridicule as how we evaluate the perform-
ance of classroom teachers. The evaluation of
teachers is criticized as perfunctory and hap-
hazard, relying on limited information and
subject to the whims of the evaluators. The im-
age of a principal dropping into a classroom,
watching from the back of the class for 10 min-
utes, and then filling out a form rating the
teacher’s performance as satisfactory may seem
like a caricature, but there’s enough truth in
the depiction to cause anyone concerned with
teaching and learning to squirm in discomfort.
When some school districts hand out top rat-
ings to nearly all of their teachers, even when
their evaluations of student achievement aren’t
nearly so positive, something is seriously amiss.
The recognition that teacher evaluation is
broken has given rise to the view — not nec-
essarily true — that almost any alternative
would be better than the status quo. But sev-
eral key issues must be considered when de-
signing an effective teacher evaluation system.
Design evaluation systems to promote clear
purposes. There are many reasons to evaluate
teacher performance, and the features of an
evaluation system may support some of these
better than others. Evaluations can be used to
certify teachers as competent (for example, the
award of tenure) or to select them for oppor-
tunities or rewards (for example, professional
development to address weaknesses or a bonus
based on excellent performance). Evaluations
can also direct the attention of teachers and ad-
ministrators to what the schools deem impor-
tant (for example, raising students’ test scores or
maintaining order in the classroom). And eval-
uations, especially when coupled with low- and
high-stakes rewards and punishments, might
motivate teachers to perform at high levels.
Beyond the consequences of evaluation for
individual teachers, there are arguments that
teacher evaluation can transform the broader
AARON M. PALLAS is a professor of sociology and ed-
ucation at Teachers College, Columbia University, New
York.
kappanmagazine.org
education landscape. A well-designed teacher
evaluation system can signal to the public, and
to young adults considering teaching as an oc-
cupation, that teacher performance is taken se-
riously. In this way, teacher evaluation might
have longer-term effects on who enters teach-
ing, and on the distribution of good teaching
in U.S. schools.
Tensions often exist among the diverse pur-
poses of teacher evaluation, especially when
principals have primary responsibility for eval-
uating teachers. Some systems ask principals to
police teachers’ performance through evalua-
tion so that poor performers get drummed out.
Others ask principals to support teachers and
guide them toward more effective practices.
Principals report discomfort with the “cop vs.
coach” roles thrust on them by teacher evalu-
ation systems, and a recent study of a teacher
evaluation system in Chicago shows that they
may manage this discomfort by inflating their
evaluations of teacher performance to main-
tain the trust and support of teachers (Sartain,
Stoelinga, and Brown 2009). Thus, the certi-
fication and selection functions of teacher eval-
uation may be weakened to support the moti-
vational and directional functions.
In addition, the number of categories used
to describe teacher performance, and the la-
bels associated with them, may differ depend-
ing on the evaluation’s purpose. Certifying
teachers as competent, for example, might re-
quire evaluators simply to differentiate those
who meet some threshold for competence
from those who do not. In other cases, evalu-
ators might need to distinguish teachers who
are highly skilled from those who are compe-
tent.
For some purposes, evaluators might need
to compare or rank teachers’ performances
against one another; for other purposes, they
might need to compare a teacher’s perform-
ance against an absolute standard. In assessing
a teacher’s contribution to student learning, for
example, an evaluator might hold samples of
student work to an absolute standard, then
judge whether each teacher has met the stan-
dard. The Performance Evaluation Program
in Alexandria, Va., used this strategy and had
teachers set annual goals for improving student
achievement (for example, “90% of my stu-
dents will be able to solve inequalities using
multiplication and division”) and used teacher-
constructed tests and other artifacts to meas-
ure progress toward these goals.
In contrast, if teachers are ranked against
one another in their contributions to student
learning, as is commonly done in value-added
assessment systems (Harris 2010), there may
be winners and losers. Washington, D.C.’s
highly touted IMPACT system for teacher
evaluation, for example, classifies 50% of the
teachers in grades four through eight as inef-
fective or minimally effective and 50% as ef-
fective or highly effective, regardless of the ab-
solute level of students’ learning. In contrast,
the program’s approach to classroom observa-
tion does not impose these kinds of constraints,
allowing all teachers to be judged effective or
highly effective. These two features of the IM-
PACT evaluation system address different pur-
poses.
Recognize that teacher evaluation identi-
fies the kind of learning we value. The cur-
rent wave of teacher evaluation systems em-
phasizes two distinct features of teachers’
work: their effect on student learning and their
classroom practices. For example, in the 2010-
11 school year, D.C. IMPACT evaluates and
labels general education teachers in grades four
through eight as ineffective, minimally effec-
tive, effective, or highly effective. This evalu-
ation is based on four factors: individual value-
added student achievement data (50%), class-
room observations based on a teaching and
learning framework (35%), a principal’s assess-
ment of a teacher’s commitment to the school
community (10%), and school-level value-
added achievement data (5%). (Teachers’ final
scores can be adjusted downward if they don’t
meet four standards for core professionalism.)
But teachers who teach grades or subjects for
which individual value-added scores aren’t
available have slightly different criteria, and
different weights: 75% of their overall evalua-
tion is based on classroom observations keyed
to the teaching and learning framework, 10%
is based on teachers’ assessments of student
learning, 10% is based on commitment to the
school community, and 5% is based on school-
level value-added achievement data.
The conventional interpretation of such
differences between weighting schemes is that
they are purely a technical issue of data avail-
ability; if there were annual tests in grades
other than three through eight that could be
used to generate individual value-added scores
for teachers, presumably these data would as-
sume the same importance (that is, 50% of the
overall evaluation). But, this way of thinking
disguises the value judgments that are embed-
ded in the criteria for evaluating teacher per-
formance and the weights assigned to these cri-
teria.
One key but problematic assumption is that
standardized test scores are satisfactorily com-
plete measures of student learning. To be sure,
students’ scores on NCLB-style standardized
tests are reliable, are designed to align with Since novice
state curricular standards, and are moderately teachers’
good predictors of future educational success. performance can
But, some dimensions of student learning for
change quickly,
which teachers are responsible may not easily
be captured by these tests. To rely solely on these evaluators should
tests as measures of teachers’ contributions to observe their
student learning is to implicitly devalue other performance
dimensions of student learning. Rothstein and frequently, perhaps
Jacobsen (2006) found that many stakeholders several times a year.
value critical thinking and problem solving, so-
cial skills and a work ethic, and citizenship and
community responsibility as goals of public ed-
ucation. If we are to hold schools accountable
for such diverse student learning goals, they
should be reflected in the learning measures
used to evaluate teachers.
Sometimes, evaluations implicitly value an
instantaneous effect on student learning. In the
effort to isolate a teacher’s contribution to stu-
dent learning, value-added methods typically
look at the change in students’ achievement
levels from one year to the next. But, a teacher’s
effect on students is not instantaneous, and
measures of that effect may differ after two
years or five years. Brian Jacob and his colleagues
found that even one year after a value-added
effect was measured, only 20% of the original
effect persisted, and after two years, only one-
eighth of the effect persisted (Jacob, Lefgren,
and Sims forthcoming). This means that dif-
ferences in teachers’ effects on student learn-
ing that appear relatively large may decline
sharply over time. If we value teachers’ contri-
butions to student learning in the long run, we
must design evaluations that assign an appro-
priate value to those contributions. But, there
are always trade-offs. If the long-term effects
of a teacher’s contribution to a student’s learn-
ing aren’t observed for five years, the measures
of this long-term influence might not be avail-
kappanmagazine.org V92 N4 Kappan 69
 able when a teacher is evaluated for tenure, of-
ten around the third year of teaching.
To use a particular student assessment or se-
ries of assessments to measure what teachers
contribute to student learning is to implicitly
value what those tests measure — and implic-
itly devalue what they do not. Some assess-
ments are more comprehensive than others in
their coverage of learning objectives; its breadth
Thinkstock/liquidlibrary
is one reason the National Assessment of Ed-
ucational Progress is often called the “gold
standard” of assessments in relation to NCLB-
style state tests. New York, by contrast, has 48
8th-grade mathematics standards, but the state’s
2009 test allowed just seven of those standards
to account for 50% of the total points available.
If we value the contribution that teachers make
to students’ mastery of standards that don’t ap-
pear on an assessment, we shouldn’t design
evaluation systems that give that mastery no
If we are to hold weight. The next generation of assessments
schools accountable will be aligned with the Common Core stan-
for diverse student dards that many states adopted in 2010 and may
learning goals, those thus allow evaluators to assess teachers’ con-
tribution to the learning of most standards.
goals should be
But, it will be several years before they appear.
reflected in the Recognize that teacher evaluation expresses
learning measures what we value as good teaching practice. We
used to evaluate have made great strides toward establishing
teachers. what counts as good teaching, but there is still
much we don’t know about how to measure it.
And though we hope that good teaching will
result in substantial learning, an uncomfort-
able circularity is built into identifying effec-
tive teaching practices based on their associa-
tion with value-added measures of student
learning on tests that assess only a small frac-
tion of the learning that we value. Are some
practices universally good or bad? How much
does good practice depend on the context in
which it is observed? The assessment of teach-
ers’ classroom practices will not wait until we
have comprehensive and validated measures of
good practice. Instead, the state of the art in
assessing teachers’ practices is to rely on class-
room observation protocols.
Even classroom observation protocols, how-
ever, are an expression of which dimensions of
teachers’ work are to be valued. Teachers’ work
includes planning and classroom assessment
skills (among many others), and to rely solely
on classroom observation protocols to repre-
sent teachers’ practices is to implicitly devalue
other dimensions of teachers’ professional
practice. If a feature of teachers’ practices gets
no weight in the evaluation system, the evalu-
70 Kappan December 2010/January 2011 kappanmagazine.org
ation system gives it no value.
The problem of overlooking important fea-
tures of teachers’ practice may be most acute
in secondary schools, where disciplinary
knowledge, and knowing how to teach disci-
plinary concepts, is so central to teachers’
work. As the National Comprehensive Center
for Teacher Quality points out, many of the
frameworks for teaching and learning that
serve as the foundation for classroom observa-
tion rubrics give short shrift to the knowledge
and practices that might make teachers suc-
cessful in teaching trigonometry, or chemistry,
or Shakespeare (Goe, Bell, and Little 2008).
Classroom observation rubrics are often ori-
ented toward the presence or absence of low-
inference, observable student behaviors. For
example, the recent design standards for
teacher evaluation created by The New
Teacher Project suggest rating the percentage
of students who raise their hands when the
teacher poses a question (The New Teacher
Project 2010). It’s not evident that effective
practice in the teaching of complex subject
matter can be measured in ways that don’t rely
on complex inferences. For this reason, some
districts hire experienced secondary school
subject-matter teachers, who presumably have
deep content knowledge, to observe secondary
classroom teachers.
The good news is that, at least at the ele-
mentary level, evidence exists that principals
and other observers can be trained to use class-
room observation protocols reliably, such that
two observers watching the same teaching les-
son can generate similar ratings. A key element
of this training is having observers watch and
rate videotaped lessons for which there is con-
sensus among experts about whether the teach-
ers’ practices are desirable or undesirable, and
providing feedback to the observers that allows
them to align their ratings to the expert judg-
ments (Sartain, Stoelinga, and Brown 2009).
Synchronize data collection with reason-
able beliefs about how quickly teachers’ per-
formance changes. We’re accustomed to using
a single school year as the timeframe for eval-
uating teacher performance. However, re-
searchers, administrators, and experienced
teachers recognize that the performance of
novice teachers can change rapidly over the
first few years, but that after 10 or 15 years of
teaching, additional years of experience don’t
seem to matter as much for outcomes such as
students’ performance on standardized tests.
Because novice teachers’ performance can
change quickly, evaluators should observe their
performance frequently, perhaps several times
a year. Conversely, since the performance of
experienced teachers is more stable, evaluators
should observe them less frequently.
Value-added assessments of teacher per-
formance have shown us that a teacher’s rank-
ing compared to other teachers is not stable
from one year to the next, even among expe-
rienced teachers. Using a single year of student
test scores to evaluate teachers will, therefore,
make it appear as though a teacher’s perform-
ance varies a great deal from one year to the
next. One reason for this is that many teachers
teach a relatively small number of students
each year. Averaging value-added estimates of
teacher performance over several years in-
creases the sample size and makes estimates
more stable, but doing so may sacrifice infor-
mation about a teacher’s current or recent rel-
ative standing.
The problems of designing good teacher
evaluation systems can be conquered. We’ve
made great strides in developing ways to assess
teachers’ contributions to student learning and
to observe the extent to which teachers engage
in classroom practices that are widely recog-
nized as desirable. The challenges I’ve described
aren’t roadblocks to continued improvement.
But, it may be costly, in terms of time and other
resources, to make continuous progress both
in developing measures of what is to be assessed
and in how to produce reliable and valid indi-
cators. Until such improvements are in hand,
we shouldn’t assume that the current genera-
“What do you mean, it’s the wrong kind of right?”
tion of teacher evaluations is more accurate or
precise than is justified by what goes into them,
even when they’re transformed into numbers.
Acknowledging the limits to what we know
about teachers’ performance based on the cur-
rent technologies for measuring their contri-
butions to student learning and for assessing
teachers’ professional practices does not re-
quire us to return to the bad old days of teacher
evaluation. K
We have made great
REFERENCES
strides toward
Goe, Laura, Courtney Bell, and Olivia Little. Approaches establishing what
to Evaluating Teacher Effectiveness: A Research
counts as good
Synthesis. Washington, D.C.: National Comprehensive
teaching, but there is
Center for Teacher Quality, 2008.
still much we don’t
Harris, Douglas N. “Clear Away the Smoke and Mirrors
know about how to
of Value-Added.” Phi Delta Kappan 91, no. 8 (May
2010): 66-69.
measure it.
Jacob, Brian, Lars Lefgren, and David Sims. “The
Persistence of Teacher-Induced Learning Gains.”
Journal of Human Resources, forthcoming.
The New Teacher Project. Teacher Evaluation 2.0: Six
Design Principles. Brooklyn, N.Y.: The New Teacher
Project, 2010.
Rothstein, Richard, and Rebecca Jacobsen. “The
Goals of Education.” Phi Delta Kappan 88, no. 4
(December 2006): 264-272.
Sartain, Lauren, Sara R. Stoelinga, and Eric Brown.
Evaluation of the Excellence in Teaching Pilot: Year One
Report. Chicago, Ill.: Consortium on Chicago School
Research, 2009.
kappanmagazine.org V92 N4 Kappan 71