Journal of Personnel Evaluation in Education 15:1, 31±48, 2001

# 2001 Kluwer Academic Publishers. Manufactured in The Netherlands.

How Valid are the National Board of Professional

Teaching Standards Assessments for Predicting the
Quality of Actual Classroom Teaching and Learning?
Results of Six Mini Case Studies
JONELLE E. POOL*
Gettysburg College

CHAD D. ELLETT
Louisiana State University

SALVATORE SCHIAVONE
Miami-Dade County Public Schools

CHARMAINE CAREY-LEWIS
Miami-Dade County Public Schools

Abstract

Six mini case studies of teachers certi®ed by the National Board of Professional Teaching Standards (NBPTS)
were completed using systematic classroom observations, individual teacher interviews, and semi-structured
individual and focus group interviews with school site administrators and colleague teachers. Classroom teaching
practices of six teachers (2 elementary, 2 middle/Jr. High, 2 High School) in a large southeastern urban district
were assessed by two trained observers, and semi-structured interviews with school-site personnel were
completed by two additional trained researchers. The ®ndings clearly showed considerable variation among these
teachers in the quality of teaching and learning in their daily practices. Two teachers were clearly exemplary, two
were rather average, and two were considered rather ineffective in the quality of classroom teaching and learning.
Interviews with school-site personnel were somewhat mixed in corroborating ®ndings from actual classroom
observations and assessments. Implications of the ®ndings for the validity of NBPTS certi®cation in everyday
practice, the meaning of NBPTS certi®cation, and policy decisions such as performance-based pay supplements
for NBPTS certi®ed teachers are discussed.

As we enter the new millennium, the national emphasis on educational reform and
increased accountability in public education continues. Early on in the new administration,
President George W. Bush proposed voucher systems and national standardized testing as
means to improve educational opportunity and the quality of America's schools. Included
in the continuing push for educational improvement, is concern about teacher supply and
demand, and the quality of America's teachers. As has been previously noted (Cuban,

*All correspondence should be addressed to: Jonelle E. Pool, Gettysburg College, Education Department, 300
North Washington St., Gettysburg, Pennsylvania, 17325.
32 J.E. POOL ET AL.

1990), politically-based calls for educational reform are certainly not new, and reforms,
like harmonic waves, have moved back and forth over the past century in their primary
focus on either student accountability or teacher quality.
A key element of educational reform during the past two decades, both nationally and at
the state level, has been concern for teacher preparation, credentialing, and continued
professional development (Darling-Hammond, 2000). Recent studies have documented
the role of teachers in affecting the quality of teaching and learning in classrooms (Sanders
& Rivers, 1998; Wenglinsky, 2000). Not only is effective teaching at the heart of student
learning, it is also a factor in the success or failure of school reforms. For these reasons, the
stakes are high for students; their learning may be directly enhanced by effective daily
classroom practices. Conversely, the cumulative effects of ineffective classroom practices
may result in irreparable damage to learners and to the perpetuation of learning inequity
(Sanders & Rivers, 1998).
Improving teacher quality has received national attention from a variety of sources
including federal initiatives calling for written examinations for licensing, educational
summits, candidates' campaign platforms, and state proposals designed to encourage
higher standards in the teaching force. Recent arguments have been made that national,
professional certi®cation models for teachers (i.e., the National Board of Professional
Teaching Standards) can move the teaching profession toward a true staged career if
credentialing bodies, local districts, and teachers meet certain conditions (Johnson, 2001).
At the heart of increasing requirements for teacher preparation, licensing, and
certi®cation is the assumption that better preparation and advanced credentialing will
enhance the quality of everyday teaching, and subsequently student learning. While the
argument for advanced credentialing of teachers as a means of improving student learning
is quite logical and rather seductive, recent research (Goldhaber & Brewer, 2000) raises
serious issues about the yield in student learning and achievement that is associated with
traditional avenues to teacher certi®cation when these are compared to non-traditional
certi®cation procedures. Thus, while the tacit assumption has always been that meeting
higher licensing and certi®cation standards will ultimately result in better quality teaching
and enhanced student learning, few empirical studies have been completed to directly test
this assumption.

Purpose

The purpose of this study was to examine variation among the professional practices of a
small sample of teachers recently certi®ed by the National Board of Professional Teaching
Standards (NBPTS) using direct, systematic classroom observations and individual and
focus group interviews. Of particular interest were differences among these teachers
within a variety of teaching and learning contexts, and the implications of these
differences for the job-related validity of the NBPTS assessments. Also of interest was the
extent to which these NBPTS certi®ed teachers were being used as a professional resource
by their individual schools and the larger school district.
NBPTS VALIDITY 33

NBPTS Credentialing of Effective Teachers

The National Board of Professional Teaching Standards (NBPTS) provides one avenue for
designating professional accomplishment in teaching. Established in 1987, the NBPTS
was one of the ®rst organizations to employ a content speci®c, standards-based approach
for identifying highly effective teachers. This independent, nonpro®t, nonpartisan
organization has developed a rigorous teaching assessment centering on the following
®ve core propositions found in a policy statement entitled What Teachers Should Know
and Be Able to Do that outline the National Board's values and beliefs about exemplary
teaching:

* Teachers are committed to students and their learning.

*
Teachers know the subjects they teach and how to teach those subjects to students.
*
Teachers are responsible for managing and monitoring student learning.
* Teachers think systematically about their practice and learn from experience.
* Teachers are members of learning communities.

Before being approved by the NBPTS board of directors, these standards were
developed through a process relying heavily on extensive professional consensus from
teachers and other experts both internally and during a public comment period. At this
time, the NBPTS has developed advanced standards for teachers in 21 certi®cation ®elds
and is in the process of developing advanced standards for teachers structured around
student developmental levels and subject areas in more than 30 certi®cation ®elds
(www.nbpts.org).
The mission statement of the NBPTS articulates the establishment of high and rigorous
standards for what accomplished teachers should know and be able to do, the development
and operation of a national, voluntary system for assessing and certifying teachers who
meet those standards, and the advancement of related education reforms for the purpose of
improving student learning in American schools. The goal is to improve student learning
by strengthening teaching. The National Board Standards were not intended to replace
state licensing for entry into the profession, but to complement state licensing through
establishing advanced standards for experienced teachers.
The assessment processes for certi®cation of NBPTS teachers are designed to capture
the complex nature of teaching by focusing on how teachers work and how they decide on
appropriate courses of action in their classrooms. The products for teaching assessments
include teacher construction of a portfolio analyzing classroom work that taps the
knowledge, skills, dispositions, and professional judgements that distinguish their
practice.
According to the organization's most recent press release, the number of National Board
Certi®ed Teachers has nearly doubled (4,694) since last year's announcement (2,970).
Figures for the year 2000 indicate that the total number of National Board Certi®ed
Teachers is an unprecedented 9,498, and they hail from 49 states and the District of
Columbia. Both public and private sector constituents are offering support to NBPTS
candidates, with incentives ranging from fee supports and salary increases to support for
34 J.E. POOL ET AL.

the National Board and its mission. (www.nbpts.org, 11-30-00). This study examined the
classroom practices of 6 NBPTS certi®ed teachers in a large, urban district in the southeast
that provides a $7,000.00 base salary supplement for successful completion of the National
Board certi®cation process.

Substantiating Effective Teaching and Learning

A Large Comparison Group Study

A recent, large-scale study comparing 65 NBPTS teachers and non-NBPTS teachers'

practices (Bond et al., 2000) provided support for the validity of the NBPTS certi®cation
process for insuring teacher quality. In this study, the practices of participating teachers
were examined using multi-method data sources including observational visits, lesson
plan artifacts, and teacher and student scripted interviews. Results from the Bond et al.
study indicated that NBPTS certi®ed teachers as a group, possessed, to a considerably
greater degree than non-NBPTS certi®ed teachers, attributes of expert teaching. Bond et
al. concluded that NBPTS teachers

* possessed pedagogical content knowledge that is more ¯exibly and innovatively

employed in instruction;
*
were more able to improvise and to alter instruction in response to contextual features
of the classroom situation;
* understood at a deeper level the reasons for individual student success and failure on
any given academic task;
* understood students developmentally and could provide them with appropriate learning
tasks that engaged, challenged, and even intrigued students, but neither bored nor
overwhelmed them;
* anticipated and planned for dif®culties students might encounter with new concepts;
*
more easily improvised when things weren't running smoothly;
* generated accurate hypotheses about the causes of student success and failure; and
* brought a distinct passion to their work (Bond et al., 2000, ix).

These ®ndings provide support for linking the NBPTS certi®cation process with
accomplished teaching and learning practices, and they contribute to justi®cation for
rewarding successful NBPTS teachers with credentials-based pay supplements. It should
be noted however, that little mention was made in the Bond et al. study of within group
differences among the NBPTS teachers.

A Small Qualitative Study

In the current study, the teaching practices of six NBPTS certi®ed teachers were examined
using multiple methods and data sources. Of particular interest were differences among
NBPTS VALIDITY 35

teachers in their practices in a variety of teaching and learning contexts. In this study, the
quality of teaching and learning practices among teachers was described using the
Professional Assessment and Comprehensive Evaluation System (PACES) (Ellett, 2000).
The PACES is a comprehensive, classroom-based observation and assessment system
currently in its 4th pilot year in a large, urban district in the southeast. The PACES is a
``comprehensive, learner-centered, classroom-based assessment system that is designed to
provide teachers, administrators, and other educators with information useful for
improving teaching and learning in classrooms and schools'' (Ellett, 2000). The PACES
was designed to assess far more than fundamental teaching skills re¯ected on the vast
majority of instruments available to evaluate teaching (Ellett, 2000). It represents a set of
integrated teaching and learning concepts extending from comprehensive planning and
re¯ective practice to the active engagement and involvement of learners in the
development of higher order thinking skills.
In addition to identifying seven major domains of learning (Planning for Teaching and
Learning, Managing the Learning Environment, Teacher/Learner Relationships,
Enhancing and Enabling Learning, Enabling Thinking, Classroom-based Assessment of
Learning, and Professional Responsibilities), the PACES includes a variety of subsumed
Teaching and Learning Components, each of which is further operationalized by a set of
assessment indicators. Included in the PACES assessment manual as well, are extensive
explanations and examples to clarify the meaning of the various assessment indicators.
These indicators are the fundamental units for making assessment decisions about the
quality of teaching and learning. A core element of the total PACES process is the
encouragement and enhancement of self-re¯ection and professional growth. At the school
wide level, the total PACES effort in the district is designed to facilitate the development
of a culture that supports a community of learners among a variety of constituents
including teachers, administrators, learners, and parents.
An example from the PACES Teaching and Learning Component, Domain II.A: Time
Management follows to illustrate the organization and content depth of a typical domain:

Component description. Teaching and learning activities reasonably re¯ect allocated

time, begin promptly, proceed ef®ciently with smooth transitions and no undesirable
digressions, and allow for maximum opportunities for learner engagement in
learning. Activity refers to all things teachers and learners do in the classroom.

Assessment indicator. II.A.1 Learning begins promptly.

This indicator focuses on the beginning of the lesson. Learning activities (teaching
methods and learning tasks) should begin with little time spent on organizational
activities such as roll taking and distributing materials and supplies. The issue in this
indicator is not simply whether teaching starts on time, but whether learning begins
promptly. Both teaching and learning should be considered depending upon the context.
For example, learners might begin a learning task while the teacher ef®ciently calls roll
and /or attends to other organizational activities. On the other hand, teaching methods
might begin promptly without engagement of learners in learning tasks.
36 J.E. POOL ET AL.

The total PACES document includes a large number of assessment indicators …n ˆ 106†,
all of which are considered important elements of teacher self-assessment, re¯ection, and
continuous professional growth. The section that follows describes in greater detail the
methodology of the study.

Methodology

Sample

NBPTS teachers. Participants in this study included 6 NBPTS certi®ed teachers (two
teachers from each school level (elementary, middle, and high school) employed in six
schools in a large urban district in the southeast. Teachers taught a variety of subject
matters including science, art, language arts for gifted students and all subjects in the
traditional, self-contained elementary school model. The average number of years of
teaching experience among teachers in this sample was 13.8 years. All teachers had
master's degrees ( four had graduate work beyond the Master's), and all but one teacher
was female. Two were hispanic and four were white. The six participating schools
ranged in size from 725 to 2,300 students; they varied greatly in terms of ethnicity of
student body and percentages of students receiving free and reduced cost lunch, and
academic achievement as measured by the SAT-8.

Description of school personnel interviewees. Thirty-one school personnel provided

perspectives on the practices of the NBPTS teachers studied. At each site, the principal,
assistant principal, teacher union building steward, and colleague teachers were inter-
viewed. Contributing school personnel who were familiar with the NBPTS teacher's
practice were selected for interviews by the principal or the assistant principal at the
school site.
On-site, semi structured interviews were used to collect information from teachers and
school personnel. Additionally, teachers were observed for the full period of a planned
lesson by two members of the research team consisting of a PACES assessor and an
external qualitative researcher. During the teacher observation and interviewing process,
a second interview team consisting of two PACES-trained educators interviewed
school personnel about the NBPTS teacher's practice. Both research teams visited each
school simultaneously to gather participant data. In order to insure honest and forthright
discussions, all participants were assured that their names would remain con®dential.

Research Teams and Study Orientation to the Study

The team of four ®eld researchers met for orientation to the study, preparation and
scheduling of data collection activities, and to select schools representative of the district
that had at least two NBPTS certi®ed teachers. The general purpose of the study was
NBPTS VALIDITY 37

discussed and a questioning framework to guide semi-structured interviews was developed

by the team for use in the study (see Appendix A).
Two, 2-member research teams collected data for the study. Team 1 completed
classroom observations and the interviews of the NBPTS teachers; this team consisted of
an expert PACES assessor and an external, teacher educator/qualitative researcher. Both
members of the ®rst research team held doctorates in Educational Psychology. The second
team (TEAM 2) collected interview attestations about the NBPTS teachers' practices from
school personnel at the school site. Both members of the second research team were staff
members employed by the district and both were highly knowledgeable about and
experienced with PACES.

Data Collection Procedures

Data were collected using direct classroom observations and assessments with the PACES
and individual and focus group interviews. The NBPTS certi®ed teachers were observed
and interviewed by Team 1 researchers and school personnel were interviewed by Team 2
researchers. The purpose of the study was explained and participants were assured that
their responses were con®dential. Questions were not speci®ed in advance and the semi-
structured format developed guided the interview process by creating a conversational
tone.
Observations were completed in each teacher's classroom for the duration of a regularly
scheduled lesson. Teachers were not asked to do any special preparation beyond their
normal classroom duties. Prior to the interview, Team 1 researchers questioned the teacher
about the goals of the lesson to be observed, as well as prior preparation for the lesson, and
notable special needs or learner characteristics of the class. During the classroom
observations, observers took independent notes and regularly recorded scans of teacher
and learner behaviors. After the lesson was completed, Team 1 researchers interviewed the
teacher and clari®ed any questions relating to the observed lesson. Team 2 researchers
interviewed school personnel about the NBPTS teachers' practices and organized their
notes by respondent job roles. All researchers took independent interview notes that were
used later by the teams to synthesize patterns, themes, and discrepancies emerging from
the data.

Data Analysis

Data were veri®ed and triangulated as recommended by Patton (1990) from the various
data sources, describing the teaching and learning practices of the 6 NBPTS certi®ed
teachers. A school level summary was developed from the synthesized data categorized by
data sources including classroom observations, NBPTS teacher interviews, and
attestations about the NBPTS teachers' practices. Attestations were categorized by job
roles and merged to re¯ect administrator comments ( principal and assistant principal), and
teacher comments (union steward and teacher colleagues).
38 J.E. POOL ET AL.

Results

The results section is organized by school level (elementary, middle and secondary), data
sources (observation and teacher interviews), and attestation data. Signi®cant patterns,
themes, and discrepancies are presented in each school level summary. The two observed
teachers within each school level contrasted sharply in terms of professional effectiveness
as de®ned by PACES observations and assessments. To facilitate reporting of results, the
teacher judged as more effective for a given school level will be identi®ed as Teacher 1;
the teacher judged as less effective is identi®ed as Teacher 2. The results presented are not
exhaustive. They are included to illustrate the most obvious differences between the
quality of teaching and learning practices observed, and the comments and themes derived
from analyses of the interview data.

Elementary School Level Summary

NBPTS teacher observations and individual interviews. There was considerable varia-
tion observed between the effectiveness of the elementary teachers' practices as
assessed with the PACES. Teacher 1 displayed a clear understanding of how chosen
teaching strategies and sequencing decisions could enhance learner understanding. As a
result, Teacher 1 asked more higher level questions and was able to pay considerable
attention to every member of the class. The class enthusiastically responded to the
lesson saying, ``we like developing these questions.'' Class interest was veri®ed by an
extremely high rate of engagement in learning during classroom scans.
In contrast, Teacher 2 was tentative about remediating student misconceptions
regarding lesson content, resulting in some dif®culties with management, practice,
questioning and learning equity. Approximately one-third of the class was inattentive to
the lesson. Teacher 2 involved only those students closest to the overhead projector, and
consistent off-task behavior was observed from 3±5 students per classroom scan.
During post-observation interviews, both of these elementary teachers articulated that
personal/professional re¯ection was important to their practices, but only Teacher 1
demonstrated a capacity for self re¯ection by describing examples from her practice,
supporting research ®ndings by Sparks-Langer, et al. (1990) that re¯ective thinking at high
levels is a developmental process.
The interviews further revealed support for professional practices and values displayed
in the classroom and between these teachers' reasons for pursuing NBPTS certi®cation.
Teacher 1 stated that ``students should be learners ®rst'' and expanded that comment to
include the importance of critical re¯ection to personal professional practice. In contrast,
Teacher 2 was motivated to pursue NBPTS certi®cation ``for the prestige and the money.''
Both teachers credited the NBPTS process with professional bene®ts, but Teacher 1 gave
richer and more speci®c examples from practice to illustrate how, what, and why
professional development and growth had been enhanced by the NBPTS process. Teacher
1 approached teaching as a collaborative learning enterprise, while Teacher 2 seemed to
view her personal growth as ®nished by virtue of NBPTS certi®cation, even though a high
NBPTS VALIDITY 39

degree of professional expertise was not evident during the observations of Teacher 2's
classroom.

Attestation interviews. Attestations provided by administrators provided additional

support for patterns observed during classroom observations. Administrators viewed
Teacher 1 as highly competent (``she is the best teacher in the school''), but they recog-
nized that ``National Board Certi®cation does not make the teacher; it only validates
the teacher's worth.'' In contrast, administrators acknowledged Teacher 2's overall con-
tribution to the faculty, but they quali®ed their comments in revealing ways (e.g.,
``there are other teachers in the school who have more to offer than this teacher'' and
this teacher's practices changed to re¯ect the business of educating children after
NBPTS certi®cation was over'').
Teacher colleagues interviewed provided additional support for patterns in the data
established through the classroom observations, NBPTS teacher interviews and
administrator sources. Teacher 1's colleagues mirrored administrators' comments about
the source of teaching effectiveness and further acknowledged that ``excellent teaching
went on at the school'' (i.e., in other teachers' classrooms). In contrast, Teacher 2's
colleagues said that ``other teachers at our school were just as shining as this teacher.'' At
the elementary level, teachers knew each other well and apparently discussed their
practices during cooperative planning. Even though these elementary teachers were
teaching in self-contained environments, most of them were able to speak knowledgeably
about the teaching practices of their colleagues to a greater extent than middle school or
high school teachers.

Middle School Level Summary

NBPTS teacher observations and interviews. The two Middle Level NBPTS teachers
participating in this study sharply contrasted in terms of teaching effectiveness as deter-
mined by teacher interviews and classroom observations and re¯ective thinking on their
practices. Teacher 1 demonstrated no apparent weaknesses and many strengths during
the classroom observations including high levels of content knowledge as evidenced by
rich examples and non examples used in explanations and discussions with learners.
Teacher 1 was a master at informal assessment, requiring learners to monitor their meta-
cognition and to elaborate their thinking (``use your own experiences and try to think
of ways to remember the formula''). Classroom scans revealed 100 per cent student
engagement and a positive classroom climate that af®rmed the learners' worth (``Class,
you have a great memory!'').
Teacher 2's classroom exhibited severe organizational and classroom management
problems. Classroom routines and rules actually prevented students from learning. For
example, four tardy students were asked to stand at attention at the board for the ®rst ten
minutes of class. Management in this classroom was based on the teacher's reaction to
student misbehavior. Teacher 2 continually interrupted and stopped teaching and learning
activities in attempts to desist and redirect inappropriate and disruptive behavior. Most
40 J.E. POOL ET AL.

often learner redirections were threats or put downs with few desirable results (``mind
your own affairs''. . . or . . . ``if you would like to participate, great; if not keep your mouth
shut''). Off-task rates in this classroom were 50±60 per cent as documented by classroom
scans. The vast majority of learners were only minimally engaged in learning tasks.
Several times during the lesson, Teacher 2 gave inaccurate, untimely, and/or vague content
information to students ( for example, ``electrons are the smallest particles known to
man''). In this teacher's class, there was also no attempt to communicate what was
important to learn during the lesson, and no emphasis was given to various elements of the
subject matter. Teacher 2's choices for instructional methods and strategies were not
effective in engaging learners, and they provided little intellectual challenge (e.g., a seat
work activity was described as ``building molecule models like you may have done in
second grade with marshmallows''). There was no emphasis given to engaging learners in
the development of higher order thinking skills. This teacher made no attempt to
accommodate individual differences among learners. In this lesson, all learners did the
same thing, and at the same time.
Discrepancies between the middle level teachers' practices were repeated in the post
observation interviews. Teacher 1 felt ``guilty'' if teaching and learning were not
successful, responding with continuous re¯ections on personal practices (``am I doing this
the best way possible?''). Recognizing the power teachers possess in the classroom,
Teacher 1 discussed how professionals ``should not be on auto pilot; re¯ection is the most
important NBPTS standard,'' and emphasized and gave examples from personal
experience about how ``inquiry and collaboration were essential to professional
growth.'' Teacher 1 discussed the importance of individualizing instruction for second
language learners, and gave several examples to illustrate how this was done.
Three key features of Teacher 1's practice that distinguished this teacher from all other
teachers in the sample were: (1) extensive knowledge of the PACES, (2) considered
thoughts about ®nancial incentives for NBPTS certi®cation, and (3) the utilization of
professional expertise. Teacher 1 framed both the NBPTS standards and the PACES
indicators as complementary, crediting the total PACES effort in the district with
considerable enhancement of professional growth (``PACES taught me more than my
entire Master's program''). Teacher 1 worried about the impact that supplemental salary
awards and other performance-based pay incentives might have on the NBPTS
certi®cation process (``will the huge number of applicants water down the thoroughness
of the assessment process?''). Teacher 1 believed that new, personal expertise developed
through the NBPTS process was utilized outside the school (conducting workshops for
other professionals), but not within the school.
In contrast, Teacher 2's interview provided more support for the disorganization and
ineffectiveness patterns evident during classroom observations. Teacher 2 did not re¯ect
deeply on practice. When queried regarding the choice of activities for the lesson
observed, Teacher 2 ``chose the activity because I used it before and the kids liked it''.
Teacher 2 appeared to excuse students from learning because: the kids were ``middle
school kids,'' the observation occurred after lunch when the students were mostly rowdy,
and the students came from unsupportive environments (``my students are not motivated
by grades so that's a problem''). In direct contrast to Teacher 1, Teacher 2 summarized the
NBPTS VALIDITY 41

bene®ts of NBPTS certi®cation in terms of ``certi®cates, awards, recognition, receptions''

and discussed the new, personal role of NBPTS certi®cation in the building as a mentor for
others seeking NBPTS certi®cation.

Interview attestations. Interviews with Teacher 1's administrators supported ®ndings

from the classroom observations and teacher interviews. They corroborated this
teacher's effectiveness and professionalism. This teacher was described as ``very pro-
fessional, learner-centered, and tremendously motivated and excellent in producing
higher order thinking from students.'' Teacher 1's reputation of excellence extended
to the community as parents ``often requested her to teach their children.'' One col-
league, however, interpreted Teacher 1's motivation for completing the NBPTS
certi®cation process as primarily driven by monetary concerns. Later during this inter-
view, this colleague teacher revealed that she had recently failed to pass the NBPTS
assessment process.
Notable discrepancies in data patterns established from interviews and observations of
Teacher 2 surfaced as the administrator and teacher colleague data were considered. Both
data sources considered Teacher 2 as ``highly effective, strong in higher order thinking
skills and re¯ective, qualities they attributed to the NBPTS certi®cation process. Teacher
2 was viewed by both administrators and teacher colleagues alike as ``responsible,
student-oriented, creative, and engaging.'' These attestations were in sharp contrast to
the actual classroom observation data collected in which 50±60 per cent of learners were
consistently off task from one scan to the next, and classroom and behavior management
problems were rampant. These ®ndings suggest that halo effects within schools can
result from the NBPTS certi®cation process, and these may have in¯uenced other
educators' perceptions of this particular teacher. Of course, colleague perceptions of
these NBPTS certi®ed teachers might also be in¯uenced by a variety of interpersonal
factors as well, such as personal/professional jealousy. What ever might be hypothesized
to explain differences between educator perceptions of this teacher and actual classroom
observations and PACES assessment ®ndings, these differences were rather extreme and
quite obvious.

High School Level Summary

NBPTS teacher observations and interviews. Like those NBPTS teachers observed for
this study at both the elementary and middle school levels, the two high school level
NBPTS certi®ed teachers were markedly different in their personal perspectives and
professional effectiveness. They also differed dramatically in their management of class
time, their expectations for learners, and in their structuring of the learning environment
in ways that enhanced the quality of teaching and learning.
Teacher 1's classroom observations documented questioning strategies, the structuring
of knowledge for learners, and routines for monitoring and providing feedback about
learning. Teacher 1 asked a variety of questions, but also answered them for students. Most
questions were at the knowledge or comprehension level, but some higher order thinking
42 J.E. POOL ET AL.

questions (e.g., analysis, synthesis, evaluation) were also included. In going over answers
to a recent test, learners were required by Teacher 1 to elaborate their answers in class
(``think back about thisÐwhy is color a factor for. . .?''). Scans of learner behavior during
the lesson observed revealed high rates of on-task behavior and engagement in learning.
Only a few (4±5) learners were momentarily off task at points in the lesson and they self-
monitored and redirected themselves, with no action beyond monitoring by the teacher
necessary. Teacher 1 used humor effectively with learners, but also communicated a
seriousness of purpose about the content and its application (``I want a legitimate science
experiment for the assignment''). The most striking features of Teacher 1's observations
were the excellent command of content displayed, the rich use of vocabulary from the
content used in class discussions, the strength of examples and aides (study guides)
generated for clari®cation of major points, and the many interdisciplinary connections that
the teacher (and the learners) made (examples: genetics, statistics, ecology).
Classroom observations of Teacher 2 revealed major de®ciencies in teaching
effectiveness related to time and classroom management, and the structure of knowledge
and content. A measurement activity designed so that students could ``work on their
measurement skills'' was permeated by high levels of off-task behavior, misbehavior such
as laughing and chatting, and little or no learning. The teacher monitored very little
(neither behavior nor learning), provided essentially no feedback to students about
measurements being made, and allowed some of the measurements to be recorded
inaccurately on a wall chart. A large number of students in this classroom essentially did
nothing related to learning for the ®rst 20 minutes of the class period, abused the time
allotted by engaging in off-task and irrelevant behaviors, side conversations and foul
language, pencil sharpening, sleeping, etc. A high off-task rate for the period (70±80 per
cent) was observed during class scans. Teacher 2 provided few redirections for off-task
behavior and failed to manage numerous disruptions and misbehaviors (e.g., one of the
few students actually engaged in the learning task was hit by a paper wad when Teacher 2
was monitoring learning and providing feedback to another student and this misbehavior
was simply ignored by the teacher). During the classroom observation period, several
students from an adjacent classroom entered this teacher's classroom. The teacher
commented that they ``should be in (your) own classroom. . .and should leave.'' However,
these students simply ignored the teacher's comments, remained in the classroom for 40±
50 minutes, participated in no learning activities, and continuously disrupted the work of
other students by engaging in loud side conversations, moving about and bantering with
other students, and making noises to distract other students. The teacher allowed these
students to remain in the classroom and to disrupt learning, and provided no further
redirection for these students to return to their own classrooms.
Students in this class appeared to have learned that nothing academic was expected from
them. For example, the teacher called for overdue homework that only one student had
submitted, and then extended the deadline for turning in a rather simple homework task for
an additional ten days. Teacher 2 gave no content related purpose for the lesson except that
``neatness and precision were important in business.'' Directions for the day's activity
were confusing and lengthy (Teacher 2 spent 25 ‡ minutes explaining what students were
to do). This teacher also completed learners' work rather than requiring learners to
NBPTS VALIDITY 43

complete their own work, lacked enthusiasm for learning and for the subject matter, and
simply complacently accepted whatever happened during the lesson (including
considerable student misbehavior).
The quality of teaching and learning synthesized from the classroom observation data
for these two high school teachers appeared well supported by their interviews. The
individual teacher interview data showed that both teachers strongly believed that the
NBPTS process had strengthened their professional practices, both mentioned student
assessment as a primary area of personal learning and professional development as a result
of the NBPTS process, both were now mentoring colleague teachers seeking NBPTS
certi®cation, and both attested that they were incorporating expanded measures and
procedures for assessments of student learning and achievement. Neither of these two
teachers professed to be knowledgeable about the PACES.
However, large differences existed between these two teachers when they expressed
their reasons for seeking NBPTS certi®cation and when they articulated what they had
learned from the total NBPTS process. Teacher 1 sought NBPTS certi®cation to study
professional practice; paying the fee personally, (``I wanted a self-directed examination of
my pedagogy that I could control''). The certi®cation process assisted understanding the
individual needs of students (``how I should change instruction when a kid isn't `getting
it'?''). For Teacher 1, NBPTS certi®cation had opened professional doors (for example,
this teacher had received a new grant through NASA to support instruction). While
Teacher 1 was not the most re¯ective teacher interviewed, examples of teaching practices
described suggested that this teacher was developing along a dimension of critical self
re¯ection as a means of professional development.
In contrast, Teacher 2 clearly stated that NBPTS certi®cation was a basically ``a
monetary issue'' sought after dropping out of a doctoral program. In response to the
researchers' questions, this teacher discussed strategies and classroom decisions
candidly, but no critical re¯ection relative to professional growth was evident in
verbal responses: (``I learned more about what I was already doing and how to reach
parents of at-risk students''). Additionally, Teacher 2's strategy for accommodating
individual differences was to ``teach for the lowest level and give opportunities for
more,'' which admittedly ``caused personal angst for those students in the middle.'' In
the area of assessment practices, Teacher 2's experience with the NBPTS portfolio
helped ``me understand how to get on paper for my students what I was thinking; now
students know what I expect''.

Interview attestations. The interviews with school administrators, colleague teachers

and union stewards for these two high school teachers generated some interesting, but
contrasting ®ndings. Teacher 1 was viewed as ``highly effective'' and popular among
students ``they clamor for the class.'' The assistant principal noted Teacher 1's ``love
of teaching,'' and high engagement, and professionalism at the school, and ranked
Teacher 1 in ``the top three teachers in this school who would re¯ect PACES excel-
lence.'' Teacher colleagues interviewed concluded that Teacher 1 ``has lots of passion
for teaching and learning.'' In contrast, administrator attestations for Teacher 2 revealed
rather low level concerns about the teacher's behavior (e.g., ``plans more, wants
44 J.E. POOL ET AL.

feedback from students, more in tune with trends in education, work habits and atten-
dance have improved since NBPTS certi®cation''). Comments about Teacher 2 showed
little focus on improvement in the quality of actual classroom practices. A lone attesta-
tion from one teacher colleague credited the NBPTS process for renewing Teacher 2's
interest in teaching, and increasing this teacher's ``professional con®dence.''
Additionally, this teacher colleague validated the quality of Teacher 2's classroom prac-
tices for providing ``resources outside the realm of the norm, utilizing real life
experiences and environmental applications in lesson content.'' It was the opinion of
this respondent that students in Teacher 2's class were ``engaged and excited, and
involved in the wider community [for subject taught]. The rather lengthy (90 minutes)
classroom observations completed in no way corroborated the perspectives and attesta-
tions offered by this teacher about the quality of teaching and learning in Teacher 2's
classroom.

Discussion and Implications

Findings from this set of six mini case studies of the daily practices of NBPTS certi®ed
teachers revealed that considerable variability exists in effective teaching practices and the
quality of teaching and learning across all grade levels. In contrast to the Bond et al. (2000)
study cited earlier, many of the practices observed in this study re¯ected the full range of
teaching and learning quality from expert to novice as depicted by Berliner (1988).
Teachers in this sample judged as rather ineffective by the PACES observations and
assessments were characterized by dif®culties traditionally associated with beginning
(novice) teachers including problems with classroom management, clarity in explaining
expectations for behavior, structuring content, lack of emphasis on higher order thinking,
etc. In contrast, two teachers judged as rather exemplary (expert) far exceeded the current
PACES expectations for the annual evaluation of all teachers in this district. Two teachers
were judged as rather average in the quality of teaching and learning in everyday practice.
While this study did not compare the practices of NBPTS teachers with non-NBPTS
teachers, the ®ndings strongly documented that wide and consistent variability exists in the
daily classroom practices among these six NBPTS certi®ed teachers.
Considered collectively, the responses of these NBPTS teachers indicated that those
who valued the core of the NBPTS certi®cation philosophy and process maintained higher
quality teaching and learning environments in their classrooms than those who did not
articulate these values. These teachers were also more open to professional discussions
about the NBPTS process and learning about the new PACES being piloted in the district.
For the most part, classroom observations of the quality of teaching and learning in these
teachers' classrooms corroborated data from the teacher interviews. In two cases, the
positive comments from administrators and colleague teachers appeared reputational, and
were clearly not corroborated by the classrooms observations. Interestingly, those teachers
who cited monetary gain as the primary reason for seeking NBPTS certi®cation also
demonstrated dif®culties with content structure, classroom management, articulation of
higher order thinking questions, and other elements of effective teaching. While all of the
NBPTS VALIDITY 45

teachers in this sample greatly appreciated the monetary rewards they received from the
district ($7,000.00 addition to base salary) as a result of NBPTS certi®cation, the teachers
judged as exemplary (expert) discounted these awards as a primary motive for their having
pursued NBPTS certi®cation. These two teachers also expressed considerable concern that
®nancial awards associated with NBPTS certi®cation might impact its professional
meaningfulness and ``water down the NBPTS assessment process.'' They also expressed
concern that the large number of teachers predicted to pass NBPTS certi®cation
requirements in the months ahead (2001±2002) would ``professionally devalue'' the
NBPTS certi®cate.
Study participants viewed the NBPTS process as a good ``tool'' for validating
teaching abilities and they cited professional growth and recognition as important
reasons for seeking NBPTS certi®cation. Principals and teachers expressed near
universal support for the worth of the certi®cation process, but data from multiple
sources indicated that it was not the only source for recognized teaching excellence. In
fact, many teachers that were not NBPTS certi®ed were identi®ed as superior by
participants in this sample.
Even though their certi®cation through the NBPTS has validated these teachers as
excellent in their schools (at least reputationally), the expertise of most of these NBPTS
teachers was not being fully utilized in their schools. This ®nding supports results reported
by Bond et al. (2000) that many NBPTS teachers are under-utilized as resources in their
districts, regions, and state education systems. All of the teachers in this study were involved
in mentoring teacher colleagues who were seeking NBPTS certi®cation, but few were used
in other capacities such as mentoring new, beginning teachers or modeling teaching and
learning practices for colleagues (e.g., demonstration lessons, co-teaching). Though greater
use of the collective expertise of NBPTS certi®ed teachers by districts might be
recommended, the results reported here suggest that school districts may well bene®t
from proceeding cautiously in assigning leadership roles to NBPTS certi®ed teachers. In
this study, the data showed that these NBPTS teachers varied widely in the quality of
teaching and learning observed in their classrooms in everyday practice, and in the
attestations about their expertise offered by school administrators and colleague teachers.
All teachers eventually develop a reputation within their schools related to the quality of
their teaching and learning practices. Consistent with prior classroom-based observation
studies using much larger samples of teachers (Ellett et al., 1994), the results of this study
of NBPTS teachers suggests that reputations of teachers do not exactly predict the quality
of their teaching and learning environments. This ®nding was born out through the PACES
observations, and assessments completed, as well as the many comments from those
interviewed that a teacher does not have to be a NBPTS certi®ed teacher to be an effective
teacher. Considered collectively, the ®ndings of this study suggest that NBPTS
certi®cation may be neither a necessary nor a suf®cient condition to guarantee effective
classroom practices.
From a more technical perspective, the results reported here raise a considerable number
of questions about the job-related validity of the NBPTS certi®cate and assessment process
and the degree to which these can either corroborate, or predict, the quality of teaching and
learning in classrooms in everyday practice. It may be that some of the teachers (and
46 J.E. POOL ET AL.

learners) in this study had a bad day. However, the ®ndings show that some of these
NBPTS certi®ed teachers maintained rather ineffective teaching and learning environ-
ments (some for a full 90 minutes), and others maintained exemplary teaching and
learning environments in everyday practice.
The results of this study also call into question performance-based pay supplements
offered by many states and/or districts to teachers who have received NBPTS certi®cation.
These awards are symbolic and attest to a teacher's having demonstrated the ability,
typically on one occasion, to meet currently established NBPTS assessment standards.
However, the extent to which these awards re¯ect the quality of everyday teaching and
learning in NBPTS certi®ed teachers' classrooms is called into question by the ®ndings of
this study. The results described here con®rm that credentials-based pay for NBPTS
certi®cation does not necessarily insure that high quality teaching and learning occur in
classrooms on a daily basis. Of course, neither do advanced degrees nor any other
professional credential guarantee excellence on a daily basis.
This study used a small sample of teachers …n ˆ 6† and case study methodologies to
explore the daily practice of NBPTS teachers. While the sample size is small, the ®ndings
reported are clear enough to warrant further study of NBPTS teachers' practices. Future
studies should include teachers who have not attained NBPTS certi®cation in a single
blind design so that researchers have no prior knowledge of whether a particular teacher is
or is not NBPTS certi®ed. Single lessons were observed in this study. Future studies might
also incorporate multiple observations, over multiple occasions, using multiple observers
to provide reliability checks for decisions about the quality of daily teaching and learning
practices. With these kinds of designs, the validity of inferences made about the quality of
teaching and learning in classrooms would also predictably be improved.
The results of this study showed that the PACES can differentiate, to a high degree, the
quality of teaching and learning among teachers previously identi®ed by the National
Board of Professional Teaching Standards as worthy of national certi®cation. This ®nding
provides support for the validity and use of the PACES as a classroom-based observation
and assessment system. However, this ®nding also raises considerable concern about the
validity of the NBPTS certi®cate relative to the quality of everyday practice in classrooms.
It suggests that the validity of performance-based pay decisions (salary increases) for
NBPTS certi®cation may well be strengthened by supplemental data derived from
assessments designed to more directly re¯ect the quality of everyday teaching and learning
in classrooms of teachers targeted for such awards.
In the current era of increased educational accountability, reform, and professionaliza-
tion of teaching, teacher credentialing and compensation are likely to continue as topics of
considerable discussion and debate. The results reported here suggest that education
policymakers and others should move cautiously in endorsing and investing in credentials-
based pay incentives solely grounded in passing new forms of national assessment.
Assessments like the NBPTS certi®cation process can tell us much about the personal
commitment of selected teachers to strive to obtain professionally endorsed national
credentials that have high symbolic (and economic) value. However, there appear to be no
assurances that attaining these credentials guarantees quality teaching and learning
environments in everyday practice. What seems needed are newer models of credentialing
NBPTS VALIDITY 47

and accountability for teachers that better integrate the roles of state licensing, local
district evaluations, national certi®cation, and continuous professional growth, with career
development paths, options, and incentives.

Appendix A

Semi-Structured Interview Framework for NBPTS Study

* General philosophy of teaching and major principles guiding teaching

* Conceptions about student learning
* The role of planning
*
Professional self-re¯ection
*
Motivation of students
* Assessment practices (formal and informal)
* Higher order thinking skills
* Individual differences
* Multicultural concerns
* Teaching and learning strategies*
* Use of technology
*
Relationships with colleagues*
*
Relationships with administrators
* Relationships with parents*
* Relationships with students*
* Classroom environment and climate
* Learning equity
* Student self-ef®cacy
* Professional development (group vs individual)*
*
Future goals and plans
*
Leadership roles (school site, region, district)*
* Major job facilitators and impediments
*Denotes primary interview focus for school personnel ( principal, assistant principal, Union steward, teacher
colleague) attestations.

References

Berliner, D. (1988, Feb). The Development of Expertise in Pedagogy. Paper presented at the annual meeting of
the American Educational Research Association, Washington, D.C.
Bond, L., Smith, T., Baker, W., & Hattie, J. (2000). The Certi®cation System of the National Board for
Professional Teaching Standards: A Construct and Consequential Validity Study. Center for Educational
Research and Evaluation, University of North Carolina at Greensboro.
Brophy, J. (1983). Classroom Organization and Management. Elementary School Journal, 83(4), 265±285.
Cuban, L. (1990). Reforming Again, Again, and Again. Educational Researcher, 19(1), 3±13.
Darling-Hammond, L. (2000). Solving the Dilemmas of Teacher Supply, Demand, and Standards: How Can
48 J.E. POOL ET AL.

We Ensure a Competent, Caring, and Quali®ed Teacher for Every Child? New York: National Commission on
Teaching and America's Future.
Ellett, C.D. (2000). Professional Assessment and Comprehensive Evaluation System: PACES. Baton Rouge,
LA: CDE Research Associates, Inc..
Ellett, C.D., Loup, K.S., Evans, L., Chauvin, S.W., & Naik, N. (1994). A Study of Teacher Nominations of
Superior Colleagues: Implications for Teacher Evaluation Programs and the Construct-Validity of Classroom-
Based Assessments of Teaching and Learning. Journal of Personnel Evaluation in Education, 8(1), 7±28.
Goldhaber, D.D., & Brewer, D.J. (2000). Does Teacher Certi®cation Matter? High School Teacher
Certi®cation Status and Student Achievement. Educational Evaluation and Policy Analysis, 22(2), 129±146.
Johnson, S.M. (2001). Can Professional Certi®cation for Teachers Reshape Teaching as a Career? Phi Delta
Kappan, 393±399.
National Board of Professional Teaching Standards. (11-30-00). About the National Board. [on-line]
Available: www.nbpts.org
Patton, M.Q. (1990). Qualitative Evaluation and Research Methods. Thousand Oaks, CA: Sage.
Sanders, W., & Rivers, J. (1998). Cumulative and Residual Effects of Teachers on Future Academic
Achievement. In Education Trust. Thinking K-16: Good Teaching Matters: How Well Quali®ed Teachers Can
Close the Gap.
Sparks-Langer, G., Simmons, J., Pasch, M., Colton, A., & Starko, A. (1990). Re¯ective Pedagogical Thinking:
How Can We Promote it and Measure it? Journal of Teacher Education, 41(4), 23±32.
Wenglinsky, H. (2000). Teaching the Teachers: Different Settings, Different Results. (Policy Information
Center), Princeton, NJ: Educational Testing Service.