Book Reviews 247

Kellaghan, T., Stufflebeam, D.L., & Wingate, L.A. (2003). International handbook of
educational evaluation. Dordrecht: Kluwer, 1040 pp., $699.
DOI: 10.1177/1098214009334509

The International Handbook of Educational Evaluation spans two volumes, each with five
sections, totaling 43 chapters and nearly 1,000 pages of text. The editors say in their introduc-
tion that ‘‘the individual chapters can stand on their own as reference works on a wide array of
topics.’’ Thus, we have adopted the structure of the Handbook to organize our review. We
begin with a brief statement about the contributing authors, then describe each individual
chapter. Our presentation uses the same structure as the Handbook, describing the 1st volume,
perspectives, and its five sections—(1) theory, (2) methodology, (3) utilization, (4) profession,
and (5) contexts—and then the 2nd volume, practice, and its five sections—(6) student
evaluation; (7) personnel evaluation; (8) program/project evaluation; (9) challenges for eva-
luation in schools; and (10) local, national, and international levels of system evaluation.
We conclude with additional commentary on strengths and weaknesses of the Handbook.

Contributing Authors

The Handbook features strong contributing authors, most of whom are publicly recognized
for their contributions to the field and expertise in the areas they were invited to address. In
the 1st volume’s section on theory, each theory is presented by its creator. The authors in the
Methodology section each contributed to the development of the methods they present. The
Utilization section features renowned authors who have made significant conceptual and
empirical contributions to the topic. The authors in the section on the Profession are senior
theorist-practitioners well recognized as leaders in the U.S. evaluation community. The mix
of U.S. and international authors in the Context section offers valuable perspectives on the
complexities of evaluation contexts. The authors in 2nd volume are also experts in their field,
although their names may be less familiar to program and policy evaluators. This is partly
because this 2nd volume on Practice emphasizes student assessment and school-based evalua-
tion in the United States, which are specialized topics about which the broad international
community of evaluators may be unfamiliar.
Across both volumes, a sizeable proportion of the chapters’ authors share the editors’ eva-
luation standpoints, including a preference for the Context Input Process Product (CIPP) eva-
luation model and a decision- and utilization-oriented evaluator role. Many of the authors have
experience working in international contexts. However, only 8 of the 61 authors were affiliates
of institutions outside the Uniteed States at the time of the book’s publication.

Volume One: Perspectives

The five sections of the Perspectives volume take a historical and scholarly look at core
conceptual ideas in evaluation. The focus, perhaps because of the historical lens, is predomi-
nantly United States.
Section 1: Theory presents five chapters that represent select evaluation theories and per-
spectives on the development of evaluation theory. Chapter 1 reviews and critiques the historic
range of evaluation theories as being ‘‘both simpler and more complex than their individual
conclusions: simpler at the meta-level, more complex in detail’’ (p. 26). Chapter 2 is a detailed
description of the CIPP model, along with examples of how CIPP works in practice. Chapter 3

Downloaded from aje.sagepub.com at UNIV OF MONTANA on April 1, 2015

248 American Journal of Evaluation / June 2009

presents the intellectual roots, methods, issues, and justification for responsive evaluation as
grounded in the author’s personal history in educational evaluation. Chapter 4 challenges early
realist approaches to evaluation with constructivist theories of knowing, including alternative
criteria by which the quality and rigor of constructivist evaluations can be judged. Chapter 5
balances practicality and philosophy while discussing strengths and criticisms of core princi-
ples of Deliberative Democratic Evaluation—inclusion, dialogue, and deliberation. The five
pivotal perspectives of this Theory section represent important differences in how to approach
the purpose, design, and conduct of educational evaluation. Yet, some highly influential con-
tributions to the development of evaluation that also influence present practice are omitted,
notably Campbell’s vision of an experimenting society and Cronbach’s and Weiss’ promotion
of evaluation as a fundamentally educative enterprise.
Section 2: Methodology highlights four methodologies associated with educational evalua-
tion. Chapter 6 plumbs randomized field trials (RFTs) in education. Because true RFTs are
rare, the author is able to present much of the available work on RFTs in educational and social
service evaluation. Chapter 7 is a comprehensive introduction to cost-effectiveness, cost-utility,
and cost-benefit analyses, laying out the benefits, limitations, and the constitutive analyses of
each approach in progressive depth. Chapter 8 shares an evaluation methodology for connois-
seurship and educational criticism, focusing on how description, interpretation, evaluation, and
thematics contribute to the reader’s sense of the program. Chapter 9 briefly but broadly presents
the epistemological orientation, methods of data collection, interpretation, and reporting associ-
ated with qualitative methods. Because evaluators use more than these four methodologies,
and often use them in combination, the inclusion of only these four reinforces a historically
problematic qualitative/quantitative schism. Available space limits the depth of these chapters,
but each includes a list of valuable recommended references and additional resources that
strengthen these chapters.
Section 3: Utilization explores select empirical research that has informed evaluation utili-
zation. Chapter 10 summarizes the history of and research on evaluation utilization, then chal-
lenges the field to engage a lens of knowledge utilization to gain further insight into the issue
of evaluation use. Chapter 11 presents a logical warrant for utilization-focused evaluation and
describes challenges and solutions to conducting good utilization-focused evaluation. Chapter
12 takes the novel approach of creating a logic model for practical participatory evaluation,
testing the coherence and validity of the approach against empirical research, to reach the con-
clusion that the logic of participatory evaluation is sound, and credibly enhances instrumental,
educative, and especially process use. Allocating an entire, albeit small, section to the topic of
utilization as opposed to devoting a chapter to it in either the Theory or Profession sections
reflects the editors’ decision- and utilization-focused evaluation stance.
Section 4: Profession contains six chapters that together reflect on the status, ethics, issues,
and future of evaluation as a profession. Chapter 13 reviews the development of multiple sets
of evaluation standards, contrasting two—the Joint Committee Standards with the AEA Guid-
ing Principles—and acknowledging neither has had significant impact on practice. Chapter 14
offers a triptych critique of the ethical landscape of evaluation by considering the ethical
implications of (a) stakeholder involvement, (b) utilization research, and (c) ways professional
standards lay a path to ‘‘occasions for sin’’ (p. 318). Chapter 15 argues for establishing a sys-
tem of qualification to practice evaluation along with a pragmatic assessment of why such
measures have not been implemented. Chapter 16 offers another historical review focused
on ways in which U.S. federal and state governments have influenced the evaluation profes-
sion and vice versa. Chapter 17 looks forward, laying out a heartfelt vision of evaluation
practitioners as a sustainable learning community and an action plan that could enact this
vision. Chapter 18 also considers the future of the field, highlighting enduring issues and

Downloaded from aje.sagepub.com at UNIV OF MONTANA on April 1, 2015

 Book Reviews 249

themes (e.g., qualitative vs. quantitative methods), new developments (e.g., technology), new
challenges (e.g., globalization), and selected criticisms of the field (e.g., shortsighted planning
for the education and training of evaluators).
Section 5: Contexts presents the international perspective in volume one with five chapters
reflecting on globalization, education, and evaluation in the United States, Europe, Africa, and
South America. Chapter 19 surveys globalization through a brief case of evaluation capacity
building across language barriers in non-Western contexts. Chapter 20 analyzes how the struc-
ture of education in the United States influences the conduct of American educational evalua-
tion, which is useful in highlighting the limitations of translating U.S. evaluation ideas to
non-U.S. contexts. Chapter 21 portrays European evaluation influences via an even-handed
review of the historic evolution of new public management evaluation and democratic evalua-
tion. Chapter 22 moves to South America. With vivid cases and historical critique, the author
reveals the political and ideological factions that both enable and hinder, often simultaneously,
evaluation practice in these contexts. Chapter 23 analyzes the history of education, evaluation,
and their intersection in Africa, weaving native student assessment practices and participatory
evaluation with effects of Western and Muslim colonization. The richness of these chapters
makes the absence of contributions on Asian and Middle Eastern contexts appear sharper.

Volume Two: Practice

The five sections of the Handbook’s 2nd volume, Practice, have a decided focus on educa-
tion as K-12 schooling and on educational evaluation as an assessment of the (CIPP-framed)
inputs, processes, and especially products or outcomes of schooling. The five sections of this
volume focus on ‘‘typical objects of [educational] evaluation: students, personnel, programs/
projects, schools, and education systems’’ (p. 3). There is no attention in the Practice volume to
the learning of children and youth in informal educational contexts or to adult learning in
any context. The volume pays only modest attention to the cultural, social, and economic
dimensions of the contexts within which teaching and learning take place.
Section 6: Student Evaluation offers four chapters that could readily be presented in a
handbook on educational measurement. The centrality of measurement to the editors’ concep-
tualization of educational evaluation—‘‘contemporary educational evaluation is rooted in
student assessment and measurement’’ (p. 2)—is reflected in the content and prominent
location of this section within the Practice volume. The four chapters in this section are uneven
in focus, coverage, and especially intended audience. Chapter 24 offers a comprehensive,
high-level evidentiary argument about assessment quality. The authors’ discussion of core
psychometric principles in student assessment—validity, reliability, comparability, and fair-
ness—is directed primarily at students and practitioners of psychometrics. Chapter 25 presents
five domains of guidance on classroom assessment, aimed to assist teacher classroom decision
making. Chapter 26 surveys forms and uses of alternative assessment targeted primarily at
students and users of assessment. Chapter 27 returns to psychometrics in discussing the impor-
tance of content validity to the quality of selection exams used in external examination systems
from countries in Europe, Africa, and Asia. Muted in all chapters is attention to the politics of
student assessment (e.g., who decides what is important to test?), and muted in all but chapter
26 is consideration of the influences of external accountability demands on student assess-
ments of all forms.
There are three chapters in section 7: Personnel Evaluation. We agree with the authors in
this section that personnel evaluation is an under-recognized but critically important facet of
educational quality. Thus, we applaud this effort to promote ‘‘more and deeper exchange on

Downloaded from aje.sagepub.com at UNIV OF MONTANA on April 1, 2015

250 American Journal of Evaluation / June 2009

the issues in educational personnel evaluation’’ (p. 603). Chapter 28 focuses on key issues in
teacher evaluation, for example, the use of student achievement to evaluate teacher per-
formance and accountability pressures on teacher evaluation. Chapter 29 focuses on concep-
tualizing the newer domain of principal evaluation through various frameworks, including
role-based, standards-based, and outcomes-based evaluation frameworks. This integrated dis-
cussion offers multiple promising avenues for evaluating school leaders. Chapter 30 is a
mostly successful advocacy argument for evaluating educational specialists and their contri-
butions to student learning.
Section 8: Program/Project Evaluation addresses program and project evaluation in four
global locales. In a sensitive and thoughtful discussion, chapter 31 engages the structural,
socio-political, and especially ethical dimensions of evaluating educational programs in the
‘‘third world.’’ The author observes that ‘‘the determination of merit and worth is intricately
interwoven into the realms of values’’ (p. 715). Importantly, she wonders whose concerns
should take precedence in evaluation—the donor’s or the recipient’s. Chapter 32 offers
another general description of educational program/project evaluation in the United States.
Chapter 33, authored by a Canadian politician, adopts systems and policy perspectives on edu-
cational program/project evaluation, with two richly illustrated cases from the provinces of
Quebec and Manitoba. Similarly, in chapter 34, three cases illustrate selected critical issues
in Australian educational program evaluation—evaluating innovations, engaging with
accountability, and creating an evaluative culture within an educational organization. These
chapters differ considerably in approach, perspective, and depth of coverage but add little
international diversity as three of four are from Western and English-speaking settings.
Section 9: Challenges for Evaluation in Schools examines elements of school level evalua-
tion. Most of the authors in this section subscribe to a similar theory of evaluation (CIPP).
Thus, the reach of this section is limited and does not match the relative diversity of theoretical
stances and methodologies presented in the 1st volume. Moreover, this section again features
predominantly U.S. contexts and U.S. authors. Both chapters 35 and 36 offer quite similar
models for school evaluation in the CIPP tradition. Chapter 35 offers an additional discussion
of how such a model could be institutionalized. Chapter 37 employs the CIPP approach to offer
guidance on the development and use of school profiles (public reports on school accomplish-
ments) focusing on appropriate indicator development. Chapter 38 has a more specific, dual
focus on evaluating the institutionalization of technology in schools and the human capital
development needed for such institutionalization.
Section 10: Local, National and International Levels of System Evaluation returns the
Handbook to its dominant practical interest in student assessment. Its five chapters cover
assessment systems in the United States; national assessment in England; and international
studies of student achievement and curricula. The section’s introduction notes the challenge
to assessment system evaluations to contribute to both performance monitoring and instruc-
tional diagnosis. Chapter 39 offers a detailed descriptive history of the U.S. National Assess-
ment of Educational Progress, with specific attention to the various groups and individuals
involved. Chapter 40 presents a narrative, issue-oriented account of the evolution of the
national assessment system in England, describing political and economic pressures that led
to the development of national educational standards, a national curriculum, and then a
national system to assess student mastery of the curriculum, tracing how the resulting system
emphasized accountability over all other intents. Chapter 41 describes state and school level
evaluation systems in the United States, primarily from the vantage point of a large urban
school district, emphasizing the importance of assessing the ‘‘value added’’ contributions of
schooling and lamenting the dominance of accountability in most system evaluations over ser-
vice to decision making. Chapter 42, written by an international trio of authors, describes

Downloaded from aje.sagepub.com at UNIV OF MONTANA on April 1, 2015

 Book Reviews 251

major large-scale international assessment systems, including the Trends in International

Mathematics and Science Study (TIMSS) and the Programme for International Student
Assessment (PISA), and their methodologies, clearly articulating the comparative function
of these systems. The final chapter concentrates on three types of cross-national curriculum
evaluation, specifically, curricula as the evaluand, as the criterion measure, and as the context
for evaluation, drawing illustrations from TIMSS.

Commentary

What is the character and role of a handbook today? In times past, handbooks offered
definitive, state-of-the-art, comprehensive statements of a field intended to be of some
duration (often a decade or more). Presently, there are scores of handbooks published every
year. Despite expansive titles like the handbooks of yore, today’s handbooks demark
smaller territories and promote narrower visions of a field. So, what territory and vision does
this two-volume Handbook offer?
First, this is a handbook primarily about U.S. K-12 school-based evaluation both in content
and in authorship. International contexts and perspectives are under-represented, as are
educational contexts outside of formal elementary–secondary schooling in the United States.
Second, this Handbook privileges a decision- and utilization-oriented vision of evaluation,
anchored around Daniel Stufflebeam’s CIPP evaluation model and the Joint Committee
Standards. Third, especially in the Practice volume, the Handbook explicitly emphasizes stu-
dent assessment as the core of educational evaluation, evoking earlier understandings of the
interdependence of measurement and evaluation. Fourth, this Handbook emphasizes historical
reviews and descriptions of educational evaluation theory and practice and underplays analysis
and critique thereof.
The two Perspectives and Practice volumes—and many of the individual chapters—can
stand independently, as per the editors’ intent. There are few explicit linkages between, or
within, the two volumes, and this enables readers to read chapters selectively or sections that
seem personally relevant. Still, the Handbook could benefit from greater justification for the
selection and organization of topics, as the distinctions between theory, method, and practice
and between testing/measurement and evaluation seem capricious at times. Notable examples
include chapter 2: The CIPP Model for Evaluation in the Theory section, which includes
detailed prescriptions for good practice; chapter 8: Educational Connoisseurship and Educa-
tional Criticism, an Arts-based Approach to Evaluation, which is often regarded as a theory
of evaluation rather than a method; and chapter 24: Psychometric Principles in Student Assess-
ment, which concentrates on the technicalities of testing not the broader parameters of educa-
tional evaluation. This overlap reflects, in part, the nature of the field but makes offering an
explicit rationale for the book’s organization even more salient.
Since these chapters were published in 2003 many features of the international educational
landscape have changed, notably, the consequential character of accountability demands, con-
troversies over what constitutes credible evidence about educational quality, and the interna-
tional status and power of countries and regions. Even so, this Handbook offers a useful
introduction to educational evaluation for new students of this social practice, along with some
particular discursive jewels. Although some chapters are written for an expert audience, the
sections on theory, methodology, student evaluation, and personnel evaluation are concise,
comprehensible, and offer students and practitioners a clear, if not complete, orientation to the
landscape of perspectives and practices in the field of educational evaluation. Individual chap-
ters and sections that point the reader to recommended references and additional resources are

Downloaded from aje.sagepub.com at UNIV OF MONTANA on April 1, 2015

252 American Journal of Evaluation / June 2009

particularly beneficial. In sum, the editors and authors have crafted a valuable but bounded
introduction to the field of educational evaluation, with special relevance for newcomers and
evaluation practitioners in U.S. school-based settings.
A. Rae Clementz
Jennifer C. Greene
University of Illinois, Urbana-Champaign

Russ-Eft, D., Bober, M. J., de la Teja, I., Foxon, M. J., & Koszalka, T. A. (2008).
Evaluator competencies standards for the practice of evaluation in organizations.
San Francisco: Jossey-Bass, 240 pp., $50.
DOI: 10.1177/1098214009334510

This was a fascinating book to review. It was an inspiring, thought-provoking read and
extremely useful for evaluators within and, I believe, beyond organizations. The book contains
eight chapters, beginning with a chapter on the practice of evaluation in organizations, fol-
lowed by discussions of the development, application, foundational research, and validation
of evaluator competencies, and ending with future directions. For readers interested in com-
petency development, the use of competencies, and their validation, the book may be read
cover-to-cover. Those who would like to quickly reference or compare evaluator competencies
will be pleased to find a concise list at the end of chapter 3. However, it is indeed advisable to
also read the underlying assumptions and the implementation section in this chapter as well as
the compelling elaborations of the competencies and performance statements in chapter 4.
The evaluator competencies are the fourth in a series of standards endorsed by the Interna-
tional Board of Standards for Training, Performance, and Instruction (IBSTPI). The other
three sets are related to instructional design, training, instruction, and learning. IBSTPI is a
not-for-profit organization providing leadership to the human learning and performance com-
munities by setting international standards. In this sense, the competencies are written for
human resource professionals and for human resource development applications, but inasmuch
as they can be validated by the larger evaluation profession and other evaluation standards,
they do have broader application.
The competencies define the knowledge, skills, and attitudes required to conduct internal
evaluations in organizations. A broad range of organizations are identified as potential audi-
ences, including for-profit organizations, which have been very much on the fringes of the eva-
luation profession. The use of the competencies has additionally been applied to resourcing job
requirements, position descriptions, curriculum development, teaching, student assessment,
and best-practice models.
It is indeed timely and opportune to define evaluator competencies related to business
processes and the learning organization. The first chapter outlines the difference between
evaluations within organizations and those examining local, state, or national programs.
Within-organization evaluations are seen as ‘‘practice oriented atheoretical,’’ ‘‘process-driven
operational,’’ and ‘‘research-oriented, practice-based comprehensive.’’ These categories are
adopted from the work of Wang and Spitzer (2005). Russ-Eft et al. summarize 17 models born
of the 3 categories. Although evaluations within organizations are well described, the differ-
ences between them and large-scale evaluations of government programs is rather cursory,
using the example of ‘‘reducing adult literacy programs’’ as an evaluation not internal to an
organizations. The authors also propose that the differences between internal organizational

Downloaded from aje.sagepub.com at UNIV OF MONTANA on April 1, 2015