You are on page 1of 2

Evaluu~ion and Program Pianning, Vol. 5, pp. 371-375, 1982 0149-7~89/82/~371-~03.

Printed in the U.S.A. All rights reserved. Copyright @ 1983 Pergamon Press Ltd


Meta-Analysis in Social Research, Gene V. Glass, Barry S. McGaw and Mary Lee Smith. Beverly Hills: Sage
Publicatiork, Inc., 1981, 273 pp., $25.00 (hardcover).

Reviewers: David A. Kenny and Jeffrey S. Berman

Looking back over the 1970s there were a number of As mentioned earlier, the Glass, McGaw, and Smith
major developments in methodology. These book is not the final word. We wish to raise three
developments include work in log linear analysis, issues that we hope that they would address in their
structural equation modeling, and time-series analysis. second edition.
However, it seems to us that the most important work Because quantitative reviews are relatively new,
in the past decade was in the area of summarizing there has been considerable variability in how such
results across studies by quantitative methods. This analyses are handled and reported. Researchers seem
area has been called meta-analysis. This effort is the to be adopting the “kitchen sink” approach and they
most promising not because of its complexity in overwhelm readers with ail sorts of numbers in their
statistical analysis, but rather because it takes standard meta-analyses. Instead of doing all possible types of
methods and applies them to important questions in analyses (e.g., combining p values, computing effect
social science. No longer are we limited in thinking of sizes, and multiple regression analyses), researchers
persons within a study as the unit of analysis but now need some very clear guidelines about which pro-
the study itself becomes the unit of analysis. Meta- cedures are appropriate. While somewhat helpful
analysis represents a bold new step in the analysis of Glass, McGaw, and Smith do not provide these
data. guidelines.
The area of meta-analysis is developed enough The purpose of an effect size measure is to provide a
where a book is in order and Glass, McGaw, and standardized estimate of treatment effects that can be
Smith have given us one. As they mention in their compared across studies. For example, the Glass,
preface they have given us a unified perspective on McGaw, and Smith’s effect size measure is the dif-
meta-analysis but not the final word. They cover the ference between treatment groups relative to the con-
basic topics of research integration: how to locate trol group standard deviation. (We find it quite
studies, how to classify studies, how to measure ef- curious that the authors do not acknowledge that the
fects, and how to analyze studies. Moreover, they pro- measure of effect size preferred by most experts in the
vide a rationale for meta-analysis as well as respond to area has as its denominator the pooled within groups
the critics of the enterprise. standard deviation and not the control group standard
For the novice the book provides a superb introduc- deviation.) The difficulty with this approach,
tion to the topic. Real examples are continually used however, lies in the possibility that within-group
and although there are a few obscure passages here variability may not be equivalent across the studies in
and there, the book is very readable to anyone familiar the domain. This nonequivalence problem is especially
with multiple regression and most of the book is in- im~rtant since most of the effort in a meta-analysis is
telligible to those familiar with only analysis of devoted to a comparison of effect sizes. If these effect
variance. To someone planning to do a meta-analysis sizes differ because of design or other extraneous
the book is particularly lucid since so many concrete features of the study, these comparisons can be
examples are used. The authors even provide a sample misleading. This nonequivalence of within-group
form for coding studies. To those who already have variance may result from differences in the subject
some knowledge of the literature, the book is not as in- populations (e.g., some studies may have more
structive since very little new ground is broken. A good homogeneous samples than others), differences in the
deal of the material has been published elsewhere. It is, reliability of the dependent measure, or in the way er-
however, useful to have all of this material in one ror variance is computed (e.g., effect sizes in some
volume and there is some new material (e.g., work on studies may be based on ANCOVA or repeated-
nonindependence). measures ANOVA, resulting in much smaller


to students of evaluation. authors also note a problem with alternatives: it is termediate programmatic. problematic approach would be to compute a single tions often require information that is not typically measure of effect size from each study. as it claims to. describes the authors’ Hammond and Knott believe that in single-mindedly research procedures: a careful analysis of all available focusing on the economic costs and benefits of any case studies on the use of Zero-Base Budgeting (ZBB) given expenditure.for example. with enough examples. It is not clear to us that guessing is the best the same domain. the proposed solu. Planning Plus Scarce Resources Equals tensive past experience and from the literature. Hammond and Knott point out that instead ZBB with broader issues in politics and public ad. and a systematic the political costs and benefits of categorizing any effort to relate these findings to what is already known given activity in different ways. The sensus on goals as unnecessary if there is already agree- Hammond and Knott book will be valuable to practic. of integrating planning with budgeting. Reviewer: William R. those concerning per. A less recognize this problem. ministration. a masterful analysis of zero-base can make the competition for scarce resources more budgeting. Meyers This important book. quiring that we first choose our preferred way of ac- Chapter 2. organizations. The authors’ recommen. program the issues they raise for the public sector are relevant categories that can logically be “prioritized. procedures that has Meta-Analysis in Social Research is a book to be raised controversy is their use of data from the same read and studied by social scientists. If the different dependent parability of effect sizes from different studies may measures are assessing different constructs. come. sonnel selection and training-that must intervene Moreover. and gives some genera1 conclusions. considerations .” and the use of performance measures in preferences and amount of money available simulta- governmental organizations. And there is little doubt that many of gregated into decision packages. Thus Budgeting. The noncom. considers allocation of dollars among the the authors are able to link their conclusions about packages. and they view achieving con- administrative realities of the budgeting process. and also to tivities and functions can be sharply defined and they managers and administrators who have budgetary reject entirely the idea that they can or should be ag- responsibilities. Traditional budgeting considers packages. Thomas H. neously. that is. and politics from ex. New Brunswick: Trans- action Books. to which particular goals has not been fully resolved. since alternative means reach different goals. As the be written sometime during the next five years.95 (hardcover). Knott. provides much insight into the political and intense and acrimonious.” The also to the private sector because of the complex in. the with how much money to spend carrying out the task grouping of “activities” and “functions” into “decision in this preferred way. about budgeting. Hammond and Jack H. state. However. A Zero-Based Look at Zero-Base Budgeting. authors note this nonindependence can artificially dations for extracting effect sizes (see chapter 5) do underestimate the variance of the effect sizes. considers the specification of objectives. The Introduction also provides an over.. 1980. complishing a task. Introduction. which has the effect of uniting planning and Hammond and Knott are skeptical about the value of budgeting. clarifying goals. and local levels. Chapter 3. ZBB actually separates planning from budgeting by re- view of the book.372 BOOK REVIEWS estimates of error variance). the question of which specific alternatives lead turn out to be. novice with a good introduction and the practitioner ment of these estimates as if they were independent. As political scientists. a better procedure. puted for each study. and only then concern ourselves cedures. $14. ment on policy. ZBB advocates blind themselves to on the federal. owing to lack of data and lack of causal before one can anticipate what the “bottom line” will theory. Chapter 1. Let us hope it acts like John the With this method it is not uncommon for there to be Baptist in preparing for the gospel of meta-analysis to twice as many effect size measures as studies. If multiple available from published reports and must therefore dependent variables are available and they are all in be guessed. separate end up being one of the more difficult problems facing analyses could be carried out for each type of out- those who conduct quantitative reviews. for often bringing out divergent views The authors believe that decision packages are in- . It provides the study to generate multiple effect sizes and their treat. One aspect of the Glass et al. A Political Analysis of ZBB’s Pro. an average effect size could be com- way to make effect sizes comparable. 148 pp. planning and budgetary hard to specify alternative means to the same goal. They are even more skeptical that ac- ing evaluators.