‘School Elfectiveness and School Improvement (0924-3453/960703.0197812.00 1996, Vol. 7. No. 3, pp. 197-228 © Swois & Zeitlinger School Level Conditions Affecting the Effectiveness of Instruction Bert P.M. Creemers and Gerry J. Reezigt GION, University of Groningen ABSTRACT In this articte, the current status of school level factors. as they appear in research reviews and in school effectiveness models is criticised both from a theoretical and trom an empirical perspective, School level factors are ofien related to student achievement with- ‘ut taking into account the classroom level. where teaching and learning primarily take place. As a theoretical alternative, an overview of school level Factors that enhance the quality of instruction, time for learning and opportunity (o Jearn at the classroom level is provided, ‘These factors are supposed to contribute to the explanation of differences in student achievement. Some empirical support for these ideas is found in educational research studies that have made use of three-level analyses. INTRODUCTION School effectiveness research started as a reaction to the disappointing results of educational research with respect to the influence of schools and teachers. Carefully designed research with respect to teacher behav- iour in classrooms and the structure and organisation of schools could not avoid the disappointing conclusion that student characteristics (such as abilities and social background) accounted for major proportions of vari ance in student outcomes, even though research into the relationship be- tween teacher behaviour and student outcomes proved thal teacher behav- iour could account for a small proportion of the variance in student out- comes. Correspondence: GION, Groningen Institute for Educational Research, University of Gro- ingen, P.O. Box 1286, 9701 BG Groningen, The Netherlands. Tel: (1) 50 636635 636660. Fax: (31) 50 636670. E-mail: g.j reczigt@ ppsw.rug.nl Manuscript submitted: December 15, 1994 Accepted for publication: May 14, 1996198 BPM, CREEME {SAND GJ. REEZIGT School effectiveness research was a reaction to the common fecling that schools and teachers did not matter. Initially, no distinction was made between the different levels within schools, such as departments and classrooms, in order to make the point as strong as possible: schools matter. The factors connected with educational effectiveness were a com- bination of all kinds of factors, as can be seen in an analysis of the so- called five-factor model and other models (Creemers & Scheerens, 1991: Scheerens, 1992). Later. the loose theoretical framework of school effec- tiveness was criticised and as a result more emphasis was given to the development of a conceptual framework that contains more levels in ad- dition to the student level. At least two levels: the classroom level within the school and the school level were discerned. Other levels, for example the departmental level in the schoo! organisation can also be discerned as well as levels immediately above the school, such as the school board. In the effectiveness models developed later on (Scheerens, 1989: Creem- ers, 1991; Slater & Teddlie, 1992; Stringfield & Slavin, 1992) a relation- ship between the classroom and the school level can be found. The educa- tional processes ~ instruction and learning — take place at the classroom level and the other levels are supposed to provide the conditions for instruction at the classroom level. This conceptual framework, however, did not stop the search for all kinds of correlates of effectiveness. and the list of possible correlates for effectiveness continued to grow. As a result of the methodology used, these correlates can be found at all levels, the classroom, the school, the school board, the district, and the national level. One of the criticisms against the correlates is that it is not clear how the relationships between these correlates might be, and what goes on at the classroom level Research including both the school level and the classroom level showed that the classroom level was important, quite often even more so than the school level (see, for example, Scheerens et al., 1989). Also, theory- oriented publications stressed the predominance of the classroom level ‘This implies that in school effectiveness research attention for the school level is diminishing whereas the classroom level gets more attention than before. In fact, we could be back at the stage of the pre-school-effective- hess-movement: teachers and classrooms make a difference, albeit quite small, However, even when learning and teaching take place primarily at the classroom level, educational effectiveness means more than outcomes in just one class, Educational effectiveness has to do with outcomes and effectiveness in successive classes in a school and in the educational system as a whole, The classroom level alone cannot guarantee adequateSCHOOL LEVEL. CONDITIONS 199 transitions between different classes. Effectiveness of education there- fore is more than effectiveness at the classroom level. This raises the following questions: 1. Which unique contribution, in terms of proportions of variance ac- counted for, can be expected from the school level when it is analysed simultaneously with the classroom level? 2. Taking the classroom level into account, which school factors can be supposed to contribute to educational effectiveness, and what is the relationship between factors at the school level and factors at the class- room level? To answer these questions, we will deal with the factors at the school level discovered in research, and the supposed effects of these factors on outcomes (in the next section). In doing so, for the sake of clarity we will focus on general effects of factors on cognitive achievement criteria, which means that issues of differential effects of factors and the potential consistency of factors across different types of outcomes will not be touched upon. The research results can be criticised on theoretical and empirical grounds (see the section on theoretical and empirical consider- ations). A theoretical basis for schoo! level factors, connecting the school level and the classroom level, will be outlined (sce the section entitled “A theoretical basis for school factors”). Finally, some examples of research into school and classroom effectiveness are discussed (sce the section on three level research on school level factors) and guidelines for future research are given (see the summary and conclusion). THE CURRENT STATUS OF SCHOOL FACTORS IN SCHOOL EFFECTIVENESS THEORY AND RESEARCH School effectiveness research has yielded impressive numbers of school factors related to student achievement. Two types of factors can be dis- cerned: school factors that are summed up in reviews of research (sup- posed to be empirically and directly related to achievement), and school factors that are included in school effectiveness models (presumed to influence classroom processes and therefore indirectly related to student achievement). Reviews of Research The school factors in research reviews are mostly derived from case study research and correlational research with an aggregated index of student achievement as the criterion for effectiveness. The school factors repre-200 P.M. CREEMERS AND GJ. REEZIGT sent a wide variety of characteristics, ranging from very global (e.g. school demographics; Wang, Haertel, & Walberg, 1993) to very specific (c.g., communication with the school adviser: Spade, Vanfossen, & Jones, 1985). The main categories of school factors, mentioned by several re- viewers such as Purkey and Smith (1983), Mortimore, Sammons, Stoll. Lewis, and Ecob (1988). Levine and Lezotte (1990), Scheerens (1992), and Wang et al. (1993) are the following: 1. orderly environment/school climate, consensus and cooperation between teachers, 3. focus on basic skills/learning time, 4, monitoring of student progress/evaluation, 5. school educational/administrative leadership, 6. policy on parental involvement, 7. high expectations. Many authors, especially Levine and Lezotte (1990), mention subcatego- ries (such as support of teachers, acquisition of resources, utilisation of instructional support personnel as refinements of the main category of school leadership), or additional school factors (such as teacher experi- ence, changes in staff, or urbanisation level of the school environment. (Scheerens, 1992). These subcategories and additional school factors are not so consistently reported as the main categories mentioned above. The main categories, with the exception of consensus and cooperation be- tween teachers and policy on parental involvement, reflect the five fac- tors suggested by Edmonds (1979). This finding might indicate the ro- bustness of these factors, However. the similarity of categories may also be attributed to reviewers who continuate to study a set of literature which has been reviewed before. thereby creating a self-repetition of findings. It may also point at a research artefact: because these factors were supposed to influence student achievement, they were studied over and over again, and indeed were found to influence student achievement from time to time. The factors therefore are certainly empirically support- ed, but other factors, that have not been studied so often, might also be important School Effectiveness Models Not all of the models that are cited in the literature on educational effec- tiveness specity school level factors. Relatively early models (e.g., Slavin, 1987; Walberg, 1984) are restricted to the student level and the teacher! classroom level. More recently, several models for educational ef! ness that do specify school level factors were proposed as a contribution to theory development (Scheerens, 1992; Slater & Teddlie, 1992; String- eSCHOOL LEVEL CONDITIONS 201 Suringfield & Slavin, 1992 = meaningtul goals + attention to academic functioning in all classes (leadership) ~ coordination of curricula i instruction across ‘classes. programs, and grades recruitment and develop: ment of staff ~ efficient organising of school funetioning ~ student teacher/elasstoom schoo! context ‘hol Level Factors + achievement-oriemted policy + educational leadership ~ consensus/cooperative planning + quality of school curricula + orderly atmosphere + evaluative potential Levels in the Model = student > teacher/elassroom = school = context administrative appropriateness management schoo! structure) leadership (influencing school culture) + student + teacher/elassroom = school Basic Assumptions about the Role of School Level Factors + indirect effects on student achievement by managerial and organisational processes that influence quality and appropriateness of instruction, incentives, and time (gait) atthe classroom level ~ indirect effects on student achievement: school factors as conditions (reinforcers! buffers) for classroom level processes indirect effects on student achievement; school Factors interact with classroom level factors: effects of interactions explain changing effectiveness Criteria for the Inclusion of School Level Factors + theory on schoo! earning empirical evidence (Louisiana studies) = eclectic: organisational, economical, polities theories ~ empirical evidence (reviews) ~ organisational theories, + empirical evidence (Louisiana Studies) [. Schoo! level factors in recent educational effectiveness models.202 B.P.M. CREEMERS AND G.I. REEZIGT field & Slavin, 1992), Although the school level factors described in these models are partly selected on the basis of their empirical support (often postulated as direct relationships with student achievement), they are mainly included in the models because of their presumed influence on classroom processes (and therefore their indirect relationships to student achievement). Figure | shows the school level factors in these three mod- els and the assumptions that led to their inclusion Although the models differ with regard to the exact number of levels included, the assumptions about interactions between levels, and the the- ories used for the selection of school factors, the selected school level factors in the models are not essentially different in nature. They are also not so very different from the school factors in research reviews. For example, the factors “attention to daily academic functioning in all class- es’ (Stringfield & Teddlie, 1991) and ‘recruitment and development of staff” are clements of the main category of leadership in the review by Levine and Lezotte (1990). The only factor that is often mentioned in reviews but not included in the models is a school policy on parental involvement. Factors in the models that are not clearly discerned in re- search reviews are: 8. coordination of curricula and instruction across classes. programs and grades (Stringfield & Slavin, 1992); and 9. quality of school curricula (Scheerens, 1992). The school factors in research reviews and in effectiveness models are largely the same. This suggests that there is some general agreement about the effectiveness of a set of, in total, 9 school factors. The Size of Effects of School Factors The general agreement about the set of school factors does not necessar- ily reflect the actual importance of school factors for student achieve- ment as indicated by the size of their effects. Information about the size of effects is necessary to judge the relative importance of isolated fac- tors. Despite the rich literature on school effects. this kind of factual information is not easy to obtain. Research reviews not always mention the size of correlations; they often include school factors merely because they were studied in a lot of projects and then were repeatedly found to have some kind of positive influence on an achievement indicator, but not on behalf of their effect sizes. For example, Levine and Lezotte do not give any information about the size (or the average size) of corrcla- tions between school factors and student achievement, Their factors are often derived from small-scale case study research, which does not allow for the computation of correlations or effect sizes (Levine & Lezotte, 1990).SCHOOL LEVEL CONDITIONS 203 Sometimes correlations, regression results or results of two-level re- search of individual studies are reported. For example, Fraser, Walberg, Welch, & Hattie (1987) mention an average correlation of school charac- teristics (such as aims and policy) and student achievement of .12; Schee- rens (1992) mentions correlations ranging from .02 (heterogeneity of classrooms) to .79 (high expectations). According to Wang ct al. (1993), the most important school factor is school culture (average T'score of 53), followed by teacher/administrator decision making (7=48), a policy on parental involvement (7=46), and school demographics (T=41). The school factor with the smallest impact on student achievement is school policy and organisation (7=37). Teddlie and Stringfield (1993) find correlations of school factors ranging from .28 (parental support) to .49 (perception of school success by the principal). They compare their findings with corre- lations found by Brookover, Beady, Flood, Schweitzer, & Wisenbaker (1979), and they report some stability with regard to the size of correla- tions. These kinds of comparisons, the same group of factors in one study compared to the same group of factors in another study, are rather rare and because of the different combinations of factors in different studies, it is difficult to find out which factors are stable. Also, correlations of isolated factors with student achievement may hide problems of multicol- linearity. Such problems are somewhat smaller when regression tech- niques or two-level analyses are used. but even then it is difficult to assess the relative importance of school factors because of the differences in school factors represented in several studies, and because of the differ- ences in indicators of achievement (student scores on standardised tests, exam results, attained educational level). Generally it is assumed, at least in the Western hemisphere, that 10 to 20 per cent of variance in student achievement can be attributed to school factors after correction for student intake measures such as aptitude or social class (Bosker & Scheerens, 1994; Mortimore et al., 1988; Scheer- ens, 1992). It is important to note that these percentages only set the boundaries for the potential impact of school factors. The question as to Which school factors are most important for student achievement still is largely unanswered. THEORETICAL AND EMPIRICAL CONSIDERATIONS Taking the set of 9 school factors discerned in research reviews and in educational effectiveness models as a point of departure, critical remarks can be made from a theoretical perspective and from an empirical per- spective.204 D.P.M, CREEMERS AND G.I, REEZIGT Theoretical Perspective Theoretical considerations refer to the theoretical basis of school factors, the conceptual range of school factors, the mutual relationships between school factors, and the relationships between school facto room level factors, and class- The Theoretical Basis of School Factors The five factors suggested by Edmonds have dominated school effective- ness research for years, although initially the five factors were neither based on a specific theory nor abundantly supported by research. Empiri- cal support has grown over the years, but still the factors are not firmly embedded in a set of theoretical concepts that can explain the differences in student achievement as far as these are caused by schools. The educa- tional effectiveness models developed so far study school factors from varying theoretical perspectives (learning theory, eclectic approach, or- ganisational theory), but the actual factors in the models were partly included because of their empirically supported status. This explains the similarities of the models, despite the different theories that they are based upon. The translation of concepts from organisational and econom- ical theories to the educational domain, as suggested by Scheerens (1992), has not yet led to a new set of concepts or a different view on the impor- lance of the traditional school factors. The Conceptual Range of the School Factors the lack of a theoretical basis explains the wide conceptual variety of school factors, Even when only the main categories of factors in research reviews and models are considered, the factors represent quite different dimensions. There are factors referring to the behavior of persons in the organisation (mainly the principal), to persons related to the school (par- ents), to material conditions (quality of school curricula), to organisation- al elements (recruitment of staf), to schoo] culture (school climate. high expectations), and to educational aspects (monitoring and evaluation) Although all of these dimensions may be important for school effective- ness, it is necessary to arrange them according to theoretical notions to make their importance for student achievement easier to understand. ‘The Mutual Relationships Between School Factors Because of the variety in dimensions, the relationships between school factors are not very clear. Some factors are partly overlapping (¢.g.. mon- itoring/evaluation and achievement-oriented policy), some factors are caused by or at least influenced by other factors (¢.g.. an orderly environment bySCHOOL LEVEL CONDITIONS 205 educational leadership), and some factors seem to be conditional for each other (c.g., high expectations and a focus on basic skills/learning time). More clarity about these relationships is needed to interpret effects of school factors correctly The Relationships of School Factors with Classroom Level Factors Some school level factors, for example focus on basic skills, monitoring! evaluation, and high expectations, refer to processes that seem to be merely essential at the classroom level. When teachers are not focusing on basic skills. il is hard to imagine what the special contribution of the school level could be in this respect and what the concept might actually mean at the school level. Such factors derived from classroom level processes are somewhat problematic because it is not clear if they refer to more than a mean score of teacher behaviour indicators School factors are generally described as conditional for classroom factors (Scheerens, 1992: Stringfield & Slavin, 1992), or interacting with classroom level factors (Slater & Teddlie, 1992) but the relationships of school factors with classroom level factors are not delineated in detail in the models. In general, one block of school level factors is depicted as influencing one block of classroom level factors. It is not clear which subgroups of school factors are supposed to influence which subgroups of classroom level factors and in what way they are supposed to do so (Bosker & Scheerens, 1994) Empirical Perspective The empirical considerations about school factors concern the lack of information about their influence on student achievement controlled for the influence of classroom factors. Research in the field of school effec- tiveness is often criticised. but despite all these criticisms the school fac- tors have obtained a status which is not sufficiently supported by empirical findings, probably because they appeal to practitioners and school improv- ers as important or essential factors. Whether school effectiveness find- ings can really be attributed to school level factors or are confounded with classroom level factors, and therefore should be attributed to some extent to classroom level factors is not a regular topic for discussion However, the design of most studies. i.e. school factors directly relat- ed to student achievement without controlling for the influence of class- room factors leaves questions on the validity of school level findings unanswered. Because of the separate research traditions in the fields of school and classroom effectiveness, up till now very few studies have collected data at both levels. According to Teddlie and Stringfield (1993),206 BPM. CRFEMERS AND Gi. REE only nine studies on school and teacher effects in a total of 328 that were reviewed in the 1980s involved both levels. And even when studies have collected data both at the school and the teacher level, school effects and classroom effects were often analysed separately instead of simultane- ously (Brandsma, 1993; Mortimore ef al., 1988; Van der Werf & Gulde- mond, 1994). Although school factors are theoretically thought to influ- ence student achievement indirectly through classroom factors, rescarch concentrated on direct effects and as a consequence may have under- or overestimated the actual influence of school factors Factual information on the relative impact of school and classroom factors is provided by Wang et al. (1993). In their overview of the influ ence of six theoretical constructs on student learning (the average T score set at 50), school factors show an average score of 45. They are surpassed subsequently by student characteristics (T score 55), classroom practices (T score 53), home and community educational contexts (7 score 51), and the design and delivery of curriculum and instruction (T score 47). The only construct ranging lower than school factors is state and district gov- emance and organisation (T score 35). Teddlie and Stringfield (1993) performed regression analyses with mean student achievement as the de- pendent variable and 17 factors derived from student, teacher and school level data as the predictors. Each of the 17 factors initially correlated significantly with achievement, but the regression procedure showed that in the end only 3 factors, all of them at the student level, contributed significantly to the measurement model in the end. Similar results were found in a two-level HLM reanalysis. Student and classroom level factors are potentially more powerful predictors of student achievement than school level factors are. Estimating the Impact of School Level Factors The degree of inaccuracy in the estimation of the impact of schoo! level factors ideally can be established by performing two-level (school and stu- dent) and three-level (school, classroom, and student) analyses on an iden- tical dataset and then comparing the estimations of the proportions of school level variance. Even more ideally, achievement measurements over several points in time would be included which would lead to a four-level analysis (time points nested within students within classrooms within schools). Four- level analyses, although they can be performed by some multilevel packag- es have not been published so far. Even three-level analyses are hardly available in the educational effectiveness literature, and it is even harder to find comparisons between two-level and three-level estimations, as described above. Some studies do offer information on the proportions of variance atSCHOOL LEVEL CONDITIONS 207 the school level when estimated simultaneously with the classroom level in a two-level analysis and when estimated separately in a three-level analy- sis. For example, the proportion of school level variance in a simultaneous estimation was 15 for language achievement and 24 for mathematics achieve- ment (Reezigt. 1993). When the classroom level was introduced in analy- ses of roughly the same dataset, the proportions of school level varianc: were reduced to 13 and 15, respectively. while the proportions of class- room level variance amounted to 6 and 11 (Bosker, 1991). So, the separate estimation of the classroom level and the school level variance shows other results than does the simultaneous estimation, In the IEA-studies on mathematics achievement in secondary educa- tion (Scheerens et al. 1989), the mean proportion of variance at the school level when estimated simultaneously with the classroom level is 42 (8 countries, range 8-64). The classroom and the school level variance were estimated separately in 9 countries. The mean proportions arc 33 for the classroom level (range 17-46) and 7 for the school level (range 0-15) These analyses point at an overestimation of the school level variance when the classroom level is not analysed separately. The figures men- tioned so far only concern unconditional models. It is still not clear which specific factors might account for the variance at the different levels. To overcome the theoretical and empirical considerations described above, firstly school factors have to be related to classroom level factors on the basis of a theory that can explain differences in student achieve- ment by the selection of factors at different educational levels and that rationally relates factors both within and across levels to each other Secondly, empirical support for the school factors that are selected ac cording to this procedure might be found in research that explicitly takes, the hierarchieal structure of education into account. The following sec- tions will deal with these topics A THEORETICAL BASIS FOR SCHOOL FACTORS In the previous sections we have seen that research on school effective- ness revealed a lot of effective school factors and present research goes on detecting even more factors that are connected in one way or another to student outcomes. Research can be eriticised for methodological rea- sons but also from a theoretical point of view. Recently, based on a plea for a theoretical foundation of school effectiveness research, several the- oretical models were developed that take into account the difference be- tween the classroom level, where instruction and learning take place, and