Differential Secondary School Effectiveness (Comparing The Performance of Different Pupil Groups)

Differential Secondary School Effectiveness: Comparing the Performance of Different
Pupil Groups
Author(s): Sally Thomas, Pam Sammons, Peter Mortimore and Rebecca Smees
Source: British Educational Research Journal , Sep., 1997, Vol. 23, No. 4 (Sep., 1997), pp.
451-469
Published by: Wiley on behalf of BERA
Stable URL: https://www.jstor.org/stable/1502081
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms
Wiley and are collaborating with JSTOR to digitize, preserve and extend access to British
Educational Research Journal
This content downloaded from

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
All use subject to https://about.jstor.org/terms
British Educational Research Journal, Vol. 23, No. 4, 1997 451
Differential Secondary School Effectiv

comparing the performance of differen
groups
SALLY THOMAS, PAM SAMMONS, PETER MORTIMORE & REBECCA

SMEES, International School Effectiveness and Improvement Centre, Univer
London Institute of Education
ABSTRACT This article reports the results of an Economic and Social Rese
Council (ESRC) funded study which focuses on the differential academic achievem
different groups of pupils. The paper describes the findings on the size and exte
school effects across 3 years (1990, 1991, 1992) for different groups of pupils (cl
by gender, eligibility for free school means [FSM], ethnic group and by prior att
ment). Pupils' overall General Certificate of Secondary Education performan
their performance in selected subjects (English, English literature, French, h
mathematics and science) have been analysed using multilevel modelling, employi
total sample of 94 inner London secondary schools. A 'value added' approac
adopted, controlling for selected student background measures of prior attainme
secondary transfer), gender, age, ethnicity and low income to provide statistical co
for differences between schools in the characteristics of their intakes. Differential
and departmental effects were identified for all pupil groups examined. However
strongest evidence of differential effects was found for groups classified by
attainment and ethnicity. Overall, the findings indicate that schools that appear
more or less effective for a particular group of pupils, such as non-FSM pup
likely to be more or less effective for all pupils. However, in some schools, subst
differences between groups were identified. The implications of these findings f
debate about the publication, presentation and interpretation of schools' examinat
results and the extent to which the overall concepts of 'effective' or 'ineffective' s
can be applied, are discussed.
Introduction
This article [1] reports results of an Economic and Social Research Council (ESR
0141-1926/97/040451-19 ?1997 British Educational Research Association

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
452 S. Thomas et al.
sponsored study of Differential School Effectiveness focusing on the issue

and consistency of overall school effects versus departmental differences
pupils' General Certificate in Secondary Education (GCSE) performanc
employs data collected for the 1990, 1991 and 1992 studies of GCSE results
the Association of Metropolitan Authorities (AMA) and funded by ind
education authorities (LEAs) (Nuttall et al., 1992; Thomas et al., 1993
project was instituted to investigate four crucial issues of effectiveness in
city schools:
* subject differences (or consistency in effectiveness for different academ

* differential academic achievement of different groups of pupils;
* stability over time of results; and
* school and departmental processes affecting GCSE attainment.
The overall study has three major aims. The first of these is to extend curr
about the size, extent and stability over time of differences between secon
in their overall effectiveness in promoting pupils' GCSE attainments.
investigating the applicability of the concept of overall school effecti
secondary sector and whether schools can be divided validly into broadly e
ineffective groups. Secondly, the study is exploring the extent of interna
school effectiveness, (i) at the departmental level and (ii) for different gr
classified by gender, eligibility for free school means, ethnic group a
attainment. Comparisons of departmental, group and school differences ar
to establish the extent to which schools are differentially effective in cer
and/or for particular pupil groups over a period of 3 years. The foc
consistency in schools' effects on different outcomes and for different gro
Thirdly, the study is investigating the reasons for any differences in effec
extent to which overall school differences in effectiveness and differences at the
departmental level and for different pupil groups is being explored in relation to spe
school status, organisational and other process characteristics. In particular, the stu
investigating the extent to which different factors and processes are importan
accounting for variations in school and departmental effectiveness (see Sammons et
1995a, 1995b, 1997).
However, the main focus of this article concerns the second of these key aim
describes the findings on the size and extent of school effects across three GCSE coho
(1990, 1991, 1992) for different groups of pupils (classified by gender, eligibility for
school meals, ethnic group and by prior attainment) and focuses on pupils' overall G
performance and their performance in selected subjects chosen because of the relati
high number of entants (English, English literature, French, history, mathematics
science). Previous reports describe the conclusions concerning stability and consisten
of school and departmental effects for all pupils in different academic outcomes ov
3 years (Thomas et al., 1994b, 1997).
Two key questions are addressed in this article:
1. Is there any evidence that some schools and departments are differentially effec
in promoting GCSE attainment for different groups of pupils (categorised by gend
ethnicity, prior attainment and entitlement to free school meals) or are secondar
schools and departments equally effective (or ineffective) in promoting attainment
all groups?
2. If schools or departments are differentially effective, what is the pattern of result

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
Differential Secondary School Effectiveness 453
different groups of pupils (categorised by gender, ethnicity, prior attainment and

entitlement to free school meals) across different GCSE outcomes (i.e. in terms of
both departmental results and the overall GCSE measure)?
To address these research questions multilevel analyses of pupils' GCSE examination

results over 3 years have been conducted for a sample of 69 (1990), 94 (1991) and 77
(1992) inner London secondary schools drawn from eight LEAs. The analyses build on
the results of previous work concerned with the stability and consistency of school
effects (Thomas et al., 1997) and, crucially, adopt the same 'value added' approach,
controlling for selected student background measures of prior attainment (at secondary
transfer), gender, age, ethnicity and low income to provide statistical controls for
differences between schools in the characteristics of their intakes. A large body of
evidence is now available concerning the importance of controlling for pupil prior
attainment and other background factors (outside the control of the school) in the
calculation of school effectiveness measures (Nuttall et al., 1989; McPherson, 1992;
Goldstein et al., 1993; Thomas et al., 1995; Sammons et al., 1996).
An analysis for each outcome measure (English, English literature, French, history,
mathematics and science [highest score in any science subject]) has been carried out,
which estimates, separately, the variation in pupil scores due to the school, cohort and
pupil levels in order to provide a single measure of school or departmental effectiveness
that reflects average performance over the 3-year period. In addition, the 3-year models
are extended to investigate schools' effects for different groups of pupils in order to
address the question of whether some schools or departments are more or less effective
for particular types of pupil. A separate but related issue is also examined; whether some
schools or departments appear to widen or narrow the gap in attainment between
different pupil groups (such as boys and girls).
The main objective, therefore, is to provide new findings that will be relevant to the
debate about the publication, presentation and interpretation of schools' examination
results, and the extent to which the overall concept of 'effective' or 'ineffective' schools
can be applied. A further objective is to provide new procedures and evidence for
evaluating the impact of equal opportunities policies at the school and departmental
level. Before reporting the results, a brief summary of relevant previous research which
has addressed the issue of differential effectiveness for different groups of pupils, is
outlined as context for the present inquiry.
Previous Research
The importance of differential school effects is a topic of increasing interest in

effectiveness research, especially in the UK field. Differential school effects conc
existence of systematic differences in attainment within schools for different
groups (those with different levels of prior attainment or different background ch
istics), once the average differences between these groups has been accounted fo
following overview focuses only on research at the secondary level. However, re
on differential effects has also been carried out at the primary level (see, for ex
Sammons et al., 1993).
Prior Attainment
The study by Rutter et al. (1979) did not utilise multilevel techniques; however, it did

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
examine schools' examination results for the most and least able childre
the results for children of different levels of prior ability (using a three
at intake-Verbal Reasoning [VR] band). It was found that 'the patter
each school was broadly similar in all three bands' (p. 86). However,
based on a very small sample of 12 schools.
Smith & Tomlinson's (1989) study of multiracial comprehensives p
evidence of differential effectiveness for pupils with different levels of
(measured by second year reading test). In particular, differences in Engl
results between schools were found to be greater for pupils with above-
pupils with below-average second year reading scores. The authors concl
'largely because the exams are such that even the best school cannot
with a pupil having a below-average reading score' (p. 273). How
Tomlinson found little evidence that the slopes of schools' individ
examination results cross over. The same schools are most successful with
less able pupils, 'but a more able pupil gains a greater advantage tha
from going to a good school' (p. 273). The findings for mathematics
those for English.
Nuttall et al.'s (1989) secondary school analyses report evidence t
performance varies differentially, some schools narrowing the gap betw
high and low attainment on entry. The results suggest that variability i
pupils between schools is much larger than that of low ability pupils
(1993) original analysis of the AMA data employed a different appro
differential effects by fitting separate school intercepts for pupils categ
prior attainment groups. The results showed that the correlations b
value-added results for pupils in VR bands 1 and 3 were 0.73 for the tot
0.76 for English and 0.73 for mathematics also indicating the exis
differential effectiveness for pupils of different ability on entry to secon
study was limited, however, because the only prior attainment data avai
crude categorisation of three VR bands.
In the Scottish context, Willms & Raudenbush (1989) also report so
differential school effectiveness for pupils of different prior attainment
in an earlier study of Scottish secondary schools Willms (1986) concluded
school relationships between outcomes and pupil characteristics did
across schools' (p. 239).
Jesson & Gray (1991) investigated the issue of differential school effe
pupils with different levels of prior achievement at the secondary level
suggest that there is no conclusive evidence for the existence of differen
research provides evidence of modest differential slopes but, alth
different prior attainment levels did slightly better in some schools th
schools which were more effective for one group of pupils were generally
effective for other groups as well' (p. 46). This conclusion is broadly in
of Smith & Tomlinson (1989). Jesson & Gray (1991) suggest a num
reasons for the difference between Nuttall et al.'s (1989) and their own
draw particular attention to the high degree of social differentiation in i
and to the crude measure of prior attainment in Nuttall et al.'s (1989) re
conclude that the use of a crude grouped measure rather than a finely d
measure of prior attainment may affect findings about the nature
differential school effectiveness. However, more recent research has established
significant differential secondary school effects using finely graded prior attainment

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
measures in both inner city areas (Goldstein et al., 1993) and in county LEAs (Thomas
& Mortimore, 1996) as well as at the post-16 level in the analysis of GCE A level results
(Goldstein & Thomas, 1996). Overall, there is an increasing body of evidence showing
that some schools are not equally effective in promoting the attainment of all pupils,
irrespective of their previous strengths or weaknesses.
Gender
A few studies have pointed to the existence of differential school effects related to pupil
gender (after taking account of the impact of gender at the level of the individual). For
example, Nuttall et al.'s (1989) study of examination results over 3 years in inner
London points to the existence of such differential effects in terms of total examination
scores, 'some schools narrowing the gap between boys and girls... and some widening
the gap, relatively speaking' (p. 774). However, in the Scottish context Willms &
Raudenbush (1989), who noted differential effects for prior attainment, did not identify
any differential effects for other background characteristics, including gender. However,
a more recent study of A level results has indicated small differential effects for boys
in comparison to girls (Goldstein & Thomas, 1996).
Ethnicity
Several studies at the secondary level point to the existence of differential school effects
for pupils of different ethnic backgrounds. Nuttall et al. (1989) reported within-school
Caribbean-English, Scottish, Welsh (ESW) differences in effectiveness and comment
that other ethnic differences vary across schools even more than the Caribbean-ESW
differences: 'the Pakistani-ESW difference has a standard deviation of some 3 score
points across schools' (p. 775). However, the authors draw attention to the lack
individual-level data about the socio-economic level of pupils' families which cou
confound ethnic differences with socio-economic differences.
Elsewhere in the UK, Smith & Tomlinson's (1989) study also produced evidence of
differential school effectiveness for children of different ethnic groups, although these
differences were found to be 'small compared with differences in overall performance
between schools' (p. 268). The authors make a general conclusion about schools in their
sample: 'the ones that are good for White people tend to be about equally good for Black
people' (p. 305). These inconclusive results point to the need for further research
into ethnic differential school effectiveness using data sets which contain adequate
individual-level socio-economic as well as ethnic data.
Socio-economic Indicators
Few studies have examined the extent of within-school differential effects related to
socio-economic factors. Willms & Raudenbush (1989) report compositional effects
related to socio-economic status (SES), but no within-school differences related to such
characteristics. For differential school effects (within school differences) the few existing
secondary studies suggest that gender, ethnicity and prior attainment may all be relevant.
However, for prior attainment, it is important that the control measure adopted is
adequate (finely differentiated).
This brief review of previous research highlights the need for further detailed
longitudinal research to address the issue of differential school and departmental effects

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
for different groups of pupils whilst making adequate control for studen
believe that the results of the Differential School Effectiveness project p
contribution to current knowledge concerning this issue.
The Analysis of Differential Effectiveness

Data
The data on the GCSE examination performance, gender and age of GCSE candi
in 69 (1990), 94 (1991) and 77 (1992) [2] inner London schools (from a total of e
LEAs) were provided by the National Consortium for Examination Results. To
information was added: the London Reading Test standardised score (LRT) centr
zero, the Verbal Reasoning (VR) band assigned to each student as part of the second
school transfer procedure (i.e. a measure of general attainment on entry to seconda
school), the student's ethnicity, and whether or not he or she was entitled to free sch
meals (FSM, an indicator of low family income). In addition, the percentage of pupi
entitled to FSM (an overall approximate measure of disadvantage in a school)
provided for each school.
Seven measures of pupil performance were employed: the total examination s
(Tscore) and six subject scores (mathematics, English, English literature, scie
[maximum grade obtained], history, French). These scores were calculated in a stand
way by assigning each GCSE examination grade a numerical score (A = 7, B =
C = 5 and so on-see Nuttall et al. [1989] for a description of performance sca
employed). The mean, standard deviation [SD] and sample n for each score [3] is show
in Table I, along with the correlation between each measure and the LRT standardise
score. Some 36% of the original total sample was excluded due to incomplete
records. However, our ESRC sample in total is sufficiently representative (over 20,0
pupils) to draw valid conclusions from the results [4].
For the English and mathematics analyses the GCSE candidates in the 15 +
cohort who were not entered (or absent) for GCSE examinations in these subjects we
given an examination score of zero. The rationale underlying this procedure is that
pupils are taught English and mathematics and therefore the achievement or n
achievement of all pupils (where possible) should be included in these measur
However, for the remaining optional subjects (English literature, science, history an
French) only pupils entered for these individual GCSE examinations were included.
The Analysis
Multilevel modelling was used as the method of analysis. In its detail, this is a relati
new approach to the analysis of hierarchically structured (e.g. pupils within coh
within schools), but in broad terms is an elaboration of multiple regression to incorpo
the hierarchical structure of data (Goldstein, 1995, or for a brief guide see Paterson
Goldstein, 1991). The advantage of multilevel modelling is that it explicitly models t
hierarchical structure of the data, and also, like multiple regression, it can be used to
at potentially interesting differences, such as those between the performance of m
and females, having taken account of their attainment on entry, thus allowing a fa
comparison of like with like. Once the impact of a range of different factors has b
established, the analysis can provide an estimate of the residual (i.e. value added scor
for each school crucially. A measure of uncertainy (a 95% confidence interval) can a
be attached to each school's results, so that the residuals can be compared.

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
oHE-
%n tn %n Jn %n vo xn
0013 00 0 1 00
"04 in in W4 r
I) In in 00- Cs
N - 000Nm
13 lin i W)C 0
6 44
(5 00 0 00I 0
g 00 15C2
N ?, L 11 Xt
34 0003o-~gF r
00
w in W) " t? r IlzrP 1S. I

VII
~c3
Co~
4)
3' -n
C
'flW n~fltV'-""-' E
C)15vs1 C 4) r
(S ~ 4)3 3* n~o?
3 000 0\1 *U6
~3 c
r~S
~S ~4)
t;EI" 0
HX HiC Z3 4

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
By employing a three-level model, the remaining unexplained varia

scores, after controlling for pupil intake, can be separated into that a
(i) differences between schools (or departments), (ii) differences betwe
years), and (iii) differences between pupils. In other words the three-l
provide a single measure of effectiveness (i.e. residual) for each school (or
which represents the average performance across the three years.
The three-level [5] models employed for the analysis specified the same
for student intake across all seven performance measures (Total, Eng
literature, Mathematics, Science, History, French). The procedure of
consistent set of explanatory variables in the fixed part of the models enab
valid comparisons to be made across the school residuals for the total GCS
individual subject scores. In other words, identical statistical controls for pu
employed for each performance measure. The final models for all measure
concerning each pupil's gender, age (in months), LRT standardised sco
ethnicity and entitlement to FSM. In addition, a contextual school factor w
(the percentage of pupils entitled to FSM in each school), as well as an indi
year of examination. The overall 'goodness of fit' of the models is illus
overall reduction in total variance: total GCSE score 45.6%; mathematics 36
40.5%; English literature 40.3%; French 37.7%; history 39.0%; science
proportions of variance were accounted for by the set of intake measures
performance (Tscore) and the two English measures and lower proportions
matics, reflecting the stronger impact of background factors on performan
in language measures (see Scheerens, 1992; Sammons, 1995). However, t
a relevant measure of mathematical achievement at secondary transfer ma
impact on this finding (see Thomas et al., 1995).
Once the fixed part of the models had been established, it was then possib
existence of differential school and departmental effects for the follo
characteristics: (i) prior attainment (LRT), (ii) gender, (iii) ethnicity,
economic status (FSM). The results are presented in two sections:
(i) the extent and significance of the variance in pupil attainment across s
different pupil groups on each performance measure (addressing key q
and
(ii) the correlation between the school and departmental residuals-for diff
of pupils on each performance measure (addressing key question 2).
The Findings
(i) The Extent and Significance of Differential School and Departmenta

Total GCSE Score and Individual Subject Scores
Key question 1: Is there any evidence that some schools and departments a
tially effective in promoting GCSE attainment for different groups of
gorised by gender, ethnicity, prior attainment and entitlement to free sc
are secondary schools and departments equally effective (or ineffective)
attainment for all groups?
The fixed parameter estimates of the three-level multilevel models employe

examination score and subject score analyses have been described in de

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
TABLE II. Analysis of total GCSE score (1990-92)
Fixed part Estimate Standard error
Intercept 14.32 0.67

LSLRT 0.44 0.01
LSLRT2 0.00 0.00
VR1-VR3 13.38 0.35
VR2-VR3 5.50 0.25
Girls-Boys 2.45 0.29
Age-mth 0.05 0.02
Year91 2.59 0.31
Year92 3.85 0.33
PCTFSM 0.01 ns 0.01
FSM-no FSM - 3.33 0.29
African-white 5.47 0.51
Caribbean-white 0.63 ns 0.34
Indian-white 8.22 0.52
Pakistani-white 10.50 0.66
Bangladeshi-white 9.24 0.64
Other-white 5.87 0.35
Notes: Pupils = 17,850; schools = 94

Where: LSLRT = London Standardised Lond
VR1 = Verbal Reasoning Band 1 (appr
FSM = Entitlement to free school meals
PCTFSM = Percentage of pupils in school entitled to FSM
ns = Not significant (at 0.05 level)
(Thomas et al., 1997). However, as an example of the results, Table II shows the average
impact of each explanatory variable on one outcome measure, the total GCSE score. As
reported in previous studies (e.g. Nuttall et al., 1989; Goldstein et al., 1993; Thomas et
al., 1994a, 1995), on average, girls perform at a higher level than boys. In contrast,
disadvantaged pupils (i.e. those entitled to FSM) as well as pupils with low LRT
attainment perform at a lower level at GCSE than all other pupils.
Moreover, on average, all ethnic groups perform at a higher expected level than the
white group (all except Caribbean are significant at 0.05 level) given that any differences
between these groups in their attainment on entry to school have already been taken into
account. One explanation for these findings is that some bilingual pupils (from particular
ethnic groups) start school as relatively low attainers (in verbal reasoning) and make
substantial progress in language skills while attending secondary school. This important
issue is discussed in more detail by Thomas et al. (1997) and Sammons (1995).
Once the average impact of the pupil intake factors has been established, it is then
possible to estimate accurately the differential effects in school performance over a
3-year period for different groups of pupils. By utilising data over 3 years the statistical
power of the analysis is boosted, as well as strengthening the specification and
generalisability of the multilevel model.
The initial results of the multilevel analyses provide strong evidence of significant
differences between schools in their effects on all pupils' total GCSE performance and
subject scores. This is illustrated in Table III for the 1990-92 results [6], which shows
the overall percentage of variation in pupil outcomes due to (i) differences between

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
TABLE III. Percentage of variance in pupils' total GCSE score and subject score due to
% variance due to school % variance due to year

1990-92 1990-92
Total GCSE performance score 6.2 1.1

English 4.1 1.8
English literature 6.9 3.4
Mathematics 5.9 3.6
Science 6.1 4.7
French 7.8 7.8
History 15.3 3.6
Note: 3-level model using
schools (or department

to examine differenti
whether the variatio
existence of differen
some schools. As a pr
appears to be highly e
in fact, be significant
Previous research usin
has reported significan
level according to pup
in general, the more e
ability pupils to a grea
intake characteristic
confirmed and it was f
level) for total GCSE sc
by showing the scho
variables) for the least
able pupils on entry
represents one school.
10-
ci, *
= 5-. - "
m -
a a,? ? +
-15~~? , .... ... -
-5 -4 -3 -2 -1 0 1 2
Low LR puil
FI. . otl CS coe.Scoo rsdulsfo lw RTsorngan hghLT coin ppis
- ---I 1 1
. -5 3 -2 0C2 eai 4 5
Low LRT pupil

"FIG. 1. Total GCSE score.

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
schools are better at promoting examination success for less able pupils than for more
able pupils (and vice versa).
In addition to the LRT student intake measure, differential school and departmental
effects were simultaneously tested for (i) boys vs girls, (ii) white vs other [7] and white
vs Caribbean, and (iii) pupils not entitled to FSM vs pupils entitled to FSM. It is
important to note that the findings reflect the significance of pupil grouping factors tested
simultaneously in order to provide a stringent test of differential effects and also to allow
for the possibility of variables being confounded (e.g. pupils entitled to FSM are more
likely to be low attainers).
Table IV shows that in addition to the prior ability measure school and departmental
results were found to vary differentially (significant at 0.05 level) in the performance of
pupils entitled to FSM (versus pupils not entitled to FSM) for total GCSE score, English
and mathematics. For pupils of Caribbean (or other) ethnic background significant
differential school effects (at 0.05 level) were found for total GCSE score, mathematics,
English, English literature, French and science. Gender was also found to be significant,
with differential effects being identified for total GCSE score, English, English literature
and mathematics. Table IV also shows the extent of the variation (i.e. standard deviation)
in school and departmental residuals estimated by the multilevel analysis. The standard
deviation can be interpreted as a measure of the range in the expected GCSE point scores
for the most effective and least effective schools (and departments) for particular pupil
groups. For example, the results show that having taken into account the fixed and
random effects of all other factors, the total GCSE performance of the average LRT
pupil has a standard deviation of 3.5 score points across schools.
Importantly Table III shows that some of the variation in student scores is
attributable to average changes in results over time (3 years) across all schools
(ranging from 1.1% to 7.8%). Therefore, it was also necessary to test whether some
school results varied much more (or less) over time than other schools. For this
purpose a year indicator (1990 = 0; 1991 = 1; 1992 = 2) was fitted as a random
coefficient at the school level to examine whether the time trend for each performance
measure varied significantly across schools (over and above the average variation due
to time estimated at level 2). Significant differences across schools were found in
trends over the 3-year period for English, English literature and mathematics but not
for total GCSE score, French, history and science. This finding shows that for three
outcome measures (English, English literature and mathematics) significant increasing
or decreasing trends in GCSE attainment can be identified for some schools. In other
words some schools appear to be improving over time (in comparison to the average
school) in terms of promoting student GCSE attainment in one of the core curriculum
subjects of English and mathematics and others appear to be declining. This finding
highlights the need for further research into the processes associated with improving
schools and departments (see Gray et al., 1996 for an analysis and discussion of
schools' results over 5 years).
Thus, in terms of key question 1 it appears that schools and departments (especially
in terms of total GCSE score, English and mathematics) are indeed differentially
effective for different groups of pupils and this finding strongly suggests the need for
schools to examine in detail the value added performance of different groups of pupils
in their schools, especially by level of prior attainment, gender, low income and
ethnicity. They also demonstrate the existence of differences between schools in perform-
ance over time (the year or cohort effect) and the importance of monitoring trends in
GCSE performance using overall measures, and specific subjects over several years.

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
N 0 0, 0tl tlt,
C 6i dC d V dV
Ma
CC
?s - CC z, cn 5,
M3
U)L
S zr
r.,-e4e4 dd o
o ~ Z~ 13 0
0
._r.
C)
C,, C4 V),,Vo
U)
V) d-0
3 C#) -002Z c Z. C 6
U)a 9
Cd
C) E
C/) Oh.Un
U) E
Su
C-E U)cbE
3 c. E
otn 'tt y)
U 'T Nc
) u 0
t io
~ 63C )
0j C-' \Od\ v r Eo
5 o
E ~ ~ ~ - U)11 U) lm m L
pi C: V y V V1V~~ V V V
H, 'LCL Zc EZC#~

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
11
CL
Z %
C
u.
30 -1.
U-
0) ?4I
E
-41
C I
M Sa a C~orrelation_=O.90~
-63 a? "a
7 ? - - - - __-_1-,_- ,_I
4 6 8 10 12 14
Gap in atta
FIG. 2. Total
(ii) Compar
and Subjec
Key questio
of results
attainment
in terms o
The multile
effects (or,
minus non-
ation of th
be describe
Fairly stro
intercepts
French and
Tscore) wer
LRT pupils)
vice versa).
in school r
'Caribbean
mathematic
Interestingl
groups (see
difference
attainmen
versa). This
residuals fo
disadvantag
(and vice ve
for FSM p
Negative co

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
V-4 M "m4~ o d
~ 2 T: $ ZZ~
00
4). AC..AZ ZZZZ ZZZ ZZ Z
Q " ZZZ~,Z 2 3:S:ZZ ZZ Z:

0 0 tON
z izo zzz zz z
C? W) (ON C h 0 N %n
. o d d o d d'.dd cr' d
40-
o z
o vl (odd ~n~ d~ rY vrr?-1 fL

4) c,,
SO0 44pr
mt \ I r-Im (I 1 %n It 0 !O
d .- d dd dc r-0 -4 ON
C3. .Y i t t l
- 0 0 \ N (CI'0 00 Ocz -0r \O?J t~~t d Vc 3C

o 666 oo ,66
o n cn c l MI II I
r . 00Ch J \O00 r ~J ON '00,~ 00-Cr tf
lit ii lii
C. oo C) C o C ~ CS
vl p-.-dd dc d
0. Z
00m T. ' i
4) H z
U'- 0L OV~- 1 OZ m

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
10
8 - Correlation =-0.73
6-
Cl) . 3 ? ?
"a. 4 3 3
0 so
a) * mm
S> -41
-861
-10
-7 -6 -5 -4 -3 -2 -1 0
Gap in attain
FIG. 3. Total GCSE sc
of FSM and non-FS
minority groups
Tscore, English
findings suggest
attainment (for a
between disadvan
FSM coefficient
minority and 'wh
high GCSE atta
between disadva
narrowing the ga
in Fig. 3 for FSM
However, in line
residuals for spec
ethnic minority
example, the co
and non-FSM pu
GCSE residuals f
These findings a
added scores fo
effects for boys
the correlation
positive (0.97).
These findings a
departments) tha
in overall GCSE a
the gap in attain
pupils is smaller
of pupils who are
pupils in more ef
pupils) boost the
though the gap

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
8 ,
6-
2 0L
C -2i 9
'Other' e
FIG. 4. Total G
analyses on
inclusion in
parental educ
with parents
likely to per
more effect
analyses that
of social cla
Thus, in ter
some extent
different lev
low attainer
and departm
relatively w
are likely t
effective sch
likely to per
'other' ethni
interpretatio
and identify
However, w
analysis of (
(or gap) in a
Conclusions and Discussion
We hope that our detailed analysis of GCSE results based on 77 schools in 1

1991 and 69 in 1990 provides a useful contribution to the debate about the inter
of schools' examination results. The findings are fairly complex and extend th
of a previous analysis (Thomas et al., 1997). Our conclusions are summaris
There is strong evidence of differential school effects and department effect
attainment and ethnicity for total GCSE score, English, mathematics, English

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
French and science results across 3 years. Furthermore, these differences may be masked
in analyses which use only a single measure of effectiveness for all pupils within a
school or department. Evidence for differential school and departmental effects was also
found for groups of pupils according to low income (effects were identified for total
GCSE score, English, mathematics) and gender (effects were identified for total GCSE
score, English, mathematics and English literature).
The correlations reported between schools' value added scores (i.e. residuals) for
different groups of pupils tend to be strong and positive but they are not perfect (e.g.
r = 0.81 shown in Fig. 1, r = 0.89 shown in Fig. 4). This suggests that in more effective
schools (for average LRT pupils) all pupils appear to perform well but that some pupils
(e.g. non-FSM pupils) are likely to benefit more than others. In contrast, it appears that
in less effective schools (for average LRT pupils) all pupils are more likely to perform
poorly but that some pupils (e.g. Asian pupils) are less likely to fail in terms of GCSE
performance. These findings relate to inner London schools and require further
confirmation and clarification in studies that include the GCSE results of a wider range
of schools (such as in county LEAs).
Despite evidence of broad stability from previous research (Thomas et al., 1997), the
results indicate that there is also a substantial degree of improvement or decline over
time in some schools' and departments' performance (over and above average changes
identified for all schools) and point to the value and necessity of looking at results over
several years. Our conclusions that some schools appear to be improving (or declining)
are in line with previous studies reporting estimates of change in school results over time
(such as Gray et al., 1993, 1996; Goldstein et al., 1995). However, as Gray & Wilcox
(1995) have pointed out, how an 'ineffective' school improves may well differ from the
ways in which more effective schools maintain their effectiveness. Reynolds (1992) has
drawn attention to the fact that ineffective schools may differ in fundamental ways from
those which are more effective. Moreover, school improvement is a (relatively) less
well-researched area than school effectiveness. Further research is required to examine
the pattern of changes in schools' effectiveness from year to year in relation to school
policy and process factors.
We conclude that the importance of taking account of background factors and prior
attainment using appropriate value added models in order to estimate the influence of the
school is clear. Also, there is only limited value in trying to make judgements about
schools' examination performance in any one year or using only one performance
measure. Instead the monitoring of internal variations in performance (adjusted for prior
attainment) in different departments and for different groups of pupils in any year and
across years should be encouraged. For example, Thomas et al. (1997) found that only
9% of schools were categorised as either highly effective or highly ineffective across
three separate GCSE cohorts and across a total GCSE measure and six GCSE subject
measures. These findings are important in highlighting the lack of detail and usefulness
of the crude school league tables published annually by the Department for Education
and Employment. In contrast, LEAs, such as Lancashire LEA, have for several years
supported and developed projects which provide secondary schools with detailed value
added measures and the associated confidence intervals required to evaluate school and
departmental effects for different groups of pupils and trends over time (see Thomas &
Mortimore, 1996). We would argue that this approach is extremely valuable both in
stimulating school improvement efforts generally and in providing a means to examine
the impact of specific policies for equal opportunities and access in education.
Finally, the results point to the broader picture and the need for further comparative

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
work in a range of socio-economic and geographical contexts. Most imp

researchers have noted that further evidence is required on the topic of
and stability of schools' effects in terms of a wider range of outco
vocational and social measures in addition to academic measures-to refle
the aims of schooling (e.g. Scheerens, 1992; Mortimore et al., 1994
Sammons et al., 1995c; Gray & Wilcox, 1995).
Correspondence: Sally Thomas, International School Effectiveness an

Centre, University of London Institute of Education, 20 Bedford Way,
0AL, UK.
NOTES
[1] Funding for this research has been provided via grants awarded by the Economic and Social
Research Council (R000234130 and H53627504395).
[2] Sixty-nine schools provided data for 1990, 1991 and 1992.
[3] The scores were approximately normally distributed and preliminary analyses indicated that th
of normal scores (with mean zero and standard deviation 1) had little impact on the significan
relative magnitude of estimates. Not using normal scores has the added benefit that the result
be interpreted more easily in terms of GCSE grades.
[4] Specific details regarding the multilevel sample can be found in Thomas et al. (1997).
[5] Level 1: candidates; level 2: cohorts; level 3: schools.
[6] The percentage of variance attributable to years and schools was calculated using the
variance components model (i.e. intercept only fitted in random part at level 1: candidates, 2:
and 3: schools) including fixed part explanatory variables: LRT, LRT', VR Band, gender, age
ethnicity, %FSM, Year 1991, Year 1992 (the final model).
[7] Earlier analyses indicated that it was necessary to collapse the original ethnicity categ
as follows: Caribbean = Caribbean; other = African, Indian, Pakistani, Bangladeshi,
white = white.
[8] See note 7-i.e. not Caribbean and not white.
REFERENCES
GOLDSTEIN, H. (1995) Multilevel Statistical Models, 2nd edn (London, Edward Arnold
Halsted Press).
GOLDSTEIN, H. & THOMAS, S. (1996) Using examination results as indicators of sch
performance, Journal of the Royal Statistics Society A, 159, pp. 149-163.
GOLDSTEIN, H., GRAY, J. & JESSON, D. (1995) Changes and improvements in schools'
trends over five years, paper presented at the British Educational Research Asso
Conference, Bath, 14-17 September.
GOLDSTEIN, H., RASBASH, J., YANG, M., WOODHOUSE, G., PAN, H., NUTTALL, D. & TH
A multilevel analysis of school examination results, Oxford Review of Education, 1
GRAY, J. & WILCOX, B. (Eds) (1995) Good School Bad School: evaluating performance an
improvement (Buckingham, Open University Press).
GRAY, J., GOLDSTEIN, H. & JESSON, D. (1996) Changes and improvements in schools'
trends over five years, Research Papers in Education, 11, pp. 35-51.
GRAY, J., JESSON, D., GOLDSTEIN, H., HEDGER, K. & RASBASH, J. (1993) A multi-level an
improvement: changes in schools' performance over time, paper presented at th
Conference of the European Association for Research on Learning and Instruction, A
France, 3 September.
JESSON, D. & GRAY, J. (1991) Slants on slopes: using multi-level models to investigate dif
effectiveness and its impact on pupils' examination results, School Effectiveness and
ment, 2, pp. 230-271.
MCPHERSON, A. (1992) Measuring added value in schools, National Commission on Edu
No. 1, February (London, NCE).

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC
MORTIMORE, P., SAMMONS, P. & THOMAS, S. (1994) School effectiveness and value added measures,
Assessment in Education, 1, pp. 315-332.
NUTrALL, D., GOLDSTEIN, H., PROSSER, R. & RASBASH, J. (1989) Differential school effectiveness,
cha. 6 in International Journal of Educational Research, special issues: Developments in School
Effectiveness Research, 13, pp. 769-776.
NUTrALL, D., THOMAS, S. & GOLDSTEIN, H. (1992) Report on Analysis of 1990 Examination Results,
January 1992 (London, Association of Metropolitan Authorities).
PATERSON, L. & GOLDSTEIN, H. (1991) New statistical methods of analysing social structure: an
introduction to multilevel models, British Educational Research Journal, 17, pp. 387-393.
REYNOLDS, D. (1992) School effectiveness and school improvement: an updated review of the British
literature, in: D. REYNOLDS & P. CUTTANCE (Eds) School Effectiveness Research, Policy and Practice
(London, Cassell).
RUTTER, M., MAUGHAN, B., MORTIMORE, P. & OUSTON, J. (1979) Fifteen Thousand Hours: secondary
schools and their effects on children (London, Open Books).
SAMMONS, P. (1995) Gender, socio-economic and ethnic differences in attainment and progress: a
longitudinal analysis of student achievement over 9 years, British Educational Research Journal, 21,
pp. 465-485.
SAMMONS, P., HILLMAN, J. & MORTIMORE, P. (1995c) Key Characteristics of Effective Schools: a review
of school effectiveness research (London, Office for Standards in Education).
SAMMONS, P., NUrrALL, D. & CUTTANCE, P. (1993) Differential school effectiveness: results from a
reanalysis of the Inner London Education Authority's Junior School Project Data, British Educational
Research Journal, 19, pp. 381-405.
SAMMONS, P., NUTrrTALL, D., CUTTANCE, P. & THOMAS, S. (1996) Continuity of school effects: a
longitudinal analysis of primary and secondary school effects on GCSE performance, School Effective-
ness and School Improvement, 6, pp. 285-307.
SAMMONS, P., THOMAS, S. & MORTIMORE, P. (1995b) Accounting for variations in academic effectiveness
between schools and departments: results from the 'Differential Secondary School Effectiveness
Project'-a three year study of GCSE performance, paper presented at the European Conference on
Educational Research/British Educational Research Association Annual Conference, Bath, 14-17
September.
SAMMONS, P., THOMAS, S. & MORTIMORE, P. (1997) Forging Links: effective schools and effective
departments (London, Paul Chapman) (in press).
SAMMONS, P., THOMAS, S., MORTIMORE, P., CAIRNS, R., BAUSOR, J. & WALKER, A. (1995a) Understanding
school and departmental differences in academic effectiveness: findings from case studies of selected
outlier secondary schools in inner London, paper presented at the Annual Conference of the
International Congress of School Effectiveness and Improvement, Leeuwarden, Netherlands, 3-6
January 1995, and submitted to School Effectiveness and School Improvement.
SCHEERENS, J. (1992) Effective Schooling: research, theory and practice (London, Cassell).
SMITH, D. J. & TOMLINSON, S. (1989) The School Effect: a study of multi-racial comprehensive (London,
Policy Studies Institute).
THOMAS, S. (1995) Optimal Multilevel Models of School Effectiveness Comparative Analyses across
Regions, ESRC Fellowship Proposal (London, Institute of Education, University of London).
THOMAS, S. & MORTIMORE, P. (1996) Comparison of value added models for secondary school
effectiveness, Research Papers in Education, 11, pp. 5-33.
THOMAS, S., NUTrALL, D. & GOLDSTEIN, H. (1993) Report on Analysis of 1991 Examination Results, June
1993 (London, Association of Metropolitan Authorities).
THOMAS, S., PAN, H. & GOLDSTEIN, H. (1994a) Report on Analysis of 1992 Examination Results, June
1994 (London, Association of Metropolitan Authorities).
THOMAS, S., SAMMONS, P. & MORTIMORE, P. (1994b) Stability and consistency in secondary schools'
effects on students' GCSE outcomes: initial results, paper presented at the Annual Conference of the
British Educational Research Association, St Anne's College, University of Oxford, 9 September.
THOMAS, S., SAMMONS, P. & MORTIMORE, P. (1995) Determining what 'adds value' to student achieve-
ment, Educational Leadership International, 52, pp. 19-22.
THOMAS, S., SAMMONS, P., MORTIMORE, P. & SMEES, R. (1997) Stability and consistency in secondary
school effects on students' outcomes over 3 years, School Effectiveness and School Improvement, 8,
pp. 169-197.
WILLMS, J. D. (1986) Social class segregation and its relationship to pupils' examination results in
Scotland, American Sociological Review, 51, pp. 224-241.
WILLMS, J. D. & RAUDENBUSH, S. W. (1989) A longitudinal hierarchical linear model for estimating
school effects and their stability, Journal of Educational Measurement, 26, pp. 209-232.

62.228.110.229 on Sat, 20 Nov 2021 13:24:51 UTC

Differential Secondary School Effectiveness (Comparing The Performance of Different Pupil Groups)

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Differential Secondary School Effectiveness (Comparing The Performance of Different Pupil Groups)

Uploaded by

Copyright:

Available Formats

Differential Secondary School Effectiveness: Comparing the Performance of Different

Stable URL: https://www.jstor.org/stable/1502081

This content downloaded from