Teaching and Teacher Education 88 (2020) 102971

Contents lists available at ScienceDirect

Teaching and Teacher Education

journal homepage: www.elsevier.com/locate/tate

Professional development on fostering students’ academic language

proficiency across the curriculumdA meta-analysis of its impact on
teachers’ cognition and teaching practices
Eva Kalinowski a, *, Franziska Egert b, Anna Gronostaj a, 1, Miriam Vock a
a €t Potsdam, Empirische Unterrichts- und Interventionsforschung, Karl-Liebknecht-Str. 24e25, 14476 Potsdam, Germany
Universita
b
Katholische Stiftungshochschule München, Don-Bosco-Str. 1, 83671 Benediktbeuern, Germany

h i g h l i g h t s

Focuses teacher training on promoting academic language support in content areas.

Effects of 11 training interventions for in-service teachers are aggregated.

Meta-analyses reveal effects on teachers’ cognition und teachers’ practices.

Methodological quality of studies moderates the effect magnitude.

a r t i c l e i n f o a b s t r a c t

Article history: This meta-analysis aggregates effects from 10 studies evaluating professional development interventions
Received 25 November 2018 aimed at qualifying in-service teachers to support their students in mastering academic language skills
Received in revised form while teaching their respective subject areas. The analysis of a subset of studies revealed a small non-
23 September 2019
significant weighted training effect on teachers’ cognition (g’ ¼ 0.21, SE ¼ 0.14). An effect aggregation
Accepted 7 November 2019
including all studies (with 650 teachers) revealed a medium to large weighted overall effect on teachers’
Available online 2 December 2019
classroom practices (g’ ¼ 0.71, SE ¼ 0.16). Methodological variables moderated the effect magnitude.
Nevertheless, the results suggest professional development is beneficial for improving teachers’ practice.
Keywords:
Professional development
© 2019 Elsevier Ltd. All rights reserved.
Language
Cross-curriculum
Content areas
In-service teacher training

1. Introduction English proficiency is substantially influenced by a language other

The ability to use the type of language used in school is
considered a major prerequisite for educational success
(Schleppegrell, 2004). This appears problematic in light of the
number of students who have difficulty with the language used at
school. For example, in the US, English language learners (ELLs)
alone, whose native language is not English or whose level of
* Corresponding author.
E-mail addresses: eva.kalinowski@uni-potsdam.de (E. Kalinowski), franziska.
egert@ksh-m.de (F. Egert), anna.gronostaj@deutsche-schulakademie.de
(A. Gronostaj), miriam.vock@uni-potsdam.de (M. Vock).
1
Author’s affiliation has changed since the work described in the article was
conducted. Present address: Die Deutsche Schulakademie gGmbH, Hausvogteiplatz
12, 10117 Berlin, Germany.
than English, make up approximately 10% of all public school stu-
dents (McFarland et al., 2018). In fact, there is a persistent
achievement gap in educational attainment between such students
and those who have access to the language of instruction in their
home contexts. This also applies to children from socially disad-
vantaged families (McFarland et al., 2018; OECD, 2016). To close
these gaps, policies and concepts calling on teachers to integrate
language support into their subject area teaching have been
established (see e.g., Carnegie Council on Advancing Adolescent
Literacy, 2010; Cheuk, 2016; Van Roekel, 2011). Considerable ef-
forts and investments in teacher professional development (PD) are
made to qualify teachers for this challenging task (see e.g., National
Clearinghouse for English Language Acquisition, 2017; U.S.
Department of Education, 2017; and Education through Speaking
and Writing [BiSS], 2014, in Germany). In most US states and in

https://doi.org/10.1016/j.tate.2019.102971
0742-051X/© 2019 Elsevier Ltd. All rights reserved.
2 E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971
other countries as well, teachers are required to participate in some
amount of PD, e.g., to maintain their teaching licenses (Hoffman &
Harris, 2018; OECD, 2009). PD endeavors are rooted in the
assumption that they benefit teachers’ competence development
and improve teaching quality. Although there is evidence that
teacher PD can foster positive change (Hattie, 2009), it is not
necessarily effective in all cases (Jacob & McGovern, 2015; Wei,
Darling-Hammond, Andree, Richardson & Orphanos, 2009).
Instead, PD success depends on various factors, such as character-
istics of the PD intervention and individual school and teacher
characteristics, and can differ substantially across domains
(Lipowsky, 2014; Timperley, Wilson, Barrar, & Fung, 2007). Never-
theless, policymakers and school leaders need to ensure that
available PD opportunities are effective (OECD, 2009). In the spe-
cific field of language development in academic content classes,
individual PD programs have been evaluated, but to our knowledge
no comprehensive overview of the entirety of existing evaluation
studies exists for this specific field until now. Thus, the aggregate
effect of such PD initiatives is unclear. The present study was
conducted to close this research gap and investigate what quanti-
tative studies tell us about the effectiveness of PD aimed at
fostering students’ language proficiency across the curriculum, and
more specifically, the effect such PD has on teachers.
1.1. Language proficiency in teaching and learning processes
Language is an essential medium for teaching and learning.
Whereas language skills used in everyday or home life also play a
role at school, typical tasks at school, such as analyzing a text,
explaining a phenomenon, or writing an essay, require proficiency
in academic language (Cummins, 2008; Francis, Rivera, Lesaux,
Kieffer, & Rivera, 2006; Schleppegrell, 2001, 2009), “the language
used in schooling for purposes of learning” (Schleppegrell, 2009, p.
3). Academic language is characterized by specific grammatical,
lexical and discursive features associated with academic contexts,
involves mastery of written and oral discourse and encompasses
literacy focused on higher-order skills instead of basic or technical
skills, such as decoding single words (Fillmore & Snow, 2000;
Scarcella, 2003; Schleppegrell, 2001). Anstrom et al. (2010)
comprehensively describe different conceptualizations of aca-
demic language.
Studies indicate that academic language poses more difficulties
than everyday language for all learners. Furthermore, students’
sociocultural backgrounds seem to affect their language proficiency
(Heppt, Henschel, & Haag, 2016; Townsend, Filippini, Collins, &
Biancarosa, 2012; Uccelli, Galloway, Barr, Meneses, & Dobbs,
2015): Students whose language experiences at home are not
aligned with those at school are less likely to meet the re-
quirements essential for school success. This particularly concerns
students from low socioeconomic backgrounds and other students
growing up in an environment where a different language than the
one used in educational institutions is dominant, such as ELLs. Since
language proficiency and particularly academic language profi-
ciency affects students’ academic performance (Gogolin & Lange,
2011; Heppt et al., 2016; Townsend et al., 2012) but academic lan-
guage skills are usually not explicitly taught in schools but rather
implicitly presupposed (Brisk & Zisselsberger, 2011; Feilke, 2012;
Schleppegrell, 2001; Schmo € lzer-Eibinger, 2013), these students are
at particular risk of low academic achievement (McFarland et al.,
2018; Stanat, Weirich, & Radmann, 2012).
The situation that teachers usually do not explicitly teach the
language skills fundamental for learning at school comes with the
fact that most teachers are ill-prepared for this challenging task
(Bunch, 2013; Quintero & Hansen, 2017; Samson & Collins, 2012).
To make content accessible to all students and reduce the above-
mentioned risk, teachers should provide directed support to stu-
dents at all grade levels in elementary and high schools and from all
socioeconomic and linguistic backgrounds, and especially to stu-
dents who do not encounter academic language at home. In order
to better account for the interconnected development of language
and content knowledge and to avoid segregation, many policies and
researchers recommend instructional approaches that integrate
academic language and literacy instruction into subject-area
teaching (Becker-Mrotzek, Schramm, Thürmann, & Vollmer, 2012;
Chamot & O’Malley, 1987; Fillmore & Snow, 2000; Lucas & Villegas,
2011; U.S. Department of Education, 2016; Valde s, Kibler, & Walqui,
2014). Corresponding programs and concepts have been developed
in recent years. These programs are based on the simultaneous
learning of academic language and subject-area content, thus
making content accessible to all students. Effective teacher PD in
this field is essential in order to qualify teachers.
1.2. Effectiveness of teacher professional development
The term PD is used inconsistently in the literature. In this study,
it “designates any purposeful, to some extent face-to-face,
formalized and organized learning and/or training opportunity
for in-service teachers” (Kalinowski, Gronostaj, & Vock, 2019, p. 3).
Researchers, particularly in the US, have started to examine the
effectiveness of PD specifically focusing on language support in
subject-area teaching. Parts of this evaluation literature have
already been summarized. For instance, Knight and Wiseman
(2005) examined several peer-reviewed US studies on PD for
teachers of (linguistically) diverse students. They assumed that
teacher PD that is not tailored to the specific target group might be
inappropriate for diverse student groups and concluded that
research on such PD and evidence for determining the effectiveness
of different PD approaches was lacking. A few years later, Bunch
(2013) focused on the pedagogical language knowledge main-
stream teachers of English learners should possess and found
research claims regarding outcomes to be quite varied and limited.
DiCerbo, Anstrom, Baker, and Rivera (2014) reviewed the literature
on teaching academic English to ELLs, including a few studies on
teacher PD. They conclude that the reviewed studies suggest
beneficial impacts of high-quality PD. Zhang (2014) synthesized 25
studies on PD for teachers of ELLs and found that these studies
primarily reported successes. Similarly, a systematic review by
Kalinowski, Gronostaj, & Vock, 2019 on teacher PD to foster stu-
dents’ academic language proficiency in content area classes found
some degree of positive impact for all evaluated interventions,
although much of the primary research was based on small sample
sizes. To date, none of the existing research syntheses have quan-
tified the effects of PD interventions that specifically focus on
integrating academic language and literacy instruction into
subject-area teaching in a meta-analytic way.
1.2.1. Outcomes of professional development
Generally, the outcome levels and constructs used in PD eval-
uation studies vary greatly. Teacher PD effectiveness is usually
measured on one or several of the following levels: (1) teachers’
satisfaction with or acceptance of the PD intervention; (2) teachers’
cognition, i.e., changes in knowledge, beliefs, motivation, etc.; (3)
teachers’ classroom practices; and (4) student achievement and
development (e.g., Guskey, 2000; Lipowsky & Rzejak, 2015; Wade,
1984). Across various domains, teacher PD programs have been
found to have an impact on the student level, which can be
considered the ultimate goal of teacher PD (Birman, Desimone,
Porter, & Garet, 2000). In his meta-analysis, Hattie (2009)
computed a substantial overall teacher PD effect of d ¼ 0.62.
However, meta-analytical evidence on the effects of PD varies
 E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971
considerably across domains, such as literacy and science (e.g.,
Timperley et al., 2007). We will present the results of primary
studies quantitatively evaluating PD aimed at qualifying teachers to
foster students’ language and literacy skills in subject areas ac-
cording to the four aforementioned levels as follows.
Level 1: Satisfaction and acceptance. Some evaluation studies
have examined teachers’ reactions to and satisfaction with the
intervention in question in a descriptive way. They mostly reported
positive teacher feedback (e.g., He, Prater, & Steed, 2011; McIntyre,
Kyle, Chen, Mun ~ oz, & Beldon, 2010). However, it is generally not
possible to quantify effects on this level in a (quasi-)experimental
way, because it is obviously not meaningful to compare teachers’
assessment of a given PD program with their assessment before the
intervention or with non-participants.
Level 2: Teachers’ cognition. With respect to cognition, there are
several areas that one would expect to be affected by PD in-
terventions focused on fostering students’ language skills in
subject-area classes, in particular teachers’ self-efficacy, beliefs, and
attitudes, as well as their knowledge. Cognition may be defined as
comprising beliefs as well as knowledge (see e.g., Borg, 2006) and
self-efficacy may be defined as a belief (Guskey & Passaro, 1994). To
add to this, according to Bandura’s (1993) social-cognitive theory,
self-efficacy is cognitively generated. Some positive findings for
these areas of teacher cognition exist. The results vary in terms of
statistical significance and effect size, however. For example,
Cantrell and Hughes (2008) studied an intervention’s impact on
different measures of teachers’ self-efficacy, i.e., their “belief or
conviction that they can influence how well students learn”
(Guskey & Passaro, 1994, p. 628), for literacy teaching. They re-
ported significant improvements from pre- to post-test. Hart and
Lee (2003), as another example, examined PD impact on the
importance teachers ascribed to incorporating language and liter-
acy into science instruction and found mixed results: Pre- and post-
data comparisons indicated significant increases for reading and
writing with a small effect, but no significant change for grammar.
Greenleaf et al. (2011) studied effects on different aspects of
teachers’ teaching philosophy (among various other constructs),
including the degree to which teachers view diverse language
backgrounds as a potential asset. They reported effects of inter-
vention and control group comparisons between non-significant
ES ¼0.24 and significant ES ¼ 0.76. Lee and Maerten-Rivera
(2012) examined teachers’ knowledge of the subject matter.
Teachers need subject knowledge to be able to enhance students’
language skills and understanding of the subject at the same time.
Lee and Maerten-Rivera (2012) found slight but non-significant
growth in teachers’ science knowledge during the first year of the
intervention but no further increase in subsequent years.
Level 3: Classroom practices. During PD, teachers may also be
expected to develop specific skills to foster (academic) language
proficiency across the curriculum. According to the empirically
based Teaching through Interactions model, which considers
teacher-student interactions to play a crucial role in student
learning in lessons (Hamre et al., 2013), students’ language learning
is related to the quality of language modeling and instructional
support in teacher-student interactions (e.g., opportunities to ex-
press existing skills, feedback and scaffolding). PD effects on
language-supportive teaching practices are measured in different
ways (based on different theories of language and literacy), but the
techniques examined are instructional strategies that likely predict
the development of students’ language and literacy skills. Such
strategies are manifold and include the promotion of inquiry,
proper use of language structures, engaging students in literacy-
related activities, and providing students with opportunities for
meaningful and authentic academic talk, such as making reasoned
arguments and negotiating ideas (Greenleaf et al., 2011; Lee &
3
Maerten-Rivera, 2012; Shanahan & Shea, 2012).
Overall, the results concerning these sort of classroom practices
have been heterogeneous, that is to say, they do not point in one
direction, as illustrated by the following examples. Greenleaf et al.
(2011), who examined “the literacy challenge offered to students”
(p. 680), reported significantly higher ratings on some measures for
intervention group teachers compared to control group teachers,
but also several non-significant results. Other examined teaching
practices include initiating collaborative activities, introducing key
vocabulary, using visuals, and employing a slower pace of discourse
and clearer enunciation (Choi & Morrison, 2014; Echevarria &
Graves, 2007; Echevarria, Vogt, & Short, 2004; Gibbons, 2006;
Haynes, 2007; Herrera & Murry, 2005; Miramontes, Nadeau, &
Commins, 1997). These practices are variously referred to as En-
glish-as-a-second-language strategies (Anderson, 2009) and features
of sheltered instruction (Crawford, Schmeister, & Biggs, 2008).
Another term used is linguistic scaffolding (Hart & Lee, 2003; Lee &
Maerten-Rivera, 2012), which emphasizes the responsiveness of
teachers’ support to the particular demands made on students
(Gibbons, 2015). For example, Hart and Lee (2003) reported pre-
post mean differences indicating significant positive change in
the linguistic scaffolding provided by teachers with a small effect.
In contrast, they did not find significant change in instructional
strategies to encourage reading and writing. Other components
that have been studied include teachers’ use of cognitive strategies
in literacy activities during their lessons (Olson et al., 2012) and
engaging students in metacognitive conversations about reading
processes or reflections on learning (Cantrell & Hughes, 2008;
Greenleaf et al., 2011). Considerable PD effects have been demon-
strated for different measures related to both. Another language
support strategy studied in existing primary research is teachers’
use of reasoning and sophisticated and varied vocabulary. Henrichs
and Leseman (2014) reported intervention effects between non-
significant ES ¼0.27 and significant ES ¼ 1.25 for several specific
topics and areas. These examples show the wide variety of
dependent variables that can be used to capture teachers’ class-
room practices.
Level 4: Student achievement. Several studies that evaluated PD
aimed at increasing students’ comprehension of content while
facilitating language acquisition reported positive changes in stu-
dent outcomes, e.g., students’ language proficiency and content
knowledge (e.g., August et al., 2014; Lara-Alecio et al., 2012).
However, using this level of evaluation has potential risks, partic-
ularly when no data on the teacher level is available. Teacher PD
and student achievement are assumed to be linked only indirectly
(Yoon, Duncan, Lee, Scarloss, & Shapley, 2007). In the case of null
results, it is impossible to determine whether improvements at the
teacher level emerged that just did not further affect student
achievement. Furthermore, PD effectiveness on the student level
might be confounded with several other factors, such as the
effectiveness of the student support activities teachers are sup-
posed to learn about in PD. The latter is particularly problematic in
this field, because the effectiveness of language support strategies
and programs has not yet been sufficiently evaluated (Paetsch,
Wolf, Stanat, & Darsow, 2014). Thus, differences in the effective-
ness of different student support activities might affect how stu-
dents benefit from teacher PD programs.
1.2.2. Potential moderators and sources of bias
Due to different outcome constructs and measures, it remains
unclear how effective PD aimed at qualifying teachers to foster
students’ language and literacy skills in subject-area classes is
overall. Additionally, a number of factors can potentially affect
evaluation outcomes, making it even more difficult to determine
the effectiveness of these programs. Effect sizes may vary
4 E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971
depending on characteristics of the study as well as the PD program
in question.
Study characteristics. Wilson and Lipsey (2001) showed that
several factors regarding study quality and methodology can ac-
count for almost as much variation in outcomes in meta-analyses as
intervention features. For instance, different research designs can
explain differences in effect sizes between studies (Higgins &
Green, 2011). Wilson and Lipsey (2001) found that one-group
pre-post designs tend to overestimate intervention effects
compared to comparison group designs. Furthermore, quasi-
experimental designs may inflate intervention effect estimates.
For example, in terms of selection bias, higher motivation among
participants in the intervention group could result in higher effects
(Borenstein, Hedges, Higgins, & Rothstein, 2009; Cheung & Slavin,
2016). Rigorous research designs, such as randomized controlled
trials, can reduce the risk of such alternative explanations for gains
(WWC, 2017; Yoon et al., 2007). However, even in randomized
trials, methodological variables can affect effect sizes when
randomization occurs at the group (e.g., school) rather than the
individual (e.g., teacher) level. Such cluster-randomized trials may
be associated with greater effects, e.g., due to motivation bias
resulting from greater compliance in implementing new methods
in preexisting groups. In contrast, randomized trials tend to un-
derestimate intervention effects when teachers from the same
school are randomized individually, and PD content is passed on
from intervention group to comparison group teachers (Higgins,
Deeks, & Altman, 2011). Furthermore, the measures and scales
used to evaluate the intervention impact matter. For example, in-
struments specifically developed for the study in question may
yield larger effects than standardized/published instruments due to
closer alignment with the content of the PD intervention (Blank &
de las Alas, 2009; Cheung & Slavin, 2016; Gersten et al., 2005;
Wilson & Lipsey, 2001). While instruments generally need some
degree of alignment and sensitivity to identify effects, unstan-
dardized and unestablished instruments can fall prey to insufficient
broadness and unknown validity (Gersten et al., 2005; Yoon et al.,
2007). Moreover, there may be issues associated with the reli-
ability of outcome measures (Hunter & Schmidt, 2004). For
example, scales with low internal consistency or low interrater
reliability may reduce precision (WWC, 2017). Furthermore, the
information source may affect the magnitude of the effect (Wilson
& Lipsey, 2001). For example, unlike an independent rater, a person
involved in the development or delivery of an intervention might
not be able to judge that intervention without bias; thus, larger
effects may be expected if the information source is affiliated with
the intervention. Similarly, the allegiance of the researchers con-
ducting the evaluation can potentially influence PD outcomes
(Luborsky et al., 1999; Munder, Brütsch, Leonhart, Gerger, & Barth,
2013).
PD intervention characteristics. Features of successful teacher
training across disciplines are well known. One of the most often-
cited features is program duration. There is consensus that PD
should be intensive and sustained over time (Garet, Porter,
Desimone, Birman, & Yoon, 2001; Guskey & Yoon, 2009). Howev-
er, findings on the relationship between the duration of a PD pro-
gram and its effectiveness are inconsistent (Lipowsky & Rzejak,
2015). In the area of academic language across the curriculum, PD
programs thought to be effective differ greatly in duration,
although most are long-term Kalinowski, Gronostaj, & Vock, 2019.
According to Timperley et al. (2007), a certain amount of time and
an extended period of PD are important, yet not sufficient. There
seems to be no ideal amount of time and no linear relationship
between PD duration and effectiveness in general. Another aspect
related to PD effectiveness is program format. For example, tradi-
tional workshops versus more innovative approaches have been
discussed (Birman et al., 2000). The PD interventions studied in the
review by Kalinowski, Gronostaj, & Vock, 2019 encompassed a wide
range of formats, with workshops and coaching being the most
common components. Coaching may be of particular interest, as it
has been found to be a particularly effective form of PD for language
and literacy practicesdat least among early childhood educators
(Markussen-Brown et al., 2017; Neuman & Wright, 2010). Beyond
the type of format, the number of formats included may also be
important. Markussen-Brown et al. (2017) found a significant
positive association between the number of formats used in PD and
educators’ practices.
In summary, the information found in the literature to date on
the effects of teacher PD aimed at fostering students’ language skills
across the curriculum is heterogeneous. Variations in study quality
and methodological factors as well as intervention features
complicate the interpretability of current primary PD research.
Thus, the overall effectiveness of PD aimed at qualifying teachers to
foster language development in academic content classes and the
extent to which study and intervention features relate to the
magnitude of effects is unclear. Reviews and syntheses can help to
integrate existing literature, estimate overall effects and reveal
variables that can potentially account for differences in the results
of studies (Cooper & Hedges, 2009). Moreover, they can increase
statistical power and the precision of intervention effect calcula-
tions (Cohn & Becker, 2003; Mulrow, 1994).
1.3. Aim of this study
Given the lack of syntheses of existing primary studies and the
lack of information on the overall effectiveness of PD interventions
to help teachers integrate the development of language necessary
for schooling into their teaching of academic content, our research
objectives were twofold. We aimed (1) to estimate the effect of such
PD programs across existing studies on (a) teachers’ cognition and
(b) teachers’ classroom practice and (2) to examine the distribution
of effect sizes and gather evidence on potential effect modifiers.
This is of great practical relevance, as it can help us better under-
stand PD effectiveness and show what is important in PD in-
terventions. Assuming a sequential relationship between teacher
change and student outcomes (Yoon et al., 2007), it seems neces-
sary to carefully examine the teacher level beforedwhere appro-
priatedturning to the student level in subsequent research. While
meta-analyses focusing on teacher PD aimed solely at fostering
students’ general and basic literacy skills (without integration into
content-area teaching) already exist (e.g., Basma & Savage, 2018;
Timperley et al., 2007), this study specifically focuses on teacher PD
to promote academic language support in content areas. In contrast
to some of the non-quantitative research syntheses cited above
(e.g., Zhang, 2014), this study’s focus is not limited to PD geared to
ELLs.
2. Method
2.1. Inclusion criteria
We applied a meta-analytic method to meet these research
objectives. After perusing the relevant literature, we developed the
following selection criteria for study inclusion: The study must (a)
evaluate the effectiveness of a PD intervention for in-service
teachers aimed at helping them integrate students’ development
of the language necessary for schooling into their academic content
instruction; (b) have been published between January 2000 and
January 2016; (c) be written in English; (d) and be a quantitative
empirical study that reported either effect sizes or statistical data
allowing effect sizes to be computed from pre- and post-data and/
 E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971
or intervention and comparison group data. Further, the examined
PD intervention (e) must have been conducted among teachers at
general education schools, including kindergarten if part of primary
education; and (f) be described in some detail. Criterion (a) was
further specified in that the study’s focus had to be the evaluation
of the PD intervention itself rather than the student support pro-
gram teachers were trained to use in the intervention. In line with
the above definition of academic language, we excluded PD initia-
tives focusing on initial or basic reading and writing, foreign-
language learning, and children with disabilities. Whereas some
researchers advocate using only studies with rigorous research
designs (such as randomized controlled trials) in a meta-analysis,
this can exclude much of the existing research. Instead of
excluding studies beforehand, it can be more informative to
examine systematic differences in study characteristics among the
included studies as part of the meta-analysis (Chambers, 2004;
Hattie, 2009; Lipsey & Wilson, 2001; Pant, 2014).
2.2. Search procedure
The search for studies was conducted using a systematic multi-
step process. To define our key concepts and search terms, we
conducted preliminary searches and used database thesauri. We
then developed a comprehensive search syntax2 with Boolean
operators including terms describing academic language or the
student population of interest (e.g., “language,” “literacy”,
“linguistically diverse”, “bilingual”) and terms to indicate that the
language support took place in the subject areas (e.g., “across the
curriculum,” “mainstream”, “sheltered instruction”). We used
various combinations of these keywords as appropriate and further
combined them with “teacher professional development” and
synonyms (e.g., “teacher* professional learning”, “teacher*
training”) as well as “effectiveness” and alternate terms (e.g.,
“impact”, “effic*”). Using this syntax, we searched the databases
Educational Resources Information Centre and EBSCOhost, including
PsycINFO, PsycARTICLES, Education Full Text (H.W. Wilson), and Psy-
chology and Behavioral Sciences Collection. We identified 1,778 re-
cords. The first author of this paper screened and applied the
inclusion criteria to the titles and abstracts of the identified records.
A trained research assistant independently rated 27% of the records.
Inter-rater reliability was almost perfect, as indicated by Cohen’s
k ¼ 0.81 (Landis & Koch, 1977). Next, the full texts of the approved
documents were screened.3 Disagreements and uncertainties about
whether a study should be included were discussed until consensus
was reached. We also conducted a manual search in relevant
journals and examined the reference lists of all eligible studies.
Additionally, we asked experts in the field for suggestions of
pertinent studies, attempting also to find unpublished studies and
reduce the risk of publication bias, which is “the tendency [ …] to
submit or accept manuscripts for publication based on the direction
or strength of the study findings” (Dickersin, 1990, p. 1385),
generally in favor of positive or statistically significant results. The
71 articles identified through these processes were also examined
using the assessment procedure outlined above. The main reasons
for excluding articles after full-text screening stemmed from the
inclusion criteria: Many studies lacked a study design (e.g., only
post-intervention surveys on perceived PD impact) or statistical
data allowing for the estimation of effect sizes (e.g., Brisk &
2
The syntax will be provided by the first author upon request.
3
The full text of seven potentially relevant dissertations could not be obtained
even after contacting the authors or their chairs (when contact details were
available). Further details on the inclusion process and a list of excluded studies can
be provided upon request.
5
Zisselsberger, 2011; McIntyre et al., 2010). Other studies were not
proper empirical studies (but rather short descriptive reports, for
instance) (e.g., Frey, Fisher, & Nelson, 2010; Walsleben, 2008),
focused on questions other than the effectiveness of the PD inter-
vention (e.g., Patrick, 2012) or a different type of language support
(e.g., Vaughn et al., 2008), or barely described the PD intervention
(e.g., Eun, 2006; Kolano, Davila, Lachance, & Coffey, 2014). To avoid
duplicate data, we included the more recent and comprehensive of
two articles that reported on the same study (Kim et al., 2011; Olson
et al., 2012). If studies met all inclusion criteria but lacked the
required statistical information, we tried to contact the authors to
request the missing information. Fig. 1 illustrates the different
stages of the literature search with the respective numbers of ar-
ticles identified, assessed, excluded, and retrieved. As can be seen,
the studies found to meet the inclusion criteria reported effects
across different outcome levels, namely teachers’ cognition (e.g.,
knowledge, beliefs), teachers’ practice, and student outcomes. None
of the thematically relevant studies reported quantifiable data on
teachers’ reactions to PD. In this paper, we focus on the available
results for the teacher level.
2.3. Data coding
Following recommendations by Lipsey and Wilson (2001), we
coded study and PD intervention variables (derived from the
literature cited in Section 1.2) as well as effects (separated by
outcome levels). Table 1 lists the coded variables and their
respective definitions.4 Following the guidelines of The Campbell
Collaboration (2014), all included studies were independently
coded by the first and second author of this paper, who were both
research associates with several years of experience in the field of
research on educators’ PD, to ensure the reliability of coding. We
used a coding form for all codes on the study and intervention level.
Interrater reliability, estimated using SPSS 24.0 (IBM Corp, 2016),
was excellent, with Cohen’s k ¼ 0.87 (calculated for nominal vari-
ables) and intraclass correlation ICC ¼ 0.87 (calculated for interval
variables). For all codes on the effect size level, one author entered
the data directly into a spreadsheet in the Comprehensive Meta-
Analysis (CMA) software, Version 3 (Borenstein, Hedges, Higgins,
& Rothstein, 2013), whereupon the other author checked the en-
tries and marked discrepancies. Since none of the studies provided
correlations between pre- and post-measurements, we inserted a
conservative estimate of r ¼ 0.50 (Fukkink & Lont, 2007). Interrater
reliability for codes on the effect size level was 99.4%. Disagree-
ments in coding were solved through discussion. A short descrip-
tion of the PD programs’ content and specification of the outcome
construct was added.
2.4. Analysis
The statistical data provided in various formats in the primary
studies was transformed into the standardized mean difference
corrected for bias, Hedges’ g. This is the appropriate measure when
sample sizes are small (Borenstein et al., 2009). Mean differences
were based on data on pre-post changes or contrasts between
intervention and comparison groups. If pre- and post means and
standard deviations for both the intervention and comparison
group were given, we standardized by change score standard de-
viation. If multiple outcome measures were available on the same
outcome level, we nested the effect sizes under PD intervention and
calculated an aggregate effect size per intervention. We applied a
4
Some additional characteristics were coded but not analyzed because of
missing data and the small number of studies.
6 E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971

Fig. 1. Literature search process with numbers of records considered (chart adapted from Moher, Liberati, Tetzlaff, Altman, & The PRISMA Group, 2009).

Table 1
Overview and description of codes used for included studies.

Code Description

Study
Publication year Year the study was published
Sample size Number of participating teachers at beginning of the study
Educational stage Possible categories: (1) elementary (kindergartenegrade 6), (2) secondary (grades 7e12), (3) mixed (kindergartenegrade 12)
Design General study design. Possible categories: (1) one-group pre-post measurement, (2) pre-post comparison group design, (3) comparison
group design without pre-measurement
Assignment to conditions Applicable if comparison group design was used. Possible categories: (1) non-random assignment and no matching of groups prior to
intervention, (2) non-random assignment but comparison group matched to intervention group, (3) random assignment at group/school
level, (4) random assignment at individual level
Instruments Instruments used for data collection. Possible categories: (1) survey, (2) interview, (3) observation, (4) assignment/test
Standardized instrument Standardized/established instruments that were used/validated in previous studies. Possible categories: (1) not included (i.e., all instruments
used were developed or adapted for the study in question), (2) included in study
Reliable data collection Reliable measures (internal consistency of scale with Cronbach’s alpha .65)/data collection process (interobserver agreement of ICC/
Kappa .70). Possible categories: (1) not included (i.e., reliability lower than cut-off values or not reported), (2) included in study
Independent information Independence of the person who provided data. Possible categories: (1) not included (rating solely by persons involved in PD, e.g., mentor, PD
source facilitator), (2) included (study included rating by independent persons, e.g., external researcher, student)
Independent evaluation Evaluation of PD intervention and evaluation financing by actors not involved in the development/delivery of the PD intervention. Possible
categories: (1) no, (2) missing, (3) yes
Intervention
Format Delivery format(s) employed. Possible categories: (1) workshop/course; (2) coaching/mentoring; (3) web-mediated/online training; (4)
classroom demonstrations; (5) other group meetings (e.g., PLG); (6) provision of curricular units
Time span Total time span in months in which PD took place. If a study did not report the time span in months, we calculated it ourselves assuming one
school year ¼ 9 months.
Duration Duration of PD workshops/courses in hours. If studies did not report workshop duration in hours, total hours were approximated by
assuming one full day ¼ 8 h. Other supports like coaching and team meetings could not be considered because their duration was often not
specified.
Effect size level
Outcome name Outcome as labeled in the report
Outcome level Level the provided outcome data applicable for meta-analysis matched. Possible categories: (a) teacher cognition (i.e., beliefs, attitudes,
knowledge), (b) teachers’ practices (i.e., techniques and instructional strategies to support students’ language development and make
content accessible to all learners)
Statistical data Outcome data applicable for meta-analysis. Format dependent on data given in reports, including sample sizes, means, standard deviations,
pre-post correlations

Note. All codes from Design through Independent information source were coded separately for (a) teacher cognition and (b) teachers’ practices.

separate meta-analysis for each outcome level (teachers’ cognition
and teachers’ practices). We integrated the intervention effects into
one weighted summary effect size g’ per outcome level by con-
ducting random-effects meta-analyses with interventions
weighted on the basis of the variance of their effect sizes. Random-
effects models were used because variables that may have
impacted the magnitude of the effect size, such as samples and
interventions, varied from study to study. Thus, between-study
variance had to be taken into account in addition to within-study
variance (Borenstein et al., 2009). We estimated the homogeneity
of effects using Q-statistics and I2 (Higgins, Thompson, Deeks, &
Altman, 2003), which relate the observed variance to the within-
study variance, and examined potential sources of heterogeneity,
i.e. effect modifiers. We first investigated study characteristics,
followed by PD features. If variables were dichotomous, mixed-
effects subgroup analyses assuming a common among-study vari-
ance across subgroups were used to test whether the mean effects
for each subgroup were different from each other. For continuous
variables, separate random-effects meta-regressions were per-
formed, taking into consideration that at least around 10 studies
 E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971 7

should be available per covariate (Borenstein et al., 2009; Deeks, like self-efficacy, teaching philosophy and content knowledge.
Higgins, & Altman, 2011). In sensitivity analyses, i.e., repeating an Teachers’ practices, by contrast, comprised instructional practices
analysis while substituting alternatives for unclear or arbitrary and language input provided, both in the sense of practices to
decisions, we tested whether and how the effects changed if improve lesson quality.
particular studies (e.g., outliers) and values were omitted from the
analyses (Deeks et al., 2011). In addition, we assessed the potential 3.2. Effects of PD on teachers’ cognition
impact of publication bias using a funnel plot, for which a study’s
intervention effect is plotted against its standard error (as an index Four studies with N ¼ 378 teachers could be included to esti-
of precision). In the absence of publication bias, one would expect mate the effect of the examined PD programs on teachers’ cogni-
the plot to resemble a symmetrical funnel with the results of less tion. Eleven individual effect sizes were calculated for this outcome,
precise studies (which are smaller and have larger standard errors) ranging from one to four per study. Table 3 and Fig. 2 show that
being more widely scattered and allocated at the bottom of the plot estimates indicate negligible (g ¼ 0.02) to medium positive impacts
than those of more precise studies. Conversely, asymmetry may (g ¼ 0.63), according to the rules of thumb established by Cohen
indicate publication bias. We also applied Egger’s regression, which (1988). The meta-analysis showed a non-significant weighted
can be considered a statistical analogue of a funnel plot, with sig- overall effect of PD interventions on teachers’ cognition of g’ ¼ 0.21
nificance indicating asymmetry (Egger, Smith, Schneider, & Minder, (SE ¼ 0.14, p ¼ .120). Heterogeneity analysis to test whether all the
1997; Sterne, Egger, & Smith, 2001). All analyses were conducted dispersion of the interventions’ effect sizes was due to random
using CMA (Borenstein et al., 2013). error was non-significant (Q ¼ 6.03, df ¼3, p ¼.110, T2 ¼ 0.04,
SE ¼ 0.06). However, power may be low when one only has a small
number of studies to work with (Higgins et al., 2003). An I2 of 50.28,
3. Results
suggesting that 50.28% of the observed dispersion reflected real
differences in effect sizes. According to Deeks et al. (2011), this
3.1. Description of included studies and PD interventions
proportion may be interpreted as moderate heterogeneity, indi-
cating that the studies included in this analysis differ in some re-
Table 2 gives an overview of the characteristics of the 10 studies
spects, which could be clarified through further analyses. However,
and PD interventions included. The studies were published be-
because of the very small number of studies, heterogeneity was not
tween 2003 and 2014. All studies were conducted in the US, with
investigated further on this level.
the exception of one from the Netherlands. Nine were journal ar-
A one-study-removed procedure revealed that the aggregated
ticles and one was a dissertation. A total of N ¼ 650 teachers
effect was not very robust. In particular, if the intervention with the
participated in the studies at the beginning of the investigations.
highest effect size (Cantrell & Hughes, 2008) was removed, the
Teacher sample sizes ranged from 21 to 198 and were relatively
summary effect decreased to g’ ¼ 0.08 (SE ¼ 0.10, p ¼ .431).
small in many of the included studies. Six studies used a one-group
Conversely, removing the intervention with the lowest effect size
pre-post design and four used a pre-post comparison group design,
(Lee & Maerten-Rivera, 2012) resulted in a total effect of g’ ¼ 0.31
of which two also included group comparisons without pre-
(SE ¼ 0.18, p ¼ .098).
measurement. Of the controlled trials, one used matched groups,
We further analyzed the risk of publication bias. The funnel plot
two were randomized at the school level, and one was randomized
(Fig. 3) displays the relationship between the standard error and
at the individual level. Teachers’ cognition was assessed via surveys.
the effect size for each trial. In general, trials with small sample
Classroom practices were sometimes assessed via surveys and in-
sizes tend to have larger standard errors, i.e., to be less precise, and
terviews, but in most studies teachers were observed. In one study,
appear at the bottom of the figure. Although some small studies
teachers completed assignments instead. It was striking that only
with large standard errors are included in the meta-analysis,
one study included a standardized instrument and only one study
Egger’s regression suggests symmetry of results (Intercept ¼ 4.32,
clearly stated that the evaluators were not involved in the PD
SE ¼ 2.15, p ¼ .182).
intervention. Basically, the 10 studies evaluated 10 different PD
interventions. However, Shanahan and Shea (2012) compared data
3.3. Effects of PD on teachers’ classroom practices
from teachers who attended a certain amount of PD to data from
teachers who attended less than this amount. Considering the
Another meta-analysis on the basis of 11 trials examined out-
different amount of time the two groups participated in PD, we
comes related to teachers’ classroom practices. For this outcome, 74
viewed the study as evaluating two separate interventions, i.e., as
effect sizes were extracted, between one and 30 per trial. As dis-
two different trials. Thus, a total of k ¼ 11 PD interventions were
played in Table 4 and Fig. 4, aggregated intervention effects were all
included in our analysis. Elementary and secondary grade levels
positive and ranged from g ¼ 0.11 to g ¼ 2.17. The overall pooled
were represented. The time span of the in-service training and the
effect was g’ ¼ 0.71 (SE ¼ 0.16, p < .001), indicating a significant
duration of the workshops varied greatly, from half a day to three
medium to large PD effect on teachers’ classroom practices. How-
years and from three to 112 h respectively. Most of the in-
ever, there was significant variation between effect sizes (Q ¼ 41.41,
terventions applied multiple delivery formats. Workshops were
df ¼ 10, p <.001, T2 ¼ 0.19, SE ¼ 0.12) with I2 ¼ 75.85, suggesting that
part of all programs; coaching was the second most frequently-
a substantial percentage of the variation in individual trial results
used format. PD programs addressed various language support
cannot be explained by chance.
approaches and strategies, such as sheltered instruction. Six in-
In a sensitivity analysis, we removed the intervention studied by
terventions were related to a specific subject, which was science in
Choi and Morrison (2014), which exhibited an exceptionally large
five cases. All 11 trials evaluated PD program effectiveness on
effect, from the analysis. The results revealed a smaller, yet still
teachers’ practices; four studies additionally examined effects on
considerable overall effect of g’¼ 0.54 (SE ¼ 0.10, p < .001). The
teachers’ cognition.5 The latter were measured using constructs
removal of the study with the smallest effect (Hart & Lee, 2003)
only slightly changed the results; the aggregated effect was
5
Four of the included articles also considered the student level. For two other
g’ ¼ 0.78 (SE ¼ 0.16, p < .001). We conducted further explorative
studies, student outcomes were presented in separate articles identified during our analyses to examine potential bias in our data. From Lee and
search process. Maerten-Rivera (2012), we included science knowledge in the
 8
Table 2
Overview of included studies and PD interventions.

Study Country Publication Sample Educational Design Instruments Standardized Reliable Independent Independent Time Duration Formats Content Outcomes
type stage instrument data information evaluation span
collection source

Anderson USA Disser- 32 Secondary Pre-post Observation Not incl. Incl. Not incl. No 9 80 Workshops, SIOP, ESL strategies, Practice
(2009) tation comparison group coaching, guided language (instruction)
(matched) demonstrations acquisition design
Cantrell and USA Journal 22 Secondary One-group pre- Survey, Not incl. Incl. Incl. Yes 9 64 Workshops, Content literacy Cognition (self-
Hughes post observation coaching, techniques efficacy),
(2008) groups practice
(instruction)
Choi and USA Journal 33 Mixed One-group pre- Observation Not incl. Not incl. Not incl. No 18 90 Workshops, SIOP Practice
Morrison post coaching, (instruction)
(2014) online, groups
Crawford et al. USA Journal 23 Elementary One-group pre- Observation Not incl. Not incl. Incl. Missing 4.5 45 Workshops, Sheltered instruction, Practice
(2008) post coaching total integrated (instruction)
language approach

E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971

Greenleaf et al. USA Journal 105 Secondary (Pre-post) Survey Not incl. Incl. Incl. No 12 80 Workshops, Reading Cognition
(2011) Comparison (teachers, coaching apprenticeship, (teaching
group students), inquiry-based philosophy),
(randomization at interview, science practice
school level) assignment (instruction)
Hart & Lee USA Journal 53 Elementary One-group pre- Survey, Not incl. Incl. Incl. No 9 32 Workshops, Scaffolding, inquiry- Cognition
(2003) post observation units based science (knowledge,
beliefs),
practice
(instruction)
Henrichs & NL Journal 59 Elementary Pre-post Observation Not incl. Incl. Not incl. No 0.03 3 Workshop Academic language Practice
Leseman comparison group input, science (language
(2014) (randomization at input)
school level)
Lee and USA Journal 198 Elementary One-group pre- Survey, Not incl. Incl. Incl. Missing 36 112 Workshops, Scaffolding, literacy Cognition
Maerten- post observation units development and (knowledge),
Rivera ESOL strategies, practice
(2012) inquiry-based (instruction)
science
Olson et al. USA Journal 103 Secondary (Pre-post) Observation Incl. Incl. Incl. No 9 46 Workshops, Cognitive strategies Practice
(2012) Comparison coaching approach to literacy (instruction)
group instruction, English
(randomization at
individual level)
Shanahan and 10 Observation Student-talk Practice
Shea (2012) strategies, Five E (instruction)
(a) model, inquiry-based
Shanahan and USA Journal 11 Elementary One-group pre- Not incl. Not incl. Incl. Missing 8 18 Workshops science
Shea (2012) post
(b)

Note. SIOP ¼ Sheltered Instruction Observation Protocol. ESL ¼ English as a second language. ESOL ¼ English for speakers of other languages.
 E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971 9

Table 3
Effects of PD on teachers’ cognition in random-effects meta-analysis.

Study g SE Lower CI limit Upper CI limit p Relative weight

Cantrell and Hughes (2008) 0.63 0.23 0.19 1.08 .006 20.75
Greenleaf et al. (2011) 0.33 0.26 0.19 0.85 .212 17.41
Hart & Lee (2003) 0.05 0.15 0.23 0.34 .709 31.93
Lee & Maerten-Rivera (2012) 0.02 0.16 0.30 0.33 .921 29.90
Combined 0.21 0.14 0.06 0.48 .120

Fig. 2. Forest plot showing effects of PD on teachers’ cognition in random-effects meta-analysis. Squares represent the aggregated effect per trial with horizontal lines showing the
respective confidence interval. The diamond represents the summary effect size.

Fig. 3. Funnel plot for the cognition outcome. Circles represent trials; diamond represents the summary effect size and its confidence interval.

Table 4
Effects of PD on teachers’ practices in random-effects meta-analysis.

Study g SE Lower CI limit Upper CI limit p Relative weight

Anderson (2009) 0.98 0.37 0.26 1.70 .008 7.35

Cantrell and Hughes (2008) 0.46 0.22 0.04 0.89 .033 10.05
Choi and Morrison (2014) 2.17 0.32 1.54 2.80 .000 8.16
Crawford et al. (2008) 1.10 0.26 0.59 1.60 .000 9.28
Greenleaf et al. (2011) 0.49 0.28 0.05 1.03 .074 9.01
Hart & Lee (2003) 0.11 0.16 0.20 0.42 .492 11.08
Henrichs & Leseman (2014) 0.81 0.28 0.26 1.35 .004 8.91
Lee & Maerten-Rivera (2012) 0.39 0.18 0.04 0.73 .028 10.76
Olson et al. (2012) 0.45 0.28 0.10 1.00 .106 8.92
Shanahan & Shea (2012) (a) 0.51 0.31 0.10 1.12 .102 8.34
Shanahan & Shea (2012) (b) 0.75 0.32 0.12 1.38 .020 8.16
Combined 0.71 0.16 0.41 1.01 .000

original meta-analysis, a construct that might not be as closely
related to teachers’ support of students’ language as the other
included outcomes on the level of teachers’ classroom practices.
When excluding this construct in a sensitivity analysis, the sum-
mary effect remained the same as when the construct was
included.
The risk of publication bias can be derived from the funnel plot
(Fig. 5). Egger’s regression (Intercept ¼ 5.02, SE ¼ 1.77, p < .020)
confirmed the asymmetry that can be seen in the plot, with a
tendency towards more studies with larger standard errors and
larger effect sizes. In this case, small studies might lead to less
precise results and overestimated effect sizes.
10 E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971

Fig. 4. Forest plot showing effects of PD on teachers’ practices in random-effects meta-analysis. Squares represent the aggregated effect per trial with horizontal lines showing the
respective confidence interval. The diamond represents the summary effect size.

Fig. 5. Funnel plot for teachers’ practices outcome. Circles represent trials; diamond represents the summary effect size and its confidence interval.

3.3.1. Study characteristics and teachers’ classroom practice
Table 5 presents the results of subgroup analyses on the level of
teachers’ classroom practices. Subgroup analyses were first per-
formed to determine whether the effect sizes varied depending on
study characteristics. Comparing the summary effect of in-
terventions studied using pre-post designs to those studied using
comparison group designs yielded no significant difference, justi-
fying the combination of both study types in further analyses.
Comparing the four controlled trials according to method of
assignment to conditions, the non-random matched group trial
yielded higher values than the two school level-randomized trials
and the individual-level randomized trial, but the differences were
not statistically significant. Another subgroup analysis showed a
non-significant difference between the majority of interventions
evaluated with unstandardized instruments and the one inter-
vention evaluated with standardized measures, with the aggre-
gated effect size favoring the former. The effects of interventions
evaluated solely with instruments and data collection processes for
which reliability was low or not given were significantly higher
than the effect sizes for interventions evaluated including reliable
measurements. Furthermore, interventions evaluated solely via
observations by mentors or other persons involved in the inter-
vention yielded effect sizes significantly higher than those (also)
assessed by independent personnel. Finally, the effects of in-
terventions evaluated by personnel involved in their development
and for which information about evaluator involvement was not
given tended to be higher than those of interventions assessed by
independent evaluators. The differences were not statistically sig-
nificant, however.
3.3.2. PD intervention characteristics and teachers’ classroom
practice
In this section, we present the results of the moderator analyses
for all 11 trials investigating PD intervention characteristics as po-
tential moderators of PD effectiveness. Subgroup analysis showed
that the impact of the six interventions including coaching was
higher than that of the five interventions without coaching, but the
difference was not statistically significant (see Table 5). Meta-
regressions (see Table 6) showed that neither the time span of
the PD intervention nor the total workshop hours were linearly
related to PD effects on teachers’ practices. There was a positive
association between the number of formats used in the PD in-
terventions and the effect size, with the effect size increasing by
0.32 with every additional PD format included. However, this as-
sociation did not reach statistical significance.
Given the high effect size and the type of instruments and
methods used (e.g., no standardized instruments or reliable data,
no external information source or evaluation) in the study by Choi
and Morrison (2014), we repeated the analyses omitting this trial.
In these analyses, the difference between interventions with
 E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971 11

Table 5
Results of subgroup analyses on the level of teachers’ practices.

Groups Number of trials Effect size Test of null (2-tail) Heterogeneity

k g SE p Q df p

Study characteristics
Design
Pre-Post 7 0.74 0.20 .000
Comparison group 4 0.67 0.28 .016 0.04 1 .841
Assignment to conditions
Non-random, matched 1 0.98 0.37 .008
Random, school level 2 0.65 0.20 .001
Random, individual level 1 0.45 0.28 .106 1.31 2 .520
Standardized instrument
Not included 10 0.74 0.17 .000
Included 1 0.45 0.54 .407 0.26 1 .612
Reliable data collection
Not included 4 1.12 0.22 .000
Included 7 0.48 0.15 .002 5.73 1 .017
Independent information source
Not included 3 1.31 0.26 .000
Included 8 0.50 0.14 .000 7.59 1 .006
Independent evaluation
No 6 0.79 0.24 .001
Missing 4 0.68 0.30 .022
Yes 1 0.46 0.57 .417 0.31 2 .856
PD intervention characteristics
Format
Coaching 6 0.91 0.20 .000
No Coaching 5 0.48 0.21 .023 2.17 1 .141

Table 6 impact when it came to improving teachers’ ability to facilitate

Results of random-effects meta-regression analyses. students’ acquisition of content and language simultaneously. The
k B SE 95% CI p R2 analog aggregate effect estimate varied in some of the sensitivity and
Time span 11 .00
moderator analyses. The original estimate of g’ ¼ 0.71 corresponds
Intercept 0.73 0.27 0.21, 1.25 to a 26-percentile point difference on a normal distribution, while
Time span 0.00 0.02 0.04, 0.03 .937 conservative estimates from studies randomized at individual level,
Duration 11 .00 standardized instruments, reliable data collection, independent
Intercept 0.53 0.30 0.10, 1.12
information sources, or external evaluation suggest an overall PD
Duration 0.00 0.00 0.01, 0.01 .454
Number of formats 11 .15 effect ranging from g’ ¼0.45 to g’ ¼0.50 on teachers’ classroom
Intercept 0.05 0.39 0.71, 0.80 practice indicating a maximum of a 19-percentile point difference.
Number of formats 0.32 0.17 0.02, 0.65 .064 Our summary effect size estimate is in line with results from other
meta-analyses: Across various domains, Wade (1984) found a PD
effect of d ¼0.60 on teachers’ practices. In a meta-analysis on lan-
coaching (k¼ 5, g’ ¼ 0.67, SE ¼ 0.15) and without coaching was guage- and literacy-focused PD for early education teachers,
smaller (k ¼ 5, g’ ¼ 0.43, SE ¼ 0.13, p ¼ .228), and unlike in the Markussen-Brown et al. (2017) found an overall effect of g’ ¼0.59 on
original results, the number of formats used in the interventions a measure of educators’ practice.
was not associated with the effect size (B¼ 0.04, SE ¼ 0.17, Given the significant heterogeneity, however, Borenstein et al.
p ¼ .796, R2analog ¼ .00). Apart from that, the results did not (2009) recommend focusing on factors that can explain differ-
substantially differ from the results of the original analyses. ences in effect sizes rather than immediately interpreting the
combined effect size as meaningful. Indeed, subgroup analyses
showed some of the study characteristics to moderate the out-
4. Discussion
comes on teachers’ practices. In line with other meta-analyses,
methodological variables affected the magnitude of the effect size
The results of our meta-analysis on teachers’ cognition indicate
((Egert, Fukkink, & Eckhardt, 2018); Cheung & Slavin, 2016;
that all four evaluated interventions tended to show at least posi-
Markussen-Brown et al., 2017; Wilson & Lipsey, 2001). In particular,
tive results. However, the weighted summary effect of PD was small
studies using reliable data collection had a summary PD effect that
and non-significant. This overall estimate appears to be in line with
was significantly lower (g’ ¼ 0.48) than that from studies that did
findings by Markussen-Brown et al. (2017), for instance, who re-
not report reliable data collection and also lower than the original
ported a non-significant ES of 0.12 for language- and literacy-
overall estimate. The larger effect of unreliable measures was
focused PD on early educators’ knowledge. However, although all
produced by trials that used observations but did not report
sensitivity analyses indicated a positive overall effect, the magni-
interrater reliability. Furthermore, effect sizes were significantly
tude of the effect varied from negligible to small. Since only four
larger (g’ ¼ 1.31) in studies, where only non-independent data from
studies could be included in this meta-analysis, it may not be
sources involved in the PD intervention were used. On the one
possible to interpret the results with confidence and further
hand, this might be explained by the assumption that a person who
research is needed.
is involved in a program has particular interest in finding their work
On the level of teachers’ classroom practices, our meta-analysis
to be effective. On the other hand, they could simply be able to
revealed a significant medium to large effect and substantial het-
better identify important aspects/outcomes of that program.
erogeneity. Again, all interventions tended to exhibit a positive
12 E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971
Moderator analyses indicated several further tendencies for dif-
ferences in effect sizes, but the alpha-level was non-significant.
However, non-significant results of subgroup analyses can be due
to small sample sizes and, as a consequence thereof, low test power
(Borenstein et al., 2009). Considering just the differences in average
effect sizes, our results on effect modifiers point in the expected
direction and are in line with the findings on methodological
moderators outlined in the introduction.
None of the subgroup analyses yielded significant results
regarding PD intervention characteristics. Our findings on coaching
are somewhat surprising in light of the results of meta-analyses in
related fields showing coaching to be an effective approach to
improve classroom quality (Egert, Fukkink, & Eckhardt, 2018;
Markussen-Brown et al., 2017). Our results exhibit a trend in the
expected direction but did not reach statistical significance. Thus,
we cannot definitively conclude whether the descriptive advantage
of coaching was due to chance or to the assumed effectiveness of
coaching. While the method might be beneficial in principle, its
ultimate effectiveness might depend on how it is implemented
(Sanetti & Kratochwill, 2014). It is possible that the coaching
implemented in (some of) the evaluated interventions lacked some
important elements. However, vague and insufficient reporting in
several included studiesda problem regularly encountered by au-
thors of meta-analyses (Lipsey & Wilson, 2001)dprevented us
from assessing and comparing the quality and extent of the
implementation of elements such as coaching.
Meta-analytic evidence on PD intensity and duration is
controversial (Egert, Fukkink, & Eckhardt, 2018; Lipowsky & Rzejak,
2015; Markussen-Brown et al., 2017). In our study, PD duration was
not systematically related to the PD effect. This could potentially be
explained by the fact that we were only able to code the total hours
spent inworkshops/courses, and not the duration of other support
elements like coaching due to vague reporting (see Table 1).
Another reason could be that different PD aims and contents
require different program durations, as Lipowsky and Rzejak (2015)
assume. We further found a non-significant association between
the number of formats and the magnitude of the PD effect, which
disappeared after removing an outlying study. It might have been
meaningful to further research the relationship between PD con-
tent and intensity. Since our study’s small sample size prevented us
from testing methodological moderators in meta-regressions as
control variables, the findings on moderators regarding PD char-
acteristics could potentially be confounded.
The relatively small number of studies that fall under the scope
of our research topic is somewhat surprising. No studies on bilin-
gual programs or from any countries other than the US or the
Netherlands met the inclusion criteria for our analysis. One diffi-
culty of this research project certainly lay in defining PD and the
type of language support to be considered in the synthesis, since a
great variety of concepts exist, many of which are not clearly
defined and are used inconsistently (in international research as
well as in the field of education in general). Thus, creating an
appropriate search syntax and deciding what studies to include
involved extensive literature searches and numerous decisions. In
light of our thorough search for studies, an explanation for the
small study sample size may be the relative novelty of the topic
(DiCerbo et al., 2014) and the fact that language support strategies
need to be developed and, ideally, evaluated before corresponding
PD initiatives can emerge and evaluation studies are useful. The
preponderance of studies from the US may reflect that the US
seems to have taken a pioneering role in evaluating PD in this field.
Whether the results of our analysis also pertain to other countries is
unclear. As many interventions in this analysis were related to
science, generalizability might also be limited in terms of subject
areas.
There were several additional limitations to our study. For one,
we were not at all close to taking all potential effect modifiers into
account. For instance, individual school and teacher characteristics
also play a role in the effectiveness of teacher PD (Lipowsky &
Rzejak, 2015). Insufficient reporting prevented some features
from being coded and small sample sizes, too little variance or too
much variance across studies concerning a given feature, as well as
potential confounding, prevented others from being analyzed, in
particular further intervention characteristics. Finally, some infor-
mation had to be summarized when aggregating the data, which
necessarily led to some simplification. For example, some studies
reported data for multiple time points (e.g., Lee & Maerten-Rivera,
2012; Olson et al., 2012), but the development of effects over time
was disregarded in our analysis as it was not our study’s focus.
Similarly, we summarized findings across different subject areas
(note that some studies did not distinguish between subject areas
either) as well as across constructs and their respective oper-
ationalizations. Considering that different outcome constructs can
explain a substantial amount of variance in effect sizes even within
one domain (Wilson & Lipsey, 2001), a more differentiated analysis
of outcomes might have been useful.
5. Conclusion
The positive results of our analysis support the idea that PD
plays an important role in supporting teachers in fostering stu-
dents’ academic language proficiency across the curriculum. The
quantitative meta-analytical estimates may help identify a more
precise estimate of the benefits of PD than single studies and
complement existing qualitative research syntheses (e.g.,
Kalinowski, Gronostaj, & Vock, 2019; Zhang, 2014). Taken together,
the findings of our meta-analysis show that research on the effects
of such PD is still at an early stage. Drawing upon existing studies
evaluating PD for in-service teachers at general education schools
in the field of language support in subject areas, this study identi-
fied the magnitude of the effect of such PD interventions on
teachers’ cognition and classroom practices. We explored the field
on the basis of a few studies with heterogeneous designs, methods,
and quality. The magnitude of PD effects might be biased due to the
inclusion of non-reliable instruments and non-independent infor-
mation sources. Our study confirms the importance of rigorous
evaluation research to reduce the risk of bias. Without doubt,
investigating changes in teachers’ classroom performance is
methodologically challenging. For instance, it may be difficult to
acquire randomized control groups in real-world educational set-
tings (Lipowsky, 2010). Nevertheless, valid instruments and
rigorous research processes, such as reliable observations by in-
dependent raters, should be established in future research on the
effectiveness of teacher PD so that conclusions can be drawn with
greater confidence. For instance, a number of recent studies, pub-
lished after our systematic literature search, used randomized
controlled trials (at the school level) as well as multiple observa-
tions (Babinski, Amendum, Knotek, Sa nchez, & Malone, 2018; Tong
et al., 2018). Furthermore, methods, interventions and outcomes
should be described as clearly and comprehensively as possible in
order to facilitate comparisons. Further analyses based on a greater
body of primary research in this field are needed to resolve the
remaining uncertainties in interpretation. In particular, since our
analysis does not clearly suggest whether different training char-
acteristics yield different effects, study and PD intervention char-
acteristics need to be disentangled and the relationship between
PD features and effect sizes needs to be further explored. On the
basis of a greater number of studies, it might also be informative to
look at outcome constructs and subject areas separately and to
meta-analytically study teacher change over time. An important
 E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971
next step will be to examine student outcomes as well as their
relation to teacher outcomes.
The small overall effect on the level of teachers’ cognition, which
was based on a small number of studies and might be instable,
should be investigated further as soon as more primary research on
this outcome is available. On the level of teachers’ practices, sub-
stantial heterogeneity was indicated and we were able to examine
study and PD intervention characteristics that could help to explain
differences in effects. Significantly larger effect sizes among
potentially biased trials were found. As a result, the aggregate effect
in our original meta-analysis might be slightly overestimated and
the results on moderating effects of PD intervention characteristics
could not be interpreted with confidence. Even when taking these
uncertainties into account, however, we can assume a small to
medium positive effect of PD in this particular field with some
confidence.
Thus, our findings on PD effects are promising and of practical
relevance. It seems that PD is able to positively affect some aspects
of teachers’ beliefs and knowledge and to support teachers in
integrating language and literacy teaching into their subject-area
teaching. This could inform arguments about providing resources
for teacher PD. Our analyses may also support administrators and
educators seeking to design or implement teacher PD focused on
providing all students with access to the specific kind of language
used in education. Although many factors influence students’ lan-
guage learning (Texas Education Agency, 2000) and teacher PD is
only one element involved in fostering it, investments in this area
generally seem worthwhile and, in the long term, may help to
reduce the disadvantages students from minority language groups
and families with lower levels of education have in the educational
system.
Funding
This work was supported by the German Federal Ministry of
Education and Research, Germany [01JI1501A]. The funding source
had no involvement in any of the stages of this study.
Acknowledgements
Our thanks to Keri Hartman who provided indispensable lan-
guage assistance and to Max Borchelt for rating hundreds of
studies.
References*
*Anderson, E. M. (2009). Teacher change: The effect of a professional development
intervention on middle school mainstream teachers of English language learners.
Doctoral dissertation. Denton: University of North Texas.
Anstrom, K. A., DiCerbo, P. A., Butler, F., Katz, A., Millet, J., & Rivera, C. (2010).
A review of the literature on academic English: Implications for K-12 English lan-
guage learners. Arlington, VA: The George Washington University Center for
Equity and Excellence in Education.
rdenas-Hagan, E., Francis, D. J., Powell, J., Moore, S.,
August, D., Branum-Martin, L., Ca
et al. (2014). Helping ELLs meet the Common Core State Standards for literacy in
science: The impact of an instructional intervention focused on academic lan-
guage. Journal of Research on Educational Effectiveness, 7(1), 54e82.
Babinski, L. M., Amendum, S. J., Knotek, S. E., Sanchez, M., & Malone, P. (2018).
Improving young English learners’ language and literacy skills through teacher
professional development: A randomized controlled trial. American Educational
Research Journal, 55(1), 117e143.
Bandura, A. (1993). Perceived self-efficacy in cognitive development and func-
tioning. Eductional Psychologist, 28(2), 117e148.
Basma, B., & Savage, R. (2018). Teacher professional development and student lit-
eracy growth: A systematic review and meta-analysis. Educational Psychology
Review, 30(2), 457e481.
Becker-Mrotzek, M., Schramm, K., Thürmann, E., & Vollmer, H. J. (2012). Sprache im
* Fillmore, L. W., & Snow, C. E. (2000). What teachers need to know about language.
References marked with an asterisk are included in the meta-analysis
13
Fach. Sprachlichkeit und fachliches Lernen [Language in subject areas. Linguisti-
cality and subject-specific learning]. Münster: Waxmann.
Birman, B. F., Desimone, L. M., Porter, A. C., & Garet, M. S. (2000). Designing pro-
fessional development that works. Educational Leadership, 57(8), 28e33.
Blank, R. K., & de las Alas, N. (2009). Effects of teacher professional development on
gains in student achievement: How meta analysis provides scientific evidence
useful to education leaders. Washington, DC: The Council of Chief State School
Officers.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction
to meta-analysis. Chichester, UK: Wiley.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2013). Compre-
hensive meta-analysis. Englewood, NJ: Biostat.
Borg, S. (2006). Teacher cognition and language education: Research and practice.
London, New York, NY: Continuum.
Brisk, M. E., & Zisselsberger, M. (2011). “We’ve let them in on the secret”: Using SFL
theory to improve the teaching of writing to bilingual learners. In T. Lucas (Ed.),
Teacher preparation for linguistically diverse classrooms: A resource for teacher
educators (pp. 111e126). New York: Taylor & Francis.
Bunch, G. C. (2013). Pedagogical language knowledge: Preparing mainstream
teachers for English learners in the new standards era. Review of Research in
Education, 37(1), 298e341.
*Cantrell, S. C., & Hughes, H. K. (2008). Teacher efficacy and content literacy
implementation: An exploration of the effects of extended professional devel-
opment with coaching. Journal of Literacy Research, 40(1), 95e127.
Carnegie Council on Advancing Adolescent Literacy. (2010). Time to act: An agenda
for advancing adolescent literacy for college and career success. New York, NY:
Carnegie Corporation of New York.
Chambers, E. A. (2004). An introduction to meta-analysis with articles from the
Journal of Educational Research (1992-2002). The Journal of Educational
Research, 98(1), 35e44.
Chamot, A. U., & O’Malley, J. M. (1987). The cognitive academic language learning
approach: A bridge to the mainstream. Teachers of English to Speakers of Other
Languages, 21(2), 227e249.
Cheuk, T. (2016). Discourse practices in the new standards: The role of argumen-
tation in Common Core-era Next Generation Science Standards classrooms for
English language learners. Electronic Journal of Science Education, 20(3), 92e111.
Cheung, A. C. K., & Slavin, R. E. (2016). How methodological features affect effect
sizes in education. Educational Researcher, 45(5), 283e292.
*Choi, D. S.-Y., & Morrison, P. (2014). Learning to get it right: Understanding change
processes in professional development for teachers of English learners. Pro-
fessional Development in Education, 40(3), 416e435.
Cohen, J. (1988). Statistical power analysis for the behavioural sciences. Hillsdale, NJ:
Lawrence Erlbaum Associates.
Cohn, L. D., & Becker, B. J. (2003). How meta-analysis increases statistical power.
Psychological Methods, 8(3), 243e253.
Cooper, H., & Hedges, L. V. (2009). Research synthesis as a scientific process. In
H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research syn-
thesis and meta-analysis (2nd, pp. 3e16). New York: Russel Sage Foundation.
*Crawford, L., Schmeister, M., & Biggs, A. (2008). Impact of intensive professional
development on teachers’ use of sheltered instruction with students who are
English language learners. Journal of In-Service Education, 34(3), 327e342.
Cummins, J. (2008). BICS and CALP: Empirical and theoretical status of the
distinction. In B. Street, & N. H. Hornberger (Eds.), Literacy (2nd ed., Vol. 2, pp.
71e83). New York: Springer Science þ Business Media. Encyclopedia of lan-
guage and education.
Deeks, J. J., Higgins, J. P. T., & Altman, D. G. (2011). Analysing data and undertaking
meta-analyses. In J. P. T. Higgins, & S. Green (Eds.), Cochrane handbook for sys-
tematic reviews of interventions Version 5.1.0. .
DiCerbo, P. A., Anstrom, K. A., Baker, L. L., & Rivera, C. (2014). A review of the
literature on teaching academic English to English language learners. Review of
Educational Research, 84(3), 446e482.
Dickersin, K. (1990). The existence of publication bias and risk factors for its
occurrence. The Journal of the American Medical Association, 263(10), 1385e1389.
Echevarria, J., & Graves, A. (2007). Sheltered content instruction: Teaching English
language learners with diverse abilities (3rd ed.). Boston: Pearson/Allyn and
Bacon.
Echevarria, J., Vogt, M. E., & Short, D. J. (2004). Making content comprehensible for
English learners: The SIOP model (3rd ed.). Boston: Pearson/Allyn and Bacon.
Education through Speaking and Writing [BiSSeBildung durch Sprache und Schrift].
(2014). Offizieller Auftakt der Bund-L€ ander-Initiative in Berlin. [Official kick-off
of the joint federal-states initiative in Berlin]. Retrieved on 22 November, 2018
from http://www.biss-sprachbildung.de/neuigkeit.html?Id¼11.
Egert, F., Fukkink, R. G., & Eckhardt, A. G. (2018). Impact of in-service professional
development programs for early childhood teachers on quality ratings and child
outcomes: A meta-analysis. Review of Educational Research, 88(3), 401e433.
https://doi.org/10.3102/0034654317751918.
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis
detected by a simple, graphical test. British Medical Journal, 315(7109), 629e634.
Eun, B. (2006). The impact of an English as a second language professional develop-
ment program: A social cognitive approach. Doctoral dissertation. Chapel Hill:
University of North Carolina.
Feilke, H. (2012). Bildungssprachliche Kompetenzen  fo € rdern und entwickeln.
[Fostering and developing academic language proficiency.]. Praxis Deutsch, 233,
4e13.
14 E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971
Washington, DC: Center for Applied Linguistics.
Francis, D. J., Rivera, M., Lesaux, N., Kieffer, M., & Rivera, H. (2006). Practical
guidelines for the education of English language learners: Research-based
recommendations for instruction and academic interventions. Retrieved on
22 November, 2018 from http://www.centeroninstruction.org/files/ELL1-
Interventions.pdf.
Frey, N., Fisher, D., & Nelson, J. (2010). Lessons scooped from the melting pot:
California district increases achievement through English language develop-
ment. JSD, 31(5), 24e28.
Fukkink, R. G., & Lont, A. (2007). Does training matter? A meta-analysis and review
of caregiver training studies. Early Childhood Research Quarterly, 22(3), 294e311.
Garet, M. S., Porter, A. C., Desimone, L. M., Birman, B. F., & Yoon, K. S. (2001). What
makes professional development effective? Results from a national sample of
teachers. American Educational Research Journal, 38(4), 915e945.
Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. S.
(2005). Quality indicators for group experimental and quasi-experimental
research in special education. Exceptional Children, 71(2), 149e164.
Gibbons, P. (2006). Bridging discourses in the ESL classroom: Students, teachers and
researchers. London, New York: Continuum.
Gibbons, P. (2015). Scaffolding language, scaffolding learning. Teaching English lan-
guage learners in the mainstream classroom. Portsmouth: Heinemann.
Gogolin, I., & Lange, I. (2011). Bildungssprache und Durchga €ngige Sprachbildung
[Academic language and continuous language education]. In S. Fürstenau, &
M. Gomolla (Eds.), Migration und schulischer Wandel: Mehrsprachigkeit (pp.
107e127). Wiesbaden: VS, Springer Fachmedien.
*Greenleaf, C. L., Litman, C., Hanson, T. L., Rosen, R., Boscardin, C. K., Herman, J., &
Jones, B. (2011). Integrating literacy and science in biology: Teaching and
learning impacts of reading apprenticeship professional development. American
Educational Research Journal, 48(3), 647e717.
Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks: Corwin.
Guskey, T. R., & Passaro, P. D. (1994). Teacher efficacy: A study of construct di-
mensions. American Educational Research Journal, 31(3), 627e643.
Guskey, T. R., & Yoon, K. S. (2009). What works in professional development? Phi
Delta Kappan, 90(7), 495e500.
Hamre, B. K., Pianta, R. C., Downer, J. T., DeCoster, J., Mashburn, A. J., Jones, S. M.,
et al. (2013). Teaching through Interactions. Testing a developmental frame-
work of teacher effectiveness in over 4,000 classrooms. The Elementary School
Journal, 113(4), 461e487. https://doi.org/10.1086/669616.
*Hart, J. E., & Lee, O. (2003). Teacher professional development to improve the
science and literacy achievement of English language learners. Bilingual
Research Journal, 27(3), 475e501.
Hattie, J. A. C. (2009). Visible learning: A synthesis of over 800 meta-analyses relating
to achievement. London: Routledge.
Haynes, J. (2007). Getting started with English language learners: How educators can
meet the challenge. Alexandria, VA: Association for Supervision and Curriculum
Development.
*Henrichs, L. F., & Leseman, P. P. M. (2014). Early science instruction and academic
language development can go hand in hand. The promising effects of a low-
intensity teacher-focused intervention. International Journal of Science Educa-
tion, 36(17), 2978e2995.
Heppt, B., Henschel, S., & Haag, N. (2016). Everyday and academic language
comprehension. Investigating their relationship with school success and chal-
lenges for language. Learning and Individual Difference, 47, 244e251.
He, Y., Prater, K., & Steed, T. (2011). Moving beyond “just good teaching”: ESL pro-
fessional development for all teachers. Professional Development in Education,
37(1), 7e18.
Herrera, S. G., & Murry, K. G. (2005). Mastering ESL and bilingual methods: Differ-
entiated instruction for culturally and linguistically diverse (CLD) students. Boston:
Pearson.
Higgins, J. P. T., Deeks, J. J., & Altman, D. G. (2011). Special topics in statistics. In
J. P. T. Higgins, & S. Green (Eds.), Cochrane handbook for systematic reviews of
interventions. Retrieved on 22 November, 2018 from Version 5.1.0. https://
training.cochrane.org/handbook.
Higgins, J. P. T., & Green, S. (Eds.). (2011). Cochrane handbook for systematic reviews of
interventions. Retrieved on 22 November, 2018 from, 5.1.0 https://handbook-5-1.
cochrane.org/.
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring
inconsistency in meta-analyses. BMJ Clinical Research Ed., 327(7414), 557e560.
Hoffman, M., & Harris, W. (2018). A complete guide to continuing education for
teachers. Retrieved on 22 November, 2018 from https://www.teachtomorrow.
org/continuing-education-for-teachers/.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis. Correcting error and
bias in research findings. Thousand Oaks: Sage Publications.
IBM Corp. (2016). IBM SPSS statistics for windows. Armonk, NY: IBM Corp.
Jacob, A., & McGovern, K. (2015). The Mirage. Confronting the hard truth about our
quest for teacher development. Brooklyn: TNT.
Kalinowski, E., Gronostaj, A., & Vock, M. (2019). Effective professional development
for teachers to foster students’ academic language proficiency across the cur-
riculum: A systematic review. AERA Open, 5(1), 1e23. https://doi.org/10.1177/
2332858419828691.
Kim, J. S., Olson, C. B., Scarcella, R., Kramer, J., Pearson, M., van Dyk, D. A., &
Land, R. E. (2011). A randomized experiment of a cognitive strategies approach
to text-based analytical writing for mainstreamed Latino English language
learners in grades 6 to 12. Journal of Research on Educational Effectiveness, 4(3),
231e263.
Knight, S. L., & Wiseman, D. L. (2005). Professional development for teachers of
diverse students: A summary of the research. Journal of Education for Students,
10(4), 387e405.
Kolano, L. Q., Da vila, L. T., Lachance, J., & Coffey, H. (2014). Multicultural teacher
education: Why teachers say it matters in preparing them for English language
learners. CATESOL Journal, 25(1), 41e65.
Landis, J. R., & Koch, G. G. (1977). An application of hierarchical Kappa-type statistics
in the assessment of majority agreement among multiple observers. Biometrics,
33(2), 363.
Lara-Alecio, R., Tong, F., Irby, B. J., Guerrero, C., Huerta, M., & Fan, Y. (2012). The effect
of an instructional intervention on middle school English learners’ science and
English reading achievement. Journal of Research in Science Teaching, 49(8),
987e1011.
*Lee, O., & Maerten-Rivera, J. L. (2012). Teacher change in elementary science in-
struction with English language learners: Results of a multiyear professional
development intervention across multiple grades. Teachers College Record,
114(8), 1e44.
Lipowsky, F. (2010). Lernen im Beruf e empirische Befunde zur Wirksamkeit von
Lehrerfortbildung [Vocational learningdempirical evidence on the effective-
ness of teacher training]. In F. H. Müller, A. Eichenberger, M. Lüders, & J. Mayr
(Eds.), Lehrerinnen und Lehrer lernen: Konzepte und Befunde zur Lehrerfortbildung
(pp. 51e72). Münster: Waxmann.
Lipowsky, F. (2014). Theoretische Perspektiven und empirische Befunde zur Wirk-
samkeit von Lehrerfort- und -weiterbildung [Theoretical perspectives and
empirical evidence on the effectiveness of teachers’ professional development
and training]. In E. Terhart, H. Bennewitz, & M. Rothland (Eds.), Handbuch der
Forschung zum Lehrerberuf (pp. 511e541). Münster: Waxmann.
Lipowsky, F., & Rzejak, D. (2015). Key features of effective professional development
programmes for teachers. Ricercazione, 7(2), 27e51.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Applied social research
methods series: Vol. 49. Thousand Oaks, CA: Sage Publications.
Luborsky, L., Diguer, L., Seligman, D. A., Rosenthal, R., Krause, E. D., Johnson, S., &
Schweizer, E. (1999). The researcher’s own therapy allegiances: A “wild card” in
comparisons of treatment efficacy. Clinical Psychology: Science and Practice, 6(1),
95e106.
Lucas, T., & Villegas, A. M. (2011). A framework for preparing linguistically
responsive teachers. In T. Lucas (Ed.), Teacher preparation for linguistically diverse
classrooms: A resource for teacher educators. New York: Taylor & Francis.
Markussen-Brown, J., Juhl, C. B., Piasta, S. B., Bleses, D., Højen, A., & Justice, L. M.
(2017). The effects of language- and literacy-focused professional development
on early educators and children: A best-evidencemeta-analysis. Early Childhood
Research Quarterly, 38, 97e115.
McFarland, J., Hussar, B., Wang, X., Zhang, J., Wang, K., Rathbun, A., & Bullock
Mann, F. (2018). The condition of education 2018. Washington, DC: U.S. Depart-
ment of Education. Retrieved on 22 November, 2018 from https://nces.ed.gov/
pubs2018/2018144.pdf.
McIntyre, E., Kyle, D., Chen, C.-T., Mun ~ oz, M., & Beldon, S. (2010). Teacher learning
and ELL reading achievement in sheltered instruction classrooms: Linking
professional development to student development. Literacy Research and In-
struction, 49(4), 334e351.
Miramontes, O. B., Nadeau, A., & Commins, N. L. (1997). Restructuring schools for
linguistic diversity: Linking decision making to effective programs. Language and
literacy series. New York: Teachers College Press.
Mulrow, C. D. (1994). Rational for systematic reviews. British Medical Journal,
309(6954), 597e599.
Munder, T., Brütsch, O., Leonhart, R., Gerger, H., & Barth, J. (2013). Researcher
allegiance in psychotherapy outcome research: An overview of reviews. Clinical
Psychology Review, 33(4), 501e511.
National Clearinghouse for English Language Acquisition. (2017). National profes-
sional development program. Retrieved on 22 November, 2018 from https://
ncela.ed.gov/national-professional-development-program.
Neuman, S. B., & Wright, T. S. (2010). Promoting language and literacy development
for early childhood educators: A mixed-methods study of coursework and
coaching. The Elementary School Journal, 111(1), 63e86.
OECD. (2016). PISA 2015 results. In Excellence and equity in education (Vol. I)Paris:
OECD publishing.
OECD [Organisation for Economic Cooperation and Development. (2009). Creating
effective teaching and learning environments: First results from TALIS. OECD
publishing. Retrieved on 22 November, 2018 from https://www.oecd-ilibrary.
org/education/creating-effective-teaching-and-learning-environments_
9789264068780-en.
*Olson, C. B., Kim, J. S., Scarcella, R., Kramer, J., Pearson, M., van Dyk, D. A., &
Land, R. E. (2012). Enhancing the interpretive reading and analytical writing of
mainstreamed English learners in secondary school: Results from a randomized
field trial using a cognitive strategies approach. American Educational Research
Journal, 49(2), 323e355.
Paetsch, J., Wolf, K. M., Stanat, P., & Darsow, A. (2014). Sprachfo €rderung von Kindern
und Jugendlichen aus Zuwandererfamilien. [Language support for children and
adolescents from immigrant families]. Zeitschrift für Erziehungswissenschaft, 24,
315e347.
Pant, H. A. (2014). Aufbereitung von Evidenz für bildungspolitische und
p€adagogische Entscheidungen: Metaanalysen in der Bildungsforschung. [Pre-
paring evidence for education policy and pedagogical decisions: Meta-analyses
in education science]. Zeitschrift für Erziehungswissenschaft, 17(4), 79e99.
Patrick, V. D. (2012). The role of superintendents as instructional leaders facilitating
 E. Kalinowski et al. / Teaching and Teacher Education 88 (2020) 102971
student achievement among ESL/EL learners through school-site professional
development. Doctoral dissertation. University of Southern California.
Quintero, D., & Hansen, M. (2017). English learners and the growing need for
qualified teachers. Retrieved on 22 November, 2018 from https://www.
brookings.edu/blog/brown-center-chalkboard/2017/06/02/english-learners-
and-the-growing-need-for-qualified-teachers/.
Samson, J. F., & Collins, B. A. (2012). Preparing all teachers to meet the needs of
English language learners: Applying research to policy and practice for teacher
effectiveness. Retrieved on 22 November, 2018 from http://files.eric.ed.gov/
fulltext/ED535608.pdf.
Sanetti, L. M. H., & Kratochwill, T. R. (2014). School Psychology book series. Treatment
integrity: A foundation for evidence-based practice in applied psychology. Wash-
ington, D.C.: American Psychological Association.
Scarcella, R. (2003). Academic english: A conceptual framework. University of Cali-
fornia linguistic minority research Institute.
Schleppegrell, M. J. (2001). Linguistic features of the language of schooling. Lin-
guistics and Education, 12(4), 431e459.
Schleppegrell, M. J. (2004). the language of schooling: A functional linguistics
perspective. Mahwah, New Jersey: Erlbaum.
Schleppegrell, M. J. (2009). In Language in academic subject areas and classroom
instruction: What is academic language and how can we teach it? Paper presented
at the workshop the role of language in school learning by the national academy of
sciences. Menlo Park, CA. Retrieved on 22 November, 2018 from https://www.
rcoe.us/educational-services/files/2012/08/What_is_Academic_Language_
Schleppegrell.pdf.
Schmo €lzer-Eibinger, S. (2013). Sprache als Medium des Lernens im Fach. [Language
as the medium for learning in the subjects.]. In M. Becker-Mrotzek, K. Schramm,
E. Thürmann, & H. J. Vollmer (Eds.), Sprache im Fach. Sprachlichkeit und
fachliches Lernen (pp. 25e40). Münster: Waxmann.
*Shanahan, T., & Shea, L. (2012). Incorporating English language teaching through
science for K-2 teachers. Journal of Science Teacher Education, 23(4), 407e428.
Stanat, P., Weirich, S., & Radmann, S. (2012). Sprach- und Lesefo € rderung [Language
and reading support]. In P. Stanat, H. A. Pant, K. Bo €hme, & D. Richter (Eds.),
Kompetenzen von Schülerinnen und Schülern am Ende der vierten Jahrgangsstufe
in den Fa €chern Deutsch und Mathematik: Ergebnisse des IQB-La €ndervergleichs
2011 (pp. 251e276). Münster: Waxmann.
Sterne, J. A. C., Egger, M., & Smith, G. D. (2001). Investigating and dealing with
publication and other biases in meta-analysis. British Medical Journal,
323(7304), 101e105.
Texas Education Agency. (2000). The Texas Successful Schools Study: Quality educa-
tion for limited English proficient students. Austin, TX.
The Campbell Collaboration. (2014). Campbell Collaboration systematic reviews:
Policies and guidelines. Campbell Policies and Guidelines Series No. 1: The
Campbell Collaboration. Retrieved on 22 November, 2018 from https://www.
campbellcollaboration.org/library/campbell-collaboration-systematic-reviews-
policies-and-guidelines.html. Version 1.3.
Timperley, H. S., Wilson, A., Barrar, H., & Fung, I. (2007). Teacher professional learning
and development: Best evidence synthesis iteration. Wellington: Ministry of
Education.
Tong, F., Irby, B. J., Lara-Alecio, R., Guerrero, C., Tang, S., & Sutton-Jones, K. L. (2018).
15
The impact of professional learning on in-service teachers’ pedagogical delivery
of literacy-infused science with middle school English learners: A randomised
controlled trial study in the U.S. Educational Studies, 21(2), 1e21.
Townsend, D., Filippini, A., Collins, P., & Biancarosa, G. (2012). Evidence for the
importance of academic word knowledge for the academic achievement of
diverse middle school students. The Elementary School Journal, 112(3), 497e518.
Uccelli, P., Galloway, E. P., Barr, C. D., Meneses, A., & Dobbs, C. L. (2015). Beyond
vocabulary: Exploring cross-disciplinary academic-language proficiency and its
association with reading comprehension. Reading Research Quarterly, 50(3),
337e356.
U.S. Department of Education. (2016). English learner tool kit for state and local ed-
ucation for State and Local Education Agencies (SEAs and LEAs). Washington, DC.
Retrieved on 22 November, 2018 from https://www2.ed.gov/about/offices/list/
oela/english-learner-toolkit/eltoolkit.pdf.
U.S. Department of Education. (2017). National professional development program.
Retrieved on 22 November, 2018 from https://www2.ed.gov/programs/nfdp/
funding.html.
Valdes, G., Kibler, A., & Walqui, A. (2014). Changes in the expertise of ESL pro-
fessionals: Knowledge and action in an era of new standards. Alexandria, VA:
TESOL International Association.
Van Roekel, D. (2011). Professional development for general education teachers of
English language learners. Washington, D.C: NEA Quality School Programs and
Resources Department.
Vaughn, S., Linan-Thompson, S., Woodruff, A. L., Murray, C. S., Wanzek, J.,
Scammacca, N., & Elbaum, B. (2008). Effects of professional development on
improving at-risk students’ performance in reading. In C. R. Greenwood,
T. R. Kratochwill, & M. Clements (Eds.), Schoolwide prevention models: Lessons
learned in elementary schools (pp. 115e142). New York: Guilford Press.
Wade, R. K. (1984). What makes a difference in inservice teacher education? A
meta-analysis of research. Educational Leadership, 42(4), 48e54.
Walsleben, L. (2008). Signs of success: District gets specific about the needs of
English language learners. JSD: English Language Learners, 29(1), 18e22.
Wei, R. C., Darling-Hammond, L., Andree, A., Richardson, N., & Orphanos, S. (2009).
Professional learning in the learning profession: A status report on teacher devel-
opment in the United States and abroad. Dallas, TX: National Staff Development
Council. Retrieved on 10 September, 2019 from https://edpolicy.stanford.edu/
sites/default/files/publications/professional-learning-learning-profession-
status-report-teacher-development-us-and-abroad.pdf.
Wilson, D. B., & Lipsey, M. W. (2001). The role of method in treatment effectiveness
research: Evidence from meta-analysis. Psychological Methods, 6(4), 413e429.
WWC [What Works Clearinghouse]. (2017). Standards handbook. Washington, DC:
U.S. Department of Education. Version 4.0.
Yoon, K. S., Duncan, T., Lee, S. W.-Y., Scarloss, B., & Shapley, K. (2007). Reviewing the
evidence on how teacher professional development affects student achievement.
Washington, DC: Institute of Education Sciences. Retrieved on 22 November,
2018 from https://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/REL_2007033.
pdf.
Zhang, Y. (2014). Investigating the impact of a university-based professional develop-
ment program for teachers of English language learners in Ohioda mixed methods
study of teacher learning and change. Doctoral dissertation.