You are on page 1of 22

Learning Disability Quarterly © 2 0 1 1 C o u n c i l for Learning Disabilities

2 0 1 1 , Vol. 3 4 , No. 1, 5 1 - 7 2

GRAPHIC ORGANIZERS AND STUDENTS WITH


LEARNING DISABILITIES: A META-ANALYSIS

Douglas D. Dexter and Charles A. Hughes

Abstract. T h i s m e t a - a n a l y s i s r e v i e w s e x p e r i m e n t a l a n d q u a s i -
e x p e r i m e n t a l studies in w h i c h upper-elementary, i n t e r m e d i a t e ,
a n d s e c o n d a r y students w i t h l e a r n i n g disabilities l e a r n e d f r o m
g r a p h i c organizers. Following a n e x h a u s t i v e search for studies
m e e t i n g specified design criteria, 5 5 s t a n d a r d i z e d m e a n effect
sizes w e r e e x t r a c t e d f r o m 1 6 a r t i c l e s i n v o l v i n g 8 0 8 p a r t i c i p a n t s .
Students a t levels r a n g i n g f r o m g r a d e 4 t o g r a d e 1 2 u s e d g r a p h i c
o r g a n i z e r s t o l e a r n in c o r e - c o n t e n t classes ( E n g l i s h / r e a d i n g , sci­
ence, social studies, m a t h e m a t i c s ) . Posttests m e a s u r e d n e a r a n d far
transfer. Across several c o n d i t i o n s , settings, a n d features, t h e use
of g r a p h i c organizers w a s associated w i t h increases in v o c a b u l a r y
knowledge, comprehension, a n d inferential knowledge. M e a n
e f f e c t sizes v a r i e d f r o m m o d e r a t e t o l a r g e b a s e d o n t y p e o f m e a s ­
ure, type of g r a p h i c organizer, a n d subject area. Conclusions,
implications for future research, a n d practical r e c o m m e n d a t i o n s
are presented.

DOUGLAS D. DEXTER, Ph.D., College of Education, The Pennsylvania State University, University Park, Pennsylvania.
CHARLES A. HUGHES, Ph.D., College of Education, The Pennsylvania State University, University Park, Pennsylvania.

As students enter the upper-elementary, intermediate, skills (Deshler, Ellis, & Lenz, 1996). These students gen­
and secondary grades, academic demands are height­ erally have difficulty connecting new material to prior
ened as material becomes more complex and the cur­ knowledge, identifying and ignoring extraneous infor­
riculum is driven by higher-order skills and advanced mation, identifying main ideas and supporting details,
concepts (Fletcher, Lyon, Fuchs, & Barnes, 2 0 0 7 ) . drawing inferences, and creating efficient problem-solv­
Students at these levels typically receive less individual ing strategies (Baumann, 1984; Holmes, 1985; Johnson,
attention than in primary grades (Hughes, Maccini, & Graham, & Harris, 1997; Kim, Vaughn, Wanzek, & Wei,
Gagnon, 2003), and are often required to learn prima­ 2004; Williams, 1993). Because many textbooks are
rily through didactic lecture and expository text presen­ written above grade-level reading ability and lack orga­
tation (Gajria, Jitendra, Sood, & Sacks, 2007; Minskoff nizational clarity (Gajria et al., 2007), these learning
& Allsopp, 2003). This shift in learning presentation, difficulties make interpreting and comprehending
rife with abstract concepts, unfamiliar content, and expository text especially challenging (Bryant, Ugel,
technical vocabulary (Armbruster, 1984), may seem Thompson, & Hamff, 1999).
daunting to most students, but especially so to students Students with LD need explicit content enhance­
with learning disabilities (LD). ments to assist in verbal (e.g., text or lecture) compre­
Students with LD often have difficulty with basic aca­ hension, and graphic organizers (GOs) have often been
demic skills (e.g., reading) and organizational/study recommended as an instructional device to assist these

51
52 1)1 X I I It & III (,1 IIS

Figure 1. Cognitive mapping example (science).

trees

rain
plants earth oceans
30% T 97.5%

clouds

atmosphere |
.001%

Downloaded from: http://www.studygs.net/mapping/

students in understanding increasingly abstract con­ may support and develop organizational strategies
cepts (Bos & Vaughn, 2002; Hughes et al., 2003; Ives & essential to both reading and writing of content area
Hoy, 2003; Kim et al., 2004; Nesbit & Adesope, 2007; material by bridging the gap between idea organization
Rivera & Smith, 1997). and written language structures (Pehrssen & Denner,
1988). Further, Boyle and Yeager (1997) pointed out
W h a t Are Graphic Organizers?
that minimizing sentences and details in the GO is an
GOs are visual and spatial displays that make rela­
essential component of cognitive mapping. They rec­
tionships between related facts and concepts more
ommend using keywords and simple drawings rather
apparent (Gajria et al., 2007; Hughes et al., 2003; Kim
than complex sentences or elaborate drawings.
et al., 2004). They are intended to promote more mean­
Teachers often make cognitive mapping GOs to use as
ingful learning and facilitate understanding and reten­
advance organizers for challenging text. Students also
tion of new material by making abstract concepts more
can fill in teacher-prepared blank cognitive mapping
concrete and connecting new information with prior
GOs during or after attending to challenging verbal
knowledge (Ausubel, 1968; Mayer, 1979). While there
material. An example of a cognitive mapping GO is
is inconsistency in the definitions of types of GOs found in Figure 1.
(Rice, 1994), we were able to classify all the studies on
Semantic Mapping. Semantic mapping (SM) is a
students with LD where effects could solely be attrib­
heuristic that enables students to recognize relevant
uted to the GO into the following five general cate­ information from lecture and text (e.g., main ideas,
gories. important supporting details), delete isolated details
Cognitive Mapping. Cognitive mapping assists in that may not be relevant to overall understanding, and
making major ideas and relationships explicit by using highlight key concepts that may have not been fully
"lines, arrows, and spatial arrangements to describe developed in a lecture or text (Bos & Anders, 1990).
text content, structure, and key conceptual relation­ Unlike cognitive mapping, when using SM, students
ships" (Darch & Eaves, 1996, p. 3 1 0 ) . By graphically and the teacher actively create a visual representation
displaying important information, cognitive mapping (e.g., relationship map or web) to represent the rela-
GRAPHIC ORGANIZERS AND LD S3

Figure 2. Semantic mapping example (social studies).

Nobles Fiefs

Vassals Farming

Serfs Castles on hills

tionships among concepts. Typically, concepts are superordinate concept. Finally, subordinate concepts
listed, and the teacher and students make predictions (e.g., concepts representing the coordinate concept) are
about how the concepts can be arranged to demonstrate listed below each coordinate concept (Bos & Anders,
those relationships on the GO (Bos & Anders, 1992). 1990, 1992; Pearson & Johnson, 1978). Figure 2 pro­
A well-made GO consists of a superordinate concept vides an example of an SM GO.
(e.g., main idea, topic) placed in an oval in the middle Semantic Feature Analysis. Semantic feature analy­
or top of the page. Coordinate concepts (e.g., categories sis (SFA) is similar to SM by helping students recognize
representing related concepts) are placed in ovals sur­ relevant information from lecture and text. This is
rounding or underneath the superordinate concept and done through a presentation of related concept charac­
connected by lines. Coordinate concepts can include a teristics in a matrix form. In SFA, unrelated concepts
variety examples, functions, or characteristics of the can be inferred directly from the chart (Darch &

Figure 3. Semantic feature analysis example (science).

Comparison of Dog Breeds


Basset Hound Old English Sheepdog Brittany Spaniel Border Collie

Energetic - + + +

Weekly Grooming + -

Good with + + ? -
Kids/Other Pets

Note. + = feature present; - = feature not present; ? = not sure.


54 DEXTER & HUGHES

Gersten, 1986). Typically, a relationship matrix is con­ ways: temporal (e.g., timeline), spatial (e.g., decision
structed with vocabulary representing the coordinate tree), sequential (e.g., flowchart), hierarchal (e.g., tax­
concepts placed along the top of the matrix, and the onomy), or comparative (e.g., Venn diagram). An
vocabulary representing the subordinate concepts example of a visual display GO is found in Figure 5.
placed along the side (Bos & Anders, 1990) The teacher Previous Research
and students can then make predictions and confirma­ Several groups of researchers have conducted reviews
tions of relationships (e.g., related, not related, not and meta-analyses of the effectiveness of using GOs
sure) between the coordinate and subordinate concepts with nondisabled students (e.g., Ausubel, 1 9 6 8 ;
(Bos &Anders, 1992). The superordinate concept serves Kulhavy, Stock, & Caterino, 1994; Mayer, 1979; Moore
as the title. Figure 3 provides an example of an SFA G & Readence, 1984; Nesbit & Adesope, 2007; Robinson,
S y n t a c t i c / S e m a n t i c Feature Analysis. Syntactic/ Katayama, DuBois, & Devaney, 1998). Based on these
semantic feature analysis (SSFA) is nearly identical to examinations of both the benefits of GOs and the
SFA but with the addition of cloze-type sentences writ­ effective design of GOs, several key findings are con­
ten based on the matrix (Bos & Anders, 1990). Cloze sistently replicated: (a) students with low verbal ability
sentences contain blank spaces replacing new vocabu­ gain more from GOs than students with high verbal
lary words. Students must use the context of the sen­ ability; (b) students with little or no prior knowledge
tence and the SFA matrix to fill in the blanks. An in a subject gain more from GOs than students with an
example of an SSFA GO is found in Figure 4. abundance of prior knowledge in a subject; (c) GOs are
Visual Display. Visual displays present concepts or especially helpful in assisting students with far-transfer
facts spatially, in a computationally efficient manner. tasks, in addition to near-transfer tasks and factual
That is, relationships between concepts are made recall; (d) GOs should be explicitly taught to students
apparent and clear by their location on the display. for maximum impact; (e) GOs should spatially group
According to Hughes et al. (2003), in a visual display, together or connect concepts so readers are more likely
facts or concepts are typically presented in one of five to perceive them as being interrelated and to draw

Figure 4. Syntactic/semantic feature analysis example (science).

Comparison of Dog Breeds


Basset Hound Old English Sheepdog Brittany Spaniel Border Collie

Energetic

Weekly Grooming

Good with Kids/ +


Other Pets

Note. + = feature present; - = feature not present; ? = not sure.

A(n) is the least energetic breed we have discussed.

A(n) requires at least weekly grooming.

A(n). and are good with other pets.


GRAPHIC ORGANIZERS AND LD 55

Figure 5 . Visual dislay example (English/language arts).

Simile Metaphor

Similarities

perceptual inferences about their relationships; (f) GOs Drawing definitive conclusions from these reviews is
should not be clustered with a lot of information; read­ problematic for several reasons, however. First, the
ers should easily perceive the phenomena or relations reviews did not take sample sizes into consideration
that are important; (g) GOs are effective because of when calculating effect sizes and comparing studies.
their computational efficiency, minimizing stress on Estimations of effect were based on individual study
the working memory; and (h) GOs can be effective characteristics without standardization. The use of
when used before, during, or after a lesson. While meta-analysis allows for statistical standardization of
these findings are promising, the vast majority of the findings, resulting in numerical values that are inter-
studies reviewed used college students as participants, pretable in a consistent pattern across all studies, thus
and few comparisons were made to students identified controlling for an individual study's sample size (Lipsey
as LD. & Wilson, 2001). Furthermore, according to Lipsey and
Two research syntheses of school-aged children with Wilson, meaningful effects and relationships where
LD (Gajria et a l , 2007; Kim et al., 2004) have focused on multiple quantitative studies with varying sample sizes
GOs. In Gajria et al., as part of an examination of several agree, as well as differential effects of study differences,
content enhancements, GOs were found to have large are more likely to be identified by meta-analytic proce­
effects for comprehension of expository text for upper- dures than by less systematic and analytic approaches.
elementary, intermediate, and secondary students with Second, the reviews focused solely on factual com­
LD. Likewise, Kim et al. found large effects for GOs on prehension measures, not on vocabulary, inference, or
the reading comprehension of upper-elementary, inter­ relational comprehension. While information on fac­
mediate, and secondary students with LD. tual comprehension is important, it may be useful to
56 DEXTER & HUGHES

measure the utility of GOs for students with LD on mul­ Research & Practice, Remedial and Special Education, and
tiple constructs rather than only on the ability to Reading Research Quarterly. This process yielded a total
answer fact-level questions about a topic. The utility of of 27 published articles to analyze, many (e.g., 20)
an intervention (e.g., GO) on multiple constructs is con­ including more than one study.
sidered important for high-quality research (Gersten, Inclusion Criteria
Baker, & Lloyd, 2000). Thus, the effectiveness of an We used six criteria to evaluate the appropriateness
intervention should be examined on all possible con­ of each study. First, the study must have included
structs. a dependent measure of near or far transfer of verbal
Third, no systematic analysis of effects was conducted (e.g., text or lecture) material and a GO as the inde­
by type of measure (e.g., near or far transfer), type of pendent variable. Studies with a mnemonic illustration
GO, subject area, or student stage of attending to verbal rather than GO as the content enhancement (e.g.,
material. This lack of clarity makes it difficult to identify Brigham, Scruggs, & Mastropieri, 1995; Mastropieri,
why the GO interventions were so effective. A more sys­ Scruggs, & Levin, 1 9 8 7 ) were excluded, as effects
tematic analysis of GO studies may allow for more pre­ could not be attributed solely to the GO. Likewise,
cise explanations of effect based on multiple criteria, studies including GOs associated with Content En­
and lead to a more robust understanding of how and hancement Routines (Bulgren, Deshler, Schumaker, &
why GOs are effective for students with LD. Lenz, 2 0 0 0 ; Bulgren, Lenz, Schumaker, Deshler, &
In the present study, we conducted a meta-analysis of Marquis, 2 0 0 2 ; Bulgren, Marquis, Lenz, Schumaker,
GO research to address the following questions: & Deshler, 2009; Bulgren, Schumaker, & Deshler, 1988;
1. What are the overall effects of GOs on the posttest Scanlon, Deshler, & Schumaker, 1996) were excluded,
performance of students with LD? because effects could not be attributed solely to the
2. Do these effects maintain over time? GO, even though GOs were vital components of each
3. Are there differential effects by type of measure of these routines. Further discussion about Content
(near or far transfer)? Enhance- ment Routines is included in the Limitations
4. Are there differential effects by type of GO? section of this meta-analysis.
5. Are there differential effects by subject area?
6. Are there differential effects by stage of attending Second, the study must have taken place in upper-
to verbal material (before, during, after instruc­ elementary-, intermediate-, or secondary-level class­
tion)? rooms (e.g., grades 4-12). This grade range was selected
because it represents a time when curricula typically
Method become more complex and students are required to
Literature Search P r o c e d u r e learn primarily through didactic lecture and expository
A three-step process was used to identify studies text presentation (Fletcher et al., 2007; Hughes et al.,
using GOs with upper-elementary, intermediate, and 2003; Minskoff & Allsopp, 2003).
secondary students with LD. First, we conducted a com­ Third, based on the recommendations of Rosenthal
prehensive computerized search of Psyclnfo, ERIC, and (1994) and Lipsey and Wilson (1993), only studies
Social Science Citation Index databases for studies from using experimental or quasi-experimental group
1975 (e.g., year LD was officially recognized by passage designs with control groups were included. Therefore,
of P.L. 94-142) to October 2009, using a list of search single-subject research studies (e.g., Gardill & Jitendra,
terms generated from previous reviews of GO studies 1999; Idol & Croll, 1987), repeated-measures studies
(e.g., Horton et al., 1993; Moore & Readence, 1984; (e.g., Lenz, Adams, Bulgren, Poulit, & Laraux, 2007),
Nesbit & Adesope, 2006). W e used the following com­ and single-group studies (e.g., Bergerud, Lovitt, &
bination of descriptors: graphic organ*, expository, verbal, Horton, 1988; Boon, Fore III, Ayres, & Spencer, 2005;
learning disab*, concept map, cognitive map, adolesc*, Horton, Lovitt, & Bergerud, 1 9 9 0 ; Lovitt, Rudsit,
semantic map, semantic feature analysis, visual display. Jenkins, Pious, & Benedetti, 1 9 8 6 ; Sinatra, Stahl-
Second, we conducted ancestral searches of identified Gemake, & Berg, 1984; Sturm & Rankin-Erickson, 2002)
articles, as well as the two most recent reviews of con­ were excluded.
tent enhancements used with students with LD (e.g., Fourth, the study must have provided sufficient quan­
Gajria et al., 2007; Kim et al., 2004). titative information (e.g., group means and standard
Finally, we conducted hand searches of the following deviations; F statistic) to permit calculation of an effect
journals to locate the most recent literature: Exceptional size (ES). One experimental study (Boyle & Weishaar,
Children, Journal of Educational Psychology, Journal of 1997) was excluded because it provided only multivari­
Learning Disabilities, The Journal of Special Education, ate analysis of covariance (MANCOVA) data without
Learning Disability Quarterly, Learning Disabilities group means and standard deviations. According to
GRAPHIC ORGANIZERS AND LD 57

Hunter and Schmidt (2004), there is no algebraically effect size estimate was computed using the following
equivalent method to compute a comparable £ 5 in formulae:
such an instance. f 3 1
Fifth, participants in the experimental and control E S
' s m = 1
" 4N- 9 E S $ m
'
groups must have included students with LD. W e
defined LD the same way as Kim et al. (2004) in their
research review and Swanson, Hoskyn, and Lee (1999) WG1 + G 2
SE = J " -J^ZI ,
in their meta-analysis (e.g., average intelligence and sm

1 n n
+

2(n + n )
G1 G2 G1 G2
poor performance in at least one academic or related
behavioral area). In each of the studies included, all
1 2n

participants were identified as students with LD. w = = G\n {n


G2 Gl + n )
G2

2 z

Finally, the study must have been published in a SE\ m 2{n Gl +n ) G2 + n n (ES' )
G1 G2 sm '
peer-reviewed journal and in English. This excluded
any studies in Dissertation Abstracts International and where N is the total sample size, ES sm is the biased stan­
unpublished studies from researchers in the field. dardized mean difference, and
While this criterion ensures that only the highest qual­
1
ity research was included in the meta-analysis (Slavin,
1995), it also represents a potential publication bias °'-'sm

(Lipsey & Wilson, 1993, 2001). This will be discussed


is the inverse variance weight used to calculate the
further in the Limitations section.
weighted mean effect size. According to Hedges,
Study C o d i n g Shymansky, and Woodworth (1989), the inverse vari­
The first author coded pertinent study features, ance weight is a better approach to accounting for the
including participant characteristics (i.e., grade level, sample size of a given study than the simpler approach
disability classification), subject area, type of GO, of weighting by sample size.
stated purpose, study contrasts, dependent measures,
Outliers
and reported findings. A graduate research assistant
Prior to analyzing the weighted mean effect size,
double-coded this information, resulting in an inter-
extreme effect sizes that may have disproportionate
rater reliability of .97. After discussion and clarification
influence on the analysis were eliminated (Lipsey &
to resolve disagreements in coding, interrater reliability
Wilson, 2001). Based on the recommendation of Burns
reached 1.00.
(2004), eliminated outliers were effect sizes that were
Individual Effect Size C a l c u l a t i o n greater than 1.5 times the mean effect size.
Using methods described by Lipsey and Wilson
(2001), standardized mean difference effect size was Data Analysis
computed using pooled standard deviation. The for­ Following transformations and outlier elimination,
mula used was: data were analyzed by computing the weighted mean
effect size:
}
*-" sm = , I(wx ES)
S p
ES =
,w
where " X i is the mean for group 1, X is the mean
G G2

for group 2, and S is the pooled standard deviation"


p
where I ( w x £5) is the summed product of the effect
(Lipsey & Wilson, p. 4 8 ) . W h e n studies only provided size and inverse variance weight, and I w is the
an F statistic, effect size was computed using a formula summed inverse variance weight.
recommended by Thalheimer and Cook (2002). The Next, the standard error of the mean effect size was
formula used was: computed using the following formula:

n + n n + n
d= t c t c

nn t c n + n -1
t r

where n is the number of treatment subjects, and n is


t c

the number of control subjects. where


Next, to correct for upwardly biased effect sizes due
to small samples, a Hedges correction (Hedges, 1 9 8 1 ;
Lipsey & Wilson, 2 0 0 1 ) was utilized. The unbiased
58 DEXTER & HUGHES

is the square root of one divided by the summed inverse sampling variance plus a constant that represents the
variance weight. The z-test for the weighted mean effect variability across the population effects (Lipsey &
size was then computed by dividing the mean effect size Wilson, 2001). The formulae are as follows:
by the standard error of the mean effect size, or in sta­
1
tistical notation: Wj = ,
set + vg

ES
sees Qr-k-l
Vg= — ,

Finally, using the z-test, the 9 5 % confidence interval for


the weighted mean effect size was computed using the 2>- = —
following formulae:
v
where e is the random-effects variance component.
Lower = ES - 1.96(se^s) We reran the analysis using this new weight to fit the
Upper = ES + 1.96{seg) random-effects model.
The preceding analyses were conducted for posttest
measures, maintenance measures, and differential
H o m o g e n e i t y Analysis effects of individual levels of independent variables
Homogeneity analysis tests whether the assumption (e.g., GO type, subject area) and dependent variables
that all of the effect sizes estimate the same population (e.g., near- or far-transfer measures). Because it was
mean is reasonable (Hunter & Schmidt, 2004). Lipsey possible to fail to reject the null hypothesis of homo­
and Wilson (2001) contend that single mean effect sizes geneity, we will present results for both the fixed-
by themselves are not sufficient descriptors of the dis­ effects and random-effects model for comparison.
tribution. Therefore, we computed a Q statistic to test Cohen's (1988) criteria for interpreting strength of
homogeneity, using the formula: effect sizes (small ES < .20, medium £5 = .50, large ES >
.80) were used to gauge the magnitude of the findings
[I(wx£S)] in this analysis.

Results
A total of 55 unique posttest effect sizes were
The Q statistic is distributed as a chi-square with degrees extracted from studies in 16 published articles meeting
of freedom (df) equaling number of ESs - 1. In our our inclusion criteria. For the purposes of this analysis,
analysis of all studies, the critical value for a chi-square each unique effect size was considered an individual
with df = 47 and p = .05 is 64. Because our calculated Q estimate of effect. (Included articles are marked with an
statistic (57.2) is less than this critical value, we can fail asterisk in the References section.) In addition, eight of
to reject the null hypothesis of homogeneity and the published articles included maintenance data ren­
assume a fixed-effects model, under which the variabil­ dering 29 more unique effect sizes. Table 1 includes
ity across effect sizes does not exceed what would be detailed information on each study, participants, vari­
expected based on sampling error. However, Lipsey and ables, measures, and individual effect sizes.
Wilson (2001) warned that a nonsignificant Q statistic
"does not always provide great confidence that a fixed Instructional Context
effects model is justified" (p. 117). Because we do not Each of the studies included instruction on the use of
have a large number of effect sizes and the correspon­ a GO. The majority of studies (Anders, Bos, & Filip,
ding samples are relatively small, the Q statistic may not 1984; Bos & Anders, 1990, 1992; Boyle, 1996, 2000,
have sufficient statistical power. Therefore, we also fit Darch & Carnine, 1986; Darch, Carnine, & Kame'enui
a random-effects model, which assumes sampling error 1986; Darch & Eaves, 1986; Darch & Gersten, 1986
plus other sources of variability are randomly distrib­ DiCecco & Gleason, 2002; Englert & Manage, 1991
uted (Lipsey & Wilson). The random-effects model is Griffin, Simmons, & Kame'enui, 1991; Hudson, 1996
also a more conservative estimate than the fixed-effects Ives, 2007; Reyes, Gallego, Duran, & Scanlon, 1989)
model of differences between moderating variables incorporated aspects of direct, explicit instruction (e.g.,
(Hunter & Schmidt, 2004). modeling, prompted practice). The authors of the
remaining study (Bos, Anders, Filip, & Jaffe, 1989)
Whereas the fixed-effects model weights each study reported that written guidelines for teaching the GO
by the inverse of the sampling variance, the random- were developed; however, these guidelines were not
effects model weights each study by the inverse of the included in the article.
GRAPHIC ORGANIZERS AND LD 59

Table 1
Independent Variables, Dependent Measures, and Effect Sizes of Individual Experiments

Subject/GO Near or Effect Size


Type/Study/ Control Far Dependent Posttest Maintenance
Participants Condition Transfer Measure M M

SOCIAL STUDIES
Semantic Mapping

Bos & A n d e r s ( 1 9 9 2 ) ; Normative Near Researcher-generated .42 .62*


Study 5: c o m p r e h e n s i o n test
2 6 upper-elementary
students w i t h LD

Normative Near Researcher-generated .30 .06

c o m p r e h e n s i o n test
Semantic Feature Analysis

Anders, Bos, & Filip Definition Near Researcher-generated 1.71*** N/A


(1984); 6 2 high school instruction c o m p r e h e n s i o n test
students with LD
Definition Near Researcher-generated 1.49*** N/A
instruction v o c a b u l a r y test

Bos, Anders, Filip, & Dictionary Near Researcher-generated 1.67*** .00


Jaffe ( 1 9 8 9 ) ; 5 0 h i g h method c o m p r e h e n s i o n test
school students
w i t h LD

SM, SFA, SSFA Combination

Bos & A n d e r s ( 1 9 9 2 ) ; Definition Near Researcher-generated .81 .86


Study 1: 4 2 u p p e r - instruction c o m p r e h e n s i o n test
elementary students
w i t h LD

Definition Near Researcher-generated .50 1.15


instruction v o c a b u l a r y test

Bos & Anders ( 1 9 9 2 ) ; Definition Near Researcher-generated 1.46 N/A


Study 3: 4 7 upper- instruction c o m p r e h e n s i o n test
elementary students
w i t h LD

Definition Near Researcher-generated 1.28 N/A


instruction v o c a b u l a r y test

Visual Display

Darch & Carnine Text-only Near Researcher-generated 1.73*** N/A


( 1 9 8 6 ) ; 2 4 upper- c o m p r e h e n s i o n test
elementary students
with LD

Text-only Far Researcher-generated .64 N/A


far-transfer
c o m p r e h e n s i o n Test
continued next page
60 DEXTER & HUGHES

Table 1 c o n t i n u e d
Independent Variables, Dependent Measures, and Effect Sizes of Individual Experiments

Subject/GO Near or Effect Size


Type/Study/ Control Far Dependent Posttest Maintenance
Participants Condition Transfer Measure M M

SOCIAL STUDIES continued


Visual Display continued

Darch, Carnine, & Directed reading Near Researcher-generated N/A


Kame'enui (1986); c o m p r e h e n s i o n test
8 4 junior high school
s t u d e n t s w i t h LD

Directed r e a d i n g Far Researcher-generated .82*** N/A


far-transfer
c o m p r e h e n s i o n test

DiCecco & Gleason Guided Near Researcher-generated .05 N/A


( 2 0 0 2 ) ; 2 4 junior h i g h discussion fact quizzes
s t u d e n t s w i t h LD

Guided Far Relational s t a t e m e n t s 94*** N/A


discussion in t w o w r i t t e n essays

Hudson (1996); 21 Note-taking Near Researcher-generated 1.58*** 2.37***


junior h i g h s t u d e n t s guide k n o w l e d g e test
w i t h LD

Note-taking Far Researcher-generated 1.00** 2.08***


guide i n f e r e n c e test

SCIENCE
Semantic Mapping

Bos & A n d e r s ( 1 9 9 0 ) ; Definition Near Researcher-generated 1.27*** .94**


61 junior high instruction v o c a b u l a r y test
students with LD

Definition Near Researcher-generated 1.33*** .54


instruction c o m p r e h e n s i o n test

Bos & A n d e r s ( 1 9 9 2 ) ; Normative Near Researcher-generated 2.16*** .78*


Study 6: 2 2 j u n i o r c o m p r e h e n s i o n test
h i g h s t u d e n t s w i t h LD

Reyes, Gallego, D u r a n , Definition Near Researcher-generated 1.27*** .94**


& Scanlon (1989); instruction c o m p r e h e n s i o n test
6 1 junior h i g h s t u d e n t s
w i t h LD

Definition Near Researcher-generated 1.33*** .47


instruction v o c a b u l a r y test

continued next page

It
GRAPHIC ORGANIZERS AND LD 61

Table 1 c o n t i n u e d
Independent Variables, Dependent Measures, and Effect Sizes of Individual Experiments

Subject/GO Near or Effect Size


Type/Study/ Control Far Dependent Posttest Maintenance
Participants Condition Transfer Measure M M

SCIENCE continued
Semantic Feature Analysis

Bos & Anders ( 1 9 9 0 Definition); Near Researcher-generated 1.03** .66*


61 junior h i g h instruction v o c a b u l a r y test
students w i t h LD

Definition Near Researcher-generated 1.47*** .44


instruction c o m p r e h e n s i o n test

Bos & Anders ( 1 9 9 2 ) ; Normative Near Researcher-generated .17 .73*


Study 6: 2 2 junior c o m p r e h e n s i o n test
h i g h students w i t h LD

Reyes, Gallego, Duran, Definition Near Researcher-generated 1.03*** .66*


& Scanlon (1989); instruction c o m p r e h e n s i o n test
6 1 junior h i g h
s t u d e n t s with LD
Definition Near Researcher-generated 1.47*** .44
instruction v o c a b u l a r y test

Syntactic/Semantic Feature Analysis

Bos & Anders ( 1 9 9 0 ) ; Definition Near Researcher-generated .64 1.38***


6 1 junior h i g h instruction v o c a b u l a r y test
students with LD

Definition Near Researcher-generated 1.18** 1.40***


instruction c o m p r e h e n s i o n test

Reyes, Gallego, Definition Near Researcher-generated .64 1.38***


Duran, & S c a n l o n instruction c o m p r e h e n s i o n test
( 1 9 8 9 ) ; 6 1 junior h i g h
students w i t h LD
Definition Near Researcher-generated 1.18*** 1.40***
instruction v o c a b u l a r y test

SM/SFA/SSFA Combination

Bos & Anders ( 1 9 9 2 ) ; Definition Near Researcher-generated 1.22 .78


Study 2: 6 1 junior instruction c o m p r e h e n s i o n test
h i g h students w i t h LD
Definition Near Researcher-generated .92 1.01
instruction v o c a b u l a r y test

Bos & Anders ( 1 9 9 2 ) ; Definition Near Researcher-generated 1.51 1.51


Study 4: 5 3 junior instruction c o m p r e h e n s i o n test
h i g h students w i t h LD

Definition Near Researcher-generated .83 .79


instruction v o c a b u l a r y test
continued next page
62 DEXTER & HUGHES

Table 1 c o n t i n u e d
Independent Variables, Dependent Measures, and Effect Sizes of Individual Experiments

Subject/GO Near or Effect Size


Type/Study/ Control Far Dependent Posttest Maintenance
Participants Condition Transfer Measure M M
SCIENCE continued
Visual Display

D a r c h & Eaves ( 1 9 8 6 ) ; Text-only Near Researcher-generated 1.29*** .35


2 2 high school c o m p r e h e n s i o n test
s t u d e n t s w i t h LD

Text-only Near Researcher-generated .64 N/A


far-transfer
c o m p r e h e n s i o n test

Griffin, S i m m o n s , & N o visual Near Researcher-generated .51 N/A


Kame'enui (1991); display c o m p r e h e n s i o n test
2 8 upper-elementary
s t u d e n t s w i t h LD

SCIENCE/SOCIAL STUDIES
Visual Display

Darch & Gersten Basal Near Researcher-generated 1.72*** N/A


(1986); 2 4 high school instruction c o m p r e h e n s i o n test
s t u d e n t s w i t h LD

Semantic Feature Analysis

Darch & Gersten Dictionary Near Researcher-generated 1.66*** .00


(1986); 2 4 high school method v o c a b u l a r y test
s t u d e n t s w i t h LD

ENGLISH/LANGUAGE ARTS
Cognitive Mapping

Boyle ( 1 9 9 6 ) ; 3 0 j u n i o r Typical Near Formal reading .34 N/A


high school students reading inventory
w i t h LD o r M M R instruction

Typical Near Below-grade-level .87** N/A


reading literal c o m p r e h e n s i o n
instruction test

Typical Near On-grade-level 1.33*** N/A


reading literal c o m p r e h e n s i o n
instruction test

Typical Far Below-grade-level .76** N/A


reading inferential
instruction c o m p r e h e n s i o n test

Typical Far On-grade-level .96*** N/A


reading inferential
instruction c o m p r e h e n s i o n test
continued next page
GRAPHIC ORGANIZERS AND LD 63

Table 1 c o n t i n u e d
Independent Variables, Dependent Measures, and Effect Sizes of Individual Experiments

Subject/GO Near or Effect Size


Type/Study/ Control Far Dependent Posttest Maintenance
Participants Condition Transfer Measure M M

ENGLISH/LANGUAGE ARTS continued


Cognitive Mapping

Mastropieri & Peters List Near Research-generated 1.42*** N/A


( 2 0 0 3 ) ; 2 0 junior h i g h illustration featured i t e m recall
students w i t h LD test

List Far Research-generated 1.06** N/A


illustration featured i t e m recall
test
Semantic Mapping

Englert & M a r i a g e Typical r e a d i n g Near W r i t t e n free recall 1.84*** N/A


( 1 9 9 1 ) ; 2 8 upper- instruction
elementary students
with LD

Visual Display

Boyle ( 2 0 0 0 ) ; 2 4 h i g h No training Near Literal c o m p r e h e n s i o n 1.14*** N/A


school students with test
LD o r M M R

No training Near Relational .91** N/A


c o m p r e h e n s i o n test

No training Far Inferential .51 N/A


c o m p r e h e n s i o n test

MATHEMATICS
Visual Display

Ives ( 2 0 0 7 ) ; Study 1: Control Near Teacher-generated .67* N/A


3 0 high school test
students w i t h LD
Control Near Researcher-generated 1.06** .94**
test ( c o n c e p t s )
Control Far Researcher-generated .16 .00
test ( s y s t e m solving)

Ives ( 2 0 0 7 ) ; Study 2: Control Near Researcher-generated .49 N/A


2 0 high school test
students w i t h LD

"'p < .001; "p < .05; *p < .1; all GOs were created by the researchers.

II
64 DEXTER & HUGHES

Generally, instruction for the experimental groups studies were conducted in a resource classroom during
included one to two sessions focused solely on how to or after the regular school day.
use the GO, one to two sessions of prompted practice W h a t Are t h e Overall Effects of GOs o n t h e
using the GO, and independent student use of the GO Posttest P e r f o r m a n c e o f Students W i t h LD?
for the remainder of sessions. During the initial ses­ After the removal of six outliers, there was a large
sions, the teacher or researcher presented the GO to overall standardized effect of GOs on the posttest per­
students and described how it illustrated relationships. formance (e.g., multiple-choice comprehension, vocab­
For example, Darch and Carnine (1986) presented their ulary, written recall) of students with LD across all
visual display via an overhead projector, and students studies (£5 = .91, SE = .062) for both random- and fixed-
followed along while the teacher used a script to effects models and a 9 5 % confidence interval of .79,
describe the various cells in the display and the inter­ 1.03 for the random effects model. Table 2 provides the
relationships between them. full comparison between the random- and fixed-effects
The following sessions generally included the instruc­ models.
tor explicitly guiding the students in creating or filling
out the GO. For example, Bos and Anders (1990) explic­ Do These Effects M a i n t a i n Over T i m e ?
itly prompted the students in each step of creating a Twenty-nine studies included maintenance measures.
hierarchical semantic map from a vocabulary list. This In each of the studies, measures consisted of multiple-
level of assistance was gradually faded. For instance, choice comprehension or vocabulary items. These
Darch and Gersten (1986) first presented a visual dis­ measures were given to students one to four weeks after
play with all the cells labeled, and prompted the entire the conclusion of the intervention.
group in answering questions about specific facts in The test of homogeneity for overall maintenance
the GO. The researchers followed this by guiding the effects produced a nonsignificant Q statistic (Q = 24.49,
students through a visual display that did not provide cv = 35.17). Therefore, similar to the overall posttest
cell labels. Finally, individual students were prompted effects, results of both the random- and fixed-effects
in labeling blank visual displays. Instruction in the models are reported. After removal of five outliers, there
remaining sessions generally focused on independent was a medium overall effect for maintenance across all
use of the GO by the students, in addition to text or lec­ studies (ES = .56, SE = .074) with a 9 5 % confidence inter­
ture presentations. However, in each of the visual dis­ val of .41, .70 for the random-effects model. Table 3 pro­
play studies all of the content was presented solely vides the full comparison between the random- and
through the GO. fixed-effects models.
Each of the interventions lasted between one and Differential Effects
seven weeks, with an additional one to four weeks Based on number of effect sizes and significant Q-val-
between posttest and maintenance measures. All of the ues, the remaining analyses of differential effects are

Table 2
Overall ES for Fixed and Random Models

9 5 % CI

ES SE ofES Z-test Lower Upper

Fixed M o d e l .9127 .062 14.677* .79 1.03

Random Model .9061 .062 14.615* .78 1.02

*/i < .001.


GRAPHIC ORGANIZERS AND LD 65

Table 3
Maintenance ES for Fixed and Random Models

9 5 % CI
£5 SE ofES Z-test Lower Upper
Fixed M o d e l .5625 .077 7.305* .41 .71

Random Model .5614 .074 7.586* .41 .70

* p < -001.

reported as the random-effects model. This provides a respectively. Overall mean effect size was 1.07 for
more conservative estimate of effect. posttest and .78 for maintenance.
Are There Differential Effects by Type of Measure Ten individual estimates of effect were calculated for
(Near or Far Transfer)? Forty-five individual estimates far-transfer measures at posttest and two individual esti­
of effect were calculated for near-transfer measures at mates of effect for maintenance. These measures tested
posttest and 27 individual estimates of effect for main­ students' ability to apply knowledge to situations not
tenance. In all but two articles, near-transfer measures directly covered in the text or lecture. For example,
consisted of researcher-generated multiple-choice ques­ Hudson (1996) included the question "Name one way
tions on material directly covered in the lessons. In the the environment influenced the culture of the Arctic
remaining two articles, Boyle (1996) used a standardized tribes" (p. 81). The teacher never stated the causal rela­
measure of reading comprehension and Englert and tion between environment and culture; only facts about
Mariage (1991) used a measure of written free recall, environment and culture were stated.

Table 4
Near- and Far-Transfer - Random-Effects Model

Posttest Maintenance
9 5 % CI 9 5 % CI
ES Lower Upper £5 Lower Upper

Near Transfer 1.065 .94 1.19 .7809 .63 .93


n = 45 « = 27

Far Transfer .6127 .36 .87 .6886 .07 1.31


n = 10 n = 2
Note, n = number of ESs.
66 DEXTER & HUGHES

Table 5
Type of Graphic Organizer - Random-Effects Model

Posttest Maintenance
95% CI 95% CI
ES Lower Upper ES Lower Upper
Cognitive .8914 .58 1.21 N/A N/A N/A
Mapping n = 7 n = 0

Semantic 1.251 .94 1.56 .6925 .38 1.01


Mapping n = 7 n = 6

Semantic Feature 1.187 1.06 1.32 .3695 .23 .51


Analysis n = 10 n = 8

Syntactic/Semantic .91 .82 .99 1.39 .95 1.83


F e a t u r e Analysis n = 4 M= 4

SM/SFA/SSFA 1.062 .93 1.19 1.013 .86 1.17


Combination n = 8 n = 6

Visual Display .7486 .56 .93 .7877 .42 1.16


n= 1 9 w= 5

Note, n = number of ESs.

As for near-transfer, far-transfer measures consisted of nance measures, SSFA and SM/SFA/SSFA Combination
researcher-generated multiple-choice questions in all had significantly larger effects than the other GO types
but one study. Boyle (1996) used a standardized meas­ (e.g., 1.39, 1.01). Table 5 provides the full comparison
ure for far transfer. The overall mean effect size was .61 between types of GOs.
for posttest and .69 for maintenance. Table 4 provides Are There Differential Effects by Subject Area?
the full comparison between near- and far-transfer Posttest effects were calculated for the subject areas of
measures. English/writing/reading, mathematics, science, and
Are There Differential Effects by Type of GO? The social studies. Large posttest effects were found for all
types of GOs used in the studies matched the defini­ subject (e.g., .96-1.05) areas except mathematics (e.g.,
tions in the introduction to this analysis (e.g., cognitive .59). Mathematics had a moderate posttest effect that
mapping, SM, SFA, SSFA, visual display). However, in was significantly smaller than the other subject areas.
one article containing eight studies (Bos & Anders, Maintenance effects were calculated for mathematics,
1992), the researchers used a combination of SM, SFA, science, and social studies. Science had a large mainte­
and SSFA. The method they utilized to present their nance effect (e.g., .80) that was significantly larger than
results prohibited disaggregation of the findings. the moderate effects for mathematics and social studies.
Therefore, a sixth category (SM/SFA/SSFA Combination) Table 6 provides the full comparison between GOs by
was added to the analysis. Large posttest effects (e.g., subject area.
.74-1.2) were found for all types of GOs except visual Are There Differential Effects by Stage of
displays. Visual displays had a moderate effect (e.g., Attending to Verbal Material (Before, During, After
.74). There were no statistically significant differences Instruction)? There was not enough information to
between GOs with large posttest effects. For mainte­ quantify differential effects by stage of instruction. All
GRAPHIC ORGANIZERS AND LD 67

but one study included GOs before and during instruc­ consistent with the theories of Ausubel (1968) and
tion, and not enough information was provided to dis­ Mayer (1979) that GOs may especially assist lower abil­
aggregate these data. One study, Englert and Mariage ity learners in both basic and higher level skills by cre­
(1991), used GOs after instruction. The unstandardized ating an easier context for assimilating information into
effect size for a near-transfer, written free recall measure their memory.
was large (ES = 1.84). Do These Effects M a i n t a i n Over T i m e ?
There was a moderate mean effect for maintenance
Discussion
(ES = .56, SE = .07) of students with LD. The significant
As was the casae in previous research syntheses (e.g.,
drop-off from posttest to maintenance is consistent
Gajria et al., 2007; Kim et al., 2004; Moore & Readence,
with the findings of the other GO research syntheses.
1984), findings from this meta-analysis indicate that The drop has been attributed to lack of clarity regarding
GOs improve the factual comprehension of upper- the duration and length of intervention sessions needed
elementary, intermediate, and secondary students with to positively affect maintenance (Gajria et al., 2007;
LD. Unlike these previous reviews, this analysis also Gersten, Fuchs, Williams, & Baker, 2001). The relatively
indicates that GOs may improve vocabulary and infer­ short duration of the intervention studies (e.g., 1-7
ence/relational comprehension for students with LD. weeks) may not have provided sufficient instruction
W h a t Are t h e Overall Effects of GOs o n t h e time for students to use GOs independently. However, a
Posttest P e r f o r m a n c e of Students W i t h LD? closer look at effects by type of GO shows that effect
Overall, there was a large mean effect for posttest per­ sizes for visual displays and SSFA were larger for main­
formance (ES = .91, SE = .06) of students with LD using tenance than posttest. This may lend support to the
both the fixed-effects and random-effects models. The visual argument hypothesis (Waller, 1981), which posits
immediate posttest performance spanned multiple con­ that GOs that are structured in a way that facilitates
structs, including multiple-choice factual comprehen­ understanding and perception of concept relationships
sion, vocabulary, and written recall requiring relational are superior to more complicated GOs that may require
comprehension. This suggests that GOs are effective in instruction to recognize conceptual relationships
not only improving basic skills (e.g., factual recall) but (Dexter, 2010). The visual displays and SSFA were more
also higher-level skills (e.g., inference). This finding is computationally efficient than the other GOs. That is,

Table 6
Subject Area - Random-Effects Model

Posttest Maintenance
95% CI 95% CI
ES Lower Upper ES Lower Upper

English/Writing/ .9612 .72 1.20 N/A N/A N/A


n = 11 n = 0

Mathematics .5942 .21 .98 .4559 .07 .99


n - 4 n = 2

Science 1.052 .88 1.23 .8035 .64 .97


YI = 2 3 n = 20

Social Studies 1.037 .85 1.22 .6535 .38 1.03


n = 19 n = 8

Note, n = number of ESs.


68 DEXTER & HUGHES

they were simple enough for students to recognize con­ groups in factual recall, but the GO groups significantly
ceptual relationships without teacher instruction. This outperformed the outline and text-only groups in iden­
may explain why maintenance effects were larger for tifying concept relations and making far-transfer con­
these types of GOs. cept comparisons.
Are There Differential Effects b y T y p e o f Are T h e r e Differential Effects b y Subject Area?
Measure (Near o r Far Transfer)? This analysis also examined the effects of GOs by
This meta-analysis also separated results into near- subject area. All subject areas had moderate to large
transfer and far-transfer measures. Near-transfer results effects for posttest and maintenance measures. The
(i.e., measures applying knowledge to situations largest effects were in science (ES = 1.05 for posttest, ES
directly covered in the text or lecture) indicate that = . 8 0 for maintenance) and the smallest effects in
GOs are effective strategies for improving factual recall, mathematics (ES = .59 for posttest, £5 = .46 for mainte­
factual and relational comprehension, and vocabulary nance). The large effects in science may be explained by
knowledge. Across all near-transfer studies, the mean the unfamiliar, technical vocabulary and content often
effect size was large (ES = 1.07), and m a i n t e n a n c e based on relationships between concepts (Lovitt et al.,
effects were moderate (ES = .78). Students using GOs 1986). This type of content lends itself to computa­
significantly outperformed their peers receiving typical tionally efficient GOs that make relationships explicit
classroom instruction on near-transfer measures. and clear. Also, students may rely more heavily on con­
Interestingly, more complicated GOs requiring inten­ tent enhancements like GOs when c o n t e n t seems
sive teacher instruction (e.g., SM, SFA) resulted in the strange or foreign.
largest effects for near-transfer posttest measures. This The small effects for mathematics may be explained
indicates that while these GOs are difficult to under­ by the fact that the information was much more
stand independently, with appropriate instruction they abstract than the other subject areas. It may also be
are superior to less complicated GOs for immediate fac­ explained by the fact that the mathematics study used
tual recall. a visual display, which only had a moderate overall
Far-transfer results (i.e., measures applying knowl­ effect. Use of GOs with mathematics concepts and solv­
edge to situations not directly covered in the text or ing systems of linear equations are only beginning in
lecture) indicate that GOs may also improve the infer­ the field. Ives (2007) offerred several implications for
ence skills and relational knowledge of secondary stu­ future research based on his initial study in this field. It
dents with LD. Across all far-transfer studies, the mean will take time and more study to fully understand the
effect size was moderate (£5 = .61), and maintenance effects of GOs on mathematics understanding.
effects were moderate (£5 = .69). Are There Differential Effects b y Stage of
It is interesting to note that for far-transfer measures, Attending t o Verbal Material (Before, During,
maintenance effect sizes were larger than posttest effect After Instruction)?
sizes. Previous research has indicated students with LD Finally, a previous GO research synthesis (Moore &
typically perform poorly on far-transfer tasks due to Readence, 1984) reported that GOs presented as text
their inability to detect underlying concepts in verbal summaries after instruction were more effective than
information due to difficulty assimilating verbal infor­ GOs presented before or during instruction. This find­
mation with previous knowledge (Suritsky & Hughes, ing cannot be corroborated by the present analysis
1991). The results of this analysis demonstrate that because only one study (Englert & Mariage, 1991)
GOs may bridge the gap between verbal information used GOs after instruction. While this study had an
and prior knowledge and assist students with LD in far- extremely large effect size (1.84), more studies are
transfer tasks. This finding supports Mayer's (1979) needed to confirm this finding. The current analysis
assimilation theory, which posits that GOs that assim­ points to effective instruction and choice of GO to be
ilate material to a broader set of past experiences allow more important than stage of attending to verbal mate­
superior transfer to new situations. rial in effectiveness of the intervention.
The finding is also consistent with the research of Methodological Limitations
Robinson and colleagues (Robinson et al., 1 9 9 8 ; There are two methodological limitations to the con­
Robinson & Kiewra, 1995; Robinson & Schraw, 1994; clusions of this analysis. First, there is the possibility of
Robinson & Skinner, 1996), comparing visual displays a publication bias. According to Smith (1980), as well as
(e.g., tree diagrams, matrices, network charts) with tra­ Lipsey and Wilson (1993), published articles have a
ditional, non-graphic outlines. In each of the studies, larger mean effect size than unpublished studies. This
groups of nondisabled college students using GOs and bias, also known as a "file-drawer" effect, happens
traditional outlines equally outperformed text-only because studies with null findings are less likely to be
GRAPHIC ORGANIZERS AND LD 69

published by major journals, often leaving offending tant to note that all but one study used measures that
data in a researcher's file drawer (Lipsey & Wilson, were researcher-created and closely tied to the content.
2001). In this analysis, based on the advice of Slavin While these measures should have good content valid­
(1995), we purposefully selected only published studies ity, there is no way to measure broader construct valid­
to ensure the highest quality of research designs. While ity. Only Boyle (1996) used a standardized measure for
this eliminated our ability to compare mean effect sizes reading comprehension. This fact may limit the gener­
of published studies versus unpublished studies, we alizability of these findings and questions the actual
were able to utilize Rosenthal's (1979) fail-safe N statis­ level of understanding obtained by students in the GO
tic, which was later adapted by Orwin (1983). This sta­ conditions.
tistic determines "the number of studies with an effect Finally, studies using Content Enhancement
size of zero needed to reduce the mean effect size to a Routines were not included in this meta-analysis
specified or criterion level" (Lipsey & Wilson, 2 0 0 1 , because their effects could not be attributed solely to
p. 166). the GO, even though GOs are a vital component of
The formula is as follows: each routine. Research on Content Enhancement
Routines has been conducted for over two decades,
focusing primarily on assisting all students, including
*o = * [ S - l " , those with LD, thrive in the often rigorous intermedi­
ate and secondary general education content class­

0
[ES C J
where k is the number of effect sizes of zero needed to
rooms (Bulgren, 2006; Bulgren, Deshler, & Lenz, 2007
These routines have been shown to improve both basic
reduce the mean effect size to ES (criterioneffect level),
C skills (e.g., factual knowledge, comprehension) and
k is the number of current studies, and ES is the cur­
k higher-order skills (e.g., manipulation, extension, gen­
rent weighted mean effect size. Using this statistic, eralization) for students with LD (e.g., Bulgren et al.,
reduction of our overall weighted effect size of .91 to 2000; Bulgren et al., 2002; Bulgren et al., 2009; Scanlon
.50 would take 45 additional unpublished studies with et al., 1996). While results of these studies did not fit
an effect size of zero. While this may be a possibility, into our analysis, they do serve as examples of effective
it is unlikely that 45 additional null studies exist in interventions utilizing GOs as a component of a larger
researchers' file drawers. routine.
Second, our 55 unique effect sizes or studies were
culled from only 16 published articles. While this is Implications for P r a c t i c e
an acceptable practice (Lipsey & Wilson, 2001), it does The major implication of this study for applied prac­
limit the generalizability of the findings because there tice is consistent with assimilation theory and the
were only 21 distinct samples of students with LD visual argument hypothesis; that is, more instruction-
(total N - 808). Therefore, caution should be exercised intensive types of GOs (e.g., SM, SFA) are better for
in generalizing these findings to all intermediate and immediate factual recall while more computationally
secondary students with LD. efficient GOs (e.g., visual display, SSFA) are better for
maintenance and transfer. This knowledge can help
Individual Study L i m i t a t i o n s teachers in designing GOs for initial instruction and for
Three limitations to the individual studies warrant re-teaching, studying, and retention purposes. For
consideration. First, while all of the effect sizes in the instance, a semantic map for initial instruction, fol­
present analysis were based on differences between a lowed by a simpler visual display for review and study
treatment group and a control group, it is not clear if will potentially maximize the effects of recall, mainte­
the control conditions constituted an adequate stan­ nance, and far-transfer for students with LD.
dard to measure the effects of GO interventions Another implication for practice is that, regardless of
(Gersten, Baker, & Lloyd, 2000). The control conditions GO type, a teacher must explicitly teach students how
in the studies used primarily typical classroom practices to use a given GO. Students with LD need explicit
(e.g., dictionary instruction) rather than more closely instruction to understand how concepts are related, to
comparable practices (e.g., outlines, structured over­ recognize differences between main and subordinate
views). While this provides much evidence for GO ideas, and to put all the pieces together to make a clear
effects over typical classroom practice, it does not yield picture of the content being learned no matter how
information comparing GOs with other researched implicit a GO may seem. A teacher's use of effective
practices (Kim et al., 2004) instruction practices (e.g., modeling, corrective feed­
Second, while the results indicate large effects for back, etc.) will positively impact the intervention's
vocabulary, inference, and comprehension, it is impor­ effectiveness.
70 DEXTER & HUGHES

Conclusions a n d Implications for Future Armbruster, B. B. ( 1 9 8 4 ) . The problems of "inconsiderate" text. In


Research G. G. Duffy, L. R. Roehler, & J . M. Mason (Eds.), Comprehension
instruction: Perspectives and suggestions (pp. 2 0 2 - 2 1 7 ) . New York,
This meta-analysis found that, in comparison to activ­ NY: Longman.
ities such as reading text passages, attending to lectures, Ausubel, D. P. ( 1 9 6 8 ) . Educational psychology: A cognitive view. New
and participating in typical classroom practice (e.g., dic­ York, NY: Holt, Rinehart, & Winston.
tionary instruction), GOs are more effective on posttest, Baumann, J . F. ( 1 9 8 4 ) . The effectiveness of a direct instruction par­
maintenance, and transfer measures. However, this con­ adigm for teaching m a i n idea comprehension. Reading Research
Quarterly, 20, 9 3 - 1 1 5 .
clusion must be tempered for several reasons.
Bergerud, D., Lovitt, T. C , & Horton, S. ( 1 9 8 8 ) . The effectiveness
First, each of the studies took place in self-contained of textbook adaptations in life science for high school students
resource classrooms. This may not be typical for today's with learning disabilities. Journal of Learning Disabilities, 21, 70-
upper-elementary, intermediate, and secondary stu­ 76.
dents with LD. As many students with LD are now fully Boon, R. T., Fore III, C , Ayres, K., & Spencer, V. G. ( 2 0 0 5 ) . The
effects of cognitive organizers to facilitate content-area learning
included with nondisabled peers in core c o n t e n t
for students with mild disabilities: A pilot study. Journal of
classes, it is important to closely examine how GOs will Instructional Psychology, 32, 1 0 1 - 1 1 7 .
work in this setting. The feasibility and practicality of *Bos, C. S., & Anders, P. L. ( 1 9 9 0 ) . Effects of interactive vocabu­
GOs must be examined in general education settings lary instruction o n the vocabulary learning and reading com-
and recommendations for effective use put forth. pre-hension of junior-high learning disabled students. Learning
Disability Quarterly, 13, 3 1 - 4 2 .
Second, there is great need for GO replication studies. *Bos, C. S., & Anders, P. L. ( 1 9 9 2 ) . Using interactive teaching and
Only three articles in the present meta-analysis were learning strategies to p r o m o t e text c o m p r e h e n s i o n and content
published in the past 10 years. More current group learning for students with learning disabilities. International
design, randomized control trials, is needed to fully val­ Journal of Disability, Development, and Education, 39, 2 2 5 - 2 3 8 .
idate the benefits of GOs across all secondary students *Bos, C. S., Anders, P. L., Filip, D., & Jaffe, L. E. ( 1 9 8 9 ) . The effects
of an interactive instructional strategy for e n h a n c i n g reading
with LD.
c o m p r e h e n s i o n and c o n t e n t area learning for students with
Finally, for student independent practice, it was not learning disabilities. Journal of Learning Disabilities, 22, 3 8 4 -
always clear from the studies if the GO was used cor­ 389.
rectly or at all. For instance, when students independ­ Bos, C. S., & Vaughn, S. ( 2 0 0 2 ) . Strategies for teaching students with
ently filled in a blank GO, there was no reported learning and behavior problems (5th ed.). Boston, MA: Allyn &
Bacon.
procedure for ascertaining if they were properly labeling
*Boyle, J . R. ( 1 9 9 6 ) . The effects of a cognitive mapping strategy on
main and subordinate details. Likewise, several of the the literal and inferential comprehension of students with mild
studies (e.g., Anders, Bos, & Filip, 1984; Bos & Anders, disabilities. Learning Disability Quarterly, 19, 8 6 - 9 8 .
1990; Bos & Anders, 1992) reported students had a GO *Boyle, J. R. ( 2 0 0 0 ) . The effects of a Venn diagram strategy on the
and text to study for the posttests. They did not include literal, inferential, and relational comprehension of students
with mild disabilities. Learning Disabilities: A Multidisciplinary
a procedure for making sure the students were actually
Journal, 10, 5 - 1 3 .
using the GO to study. These students may have been Boyle, J. R., & Weishaar, M. ( 1 9 9 7 ) . The effects of expert-generated
using the text as their study guide. This lack of control versus student-generated cognitive organizers on the reading
may somewhat negate the attribution of effects to the comprehension of students with learning disabilities. Learning
GO. Future research must tightly control for these Disabilities Research & Practice, 12, 2 2 8 - 2 3 5 .
potential problem areas. Boyle, J. R., & Yeager, N. ( 1 9 9 7 ) . Blueprints for learning: Using cog­
nitive frameworks for understanding. Teaching Exceptional
Taking the above issues into account, the evidence in Children, 29, 2 6 - 3 1 .
this analysis should still persuade educational practi­ Brigham, F. J . , Scruggs, T. E., & Mastropieri, M. A. ( 1 9 9 5 ) .
tioners to make well-planned and well-instructed use of Elaborative maps for enhanced learning of historical informa­
tion: Uniting spatial, verbal, and imaginable information. The
GOs. There were no significant negative effects across
Journal of Special Education, 28, 4 4 0 - 4 6 0 .
any of the categories of analysis and no other identified Bryant, D. P., Ugel, N., T h o m p s o n , S., & Hamff, A. ( 1 9 9 9 ) .
detrimental effect. A thoughtful combination of types Instructional strategies for content-area reading instruction.
of GOs will help make the learning process more effi­ Intervention in School and Clinic, 34, 2 9 3 - 3 0 2 .
cient for upper-elementary, intermediate, and second­ Bulgren, J . A. ( 2 0 0 6 ) . Integrated content e n h a n c e m e n t routines:
ary students with LD. Responding to the needs of adolescents with disabilities in rig­
orous inclusive secondary content classes. Teaching Exceptional
Children, 38, 5 4 - 5 8 .
References Bulgren, J . A., Deshler, D. D., Schumaker, J . B., & Lenz, B. K.
* Anders, P. L., Bos, C. S., & Filip, D. ( 1 9 8 4 ) . The effect of semantic ( 2 0 0 0 ) . The use and effectiveness of analogical instruction in
feature analysis on the reading comprehension of learning dis­ diverse secondary c o n t e n t classrooms. Journal of Educational
abled students. In J. A. Niles & L. A. Harris (Eds.), Changing per­ Psychology, 92, 4 2 6 - 4 4 1 .
spectives on reading/language processing and instruction (pp. Bulgren, J., Deshler, D. D., & Lenz, B. K. ( 2 0 0 7 ) . Engaging adoles­
1 6 2 - 1 6 6 ) . Rochester, NY: National Reading Conference. cents with LD in higher order thinking about history concepts
GRAPHIC ORGANIZERS AND LD 71

using integrated c o n t e n t e n h a n c e m e n t routines. Journal of 6, 1 0 7 - 1 2 8 .


Learning Disabilities, 40, 1 2 1 - 1 3 3 . Hedges, L. V., Shymansky, J. A., & W o o d w o r t h , G. ( 1 9 8 9 ) . A prac­
Bulgren, J . A., Lenz, B. K., Schumaker, J . B., Deshler, D. D., & tical guide to modern methods of meta-analysis. Washington, DC:
Marquis, J. G. ( 2 0 0 2 ) . The use and effectiveness of a comparison National Science Teachers Association.
routine in diverse secondary c o n t e n t classrooms. Journal of Holmes, B. C. ( 1 9 8 5 ) . The effects of a strategy and sequenced mate­
Educational Psychology, 94, 3 5 6 - 3 7 1 . rial o n t h e inferential c o m p r e h e n s i o n of disabled readers.
Bulgren, J . A., Marquis, J . G., Lenz, B. K., Schumaker, J . B., & Journal of Learning Disabilities, 18, 5 4 2 - 5 4 6 .
Deshler, D. D. ( 2 0 0 9 ) . Effectiveness of question exploration t o Horton, P. B., McConney, A. A., Gallo, M., Woods, A. L., Senn, G.
enhance students' written expression of content knowledge and J . , & Hamelin, D. ( 1 9 9 3 ) . An investigation of the effectiveness of
comprehension. Reading and Writing Quarterly, 25, 2 7 1 - 2 8 9 . concept mapping as an instructional tool. Science Education,
Bulgren, J. A., Schumaker, J. B., & Deshler, D. D. ( 1 9 8 8 ) . Effective­ 77, 9 5 - 1 1 1 .
ness of a concept teaching routine in enhancing the perform­ Horton, S. V., Lovitt, T. C , & Bergerud, D. ( 1 9 9 0 ) . The effective­
ance of LD students in secondary-level m a i n s t r e a m classes. ness of graphic organizers for three classifications of secondary
Learning Disabilities Quarterly, 11, 3 - 1 7 . students in content area classes. Journal of Learning Disabilities,
Burns, M. ( 1 9 9 4 ) . Empirical analysis of drill ratio research. 23, 12-22.
Remedial and Special Education, 25, 1 6 7 - 1 7 3 . "Hudson, P. ( 1 9 9 6 ) . Using a learning set t o increase the test per­
Cohen, J . ( 1 9 8 8 ) . Statistical power analysis for the behavioral sciences formance of students with learning disabilities in social studies
(2nd ed.). Hillsdale, NJ: Erlbaum. classes. Learning Disabilities Research & Practice, 11, 7 8 - 8 5 .
*Darch, C , & Carnine, D. ( 1 9 8 6 ) . Teaching content area material Hughes, C. A., Maccini, P., & Gagnon, J . C. ( 2 0 0 3 ) . Interventions
to learning disabled students. Exceptional Children, 53, 2 4 0 - 2 4 6 . that positively impact the performance of students with learn­
*Darch, C. B., Carnine, D. W., & Kame'enui, E. J . ( 1 9 8 6 ) . The role ing disabilities in secondary general education classes. Learning
of graphic organizers a n d social structure in c o n t e n t area Disabilities, 12, 1 0 1 - 1 1 1 .
instruction. Journal of Reading Behavior, 18, 2 7 5 - 2 9 5 . Hunter, J . E., & Schmidt, F. L. ( 2 0 0 4 ) . Methods of meta-analysis:
*Darch, C , & Eaves, R. ( 1 9 8 6 ) . Visual displays to increase compre­ Correcting error and bias in research findings. Thousand Oaks, CA:
hension of high school learning-disabled students. The Journal Sage Publications, Inc.
of Special Education, 20, 3 0 9 - 3 1 8 . Idol, L., & Croll, V. J . ( 1 9 8 7 ) . Story-mapping training as a means
*Darch, C , & Gersten, R. ( 1 9 8 6 ) . Direction-setting activities in of i m p r o v i n g reading c o m p r e h e n s i o n . Learning Disability
reading c o m p r e h e n s i o n : A c o m p a r i s o n of t w o approaches. Quarterly, 10, 2 1 4 - 2 2 9 .
Learning Disability Quarterly, 9, 2 3 5 - 2 4 3 . *Ives, B. ( 2 0 0 7 ) . Graphic organizers applied to secondary algebra
Deshler, D. D., Ellis, E., & Lenz, B. K. ( 1 9 9 6 ) . Teaching adolescents instruction for students with learning disorders. Learning
with learning disabilities: Strategies and methods. Denver, CO: Disabilities Research & Practice, 22, 1 1 0 - 1 1 8 .
Love. Ives, B., & Hoy, C. ( 2 0 0 3 ) . Graphic organizers applied to higher-
Dexter, D. D. ( 2 0 1 0 ) . Graphic organizers and their effectiveness for level secondary mathematics. Learning Disabilities Research &
students with learning disabilities. Thalamus, 26, 5 1 - 6 7 . Practice, 18, 3 6 - 5 1 .
*DiCecco, V. M., & Gleason, M. M. ( 2 0 0 2 ) . Using graphic organiz­ J o h n s o n , L., Graham, S., & Harris, K. R. ( 1 9 9 7 ) . The effects of goal
ers to attain relational knowledge from expository text. Journal setting and self-instructions o n learning a reading comprehen­
of Learning Disabilities, 35, 3 0 6 - 3 2 0 . sion strategy: A study with students with learning disabilities.
*Englert, C. S., & Mariage, T. V. ( 1 9 9 1 ) . Making students partners Journal of Learning Disabilities, 30, 8 0 - 9 1 .
in the comprehension process: Organizing the reading POSSE. Kim, A., Vaughn, S., Wanzek, J., 6k Wei, S. ( 2 0 0 4 ) . Graphic organ­
Learning Disability Quarterly, 14, 1 2 3 - 1 3 8 . izers and their effects o n the reading comprehension of students
Fletcher, J . M., Lyon, G. R., Fuchs, L. S., & Barnes, M. A. ( 2 0 0 7 ) . with LD: A synthesis of research. Journal of Learning Disabilities,
Learning disabilities: From identification to intervention. New York, 37, 1 0 5 - 1 1 8 .
NY: The Guilford Press. Kulhavy, R. W., Stock, W . A., & Caterino, L. C. ( 1 9 9 4 ) . Reference
Gajria, M., Jitendra, A. K., Sood, S., & Sacks, G. ( 2 0 0 7 ) . Improving maps as a framework for remembering text. In W . Schnotz & R.
c o m p r e h e n s i o n of expository t e x t in students with LD: A W . Kulhavy (Eds.). Comprehension of graphics (pp. 1 5 3 - 1 6 2 ) . New
research synthesis. Journal of Learning Disabilities, 40, 2 1 0 - 2 2 5 . York, NY: Elsevier Science.
Gardill, M. C , & Jitendra, A. K. ( 1 9 9 9 ) . Advanced story m a p Lenz, B. K., Adams, G. L., Bulgren, J. A., Pouliot, N., & Laraux, M.
instruction: Effects o n the reading comprehension of students ( 2 0 0 7 ) . Effects of curriculum maps and guiding questions o n the
with learning disabilities. The Journal of Special Education, 33, test p e r f o r m a n c e of adolescents with learning disabilities.
2-17. Learning Disability Quarterly, 30, 2 3 5 - 2 4 4 .
Gersten, R., Baker, S., & Lloyd, J. W. ( 2 0 0 0 ) . Designing high-qual­ Lipsey, M. W . , & Wilson, D. B. ( 1 9 9 3 ) . The efficacy of psychologi­
ity research in special education: Group experimental design. cal, educational, a n d behavioral t r e a t m e n t . The American
The Journal of Special Education, 34, 2 - 1 8 . Psychologist, 48, 1 1 8 1 - 1 2 0 1 .
Gersten, R., Fuchs, L. S., Williams, J . P., & Baker, S. ( 2 0 0 1 ) . Lipsey, M. W., & Wilson, D. B. ( 2 0 0 1 ) . Practical meta-analysis (Vol.
Teaching reading comprehension strategies to students with 4 9 ) . Thousand Oaks, CA: Sage Publications, Inc.
learning disabilities: A review of research. Review of Educational Lovitt, T., Rudsit, J., Jenkins, J . , Pious, C , & Benedetti, D. ( 1 9 8 6 ) .
Research, 71, 2 7 9 - 3 2 0 . Adapting science materials for regular and learning disabled sev­
*Griffin, C. C , Simmons, D. C , & Kame'enui, E. J . ( 1 9 9 1 ) . enth graders. Remedial and Special Education, 7, 3 1 - 3 9 .
Investigating the effectiveness of graphic organizers instruction Mastropieri, M. A., Scruggs, T. E., & Levin, J . R. ( 1 9 8 7 ) . Learning-
on the comprehension and recall of science content by students disabled students' m e m o r y for expository prose: M n e m o n i c ver­
with learning disabilities. Reading, Writing, and Learning sus n o n m n e m o n i c pictures. American Educational Research
Disabilities, 7, 3 5 5 - 3 7 6 . Journal, 24, 5 0 5 - 5 1 9 .
Hedges, L. V. ( 1 9 8 1 ) . Distribution theory for Glass's estimator of Mayer, R. E. ( 1 9 7 9 ) . C a n advanced organizers influence meaning­
effect size and related estimators. Journal of Educational Statistics, ful learning? Review of Educational Research, 49, 3 7 1 - 3 8 3 .
72 DEXTER AND HUGHES

Minskoff, E., & Allsopp, D. ( 2 0 0 3 ) . Academic success strategies for Rosenthal, R. ( 1 9 9 4 ) . Statistically describing and combining stud­
adolescents with learning disabilities and ADHD. Baltimore, MD: ies. In H. Cooper & L. V. Hedges (Eds.), The handbook of research
Brookes Publishing. synthesis (pp. 2 3 1 - 2 4 4 ) . New York, NY: Russell Sage Foundation.
Moore, D. W., & Readence, J . F. ( 1 9 8 4 ) . A quantitative and quali­ Scanlon, D., Deshler, D. D., & Schumaker, J . ( 1 9 9 6 ) . C a n a strat­
tative review of graphic organizer research. Journal of Educational egy be taught and learned in secondary inclusive classrooms?
Research, 78, 1 1 - 1 7 . Learning Disabilities Research and Practice, 11, 4 1 - 5 7 .
Nesbit, J . C , & Adesope, O. O. ( 2 0 0 6 ) . Learning with concept Sinatra, R. C , Stahl-Gemake, J . , & Berg, D. N. ( 1 9 8 4 ) . Improving
and knowledge maps: A meta-analysis. Review of Educational reading comprehension of disabled readers through mapping.
Research, 76, 4 1 3 - 4 4 8 .
The Reading Teacher, 38, 2 2 - 2 9 .
Orwin, R. G. ( 1 9 8 3 ) . A fail-safe iV for effect size in meta-analysis.
Slavin, R. E. ( 1 9 9 5 ) . Best evidence synthesis: An intelligent alter­
Journal of Educational Statistics, 8, 1 5 7 - 1 5 9 .
native t o meta-analysis. Journal of Clinical Epidemiology, 48,
Pearson, P. D., & J o h n s o n , D. ( 1 9 7 8 ) . Teaching reading comprehen­
9-18.
sion. New York, NY: Holt, Rinehart, & Winston.
Pehrsson, R., & Denner, P. ( 1 9 8 8 ) . Semantic organizers: Smith, M. L. ( 1 9 8 0 ) . Publication bias a n d meta-analysis.
Implications for reading a n d writing. Topics in Language Evaluation in Education, 4, 2 2 - 2 4 .
Disorders, 8, 2 4 - 3 7 . Sturm, J . M., & Rankin-Erickson, J . L. ( 2 0 0 2 ) . Effects of hand-
*Reyes, E. I., Gallego, M. A., Duran, G. Z., & Scanlon, D. J . ( 1 9 8 9 ) . drawn and computer-generated concept mapping o n the expos­
Integration of internal concepts and external factors: Extending itory writing of middle school students with learning
the knowledge of learning disabled adolescents. The Journal of disabilities. Learning Disabilities Research & Practice, 17, 1 2 4 - 1 3 9 .
Early Adolescence, 9, 1 1 2 - 1 2 4 . Suritsky, S. K., & Hughes, C. A. ( 1 9 9 1 ) . Benefits of notetaking:
Rice, G. E. ( 1 9 9 4 ) . Need for explanations in graphic organizer Implications for secondary and postsecondary students with
research. Reading Psychology: An International Quarterly, 15, 3 9 - learning disabilities. Learning Disability Quarterly, 14, 7-18.
67. Swanson, H. L., Hoskyn, M., & Lee, C. ( 1 9 9 9 ) . Interventions for stu­
Rivera, D. P., & Smith, D. ( 1 9 9 7 ) . Teaching students with learning dents with learning disabilities. New York, NY: Guilford Press.
and behavior problems (3rd ed.). Boston, MD: Allyn & Bacon. Thalheimer, W., & Cook, S. ( 2 0 0 2 , August). How to calculate effect
Robinson, D. H., Katayama, A. D., DuBois, N. F., & Devaney, T. sizes from published research articles: A simplified methodology.
( 1 9 9 8 ) . Interactive effects of graphic organizers and delayed
Retrieved from http://work-learning.com/effect_sizes.htm.
review o n concept acquisition. Journal of Experimental Education,
Waller, R. ( 1 9 8 1 , April). Understanding network diagrams. Paper pre­
67, 1 7 - 3 1 .
sented at the annual meeting of the American Educational
Robinson, D. H., & Kierwa, K. A. ( 1 9 9 5 ) . Visual argument: Graphic
Research Association, Los Angeles.
organizers are superior to outlines in improving learning from
text. Journal of Educational Psychology, 87, 4 5 5 - 4 7 6 . Williams, J. P. ( 1 9 9 3 ) . Comprehension of students with or without
Robinson, D. H., & Schraw, G. ( 1 9 9 4 ) . Computational efficiency learning disabilities: Identification of narrative themes and idio­
through visual argument: Do graphic organizers c o m m u n i c a t e syncratic text representations. Journal of Educational Psychology,
relations in t e x t t o o effectively? Contemporary Educational 85, 6 3 1 - 6 4 1 .
Psychology, 19, 3 9 9 - 4 1 5 .
Robinson, D .H., & Skinner, C. H. ( 1 9 9 6 ) . W h y graphic organizers * = Studies included in meta-analysis.
facilitate search processes: Fewer words or c o m p u t a t i o n a l l y
efficient indexing? Contemporary Educational Psychology, 21,
166-180. Please address correspondence about this article to: Douglas D.
Rosenthal, R. ( 1 9 7 9 ) . The "file-drawer" problem and tolerance for Dexter, 2 3 0 CEDAR, The Pennsylvania State University, University
null results. Psychological Bulletin, 86, 6 3 8 - 6 4 1 . Park, PA 1 6 8 0 2 ; email:Dddl76@psu.edu

You might also like