You are on page 1of 13

Gifted Child Quarterly

Volume 51 Number 3
Summer 2007 272-284
© 2007 National Association for
Gifted Children
10.1177/0016986207302722
A Study of Curriculum Effectiveness in http://gcq.sagepub.com
hosted at

Social Studies http://online.sagepub.com

Catherine A. Little
University of Connecticut
Annie Xuemei Feng
Joyce VanTassel-Baska
The College of William and Mary
Karen B. Rogers
University of New South Wales
Linda D. Avery
Private Consultant

Abstract: This quasi-experimental study examines the effects on student performance of a Javits-funded curriculum
designed to respond to the needs of high-ability students in elementary and middle school social studies. The cur-
riculum, implemented with all students in heterogeneous classrooms, addresses state standards while integrating
advanced content, higher level process emphases, and a conceptual orientation. Data collection focuses on student
performance in conceptual reasoning, critical thinking, and content learning and on teacher demonstration of specific
desired teaching behaviors. Results demonstrate significant and important differences between treatment and com-
parison groups in the area of content learning, favoring the treatment group; no significant differences are found for
the small subsample of gifted students. Subanalyses yield differential results for specific units and schools, poten-
tially indicating issues of treatment fidelity. Contextual challenges and implications of the study are discussed,
including issues related to social studies curriculum implementation and differentiation in the current standards-based
environment.

Putting the Research to Use: This study provides evidence useful in practice and in further research, in terms of
both its specific findings and the contextual issues that emerged in the implementation of the study. First, the study
supports the idea that curriculum that integrates higher level processes and specific conceptual thinking activities
with strong content may yield content gains as strong as or stronger than a more direct, knowledge-based struc-
ture for teaching to standards. In an era heavily focused on standards coverage, this finding is an important
reminder that content standards may be used as a basic frame for a larger tapestry of more complex teaching and
learning in a content area. Second, the study demonstrates that such integrated and challenging curriculum and
instruction, designed to respond to the needs of highly able learners, may also promote growth gains for students
not identified as gifted, supporting the need for high expectations for all learners. Third, the study supports claims
from the social studies literature that integration of the habits of mind of the social science disciplines is impor-
tant for content learning.
At the same time, the study reminds practitioners that although challenging curriculum designed to meet the
needs of gifted students may promote learning for nonidentified students, appropriate instructional differentiation
is still necessary to strengthen the possibility that students of varying ability levels will benefit from the learning
experience and that the challenge level for highly able students is not diluted. Focusing on a content area that often
receives limited attention and limited differentiation in the elementary and middle school years, this study sup-
ports greater research and practice attention to the needs of gifted students in their social studies learning.

Keywords: social studies; gifted; critical thinking; conceptual thinking; curriculum

272

Downloaded from gcq.sagepub.com at PENNSYLVANIA STATE UNIV on March 4, 2016


C urriculum reform is a critical element of the larger
educational reform agenda and offers tremendous
potential for classroom-level impact (Cuban, 1998).
the curriculum framework for the series of units. Most
of the units incorporated history as the central disci-
pline of study. In an effort to reflect key principles of
National Excellence: A Case for Developing America’s the history standards, they were written with a heavy
Talent (U.S. Department of Education, 1993) clearly focus on historical analysis and text interpretation,
delineated the need for higher standards and more while also incorporating interdisciplinary emphases
powerful curriculum interventions in our schools. In an and an inquiry approach. Reflecting the interdiscipli-
era of high standards and ever-increasing demands for nary nature of the various social science disciplines,
accountability, the necessity of linking strong content the units were designed to address connections
to strong learning goals to strong pedagogical practice between history and geography, between economics
is paramount. However, the complexity of translating and government and between all of these and other
national and state content standards into effective mod- subject areas, including strong connections to language
els for curriculum and instruction to reach a wide range arts and, where relevant, to mathematics, science, and
of learner abilities cannot be overstated, particularly fine arts. These interdisciplinary connections were
with regard to responding to the needs of highly able emphasized through specific content applications and
learners. through integration of the interdisciplinary concept of
The Center for Gifted Education at the College of “systems” as a major emphasis. Thus, the project
William and Mary received a grant from the Jacob K. attempted to coalesce key understandings about effec-
Javits program of the United States Department of tive social studies teaching with relevant understand-
Education to design, develop, and field-test curriculum ings about supporting the education of high-end
units in social studies responsive to curriculum reform learners.
recommendations and specifically targeted to high- Project Phoenix was designed to demonstrate how
ability learners. Under the title “Project Phoenix,” the the development and implementation of high-powered,
grant supported curriculum development and dissem- interdisciplinary curriculum in social studies can raise
ination, professional development, parent/community the threshold of performance for students at all ability
involvement, and data collection during a 3-year levels in the regular classroom, as well as effectively
period. responding to the needs of high-ability learners. By
The curriculum units for Project Phoenix were setting high standards for curriculum content and
designed using the Integrated Curriculum Model (ICM; instructional pedagogy and by working with adminis-
VanTassel-Baska, 1986, 1995), organized around trators, teachers, students, and parents to embed these
advanced content, higher level processes and product expectations in classroom practice with extensions in
development, and concept learning. The units have the home and community, the project worked to trans-
also been aligned with national standards for history late the key components of standards-based reform
(National Center for History in the Schools, 1996) and into classroom and building-level practice.
with the history and social science standards for the The study centered around three major questions:
Commonwealth of Virginia (Commonwealth of
Virginia Department of Education, 2001). Topics for Do students exposed to a specific curriculum interven-
the units were selected in collaboration with teachers tion in social studies outperform similar students
and administrators from the partner school division, not using the intervention on measures of concep-
and standards related to the content topics, to critical tual and critical thinking and content learning?
thinking in social studies, and to historical analysis and Are there differences in performance between students
primary source interpretation formed a major part of identified as gifted and students not identified?
Do teachers implementing the project pedagogy show
change over time in observed and self-reported
behaviors supporting high-end learning?
Authors’ Note: Research for this article was supported under the
Javits Act Program as administered by the U.S. Department of
Education. Portions of the findings were shared at the 2004 con-
vention of the American Educational Research Association. The
Review of Relevant Literature
authors would like to thank all of the participants in Project
Phoenix, particularly the teachers and administrators in the Social Studies Standards and Habits of Mind
school districts implementing the curriculum and the graduate
students and Center for Gifted Education staff members who Representing the collection of subject areas and dis-
worked tirelessly to make the project possible. ciplines called “social studies” in the public schools,

Little et al. / Curriculum Effectiveness in Social Studies 273

Downloaded from gcq.sagepub.com at PENNSYLVANIA STATE UNIV on March 4, 2016


five separate sets of national standards documents were content areas, and the demands of heavily fact-based
developed in the mid-1990s (Buckles, Schug, & Watts, multiple choice assessments (Buckles et al., 2001;
2001). Although the development efforts for some of Haas & Laughlin, 2001). Consequently, delivery of
these standards—particularly those in history—have the volume of facts and details often contained within
been fraught with controversy (Saunders, 1996; Saxe, social studies textbooks—and some social studies
1995; Whelan, 1995), several key interdisciplinary standards—often takes precedence in instructional
emphases have emerged across standards documents practice and student learning activities over broad-
and thus represent central areas for curriculum devel- based social science concepts and habits of mind.
opment and instructional emphasis. These areas include Thus, concept-based curriculum and instruction in
the importance of critical inquiry and conceptual learn- social studies are crucial in making this volume of
ing, opportunities for problem solving, emphasis on information meaningful, to support students in the
depth over breadth, and connections to current and real- development of useful generalizations that reflect
world issues and problems (Avery & Zuo, 2003). the larger scale of what they study (Kneip, 1989;
Standards for effective curriculum and instruction in McKinney & Edgington, 1997; Reyes & Smith, 1983;
history in particular also reflect emphases on the mean- Shaver, 1991). In addition to a conceptual orientation
ing and process of history, direct engagement with pri- to curriculum, emphases on disciplined inquiry, criti-
mary source material, and the use of essay writing cal reasoning, and conscious reflection are also
(Gregory, 2000; National Center for History in the required to promote the key social science habits of
Schools, 1996). All of these standards draw on key mind (Beyer, 1971; Brown, 1996; Paul, 1992;
habits of mind of practitioners of the social science dis- Scheurman & Newman, 1998).
ciplines, including the following:
Social Studies and the ICM
to analyze documents of all sorts to detect bias, weigh
evidence, and evaluate arguments; The social studies processes and habits of mind noted
to distinguish between fact and conjecture and between previously represent critical elements of curriculum and
the trivial and the consequential; instruction for all learners, yet they also—when com-
to view human subjects nonjudgmentally and with bined with other recommendations for differentiation—
empathy instead of present mindedness; can form a foundation for effective curriculum for
to recognize and analyze the interplay of change and high-ability learners. The ICM (VanTassel-Baska, 1986,
continuity; 1995) provides the basis for a deep integration of the
to recognize the complexity of causality and avoid easy elements of conceptual understanding and critical rea-
generalizations and stereotypes when analyzing soning with advanced content in the social studies.
how change occurs;
Curriculum and instruction designed within the ICM
to recognize that not all problems have solutions;
to understand how people and cultures differ and what
framework engage students in actively exploring, ana-
they share; lyzing, and discussing advanced materials and topics
to analyze how the actions of others, past and present, through structured activities and questions, whereby
influence our own lives; and society (Sandling, they are encouraged to reflect on central ideas within a
2003, p. 221) specific context, on connections to broad concepts that
cut across time and across discipline, and on connec-
Although emphases on constructing meaning and tions to students’ own lives and circumstances. These
critical thinking are often highlighted as key aspects of emphases on active engagement, challenge, and devel-
social studies education, conceptions of what history is oping meaning reflect key characteristics of “powerful
and what it means are by no means uniform among teaching” identified by the National Council for the
teachers, nor is there a clear understanding of how best Social Studies (NCSS, 1994). By not only providing
to help students understand history and the other social support for the social science habits of mind but also
sciences (Barton, 1997; Evans, 1988; VanSledright, using advanced and complex content, the model also
1995). Moreover, despite recommended emphases on allows the curriculum to be appropriately differentiated
social science processes and depth in learning (National for high-end learners, reflecting research supporting
Center for History in the Schools, 1996), social studies inquiry learning and curricular differentiation as effec-
teachers are also influenced in their planning by tive practices for this population (Hertzog, Klein, &
highly limited time, limited priority compared to other Katz, 1999; Shore & Delcourt, 1996).

274 Gifted Child Quarterly, Vol. 51, No. 3

Downloaded from gcq.sagepub.com at PENNSYLVANIA STATE UNIV on March 4, 2016


Previous curriculum work using the ICM has Table 1
demonstrated the effectiveness of high-powered cur- Student Participation by Unit
riculum targeted to advanced learners on student learn- Unit Treatment Group N Comparison Group N
ing in language arts (VanTassel-Baska, Johnson,
Hughes, & Boyce, 1996; VanTassel-Baska, Zuo, Avery, Civilizations 81 34
Colonial 160 29
& Little, 2002) and in science (VanTassel-Baska, Bass,
1920s 241 10
Reis, Poland, & Avery, 1998). In these studies, students Civil War 88 48
who engaged in curriculum integrating advanced con- Government 21 30
tent with higher-level processes and interdisciplinary
themes demonstrated significant learning gains over
similar students not engaged in the curriculum. Gains
were consistent for males and females and across dif- critical elements of the change process all represented
ferent grouping arrangements. foundational aspects of the research and development
efforts related to Project Phoenix. Results related to
Professional Development student learning and teacher practice are reported in
the pages that follow.
Educational research is increasingly demonstrating
the role of the teacher as the most powerful influence
on learning (Sanders, 2001; Sanders & Rivers, 1996), Method
and effective professional development represents an
important component of strengthening that influences The study used a quasi-experimental design mode
in positive ways. Within the field of gifted education (Campbell & Stanley, 1969) to investigate the effects
specifically, researchers have found that teachers who of the curriculum treatment on gifted and nongifted
have engaged in learning experiences themselves students across elementary and middle school grades.
related to the characteristics and learning needs of This study reports on student results from the final
gifted students, as well as how to address those needs year of the 3-year project and on a cross-year analy-
in the classroom setting, are more effective with this sis of teacher practices for the subsample who partic-
population than those who have received no training ipated across 3 years.
(Hansen & Feldhusen, 1994). In social studies educa-
tion, a recent survey of 98 elementary teachers who are Sample
members of the NCSS indicated that the need for more
The student sample consisted of 1,200 students in
staff development in social studies was a key concern
Grades 2 to 8 across nine schools in one urban school
among both novice and veteran teachers, a concern
district. The district is considered to be an “Enterprise
rated second only to the perceived lack of priority
Zone,” with high populations of minority and disad-
given to social studies programs (Haas & Laughlin,
vantaged students. The sample was nearly evenly dis-
2001). In the same study, another frequently cited con-
tributed by gender, with 597 boys (50.7%) and 580
cern was a need for more curriculum development
girls (49.3%). Of the 1,200 students, 949 (79.1%)
work in social studies, with corresponding professional
were in the treatment group and 251 (20.9%) in the
development around the relevant social studies content
comparison group, with similar gender distributions
and teaching strategies.
across the two groups. Forty-one students (3.4%)
The role of teacher attitudes and beliefs about
were identified as gifted from the sample; thirty-five
student learning as related to specific teaching prac-
of these were in the treatment group. Within the treat-
tices is also significant as a catalyst for implementing
ment group, sample size by unit varied from 21
reform in schools. Research efforts have demon-
students to 241 students, as noted in Table 1.
strated that changes in teacher attitudes and beliefs
Six of the 16 teachers involved in the third year of
occur not after professional development alone but
project implementation had been involved across all 3
rather after the demonstration of changes in student
years; this subgroup of six teachers formed the sam-
learning following changes in classroom practice
ple for the study reported here.
(Guskey, 2000).
These elements of reflecting strong content and
habits of mind in the subject area studied, responding
Curriculum Treatment
to the learning needs of high-ability learners, and sup- Five units, developed under Javits funding, were
porting teachers in their own growth and reflection as used as the curriculum treatment in the study. The

Little et al. / Curriculum Effectiveness in Social Studies 275

Downloaded from gcq.sagepub.com at PENNSYLVANIA STATE UNIV on March 4, 2016


curriculum framework for the units was developed using models, and creating linkages across unit goals. Thus,
the ICM, incorporating goals addressing advanced con- for example, a lesson might review the context under
tent, higher level processes, and abstract concepts. The study, introduce a relevant primary source document
five goals for the units were as follows: for analysis, and follow the analysis with a set of dis-
cussion questions to explore how the document
to develop understanding of the concept of systems and reflected the historical context and also key general-
of structure, function, and pattern as key systems izations about the central concept of systems.
elements; The five units implemented in the study, with their
to develop reasoning skills with application to social intended grade levels, were as follows:
studies;
to develop interpersonal and social group process skills; Systems of Civilization (Grades 2 to 3)
to develop skills in historical analysis and primary Building a New System: Colonial America 1607-1790
source interpretation; and (Grades 4 to 5)
to develop knowledge and understanding around unit- A System in Conflict: The American Civil War (Grades
specific content goals (each unit identified a con- 5 to 6)
tent goal and outcomes related to understanding The 1920s and 1930s in America: A System of Tensions
and using key concepts and information for the cul- (Grades 6 to 7)
ture and/or time period under study). The Road to the White House: The American System of
Representational Democracy (Grades 6 to 8)
As discussed previously, the unit developers drew
and expanded on existing state and national standards
in developing the framework and the units them- Instrumentation
selves. Grade-level content emphases were selected, The instruments used in the curriculum effective-
based on the Virginia social studies standards; devel- ness study included three assessments for students: a
opers then gave attention to advancing these content measure of students’ conceptual thinking (conceptual
topics for high-ability learners through the depth and thinking test; CNTA), a measure of critical thinking
complexity with which the topics would be addressed (critical thinking test; CRTA), and unit-specific con-
and the challenge level of the resources used. In addi- tent assessments. A final instrument was a classroom
tion, each unit incorporated a heavy emphasis on the observation form (COF), relating to the research
reasoning process (Paul, 1992) and on applying ele- question about teaching behaviors.
ments of that process to the study of primary source
documents. Primary sources were embedded as major CNTA and CRTA. The assessments of conceptual
readings and information resources across all units, thinking and critical thinking, designed by the project’s
again reflecting discipline-based standards and the external evaluator, each consisted of three multipart
need for challenge to support differentiation for high- questions. Items for the CNTA included emphasis on
ability learners. Specific teaching models related to classification of concepts, identification of concept-
reasoning and primary source analysis were incorpo- relevant characteristics, and specific application of the
rated across all units. In addition, the units used the concept of systems. For example, one item presents a list
concept of systems as a central organizer, applying a of food items and asks students to group the items and
definitional model and concept generalizations to assign categorical names to the groups; another asks
systems ranging from the classroom to systems of students to read informational text about systems and
government to systems of civilization. This concep- then determine and explain whether given examples are
tual organizer was intended to support differentiation or are not examples of systems. These emphases reflect
based on depth and also to promote interdisciplinary literature on conceptual thinking (Ehrenberg, 1981;
connections within the social studies and between Taba, 1962) and the emphasis in the curriculum inter-
social studies and other subject areas. vention on the concept of systems. In initial piloting,
Each unit was designed to represent roughly 9 to test-retest reliability during a 4-month period using
12 weeks of social studies instruction, determined Pearson Product Moment correlation reached .62. Items
based on a review of planning blueprints used by the for the CRTA emphasized use of evidence to draw con-
school district involved. Units contained 18 to 25 clusions and suggest implications, reflecting elements of
lessons, including specific lessons introducing and the Paul (1992) model of reasoning and other work on
practicing key teaching and learning models, explor- applying logical rules of reasoning in everyday life (e.g.,
ing and discussing key content topics through these Halpern, 2003). For example, in one item, students are

276 Gifted Child Quarterly, Vol. 51, No. 3

Downloaded from gcq.sagepub.com at PENNSYLVANIA STATE UNIV on March 4, 2016


given a rule and then asked to apply the rule in making “yes” or “no” response for the presence of each of the
a decision; another item asks students to provide reasons behaviors. A self-report version of the form was also
for and against a given claim. Test-retest reliability dur- used in the study; on this form, teachers responded
ing a 4-month period was moderate in the pilot stage “yes” or “no” to each of the behaviors with regard to the
(r = .50). Project staff reviewed both tests before and same lesson observed by project staff.
after the piloting, making revisions each time based on
review against the literature on critical and conceptual Procedures
thinking and on pilot data. Three forms of each test were
developed, one at each of the levels of primary, interme- All teachers involved in the project as treatment-
diate, and middle school. At each level, each test group teachers participated in 1 to 4 days of profes-
included three multipart items, as noted; complexity was sional development before and during implementation
added to the tasks with advancement of grade level of the intervention. Teachers involved in the project
through the consideration of additional variables within throughout multiple years received additional pro-
the problems. Item analyses on both tests demonstrated fessional development each year of involvement.
sufficient ceiling on most items for growth to be Professional development workshops consisted of the
measured. following key features: overview of the curriculum
model and relationship to reform principles and content
Content assessments. The content assessments standards, demonstration and practice with key teach-
were designed by each of the unit developers to ing models used in units, “walk-through” of unit with
accompany the specific units. They include both mul- attention to key features and resources needed, and dis-
tiple choice and short-answer questions specifically cussion of various implementation expectations and
tied to the content objectives of each unit, which were questions.
derived from state-level standards of learning in the The CNTA and CRTA were administered by project
state in which the project was conducted. The tests staff in both treatment and comparison classes prior to
ranged from 15 to 25 items, depending on grade level, the beginning of unit implementation in the treatment
but all had a maximum of 75 points as a score. Items classes. Teachers in both groups administered the
were scored as right or wrong or else given points content assessments. Following test administration,
against a rubric, depending on type of item. treatment group teachers conducted 20 to 25 hours of
instruction using the relevant unit for their grade level.
COF. The COF was developed and validated in the During this period, teacher observations were con-
first year of the project, with interrater reliability ini- ducted in treatment classes, with two observations per
tially established at a median kappa of .63 (VanTassel- teacher, to document specific teaching behaviors.
Baska & Avery, 2001) and since reevaluated at .82 Observations were conducted by trained project staff,
based on use in subsequent studies (Feng, 2001). Three and teachers were asked to complete a self-report form
experts in gifted education and one instructional coor- immediately following the lesson observed. The obser-
dinator in general education reviewed the 40 items of vations generally lasted about 30 to 45 minutes, and
the form for content validity, obtaining an average rat- external observers followed up the observations with
ing of 2.9, using a scale from 1 to 3, with 3 being the informal interviews to clarify any questions.
highest. Content validity was established based on a .96 At the conclusion of unit implementation, teachers
agreement of expert review (Avery, 1999). The purpose in both treatment and comparison classes again
of the form is to note the demonstration of specific administered the content assessments, and project
teaching behaviors reflecting general instructional prac- staff returned to administer the CNTA and CRTA. In
tice as well as recommendations for teaching high- addition, teachers submitted log notes and met with a
ability learners. The form consists of 40 behaviors, project staff member to discuss details of implemen-
grouped into the following nine categories: curriculum tation and report on lessons implemented.
planning, expectations for learning, accommodating Tests were scored by project staff members who
individual differences, curriculum delivery features, had received training in the scoring protocol and were
general teaching strategies, critical thinking, problem supervised during initial efforts to ensure reliability
solving, metacognition, and extensions. in the scoring process. Observation scores were com-
To use the COF, trained observers scripted what they puted by adding the items noted as present in each of
saw occurring in the classroom and then indicated a the nine categories.

Little et al. / Curriculum Effectiveness in Social Studies 277

Downloaded from gcq.sagepub.com at PENNSYLVANIA STATE UNIV on March 4, 2016


Limitations of the Study development sessions. These issues around implemen-
tation suggest a need for caution in interpreting results,
Several limitations evolved during the implementa-
while also reflecting key concerns in any study imple-
tion of the study that must be considered in terms of
mented in the real-world context of schools.
analyzing results. First, although the overall sample size
included a large number of students, the number of
Data Analysis
identified gifted students was quite small, representing
less than 4% of the total sample. Naturally, in any study Analyses of student and teacher data from the pro-
conducted in heterogeneous classroom settings, the ject were done initially by the external evaluator and
gifted students in the sample are likely to represent a extended by the Center for Gifted Education’s research
minority; however, the small size in this study and the coordinator. The major analytical method used in this
spread of the students across schools, grade levels, and study was the analysis of covariance (ANCOVA).
units prevent any conclusions regarding differential Several considerations guided the choice of separate
effects for identified and nonidentified students. ANCOVAs in lieu of multivariate statistics: (a) The
Furthermore, there was a large difference in size between researchers intended to examine the effect of the cur-
the treatment group and the comparison group, also sug- riculum intervention on each of the dependent mea-
gesting a need for caution in interpreting results. In sures respectively; (b) although there were significant
addition, validity and reliability evidence on the student correlations (p < .05) between the three dependent
assessments was limited in strength, reflecting some of measures (i.e., conceptual reasoning, critical thinking,
the challenges of assessing higher level thinking skills content knowledge), the correlation coefficients (.39 to
in younger children; the instruments themselves may .48) were much lower than the threshold of a correla-
have been a contributing factor to the small size or lack tion of .90, at which statistical problems will typically
of apparent group differences. occur because of multicollinearity (Tabachnick &
Other notable limitations in the study related to Fiddell, 1989); and (c) Bonferroni corrections were
teacher attrition and the implementation process. In the used to reduce the Type I error rate because of multi-
first year of the study, 26 teachers began implementa- ple analysis of covariances.
tion of the units; however, by the third year, only six of Analyses of covariance (ANCOVA) were conducted
this group remained, and the total number of teachers on the results for the three student tests to investigate
involved in year three was 16. Teacher attrition differences in posttest performance between the treat-
occurred as a result of a variety of influences. The ment and comparison groups for the whole sample
study occurred during a time of major transition within (Research Question 1), controlling for pretest differ-
the district and state, with changes to the standards and ences, and for treatment effect within the group identi-
an increase in state assessments and local accountabil- fied as gifted and the group not identified, respectively
ity. Although the study was conducted in grade levels (Research Question 2). Bonferroni adjustment was con-
in which the state testing did not occur, overall changes ducted to reduce the Type I error rate. The overall alpha
to the culture of the district in response to the assess- level of .05 was separately specified for the tests to
ment program caused some teachers and principals to answer Research Question 1 (.02) and Research
withdraw from the study to address the standards in a Question 2 (.03). Therefore, the alpha level specified
more step-by-step, content delivery format instead of for the three ANCOVA tests for Question 1 was set at
in the integrated approach of the treatment curriculum. .02/3 = .007; the alpha level specified for the six
Other teachers withdrew from the study because they ANCOVA tests to answer Question 2 was set at .03/6 =
changed grade levels or schools. This attrition among .005.
teachers influenced the study in several ways, limiting Additional ANCOVAs were conducted to examine
the degree to which multiyear results could be ana- treatment effect based on school and unit to explore
lyzed and also the investigation of Research Question potential differences related to the groups of students,
3, related to teacher change over time. Furthermore, the teaching context, and the specific intervention used.
observations and ongoing interactions with the project Because these tests are of an exploratory nature in this
teachers demonstrated inconsistencies in implementa- study, no Bonferroni correction of the alpha level was
tion, with some cases of incomplete implementation used to reduce the likelihood of evoking Type II error.
and other cases of lack of fidelity to the curriculum Because of the low number of participating teachers
guidelines as written and shared in the professional in all three years of the project (N = 6), only descriptive

278 Gifted Child Quarterly, Vol. 51, No. 3

Downloaded from gcq.sagepub.com at PENNSYLVANIA STATE UNIV on March 4, 2016


Table 2
Pre- and Posttest Means and Standard Deviations by Condition
Treatment Comparison

Pretest Posttest Pretest Posttest

Instrument N M SD M SD N M SD M SD

CNTA 589 11.1 4.7 13.2 4.5 139 11.5 4.6 12.6 4.5
CRTA 563 8.1 2.1 8.9 2.3 141 7.4 1.8 8.7 2.2
Content knowledge 276 18.2 13.7 28.3 15.1 63 10.5 7.7 12.9 8.6
Note: CNTA = conceptual thinking test; CRTA = critical thinking test.

Table 3
One-Way ANCOVA Conducted for Whole Sample
Dependent Variable N df F Significance Effect Size (η2)

Post Concept reasoning 728 1 5.39 .020 .007


Post Critical thinking 704 1 .59 .444 .001
Post Content knowledge 339 1 42.29* .000 .11
*p < .001.

Table 4
One-Way ANCOVA Conducted for Identified GT and Non-GT Students
Group Dependent Variable N df F Significance Effect Size (η2)

GT Post Concept reasoning 31 1 .43 .518 .020


Post Critical thinking 26 1 .05 .829 .002
Post Content knowledge 12 1 .78 .400 .080
Non-GT Post Concept reasoning 697 1 4.63 .032 .007
Post Critical thinking 678 1 .47 .492 .001
Post Content knowledge 327 1 41.57* .000 .114
Note: GT = gifted.
*p < .005.

statistics were used to examine patterns of change in the favoring the treatment group. Using eta squared as the
six teachers’ instructional practices against the nine cat- index for effect size, the treatment effect was moder-
egories of teaching behaviors delineated in the COF. ate in content knowledge (.11).1 No statistically sig-
nificant differences were found in concept reasoning
and critical thinking between the two groups.
Results Additional ANCOVAs were performed to examine
the treatment effect for identified gifted students and
Pre- and post-assessment means for the treatment nonidentified students, respectively. Table 4 summa-
and comparison groups on all three measures are rizes the results. For the gifted students, no signifi-
given in Table 2. ANCOVA tests were run to examine cant differences were detected between the treatment
differences between the treatment and comparison and comparison group in any of the three areas. It
students’ post-assessment scores in concept reason- should be noted that this result was affected by the
ing, critical thinking, and content knowledge, con- limited data available from gifted students (N < 31)
trolling their initial differences using pre-assessment for the analyses. For nongifted students, statistically
scores. As Table 3 summarizes, statistically signifi- significant differences were found in students’
cant differences in students’ post-assessment scores posttest scores in content knowledge, favoring the
were registered in content knowledge (p < .007), treatment group (p < .005). The effect sizes, using eta

Little et al. / Curriculum Effectiveness in Social Studies 279

Downloaded from gcq.sagepub.com at PENNSYLVANIA STATE UNIV on March 4, 2016


Table 5
One-Way ANCOVAs Conducted by School
School Dependent Variable N df F Significance Effect Size (η2)

School 1 (elementary) Post Concept reasoning 50 1 .06 .81 .001


Post Critical thinking 50 1 .02 .89 .000
Post Content knowledge 45 1 8.11** .00 .162
School 2 (elementary) Post Concept reasoning 108 1 .33 .57 .003
Post Critical thinking 109 1 5.27* .02 .047
School 3 (middle) Post Concept reasoning 138 1 .32 .57 .002
Post Critical thinking 139 1 2.83 .10 .020
Post Content knowledge 46 1 70.16* .00 .620
*p < .05. **p < .01.

Table 6
One-Way ANCOVAs Conducted by Unit
Unit Dependent Variable N df F Significance Effect Size (η2)

Civilization Post Concept reasoning 115 1 0.00 .98 .000


Post Critical thinking 115 1 2.26 .14 .020
Post Content knowledge 84 1 9.77** .002 .108
Colonial Post Concept reasoning 186 1 3.35 .07 .018
Post Critical thinking 189 1 2.00 .16 .011
Post Content knowledge 133 1 3.43 .07 .026
Civil War Post Concept reasoning 135 1 0.96 .33 .007
Post Critical thinking 136 1 2.52 .12 .019
Post Content knowledge 91 1 62.89** .00 .420
Government Post Concept reasoning 51 1 0.27 .61 .006
Post Critical thinking 51 1 6.37* .02 .117
*p < .05. **p < .01.

squared, was moderate (.11). No significant differ- treatment and comparison students’ post-assessment
ences were found in concept reasoning and critical in concept reasoning.
thinking between the treatment and comparison Further ANCOVAs were performed to examine the
group, with their initial differences controlled. treatment effect for four of the units administered.
ANCOVAs were also conducted to explore treatment Treatment effect could not be measured for the fifth
effect by school. Four schools hosted both treatment and unit (The 1920s), as data were incomplete for one or
comparison students; however, no post-assessment data the other group on all measures. Table 6 shows that sta-
were available for one participating school. Thus, the tistically significant differences were found in
school-level ANCOVAs were conducted for the three students’ posttest performance in content knowledge
schools for which both pre- and post-assessment data within two unit groups, Civilization and Civil War,
were available. Table 5 presents the results. A statis- favoring the treatment group. The implementation of
tically significant treatment effect was found in both units also resulted in moderate to large effect sizes
students’ post-assessment in content knowledge in (eta squared results were .11 and .42, respectively),
two schools, favoring the treatment group; the effect indicating the differences between treatment and com-
sizes were large (eta squared = .16 and .62, respec- parison groups had practical importance. For students
tively), indicating educational importance as well as who were exposed to the Government unit, statistically
statistical significance. At one school (School 2), a significant differences were also found in their
statistically significant difference was found in post-assessment in critical thinking, favoring the treat-
students’ critical thinking post-assessment, favoring ment group. The effect size using eta squared reached
the comparison group, with a moderate effect size. No .12, indicating that the magnitude of treatment
statistically significant differences were found between students’ gains in critical thinking was fairly large and

280 Gifted Child Quarterly, Vol. 51, No. 3

Downloaded from gcq.sagepub.com at PENNSYLVANIA STATE UNIV on March 4, 2016


Table 7
Continuing Teachers’ Instructional Behaviors: Years 1 to 3
Year 1 Year 2 Year 3

Pretest Posttest Pretest Posttest Pretest Posttest

COF Categories Max. Score M SD M SD M SD M SD M SD M SD

Planning 3 2.50 0.77 2.25 0.69 2.50 0.84 — — 2.33 0.52 2.17 0.98
Expectations 3 2.30 0.88 2.75 0.61 3.00 0.00 — — 3.00 0.00 2.17 0.98
Accommodating 5 2.33 0.98 2.33 0.61 3.17 0.75 — — 3.67 1.03 3.17 1.17
individual
differences
Curriculum 3 1.33 0.98 1.75 0.88 2.50 0.84 — — 2.17 0.75 1.83 0.98
delivery
features
General strategies 9 3.83 2.70 4.42 0.86 5.50 1.22 — — 6.33 1.51 4.67 2.88
Critical thinking 5 2.25 1.13 2.50 0.63 3.00 1.10 — — 4.00 2.00 2.33 1.03
Problem solving 6 1.58 1.16 2.25 0.52 2.50 1.64 — — 1.00 1.10 0.83 0.98
Metacognition 3 0.58 1.02 1.25 0.88 0.16 0.40 — — 1.50 1.05 0.67 0.52
Extensions 3 0.17 0.26 0.33 0.61 0.17 0.41 — — 0.33 0.52 0.17 0.41
Total 40 16.80 8.05 19.83 2.48 22.50 2.81 24.30 5.61 20.17 7.36
Note: COF = classroom observation form.

educationally meaningful. No significant differences four observations were conducted for each of these
were found between treatment and comparison teachers, with two (pre-observations) conducted
students who used the Colonial unit. No treatment before the implementation of the curriculum but
effect was detected in conceptual reasoning for after the training workshop on how to use the curricu-
students in these four units. lum; two observations (post-observations) followed
during the implementation of the unit. For analysis
purposes, the two pre-observations scores for the par-
Teacher Results ticipants were averaged as a Year 1 pre-observation
score; the two post-observation scores were averaged
Six teachers, out of an original sample of 26, as a Year 1 post-observation score. In Year 2 of the
remained in Project Phoenix for the 3 years of imple- project, only the observation data collected before
mentation. These six teachers were all female and var- the implementation of the unit in that year were
ied in age between younger than 25 and older than available for analysis. In Year 3 of the project, two
50. Four of these teachers taught second graders observations were conducted separately near the
(Civilizations unit) for the 3 years, and two teachers beginning and near the end of implementation.
taught fourth and fifth graders (Colonial unit). All of Because of the small number of cases available for
the teachers held a BA or BS, and one held a master’s analysis (N = 6) for the teacher behavior data, descrip-
degree. All of them reported having taken at least six tive statistics were used to document these six
credits of graduate coursework, except the youngest teachers’ instructional practice changes from Year 1 to
teacher, who had taken only one graduate course; none Year 3 implementation. Table 7 illustrates means and
of them had taken any courses in gifted education. standard deviations of instructional behaviors against
Their teaching experience ranged from 3 years to more the nine behavioral categories. The results showed that
than 20 years. All teachers had received at least 5 days there was a consistent pattern of improvement in eight
of training related to the project curriculum, with the out of the nine categories from initial observations
majority of the training conducted in Year 1 and shorter to later observations in Year 1 implementation; this
follow-up sessions conducted in the subsequent years increasing behavioral pattern continued from Year 1 to
of the project. Year 2 implementation, as recorded by external obser-
Three years of observation data for the six teachers vation results during that year. From Year 2 observa-
were available for analysis. In Year 1 of the project, tion (pre-observation only) to Year 3 pre-observation,

Little et al. / Curriculum Effectiveness in Social Studies 281

Downloaded from gcq.sagepub.com at PENNSYLVANIA STATE UNIV on March 4, 2016


the instructional improvement continued in the area of Although several results were positive in terms of
accommodation for individual differences, general demonstrating learning gains for students, the study
teaching strategies, critical thinking strategies, highlights several areas of concern and caution in
metacognition, and extensions; however, observation implementing a curriculum designed for use with
rating in the second observation of Year 3 implementa- high-ability learners yet implemented with all learn-
tion, on average, decreased from the first observation ers. First, there is a clear need to ensure fidelity of
in the same year, particularly in the area of general implementation in classrooms. Teachers are not
teaching strategies, critical thinking, metacoginition, socialized to implement packaged curriculum, which
and extensions. The data provided evidence that these results in fragmented learning. This “habit of prac-
six teachers have had ongoing positive change in tice” needs to change to ensure student growth in crit-
instructional behavior during a 3-year period, with the ical areas. Only in the final year of the project,
last semester of the implementation as an exception. It reported here, were we able to see teachers approach-
is unclear from the limited data what might have ing the prescribed minimum of 20 hours of unit imple-
caused a decreased level of instructional practices in mentation deemed necessary for treatment fidelity. In
the second observation of Year 3 implementation. The earlier years of the project, only 6 to 14 hours of
limited number of cases with 3 years’ data prevented implementation occurred, resulting in no important
the authors from making any generalizations. student growth gains. Furthermore, given the findings
related to limited growth gains for gifted students
involved in the project, there is a need to explore sev-
Discussion eral key questions, including whether modifications
made to “fit” the curriculum to a given classroom
The implementation of the curriculum intervention distort the units’ embedded differentiation for high-
resulted in significant treatment effect for students in ability learners, whether whole class delivery inter-
the area of content knowledge, supporting the potential feres in some way with advanced learners’ growth,
of the curriculum to support student growth in this and whether teacher effectiveness has a disproportion-
area. Subanalyses demonstrated statistically significant ate influence on gifted student learning.
treatment effects for given units and at given schools, The professional development model employed in
suggesting potential differential effects of the units the project, in conjunction with the curriculum itself,
and/or differential degrees of implementation. These appeared to affect teaching practice positively
results bear further investigation in future studies throughout the 3 years in heterogeneous classrooms
related to the curriculum, especially given the disparity by enhancing greater use of research-based practices
in sample size between the treatment and comparison and also increasing teachers’ accommodation to indi-
groups in this study. Results related to student perfor- vidual differences. However, the limited size of the
mance on the conceptual reasoning and critical think- teacher sample, resulting from attrition and various
ing assessment are less positive for the project; they contextual issues over the duration of the project, also
suggest the need for further investigation of how the suggests caution in drawing conclusions from the
curriculum may or may not be supporting these areas results of this aspect of the study.
as measured by the instruments. Moreover, given the
somewhat low reliabilities on the CNTA and CRTA, Implications
further investigation and development of the instru-
ments themselves may offer a greater understanding of The study suggests the need for more research on
any effects in these areas. student learning in social studies based on grouping
Given the limited number of gifted students in the assignment. It is not clear whether gifted students
overall sample, generalizations related to effectiveness would have grown more in a classroom of like-ability
with that population are problematic. However, the evi- peers in which clustering took place rather than in the
dent growth gains for students not identified as gifted heterogeneous classroom model used in this study.
are a promising indication of the effectiveness of using Larger samples of both teachers and gifted students
curriculum designed for high-ability learners with a would also strengthen the interpretation of results in
broader population of students. Further investigation to the areas of student learning and teacher change.
determine how treatment effect may relate to fidelity of It is notable that the results of the study revealed
implementation, differences among the units, or devel- significant treatment effects in the area of content
opmental differences in students is warranted. knowledge, which was the area most common

282 Gifted Child Quarterly, Vol. 51, No. 3

Downloaded from gcq.sagepub.com at PENNSYLVANIA STATE UNIV on March 4, 2016


between treatment and comparison groups. All References
students involved in the project were subject to the
same content standards of learning. The curriculum Avery, L. D. (1999). Teacher efficacy and behavior: Their rela-
intervention approached the standards in a different tionship and impact on student learning. Unpublished doc-
toral dissertation, The College of William and Mary,
way, linked with specific instruction in the areas of Williamsburg, VA.
concept development and reasoning, and the treat- Avery, L. D., & Zuo, L. (2003). Selecting resources and materi-
ment group demonstrated greater gains in their con- als for high-ability learners. In J. VanTassel-Baska & C. A.
tent knowledge. Although the differential size of the Little (Eds.), Content-based curriculum for high-ability learn-
treatment and comparison groups and some of the ers (pp. 259-277). Waco, TX: Prufrock.
Barton, K. C. (1997). “Bossed around by the queen”: Elementary
other limitations previously noted indicate the need
students’ understanding of individuals and institutions in
for cautious interpretation of results, this finding is history. Journal of Curriculum and Supervision, 12, 290-314.
still important in an era of standards-based teaching Beyer, B. K. (1971). Inquiry in the social studies classroom: A
and testing. The results bear the implication that strategy for teaching. Columbus, OH: Merrill.
teaching from an integrated curricular framework that Brown, R. H. (1996). Learning how to learn: The Amherst project
embeds attention to the standards within a structure and history education in the schools. The Social Studies, 87,
267-273.
focused on development of higher level thinking and Buckles, S., Schug, M. C., & Watts, M. (2001). A national survey
concept development may yield content gains as of state assessment practices in the social studies. The Social
strong as or stronger than a more direct, knowledge- Studies, 92, 141-146.
based structure for teaching the standards. Campbell, D. T., & Stanley, J. C. (1969). Experimental and quasi-
A third implication of the study rests with the need experimental designs for research. Chicago: Rand McNally.
Cohen, J. (1988). Statistical power analysis for the behavioral
for a system of monitoring teacher classroom behavior
sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
rather than relying on self-report of strategy use. Commonwealth of Virginia Department of Education. (2001).
Earlier findings demonstrating that teachers rated their History and social science standards of learning for Virginia
behaviors significantly higher than trained observers public schools. Richmond, VA: Author.
suggest a disparity in perception that skews the picture Cuban, L. (1998). How schools change reforms: Redefining reform
of what implemented practice is. The study also sug- success and failure. Teachers College Record, 99, 453-477.
Ehrenberg, S. D. (1981). Concept learning: How to make it hap-
gests an uneven pattern of development across years in pen in the classroom. Educational Leadership, 39(1), 36-43.
respect to teacher growth and change in behaviors. If Evans, R. W. (1988). Lessons from history: Teacher and student
we are to understand how reform is really working, we conceptions of the meaning of history. Theory and Research
need an ongoing record of teacher change and a clear in Social Education, 16, 203-225.
response to that record of change. Moreover, to support Feng, A. X. (2001). Technical report on interrater reliability of
the Classroom Observation Scale. Williamsburg, VA: Center
ongoing school reforms, valid and reliable educational
for Gifted Education.
research, and consistent services for students, educa- Gregory, T. (2000). Social studies curricula for gifted learners in
tional leaders and researchers need to address the issue elementary and middle school. Unpublished master’s thesis,
of teacher attrition from research projects as a major The College of William and Mary, Williamsburg, VA.
consideration in embarking on any study. Guskey, T. R. (2000). Evaluating professional development.
Implementing an innovation in schools is a difficult Thousand Oaks, CA: Sage.
Haas, M. E., & Laughlin, M. A. (2001). A profile of elementary
task for many reasons, including teacher and principal social studies teachers and classrooms. Social Education, 65,
attrition, teacher acceptance and support, administrator 122-126.
support, the climate for change, other competing pro- Halpern, D. F. (2003). Thought and knowledge: An introduction to
jects, and the threat of state testing. Without adequate critical thinking (4th ed.). Mahwah, NJ: Lawrence Erlbaum.
levers to accomplish specific changes, meaningful Hansen, J. B., & Feldhusen, J. F. (1994). Comparison of trained
and untrained teachers of gifted students. Gifted Child
institutional change is not possible in many schools
Quarterly, 38, 115-123.
and classrooms despite the valiant efforts of all Hertzog, N. B., Klein, M. M., & Katz, L. G. (1999). Hypothesizing
involved. Project Phoenix has yielded important and theorizing: Challenge in an early childhood curriculum.
insights on the realities of that struggle for effective Gifted and Talented International, 14, 38-49.
implementation of curricular innovations in schools. Kneip, W. (1989). Social studies within a global education. Social
Education, 53, 399-403.
McKinney, C. W., & Edgington, W. D. (1997). Issues related to
Note teaching generalizations in elementary social studies. The
Social Studies, 88, 78-81.
1. Cohen (1988) provided rules of thumb for effect sizes using National Center for History in the Schools. (1996). National stan-
eta squared: .01 = small, .06 = medium, and .14 = large. dards for history. Los Angeles: Author.

Little et al. / Curriculum Effectiveness in Social Studies 283

Downloaded from gcq.sagepub.com at PENNSYLVANIA STATE UNIV on March 4, 2016


National Council for the Social Studies. (1994). Expectations of effectiveness for high-ability students. Gifted Child Quarterly,
excellence: Curriculum standards for social studies. Washington, 42, 200-221.
DC: Author. VanTassel-Baska, J., Johnson, D. T., Hughes, C., & Boyce, L. N.
Paul, R. (1992). Critical thinking: What every person needs to (1996). A study of language arts curriculum effectiveness with
survive in a rapidly changing world. Rohnert Park, CA: gifted learners. Journal for the Education of the Gifted, 19,
Foundation for Critical Thinking. 461-480.
Reyes, D. J., & Smith, R. B. (1983). The role of concept learning VanTassel-Baska, J., Zuo, L., Avery, L. D., & Little, C. A. (2002).
in social studies textbook comprehension: A brief analysis. A curriculum study of gifted student learning in the language
The Social Studies, 7, 85-88. arts. Gifted Child Quarterly, 46, 30-44.
Sanders, W. L. (2001, June). The impact of teaching on student Whelan, M. (1995). Right for the wrong reasons: National history
achievement. Paper presented at the William and Mary School standards. Social Education, 60, 55-57.
Leadership Institute, Williamsburg, VA.
Sanders, W. L., & Rivers, J. (1996). Cumulative and residual effects Catherine A. Little is an assistant professor in educational psy-
of teachers on future student academic achievement. Knoxville: chology specializing in gifted and talented education at the
University of Tennessee Value-Added Research and Assessment University of Connecticut. She previously served as a visiting
Center. assistant professor in gifted education at The College of William
Sandling, M. M. (2003). Adapting social studies curricula for and Mary and as a curriculum coordinator at the Center for Gifted
high-ability learners. In J. VanTassel-Baska & C. A. Little Education there. She received her PhD in educational policy, plan-
(Eds.), Content-based curriculum for high-ability learners ning, and leadership from William and Mary. Her research interests
(pp. 219-254). Waco, TX: Prufrock. include professional development, teacher talent development, cur-
Saunders, R. M. (1996). National standards for United States riculum differentiation, and perfectionism.
history. Social Studies, 87, 63-67.
Annie Xuemei Feng is the research and evaluation director of the
Saxe, D. W. (1995). The national history standards: Time for
Center for Gifted Education at The College of William and Mary.
common sense. Social Education, 60, 44-48.
Her research interests include curriculum effectiveness studies,
Scheurman, G., & Newman, F. M. (1998). Authentic intellectual
program evaluation research, gender-related studies, and cross-
work in social studies: Putting performance before pedagogy.
cultural research.
Social Education, 62, 23-25.
Shaver, J. P. (Ed.). (1991). Handbook of research on social stud- Joyce VanTassel-Baska is The Jody and Layton Smith Professor
ies teaching and learning. New York: Macmillan. of Education and executive director of the Center for Gifted
Shore, B. M., & Delcourt, M. A. B. (1996). Effective curricular Education at The College of William and Mary, where she has
and program practices in gifted education and the interface developed a graduate program and a research and development
with general education. Journal for the Education of the center in gifted education. Her major research interests include the
Gifted, 20, 138-154. talent development process and effective curricular interventions
Taba, H. (1962). Curriculum development, theory, and practice. with the gifted, and she has received several Javits grants for
New York: Harcourt, Brace, and World. research and development work on curriculum. She is past presi-
Tabachnick, B. G., & Fiddell, L. S. (1989). Using multivariate dent of the Association for the Gifted of the Council for
statistics. New York: Harper & Row. Exceptional Children, and she is currently president of the National
U.S. Department of Education. (1993). National excellence: A case Association for Gifted Children.
for developing America’s talent. Washington, DC: Author.
Karen B. Rogers is currently the director of research at the
VanSledright, B. A. (1995). “I don’t remember—the ideas are all
Gifted Education Research, Resource, and Information Centre at
jumbled in my head”: Eighth graders’ reconstructions of colo-
the University of New South Wales. She is a leading authority on
nial American history. Journal of Curriculum and Supervision,
the development of differentiated curriculum for gifted and tal-
10, 317-345.
ented students. She has worked extensively with schools on the
VanTassel-Baska, J. (1986). Effective curriculum and instruc-
development of programs of identification, curriculum differenti-
tional models for talented students. Gifted Child Quarterly,
ation, and evaluation in gifted education. She has served as the
30, 164-169.
evaluator for a number of Javits grants, including several at the
VanTassel-Baska, J. (1995). The development of talent through
Center for Gifted Education at William and Mary.
curriculum. Roeper Review, 18, 98-102.
VanTassel-Baska, J., & Avery, L. D. (2001). Investigating the Linda D. Avery received her PhD from The College of William
impact of gifted education: Evaluation at state and local and Mary. For several years, she served as manager of the Center
levels: Problems with traction. Journal for the Education of for Gifted Education at William and Mary and as a project man-
the Gifted, 25, 153-176. ager for Project Phoenix. Her interests include program evalua-
VanTassel-Baska, J., Bass, G. M., Reis, R. R., Poland, D. L., & tion and curriculum reform. She currently works as a private
Avery, L. D. (1998). A national pilot study of science curriculum consultant and writer.

284 Gifted Child Quarterly, Vol. 51, No. 3

Downloaded from gcq.sagepub.com at PENNSYLVANIA STATE UNIV on March 4, 2016

You might also like