830772

research-article2019
BULXXX10.1177/0192636519830772NASSP BulletinNehring et al.

Article
NASSP Bulletin
2019, Vol. 103(1) 5­–31
Redefining Excellence: © 2019 SAGE Publications
Article reuse guidelines:
Teaching in Transition, sagepub.com/journals-permissions
DOI: 10.1177/0192636519830772
https://doi.org/10.1177/0192636519830772
From Test Performance journals.sagepub.com/home/bul

to 21st Century Skills

James H. Nehring1, Megin Charner-Laird2,

and Stacy A. Szczesiul1

Abstract
This study examined classrooms in three, high performing, public secondary schools
serving high need communities. Of 22 classes observed, we found approximately
one third exhibited an instructional demand for 21st century skills. In many of the
remaining classes, teachers appeared to apply misconceptions of 21st century skills,
and unintentionally deployed those misconceptions to reinforce the narrow and
shallow skill set associated with test-based accountability. Themes from teacher and
administrator interviews support this finding. School and system level implications are
discussed.

Keywords
urban education, 21st century skills, deeper learning, teacher practice, accountability

Policy focus on both test-driven accountability and 21st century skills (sometimes
called deeper learning) is accelerating (Every Student Succeeds Act, 2016; No Child
Left Behind, 2002; Voogt & Roblin, 2012). Twenty-first century skills is a construct
well accepted within the international education policy world that refers to higher
level cognitive, interpersonal, and intrapersonal skills, seen as increasingly relevant to
public education in a global economy (Voogt & Roblin, 2012). This trend is evident
among member nations of the Organization for Economic Cooperation and
Development (OECD, 2010). For nations with large disadvantaged student

1University of Massachusetts Lowell, Lowell, MA, USA

2Salem State University, Salem, MA, USA

Corresponding Author:
James H. Nehring, College of Education, University of Massachusetts Lowell, 520 O’Leary Library, 61
Wilder Street, Lowell, MA 01854, USA.
Email: james_nehring@uml.edu
6 NASSP Bulletin 103(1)

populations and a history of differential educational outcomes, it is uncertain, given

current trends, whether this twin policy focus will lead to greater equity or will serve
to solidify and deepen existing patterns of privilege and marginalization.
We conducted a series of studies beginning in 2012 that asked several related ques-
tions (Nehring & Szczesiul, 2015; Szczesiul, Nehring & Carey, 2015). By way of
introduction, we summarize our previous studies to place this current study in context.
Initially, we were curious about the inherent conflict between the twin policy demands,
given the great deal of research showing that the emphasis on testing has fostered
instruction focused on a narrow and shallow skill set, especially in schools serving
high needs communities. We wondered, “Are there outlier schools serving high needs
communities that have figured out how to meet both policy demands (i.e., perform
well on tests) while also teaching the broader and deeper skill set that constitutes 21st
century skills?” If so, what might these schools teach us about the practical means of
accomplishing such a feat and the implications for policy? In one of the earlier studies,
we identified nine secondary schools in Massachusetts, serving high need communi-
ties, that reliably outperformed their demographic peers on the Massachusetts
Comprehensive Assessment System (MCAS). MCAS is the state-level series of stan-
dardized exams required for all students in public schools. We then assessed the
instructional demand in these schools by collecting 1 week worth of classroom materi-
als from as many teachers as would agree to participate. By instructional demand, we
mean teacher initiated tasks that require students to practice a particular skill. From the
nine schools, we collected 155 classroom artifacts (e.g., worksheets, rubrics, tests,
assignment descriptions) from a total of 73 teachers and analyzed them for instruc-
tional demand using an original rubric of deeper learning (see Theoretical Framework
and the appendix). Across the 1,729 distinct tasks embedded in the materials, we found
that none of the schools regularly or systematically taught deeper learning skills, based
on the evidence. All focused overwhelmingly on lower level skills required for test
performance (Szczesiul, Nehring & Carey, 2015). However, based on the results of
interviews with principals from the participating schools, we identified three schools
that claimed to be committed to developing an instructional focus on 21st century
skills through school goals and professional development.
In the study reported here, we took a deeper dive in these three schools by observing
classroom practice, focused by the following question: What are the classroom instruc-
tional patterns in schools serving high needs communities that regularly outperform their
demographic peers on state tests and express a school-level commitment to adapting
instruction to teach 21st century skills? In each school, we invited all teachers to take
part and particularly encouraged them to invite us to classes where they believed they
were teaching 21st century skills. Of the 22 classes we observed in the three schools, we
found that approximately one third exhibited an instructional demand for 21st century
skills. In many of the remaining classes, we found that teachers appeared to apply mis-
conceptions of 21st century skills, and unintentionally deployed those misconceptions to
reinforce the narrow and shallow skill set associated with test-based accountability. This
finding is consistent with patterns we encountered in the administrator and teacher
Nehring et al. 7

interviews we conducted. The article that follows lays out the theoretical framework for
our study, the methodology, findings, discussion, and implications.

Theoretical Framework
This study locates itself at the intersection of two streams of research activity from edu-
cation scholars around the world, stretching back several decades. They are as follows:

1. Research in the area of 21st century skills (Partnership for 21st Century
Learning, n.d) and the emerging construct of deeper learning (William & Flora
Hewlett Foundation, n.d.).
2. Research into the impact of test-based accountability on instructional practice
and student learning.

21st Century Skills and the Emergence of the Deeper Learning

Construct
Since the early 1990s, labor economists have theorized the emergence of a skill set
associated with a growing number of jobs in the United States and international labor
markets, resulting from economic globalization and advancing digital technology. The
U.S. Department of Labor’s Commission on Achieving Necessary Skills (1991) pro-
duced what may be the first major report identifying a skill set which has been reiter-
ated with strikingly little variation by policy reports in the decades since. Three major
initiatives active in recent years have yielded similar analyses. The Partnership for
21st Century Learning (n.d.), the International Society for Technology in Education
(2016), and the North Central Regional Educational Laboratory (Lemke, 2002) have
each published schema outlining 21st century skills.
The rationales driving the major 21st century skill initiatives are quite similar and
are summarized well in the early U.S. Department of Labor (1991) report:

Two conditions that arose in the last quarter of the 20th Century have changed the terms
of our young people’s entry into the world of work: the globalization of commerce and
industry and the explosive growth of technology on the job. These developments have
barely been reflected in how we prepare young people for work or in how many of our
workplaces are organized. Schools need to do a better job and so do employers. Students
and workers must work smarter. Unless they do, neither our schools, our students, nor our
businesses can prosper. (p. viii)

Though perceived labor market needs have been the chief driver of policy activity sur-
rounding 21st century skills, various education stakeholders have invoked 21st cen-
tury skills as the essential abilities that foster civic engagement in a world that is
increasingly complex.
The “21st century skills” construct is emerging as a fairly stable and relatively well
accepted policy framework internationally for what students need to know and be able
to do to thrive as workers and citizens in a globalized environment (Anandiadou &
8 NASSP Bulletin 103(1)

Table 1.  Deeper Learning Domains and Related Competencies.

Domains Related competencies

Cognitive domain Deep content knowledge
Critical thinking and complex problem solving
Interpersonal domain Collaboration
Communication
Intrapersonal domain Learning to learn competencies
Academic mind-set

Source. Adapted from Huberman, Bitter, Anthony, and O’Day (2014).

Claro, 2009; Voogt & Roblin, 2012). In 2011, the National Academies of Sciences,
Engineering, and Medicine initiated a project, led by Senior Program Officer Margaret
Hilton, to clearly define 21st century skills and to learn “what is known—and what
research is needed—about how these skills can be learned, taught, and assessed” in
schools (Pellegrino & Hilton, 2012, p. 1). The first outcome of this project was a syn-
thesis of related, contemporary concepts widely in use in policy and practice. They
include the following: deeper learning, 21st century skills, college and career readi-
ness, student-centered learning, next-generation learning, new basic skills, and higher
order thinking. The National Academies researchers divided learning into three
domains: cognitive, interpersonal, and intrapersonal. Within each domain, general
competencies were identified. Table 1 provides a summary of the domains and their
related competencies.
Between 2012 and 2016, the National Academies project produced several papers,
applying the deeper learning definition to empirical research in schools. For the series
of studies, the schools chosen belonged to school reform networks committed to vari-
ous elements of the deeper learning construct (Huberman et al., 2014). Matched
schools, not in any of the networks, were also selected for comparison. Bitter, Taylor,
Zeiser, and Rickles (2014) explored whether students in the network schools experi-
enced greater opportunities to engage in deeper learning. They found significantly
greater opportunities for deeper learning in the networked schools. Zeiser, Taylor,
Rickles, Garet, and Segeritz’s (2014) study examined student outcomes with respect to
deeper learning skills. This study found that students in the networked schools per-
formed better on the OECD Programme for International Student Assessment-based
test for schools, which includes complex problem solving. Also, students self-reported
better outcomes with respect to interpersonal and intrapersonal skills, and attended
4-year colleges at a higher rate. All of these indicators not only suggest the possibility
of stronger, deeper learning outcomes but also point to the difficulty of generating
measurable results of deeper learning. Pencil and paper tests, no matter how sophisti-
cated, are hard-pressed to measure interpersonal and intrapersonal skills. Self-reports
by students and college attendance rates, without information about persistence and
success, are also limited in their ability to persuasively indicate deeper learning
Nehring et al. 9

outcomes. With the problem of measurement in mind, we turn to the second stream of
research informing this study: test-based accountability.

Impact of Test-Based Accountability on Instructional Practice and

Student Learning
21st century skills has become a construct with global currency. However, the penetra-
tion of 21st century skills into public schools internationally remains weak (Anandiadou
& Claro, 2009; Voogt & Roblin, 2012). A barrier to success, particularly for schools
serving marginalized students, is the omnipresence of high stakes, mandated tests
across many industrialized nations, which disincentivize deeper learning. As the pol-
icy world increasingly relies on tests as a public accountability metric, there is grow-
ing evidence that schools under pressure for test performance narrow curriculum and
instruction in order to boost scores. Educators often look to the tests themselves more
than the standards to drive their instruction.
Schools serving populations of students who are overwhelmingly poor, students of
color, immigrants, English learners, and so on, are more beholden to test-based
accountabilities due to historical trends of underfunding and opportunity gaps. Thus,
the test-driven policies affect these schools and communities in a disproportionate
manner (Au, 2007; Hinde, 2003; McMurrer, 2007; Moon, Brighton, Jarvis, & Hall,
2007; Schoen & Fusarelli, 2008; Szczesiul, et al., 2015). By focusing on tests, schools
diminish their capacity to foster deeper learning and intensify a learning gap between
their students and students of privilege with an increasingly important skill set that
goes largely unmeasured. Because external accountability currently focuses on test
performance, 21st century learning in many schools fails to make its way into instruc-
tional practice. Schools tend to respond to the call for 21st century skills through a
vision statement or other types of administrative planning documents, but they rarely
respond at the level of practice because there are no assessments that hold them
accountable for doing so. This problem is particularly acute in schools serving low-
socioeconomic status populations in many OECD nations which are experiencing
increasingly intense pressure for improved test performance. While documentation of
schools that perform beyond expectation on tests exists (Carter, 2000; Herman et al.,
2008; Maden, 2001), we are only beginning to understand schools that also place a
demonstrable emphasis on 21st century learning outcomes.
In all three of the schools in the current study, principals and teachers were organiz-
ing curriculum to align with the Common Core State Standards Initiative (CCSSI)
during the period we collected data (2015-2016). The CCSSI, at the time, represented
a dominant policy initiative standing at the intersection of test-based accountability
and 21st century skills. The CCSSI was spawned by the National Governor’s
Association (2008) report. The report called for states to upgrade “standards by adopt-
ing a common core of internationally benchmarked standards in math and language
arts for grades K-12 to ensure that students are equipped with the necessary knowl-
edge and skills to be globally competitive” (p. 6). They also recommended that
10 NASSP Bulletin 103(1)

“schools and systems [be held] accountable through monitoring, interventions, and
support to ensure consistently high performance” (p. 6). Working with the National
Council of State School Officers and Achieve, the National Governor’s Association
produced a set of learning standards in 2010 (CCSSI, 2018a, 2018b), which, by 2014,
was adopted by 45 states and the District of Columbia. Public controversy over gov-
ernment overreach since then reduced this number and led to degrees of adoption by
various states.
Twenty-first century skills are evident across the standards. For example, a main or
anchor literacy standard states, “Write routinely over extended time frames (time for
research, reflection, and revision) and shorter time frames (a single sitting or a day or
two) for a range of tasks, purposes, and audiences” (CCSSI, 2018a, p. 44). Establishing
writing routines over extended time frames requires persistence. Research, reflection,
and revision require planning and self-awareness, all of which represent competencies
in the intrapersonal domain (see Table 1). Another example is a literacy standard that
states, “Prepare for and participate effectively in a range of conversations and collabo-
rations with diverse partners, building on others’ ideas and expressing their own
clearly and persuasively” (CCSSI, 2018a, p. 48). This standard calls for competency
in the interpersonal domain including collaboration, listening, and oral communica-
tion. It also requires competency in the intrapersonal domain in so far as the listener
has to self-monitor and self-manage in the context of conversations that are difficult
(i.e., that challenge assumptions, beliefs, values, etc.).
Though the policy call for 21st century skills was heeded in the standards, the
recent history of assessment programs designed for the CCSS shows problems of
measuring skills beyond the narrow and shallow skill set historically associated
with test-based accountability. In 2010, two entities won Federal Race to the Top
funding to develop tests: Smarter Balance and the Partnership for Assessment of
Readiness for College and Careers (PARCC). One problem with measuring the
standards was public pressure to reduce costs associated with the assessments.
Public pressure also reduced the amount of school time devoted to testing. PARCC
responded. Gewertz (2015) reports that, in 2011, PARCC went from four tests per
year to two. In 2014, the speaking and listening sections became optional, and in
May 2015, the assessment’s Board of Governors voted to reduce the time for the
test and further reduce frequency from twice per year to a single test at the end of
the year (Gewertz, 2015). This sharp reduction in the time and nature of the test
compromised the credibility of the exam as a measure of complex 21st century
skills. In addition to the compromises to time and scope, the fundamental nature of
the assessment (i.e., a machine readable, pencil and paper, or computer-based test)
strains credulity as a measure of collaboration, listening, metacognition, and self-
awareness—indeed a host of 21st century skills. These issues surrounding CCSSI
were in play, centerfield, during our data collection period, underscoring the ten-
sion between test-based accountability and 21st century skills. Schools in our study
were actively aligning curriculum with the CCSSI during the time of data collec-
tion. At the same time, the state test (i.e., MCAS) continued to dominate instruction
and assessment decisions.
Nehring et al. 11

Study Rationale
This study began when we recognized a potential conflict between the policy demands
for test performance and 21st century skills. Our hunch was borne out by findings of
published research: the demand for test performance incentivizes instruction focused
on a shallow and narrow skill set while the call for 21st century skills demands instruc-
tion for a skill set that is deep and broad (Wilder, Jacobsen, & Rothstein, 2008). We
found in the earlier study we conducted that among schools noted for test performance
serving high needs communities, evidence of instruction in 21st century skills was
scant at best (Szczesiul, et al., 2015). In this present study, we sought to uncover the
nature of instruction in a subset of those schools, which, by their own account, were at
the early stage of adapting instruction to meet the second policy demand for 21st cen-
tury skills.

Method
We used an original taxonomy of 21st century skills developed for our earlier study
(Szczesiul, et al., 2015), based on a review of relevant scholarship (Brookhart, 2010;
Doyle, 1983; Hattie, 2009; Pellegrino & Hilton, 2012). Our taxonomy draws heavily
on the work of Pellegrino and Hilton (2012), but differs from it in the cognitive domain
where we chose to retain a hierarchy of cognitive skills, a la Bloom (1956) and as
revised by Anderson et al. (2001). The hierarchy allowed us to differentiate lower level
skills associated with test-based accountability and higher level skills associated with
deeper learning. The taxonomy follows as Figure 1. An extended version of this tax-
onomy, including definitions for all skills, appears in the appendix.

School Sites
As previously mentioned, the three schools for this study were chosen based on out-
comes of our earlier study (Szczesiul et al., 2015). The three selected school sites were
among the highest performing schools in Massachusetts, based on test results, that
served a high needs population. They also showed evidence of an early-stage school-
wide commitment to instruction for 21st century skills, based on self-reporting by the
principals. All three schools were public secondary schools in the Commonwealth of
Massachusetts. All three served urban communities with majority non-White popula-
tions. In all three schools, the largest racial groups were as follows: African American,
followed by Hispanic. A majority (or near-majority) of students in all three schools
were considered to have high needs (Massachusetts Department of Elementary and
Secondary Education, 2018). Massachusetts Department of Elementary and Secondary
Education (2018) defines high needs as:

An unduplicated count of all students in a school or district belonging to at least one of

the following individual subgroups: students with disabilities, English language learners
12 NASSP Bulletin 103(1)

Figure 1.  Student skills expressed as tasks in three domains.

Note. An asterisk (*) indicates a task that is higher level, often described in the literature as a 21st
century skill.

(ELLs) and former ELLs (FLEP), or low income students. Students may be included in
more than one category. (para.11)

One school was a large, comprehensive high school serving Grades 9 to 12 (62%
high needs), divided into five nonthemed “houses” in order to create a small school
feel. One house had participated in the previous study and continued to participate
in the current study. Another school was a public charter school serving Grades
PK-8 (49% high needs). In this school, we focused on the upper division, Grades
4 to 8, which the principal described as an “experiment” in redefining the middle/
secondary level. The third school was a theme-based, district school of choice
(lottery-based in the event of oversubscription), serving Grades 9 to 12 (69% high
needs).

Sources of Data
Our purpose was to understand the nature of instruction across a range of classrooms.
As schools wrestle with the conflicting demands of test performance and 21st century
skills, we wondered: What may we learn from schools that are already high
Nehring et al. 13

performing with respect to tests and stand at an early stage of a process of adapting
instruction to also address 21st century skills? Therefore, we used the following types
of data sources: observations and interviews.

Observations.  In each school, we asked the teaching staff to invite us to observe any
class they taught. We informed them that we were interested, particularly, in observing
classes where 21st century skills were being taught. We shared our taxonomy with
them. This approach clearly biased teachers toward classes that they believed were
focused on 21st century skills. This approach was a deliberate choice because our goal
was to observe the results of early stage efforts to adapt instruction to include 21st cen-
tury skills. The general invitation limited our ability to manage the distribution of
classes across subject, grade level, and academic track. Over a period of several months,
we scheduled 22 observations of general education teachers. The three schools together
had approximately 130 general education teachers. Each observation consisted of a
researcher sitting in the classroom for the full period, taking notes and audio recording
the entire class. A simple categorization for task demand (low, medium, high) was cre-
ated for each domain. Due to the wide range of potential activities across multiple
subjects and grade levels, identification of categories involved complex, nuanced judg-
ments. All three researchers met several times to jointly listen to audio recordings and
discuss task demand for selected observations in order to norm our expectations. Figure 2
explains how each domain is categorized and provides examples.
Audio recordings of the observations were transcribed, and, based on the field
notes and transcription, for each class, a two column set of written notes was produced
narrating events in the left column and noting task demand in the right column. Each
class was rated based upon the highest level demand observed. In other words, a class
that was low demand for most of the time with a clear instance of moderate level
demand would be rated “moderate.”
While all classes observed, according to the teacher, emphasized 21st century skills,
we did not conduct follow-up interviews with the teachers and are therefore unable to
analyze teacher beliefs and the relationship between teacher beliefs and specific instruc-
tional episodes, except to say that all teachers believed they were teaching 21st century
skills in each class, after having the opportunity to view the taxonomy used in the study.
When side conversations occurred between the teacher and researcher during the obser-
vation, these were noted in field notes or captured in the audio recording.
An approximately equal number of classrooms were observed in each school and,
though the researchers did not select classes in order to achieve a distribution across
grade level, subject, and academic track, the 22 observed classes included a variety of
subjects, grade levels, and academic tracks in each school.

Interviews.  We also conducted administrator and teacher interviews in all three schools.
Interviews were conducted individually or in groups of two or three, based on avail-
ability. A total of 11 individuals were interviewed, including several from each school.
Interview participants tended to be different individuals from those observed in the
classroom. Overlap was not tracked as the purpose of the interviews was to gain a
14 NASSP Bulletin 103(1)

Figure 2.  Domain categorization and examples.

general impression of the ways in which 21st century skills were conceptualized
school wide.

Findings and Discussion

Instructional demand in 15 of 22 classrooms was in the low to moderate range. Seven
classrooms exhibited a high level of instructional demand in at least one domain,
which was consistently, the cognitive domain. All classrooms with high demand in one
domain (cognitive in all instances) showed at least moderate demand in at least one of
the other domains (interpersonal or intrapersonal). These patterns suggest that as the
level of cognitive demand rose, the level of demand for interpersonal, and/or intraper-
sonal skills also rose. Variation of instructional demand occurred within subject areas,
grade levels, and academic tracks; in other words, no clear association exists between
Nehring et al. 15

instructional demand and any of these factors. Table 2 shows a summary of our analy-
sis for the 22 classes and provides information such as grade level, subject, academic
track, and instructional demand for each of the three domains (cognitive, interper-
sonal, and intrapersonal).

Continuum of Student Engagement

We were curious about the nature of the classrooms that exhibited high instructional
demand; in other words, classes that were clearly teaching 21st century skills. We
wondered, “What is the nature of instruction in these classrooms?” We, therefore,
conducted a deeper analysis of all the observed classrooms, with a focus on the differ-
ences exhibited by those characterized by instruction for deeper learning.
The first pattern we observed was that in classrooms where 21st century skills were
strongly in evidence, instructional demands from the teacher moved up and down the
ladder of skills in our taxonomy. In one moment, the task would involve a high-level
skill such as evaluating or creating. Several minutes later, there would be a series of
low-level demands, such as recall or application. Then, later in the class, the demand
would be high again. Sometimes, the full range of skills was involved in a single
moment as when managerial steps were explained for a complex task. This pattern of
moving up and down the ladder of skills held across classrooms where high-level skill
demand was in evidence. We gave the pattern a name. We called it complex instruc-
tional demand to distinguish it from high-level instructional demand.
Next, we noticed that in classrooms with complex instructional demand a consis-
tently strong and effective effort was put forth by the teacher to engage students with
the instructional tasks. We decided we wanted to track the pattern of engagement. We
noted engagement retroactively by examining field notes, audio evidence from the
recordings, and noting the recollections of each researcher from classes observed.
Because we had not anticipated student engagement as a focus of our study, we looked
into the literature on student engagement to assist us in creating a simple continuum of
student engagement. We learned that student engagement as a construct remains prob-
lematic as Appleton, Christenson, and Furlong (2008) note, “Although uses of this
construct have proliferated, definitional clarity has been elusive” (pp. 369-370). They
further note that reliable measures of its dimensions have yet to be devised. Despite
the shortcomings of existing research, we believe that student engagement was a cru-
cial feature of what we were observing in classrooms. In this study, we understand it
to mean observable expressions of interest by students in the instructional task.
Moreover, as we observed it, engagement was more readily discernible as an attribute
of the whole class, rather than individuals. Engagement expressed itself in often subtle
and complex dynamics among students and through interactions with their teacher.
This approach is consistent with research that views engagement as a multidimen-
sional construct with both behavioral and affective attributes (Marks, 2000; Trowler,
2010). Based on the literature (Appleton et al., 2008; Marks, 2000; Trowler, 2010), we
devised a simple, intuitively defensible continuum (see Figure 3).
16 NASSP Bulletin 103(1)

Table 2.  Analysis of Instructional Demand for Observed Classes.

Cognitive Interpersonal Intrapersonal

Class Grade School Subject Track demand demand demand
1 10-11 CO U.S. history AP Moderate Low Low
2 12 CO Spanish AP High High Low
3 10 CO Biology CP Low Low Low
4 9 CO Environmental a Low Low Low
science
5 10 CO Math a Low Low Low
6 11-12 CO Pre-calculus a Low Moderate Low
7 5 CH English Not Low Moderate Moderate
language arts tracked
8 8 CH English Not Low Low Low
language arts tracked
9 8 CH Math Not Moderate Low Low
tracked
10 4 CH Math/science Not High High Low
Tracked
11 4 CH Math/science Not High High Low
tracked
12 7 CH Social studies Not Low Moderate Low
tracked
13 10 TH Humanities Not High Moderate Moderate
Tracked
14 10 TH Biology Not Moderate Low Low
tracked
15 10-11 TH Chemistry Not None/low Moderate Low
tracked
16 10 TH English Not High Low High
language arts tracked
17 12 TH Humanities Honors High High High
18 11-12 TH Humanities Not High Moderate High
trackedb
19 9 TH Humanities Not High Moderate Low
Tracked
20 9 TH Humanities Not Low Low Low
tracked
21 11-12 TH Political AP Moderate Moderate Low
science
22 11-12 TH Spanish Not None/low Nonec None
trackedb

Note. CO = comprehensive high school; CH = charter school; TH = theme-based district school, AP =

advanced placement; CP = college preparatory.
aDenotes part of a tracked system but track for this class is not known. bDenotes not tracked but AP is

offered as alternative to this class. cDenotes none observed.

Nehring et al. 17

Figure 3.  Continuum of student engagement.

Matrix of Student Engagement and Instructional Demand

To help us more systematically analyze the interaction of instructional demand and
student engagement in the classrooms we observed, we created a matrix with student
engagement on the y-axis and instructional demand on the x-axis (see Figure 4).
We then reanalyzed our classroom observations and plotted them on the matrix in
Figure 4. Many classes were plotted at the (+1, +1) and (+2, +2) locations meaning
that instructional demand tended to be in the zone of recall, application, and occa-
sionally analysis, while student engagement was in the zone of compliance and con-
geniality. Only a few classes were plotted in negative territory (little to no instructional
demand and student disregard). What interested us, however, were the considerable
number of classes in the upper right extreme of the matrix, meaning that instructional
demand was complex and student engagement was high (see Figure 5).
18 NASSP Bulletin 103(1)

Figure 4.  Matrix of student engagement and instructional demand (ID).

Emergent Themes for Complex Instructional Demand Classes

We wanted to better understand the classrooms where students were engaged in com-
plex instructional tasks. We listened again to the audio recordings and reviewed our
field notes looking for moves made by teachers who appeared to foster complex instruc-
tional demand and deep student engagement. Our analysis revealed several themes:

1. Because classes spanned subjects, grade levels, ability levels, and all three
schools in the study, the presence of complex instructional tasks and deep stu-
dent engagement did not appear to be linked to any of those factors.
2. Teachers were highly attuned to the social-emotional dynamics of individual
students and the class as a whole. Teachers addressed the dynamics, appar-
ently, for the purpose of creating a harmonious social-emotional environment,
as well as demonstrating an understanding that doing so is a prerequisite to
academic learning and student engagement.
3. Teachers consistently matched teaching moves to the dynamic moment in the
classroom in ways that fostered social-emotional harmony and effectively
advanced the academic lesson.
4. Teachers deployed effective moves with stunning fluency and density. It was
nearly impossible during the analysis to link a single move to a single purpose.
One move dissolved into the next as multiple purposes were served from
moment to moment. Watching any of these teachers was like watching a vir-
tuoso soloist perform with a symphony orchestra.
5. Teachers exhibited a powerful desire to connect relationally with students,
both individually and collectively, and displayed a nearly palpable joy in doing
so, while keeping instruction at the center of the relationship.
6. Instruction was tied to complex assessments, often performative in nature,
designed by the teacher, and formative—in contrast to the more routine
Nehring et al. 19

Figure 5.  Matrix of student engagement and instructional demand.

test-based, summative assessments or occasional, less demanding projects in

some other classrooms, and accountability instruments required by the state.
7. Complex task demand was regularly coupled with a high level of student
engagement. While this combination of attributes defined the classes we ana-
lyzed for the above themes, we feel it is important to list it as a theme, itself, so
as not to lose sight of it as the defining feature.

As an illustration of the seven themes exhibited by this group of teachers, we offer a

peek inside one classroom, a short segment taken from a Humanities class for Grade
11 and 12 students, which was a nonhonors track in a tracked school.

Themes in Action.  It was the Monday after Thanksgiving vacation. The narrative starts
at the beginning of class, as students were nearly all seated, and focuses mainly on just
the first 4 minutes. The commentary is admittedly interpretive in places; however, it
was vetted by three researchers, including the person who observed the class, each of
whom has many years of K-12 teaching experience.
As the class gathered, the teacher was leading an informal conversation from the
front of the room about Thanksgiving dinner. Several students were complaining about
how their Thanksgiving family dinners were boring, all the same stuff as every other
year. The teacher picked up on one student who did his own cooking. She affirmed that
student’s initiative and suggested, in reply to the complaining students, that they do
likewise. Her tone was light (there was plenty of laughter throughout the exchange),
but pointed. She seized this moment as a way to connect with her students while also
teaching a small lesson in empowerment: If you do not like the way someone else is
doing something, then step up to do it yourself.
The Thanksgiving dinner conversation continued with the teacher sharing what she
made and explaining how people in her house made the meal together. Students and
their teacher shared favorite dishes. The teacher was allowing the conversation to
20 NASSP Bulletin 103(1)

continue, though the time for class had begun. By allowing it, she was expressing
value in the conversation as a way to connect with students, affirming a topic that was
of interest to them, demonstrating that it was of interest to her too, and turning it spon-
taneously toward an instructional end; teacher and students had a shared personal
interest, a basis for a relationship. She also used the opportunity to model cooperative
meal making as a further answer to the students’ earlier complaint about having the
same thing every year, thus lending to the moment an instructional purpose (Theme 5).
Next, the teacher attempted to redirect the class from the Thanksgiving conversa-
tion to the planned lesson. She transitioned, warmly, with a “welcome back” statement
and a parting nod to the Thanksgiving conversation (the “lovely mac and cheese”
mentioned by one student). Her parting nod, however, prompted further talk about
Thanksgiving. She acknowledged this talk even though it was in direct opposition to
her attempt to transition to the day’s lesson because, apparently, it touched on one
student’s sensitive relationship with her mother that was known by multiple members
of the class. By allowing it, she demonstrated to the students that she is a teacher who
balances student emotional needs with curriculum (Theme 2).
She then made a second attempt to transition to the lesson and succeeded. The
students’ emotional needs had been met. A positive relationship among students
and between the teacher and the students was in place. The class was entirely with
her (Theme 3).
She began the lesson, an introduction to a new unit on the history and culture of
China, with the following statement:

So, what I’d like you guys to do is on this Venn diagram, would you please put down just
preliminary thoughts. What do you know about Chinese culture? You know, list what you
know about Chinese culture, list what you know about American culture, and in the
middle any similarities that you think exist between these two cultures. . . . So, just
brainstorming here.

In giving the instructions for the Venn diagram exercise, she made it clear through her
choice of words, her self-revision of phrasing, and her inflection (often up), that this
exercise was low stakes and students should freely associate ideas. Thus, she was pro-
moting intellectual risk taking. As she circulated among the students, she clarified that
they need not feel certain about the claims they would write down and that, in fact, she
anticipated some of them may turn out to be stereotypes. In this way, she continued to
foster risk taking. She explained how the results of this exercise were relevant to the
lesson, and she explained how all the claims, both the accurate and inaccurate, would
be valuable and would be used as part of the learning process later in the unit of study.
She, thus, validated anything and everything that students might write while also
acknowledging that at some point, the class would distinguish what is true from what
is untrue. Here, she balanced the importance of today’s intellectual risk taking with the
necessity of finding out, ultimately, what is true.
The lesson that followed gestured back to the previous unit (a study of Africa) and
introduced the new unit (a study of China). The rear-looking gesture was the return of
Nehring et al. 21

a set of essays, to be revised, about which the teacher said to the class, “I haven’t
given you a grade . . . I know that drives you crazy, but I hope it creates incentive . . .
this is about learning to write” (Theme 6). The introductory work that followed, for
the new China unit, had students inventing questions, with careful guidance, that
would frame the unit. In the course of inventing questions, the teacher taught students
how to transform lower level questions into higher level, open-ended questions. The
intellectual demands of this class were substantial, the expectation of the teacher, was
for students to make improvement (revising the essays), which was coupled with the
belief that they were capable of doing so and would indeed do so. The class was, also,
clearly focused on deeper learning (brainstorming, creating higher level questions,
collaborating).
What is particularly remarkable about the brief observational segment is that in less
than 4 minutes, the teacher moved the class from a fairly chaotic state of postvacation
boisterousness to a socially emotionally harmonious group, ready and willing to step up
to a demanding academic task. As she moved into the academic component, what was
remarkable was not so much the techniques she employed, which were entirely appro-
priate (e.g., accessing prior knowledge, explaining the purpose of the lesson, connecting
the current task to future tasks, etc.), but the nuanced way she deployed these techniques
to achieve more subtle purposes (Theme 4). For example, she used particular phrases
(“just preliminary,” “similarities that you think exist,” “just brainstorming here”) to
lower the stakes of the task, foster free association, and promote intellectual risk taking.
She also regularly inflected her voice up, implying that the exercise is about tentative
knowledge not certainty. But she also let students know (“we’ll revisit this throughout
our unit”) that today’s answers would not be the final word, that accurate information
(“to see whether or not we’re actually accurate”) is the ultimate goal and to prepare stu-
dents for the possibility that their answers today would be wrong (“or whether, maybe,
we’re a little bit off”), and that possibility is an expected part of the exercise and not a
negative mark. All of these moves suggest a teacher with a deep repertoire of teaching
strategies, coupled with a powerful social-emotional radar, a strong desire to connect in
a relational way with students, and a driving intellectual purpose—a veritable symphony
of attributes, performed with artistry (Theme 4, Theme 7).

Emergent Themes Among Low Instructional Demand Classes

As we analyzed the classrooms with complex demand, we also wondered about the
larger group of classrooms that exhibited lower instructional demand. Because all the
teachers of all of the observed classes believed they were emphasizing 21st century
skills, we wondered if patterns existed among the low-demand classes that suggest an
alternative understanding of task demand for 21st century skills.
We reviewed audio recordings of classroom observations and were struck by fre-
quent teacher comments to the class suggesting that academic rigor was very much on
their minds while they taught. They made comments such as, “I’m pushing you here,”
and “I am demanding more now.” There was clearly plenty of language around aca-
demic demand. The question then was as follows: What was the nature of the demand
22 NASSP Bulletin 103(1)

when we looked closely at the tasks to which students were set? Two patterns emerged
which we called more stuff and more steps.

More Stuff.  In some classes we observed, teachers presented students with complex
content; however, the tasks to which students were set, did not require complex
thought. Though the content was complex, the tasks were simple, mostly requiring no
deeper skills than recall or application.
This pattern was apparent, for example, in an advanced placement government
class for 11th and 12th graders. On the day we observed, the teacher was reviewing
material from a chapter in the textbook. The lesson consisted mainly in the teacher
stating terms. Students were asked recall questions about the definition of the terms,
and then asked to apply the terms.
To be successful in this class, students needed to be familiar with a number of
terms, such as secular realignment, demographics, party identification, Democrat,
Republican, blue state, red state, and purple state. They also had to be able to apply the
terms to simple problems, such as, “What factors predict a person’s party affiliation?”
The content was complex, but the tasks that invoked the content were relatively sim-
ple. While the required vocabulary was quite extensive, no deeper learning demands
were required of the students during this lesson. Several other classes in the group of
classes with low instructional demand showed the same pattern.

More Steps.  In other classes we observed, the teacher set students to tasks that required
complex procedures, but there was little evidence of complex thought. In an elective
history class for 11th and 12th graders, for example, the teacher was transitioning
students to the next chapter in a textbook.
A student paying close attention to the teacher’s rapid-fire instructions would find
the complexity of procedures in this class daunting. We did. In the space of a few min-
utes, the teacher’s instructions referenced chapter classifications (4.0, 4.1, 4.2), videos,
articles, learning objectives, learning targets, learning outcomes, an essential question,
a guiding question, a project, online quizzes, self-pacing, corrections, and a required
100% on all quizzes. The procedures were clearly complex, but when the class turned
to a review of material studied, task demand was at a strictly recall level.
Comprehension was not required for some questions when, for example, the teacher
provided part of a technical phrase (“clouds of____”) and the student needed only to
fill in the blank with the correct word (“gas”). This class is representative of a pattern
we observed in the group of classes with low instructional demand: Procedural com-
plexity with instructional demand at the recall level.
In the group of classes with low instructional demand, it was striking to see how
much teachers talked versus how much the students talked and what the nature of the
student talk was. Student talk was limited mainly to simple, one word responses to
teacher prompts offered in rapid succession. Extended, disciplinary, student-to-student
dialogue so characteristic of deeper learning was wholly absent.
Although we did not conduct follow-up interviews in association with the class-
room observations, we did conduct interviews with administrators and teachers in the
Nehring et al. 23

schools, asking them about their understanding of 21st century skills and their experi-
ences teaching them. Many teachers and administrators referenced relevant language
such as: “Bloom’s taxonomy,” “self-regulation,” “metacognition,” “collaboration.”
At the same time, their descriptions of class activities suggested a clear absence of
relevant instruction. One teacher, for example, commented on intrapersonal skills as
follows:

So I have older students. I don’t have freshmen, but I would say that transitioning from
the direct note taking—so I always tell my students, “Make notes. These are my notes and
they make perfect sense to me, but you need to look at them, listen to what I’m saying and
make them your notes.” Then similarly, they don’t know what to do with notes, so I’ll tell
them, “You have a test Friday. That doesn’t mean you take notes Monday and you don’t
look at them again till Thursday night.”

In this instance, the teacher expected the students to reread class notes; however, the
purpose was not directed toward a higher level cognitive skill (e.g., evaluation), but
simply toward greater success with a low-level cognitive skills-memorization. Likewise,
the teacher, in this account, did not offer any strategy for the student to achieve more
effective memorization, telling the students simply, “Make them your notes.”

Implications
The schools in this study had previously demonstrated high performance on state tests.
Interviews involving the principals revealed that all three schools were in the early
stages of adapting instruction to address 21st century skills through school goals and
professional development. Observation of a sample of classes, in which teachers
believed they were teaching 21st century skills, revealed that approximately one third
did in fact teach 21st century skills, while achieving a high level of student engage-
ment, based on our analysis of instructional demand. The other two thirds demanded
much less of students instructionally, though these teachers equally believed they were
teaching 21st century skills. Related teacher and administrator interviews showed a
pattern of relevant language in use while suggesting an absence of relevant instruction
in practice.
For our discussion of implications, we will first return to the aforementioned themes
that emerged from our analysis of the observed classes with complex instructional
demand. Then, we will consider the two patterns (more stuff, more steps) that emerged
from our analysis of classes with low instructional demand.
The themes we identified for classes with complex instructional demand suggest an
agenda for change at the school level and policy level. The first theme suggests that
factors such as subjects, grade levels, and ability levels should not be seen as a barrier
to high instructional demand. Also, a presumption that students in a low track are not
capable of handling complex material is likely wrong, which adds to the large body of
literature suggesting that the practice of tracking students based on perceived ability is
misguided (Oakes, 2005; Slavin, 1987, 1990). Themes 2 through 5 highlight the
24 NASSP Bulletin 103(1)

complexity of teaching a class with complex instructional demand. Taken together,

these themes require a strong foundation of technical knowledge, a high level of social
emotional awareness, a strong ability to marry the right instructional move to the
moment, and a powerful drive to connect relationally with their students. These attri-
butes imply a demanding agenda for teacher training, professional development,
school and system leadership, and policy development. Determining which moves are
best for supporting highly competent teachers goes beyond the scope of this study, but
it is clear that simplistic approaches will not work and will likely undermine the work
of highly competent teachers. We offer two broad suggestions. Within the profession
(e.g., teacher training, professional development, school and system leadership), com-
munities of practice are well suited to the fostering of complex skills because they
bring practitioners into contact with one another through the medium of inquiry and
reflective dialogue (McLaughlin & Talbert, 2006; Wenger, McDermott, & Snyder,
2002). At the policy level, legislators and regulators must manage the regulatory levers
deftly so as not to overly constrain teaching. As this and other studies demonstrate, the
current accountability fixation is heavily constraining.
Theme 6 (instruction was tied to complex assessments, often performative in
nature) suggests that instructional demand and assessment must work together. If
assessments are simplistic, instruction will tend to follow (Wilder et al., 2008). It is
striking that even though teachers at all three schools were observed actively incorpo-
rating CCSSI (2018a, 2018b) standards into the curriculum, instruction, by and large,
did not reflect the high-level instructional demand associated with some CCSSI stan-
dards. As policy makers fold 21st century skills into endorsed curricula, it will be for
naught unless assessments are stepped up also. The behavior of teachers in these three
schools bears that out. Most teachers continued to teach the low-level skills associated
with the MCAS exam even in the presence of curriculum that includes 21st century
skills. The minority of teachers whose instruction included 21st century skills, created
their own assessments, which, by and large, were not observed to be tests. In this
regard, it is interesting to note that the networked schools in Huberman et al. (2014),
cited above, tend toward performance-based assessments such as portfolios and exhi-
bitions, in which trustworthiness is achieved not through strict psychometric proper-
ties of reliability and validity, but through peer review and juried decisions. Recent
experience with Smarter Balance and PARCC suggest that for a variety of reasons
including both political will and technical difficulties (described above), large-scale
tests, whether delivered via paper or computer screen do not assess 21st century skills
despite claims to the contrary. It may well be that the path forward for assessment of
21st century skills does not run through the realm of measurement, but instead will be
achieved through an approach to evaluation that is driven by informed judgment and
includes peer review and juried exhibitions.
We turn next to the patterns observed among the majority of teachers whose instruc-
tional demand tended toward the low end of the continuum. While all of the observed
teachers stated or implied (by their invitations for us to observe) that they were teach-
ing 21st century skills, we found two thirds of the observed classes showed no evi-
dence. What can account for this phenomenon? Because we did not conduct follow-up
Nehring et al. 25

interviews with the observed teachers, we cannot know how they would address that
gap and that is a limitation of this study. We can, however, rule out some possibilities.
Because no association was observed between grade level, academic subject (includ-
ing tested and nontested subjects), academic track and instructional demand, we can
be reasonably confident that, in these schools at least, they were not major factors.
That suggests the reasons may be more idiosyncratic, and perhaps have to do with
teacher beliefs. Teacher epistemic beliefs may influence how a teacher conceives and
enacts instruction for 21st century skills. Teacher beliefs about the nature of knowl-
edge vary widely. Abundant research demonstrates this fact, particularly when it
comes to teacher beliefs about the nature of knowledge within their own academic
disciplines (Cross, 2009; Richardson, Anders, Tidwell, & Lloyd, 1991; VanSledright,
Maggioni, & Reddy, 2011). For an excellent review of research on teacher beliefs,
overall, see Fives and Gill (2014).
Within many academic disciplines, beliefs about the nature of disciplinary knowl-
edge range widely. Researchers have derived elaborate schema to represent this vari-
ety. When comparing schema, a broad pattern is discernible. Though an
oversimplification, it is nonetheless useful to summarize the range of teacher epis-
temic beliefs as a continuum. At one end, knowledge is seen as certain and fixed; at the
other, knowledge is tentative and socially constructed. It is only a small leap to infer
that pedagogy associated with the former will tend toward transmission of information
with memorization of facts and formulas, while pedagogy associated with the latter
will tend toward inquiry, collaboration, and problem solving. This inference is sup-
ported by research demonstrating that teacher beliefs powerfully influence instruction
(Thornton, 1991). While research suggests a strong link between teacher epistemic
beliefs and instruction, it also suggests the influence is not linear.
If a teacher views knowledge as fixed and certain, and the school is advocating
greater intellectual demand, there is only one direction this view will allow the
teacher to go up. In other words, more demand means more quantity. Quantity may
be expressed in either of two ways: a greater quantity of content (i.e., more stuff), or
a greater quantity of rules and guidelines (i.e., more steps). Patterns noted above
from classes and the teacher interviews fit this pattern. On the other hand, if a teacher
views knowledge as tentative and socially constructed, and the school is advocating
greater demand, then the natural direction to move is deeper inquiry with more
extensive collaboration and more complex problem solving (i.e., deeper learning).
Schools advocating for high-level instructional demand will do well to examine
closely what they mean. Merely invoking the right language is insufficient to alter
classroom practice. Interviewees exhibited abundant relevant language, but class-
room practice told a different story. Epistemology does not readily spring to mind as
a professional development topic, but attention to the question of how we know is
crucial to the transition from test performance to 21st century skills. Professional
development in the academic disciplines needs to emphasize the tentative, con-
structed nature of knowledge and a pedagogy of inquiry and problem solving that
naturally follows.
26 NASSP Bulletin 103(1)

Recommendations for Further Research

This study offers a detailed account of teaching practice in schools deemed high per-
forming. While it points us toward some interesting possibilities, larger scale studies
are needed to produce generalizable results. For example, do the themes regarding
complex task demand uncovered in this study hold true across a broader sample of
classrooms? The same can be asked of the teacher conceptions of 21st century skills,
also uncovered in this study. Our study also raises questions about the sources of
teacher behavior that we observed, namely:

1. How do teachers who regularly teach for complex instructional demand acquire
their skills? Is it training? Is it predispositions? What is the full range of influ-
ences including, for example, the parenting they experienced, the schooling
they experienced?
2. What role, if any, do teacher epistemic beliefs play in a school’s adoption of
high-level instruction? If teacher epistemic beliefs are a factor in a school’s
transition to higher instructional demand, what steps can be taken to alter
beliefs for those teachers with a view of knowledge that is fixed and
certain?

As the knowledge required for global citizenship becomes more complex, the systems
in which we educate the rising generation must evolve. Intensifying long standing
practices of low-level instructional demand and standardized tests will not get us there.
Educational excellence must be redefined. Exceptional teachers who are already doing
the work, like those featured in this study, are a good place to begin.

Appendix
Student Skills Expressed as Tasks in Three Domains
Cognitive Domain
Cognitive Standard 1: Recall

The task asks students to recall or reproduce information gained by reading, listening,
or observing.

Cognitive Standard 2: Application

The task asks students to execute or implement a procedure to solve a problem. These
problems usually have one best answer and they can be similar to problems students
have solved before.

Cognitive Standard 3: Analysis

Nehring et al. 27

The task asks students to break information into its parts and determine how the parts are
related to each other and to the overall whole. Processes include differentiating, organiz-
ing, and attributing. “Getting to the point” of something (main idea), analyzing argu-
ments (or theses), or comparing and contrasting are all examples of an analysis task.

Cognitive Standard 4: Evaluation*

The task asks students to judge the value of material and methods for given purposes,
based on standard criteria or criteria students invent themselves. The task requires
reasoned evaluation that can be stated as a thesis or a conclusion and supported with
logic and evidence. Processes include checking and critiquing. The task asks students
to judge whether a single fact or claim is true and whether it is relevant to the argument
or problem at hand, and/or to judge whether two or more things are consistent. Tasks
may emphasize deductive reasoning—starting with one or more premises (the basis
for an argument) and then reasoning with it to draw a conclusion. Tasks may empha-
size inductive reasoning—starting from an instance or instances and moving to a prin-
ciple. (Students have to reason from various aspects of a text, for example, and
determine its meaning as a whole.) This standard also includes tasks that ask students
to evaluate the credibility of a source of information, identifying assumptions implicit
in that information, and identifying rhetorical and persuasive methods.

Cognitive Standard 5: Creative Thinking*

The task asks students to put disparate elements together to form a new whole, or
reorganizing existing elements to form a new structure. Processes include generating,
planning, and producing. Examples: Coming up with alternative hypotheses based on
criteria; devising a procedure for accomplishing a given task, such as planning a
research paper on a given topic; inventing a product, such as a habitat for a specific
purpose.

Cognitive Standard 6: Information Literacy*

The task asks students to use information literacy skills (e.g., computer literacy skills,
library literacy skills, media literacy skills, network literacy skills, visual literacy skills).
The task requires students to use these skills to answer the questions: What information
do I need and where do I get it? How do I effectively convey this information?

Note. Expanded conceptualization of information literacy to include the translation of

information from one form to another (e.g., represent the information in a graph in
narrative form).

Interpersonal Domain
Interpersonal Standard 1: Teamwork/Collaboration*
28 NASSP Bulletin 103(1)

The task asks students to work with others by effectively communicating, cooperating,
being empathetic/taking different perspectives, building trust, taking on a service ori-
entation, resolving conflict, and/or negotiating.

Interpersonal Standard 2: Leadership*

The task asks students to be responsible, assertive, and influential members of a group.

Intrapersonal Domain
Intrapersonal Standard 1: Intellectual Openness*

The task asks students to be flexible, adaptable; to appreciate diversity and value con-
tinuous learning; to demonstrate artistic and/or cultural appreciation; to show intel-
lectual interest and curiosity.

Intrapersonal Standard 2: Metacognition*

The task asks students to use self-regulation strategies and skills such as forethought,
self-reflection, self-monitoring, self-evaluation, and self-reinforcement.

Note. An asterisk (*) indicates a task that is higher level, often described in the literature
as “21st century.”

Acknowledgments
Grateful acknowledgement is made to the U.S. Department of State Fulbright Program, and the
University of Massachusetts Lowell Seed Grant Program, for support that contributed signifi-
cantly to the development of this article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship,
and/or publication of this article.

Funding
The author(s) received no financial support for the research, authorship, and/or publication of
this article.

References
Anandiadou, K., & Claro, M. (2009). 21st century skills and competences for new millen-
nium learners in OECD countries (Working Paper No. 41). Retrieved from doi:10.1787/
218525261154
Anderson, L., Krathwohl, R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich,
P. R., . . .Wittrock, M. C. (2001). A taxonomy for learning, teaching, and assessing: A
Nehring et al. 29

revision of Bloom’s taxonomy of educational objectives. New York, NY: Addison Wesley
Longman.
Appleton, J., Christenson, S., & Furlong, M. (2008). Student engagement with school: Critical
conceptual and methodological issues of the construct. Psychology in the Schools, 45, 369-386.
Au, W. (2007). High-stakes testing and curricular control: A qualitative metasynthesis.
Educational Researcher, 36, 258-267.
Bitter, C., Taylor, J., Zeiser, K., & Rickles, J. (2014). Providing opportunities for deeper learning.
Retrieved from https://www.air.org/sites/default/files/downloads/report/Report%202%20
Providing%20Opportunities%20for%20Deeper%20Learning%209-23-14.pdf
Bloom, B. S. (1956). Taxonomy of educational objectives. Boston, MA: Allyn & Bacon.
Brookhart, S. M. (2010). Assessing higher-order thinking skills in your classroom. Alexandria,
VA: ASCD.
Carter, S. C. (2000). No excuses: Lessons from 21 high performing, high poverty schools.
Washington, DC: Heritage Foundation.
Common Core State Standards Initiative. (2018a). Common core standards for English lan-
guage arts and literacy in history/social studies, science, and technical subjects. Retrieved
from http://www.corestandards.org/assets/CCSSI_ELA%20Standards.pdf
Common Core State Standards Initiative. (2018b). Common core standards for Mathematics.
Retrieved from http://www.corestandards.org/Math/
Cross, D. I. (2009). Alignment, cohesion, and change: Examining mathematics teachers’ belief
structures and their influence on instructional practices. Journal of Mathematics Teacher
Education, 12, 325-346.
Doyle, W. (1983). Academic work. Review of Educational Research, 53, 159-199.
Every Student Succeeds Act of 2015, Pub. L. No. 114-195, § 114 Stat. 1177 (2016).
Fives, H., & M. G. Gill (Eds.). (2014). International handbook of research on teachers’ beliefs.
New York, NY: Routledge.
Gewertz, C. (2015, May 21). PARCC shortens its common-core test. Education Week. Retrieved
from https://www.edweek.org/ew/articles/2015/05/21/parcc-shortens-its-common-core-test
.html
Hattie, J. (2009). Visible learning. New York, NY: Routledge.
Herman, R., Dawson, P., Dee, T., Greene, J., Maynard, R., Redding, S., & Darwin, M. (2008).
Turning around chronically low-performing schools: A practice guide (NCEE #2008-4020).
Retrieved from https://ies.ed.gov/ncee/wwc/Docs/PracticeGuide/Turnaround_pg_04181.pdf
Hinde, E. R. (2003). The tyranny of the test: Elementary teachers’ conceptualizations of the
effects of state standards and mandated tests on their practice. Current Issues in Education,
6(10). Retrieved from https://www.researchgate.net/publication/260789208_The_tyranny
_of_the_test_Elementary_teachers’_conceptualizations_of_the_effects_of_state_standards
_and_mandated_tests_on_their_practice
Huberman, M., Bitter, C., Anthony, J., & O’Day, J. (2014). The shape of deeper learning:
Strategies, structures, and cultures in deeper learning network high schools. Retrieved
from https://files.eric.ed.gov/fulltext/ED553360.pdf
International Society for Technology in Education. (2016). ISTE standards for students: A prac-
tical guide for learning with technology. Arlington, VA: Author.
Lemke, C. (2002). enGauge 21st century skills: Literacy in the digital age. Naperville, IL: North
Central Regional Educational Laboratory.
Maden, M. (2001). Success against the odds: Five years on. London, England: RoutledgeFalmer.
30 NASSP Bulletin 103(1)

Marks, H. M. (2000). Student engagement in instructional activity: Patterns in the elementary,

middle, and high school years. American Educational Research Journal, 37, 153-184.
Massachusetts Department of Elementary and Secondary Education. (2018). Massachusetts
plan for equitable access to excellent educators. Retrieved from http://www.doe.mass.edu
/educators/equitableaccess/glossary.html
McLaughlin, M., & Talbert, J. (2006). Building school-based teacher learning communities:
Professional strategies to improve student achievement. New York, NY: Teachers College
Press.
McMurrer, J. (2007). Choices, changes, and challenges: Curriculum and instruction in the
NCLB era. Washington, DC: Center on Education Policy.
Moon, T., Brighton, C., Jarvis, J., & Hall, C. (2007). State standardized testing programs: Their
effects on teachers and students. Storrs, CT: National Research Center on the Gifted and
Talented.
National Governor’s Association. (2008). Benchmarking for success: Ensuring U.S.
Students receive a world-class education. Retrieved from http://www.corestandards.org
/assets/0812BENCHMARKING.pdf
Nehring, J. & Szczesiul, S. (2015). Redefining high performance in Northern Ireland: Deeper
learning and 21st century skills meet high stakes accountability. Journal of Educational
Change,16 (3), 327-34. DOI: 10.1007/s10833-015-9250-8.
No Child Left Behind Act of 2001, P.L. 107-110, 20 U.S.C. § 6319 (2002).
Oakes, J. (2005). Keeping track: How schools structure inequality (2nd ed.). New Haven, CT:
Yale University Press.
Organization for Economic Cooperation and Development. (2010). PISA 2009 results:
Executive summary. Retrieved from https://www.oecd.org/pisa/pisaproducts/46619703.pdf
Partnership for 21st Century Learning. (n.d.). Retrieved from http://www.battelleforkids.org
/networks/p21
Pellegrino, J., & Hilton, M. (2012). Education for life and work: Developing transferable know­
ledge and skill in the 21st century. Washington, DC: National Academies Press.
Richardson, V., Anders, P., Tidwell, D., & Lloyd, C. (1991). The relationship between teachers’
beliefs and practices in reading comprehension instruction. American Educational Research
Journal, 28, 559-586.
Schoen, L., & Fusarelli, L. D. (2008). Innovation, NCLB, and the fear factor: The challenge of
leading 21st-century schools in an era of accountability. Educational Policy, 22, 181-203.
Slavin, R. E. (1987). Ability grouping and student achievement in elementary schools: A best-
evidence synthesis. Review of Educational Research, 57, 293-336. doi:10.3102/003465
43057003293
Slavin, R. E. (1990). Achievement effects of ability grouping in secondary schools: A best-
evidence synthesis. Review of Educational Research, 60, 471-499. doi:10.3102/00346543
060003471
Szczesiul, S., Nehring, J., & Carey, T. (2015). Academic task demand in the 21st century, high stakes
accountability school: Mapping the journey from poor [to fair to good to great] to excellent?
Leadership and Policy in Schools, 14(4), 460-489. doi: 10.1080/15700763.2015.1026448
Thornton, S. J. (1991). Teacher as curricular-instructional gatekeeper in social studies. In Shaver,
J. P. (Ed.), Handbook of research on social studies teaching and learning (pp. 237-248).
New York, NY: Macmillan.
Trowler, V. (2010). Student engagement literature review. Retrieved from https://www.heacad
emy.ac.uk/system/files/studentengagementliteraturereview_1.pdf
Nehring et al. 31

U.S. Department of Labor. (1991). What work requires of schools: A report of the U.S.
Department of Labor’s Commission on achieving necessary skills. Washington, DC:
United States Department of Labor.
VanSledright, B., Maggioni, L., & Reddy, K. (2011, April). Preparing teachers to teach histori-
cal thinking? An interplay between professional development programs and school-systems’
cultures. Paper presented at the annual meeting of the American Educational Research
Association, New Orleans, LA.
Voogt, J., & Roblin, N. (2012). A comparative analysis of international frameworks for 21st
century competences: Implications for national curriculum policies. Journal of Curriculum
Studies, 44, 299-321.
Wenger, E., McDermott, R., & Snyder, W. (2002). Cultivating communities of practice.
Cambridge, MA: Harvard Business Review Press.
Wilder, T., Jacobsen, R., & Rothstein, R. (2008). Grading education: Getting accountability
right. New York, NY: Teachers College Press & Economic Policy Institute.
William & Flora Hewlett Foundation. (n.d.). K-12 teaching and learning. Retrieved from
https://hewlett.org/strategy/deeper-learning
Zeiser, K., Taylor, J., Rickles, J., Garet, M., & Segeritz, M. (2014). Evidence of deeper learn-
ing outcomes. Retrieved from https://www.air.org/sites/default/files/downloads/report
/Report_3_Evidence_of_Deeper_Learning_Outcomes.pdf

Author Biographies
James H. Nehring research focuses on school and system leadership, professional learning,
classroom instructional demand, and student assessment. At the University of Massachusetts
Lowell, he serves as graduate coordinator for the Ph.D. program in the College of Education.
Megin Charner-Laird research focuses on how schools respond to adaptive challenges and
how they center teacher expertise in the process. In the School of Education at Salem State
University, she serves as the lead faculty for assessment and co-coordinates the Programs in
Educational Leadership.
Stacy A. Szczesiul is an associate professor in the College of Education at University of
Massachusetts Lowell. Her research examines how the socio-political context of schooling
mediates efforts to improve teaching and learning.