Knowing What Students Know

Author(s): JAMES W. PELLEGRINO

Source: Issues in Science and Technology , WINTER 2002–03, Vol. 19, No. 2 (WINTER
2002–03), pp. 48-52
Published by: University of Texas at Dallas

Stable URL: https://www.jstor.org/stable/43312299

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms

is collaborating with JSTOR to digitize, preserve and extend access to Issues in Science and
Technology

This content downloaded from

137.166.4.123 on Sat, 12 Aug 2023 03:44:10 +00:00
All use subject to https://about.jstor.org/terms
 JAMES W. PELLEGRINO

Knowing What
Students Know
Recent advances in
the cognitive and
measurement
sciences
Many people are simply puzzled by should be the past 30 years, there have been
the heavy emphasis on standard- major advances in the cognitive
the
ized testing of students and eager foundation for sciences that help us to understand
to find out exactly what is gained
developing a new the ways that people learn, which
by such activity. The title of the Na- in turn help identify more pre-
tional Research Council study that I system of student cisely what aspects of student
co-chaired states the goal directly: assessment achievement we should be trying
Knowing What Students Know. The to assess. During the same period,
concerns about student assessment there has been equally rapid
are quite well known: misalignment progress in the development of
of high-stakes accountability tests measurement techniques and in-
and local curricular and instructional practices, nar- formation technology that enhance our ability to col-
rowing of instruction by teaching to tests with re- lect and interpret complex data and evidence. The
stricted performance outcomes, the frequent failure committee brought together experts in measurement,
of assessments to provide timely and instructionally cognitive and developmental psychology, math and
useful and/or policy-relevant information, and the fail- science education, educational technology, neuro-
ure to make full use of classroom assessments to en- science, and education policy to determine how these
hance instruction and learning. recent developments can be applied to assessment.
The goal of our committee was not to review all The committee's stated mission was to establish a
the alleged shortcomings of past or current tests but totheoretical foundation for the design and develop-
take the opportunity to rethink assessment. During ment of new kinds of assessments that will help stu-
dents learn and succeed in school by making as clear
as possible to them, their teachers, and other education
stakeholders the nature of their accomplishments and
James W. Pellegrino (pellegjw@uic.edu), Distinguished Pro-the progress of their learning. Most assessments now
fessor of Cognitive Psychology and Education at the Univer-
in use fail to meet this objective.
sity of Illinois at Chicago, was co-chair of the National Re-
search Council committee that produced Knowing What Students We want assessment to be a facilitator of higher
Know: The Science and Design of Educational Assessment (Na-levels of student achievement. This will require a de-
tional Academies Press, 2001). parture from current practice that must be guided by

48 ISSUES IN SCIENCE AND TECHNOLOGY

This content downloaded from

137.166.4.123 on Sat, 12 Aug 2023 03:44:10 +00:00
All use subject to https://about.jstor.org/terms
STUDENT ASSESSMENT

further research as well as policy changes.mation

What about student competence. Assessments based
we've learned from research enables and compels us
on a complex and detailed understanding of how stu-
to do a better job with assessment. dents learn will not yield all the information they oth-
erwise might if the statistical tools available to inter-
What is assessment? pret the data, or the data themselves, are not sufficient
Assessment is a process of gathering information for for the task.
the purpose of making judgments about a current state We should move away from the simplistic no-
of affairs. In educational assessment, the information tion that a test is a test is a test. We have to make
collected is designed to help teachers, administrators, certain that any given assessment is designed for its
policymakers, and the public infer what students know specific purpose and that within that context all three
and how well they know it, presumably for the purpose legs of the assessment triangle are strong.
of enhancing future outcomes. Part of the confusion
that I mentioned above stems from the fact that some Scientific foundations
of these outcomes are more immediate, such as the Advances in the sciences of thinking and learning
use of assessment in the classroom to improve learn- need to be used for framing the model of cognition,
ing, and others are more delayed, such as the use of as- which must then inform our choices about what ob-
sessment for program evaluation. servations are sensible to make. Developments in
This means that in looking at any assessment, measurement and statistical modeling are essential
we have to keep in mind issues of context and pur- for strengthening the interpretation of our observa-
pose. Sometimes we're looking for insight into the tions. For a thorough description of recent develop-
state of affairs in the classroom, at other times em- ments in the cognitive sciences, I must refer you to
phasis is on the school system. The particular focus another NRC study How People Learn. I can't begin
could be to assist learning, measure individual to do justice to what's been accomplished in this large
achievement, or evaluate programs. And here's the and dynamic field, so I'll limit myself to the bare
rub: One size does not fit all. By and large, this has bones of what we need to know in order to develop
been overlooked in the United States. As a result, we sound assessments.
have failed to see that changes in the context and pur- The most critical implications for assessment are
pose of an assessment require a shift in priorities, in- derived from study of the nature of competence and
troduce different constraints, and lead to varying the development of expertise in specific curriculum
tradeoffs. When we try to design an all-purpose as- domains such as reading, mathematics, science, and
sessment, what we get is something that doesn't ad- social studies. Much is known now about how knowl-
equately meet any specific purpose. edge is organized in the minds of individuals - both
Any assessment must meld three key compo- experts and novices in a field. We know more about
nents: cognition, which is a model of how students metacognition: how people understand their own
represent knowledge and develop competence in the knowledge. We recognize that there are multiple paths
domain; observations, which are tasks or situations to competence; that students can follow many routes in
that allow one to observe students' performance; and moving from knowing a little to knowing a lot. One of
interpretation, which is a method for making sense my favorite insights is that we now know much about
of the data relative to our cognitive model. Much of the preconceptions and mental models that children
what we've been doing in assessment has been based bring to the classroom. They are not blank slates, and
on impoverished models of cognition, which has led we can't simply write over what is on those slates. In
us to highly limited modes of observation that can fact, if we fail to take into account how they know
yield only extremely limited interpretations of what things, we are very likely to fail in instruction. In ad-
students know. dition, much of students' knowledge must be under-
It does little good to improve only part of this stood as highly contextualized and embedded in the
assessment triangle. Sophisticated statistical tech- situation in which it was acquired. The fact that we
niques used with restricted models of learning or re- have taught fractions does not mean that students have
stricted cognitive tasks will produce limited infor- a broad and flexible knowledge of fractions.

WINTER 2002-03 49

This content downloaded from

137.166.4.123 on Sat, 12 Aug 2023 03:44:10 +00:00
All use subject to https://about.jstor.org/terms
 Contemporary knowledge from will require more thought and re-
the cognitive sciences strongly im- search to explore the fit between
plies that assessment practices need When we try particular statistical models and
to move beyond discrete bits and varying descriptions of compe-
pieces of knowledge to encompass
to design an tence and learning. Doing so re-
the more complex aspects of stu- all-purpose quires extensive collaboration
dent achievement, including how among educators, psychometri-
their knowledge is organized and
assessment , what cians, and cognitive scientists.
whether they can explain what they we get is something
know. There are instructional pro- Assessment design and use
that doesn't
grams and assessment practices for In moving from the scientific
areas of the curriculum based on
adequately meet foundations to the design and use
cognitive theory, but much more of effective assessment, we en-
work is needed. any specific counter four major challenges:
Interpreting data on student per- purpose. developing principles to guide the
formance is neither simple nor process of assessment design and
straightforward. Researchers have development; identifying assess-
been refining techniques for decid- ment practices that are connected
ing how data can be filtered and to contexts and purposes; explor-
combined to produce assessment results, as well asing feasibility questions, particularly the potential for
determining how much information on what types of applying technology to many new design and imple-
tasks needs to be collected. Statistical models are es-
mentation challenges; and being prepared to consider
pecially useful in four situations: high-stakes con-the possibility of a radical vision for the transforma-
texts in which we want to make decisions that have tion of assessment.
major effects on students' lives; when we know rel- Assessment design should always be based on a
atively little about the history and experiences of the student model, which suggests the most important
students being assessed; for complex models of learn-aspects of student achievement that one would want
ing; and for large volumes of data. to make inferences about and provides clues about
How do advances in measurement enable us to the types of tasks that will elicit evidence to support
go beyond simplistic models of general proficiency? those inferences. What this means for the classroom
Three general sets of measurement issues account teacher is that assessment should be an integral part of
for most of the discontent with current measurement instruction. Students should get information about
models, and each concern can be accommodated by particular qualities of their work and what they can do
various newer models. The first concern is whether to improve. Along the way, they should also be helped
we need to capture performance in terms of qualita- to understand the goals of instruction and what level
tively discrete classes or a single general continuumof performance is expected of them. Of course, this
will become possible only when cognitive science
of performance. The second is whether we need to
research findings are expressed in a user-friendly lan-
be evaluating single or multiple attributes of perfor-
mance. The third concern is whether we are evalu- guage. The implication for large-scale testing is that
ating status at a single point in time or change andan integrated approach holds promise for drawing
growth over a period of time. Dealing with these is-more valid and fair inferences about student achieve-
sues requires a progression of models and methods ment.
of But in order to implement this approach, poli-
increasing complexity. cymakers will have to relax the constraints that drive
Fortunately, we have a collection of new mea-
current practices. For example, states should be al-
surement methods that can deal with more complex lowed to administer more complex tests to a sample of
measurement concerns, but they have yet to be ap- students to acquire more fine-grained information on
what students can do or to embed assessments in the
plied completely to the practical work of assessment.
Information technology can help in this effort, butcurriculum
it in subjects such as art. The goal is to re-

50 ISSUES IN SCIENCE AND TECHNOLOGY

This content downloaded from

137.166.4.123 on Sat, 12 Aug 2023 03:44:10 +00:00
All use subject to https://about.jstor.org/terms
STUDENT ASSESSMENT

place single assessment tests with the curriculum while also devel-
systems of assessment that cut oping statistical models designed
across contexts and are more com-
We should develop to match the cognitive theories. To
prehensive, coherent, and contin- improve the fairness of the tests,
uous. A number of assessment sys- reporting we need to find reliable ways to
tems have been developed, and mechanisms that incorporate what we know of the
they give us a good starting point individual student's instructional
for further development.
enable teachers and history. And to ensure that the new
Computer and telecommuni- students to use assessments will work in practice,
cations technologies provide pow- we should form multidisciplinary
erful new tools that are needed to assessment results design teams that include practi-
meet many of the design and im- tioners as well as researchers.
plementation challenges implied by
to focus their efforts Practitioners will have to as-
merging cognitive models and mea- and improve sume a key role in research de-
surement methods. Compared to a
educational signed to explore how new forms
conventional paper-and-pencil test, of assessment can be made practi-
a test designed for administration outcomes . cal for use both in classroom and in
on a computer makes it possible to large-scale contexts and how vari-
consider novel ways of presenting ous new forms of assessment af-
questions, including a richer mix fect student learning, teacher prac-
of task designs and question formats, assessing a tice, and educational decisionmaking. This will entail
broader repertoire of cognitive skills and knowledge, developing ways to assist teachers in integrating new
recording and scoring complex aspects of behavior, forms of assessment into their instructional practices,
and embedding assessments in learning environments. exploring ways that school structures (such as class
If we let our imaginations roam a little, we can see an period length, class size, and opportunities for teacher
assessment system that is radically different from the interaction) affect the feasibility of effectively im-
annual multiple-choice test on a narrow slice of mate- plementing new types of assessment.
rial. Developers of educational curricula and class-
Rich sources of information about student learn- room assessments should create tools that will en-
ing can be continuously available across wide seg- able teachers to implement high-quality instructional
ments of the curriculum and for individual learners and assessment practices that are consistent with mod-
over extended time periods. Electronic tests can be ern understanding of how students learn and how
far more easily customized to the individual student such learning can be measured. But simply delivering
and to the particular purpose or context. As we con- these tools to teachers will not be enough. The teach-
sider how to apply the insights of cognitive and mea- ers themselves will need to be familiar with the cog-
surement science, we should not limit ourselves tonitive science from which the new tools and ap-
tinkering with testing as we know it. We should imag- proaches have emerged. We need to develop training
ine assessment that is truly useful for a number ofmaterials that will enable teachers to understand how
purposes, most importantly to aid in student learning.new research should be incorporated into their class-
room practice. Instruction in how students learn and
Road map for action how learning can be assessed should be a major com-
This vision of an assessment system designed to help ponent of teacher preservice and professional devel-
students learn will become reality only if we con-opment programs. Standards for teacher licensure
tinue making research progress, reform policies toand program accreditation should include training in
allow innovation in testing, and take the steps nec-assessment. This clearly is an enormous task and an-
essary to make changes in practice. The major re- other area in which new information technology can
search challenges include developing models of cog-play a critical role. Computer- and Internet-based
nition and learning that are particular to each area oftraining materials could make it possible to reach a

WINTER 2002-03 51

This content downloaded from

137.166.4.123 on Sat, 12 Aug 2023 03:44:10 +00:00
All use subject to https://about.jstor.org/terms
large number of teachers quickly and at relatively their ability to make decisions about education pro-
low cost. grams and the allocation of resources. We must dis-
The country is apparently committed to the need abuse them of the dangerous belief that important
for large-scale assessments of all students, but that decisions can be based on single test score. An ac-
need not be a commitment to the type of tests we use curate picture of student achievement can be attained
today. We can develop assessment systems that ex- only through a mix of assessment measures, from the
amine the broad range of competencies and forms of classroom to the national level, that are coordinated to
student understanding that research shows are im- reflect a shared understanding of how learning oc-
portant aspects of student learning. Although this can curs. Policymakers also need to understand that as-
involve some of the familiar forms of testing, it must sessment is a movie, not a snapshot. What we really
also incorporate a variety of other approaches, such as want to know is how students are progressing over
matrix sampling of items for test administration and time, not where they stand on a particular day.
curriculum-embedded measures. Finally, educators and researchers need to work in
We also need to devote more attention to ex- cooperation with the media to improve public un-
plaining our new approaches to students, teachers, derstanding of assessment, particularly as more com-
plex assessment systems are implemented. Parents
and administrators. All participants should be helped
to understand what the learning goals are and how and voters need to be able to use this information to
the assessment process advances those goals. This guide their decisions. Without sound information in
extra effort should also extend to the reporting of accessible
re- form, they will become even more puz-
sults. Too often the test score is communicated simply zled by the mysteries of assessment.
as a verdict with which one must live. We should de- If we all work together, we can replace assess-
velop reporting mechanisms that enable teachers and ments that are often narrow one-shot tests of skills
divorced from the school curriculum with an assess-
students to use the results to focus their efforts and im-
prove educational outcomes. ment system that is comprehensive in merging a mix
We need to work with policymakers to help themof measurement approaches, coherent in its link to
sophisticated models of learning and its alignment
recognize the limitations of current assessments and
to convince them to support the development of new
with the curriculum, and continuous in linking each
systems of multiple assessments that will improve
student's performance over time.

52 ISSUES IN SCIENCE AND TECHNOLOGY

This content downloaded from

137.166.4.123 on Sat, 12 Aug 2023 03:44:10 +00:00
All use subject to https://about.jstor.org/terms