Vol 453|1 May 2008
COMMENTARY
Making the grade
International testing that is used to predict the grim future of US science and technology
is being vastly misinterpreted, say Hal Salzman and Lindsay Lowell.
t’s a familiar story. Children around the
world have been tested, and the United
States is in trouble. A US Department of
Education report1 from March concludes that
“without substantial and sustained changes
to the educational system, the United States
will relinquish its leadership in the twenty-
first century”. The panic plays out in countless
newspaper articles and policy reports, recently
leading to legislative responses such as the
America COMPETES Act, which contains a
list of measures to boost average mathematics
and science test scores.
A country’s place in the new global economy
is, according to these reports, determined by its
rank in the maths- and science-score hierarchy.
Following this reasoning, one would conclude
that the US economy is threatened not only by
Japan and South Korea, but also by Finland, Sin-
gapore, New Zealand and the Czech Republic.
The rankings that engender these fears are pri-
marily based on two tests administered to mid-
dle- and high-school students since 1995: the
Programme for International Student Assess-
ment (PISA) and the Trends in International
Mathematics and Science Study (TIMSS).
Improving education should be a priority
for the nation, but erroneous interpretations
of international test scores may drive economic
and competitiveness policy in the wrong direc-
tion. When we consider that education testing
shows formidable US strength as the largest
producer of top-scoring students alongside a
significant problem at the bottom, the threat
to future competitiveness seems to be
something quite different from the
headlines2. Caution is needed
so we neither create
policies that
28
overstock the science and technology work- students is 30% lower than that of the United
C. EMIABATA
force nor unthinkingly implement the educa- States. In fact, the United States has a higher
tion and social practices in other high-scoring percentage of top-performing students than
countries. A full grasp of the meaning of test- 5 of the 14 others in the top-ranked group of
score differences should lead the next presi- countries with high average scores.
dent to address education and competitiveness Moreover, it would seem inappropriate to
problems more effectively than the recent consider the United States, a country with a
America COMPETES legislation, which is population of more than 300 million, in com-
now languishing for a lack of funding. Focus- petition with Singapore, a country of 4.5 mil-
ing the great consternation about education on lion, or with even smaller New Zealand. The
real rather than imagined problems requires a economies in these countries range from a gross
careful assessment of the evidence. domestic product (GDP) of $124 billion in New
Zealand to $236 billion in Finland, compared
Lagging behind? with the $14-trillion GDP of the United States.
It is misleading to gauge the relative position of Perhaps a more apt comparison would be Mas-
the United States in the world based on a sim- sachusetts with a population of 6.4 million and
plistic ranking of its students’ a gross state product of $338
test scores. This is much like billion, or Colorado with 4.8
measuring shoe size to pre- “Average test scores million residents and a $230-
dict runners’ future race are largely irrelevant billion state product. Although
times while ignoring their the top group also includes
past performance. There are as a measure of economic powerhouses South
substantial methodological economic potential.” Korea and Japan, which come
limitations in using these in at under a fourteenth and
tests to compare nations, less than a third, respectively,
including reporting ‘rankings’ that are based of the size of the US economy, for the most
on minute differences that are not statistically part it makes more sense to compare US state
significant3. For example, when considering economies with other countries because it is
statistically significant differences, national Massachusetts or California that is ‘competing’,
test scores can be clustered into three mean- for example, with Singapore in developing their
ingful levels and the United States consistently biotech industries.
ranks in a middle group on maths and science If, as we argue, average test scores are mostly
while being top ranked in civics4 — the study irrelevant as a measure of economic potential,
of citizenship and government. Overall, about other indicators do matter. To produce leading-
one-fifth of other nations rank better and two- edge technology, one could argue that it is the
fifths rank underneath the United States. numbers of high-performing students that is
Still, average performance tells us nothing most important in the global economy.
about the distribution of students with the very These are students who can enter the sci-
best test scores. In maths and science, when ence and engineering workforce or are likely to
looking at average scores, the United States is innovate whatever their field of study. Remark-
outranked by countries such as Finland and able, but little noted, is the fact that the United
South Korea. But the rankings change when we States produces the lion’s share of the world’s
examine the percentage of students who per- best students (see graph opposite).
form at the top, those most likely to be tomor- At the same time, low-performing students
row’s innovators. The South Korean average can hamper productivity and here, unfortu-
places it in the top-ranked group of nations, nately, the United States also stands out. The
yet its relative proportion of top performing United States produces more than one million
NATURE|Vol 453|1 May 2008 COMMENTARY

increased salaries and job openings, students openings in the US work-

respond. When the IT industry was growing, force”. Teitelbaum adds that
the number of graduates in computer science this overproduction, which
kept pace, doubling over six years. Follow- leads to increasingly longer
ing the collapse of the IT industry bubble, the postdocs in many science
number of graduates fell by 17% between 2003 fields, makes our universi-
and 2005. Employment in this field is just now ties look more like a system to
reaching the levels of the boom years but, with produce “a pool of low-cost research lab
little prospect of rapid growth, students seem workers with limited career prospects than a
to be wise in choosing other fields. Or, consider high-quality training program for soon-to-be
petroleum engineering. This is an industry academic researchers”.
that has had slow growth for two decades and,
correspondingly, undergraduate enrolments Social choices
declined 85% during that period, and master’s The beauty of brandishing a simple number
programmes instead attracted students from or a few facts is that they fit in a single head-
areas of the world with fast-growing oil explo- line and focus the reader’s attention. However,
ration. Today, 75% of US master’s graduates in before we send teams of educators to discover
petroleum engineering are foreign students on the educational secrets of Finland, Singapore,
temporary visas. Now, the New Zealand, South Korea
low-performing maths and science students US industry has a real need or Japan, we should do
each year, more than any other country in the for more engineers because “History suggests that more study into the nature
Organisation for Economic Co-operation and of increased demand for policies designed to and context of their educa-
Development except for Mexico (see graph oil and new exploration tion systems. As the PISA
overleaf). Although programmes to improve coupled with 20 years of
stockpile scientists and report6 notes, the tests do
education for low-performing students and minimal hiring and an engineers are counter- not evaluate schooling,
schools are included in the various policy ageing workforce. The oil productive.” per se, but the “cumula-
reports, they are shunted to the background industry has responded by tive impact of learning
when the headlines focus on increasing the increasing entry-level sala- experiences ... starting in
numbers of those at the top, and overall seem to ries 30–60% over the past four to five years, early childhood and up to the age of 15 and
carry little weight when they are diluted as part far greater than in other fields. As a result, embracing experiences both in school and at
of a long laundry list of recommendations. petroleum-engineering graduates have dou- home”. That is, much is made of a few select
bled in the past five years and freshmen enrol- schooling practices in each country and recom-
Market maths ments, at Texas Tech University in Lubbock for mendations are made to emulate them without
Without a doubt, science, maths and technol- example, have increased more than sixfold. considering what the effect is both on the lives
ogy education is needed in today’s society, When supply far exceeds demand, the bust of these children and on the economy.
whether for its citizens to understand enough that follows reverberates for many years and Sending children to classes six days a week,
to participate in public debate or just to operate discourages students even when demand does extra preparation courses nights and weekends,
the technology of everyday life. However, some increase later. As Michael Teitelbaum of the and having a single examinaton that decides
argue for more advanced courses as if they Alfred P. Sloan Foundation in New York notes, their fate, as is done in Japan, is not a choice
want to prepare all students to be scientists or not only is there no evidence of any widespread most US parents would make. Nor is the social
engineers. We believe that there is something shortages but “substantially more scientists discipline in Singapore that seems to keep stu-
fundamentally wrong with such an approach. and engineers graduate from US universities dents on the straight and narrow path: death
History suggests that policies designed to than can find attractive career for drug pushers, prohibitions on spit-
stockpile scientists and engineers are ting and, for offences in between such as
counter-productive. The COU N TR Y robbery after 7 p.m., at least 12 strokes
space race is typically cited P- PE RFO RM IN G STUDENTS BY of the cane. Although South Korea’s
as a success story of Ameri-
TO spectacular economic rise is held in awe,
can technological prowess, and its tenfold-per-capita GDP increase
but less often discussed is the over the past 20 years is widely praised,
impact of the workforce build- rarely noted is the close to 250% rise in
up on US engineering and sci- the incidence of suicide over the same
ence in the years that followed. period, with suicide becoming a lead-
Following a spike in the num- ing cause of death among young peo-
bers of science and engineer- ple. With South Korea are Finland and
ing college graduates in the New Zealand at the higher end of the
late 1950s and early 1960s, a global rankings of test scores and sui-
spectacular bust followed that cide rates. No single factor is respon-
led to high unemployment in sible for either high scores or suicides,
these fields. For many years but mental-health experts cite the
afterwards, fields such as physics pressure leading to one outcome as
were thought of as poor career a factor leading to the other in many
choices5 . Similar boom-and- high-scoring nations7.
–bust cycles have continued for The future educational path for
the past four decades, in engi- the United States should come from
neering, in information technol- looking within the country rather
ogy (IT) and in science. than lionizing faraway test-score
When demand is translated into champions. Our analysis3 of the
29
COMMENTARY NATURE|Vol 453|1 May 2008

LOWEST
-PERFOR
NUMBER MING ST
AND SH UDENTS
ARE OF BY COUN
ALL 30 O TOTAL (15 TRY:
RGANIS YEAR OL
AND DE ATION F DS IN
VELOPM OR ECON
ENT COU OMIC CO
NTRIES) -OPERAT
ION

data suggests two fundamental problems that discussion than the averages. So
require different approaches. First, pedago- we need to look beyond the mean
gies must address science literacy for the large to consider size and quality of the
numbers of low-performing students. Second, workforce and the content of our
education policy for our highest-performing education that will be driving
students needs to meet actual labour-market innovation. In a country that
demand. has a long history of innova-
In the United States, a decade’s worth of tion and high productivity, we
international test rankings based on slender should start by looking at how
measures of academic achievement in science our best schools educate top
and maths have been stretched far beyond performers. It is unlikely that
their usefulness. Perhaps policy-makers feel it they do so by the types of
is better to motivate policy by pointing to high- education heralded in other
scoring Czechs with fear, instead of noting our countries.
high-scoring Minnesotans as examples to Paying attention to the
emulate. But looking within the United States problems at the bottom is as
may be the best way to learn about effective important, if not more so, than focusing on
education. As the PISA authors emphasize in the top. The most innovative technology has
their report, 90% of the variance in the scores limited use if the more than 70 million work- Hal Salzman is at the Urban
is within countries rather than between coun- ers without college degrees do not have the Institute, 2100 M Street NW, Washington, DC
tries. Therefore, most of skills to use it effectively. 20037, USA. Lindsay Lowell is at the Institute
what one can learn about “A better understanding The nation’s low perform- for the Study of International Migration at
high performance is due
to the variation in factors
of the education data will ers and schools should be
a headline concern and
Georgetown University, 3300 Whitehaven Street,
Washington, DC 20007, USA.
within the nation’s borders. lead to better policy.” the remedies are often to
It would seem far more be found in schools only 1. www.ed.gov/about/bdscomm/list/mathpanel/report/
final-report.pdf
effective to transfer best practices across city a neighbourhood or town away. It will be far 2. Lynn, L. & Salzman, H. Issues in Science and Technology
and state lines than over oceans. more effective to take the best that America Winter, 74–82 (2006).
has to offer before seeking elusive and poorly 3. Lowell, B. L. & Salzman, H. In the Eye of the Storm: Assessing
the Evidence on Science and Engineering Education, Quality, and
Chasing tails understood practices found in a diverse collec- Workforce Demand (Urban Inst., Washington DC, 2007).
In America, little about the nation’s condition tion of small countries around the globe. 4. Boe, E. E. & Shin, S. Phi Delta Kappan 86, 688–695 (2005).
can be gleaned from averages, whether As advocates of evidence-based policy, we 5. Kaiser, D. Social Res. 73, 1225–1252 (2006).
6. Learning for Tomorrow’s World: First Results from PISA 2003
by assessments of income or edu- argue that competitiveness and education (OECD, Paris, 2004).
cation. Our great opportunities as policy should use the best available evidence 7. Lim, M. Asia’s Ongoing Struggle with Suicide International
well as our great limitations seem as a guide and not be driven by impressions Affairs Journal at UC Davis (29 June 2007).
to be accompanied by great dis- and rhetoric. Our analysis suggests that a better 8. PISA 2006 Science Competencies for Tomorrow’s World
(OECD, Paris, 2008).
parities. It is these extremes, the understanding of the education data will lead to
tails at either end of the distribu- better and, in many cases, different policy direc- To discuss this article or any of our education
tion, that require much more tions from those now being advocated. ■ material this week, visit http://tinyurl.com/6ndqko

30