EDUCATIONSECTOR REPORTS

Measuring skills FOr THe 21sT CenTury

By Elena Silva
hovambar 2008
A0Kh0wLF00FNFhTS
ThankstomyEducationSectorcolleaguesKellyBathgate,Kevin
Carey,StaceyJordan,RobManwaring,ReneRybak,andBill
Tuckerfortheirhelpinthinkingabouttheissuesofassessment
andstudentlearningoutcomes,andtoThomasTochandAndrew
Rotherhamfortheirsupportinthewritingandeditingofthispaper.
ThanksalsotoAbdulKargboforhissupportindesigninggraphics
andformattingthereport.RobinSmilesdeservesspecialthanks
forhercarefuleditinganddiligenceinmanagingtheproductionof
thisreport.IwouldalsoliketothankCeciliaLe,whocontributed
substantialtimeandefforttothisprojectasaFordhamFellowfor
EducationSector,andClairaWilliams,whoalsoprovidedvaluable
researchassistance.Mysincereappreciationtothemanypeople
whowerekindenoughtoreadandcommentonanearlierdraftof
thispaper.Iamparticularlygratefultothestudents,teachers,and
directorsatSt.AndrewsSchoolinMiddletown,Delaware,andat
NewTechnologyHighSchoolinNapa,California,whograciously
hostedourvisitstotheirschoolsandclassroomsandspenthours
collectivelyspeakingtousabouttheirteachingandassessment
practices.ThanksalsotoAn-MeChungoftheCharlesStewart
MottFoundationforherongoingsupportofthiswork.
ThisreportwasfundedbytheCharlesStewartMottFoundation.
EducationSectorthanksthefoundationfortheirsupport.The
viewsexpressedinthepaperarethoseoftheauthoralone.
A80UT THF AUTH08
ELENA SILVAisaseniorpolicyanalystatEducationSector,
wheresheoverseestheorganizationsteacherqualitywork.She
canbereachedatesilva@educationsector.org.
A80UT F0U0AT|0h SF0T08
EducationSectorisanindependentthinktankthatchallenges
conventionalthinkingineducationpolicy.Weareanonproft,
nonpartisanorganizationcommittedtoachievingmeasurable
impactineducation,bothbyimprovingexistingreforminitiatives
andbydevelopingnew,innovativesolutionstoournationsmost
pressingeducationproblems.
Copyright 2008 Education Sector.
Education Sector encourages the free use, reproduction, and distribution of our
ideas, perspectives, and analysis. Our Creative Commons licensing allows for the
noncommercial use of all Education Sector authored or commissioned materials.
We require attribution for all use. For more information and instructions on the com-
mercial use of our materials, please visit our Web site, www.educationsector.org.
1201ConnecticutAve.,N.W.,Suite850,Washington,D.C.20036
202.552.2840www.educationsector.org
A80UT TH|S SF8|FS
ThisreportisaproductofEducationSectorsNextGeneration
ofAccountabilityinitiative.Theinitiativeseekstostrengthen
publiceducationbyexaminingkeyelementsofaccountability,
forinstance,whoshouldberesponsibleforstudentsuccessand
howtheyshouldbeheldresponsible.Ourworkseekstobuild
onthestrengthsofcurrentschoolaccountabilitysystems,more
fullyandeffectivelymeasurethedepthandbreadthofstudents
educationalexperiences,andencourageeducators,parents,
policymakers,andthelargerpublictopursueeducationalequity
andexcellenceforallstudents.
1
Measuring Skills for the 21st Century www.educationsector.org
contentandbasicskillslikereadingandcomputationfrom
moreadvancedanalyticalandthinkingskills,eveninthe
earliestgrades.
Butstandinginthewayofincorporating21stcenturyskills
intoteachingandlearningarewidespreadconcernsabout
measurement.Thecost,timedemands,anddiffcultyin
scoringtestsoftheselesseasilyquantifedskillshave
slowedtheadoptionofsuchtests,ashaveconcerns
amongcivilrightsadvocatesthatthesetestswould
erodeprogresstowardensuringcommonstandardsof
learningforallstudents.Collectively,theseconcerns
derailedeffortsinthelate1990stomovetowardtheuse
ofperformance-basedassessmentssuchasportfolios,
exhibitions,andprojects.
NewassessmentsliketheCWRA,however,illustrate
thattheskillsthatreallymatterforthe21stcenturythe
abilitytothinkcreativelyandtoevaluateandanalyze
informationcanbemeasuredaccuratelyandina
commonandcomparableway.Theseemergentmodels
alsodemonstratethepotentialtomeasurethesecomplex
thinkingskillsatthesametimethatwemeasurea
studentsmasteryofcorecontentorbasicskillsand
knowledge.Thereis,then,noneedformoreteststo
measureadvancedskills.Rather,thereisaneedforbetter
teststhatmeasuremoreoftheskillsstudentsneedto
succeedtoday.
Unmet Challenges
Theideathatschoolsshouldfocusonmorethan
justthebasicsisnotnew.Acenturyago,leadersof
theprogressiveeducationmovement,spearheaded
byAmericanphilosopherandeducatorJohnDewey,
Studentsweregivenresearchreports,budgets,and
otherdocumentstohelpdrafttheiranswers,andthey
wereexpectedtodemonstrateprofciencyinsubjects
likereadingandmathaswellasmasteryofbroaderand
moresophisticatedskillslikeevaluatingandanalyzing
informationandthinkingcreativelyabouthowtoapply
informationtoreal-worldproblems.
Notmanypublicschoolstudentstakeassessmentslike
theCWRA.Instead,moststudentstaketeststhatare
primarilymultiple-choicemeasuresoflower-levelskillsin
readingandmath,suchastheabilitytorecallorrestate
factsfromreadingpassagesandtohandlearithmetic-
basedquestionsinmath.Thesetypesoftestsareuseful
formeetingtheprofciencygoalsofthefederalNo
ChildLeftBehindAct(NCLB)andstateaccountability
systems.Butleadersinbusiness,government,andhigher
educationareincreasinglyemphaticinsayingthatsuch
testsdontdoenough.Theintellectualdemandsof21st
centurywork,todaysleaderssay,requireassessments
thatmeasuremoreadvancedskills,21stcenturyskills.
Today,theysay,collegestudents,workers,andcitizens
mustbeabletosolvemultifacetedproblemsbythinking
creativelyandgeneratingoriginalideasfrommultiple
sourcesofinformationandtestsmustmeasure
studentscapacitytodosuchwork.
Whilemanypolicymakers,includingSecretaryofEducation
MargaretSpellings,haveemphasizedtheneedfor
schoolsto,frstandforemost,teachthebasics,learning
scienceaninterdisciplinaryfeldthatincludescognitive
science,educationalpsychology,informationscience,and
neurosciencesuggeststhatthebestlearningoccurs
whenbasicskillsaretaughtincombinationwithcomplex
thinkingskills.Decadesofresearchrevealsthatthereis,in
fact,noreasontoseparatetheacquisitionoflearningcore
When ninth-graders at st. andrews school, a private boarding school in
Middletown, Delaware, sat down last year to take the schools College
Work and readiness assessment (CWra), they faced the sort of problems
that often stump city offcials and administrators, but rarely show up on
standardized tests, such as how to manage traffc congestion caused by
population growth. i proposed a new transportation system for the city, said
one student describing his answer. its expensive, but it will cut pollution.
1

2
Measuring Skills for the 21st Century www.educationsector.org
arguedforaneducationsystemthatteachesmorethan
justthebasicsofcoreacademicsubjects.Suchcalls,
however,haveintensifedinthepasttwodecadesasthe
natureoftheeconomyandworkhaschanged.Several
majorreportsinthe1990spromptedrenewedattention
tocriticalthinkingineducation.

One,issuedbythe
U.S.DepartmentofLaborSecretarysCommissionon
AchievingNecessarySkills,challengedschoolstoteach
notonlybasicskillsbutalsotheabilitytothinkcreatively
andacquireandanalyzeinformation.
2
Morerecently,theNewCommissionontheSkillsof
theAmericanWorkforceagroupofbusinessleaders,
governors,schoolchancellors,andformersecretariesof
laborandeducationreleasedasequeltoits1990report
onthenationseducationalandeconomicchallenges.
Themessageofthe2006report,Tough Choices or
Tough Times,isclear:Basicskillsarenecessarybutnot
suffcient.
Thecommissionsreportdescribeshownewtechnology
andglobalcompetitionhavechangedthegamefor
Americanworkers.Studentsneedastrongfoundation
ofbasicskills,thecommissionasserts,butthataloneis
nolongerenoughforeconomicandjobsecurity.Itisa
worldinwhichcomfortwithideasandabstractionsisthe
passporttoagoodjob,inwhichcreativityandinnovation
arethekeytothegoodlife,inwhichhighlevelsof
educationaverydifferentkindofeducationthanmostof
ushavehadaregoingtobetheonlysecuritythereis.
3
ThisnewrealityappliestoallchildrenintheUnitedStates,
notjustaneliteclassofstudents.Nearlyeverysegment
oftheworkforcenowrequiresemployeestoknowhowto
domorethansimpleprocedurestheylookforworkers
whocanrecognizewhatkindofinformationmatters,
whyitmatters,andhowitconnectsandappliestoother
information.
RichardMurnaneandFrankLevy,botheconomistsand
professorsatHarvardandMIT,respectively,havebeen
researchingandwritingaboutworkforceskillsformore
thanadecade.Theyagreethatbasicskills,onceinhigh
demandforworkers,arenolongerwhatmattermost.There
arefewertasksrequiringonlyroutineskills,theyexplain,
andtheyareoftendonebycomputers.
4
(SeeFigure1.
ConcernsthattheUnitedStatesislosingitsglobal
competitiveedgeareheightenedbythenations
performanceonthemostrecentinternationaltests.The
ProgrammeforInternationalStudentAssessment(PISA)
andtheTrendsinInternationalMathematicsandScience
Study(TIMSS),twoofthelargesteducationsurveysin
theworld,measurehowwellearlyadolescentstudents
(PISAtests15-year-oldsandTIMSSteststherough
equivalentofeighth-graders)arefaringintheirabilitiesto
problem-solveinmathandscience.
5
TIMSSfoundU.S.
eighth-graderstobeaboveaverageperformersamong
participatingnationsandfoundsubstantialimprovement
inperformance,particularlyinscience,fromthe1999
to2003tests.ButthePISA,designedtoteststudents
applicationofmathandsciencetoreal-worldscenarios,
foundU.S.studentstobeamongtheworstperformers.
(SeeTable1.)Takentogether,theseresultsrevealthat
U.S.studentsmaybeperformingwellintheirmastery
ofinstructionalmaterialbutthatthisperformanceisnot
carryingovertotheapplicationofmaterialtoreal-world
problems.
The Must Have Skills
Itisanemphasisonwhatstudentscandowithknowledge,
ratherthanwhatunitsofknowledgetheyhave,thatbest
describestheessenceof21
st
centuryskills.
Butthatcorenotionisoftenlostinthewelterofterms
usedtodescribe21
st
centuryskillsandinthemany
Figure 1. Sk|||s for a haw Fconomy
1969 1979 1989 1999
14
12
10
8
6
4
2
0
2
4
6
8
P
e
r
c
e
n
t
i
l
e

C
h
a
n
g
e
Non-Routine Manual
Routine Manual Complex Communication
Expert Thinking
Routine Cognitive
Source:FrankLevyandRichardMurnane,The New Division of Labor:
How Computers Are Creating the Next Job Market(Princeton,NJ:
PrincetonUniversityPress,2004).
3
Measuring Skills for the 21st Century www.educationsector.org
sub-skillsoftenincludedundertheconcept.Workforce
andmanagement-traininggroupsoftencall21stcentury
skillssoftorinterpersonalskills.Vocationaleducation
programscallthemappliedskillsorworkforce
skills.Manyyouthdevelopmentprogramsrefertothem
aslifeandcareerskills.Andresearchersoftenuse
thetermnon-cognitiveskills.Technologyliteracy
isafrequentlymentioned21stcenturysub-skill.But
itsdefnedinmyriadways.Tovariouseducational
organizationsandbusinesses,itsinformation-science
skills,digitalmediafuency,advancedcomputerand
internetcommunications,andtechnacy,anewerterm
usedtodescribeadeepknowledgeoftechnological
systems.
Anumberoforganizationshavedevelopedframeworks
thatattempttoidentifytheindividualskillsandsetsof
skillsstudentsneedtosucceedandtohelpeducators
integrate21stcenturyskillsintoexistingeducation
programs.TheenGauge21stCenturySkillsframework,
forinstance,developedbytheNorthCentralRegional
EducationalLaboratory,includesdigital-ageliteracy,
inventivethinking,effectivecommunication,andhigh
productivityasthemostimportantskillsets.
6

ThePartnershipfor21stCenturySkills,anetworkof
nearly30majorbusinessesandeducationgroupsand
oneofthestrongestadvocatesforinfusing21stcentury
skillsintoeducation,hasdevelopedaframeworkfor
21stcenturylearningwiththeintenttohelpstates,
districts,andschoolsintegratecoresubjectlearning
with21stcenturyskills.Similarly,theU.S.Conferenceof
Mayorspassedapolicyresolutionin2005supportinga
frameworkfor21stcenturyskillsthatencouragescitywide
policiesandprogramsaimedatpreparingstudentswitha
morecomprehensivesetofskills.
7
Theframeworkwithpotentiallythewidestreachis
thatoftheDefnitionandSelectionofCompetencies
Project,createdbytheOrganizationforEconomic
CooperationandDevelopment(OECD),anorganization
of30industrializednations.Thisframeworkdescribes
asetofkeycompetenciesforinstance,theabilityto
considerthewidercontextofdecisionsandactions
thatmarrytheneedforbasicliteracywithessential
deepconceptualunderstanding.
8
Thisframework
helpedtodefneOECDslong-termstrategyfor
assessingcompetenciesofyoungpeople,includingits
developmentofthePISA.
A Learning Imperative
Atthesametime,studiesbynationalandinternational
researchorganizations,includingtheNationalResearch
Council,OECD,andtheInternationalSocietyfor
TechnologyinEducation,haveshownthatcomplex
thinkingandanalyticalskillsareanintegralpartoflearning
ateverystageofdevelopment.
9

Fordecades,educatorshavereliedontheprinciplesof
BloomsTaxonomyofEducationalObjectivestooutline
teachingpractices,developcurriculum,andcreatetesting
standards.
10
Thewidelyusedrubricsequenceslearningon
acontinuumfromlower-leveltohigher-levelskillsbased
onthebeliefthatlearningisalinearprocessthatthe
abilitytodevelopaparticularskill,suchasconstructinga
fowchartthatdescribeshowandwhycertainhistorical
eventsledtoothers,isnecessarilyprecededbythe
developmentofanotherparticularskill,suchasrecalling
atimelineofhistoricaldates.Teachershavebeen
trainedtomovestudentsupthroughthiscontinuumof
skills,beginningwiththeacquisitionofknowledgeand
eventuallygettingtoskillslikeanalyticalthinking.
Table 1. P|SA 2003, 15-Yaar-0|ds, Prob|am-So|v|ng
Top 10 nations and the United States
Country PISA Score
Korea 550
Finland 548
Japan 547
NewZealand 533
Australia 530
Canada 529
Belgium 525
Switzerland 521
Netherlands 520
France 519
United States 477
Note: TheOECDaverageis500.Of29OECDcountriesparticipatingin
PISA2003,onlythreecountriesGreece,Turkey,andMexicoscored
belowtheUnitedStates.
Source:Lemke,M.,Sen,A.,Pahlke,E.,Partelow,L.,Miller,D.,Williams,
T.,Kastberg,D.,Jocelyn,L.(2004).InternationalOutcomesofLearningin
MathematicsLiteracyandProblemSolving:PISA2003.ResultsFromthe
U.S.Perspective.(NCES2005003).Washington,DC:U.S.Department
ofEducation,NationalCenterforEducationStatistics.
4
Measuring Skills for the 21st Century www.educationsector.org
*Surveyofpublishedgoalsandoutcomesofafter-schoolandout-of-schoolprograms,including21stCenturyCommunityLearningCentersgrantees,
Spring,2007.
**TheAfterschoolAllianceestimatesthetotalfundingoftheafter-schoolindustrytobe$3.5billion.21stCCLCrepresentsthelargestpotoffundingfor
after-andout-of-schoolprogramming.

InterviewwithHeatherWeiss,July2008.
Withalonghistoryofimpartingskillslikeproblem-solving,
inquiry,andcriticalthinking,andconnectingtheseskillsto
academicgoals,theafter-schoolorout-of-schoolfeldseems
fertileterritoryforfndingwaystomeasure21stcenturyskills.
ButthepushforaccountabilityineducationbywayofNCLB
hasbeenamixedblessingfortheafter-schoolworld.Onthe
onehand,thedemandforresultshascompelledthefeldto
makesignifcantstridesinthedevelopmentofbetterandmore
appropriateevaluationandassessmenttools.TheNational
InstituteonOut-of-SchoolTime,forexample,recentlyworked
withtheMassachusettsDepartmentofEducationtodevelop
theAfterschoolProgramAssessmentSystem(APAS),whichis
designedtohelpafter-schoolprogramsimprovetheirquality
andreachtheirdesiredoutcomes.APASincludesatool,
theSurveyofAfter-schoolYouthOutcomes(SAYO),which
measureschangesinyouthoutcomesovertime.TheSAYO,
frstusedbyMassachusettsin2004andnowexpandingto
GeorgiaandNorthCarolina,usespre-andpost-participation
surveysofteachersandafter-schoolstafftomeasurechanges
inyouthbehaviorsthatarealignedwiththeoutcomegoals
oftheprogram.Whilethetoolisnotintendedtobeusedas
adiagnosticinstrumentforindividualyouth,itenablesthe
collectionofahugeamountofdemographicinformationand
outcomedataandisastepintherightdirectionfortheafter-
andout-of-schoolfeld.
ToolfndisanotherMassachusetts-baseddevelopment.The
UnitedWayofMassachusettsBaydevelopedtheToolfnd
databasein2006toidentifyandsharepsychometricallysound
toolsthatcanbeusedbyaveragepractitioners.Itnowsitsas
adatabaseof46testedtoolsin11outcomesareasincluding
problem-solving,positivebehavior(self-control,cooperation,
conductinschool,responsibility),leadership,learningorientation
(motivation,persistence,studyhabits),andacademicskills.
Ontheotherhand,theafter-schoolfeldisnotresistanttothe
pressuresofschool-basedaccountability.Manyprograms
aligntheirworkandassesstheirsuccessbasedprincipallyon
academicoutcomes.*CitizenSchools,anationalnetworkof
after-schooleducationprogramsformiddleschoolstudents,
providesausefulillustrationofthis.CitizenSchoolswas
designedtoimpartskillslikecriticalthinking,teamwork,and
communication.Buttheyassesstheirperformancestrictlyin
termsofschool-basedacademicmeasures,suchasreading
andmathscores.
LizReisner,principalatPolicyStudiesAssociates,conducted
arecentevaluationofCitizenSchools.Sheexplainsthatwhile
thereisstronginterestwithinCitizenSchoolstoshowhow
theprogramteachesabroadersetofskills,theirworkisstill
drivenbystakeholdersandfunderswhoexpecttoseesuccess
measuredbyacademicgains.Therearewaystomeasure
someoftheseskills,Reisnersays.Forexample,itmightbe
possibletoassessdecision-makingskillsbyanalyzingthe
middleschoolparticipantsselectionofhigh-qualitycollege
preparatoryhighschools.Wereworkingonit,Reisnersays,
butwithoutincentivestomeasuretheseskills,theresnotalot
ofattention[tomeasuringthestatedgoalsofcriticalthinking,
teamwork,andcommunication].
Measuringsuccessbyschool-basedacademicoutcomesis
understandablefortworeasons.First,thecallformeasuresof
impactinafter-schoolprogramsthataremorerigorousthan
commonlyusedsatisfactionsurveysisnotonlyanaddedtask
butadauntingone.Mostprograms,operatingindependently
withvaryingpurposesandgoals,donothavestaffwith
expertise,ortimetodevelopexpertise,inassessmentand
evaluation.Thesimplestapproachfortheseprograms,then,
istoadoptexistingacceptableindicatorsofsuccess.Inmost
cases,school-baseddataonachievement,oftenintheformof
testscores,servethispurpose.
Second,theafter-schoolworldishighlydependentonfunding
fromthe21stCenturyCommunityLearningCenters(CCLC),a
federalfundingstreamofnearly$1billionthattargetsstudents
athigh-povertyandlow-performingschools.**Whenthe
programbeganin1998,saysJenRinehart,vicepresident
forresearchandpolicyattheAfterschoolAlliance,there
wasntmuchofafocusonschoolatall.NCLBchangedthat,
andCCLCmovedfromacommunitylearningcentermodeltoa
focusalmostentirelyonacademicoutcomes,althoughnotably
didntchangeitsnametorefectthisdifference.Thatshada
majorimpactonthefocusofafter-schoolprograms.
Thisshifttoadoptschool-basedacademicgoalsisaproblem
fortheafter-schoolworld,whichhasdefneditselfandits
relevancebyitsabilitytosupportnotjusttheacademic
butalsothesocial,emotional,andphysicaldevelopment
ofchildren.Ineffect,theafter-schoolfeldrisksbeing
relegatedtoalarge-scaletutorialprogram,supportingnarrow
profciencygoalsofschoolattheexpenseofitsbroaderyouth
developmentgoals.
Thepressuresontheafter-schoolfeldtoalignwiththegoals
ofschool-basedaccountabilityaresubstantial.Buttheyalso
presentanopportunityforthefeld.Bycombiningitsdeep
rootsinyouthdevelopmentwithpromisingadvancesin
assessment,theafter-schoolfeldispoisedtoplayapowerful
roleininforminganeducationalassessmentsystemthat
measuresamorecomprehensivesetofskills.Asschoolslook
toteachmorethanthebasics,andtoevaluatetheirsuccessin
doingso,theywillneedmorethansupplementalhelp.Arecent
reportbyHeatherWeissandPriscillaLittle,bothresearchersat
theHarvardFamilyResearchProject,pointstoagrowingbody
ofresearchthatshowsthatqualityafter-schoolprogramscan
improveawiderangeofoutcomesforyouth.Weissconcludes
thatthebestscenarioforstudentsistheonewhereout-of-
schoolprogramsarerecognizedfortherangeoflearningthey
provideandworkwith,notagainst,schoolstoprovidethem.

An Aftar-Schoo| 0pportun|ty
5
Measuring Skills for the 21st Century www.educationsector.org
ButtwoformerstudentsofBenjaminBloom,the
researcherwhodevelopedthetaxonomyintheearly
1950s,publishedanewversionofthetaxonomyin2001
basedonthenewfndingsthatmostskillscanbegained
andemployedsimultaneouslyoroutoforder.Thisis
differentfromtheoldtaxonomy,whichsaid,forexample,
thatyoucannotapplyuntilyoucomprehend,orthat
youmustunderstandbeforeyoucananalyze,explains
co-authorLorinAnderson,whowithDavidKrathwohl
convenedaworkinggroupthatspentfveyearsrevising
Bloomstaxonomy.
11
Wenowknowthat,inmany
instances,theseprocessescanbelearnedatthesame
time,oreveninreverseorder.(SeeFigure2.)
Thenotionthatbasicandadvancedskillsarebestlearned
togetherisoneofthemajorfndingsofarecentreporton
mathematicseducation,fundedandreleasedbytheU.S.
DepartmentofEducation.Thebestlearninghappens,
thereportasserts,whenstudentslearnbasiccontent
andprocesses,suchastherulesandproceduresof
arithmetic,atthesametimethattheylearnhowtothink
andsolveproblems.
Themathematicsreportalsoconcludedthatthereis
nosetageordevelopmentalstagewhenchildrenare
readytogaincomplexthinkingskills.Thisisinsharp
contrasttothepreviouslyheldnotionthatveryyoung
childrenareconcreteandsimplisticthinkerswho
cannotthinkabstractlyorgaindeepunderstanding
ofconcepts.Thus,whiletherearebuildingblocks
ofknowledgestudentsmustmasteradditionand
subtractionbeforetheymultiplyordividetheideathat
studentsshouldbetaughtfactsandsimpleprocedures
beforetheygettoproblem-solvingorcriticalthinking
nolongermakessense.Thecommonideathatwecan
teachthinkingwithoutasolidfoundationofknowledge
mustbeabandoned.Somusttheideathatwecan
teachknowledgewithoutengagingstudentsinthinking.
Knowledgeandthinkingmustbeintimatelyjoined,
saysLaurenResnick,aprofessorofpsychologyatthe
UniversityofPittsburghandaleadingexpertoncognitive
science.
12

Teachingchildrenbasicfactsandsimpleproceduresina
waythathelpsthemalsolearnhowtoapplyandusethis
knowledgeandtheseskillsmirrorsthenaturalprocess
oflearning.Sotheintegrationofadvancedthinkingand
analyticalskillsintoteachingandlearningmakesiteasier
forstudentstoacquireeventhemostbasicskillsandcore
knowledge.
Thebeliefthatthereshouldbeasolid,specifc,and
sharedcorecurriculum,anideaadvancedmostnotably
bythenonproftCoreKnowledgeFoundation,founded
andledbyformerprofessorandliteraryscholar,E.D.
HirschJr.,isnotatoddswiththisapproach.TheCore
Knowledgecurriculumsupportsthepointthatlearning
factualknowledgeandtheabilitytoapply,analyze,and
solveproblemsgohand-in-hand.TeachersusingtheCore
Knowledgeapproachdonotstressrotememorizationof
facts;theyuseanarrayofstrategiesincludingworkshops,
researchprojects,dramatizations,andcollaborative
learninggroupsbecausetheyknowthatstudentswill
learnbestiftheyareexposedtobothsubjectknowledge
andwaystoapplythisknowledgeatthesametime.
Theattributesthatbusinessandhighereducationleaders
arecallingforinyoungpeoplethattheybeindependent
thinkers,problem-solvers,anddecision-makersare
capturedbytheadvancedskillsintherevisedBlooms
taxonomy,theabilitytoanalyze,evaluate,andcreate.
13
Whatisuniquetocreate,evaluate,andanalyzeis
thatthecontentisnotexplicitintheprocessorproduct
beingcreated,writeAndersonandhiscolleagues.Thisis
particularlytrueforcreate,theyexplain,whichrequires
astudenttouseexistinginformationtocomeupwith
somethingentirelyoriginalanewidea,auniqueproduct,
analternativesolutiontiedtoaspecifcpurpose.
14

Withouttheseprocesses,Andersonsays,people,
Figure 2. A haw Taxonomy
Bloom Revised Bloom
Create Evaluation
Synthesis Evaluate
Analysis Analyze
Application Apply
Comprehension Understand
Knowledge Remember
Source:ATaxonomyforLearning,TeachingandAssessing:ARevisionof
BloomsTaxonomyofEducationalObjectives,2001.

Measuring Skills for the 21st Century www.educationsector.org

whenfacedwithaproblemorachallenge,willeithercall
someoneforhelporjustquit.
Integrating21stcenturyskillsintoteachingand
assessment,then,isnotonlyaneconomicimperative,
drivenbychangesintheworkforce,butavitalaspectof
improvingstudentlearning.
Creative Measures
Yet,thereremainsanassumptionthat21stcenturyskills
cannotbefairlyorreliablymeasured.Mostexisting
testsmeasureonlywhetherastudentpossessesa
particularpieceofknowledge,notwhetherthestudent
cananalyzethisinformation,evaluateitsutility,orcreate
newknowledgefromitthecoreof21stcenturyskills.
Butnewmodelsofassessmentthatmeasurebothbasic
skillsandmoreadvancedskillsareemergingtochallenge
theassumptionthatsuchskillscannotbemeasuredand
tomoveustowardanassessmentsystemthatismore
alignedwithwhatstudentsnowneedtoknow.
TheCWRA,usedbySt.AndrewsSchoolinDelaware,
offersoneexample.Itconsistsofasingle90-minutetask
thatstudentsmustrespondtousingalibraryofonline
documents,fromone-pagenewspapereditorialsto
20-pageresearchreports.Facingproblemslikeacitybeset
bypollutionfromanow-defunctfactoryoracommunity
healthclinicstrugglingtoserveagrowingimmigrant
population,studentsmustgrapplewithreal-world
dilemmas;makejudgmentsthathaveeconomic,social,
andenvironmentalimplications;andarticulateasolutionin
writing.
TheCWRAgrewoutoftheCollegiateLearning
Assessment(CLA),developedbytheCouncilforAidto
EducationandtheRANDCorporation.(SeeLearning
FromHigherEducationsidebaronPage7.)Likethe
CWRA,theCLAisasingletestthatmeasuresanalysis
andwritingskills.But,whiletheCLAisusedbymorethan
175highereducationinstitutions,theCWRAisinuseby
onlyahandfulofprivateschools(likeSt.Andrews,which
beganusingitafewyearsago)andasingleLongIsland,
N.Y.,publicschool.
TheCWRAisintendedasatoolforschoolimprovement,
notnecessarilytomeasureindividualstudentgains.But
thosewhouseitaffrmitsvalueasanessentialmetric
forstudentlearning:Areweteachingourstudentsto
thinkintelligentlyandcritically,todomorethanjustfollow
orevenlead,buttofndnewpathstogodown?Thats
whatwelearnfrom[theCWRA],saysJohnAustin,the
academicdeanofSt.Andrews.
15

Newtechnologiesaremakingiteasiertomeasure
individualstudentmasteryof21stcenturyskills.River
City,forexample,isavirtualworldthatsimultaneously
teachesandassessesmiddleschoolsciencestudents.
Likeothersimulatedlearningprogramsineducationand
arangeofotherindustries,RiverCitypresentsstudents
withaproblemandasksthemtodevelopahypothesis
andprocedure,testitvirtuallyandthendescribetheir
fndingsandmakerecommendationsinareport.
16

Fromatechnicalstandpoint,thesemulti-uservirtual
environmenttoolsareamongthemostadvanced
performanceassessmentsthatnowexist.Theycankeep
detailedrecordsofthemoment-by-momentmovements
anddecisionsofeachparticipantintheenvironmentand
providealogforeachstudentineachsession.Teachers,
then,cantracktheprogressofindividualstudents.
ProgramslikeRiverCityaregoodfortrackingstudent
gainsattheclassroomorschoollevel,butthetruetestfor
accountabilitywillbeifassessmentslikethiscanworkon
alargerscale.Severalpromisingexamplesmovetoward
thisgoal,takingthenecessarystepoflinkingthesetypes
ofassessmentstoexistingstateornationalstandards.
ResearchersattheCenterforResearchonEvaluation,
Standards,andStudentTesting(CRESST)atthe
UniversityofCalifornia-LosAngeleshavedeveloped
anassessmentthatmeasurescomplexthinkingand
judgmentskillswithintheexistingframeworkofstate
mathassessments.Thesystem,calledPowerSource,
isfundedbyaU.S.DepartmentofEducationInstitute
ofEducationSciencesgrantandisnowbeingpiloted
innearly70schools.Designedformiddleschoolpre-
Algebra,itconsistsofmultipleinterimassessmentsthat
areformattedasnarrativethemesorgraphicnovels.Its
stillanexperiment,explainsEvaBaker,whodirectsthe
center.Butithasrealpromiseforimprovinginstruction
andfordemonstratingmasteryofabroadsetofskills.
17
PowerSourcemeasuresadvancedskillsinthecontext
ofmeasuringcontentprofciency,whichmeansitcan
demonstratestudentlearningforspecifcsubjectmatter
7
Measuring Skills for the 21st Century www.educationsector.org
whilealsotestingstudentsdevelopmentofhigher-
orderskills.Studentsareasked,forexample,toapply
algebraicprinciplesaswellasexplainwhytheychose
theprinciples.PowerSourcewasdesignedtomeasurea
broadersetofoutcomesbyfocusingonahandfulofbig
ideasratherthanaheapofdiscretefacts.
18
TheUnitedKingdomrecentlydevelopedaninnovative
nationalassessmentthatalignswithitsexistingnational
standards.TheKeyStage3(ages1213)Information
CommunicationsTechnology(ICT)LiteracyAssessment,
createdbytheBritishgovernmentsQualifcationsand
CurriculumAuthority,measuresasetoftechnicalskills
aswellasastudentsabilitytousethoseskillstosolve
asetofcomplexproblems.Studentsareprovideda
toolkitofapplicationstousetocompletetasksthat
measurelearningskillssuchasfndingthingsout,
developingideas,andexchangingandsharing
information.Studentactionsaretrackedandmapped
againstexpectedabilitiesforthatlevelofeducationand
testresultsprovidebothnationalscoresforstudentsand
detailedfeedbackaboutstudentperformancethatcan
beusedtoinformteachingandlearningattheclassroom
level.
19
TheclosestthingintheUnitedStatesisthe2009NAEP
ScienceAssessment,administeredbythefederally
fundedNationalAssessmentofEducationalProgress
(NAEP).Thetestwillforthefrsttimemeasurenotjust
studentsknowledgeofscienceprinciples,butalso
*AssociationofAmericanCollegesandUniversities(AAC&U),Greater Expectations: A New Vision for Learning as a Nation Goes to College,2002.
**U.S.DepartmentofEducation,Washington,D.C.,2006.http://www.ed.gov/about/bdscomm/list/hiedfuture/index.html.
***PersonalinterviewwithRichardShavelson,March,2008.

InterviewandcorrespondencewithRobertSternberg,February2008.

CorrespondencewithTuftsUniversityAdmissionsOffce,March,2008.
Collegesanduniversities,whilefreefromtheaccountability
pressuresfacedbyK12schoolsandafter-schoolinstitutions,
havegoodreasontofocuson21stcenturyskills.
TheAmericanAssociationofCollegesandUniversities
stressedinits2002report,Greater Expectations: A New Vision
for Learning as a Nation Goes to College,thatthecurrent
emphasisonfactualrecallisamajorbarriertosuccess
incollege.*Todayscollegestudents,thereportconcludes,
needtobeintegrativethinkerswhocanseeconnectionsin
seeminglydisparateinformationanddrawonawiderangeof
knowledgetomakedecisions.Collegesanduniversitiesare
thuslyfocusedonensuringthattheirstudentsandprospective
studentshaveasetofstrongcreativeandanalyticalskills.
Newmodelsforassessingtheseskillshaveemergedinthe
lastfewyears.TheCollegiateLearningAssessment(CLA),the
parenttestoftheCWRA,wasrecentlyhighlightedbytheU.S.
SecretaryofEducationsCommissionontheFutureofHigher
Educationasapromisingapproachfortheassessmentof
studentlearningatcollegesanduniversities.**Developedfor
theCouncilforAidtoEducationbyformerRandresearchers,
StevenKleinandRichardShavelson,theCLAiscomprised
ofasingletestthatmeasureswritingskillsandanalysisand
problem-solvingskillsatthesametime.Itisnowinuseby
morethan175highereducationinstitutions.
Wearetestingabroaderrangeofoutcomesforcollege
studentsthatarerelevanttotheirfutureasworkersand
citizens,saysShavelson.Whatweultimatelywantisfor
peopletobehaveintelligentlytothinkcritically,reason
analytically,andmakedecisionsthatarejustifable.***
RobertSternberg,deanoftheSchoolofArtsandSciences
andpsychologyprofessoratTuftsUniversity,hadasimilar
goalinmindtomeasuremorerelevantskillswhenhe
developedaseriesofalternativeassessmentsdesignedto
measurestudentcreativity,practicality,andwisdom.The
tests,administeredinwrittenformandbyvideo,askstudents
towritestories,formcaptionsforuntitledcartoons,and
solveeverydayproblemssuchasmovingalargebedupa
windingstaircase.Sternbergadministeredhisteststomore
than1,000collegefreshmenandhighschoolseniorsfrom
15schoolsaspartofwhathenamedtheRainbowProject.
Hefoundthathistestspredictedstudentgradesascollege
freshmentwiceaswellasSATscoresandhighschoolgrade
pointaverages,andthatthetestsreducedethnic-group
differenceinscorescomparedtotheSAT.

Onthe200607applicationforTuftsundergraduates,
slightlymorethanhalftheapplicantsorabout8,000chose
toparticipateintheassessment.

Sternbergsaidthat,as
predictedbytheearlierRainbowProject,Tuftsadmittedbetter-
qualifedapplicantsasdefnedbythetraditionalmeasures
ofSATscoreandgrade-pointaverageandalsoincreased
diversity,admitting30percentmoreblacksand15percent
moreHispanicsthaninthepreviousyear.
Sternbergencouragesothercollegesandhighschoolsto
assessthisbroadersetofskillsandfreelyshareshismaterials.
TheselectiveindependenthighschoolsChoateRosemary
HallinWallingford,Conn.,andPhillipsAcademyAndoverin
Massachusettsareusingsimilarmeasuresintheiradmissions
processes.Sternberglamentsthatmanyofthehighschools
makingeffortstoteachandassesstheseskillseducatean
alreadyadvancedstudentbody,whiletheprimaryfocuswith
underservedstudentsisonbasicprofciency.Thosestudents
aregoingtobetheonesmostlikelytocapitalizeontheir
practicalskills,hesaid.
Laarn|ng From H|ghar Fducat|on
8
Measuring Skills for the 21st Century www.educationsector.org
whethertheycanapplytheirknowledge.Thisisabig
changeforNAEPandrepresentsapotentialmovetoward
nationalassessmentsthatprovidearicherpictureof
studentmasteryofsciencecontentandthescientifc
inquiryprocess.
TheInternationalBaccalaureate(IB)DiplomaProgramme,
arigoroustwo-yearhighschoolcourseofstudytaught
inmorethan2,000publicandprivateschoolsin130
countriesundertheauspicesofthenonproftInternational
BaccalaureateOrganization,servesasevidencethatthe
assessmentofcorecontentandadvancedskills,aligned
withaprogramofstandardsandcurriculum,canhappen
atalarge,eveninternational,scale.The40-year-old
programisbuiltontheprinciplethatstudentscanand
shouldmasterbothbasicsubjectmatterandhigher-order
skills.Theprogramhasdevelopedcommoncurricula,
standards,andassessments,whichareusedthroughout
itsschoolnetworkandhasdevelopedstrategiesfor
ensuringthestandardizationofbothteachingand
teacher-gradedtesting.
20

Theprogramassessesstudentperformanceusinga
rangeoftechniques,bothinternal(classroom-based,
teacher-led)andexternal.Allcourses,forexample,have
threeorfourseparateassessmentcomponents,none
worthlessthan20percentormorethan50percentofthe
overallassessment.Eachcomponentincludesarange
ofperformancetasksinvariousformatsappropriateto
thesubjectmatter,whichcouldincludemultiple-choice
questions,short-responsequestions,structuredand
open-endedproblem-solvingquestions,dataanalysis
questions,casestudies,andessayquestions.
Thetestingindustry,keenlyawarethatthecallfor21st
centuryskillsmeansmoredemandforteststhatmeasure
theseskills,hasalsobeenworkinghardtodevelop
assessmentsthatmeasuremorethanthebasics.
TheEducationalTestingService(ETS),aprivate
nonproftorganizationandoneoftheworldslargest
testdevelopers,hasseveralinitiativesunderway
tomeasureabroadersetofskills,includingcritical
thinking,communication,andahostofsocio-emotional
skillslikeadaptabilityandagreeableness.Researchers
attheorganizationsCenterforNewConstructsare
studyinghowtoassessskillsthatarenotmeasuredby
theSAT,ACT,andothertraditionalstandardizedtests.
Pilotprojectsofmorethan20individualassessments
thatmeasureanalyticalandahostofotherskillslike
negotiationandteamworkaretakingplaceoncollege
campusesandinseveralschooldistricts.
21
RichardRoberts,oneofthecentersprincipalresearch
scientists,saysthatthesetests,bydescribingaproblem-
solvingscenariosuchasdecidingwhattodowhena
groupprojectthatneedsweeksofworkisdueindays,
challengestudentstothinkinwaysthatmosttestsdo
not.Wearenotjustaskingstudentswhatshouldyou
dointhissituation?butalsowhatwouldyoudointhis
situation?Studentshavetothinkdifferentlytheyhave
tothinkdeepertoanswerthesecondquestionbecause
thereissimplynorightanswer,explainsRoberts.
22

Cost, Time, Quality, and More

Measuring21stcenturyskillsonalargescaleisnot
goingtobeaneasytask.Forone,assessmentslike
theCWRAareexpensive.TheGovernmentAccounting
Offceestimatedina2003reportthatthecosttoscore
NorthCarolinasmultiplechoice,machine-scored
assessmentswasapproximately60centspertest.
23
The
costforMassachusetts,withitscombinationofmultiple-
choiceandopen-endedquestions,wasapproximately
$7pertest.Incontrast,thecosttoscoretheCWRAs
performancetaskismorethan$40pertest,although
thisisstillasmallportionoftheroughly$8000spenton
educationperstudent.Eventhisisconsideredbythose
inthetestingindustrytobeinexpensivecomparedto
thecostoflarge-scaleperformance-basedassessments
requiringhumanobservationorscoring.
Usingpeopletogradeawiderangeofopen-endedand
performance-basedassessmentsof21
st
centuryskills
raisesconcernsaboutthereliabilityofresults.People
maybeabletoassessinmoredepthandwithmore
nuancethanacomputerprogram,buthumanscorers
inevitablyintroducealevelofsubjectivityintothe
assessmentprocess.So-calledinter-raterreliabilityis
achallenge;nomatterhowclearscoringstandardsare
itsdiffculttoexpecthumanscorerstogradetestswith
perfectconsistency.Trainingandmonitoringscorerscan
betime-consumingandcostly,butitcanhelp.TheIB
DiplomaProgramme,forinstance,hasnearly5,000test
examinersworldwide.Theprogramensuresahighlevel
ofconsistencyamong itsexaminers,mostofwhomare
experiencedDiplomaProgrammeteachers,byproviding
9
Measuring Skills for the 21st Century www.educationsector.org
detailedinstructionsonscoringforeachassignment,
requiringeachexaminertosubmitsamplesoftheir
scoring,andemployingacadreofseniorexaminerswho
determinescoringstandardsandmonitorexaminerwork.
Thecostandtimerequiredtowrite,administer,andscore
open-endedquestionshasledmanystatepolicymakers
toincreasethenumberofmultiple-choicequestionsin
testing.AsThomasTochhaswrittenintheEducation
SectorreportMargins of Error: The Testing Industry in
the No Child Left Behind Era,thepercentageofpublic
schoolstudentstakingstatewidetestswithnoopen-
endedquestionsreached42percenttwoyearsago,
afgurethathascontinuedtoclimb.
24
Earlierthisyear,
statepolicymakersinKentuckylookedtosaveasmuch
as$10millionbyremovingopen-responsequestions
fromthestatesaccountabilitysystem.Isthisopen-
responsetestworththe[moneyand]sixtoeightweeksof
resourceswhenwecouldbespendingtimeontaskand
thenbegettingmorevaluableinformationfromthenorm-
referencedtest?askedstateSenateMajorityLeaderDan
Kelly,wholedthestatespushtoeliminateportionsof
theassessmentsystem.
25
Kellysquestionisafairone,
giventhescopeofstatewideassessment.Givingtests
withfewer,lessstandardizeditemsinvariablyreducesthe
reliabilityofresults.
26

Theresearchcommunityisgrapplingwithanother
reliabilityquestion:whether21
st
centuryskillscanbe
coachedorfakedonatest.Astudent,forexample,
mightanswerinwaysthatsuggestsheisananalytical
thinkerwheninfactsheismerelydemonstratingthat
shehaslearnedwhattypesofanswersmakeherseem
thatway.Thispotentialproblemisafocalpointof
researchonmeasurementandtesting.TheCenterfor
NewConstructsatETS,forexample,sponsoredanentire
conferencein2006ontheissueoffaking.
27
Withgeneral
agreementthatfakingcanhappenandthatitcanskew
results,participantsdiscussedtheutilityofnewtypes
ofquestions,includingmoresubtlequestionsormore
complexforcedchoicequestions,whichcandetectand
correctfaking.
Advancementsinassessmenttechnologycananswer
manyoftheseconcerns.Simulation-basedassessments,
forexample,likeRiverCity,areabletoassessstudents
understandingofcomplexproblemsusingmultiple-choice
formatsthatareautomaticallyscored,makingthemnot
onlycheaperandmoreeffcient,butalsomorereliable.
A 0arta|n Typa of Taachar?
TheNewTechnologyHighSchoolmodelwasfoundedin1996
whenlocalbusinessesinNapa,Calif.,begancomplaining
thatstudentswereunpreparedforhigh-techjobs.Themodel
incorporatesproject-basedlearning,smallschoolsize,one
computerforeverystudent,andanenvironmentinwhich
studentsareresponsiblefortheirownlearning.Withinthree
yearsoftheschoolsfounding,theBillandMelindaGates
Foundationsteppedintohelpreplicatethemodel,which
costsapproximately$800moreperstudentperyearthana
traditionalschool.Therearenow40NewTechnologyschools
inninestates;fvearecharterschools,and21aresmall
schoolswithinalargerhighschool.
StudentlearningatNewTechnologyisdesignedtosimulatereal
lifeandrealwork.Insteadofcompletingtraditionalworksheets
anddailyassignments,studentsareassignedperiodicprojects,
oftenasteams,andmustcompleteacombinationofproducts,
includingwrittenessaysandpracticaldemonstrations.Each
projectassignedtostudentsisaccompaniedbyasetofrubrics
thatmeasureastudentsperformanceonfundamentalskills,like
writing,aswellascriteriasuchascriticalthinking,application,
andoriginality.Studentsreceivemultiplegrades,oneforeach
criterion,foreachproject.
Inastandardschool,explainsaworldstudiesteacherat
theNapa-basedNewTechnologyHigh,youwouldreada
paperandsay,Wow.Thisstudentisnotastrongwriterbut
hassomegoodideas.Buttheresnowaytocommunicate
thatinthegrade.Here,IcouldgivethemalowCforwritten
communications,butahighergradeinanotherarea.SoIknow
andthestudentknowswhatareasweneedtoreallywork
on.
Thistypeofproject-basedlearningismadeeasierbya
suiteofWeb-basedtoolsusedtotrackandimprovestudent
performance.Teachers,students,andparentshaveconstant
accesstoanonlinesystemthatprovidesdetailedinformation
abouthowstudentsareprogressingandhowtheycan
improve.
Teachersacknowledgethatitcanbeachallengetoteachthis
way.Wetakeitforgrantedthatstudentsknowhowtolearn,
buttherealityismostofthemdont,saidaSpanishteacher.
Itsourjobtoteachthemhowtolearnbutitsnotalways
easy,particularlywiththestudentswhostrugglewithpoor
timemanagementandinitiative.Itrequiresalotofworkand
perhapsacertaintypeofteacher.Arecentsurveyofteachers
conductedbytheBuckInstituteforEducationcompared
teachersworkinginnewerreformmodelsusingproject-based
learning,withteachersworkingintraditionalhighschoolsusing
conventionalinstructionalapproaches.Teachersinthereform
modelschools,includingasampleof71NewTechnology
teachers,tendedtobenewerandyoungerteacherswithmore
traininginandexposuretoproject-basedlearningandthe
technologiesnecessarytosupportthistypeofinstruction.
NewTechnologyisexpandingquickly(itplanstoopen10-12
additionalschoolsnextyear).ButtotakemodelslikeNew
Technologytrulytoscaleinpubliceducationisnosmall
task.Itmeansfndingteacherswhoarepreparedtousenew
methodsandnewtoolstoteachnotonlythehigh-performing
studentswhocomereadyforchallengingreal-worldprojects
butalsolow-performingstudents.
10
Measuring Skills for the 21st Century www.educationsector.org
Andcomputer-adaptivetests,whichadjustthediffculty
ofquestionsbasedonstudentsperformanceonprevious
questions,cannotonlybescoredimmediatelybutalso
makefakingfarmorediffcultsincethetestchangeswith
theindividualtest-taker.Delaware,oneofseveralstates
experimentingwithcomputer-adaptivetests,recently
completedapilotof30,000studentsinfourdistricts,
andfoundthattheadaptivetestswereactuallybetterat
identifyingstudentgrowththanexistinggrade-leveltests.
28
Still,betterassessmentisonlyonepartofdelivering
betterlearningtoallstudents.Thebasicprinciplethat
thereisnorealchoicebetweenbasicand21
st
century
skillsthatbothareessentiallearningoutcomesfor
studentsmustalsoapplytostandardsandcurriculum.
Evenmoreimportant,deliveringbetterlearninghingeson
preparingandsupportingqualityteacherswhocandeliver
themusthavecombinationofbasicandadvanced
learningtoallstudents.(SeeACertainTypeofTeacher?
sidebaronPage9.)
Inthelongrun,newformsofassessment,aswellasother
yet-to-be-developedmeasures,willbecriticalformaking
assessmenteffectivebothforeducationalpurposesto
ensurethatteachersandstudentscanmonitorand
improvethelearningprocessandforaccountability
purposestoensurethatschoolsaregivingall
studentswhattheyneedtosucceed.Thiswillrequire
alargerinvestmentinthedevelopmentanddesignof
assessmentsandassessmentsystems.Itwillalsomean
morecoordinationbetweenpolicymakers,educators,
researchers,andtestdevelopers,whotoooftenworkin
isolationofoneanother.
11
Measuring Skills for the 21st Century www.educationsector.org
Endnotes
1
In-personstudentinterviewsatSt.AndrewsSchool,March
2008.
2
SCANSreports,availableonlineathttp://wdr.doleta.gov/
SCANS/whatwork/.
3
TheNewCommissionontheSkillsoftheAmericanWorkforce,
Tough Choices or Tough Times(Washington,DC:National
CenteronEducationandtheEconomy,2006).
4
RichardMurnaneandFrankLevy,The New Division of Labor:
How Computers Are Creating the Next Job Market(Princeton,
NJ:PrincetonUniversityPress,2004).
5
TrendsinInternationalMathematicsandScienceStudy
(TIMSS)2003;TheProgrammeforInternationalStudent
Assessment(PISA)2003.
6
EdCoughlin,CherylLemke,etal.enGauge 21st Century Skills:
Literacy in the Digital Age(LosAngeles,CA:TheMetiriGroup,
2003).
7
Framework for 21st Century Learning(Tucson,AZ:The
Partnershipfor21stCenturySkills,July23,2007).
8
Defnition and Selection of Key Competencies(Paris,France:
OECD,May27,2005).
9
J.D.Bransford,A.L.Brown,andR.R.Cocking(eds),How
People Learn,2000;OECD,Innovation in the Knowledge
Economy: Implications for Education and Learning,2004;and
R.Kozma,Technology, Innovation, and Educational Change:
A Global Perspective(Eugene,OR:InternationalSocietyfor
TechnologyinEducation,2003).
10
BenjaminS.BloomandDavidR.Krathwohl,Taxonomy of
Educational Objectives: The Classifcation of Educational
Goals, by a committee of college and university examiners.
Handbook I: Cognitive Domain(NewYork:Longman,Green,
1956).AlsoD.Krathwohl,B.S.Bloom,andB.B.Masia,
Taxonomy of Educational Objectives: The Classifcation of
Educational Goals. Handbook II: Affective Domain(NewYork:
DavidMcKayCo.,Inc.,1964).
11
L.W.Anderson,D.R.Krathwohl,eds,A Taxonomy for
Learning, Teaching, and Assessing: A Revision of Blooms
Taxonomy of Educational Objectives(NewYork:Allyn-Bacon
Longman,2001).
12
L.B.Resnick,MakingAmericaSmarter,Education Week
Century Series18(40)(June16,1999),3840.
13
Seediagram,Figure2.
14
L.W.Anderson,D.R.Krathwohl,eds,A Taxonomy for
Learning, Teaching, and Assessing: A Revision of Blooms
Taxonomy of Educational Objectives.
15
InterviewwithJohnAustin,March2008.
16
D.J.Ketelhut,Theimpactofstudentself-effcacyon
scientifcinquiryskills:AnexploratoryinvestigationinRiver
City,amulti-uservirtualenvironment,The Journal of Science
Education and Technology16no.1(2007):99111.Formore
informationaboutRiverCity,alsoseehttp://muve.gse.harvard.
edu/rivercityproject/.
17
InterviewwithEvaBaker,May2008.
18
EvaBaker,Moving to the Next Generation System Design:
Integrating Cognition, Assessment, and Learning(LosAngeles,
CA:NationalCenterforResearchonEvaluation,Standards,
andStudentTesting(CRESST),UniversityofCalifornia,
February,2007).
19
KeyStage3ICTAssessmentTasks,NationalAssessment
Agency,UnitedKingdom,availableonlineathttp://www.naa.
org.uk/naa_18908.aspx.
20
InternationalBaccalaureateOrganization.Diploma
ProgrammeAssessmentPrinciplesandPractice.(United
Kingdom:AntonyRoweLtd.,September2004).
21
ETSusesthetermnon-cognitivebecauseitiswidelyused
inpsychologyandmeasurement.Itis,however,amisnomer
fordescribingtheseskills,asmanyareactuallycognitivein
nature.
22
InterviewwithRichardRoberts,March2008.
23
TitleI:Characteristicsoftestswillinfuenceexpenses;
informationsharingmayhelpstatesrealizeeffciencies.
ReporttoCongressionalrequesters.(Washington,DC:
GovernmentAccountingOffce,May,2003).
24
ThomasToch,Margins of Error: The Testing Industry in the
No Child Left Behind Era(Washington,DC:EducationSector,
2006).
25
LindaJacobson,KentuckyLawmakersTakeAimatState
Tests,Education Week,March,2008.
26
AbillwassubmittedduringtheSpring2008sessionthatcalled
forthereplacementoftheKYCATSsystemwithanorm-
referencedtestandanaugmentedtest.Itdidnotmoveout
ofcommittee.Accordingtothestatesoffceofaccountability
andassessment,thestatehasestablishedataskforceon
assessmenttoconsiderfuturechangestostatetests.
27
TheTechnicalAdvisoryCommitteeonFakingon
NonCognitiveAssessments,October13,2006.Sponsoredby
theEducationalTestingService.
28
TheProposedDelawareComprehensiveAssessmentSystem
developedbytheDelawareStatewideAcademicGrowth
AssessmentPilot,Spring2008,availableonlineathttp://www.
mapuser.k12.de.us/future/index.html#delaware.