Heckman, Ichimura, Smith, Todd - Characterizing Selection Bias Using Experimental Data (1998)

Embed Doc
Copy Link
Readcast
Collections

CommentGo Back

Characterizing Selection Bias Using Experimental DataAuthor(s): James Heckman, Hidehiko Ichimura, Jeffrey Smith, Petra ToddSource:

Econometrica,

Vol. 66, No. 5 (Sep., 1998), pp. 1017-1098Published by: The Econometric SocietyStable URL:

http://www.jstor.org/stable/2999630

Accessed: 13/01/2009 10:54

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available athttp://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unlessyou have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and youmay use content in the JSTOR archive only for your personal, non-commercial use.Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained athttp://www.jstor.org/action/showPublisher?publisherCode=econosoc.Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printedpage of such transmission.JSTOR is a not-for-profit organization founded in 1995 to build trusted digital archives for scholarship. We work with thescholarly community to preserve their work and the materials they rely upon, and to build a common research platform thatpromotes the discovery and use of these resources. For more information about JSTOR, please contact support@jstor.org.

The Econometric Society

is collaborating with JSTOR to digitize, preserve and extend access to

Econometrica.

http://www.jstor.org

Econometrica, Vol. 66, No. 5(September, 1998), 1017-1098

CHARACTERIZING SELECTION BIASUSINGEXPERIMENTAL DATA1

BY JAMES HECKMAN, HIDEHIKO ICHIMURA,JEFFREYSMITH,ANDPETRATODDSemiparametric methods are developedto estimate the bias that arises from usingnonexperimental comparison groups toevaluate social programs and to test the identify-ing assumptions that justify matching,selection models, and the method of difference-in-differences. Using data from anexperiment on a prototypical social program and datafromnonexperimental comparison groups,we reject the assumptions justifying matchingand ourextensionsofit. The evidencesupports the selection bias model and theassumptions that justify a semiparametric versionof the method of difference-in-dif-ferences.We extend our analysistoconsiderapplicationsof the methods toordinaryobservational data.

KEYWORDS:

Selection bias, programevaluation, training programs, semiparametricestimation.

1.

INTRODUCTIONASTANDARD METHODFOR EVALUATING socialprograms uses the outcomesof

nonparticipantstoestimatewhatparticipants would have experienced had theynotparticipated.Thedifferencebetween participant and nonparticipant out-comesisthe estimatedgross impactofaprogram reportedinmany evaluations.The outcomes ofnonparticipants maydiffersystematicallyfrom what theoutcomesofparticipants would havebeenwithout the program, producingselection biasinestimatedimpacts.Avarietyofnonexperimentalestimators

1Apreviousversion ofthis paper appeared under the title"Nonparametric CharacterizationofSelectionBiasUsing ExperimentalData: AStudyofAdultMalesinJTPA.PartI.Definitions,ApplicationsandEmpiricalResults."AnearlierversionofitappearedinAugust, 1994,underthetitle"EvaluatingtheImpactofTrainingontheEarningsand LaborForceStatus ofYoungWomen:Better DataHelpA Lot."This researchwas supportedbyNSFSBR91-11-455,NSFSBR93-21-048andbyagrantfrom theRussellSageFoundation. Thispaperwaspresentedas aninvitedlecture atthe Latin AmericanEconometricSocietyMeeting, Caracas, Venezuela, August1994. Wehavebenefited fromcomments received fromworkshopsinSeptember, October,and November 1994 atYale, Princeton, Chicago,UC-SanDiego, USC,Rand-UCLA, UC-Irvine, UC-Riverside,Northwest-ern,and U.C.London,andworkshopsinthe Winter andSpringof1995atUC-Berkeley, Oslo,Washington-St. Louis,TelAviv,andVirginia and anNSF-sponsoredconference oneconometricsheldinMadison,Wisconsin inJune 1995.We alsopresentedthispaperin the MalinvaudWorkshopinParis, March,1995. Weare gratefultothreeanonymous referees, a co-editor, Derek Bandler,LarsHansen,BoHonor6,LanceLochner,Thierry Magnac, Christopher Taber,EdVytlacilandAdonisYatchew forhelpful comments and DerekBandler, Jingjing Hsee, Lance Lochner, andAnnieZhangforprogrammingassistance.1017

1018

J. HECKMAN, H. ICHIMURA, J. SMITH, AND P. TODD

adjust for this selection bias under different assumptions.2 Under certainconditions, randomized social experiments eliminate this bias.3Social experiments are costly and the identifying assumptions required tojustify themare notalwayssatisfied.4However,it iswidelyheldthat thereis no validalternative to experimentationasa methodforevaluating social programs(see, e.g., Burtless, 1995).In animportant paper,LaLonde(1986)combines datafrom a socialexperiment with datafromnonexperimental comparison groupsto evaluatetheperformanceofmany commonly-used nonexperimentalestimators.For theparticular groupofparametric estimators thatheinvestigates,and for hisparticularchoicesofregressors,hefindsthat the estimators chosenbyeconometric model selection criteriaproducearangeofimpactestimates that isunacceptably large.This paper uses datafroma social experimenton aprototypical socialprogram combined with dataoncomparison groupsofpersons who chose nottoparticipateintheprogramevaluatedbytheexperiment.As documentedbyHeckman, LaLonde,and Smith(1999), many programsinplacearound theworld areverysimilar to theprogramweanalyzeinthispaper.Ouranalysisis based onthefollowing principles.Neithertheexperimentalcontrol groupnor thecomparison group we analyze receives treatment,sothatdifferencesinmeasured outcomes between the twogroupscan beattributedsolelyto selection bias. Instead ofexamining the performanceofspecificparametricestimators based onspecificsetsofregressorsineliminating selec-tionbias,as LaLonde(1986)and scholars who follow him havedone, we usesemiparametriceconometricmethodsto estimatethe functionalform oftheselection biasdirectly usingavarietyofregressorsand datasets.Weusetheestimatedbias functionstotestidentifying assumptionsthat have beenmain-tainedintheliterature,and tosuggestestimators thatmightbe effectiveineliminatingselection biasinfuture evaluationsofsimilarprograms.Ourmethodforcharacterizingbiasisgeneraland can beappliedin avarietyofsettings,includingthestudyoftheanalyticallysimilarproblemofsampleattrition. By characterizingthe biasnonparametrically, and by examining the sensitivityofthe estimatedbiastomanyalternative sets ofconditioning variables,we analyzethesuitabilityofentire classes ofestimators,rather thantryingout afewparametricmembers ofthoseclasses with alimited set ofconditioningvariables. Evidence thataparticularestimatorwith aparticularsetofregressors"works" in aparticulardata set isproperlydiscountedbymost seriousanalysts.There isalwaysthesuspicionthat the success of anestimatorinaparticularinstanceistheconsequenceofadiligent specificationsearch. We avoid that

2

These estimators and the identifying assumptions that justify them are summarized in Heckmanand Robb (1985, 1986), Heckman (1990a), Heckman and Smith (1996), and Heckman, Smith, andLaLonde (1999).

3

See Heckman (1992), HeckmanandSmith (1993; 1995a), and Heckman, Smith,andLaLonde(1999) for statementsof thoseassumptions.

4

SeeTorp,etal.(1993), Heckman, Khoo, Roselius,andSmith(1996)andHeckman, Hohmann,Khoo, and Smith (1997).

of 00

Leave a Comment