Chapter 1 _ Thinking Empirically Once one starts to think empirically about the world and about the particular issues and problems that one faces, it is difficult to imagine what one did before one thought empirically. We hope that you will find that knowing how to look for and evaluate empirical evidence will broaden your horizons. It may give you access to an entirely new litera- ture that was previously beyond your ability to read and appreciate. It may allow you to ask new questions about the law and to suggest meth- ods by which those questions might be answered. It may even inspire you to do empirical work of your own and thereby contribute to our under- standing of how legal institutions work. I. What Are Empirical Methods? By the term empirical methods we mean, at the most general level, all techniques for systematically gathering, describing, and critically analyz- ing data (objective information about the world). In this book, we will take you through this process. We begin, in Parts I and II, with a discus- sion of the practice of designing empirical research and gathering, data. This is a surprisingly complex process in which a great deal of careful forethought and planning can make the actual gathering of data far less chaotic and can lead to more useful data analysis. In some instances, you will find that someone else has already gathered the data for you. There are lots and lots of publicly available datasets containing data that others have compiled, and some of them may contain exactly the information that you need. We describe where you can find and access many of these datasets for your own empirical work. _A far more daunting, but potentially exciting, situation is one in which the data that will help to answer the empirical question that you have posed are not already available. This could be because either some- one else has control of the data and is not inclined to let you use it or, more likely, no one has yet gathered the data you need. In that second instance, you will have to set out to collect the data yourself. For instance, Suppose that, like one of our students, you would like to know the extent to which certain economic concepts figure in how judges decide commer- cial law cases and controversies. You may seek to answer that question by, 5 Scanned with CamScanner6 | Empirical Methods in Law . irculating it among all or some first, drafting a questionnaire and Lapteereoeie oan ducting bi the relevant judges Themen, such as how to draft the questionnsire ier it should be mailed or e-mailed or administered by phone, whether there should be a follow-up communication, whether one should eee reser the questionnaire to all or to a random subset of the judges, and so on. Altematcly, you may seek to explore this question by lookin at opinions written by such judges for evidence of the judges’ use of the relevant concepts. This investigation, too, involves a great many details, such as what opinions to examine, how to determine whether the concept was present, and so on. All of these methodological details are matters about which we will have more to say. Once you have gathered your data, you next need to find a way to summarize the data so that you can describe it in a simple and informa. tive manner. Few readers will be interested in reviewing each Piece of the full set of data you have gathered. And most will not know what to make of a mountain of data, even if organized into neat columns and rows in a spreadsheet. The challenge is to find a way to present a summary of the data that is simple, informative, and not misleading. In addition, you will likely want to use the data to make inferences about broader Populations and to explore relationships among variables. As we shall see in Part II, there are well-established methods for accomplishing these tasks. Finally, you will want to consider effective ways in which to present empirical information. You may need to demonstrate complicated empir- ical material as part of a trial or in an administrative agency hearing or as Part of a legislative committee investigation. Those audiences may not be as facile at dealing with empirical evidence as you will have become, and there are better and worse ways to write about empirical work and to these in Part IV. ane een in the sciences to distinguish between theoretical and ee leg but also to See them as closely linked parts of a common sepa coretical Physics, for example, is a distinct specialty from experi Erysics, but there is a close working relationship between Thy eacialties. The theoretical side of most scholarly disciplines is generally speaking, concemed with articulating the fundamental propos" tons of the isitine na un Provide causal explanations and predic ons about the phenomena und i ee the form, “The weather will be « od ee oy ae fj Cold for the next few days because - « with the statements after “because” Providing an explanation for why the i Scanned with CamScannerThinking Empirically | 7 weather will be cold for the next few days. Implicit in the reasons given in explanation for the phenomena under investigation is the contention that if those reasons should appear again in the future, then we would again expect the weather to be cold for a few days. (One of the purposes of theory is the generation of testable hypotheses. We have already remarked on the fact that theories provide reasons for the phenomena under investigation and that these reasons imply predic- tions about the relationship between those reasons and the phenomena— namely, that whenever we identify these reasons as being in existence, the phenomena under discussion will arise. These predictions about causal relationships are referred to as itypotheses. Empirical research and the hypotheses that are tested are stronger when they are grounded in theory. And, as you can no doubt imagine, the worth of a theory often turns on whether the hypotheses that it implies are borne out.’ This is precisely the point at which the connection between theory and empirical work lies. If the hypotheses are confirmed by careful empirical work, then the theory may be (provisionally) accepted. If not, then the theory is re~ vised, resulting in new propositions and testable hypotheses. Il. Thinking Empirically Thinking like an empiricist differs in important ways from thinking like a lawyer. While both modes of thinking are grounded in rigorous analysis, lawyers and empiricists often have different goals and ap- proaches. Legal analysis places a premium on argumentation and appeals to authority, is frequently geared toward proving a particular view, is of- ten focused on the particulars of an individual case, and is directed at reaching a definitive conclusion. In contrast, empirical analysis places 2 premium on observation, challenges assumptions, is oriented toward the testing of hypotheses, is usually focused on describing patterns that oper- ate in the aggregate, and is a continuing enterprise in which new work builds on that which came before and generates even more questions for further investigation. ‘A. Some Characteristics of Empirical Research Empirical research is based on observation. Consider the legal profes- sion’s traditional approach to facts. Many a legal dispute revolves around whether Fact A or Fact B is true. Lawyers are hired to represent a particu- lar side of the case, and the job of each lawyer is to find evidence support™ 1 The exact relationship between theory validation and hypothesis testing is a central topic in the philosophy of science. One of the most famous entries in that literature is THOMAS S. KUHN, THE STRUCTURE OF SCIENTIFIC REVOLUTIONS (3d ed. 1996), which looked at several historical episodes, such as the shift from a geo- centric conception of our solar system to a heliocentric conception, in order to draw some general lessons about how scientists effectuate a fundamental theo- retical change. We will return to a discussion of hypothesis testing in Chapter 9. Scanned with CamScannerEmpirical Methods in Law . . ; ignaredior, ing that side of the case. Inconsistent evidence is often ignored or, if the i i idence, explained awa side uses the inconsistent evi 2, exp such an approach is the antithesis of the scientific method, the generation of hypotheses, experimentation, and the dence. n environment that rewards persuasion, the regal profession has always valued those with thetorical skills. We have seen evidence of the emph times when listening to new law professors describe tI Projects as “I am going to establish such-and-so” o} Such a description suggests the conclusion has alrea the scholar's job is merely to marshal evidence to prove other social science, there is an importantly different em tigator articulates a hypothesis about some real phen Sets out to find objective data that will confirm or ret While she hopes to persuade her audience that the search for confirmation or refutation is important does not seek to persuade an audience by mere the ultimate goal is a true account of the world, not winning Consider the example set forth by the United State: in a case called Carnival Cruise Lines v. Shute2 In that Sons for its holding, citing case precedent and st of writing the majority opinion, Justice Blackmi observation: Finally, it stands to reason that ining a forum y. Of course, which calls for weighing of evi- it is not surprising that strong oral or written hasis on rhetoric some- their first Tesearch Tw ‘ant to prove this.” dy been reached, and its truth. In every phasis: the inves- lomenon and then fute the hypothesis, hypothesis and the and interesting, she ‘etoric. The scholars an argument. s Supreme Court it case, a cruise shij run also made the following Passengers who purchase tickets con- clause like that at i at issue in this case benefit in the form of reduced fares reflecting the savi limiting the fora in which it may be sued. avi igs that the cruise line enjoys by How do we know Justice Blackmun is right? It also might stand to Teason that any cost savi ise li 18s Benerated by limiting the places in which the Cruise line could be sued are 5 ici s = to the prices charged for tick ‘© negligible that they result in no cl ge the form of hi, drafting of the 2499 U.S. 585 (1991), 31d. at 594. We are indebted to ren who discussed this parti icket: Ss. Cost savings also might be captured in igher profits, Per e . E contract clai haps instead the cruise line entrusted the ‘use to an attorney who selected Florida precise- then-Profe: 'ssor (and now Senator) Elizabeth Wat- icular Passage on the Credit Slips blog (available at http://www.creditslips.org/creditstipayone it Slips blog ( PS/2006/11/what_ is_a_facthtml) and raised ' the issue of what it meant for the empirical approach to law. Scanned with CamScannerThinking Empirically | 9 ly because litigation costs are higher in Florida, which would be more profitable for the lawyer. Or perhaps the lawyers were taking advantage of the passengers’ “rational ignorance”: almost all passengers anticipate a pleasing vacation and do not expect to have a legal controversy with the éruise line. As a result, the lawyer might have strongly suspected that the passengers would not read the fine print on the back of their tickets and would not, therefore, have thought that the clause limiting the forum in which to bring a controversy was worth anything. Justice Blackmun made a prediction about the state of the world, but it is merely a plausible prediction, not a fact about the state of the world. It must be verified in order to be accepted. Stated differently, Justice Blackmun stated a hypothesis that could be subjected to empirical testing. All of the alternatives that we mentioned in the previous paragraph are also plausible, and there may be others. Empiricists “do not accept a statement or generalization about the world because an authority or ex- pert says it is true, or because tradition and common sense says it is so, or because it seems intuitively plausible.” The best way to figure out which of these competing explanations for the limited forum clause and its ef- fects is most correct is not by close reasoning but by observation. It is with this observation that empirical research is concerned. We do not intend this one passage from the U.S. Supreme Court to serve as an exemplar of particularly poor reasoning. It is not. Rather, it is typical of many such passages in many judicial opinions and many court briefs (and in the scholarly literature) involving matters great and small in the legal system. Two things strike us about this passage. First, it is completely unnecessary to the Court's holding, but as we have noted, the legal profession has traditionally valued argumentation. Second, Justice Blackmun’s assertion that his explanation of the effect of the forum- limitation clause on cruise prices “stands to reason” was not challenged by the two dissenting justices. Empirical research is oriented toward testing hypotheses. Serious academ- ic theories are almost always logically coherent and complete, but the world may not comport with the theory. We do empirical work to see whether our theory is borne out by the facts. Engaging in empirical ob- servation can sometimes help to resolve equally poised theoretical argu- ments or can throw a surprising light on legal questions. Consider a study one of us did on the choice to file chapter 13 bank- tuptcy. Theory might suggest that chapter choice is largely determined by legal technicalities and not social circumstances. No one had consid- ered that the choice might be influenced by a debtor's race. Empirical re- Search using a national sample of bankruptcy filers, however, found that even after controlling for a variety of financial and legal factors that might Support filing under chapter 13, African Americans are more likely to file Se “ROYCE A. SINGLETON, JR. & BRUCE C. STRAITS, APPROACHES TO SOCIAL RESEARCH Gd ed. 1999), Scanned with CamScanner10 | Empirical Methods in Law 13 as compared to bankruptcy filers of other bankruptcy under Saree were obtained using experimental simula. ys who were asked to advise a hypothetical were more Wey to recommend chapter ‘i named “Reggie and Latisha” a over chepieeZaviney ee Methodist Episcopal Chusch ne a ere aieiten identical financial situation named “Todd and Al fon who attended a United Methodist aero apparent racial steering raised troubling sues given that chapter 15 tends to cost more and ees a new insight into something about the world — that a plays a role in the choices that get made about which chapter to file under in bankruptcy—a choice that has implications for the relief af. forded to debtors—and that the attorneys advising such clients may play in these racial disparities. a inal research ie concerned with aggregate effects. Legal analysis tends to be directed at reaching a resolution in a particular case or craft. ing an agreement for a particular deal—the focus is often on the details of a particular case. In contrast, empirical analysis is concerned with exam- ining patterns in the aggregate. ; ; Consider how one approaches the problem of eyewitness testimony. Of the growing number of individuals who have been convicted of a crime and then later exonerated by DNA testing, 75% had been mistaken- ly identified by an eyewitness to the crime. Perhaps not surprisingly, eyewitness testimony identifying an individual as the perpetrator of a crime is quite influential—affecting decisions made by investigators, prosecutors, judges, and jurors. In a legal case, then, one of the key ques- tions at issue may be the reliability of a particular eyewitness identifica- tion. races. Complementary resu tion methodology. Attorne) couple considering bankruptcy Empirical research on eyewitness testimony can shed light on this individual question, but it does so by focusing on aggregate data about the factors that affect the ability of eyewitnesses generally to make accu- tate identifications. Some of these factors—called estimator variables—are features of the witnessing situation and cannot be altered by those who make use of the evidence. These include such things as the stress the wit- ness was under when she viewed the crime, the presence of a weapon, the age or gender of the witness, the physical conditions of the witnessing (for example, lighting, distance, or the use of a disguise), whether the identification was a cross-race identification, or the length of time be- tween the witnessing and the identification. Other factors—called system variables—are features of the identification that are under the control of ee *Jean Braucher, Dov Cohen & Robert M. Lawless, Race, Attorney Influence, and Bankruptcy Chapter Choice, 9 J. EMPIRICAL LEGAL STUD. 323 (2012). ° See Gary L. Wells et al., From the Lab to the Police Station: A Successful Application of Eyewitness Research, 55 AM. PSYCHOL. 581 (2000); Gary L. Wells et al., Eyewitness Evidence: Improving Its Probative Value, 7 PSYCHOL, SCI. IN PUB, INTEREST 45 (2006) - Scanned with CamScannerThinking Empirically | 11 tem, These include the ways in which wi . 7 the ways in which lineups ee ae gregate patterns are relevant to ator ease—establishing what tends fo happen areas ae aan of what happened in the individual case, While this evidence ery nt be dispositive, it does help to enlighten the factual question ake also that there are different levels of aggregation. At the broadest level, the research on eyewitnesses aims to discover patterns in how peo- pie in general perform 28 eyewitnesses overall. More importantly, how- ever, the research seeks to identify factors that moderate these patterns— for example, identifying cone under which identifications tend to be e uni the legal s viewed and Note that the ag} particularly poor or thos ier which identifications tend to be relative- aapod, When such conditions are considered, the aggregate results can ely matched with the relevant factual scenario. Yvearch is an incremental and ongoing process. Much legal at reaching a final resolution on a legal question. In con- ongoing inquiry. Each new study h is one of nce, building on the empirical work that has ww questions for future inquiry: iifferent tools available to them, dif- ferent constraints, and different goals. Science demands no final deci- sions; it is an ongoing process. If the evidence is murky, scientists can wait, can reserve judgment until they can conduct further research. And they can figure out ‘what further research needs to be done to an- Swer the question, and do it. Judges can neither reserve judgment nor go beyond the data presented in court, however ambiguous those data might be. They cannot carry out further research, nor wait until others have done so; they must decide. ‘And the judge’s decision, whether the evidence is conclusive or com pletely inadequate, has the same precedential force. It is final. The sci- entist’s conclusions are never final, always tentative.” ed much as we would like difficult issues to have straightforward an- swers, once empirical techniques are brought to b i Pree s iq 2) ear on a matter, that is Relatedly, the need to come to a final i i i pth resolution on’a legal question eee thinkers to think in categorical, either-or tcome A civil de- oe ister negligent or not negligent; a criminal defendant is either fringed 7 eu oF not guilty by reason of insanity. A patent was in- study is eth inf ringed. Evidence is either admissible or inadmissible. A er valid or invalid. In contrast, an empirical approach is more 1 be more clos Empirical analysis is aimed trast, an empirical approac makes an incremental advai come before it and raising ne [T)the judge and the scientist have d — — Phoebe C. Ellsworth, Legal Reasoning, in THE CAMBRIDGE HANDBOOK OF THINK- | Ret 1B ING AND REASONING 685, 696 (Keith J. Holyoak & Robert G, Morrison cs 2005). Scanned with CamScanner12 | Empirical Methods in Law inclined toward thinking in terms of continuous distributions of oy, jables.* Scat erpiieal work on the deterrent effect of the death Penalty Such work has been going on for over 30 years? The results are mixes and shifting. Professor Isaac Ehrlich of 7 fora University of New York at Buffalo famously reported in the arly: Fees that one execution de: terred between 7 and 8 subsequent homicides." Since the 1970s, the have been several waves of research one immediately after Ehrichys pioneering study casting doubts on his conclusions, then a recent reviy) of empirical research, generally finding a deterrent effect at least as stron and perhaps much stronger than that found by Ehrlich;"" and a still moe recent literature that is critical of studies finding that the death Penalty t homicides. oa the matter is not settled, despite the assiduous ang continuing efforts of some very accomplished scholars.¥® This does my mean that the death penalty has no deterrent effect. Nor does it mean that the death penalty does have a deterrent effect. And it certainly does Tot mean that empirical evidence is useless or that the matter will never be empirically resolved. It simply means that scholarly debates (about both theoretical and empirical matters) are dynamic, ongoing affairs. Someone who had reviewed the literature on the deterrent effect of the death pen- alty in the early or mid-1980s and had concluded that the trail had gone cold, that we would almost surely never know the answer to this im. Portant question, would have been proved spectacularly wrong by the outpouring of scholarly studies of the early 2000s. ‘ dite Ellsworth, Legal Reasoning and Scientific Reasoning, 63 ALA. L. REV. 895 * See the summary of this literature in COOTER & ULEN, Law AND ECONOMICS Ch. 13 (6th ed. 2011), por Yested States Supreme Court cited this famous result in its opinion rein- sag nb the constitutionality of the death penalty in Gre; Georgia, 428 US. 153 (1976), and its companion cases, ea SB OTB : ” i Apaen See, 81, H. Naci Mocan & R. Kaj Gittings, Getting Off Death Row: Commuted Ser she . mon Hose taktsh, Paul H. Rubin & Joanna M. Shepherd, Does Capital Punish terrent cine New Evidence from Postmoratorium Panel Data, 5 AM. Donohue Il & Justi Penalty Dea, om ror eae ie Da ‘mpirical r¢ - . the fa Pear jnamined a wide range of additional issues related '0 sion of jury instructions, the wre, tS OF death qualifying jurors, comprehe™ cess, public opinion about 7 role of discrimination at various stages of the pro- about crime and punishment en eonishment, and the effects of medio reporting Punishment See Cac HaNey, DEATH ny Destew ogy Scanned with CamScanner‘Thinking Empirically | 13 Several factors may contribute to the waxi 7 intereat in a topic. For example, research opeortanti eet eealatly when the law changes. The law surrounding the death penalty Leas changed over time. Of particular note is the period 1972-1976 during Which the United States Supreme Court found that all the states death Matutes were unconstitutional, In recent years, a number of states have Sather abolished or imposed moratoria on the death penalty" In addition, GinSupreme Court has abolished the death penalty for offenders with nental disabilities, offenders who were under 18 when their crimes were sievmitted, and offenders convicted of rape.'* These changes in the law of capital punishment have provided new occasions for testing the deterrent effect of the death penalty. ; Second, improvements in methodological and statistical techniques stimulate new ways of engaging with empirical questions. The techniques available to scholars in the early part of the twenty-first century are far fnore powerful than those that were available in the 1970s and early 1980s. The new techniques allow for far more sophisticated hypothesis testing than had been possible 20 years ago. Moreover, and not insignificantly, the computing power available to the average scholar is vastly greater than it was to scholars several decades ago. One of the catchphrases of this change is that today’s laptops have the computing power that super~ computers had 25 years ago. Both of these trends are likely to continue, so that both better techniques and more powerful computers will be availa- ble in the future. So, the world keeps changing, providing us with new circumstances, new data, and new technologies and techniques for empirical study. Despite this posture of ongoing inquiry, there will at some point be enough research on a topic that useful conclusions can be drawn. When a body of carefully conducted empirical research has been developed, such work can also shed light on legal questions in ways that lead to changes in legal practices or structures. Consider, again, the problem of the relia~ bility of eyewitness testimony. In the past, not much attention was paid to the instructions that police gave to witnesses before they viewed a lineup until a series of studies was conducted that explored the effects of such instructions on eyewitness accuracy. In a typical study of eyewitness identification, researchers have individuals “witness” a crime—this is of- ten either a staged crime or a videotaped version. After the witnesses view the “crime,” they are asked to view a lineup. This procedure gives the researchers control over the crime, the witnessing conditions, and the content and conduct of the lineup—in particular, researchers know the identity of the true perpetrator and whether he or she is in the lineup. (In contrast, in the real world, we do not ever know for sure whether the real ‘ See http://www deathpenaltyinfo.org/states-and-without-death-penalty. See Atkins v. Virginia, 536 U.S. 304 (2002); Roper v. Simmons, 543 U.S. 551 (2005); Kennedy v. Louisiana, 554 U.S. 407 (2008). Scanned with CamScanner14 | Empirical Methods in Law Perpetrator is present; we know only that the suspected Perpetrator ; resent. is een a of the studies exploring the effect of lineup instruction Broup of witnesses is instructed that the suspect “may or may mai me Present in the lineup. A separate group of witnesses is not cattioned this way. These studies have found that the differences in thew two ser of instructions tendo ave litle effect onthe accuracy af wit tfcations when the perpetrator is indeed present in the Tinea sy other hand, when witnesses view a lineup that does not include the actu; perpetrator, they are much less likely to make a mistaken identification when they are warned about the possibility that the Perpetrator might n, be present than when they are not so cautioned. When Not told of te possibility that the suspect might not be present, witnesses a0 likely tg assume that the perpetrator is in the lineup and to view their task as ong in which they are to identify the member of the lineup who bears the strongest resemblance to the perpetrator. When the perpetrator is Present in the lineup, this strategy poses little problem; when the Perpetrator ig not present, however, this approach is likely to result in an increased rate of error. As a result of this line of empirical research, the Department of Justice has recommended that witnesses be instructed that “the Person who committed the crime may or may not be present.”6 Research into other aspects of the Procedures by which eyewitnesses make identifications—such as the selection of “fillers” for a lineup, the size of the lineup, the use of sequential lineup procedures, and the tse of double-blind procedures—has also contributed to an understanding of how to improve the probative value of eyewitness evidence by maximiz. ing the likelihood of correct identifications and minimizing the likelihood of erroneous identifications. In addition to changes in lineup and witness interviewing procedures, decision makers at trial now may hear expert testimony or be given judicial instructions” about the factors that influ ence witness accuracy. B. Some Limitations of an Empirical Approach As we shall emphasize, an empirical approach is a rigorous approach. As we consider issues of research design and data collection in Parts I and II, you will see that it is important to pay attention to a wide range of is sues in the design and conduct of an empirical study. How have concepts been defined and measured? How was a survey question worded? How was an experimental manipulation achieved? How was a sample of pa ticipants, cases, or documents drawn from a larger population? Such de sign choices have important implications for the interpretation of # study's results. Similarly, as we will see in Part III, the assumptions ut derlying different statistical analyses can be quite demanding. In addition ee “US, Department of Justice, Technical Working Group for Eyewitness Eviden ® EYEWITNESS EVIDENCE: A GUIDE FoR LAW ENFORCEMENT (1999). 17 See State v. Ledbetter, 881 A.2d 290 (Conn. 2005). Scanned with CamScanner