How Many Cases Are Easy

HOW MANY CASES ARE EASY?
Joshua B. Fischman*
Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

ABSTRACT
Because judges are expected to decide cases through the impartial application of existing
law, they are often reluctant to admit that they must make law in hard cases. Many judges
claim that such hard cases are rare, constituting roughly 10 percent of cases. In stark con-
trast, economic models of the selection of disputes for litigation predict that easy cases
will settle, so that only hard cases would remain in trial and appellate courts. Empirical indi-
cators, such as dissent rates or voting differences between Democratic and Republican
appointees, have yielded muddled conclusions about the proportion of easy and hard
cases in appellate courts. In fact, none of these crude statistics relate directly to the propor-
tion of easy cases. This article develops a new approach for empirically analyzing the pro-
portion of easy cases. Although the easiness and hardness of cases are subjective, it is pos-
sible to estimate feasible combinations of the proportion of hard cases and clear errors.
This approach relies only on the basic premise that reasonable judges should not disagree
in easy cases. The article then illustrates this approach using two datasets of appeals. An
analysis of asylum appeals in the 9th Circuit finds widespread disagreement, implying
high proportions of hard cases, clear errors, or both. By contrast, voting data from labor
and environmental cases in the D.C. Circuit is consistent with the claim that 90 percent of
cases are easy and 1 percent of decisions are clear errors.
1. INTRODUCTION
When President Donald Trump criticized an “Obama judge” who had ruled
against his administration’s asylum policy,1 Chief Justice John Roberts
* Professor of Law, University of Virginia School of Law, E-mail: jfischman@virginia.edu.

I thank Oren Bar-Gill, Charles Barzun, Brian Bix, Katherine Mims Crocker, Paul Edelman, Jonah
Gelbach, Nicholas Georgakopoulos, Michael Gilbert, Louis Kaplow, Margaret Lemos, Marin Levy,
Michael Livermore, Richard Re, Fred Schauer, Eric Segall, Ramsi Woodcock, and workshop partici-
pants at Duke Law School, Georgia State University, Harvard Law School, the Conference on
Empirical Legal Studies, and the American Law and Economics Association Annual Meeting for
many helpful comments. I am especially grateful to an anonymous referee for many constructive
suggestions.
1 See Adam Liptak, Chief Justice Defends Judicial Independence After Trump Attacks ‘Obama Judge’,
N. Y. Times (November 21, 2018), https://www.nytimes.com/2018/11/21/us/politics/trump-chief-
justice-roberts-rebuke.html.
C The Author(s) 2021. Published by Oxford University Press on behalf of The John M. Olin Center for Law, Economics and
V
Business at Harvard Law School.
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License
(http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any
medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com
doi:10.1093/jla/laaa010
596 ~ Fischman: How Many Cases Are Easy?
countered with a forceful defense of the judiciary. “We do not have Obama
judges or Trump judges, Bush judges or Clinton judges,” Roberts responded,
emphasizing that judges are “doing their level best to do equal right to those
appearing before them.”2 Roberts’s unusual rebuke was surely motivated by
the threat that Trump’s remarks posed to judicial independence. Because fed-
eral judges are unelected and unaccountable, their legitimacy depends largely
on the perception that they are deciding cases by neutrally applying existing

law, and not on the basis of political considerations (Tyler & Mitchell 1994,
p. 708; Farber & Sherry 2009, p. 123). As a factual matter, however, Trump’s
insinuation had a firm basis in the literature on judicial behavior, which has re-
peatedly found significant differences between Democratic and Republican
appointees (Revesz 1997; Segal & Spaeth 2002; Sunstein et al. 2006).
If Trump could threaten the judiciary by merely repeating well-known facts,
perhaps our conception of judicial legitimacy needs sturdier foundations.
Despite all evidence to the contrary, judicial nominees have repeatedly clung to
the rhetoric of neutrality, denying that value judgments play any role in their
decisions (Kahan 2011, p. 5; Sachs 2019, p. 533). In his confirmation hearing,
Roberts famously compared judges to “umpires” who call “balls and strikes.”3
Justice Sonia Sotomayor similarly claimed that judges “apply law to facts,”4
repudiating her prior observation that a Latina judge might offer a different
perspective from a white male judge.5 Justice Brett Kavanaugh insisted that a
judge must be “a neutral and impartial arbiter” who “must interpret the law,
not make the law.”6
The problem with this rhetoric is that it depends on a conception of judging
that is unattainable. Every lawyer knows that there are hard cases where legal
materials do not provide clear answers. In such cases, when judges must neces-
sarily apply contestable value judgments, it is proper to debate which values
should guide their decisions. At the same time, there are many easy cases where
the law is settled and judges ought to apply it impartially. Understanding how
many cases are easy and hard is therefore essential to any evaluation of judicial
2 Id.
3 See Confirmation Hearing on the Nomination of John G. Roberts, Jr. to be Chief Justice of the
United States Before the S. Comm. on the Judiciary, 109th Cong. 55 (2005) [hereinafter, Roberts
Hearing].
4 Confirmation Hearing on the Nomination of Hon. Sonia Sotomayor, to Be an Associate Justice of
the Supreme Court of the United States: Hearing Before the S. Comm. on the Judiciary, 111th
Cong. 120 (2009) [hereinafter, Sotomayor Hearing].
5 See id. at 123–126.
6 See Judge Brett Kavanaugh’s Opening Statement: Full Prepared Remarks and Video, N. Y. Times
(September 4, 2018), https://www.nytimes.com/2018/09/04/us/politics/judge-brett-kavanaughs-
opening-statement-full-prepared-remarks.html [hereinafter, Kavanaugh Opening Statement].
2021: Volume 13 ~ Journal of Legal Analysis ~ 597
decision making. When judges introduce controversial value judgments in easy

cases, we criticize them as activist,7 ideological (Sunstein et al. 2006, p. 148), or
unprincipled (Edwards 1991, p. 859). When judges purport to be “following
the law” in hard cases, we criticize them for being disingenuous (Chemerinsky
2006, pp. 1077–1079) or formalistic (Schauer 1988, pp. 509–517).
Legal scholarship, however, has not carefully addressed how many cases are
easy. Influential economic models of settlement bargaining predict that easy

cases should settle and that most litigated cases should therefore be hard. Yet in
public discourse, judges and commentators frequently assert that hard cases
are rare—if they acknowledge their existence at all. Going back to Judge
Benjamin Cardozo, many judges, scholars, and public figures have claimed
that roughly 90 percent of cases are easy.
In a notable example of our skewed discourse, then-Senator Barack Obama
responded to Roberts’s umpire analogy by claiming that “adherence to legal
precedent and rules of statutory or constitutional construction will dispose of
95 percent of the cases that come before a court, so that both a Scalia and a
Ginsburg will arrive at the same place most of the time on those 95 percent of
the cases.”8 In the remaining 5 percent of hard cases, Obama argued that “the
critical ingredient is supplied by what is in the judge’s heart.”9 Critics pounced
on Obama’s evocation of judicial empathy, suggesting that he was endorsing
judicial “activism” and “lawlessness” (Rollert 2010, pp. 90–93; Colby 2012, pp.
2953–2958). Obama’s own Supreme Court nominees, Justices Sotomayor and
Elena Kagan, disavowed his call for judicial empathy during their own con-
firmation hearings.10 Hardly anyone, however, took issue with Obama’s sug-
gestion that 95 percent of cases could be decided by calling balls and strikes. In
fact, scholars praised his statement for its candor (Dworkin 2009, p. 37; Rollert
2010, p. 100) and for bringing discourse about judging “into contact with
reality.”11
7 See Black’s Law Dictionary 974 (10th Edition, 2014) (defining “judicial activism” as a “philosophy
of judicial decision-making whereby judges allow their personal views about public policy, among
other factors, to guide their decisions”).
8 151 Cong. Rec. 21,032 (2005) (statement of Sen. Barack Obama).
9 Id.
10 See Sotomayor Hearing at 120 (stating that “judges can’t rely on what’s in their heart”); The
Nomination of Elena Kagan To Be an Associate Justice of the Supreme Court of the United States:
Hearing Before the S. Comm. on the Judiciary, 111th Cong. 103 (2010) (declining to endorse
Obama’s empathy standard and claiming that “it’s law all the way down”).
11 Louis Michael Seidman, Comment to The Federalist Society Online Debate Series: The Sotomayor
Nomination, Part II, Federalist Soc’y (July 13, 2009), http://www.fed-soc.org/publications/detail/
the-sotomayor-nomination-part-ii.
Given the wide gap between settlement models and public discourse on easy
cases, it would be natural to seek answers through empirical research. Yet the
empirical literature on judicial voting behavior has not successfully connected
its findings to lawyers’ understandings about the division between easy and
hard cases. Some studies have cited simple statistics, such as dissent rates or the
frequency of published decisions, to support claims about the proportion of
easy cases. As this article demonstrates, however, such statistics do not directly

relate to the proportion of easy cases.
This article develops a novel framework for empirically examining the pro-
portion of easy and hard cases, using voting data from the US Courts of
Appeals. This inquiry is especially challenging because the easiness and hard-
ness of cases are ultimately subjective. Although legal scholarship has much to
say about hard cases (e.g., Dworkin 1975) and easy cases (e.g., Schauer 1985),
there is no shared understanding about what makes a case hard or easy; this ne-
cessarily depends on contested meanings of legal materials, how they apply in
particular cases, and how these materials are perceived by judges or other
members of a legal community.
The framework developed in this article surmounts this difficulty using
methods of bounded inference. In particular, it is possible to jointly estimate
bounds on the proportion of hard cases and the proportion of “clear errors”—
cases where a court reached a wrong result in an easy case. In estimating these
bounds, it is not necessary to fully specify the definition of an easy case. The
bounds will apply to any definition of an easy case, as long as hard cases and
clear errors are counted according to the same definition.
The article begins in Section 2 by examining the different meanings of “easy”
and “hard” cases in legal discourse. A few scholars use these terms to reflect the
division between cases with correct answers and those without. Others use
these terms to denote forms of epistemic determinacy, such as whether the cor-
rect answer to a case is readily knowable to a particular interpreter. Some use
“easy” and “hard” in an individual sense to reflect the perspective of a particu-
lar judge or interpreter. Others use it in a sociological sense to refer to some
level of agreement within a community that a case is easy or hard. Still, others
view hard cases as requiring reasonable disagreement within a community.
The article then proceeds to compare three separate literatures that address
the proportion of easy cases. As with the academic debate, the article focuses
primarily on the federal courts of appeals and epistemic conceptions of “easy”
and “hard” cases. Section 3 begins by examining judges’ estimates based on
their own experiences, as described in speeches and published articles.
Although there is some variation in their estimates, many prominent circuit
judges have claimed that roughly 90 percent of cases are easy. These estimates,
which transcend circuits, time periods, and ideological divisions, have been
repeated so often that they have arguably achieved the status of conventional
wisdom. Interestingly, judges’ remarks suggest that they often do not have
shared understandings about what it means for a case to be easy. Nevertheless,
many of them agree on the proportion of easy cases—however “easy” is
defined.
Section 4 considers economic models of the selection of disputes for litiga-
tion. According to many of these models, easy cases settle and only hard cases

are litigated and appealed. Thus, if these models are accurate, most cases in the
courts of appeals should be hard and few should be easy. This prediction stands
in sharp tension with judges’ reported observations. Section 4 then examines
the assumptions underlying these models and demonstrates that they hold
only in limited circumstances.
Section 5 examines empirical approaches used in legal scholarship to esti-
mate the proportion of easy cases in the federal courts of appeals. Some
articles have cited statistics based on within-panel disagreement, such as
dissent rates. Others have considered differences between panels, typically
by comparing differences in outcomes in panels dominated by Democratic
versus Republican appointees. As a general matter, differences in voting be-
havior appear more pronounced between panels than within panels. Other
studies point to the proportion of cases decided without oral argument or
without published opinions. All of these statistics suggest very different
conclusions regarding the proportion of easy cases. Legal scholarship often
cites such statistics, without explaining why these statistics differ or what
relevance they have for our understanding about the proportion of easy
cases.
Although rates of judicial agreement and disagreement have intuitive
connections to rates of easy and hard cases, these frequently cited statistics
are not directly informative. Section 6 develops a framework for generating
inferences about the proportion of easy cases from voting behavior in the
U.S. Courts of Appeals. The key premise is that reasonable judges should
not disagree in easy cases. When the correct outcome is clear, any reason-
able judge should follow it. Thus, if two judges disagree about the outcome
of a case, then either the case is hard or one of them is making a clear error.
Judicial voting data can then determine the proportions of easy cases that
are compatible with various rates of clear errors. In particular, for any
stipulated rate of clear errors, one can estimate an upper bound on the pro-
portion of easy cases. This is true for any subjective view regarding which
cases are easy and which decisions are clear errors, as long as these are
defined consistently.
Section 6 then illustrates this method of analysis using two datasets. The first
involves asylum appeals in the 9th Circuit. The analysis demonstrates that
some panels would disagree in at least 60 percent of the asylum cases. If

90 percent were easy, then at least 7 percent would be clear errors. This is espe-
cially troubling because 92 percent of these cases were decided summarily
with unpublished opinions. Such abbreviated procedures were premised on
the notion that these were easy cases that did not warrant substantial judicial
attention.
The second dataset is comprised of appeals from the Environmental

Protection Agency (EPA) and National Labor Relations Board (NLRB) in the
D.C. Circuit. Panels at opposite extremes would disagree at least 40 percent of
the time, however, such panels are rare. The data are consistent with the claim
that 90 percent of the cases were easy and 1 percent was clear errors, although
the actual levels may be much higher.
The examples analyzed in this article provide mixed evidence regarding
judicial overconfidence about the easiness of cases. To the extent such over-
confidence exists, Section 7 discusses some of the implications. Aside from
concerns about judicial legitimacy, the findings also have important impli-
cations for judicial administration in circuit courts. To deal with excessive
caseloads, the federal courts of appeals use staff attorneys to prescreen cases.
Cases deemed to be easy are decided largely on the basis of the staff attor-
neys’ recommendations, with no oral argument and unpublished disposi-
tions. The analysis here provides one way to assess whether these cases
warrant more careful consideration.
2. WHAT MAKES CASES EASY OR HARD?
Although legal commentators often describe cases as “easy” or “hard,” there is

no accepted understanding about what these terms mean. In fact, these terms
are often used in ways that conflate distinct concepts. This section discusses the
ways the terms “easy” and “hard” are used in legal discourse and clarifies the
meanings of these different usages.
2.1 Metaphysical Determinacy

First, some legal theorists use the distinction between “easy” and “hard” to co-
incide with metaphysical legal determinacy. For example, Marmor (1990)
defines hard cases as those that are “not determined by the existing legal stand-
ards” whereas “the law can be identified and applied straightforwardly” in easy
cases. He emphasizes that the distinction between easy and hard cases “has
nothing to do at all with the amount of intellectual effort required to decide a
legal case.” Similarly, Coleman and Leiter (1993) describe the “classic positivist
account of judicial discretion” in which easy cases have “unique, determinate

outcomes” while “judges must exercise discretion” in hard cases “because the
set of legal reasons is inadequate to determine or warrant one unique
outcome.”
This conception of “easy” and “hard” cases is objective, in the sense that it is
defined by reference to existing legal sources and not the perspective of any
particular interpreter. However, the unique correct answer may not be know-

able to a particular judge, or even to any judge. Of course, observers may dis-
agree about whether a particular case is easy in the metaphysical sense. When
they do so, however, they are debating the easiness of the case as a matter of
fact, not merely as a difference of opinion.
2.2 Epistemic Determinacy

A second usage corresponds to epistemic legal determinacy: a case is easy if the
correct answer is knowable (Kress 1990; Leiter 1995, p. 485, n.17). For example,
Lyons (1984, p. 180) defines “an easy case as one in which the law is clear
enough so that it can be decided in a more or less ‘mechanical’ way, by apply-
ing relevant rules in a logically rigorous argument.” Note that the epistemic
conception of easy and hard cases must always depend on the subjective per-
spective of the interpreter or interpreters, and also on the degree of effort
expended (Kennedy 1997, p. 170). A case that initially appears hard may be-
come easy after some consideration; a judge may also exert effort to make an
easy case seem hard if doing so could avoid an undesirable result (id., p. 166).
There cannot be a unique epistemic definition of easy and hard cases because
different interpreters will have differing capacities and employ different meth-
ods to decide cases (Kress 1990).
2.3 Individual vs. Sociological

There is another prominent distinction among the epistemic usages of “easy”
and “hard.” Some commentators describe cases as “easy” or “hard” an individ-
ual sense, meaning whether a case is perceived as easy or hard (in the epistemic
sense) to a particular interpreter. At other times, they are using these terms in a
sociological sense, meaning that a case is easy if a community of interpreters
agrees how it should be decided (Schauer 1985, p. 410; Fallon 2008, p. 1134;
Bix 2016, p. 143). Similarly, a case is hard in the sociological sense if members
of the community disagree about what the outcome should be (Brink 2001,
p. 19). The individual and sociological definitions of hard cases are closely
related; if every interpreter views a case as hard in the individual sense, then it
is nearly inevitable that there will be disagreement about the correct

outcome.12
The converse, however, does not hold. A case can be hard in the sociological
sense even if multiple interpreters perceive it as easy in the individual sense.
Justice John Paul Stevens (2006) recounted a conversation he had with Justice
Lewis Powell before the oral argument in an affirmative action case, Wygant v.
Jackson Board of Education.13 Both agreed in their conversation that the case

was easy, but they later discovered that they disagreed about correct outcome
(id., p. 1565). Both justices perceived Wygant to be easy in the individual sense,
but it was a hard case for them in the sociological sense.
The sociological notion of hard and easy cases requires reference to a par-
ticular community (Tushnet 1990, p. 141). A community of politically conser-
vative lawyers may view a case as easy, while a community of liberal lawyers
might view the same case as hard. Similarly, a case that was perceived as hard
fifty years ago might seem easy today, or vice versa (id.).
2.4 Disagreement vs. Reasonable Disagreement

When commentators describe cases as easy, they do not necessarily mean to
imply that there will be complete unanimity among all interpreters. For ex-
ample, Douglas Laycock described the constitutional challenge to the
Affordable Care Act as a “very easy case,” predicting that the challengers would
“lose eight to one.”14 It is commonplace to perceive a case as easy even if one
“deviant interpreter” would advance a contrary view (Bix 1993, p. 64). Thus,
sociological conceptions of easy and hard cases often account for outlying
views. The requisite level of agreement can be specified by some level of compe-
tence, such as by reference to interpreters who are “reasonable” (Brink 2001,
p. 19; Eisgruber 2007, p. 28; Shapiro 2011, p. 234), “serious” (Harris 1979, p. 6),
or “skilled” (Rosati 2004, p. 279). Alternatively, the level of agreement could be
12 Some scholars have defined hard cases in ways that combine the individual and sociological con-
ceptions. See Eisgruber (2007, p. 28) (defining hard cases as “cases where the law’s meaning is genu-
inely in doubt” so that “reasonable judges not only could disagree but have in fact disagreed”);
Shapiro (2011, p. 234) (describing hard cases as “suits where good arguments are made by each of
the opposing parties and reasonable lawyers disagree about which side should prevail”); Rosati
(2004, p. 279) (“By ‘hard cases,’ I mean those cases in which, if there is a correct legal answer, it is
neither readily apparent nor evident upon careful reflection of the sort skilled lawyers or jurists
commonly engage in, so that reasonable, legal interpreters could advance compelling, alternative
claims about what the law is.”).
13 476 U.S. 267 (1986).
14 See Sheryl Gay Stolberg & Charlie Savage, Vindication for Challenger of Health Care Law, N. Y.
Times (March 26, 2012), http://www.nytimes.com/2012/03/27/us/randy-barnetts-pet-cause-end-
ofhealth-law-hits-supreme-court.html.
determined by a numerical threshold, such as “nine out of ten” lawyers (Neely

1981, p. 4 & n. 2), or a less precise standard, such as “most” interpreters.
When scholars claim that cases are easy in the metaphysical or individual
epistemic sense, these claims are justified from the internal point of view, on
the basis of legal sources and how they apply to particular fact patterns.
Assertions that cases are easy in the sociological sense, however, are generally
made from the external point of view; these are empirical claims about whether

interpreters would agree on particular cases. When these claims are limited to
reasonable disagreement, however, easiness is a hybrid of internal and external
perspectives. How often interpreters disagree is an empirical question, but
whether particular instances of disagreement are reasonable is a legal question.
When easy cases are defined by reference to reasonable disagreement, claims
that cases are easy are often intended as critical rather than descriptive. To say
that a case is easy is to deny that any disagreement with the speaker’s interpret-
ation could be reasonable. For example, Justice Antonin Scalia occasionally
wrote, in dissent, that cases were “easy,”15 or even “absurdly easy.”16 In such
instances, he was obviously not claiming that the justices were in agreement
about the dispositions of those cases. Nor was he boasting that he had figured
them out with less effort than his colleagues. Scalia’s usage only makes sense as
an assertion that reasonable judges should have agreed with him and that the
majority opinions in such cases were not merely mistaken, but clearly wrong.
While Scalia could be especially caustic in his dissents, he was hardly alone in
using the term “easy” in this sense. In judicial opinions or academic commen-
taries about particular cases, claims that cases are “easy” often appear in dis-
senting opinions or separate concurrences,17 and such claims only make sense
as criticism.
2.5 The Proportion of “Easy” and “Hard” Cases, Based on Different Definitions
As discussed above, an assertion that a case is “hard” can have several different
meanings. It could mean that legal materials do not provide an objectively cor-
rect answer. It could mean that a competent judge could not be expected to as-
certain the correct answer using some specified quantum of effort. It could
15 See, e.g., Vartelas v. Holder, 566 U.S. 257, 277 (2012) (Scalia, J., dissenting); McIntyre v. Ohio
Elections Commn., 514 US 334, 385 (1995) (Scalia, J., dissenting).
16 Michigan v. Bryant, 562 US 344, 384, 386 (2011) (Scalia, J., dissenting).
17 See, e.g., Maracich v. Spears, 570 US 48, 87 (2013) (Ginsburg, J., dissenting) (“This case should
therefore be easy.”); General Dynamics Land Sys., Inc. v. Cline, 540 US 581, 602 (2004) (Thomas, J.,
dissenting) (“This should have been an easy case.”); Neder v. United States, 527 US 1 (1999)
(Stevens, J., concurring in part and concurring in the judgment) (stating, in a solo concurrence
with three judges dissenting, that “[t]his is an easy case”).
mean that reasonable judges could be expected to disagree about the correct
outcome, or equivalently, that existing disagreement is in fact reasonable.
Similarly, a statement that a case is easy could mean that it has an objectively
correct answer, or that this answer is knowable to a particular judge, or that the
answer is knowable to most judges in the community.
The proportion of hard and easy cases will inevitably vary based on which defin-
ition is used. The proportion of easy cases should be smaller under the metaphys-

ical definition than under any epistemic definition, because the correct answer to a
case cannot be knowable if it does not exist (Kress 1990, p. 139).18 Similarly, the
proportion of cases perceived as easy by an individual interpreter will generally ex-
ceed the proportion that is easy in the sociological sense. The requirement that
one interpreter finds a case to be easy is generally less demanding than the require-
ment that most interpreters agree that a case is easy.
For example, suppose that each interpreter perceives 15 percent of cases to be
hard. If they all agree that the same cases are hard, then these 15 percent will be
hard in the sociological sense. However, interpreters may disagree about which
cases are hard and how they should be decided. For example, if conservative law-
yers think 15 percent of cases are hard, and liberal lawyers think a different 15 per-
cent are hard, then 30 percent could be hard in the sociological sense.
3. JUDICIAL PERSPECTIVES ON THE PROPORTION OF EASY

CASES
Judges’ perceptions about easy and hard cases are largely products of their first-
person consciousness. For this reason, judges’ internal accounts of their activity
are an essential source regarding the proportion of cases that they perceive to be
easy—at least in the individual epistemic sense. Appellate judges may also have im-
portant insights on the proportion of cases that are easy in the sociological sense
because they are uniquely positioned to observe colleagues’ views, especially in
confidential deliberations (Edwards & Livermore 2009, p. 1903).
Much of the judicial commentary on easy cases focuses on the US Courts of
Appeals, which have the final word on most disputes involving federal law.
Most judges who have opined on the issue claim that a vast majority of federal
appellate cases are easy. Whether or not these claims are accurate, they have
been repeated widely enough to constitute a conventional wisdom about easy
and hard cases.
18 An overconfident interpreter might falsely perceive indeterminate cases to be easy, in which case
this interpreter might perceive more cases to be easy in the epistemic sense than are actually easy in
the metaphysical sense.
3.1 Judicial and Scholarly Perspectives on Easy and Hard Cases

In 1921, Judge Benjamin Cardozo offered perhaps the most famous assessment
of the division between easy and hard cases. When a hard case presents con-
flicting interests, Cardozo (1921, pp. 113, 120, 165–166) stated that a judge
behaves essentially like a legislator. Contrary to the rhetoric of legal formalism
that prevailed at that time, Cardozo argued that the law “is not found, but
made” in hard cases, and that the “process, being legislative, demands the legis-

lator’s wisdom” (id., p. 115).
Cardozo’s remarks may sound bland to contemporary ears, but at the time they
were viewed as “radical” (Polenberg 1997, p. 86; Stone 1922, p. 385), “sensational”
(Schwartz 1999, p. 305), “the voice of heresy” (Hand 1922, p. 480), and even like a
“legal version of hard core pornography”(Gilmore 1977, p. 77). Cardozo sug-
gested, perhaps jokingly, that his remarks could lead to his impeachment (Corbin
1961, p. 198). It is therefore not surprising that he was careful to emphasize that
his claim was limited to a small proportion of cases. He wrote that “a majority” of
cases could only be decided one way, while in a “considerable percentage” of cases,
“the rule of law is certain,” but the outcome could depend on the application of
the law to the facts (Cardozo 1921, p. 164). It was only in a percentage of cases,
“not large indeed,” in which “the creative element in the judicial process finds its
opportunity and power” (id., p. 165). A few years later, Cardozo offered a more
precise estimate, claiming that “[n]ine-tenths, perhaps more, of the cases that
come before a court are predetermined . . . by inevitable laws that follow them
from birth to death” (Cardozo 1924, 60).
Some scholars have questioned whether Cardozo’s claim was intended to be a
precise estimate (Weisberg 1979, p. 298 n. 68; Simon 2000, p. 1052). Nevertheless,
judges and legal scholars often repeat Cardozo’s claim that 90 percent of cases are
easy. It is not clear if Cardozo’s claim serves as a focal point, if he was stating some
sort of universal truth, or if his prominence allows other judges to safely quote
him without generating controversy. Commentators may not even agree about
what makes a case easy or about which cases are easy. Nevertheless, Cardozo’s esti-
mate has been widely repeated, arguably achieving the status of conventional
wisdom.
Several contemporary circuit judges have echoed Cardozo’s claim that 90
percent of cases are easy.19 Judge Harry Edwards (1983–1984, p. 390) has
19 Judge Ruggero Aldisert (1987, p. 463) claimed that 90 percent of cases could be resolved through
the “disinterested application of known law.” Judge Alex Kozinski (2004, p. 1098) wrote that “nine
times out of ten, maybe more,” a diverse panel of judges “would reach the obvious result.” Judge
Thomas Reavley (1997, p. 1041) claimed that “10 percent of appeals present conflicting positions
of comparable merit,” another 20–30 percent “can be easily decided after getting through the
lawyer-spinning to understand the record,” and the rest are easy enough to be “screened out by
cursory review.”
estimated 85–95 percent.20 Others have given estimates that were similar21 or
slightly lower.22 Some judges eschewed precise quantification, simply stating that
“most” or a “vast majority” of cases are easy (Friendly 1961, p. 222 & n. 23;
Newman 1984, p. 204; Rubin 1987, p. 364; Coffin 1994, p. 253; King 2007,
p. 783; McKee 2007, p. 1716; Wardlaw 2010, p. 1634; Miner 2012–2013, p. 520;
Wood 2012, p. 1464). Many state court judges have repeated this claim as well,
including judges from the California Courts of Appeal (Wold 1978, p. 62); the

Supreme Court of Arkansas (Leflar 1979, pp. 736–737 & n. 66), the Supreme
Court of Illinois (Schaefer 1966, p. 4), the Louisiana Court of Appeals (Tate
1959, p. 62), and the New York Court of Appeals (Llewellyn 1960, p. 25 n. 16;
Kaye 1995, p. 35 n. 145).
Of course, claims taken from public speeches and legal scholarship do not
constitute a representative sample of judges. However, Howard (1981,
p. 165) also reported results from a survey of thirty-five circuit judges, most
of whom agreed that roughly 90 percent of cases were easy. Many scholarly
commentators have also accepted Cardozo’s 90 percent estimate (e.g.,
Mishkin 1965, p. 60; Keeton 1969, pp. 13–14; Kress 1989, pp. 324–325;
Greenawalt 1990, p. 34; Levi 2009, p. 1800; Tamanaha 2010, p. 144), or sug-
gested even higher proportions of easy cases (Martineau 1986, p. 14 n. 85;
Pozen 2010, p. 2097). Although some scholars have questioned Cardozo’s
claim (Patterson 1947, p. ix; Jones 1961, p. 803 & n. 16; Rubin 2011, p. 874),
many nevertheless acknowledge that it represents the mainstream position
(Gordon 1993, pp. 2077–2078; Schauer 2013, pp. 760–761).
Surprisingly, even some prominent legal skeptics have endorsed Cardozo’s
estimate. Richard Posner (1988) was notably outspoken about the role of judi-
cial discretion, arguing that judges “exercise considerable discretion and fre-
quently must and do decide indeterminate cases” (p. 872). Yet Posner
acknowledged that hard cases represent only a “tiny fraction” of all cases (id.,
20 Edwards used slightly different terminology. He described 5–15 percent of cases as “very hard,”
meaning that “fair application of the law to the facts leaves [him] in equipoise,” so that he “must
rely on some significant measure of discretion.” He claimed 50 percent were “easy” in the sense
that the correct answer was clear, while 35–45 percent of cases would yield a correct answer after
some effort.
21 Judge David Nelson (1995, p. 565) speculated that “Cardozo’s ninety-percent figure is probably on
the low side” for the Sixth Circuit. Judge John Noonan (2002) claimed that “96 percent of the time
the judges agree what the law is; it wouldn’t matter who was on the panel.” Judge Patricia Wald
(1992, p. 180) estimated that 15 percent of cases constitute the “segment of the caseload . . . where
judgment counts, whether your view or an opposing one will prevail often depends on the political
philosophy of the majority.”
22 Judge J. Edward Lumbard (1968, p. 36) wrote that “most appellate judges would agree that the re-
sult in about seventy-five per cent of appeals.” Judge J. Clifford Wallace (2005, p. 197) reported
that 75 percent of cases are “relatively easy” based on the assessments of 9th Circuit staff attorneys.
p. 828), even suggesting that Judge Edwards’s estimate—that 5–15 percent of

circuit court cases are hard—is too high (Posner 2010, p. 1186).
Similarly, many of the legal realists ultimately accepted Cardozo’s estimate.
In the early 1930s, Jerome Frank claimed that trial and appellate judges regular-
ly exercise discretion due to pervasive legal indeterminacy (1930, p. 138, 1931,
p. 29). After being appointed to the Second Circuit in 1941, Frank softened his
tone. He suggested that Cardozo’s estimate was “perhaps not too wide of the

mark” with regard to appellate cases, although he maintained that “no such
easy prediction can be made of most trial-court decisions.” (1948, p. 374).
Charles Clark, another legal realist who served with Frank on the Second
Circuit, similarly concurred with Cardozo’s estimate (Clark & Trubek 1961,
p. 256 n. 7).
Karl Llewellyn followed a similar trajectory. In the early 1930s, he argued
that nearly every litigated dispute was hard (Llewellyn 1931, p. 1239). In 1950,
however, he had softened his views, estimating that “six appealed cases out of
ten” were easy (Llewellyn 1950, p. 397). By 1960, he had fully joined the main-
stream, endorsing Cardozo’s estimate that 90 percent were easy (Llewellyn
1960, p. 25 n. 16).
Interestingly, a few contemporary judges, such as Frank Easterbrook
(1994, p. 61) and John Walker (2001, p. 203), have deviated from the view
that most cases are easy. Neither of these judges is known to be a legal skep-
tic. When asserting that many cases are hard, these judges were not pur-
porting to draw on their own first-hand experiences, but rather on
economic arguments that hard cases are disproportionately selected for ap-
peal. Judge Leonard Garth (1995) was one of the few to claim that most
cases are hard based on his own observations. He wrote that his court heard
a decreasing number of “slam-dunk” cases and that “[m]ost appeals are in
the gray area” (p. 1364). None of these judges offered precise estimates
about the proportion of hard cases, but these comments suggest, at a min-
imum, that there is not consensus among judges regarding Cardozo’s
estimate.
3.2 Limitations of Judicial and Scholarly Perspectives

Judicial perspectives have many strengths: judges encounter a broad
cross-sample of cases, they have full access to all relevant legal materials, and
they observe their colleagues’ behavior in confidential deliberations. Empirical
social scientists do not have comparable access to the judicial process. For this
reason, judges such as Harry Edwards (1998, p. 1364) have prominently
criticized empirical legal scholars for neglecting judges’ own accounts of their
activity.
Others, however, have been more skeptical about the reliability of judges’ in-
ternal accounts. Revesz (1999, p. 845) argued that “Judge Edwards is not a neu-
tral observer” and that he “faces strong institutional pressures to give a
particular account of the work of his court.” Judge Richard Posner (2010,
p. 1181) questioned (somewhat ironically) whether “judicial introspection . . .
is a valid source of knowledge.” He claimed that “[j]udges tend not to be can-
did about how they decide cases” and that they “like to say they just apply the

law . . . to the facts,” leading them to overstate the proportion of easy cases
(Posner 2013, p. 106). For these reasons, many scholars who study judicial be-
havior have been skeptical or even dismissive of judicial self-reports (Schubert
1968, pp. 408–409; Mackie 1977, p. 7; Shapiro 1994, p. 156; Baum 1997, p. 19;
Klein 2002, p. 138; Cross 2003, p. 1477).
4. THE SELECTION OF DISPUTES FOR APPEAL
While most judges’ accounts of their first-hand experience give the impression
that most cases are easy, the literature on the selection of cases for litigation
and appeal leads to a starkly different conclusion. According to the conven-
tional wisdom on settlement, most easy cases are settled prior to litigation. If
the parties can predict how a court would rule on their dispute, they could set-
tle for the expected damages and save on trial costs.
When there is uncertainty about the relevant facts, the parties may have
trouble reaching a settlement and will resolve their factual disputes at trial. The
parties would only file an appeal, however, when there is also legal uncertainty.
Thus, parties should drop or settle easy cases before appeal, leaving the appel-
late courts with only hard cases.
This basic intuition—that only hard cases go to trial, and only the very hard-
est cases reach appellate courts—is widely understood in both legal scholarship
and the economic literature on settlement (Solum 1987, pp. 496–497; Schauer
1988, pp. 1726–1727). Indeed, many scholars who argued that judges exercise
substantial discretion were careful to limit their argument to litigated cases
(Llewellyn 1931, p. 1239; Radin 1942, p. 1271; Schauer 2013, p. 758). This intu-
ition has been formalized in the influential hypothesis of Priest and Klein
(1984). Using economic modeling and simple assumptions, Priest and Klein
showed that litigated disputes will be a non-representative sample of all legal
disputes. Instead, disputes that go to trial will be disproportionately hard cases,
where the legal merits are a close call (id., p. 16).
The key insight of the Priest–Klein model is that cases will go to trial when
the parties have divergent expectations about the plaintiff’s likelihood for suc-
cess. If the plaintiff and defendant both have realistic expectations about the
outcome of litigation, they will be able to reach a settlement and avoid litiga-
tion costs. The parties will only litigate when both sides are overconfident. If a
case is easy, both sides should anticipate the outcome, so there would not be
divergent expectations.
The Priest–Klein model does not predict that all hard cases will be litigated.
If a case is a tossup, so that each side has a 50 percent chance of prevailing,
then the parties could settle for roughly 50 percent of the damages sought by

the plaintiff. Hard cases do not necessarily go to trial, but they present the pos-
sibility that both sides could be overconfident. When both sides can anticipate
the outcome of an easy case, there will always be a range at which both parties
would accept a settlement. Thus, the Priest–Klein model predicts that all of the
easy cases will settle, as will some of the hard cases, leaving the remaining hard
cases to be decided by the courts.
The prediction of the Priest–Klein model is in sharp tension with the
accounts provided by many judges in Section 3 of this article. If so many cases
in appellate courts are easy, why are the parties paying to litigate them? This
section considers how the Priest–Klein model and related literature can be rec-
onciled with the judicial accounts discussed in Section 3. Although most of this
literature focuses on the selection of cases for trial, similar intuitions apply to
the selection of cases for appeal.
One possible explanation for the discrepancy between judicial estimates and
economic models of case selection is that these two literatures are not always
applying identical definitions of “easy.” Judges may be discussing their own in-
terpretive perspective or offering a sociological perspective based on interactions
with other judges. Economic models of case selection, however, are only relevant
to the sociological definition. When parties decide whether to go to file an appeal
in federal court, they consider the likelihood that they will be able persuade at
least two out of three judges in a panel. If a case were sufficiently easy in the
sociological sense—that is, if every judge would reject their claim—an appeal
would have negative expected value. Litigants are not interested in whether par-
ticular judges think the case is easy; they are concerned about the level of agree-
ment in the community. Some judges may estimate a higher proportion of easy
cases because they have the individual definition in mind.
There is a second distinction between judges’ perspectives and the predic-
tions of economic models. According to the economic model, cases will only
be appealed when there is sufficient disagreement among the judges in the cir-
cuit. The economic model, however, does not consider whether the judicial
disagreement is reasonable; the only issue is whether each side would have at
least a modest chance of prevailing. When judges speak about easy cases—even
in the sociological sense—they often have in mind those cases where there is
no reasonable disagreement. A judge may think that a case is easy in the sense
that any reasonable judge would agree, but an appeal could still be profitable
from a litigant’s perspective because of the possibility of drawing an outlier
panel.
4.1 Divergent Expectations Models of Settlement

The Priest–Klein model predicts that litigants will go to trial when they have

divergent expectations, that is, when both sides are overconfident about their
chances of winning. This section provides a simplified illustration of the model.
Suppose that a plaintiff is suing for $10,000 in damages. Both the plaintiff and
defendant believe that each side has a 50 percent chance of prevailing, so the
plaintiff will win $5,000 in expectation. If each side will incur $1,000 in litiga-
tion costs, then the plaintiff will have a net gain of $4,000 and the defendant
will have a net loss of $6,000. Thus, the plaintiff would prefer any settlement
that pays her at least $4,000, and the defendant would prefer any settlement
where he pays at most $6,000. The interval between $4,000 and $6,000 would
constitute a settlement range that would be mutually acceptable to both par-
ties. As long as such a settlement range exists, it is possible in principle for the
parties to avoid litigation.
If one or both of the parties are too optimistic, however, then settlement bar-
gaining may fail. If the plaintiff believes, for example, that she has a 75 percent
chance of winning, then her expected damages would be $7,500. After subtract-
ing litigation costs, her expected gain would be $6,500. The plaintiff would not
be willing to accept less than $6,500 and the defendant would not be willing to
pay more than $6,000. Thus, there would be no settlement range and the par-
ties would go to trial.
If a case is easy, then there should be little doubt about the outcome.
Suppose both parties thought that the plaintiff would almost certainly win.
Then the plaintiff would accept any settlement exceeding $9,000, the de-
fendant would pay at most $11,000, and the parties could bargain toward
some amount within this range. When the outcome is predictable, neither
party will be overconfident. However, this model could fail for several rea-
sons discussed below.
4.2 Why Hard Cases Are Litigated

4.2.1 Lumpy Outcomes
The Priest–Klein model is based on a lawsuit for damages. This turns out to be
a key assumption, because monetary payments are always continuously divis-
ible. Whenever there is a mutually acceptable settlement, the parties will be
able to reach an agreement within the settlement range. For example, if both
parties agree that the plaintiff has a 41 percent chance of prevailing, then they
can settle for 41 percent of the damages sought.
Settlement bargaining may fail, however, when the object of the dispute can-
not be divided with such precision. A plaintiff seeking an injunction may have
a 41 percent chance of prevailing, but the parties cannot settle for 41 percent of
an injunction. An asylum petitioner cannot settle for a partial deportation and

a criminal defendant cannot plead half guilty. There may be some intermediate
outcomes; a criminal defendant can plead guilty to a lesser offense and an asy-
lum petitioner can agree to a voluntary departure. However, these outcomes
are “lumpy,” so there may not be an intermediate outcome that would be mu-
tually agreeable to both parties (Mnookin & Kornhauser 1979, p. 975). In
criminal cases, statutory minimum penalties may similarly thwart bargains
that would best reflect the parties’ expectations (Bibas 2004, pp. 2487–2488 &
n. 97).
4.2.2 Criminal Cases

The Priest–Klein hypothesis is based on a model of civil litigation. Although
many criminal cases result in plea bargains rather than jury verdicts, the
plea bargaining differs significantly from settlement bargaining in the civil
context. Plea bargains are not necessarily based on the parties’ chances of
victory at trial, but instead occur “outside the shadow of trial” (id.). Unlike
a civil litigant, a prosecutor does not have a financial stake in a criminal trial
(id., p. 2471) and does not necessarily seek to maximize the defendant’s sen-
tence (Stuntz 2004, p. 2553). Public-minded prosecutors may seek senten-
ces that they or their constituents perceive as fair (id., p. 2550). Career-
motivated prosecutors may be more concerned about maintaining a high
win rate than maximizing sentences (Bibas 2004, p. 2471.) This would lead
them to bring easy cases to trial, while offering more generous pleas to the
hardest cases (id., p. 2472). In fact, more than 85 percent of federal criminal
cases that go to trial result in convictions,23 suggesting that prosecutors are
not just trying the hard cases.
Many criminal appeals involve challenges to sentences rather than convic-
tions. Because many of these defendants are represented by court-appointed
counsel, they have incentives to file appeals even if their chances of prevailing
23 See U.S. District Courts—Criminal Defendants Disposed of, by Type of Disposition and Offense,
During the 12-Month Period Ending March 31, 2017, http://www.uscourts.gov/file/22606/down
load (documenting 1580 convictions and 266 acquittals following bench or jury trials for the one-
year period ending on March 31, 2017, representing a conviction rate of 86 percent).
are small (Solum 1988, p. 1745). In fact, criminal appeals constitute more than
20 percent of cases in the circuit courts.24
4.2.3. Litigation Subsidies

Settlement is typically efficient because it avoids litigation costs. If litigation is
subsidized, however, it may be costless or even beneficial to one of the parties.
When this occurs, there may be no incentive to avoid litigation. In some

instances, subsidies for litigation are explicit, such as when criminal defendants
are given court-appointed legal representation. In other contexts, subsidies for
litigation are implicit, such as when litigants benefit from delay (Greenawalt
1990, p. 34). Death row inmates, for example, often use litigation tactics to
postpone execution (Farbman 2019, pp. 1939–1940). In a suit for equitable re-
lief, a litigant who seeks to preserve the status quo may similarly benefit from
delay. The same logic applies to stays pending appeal: the side that is granted a
stay may have negative costs for filing an appeal. Similarly, a petitioner denied
asylum benefits from delay (Marouf, Kagan, & Gill 2014), especially if she is
not detained pending appeal.25
4.2.4. Bargaining Breakdown

The Priest–Klein model assumes that litigants will be able to reach an agree-
ment as long as a settlement window exists. Alternative models of settlement
predict that litigation occurs as a consequence of bargaining breakdown rather
than divergent expectations. Even when a settlement window exists, the parties
must figure out how to divide the surplus from settlement. Each litigant may
behave strategically to maximize its own share of the surplus (Mnookin &
Kornhauser 1979, pp. 972–973). This bargaining may prevent the parties from
reaching an agreement.
To illustrate, suppose that the plaintiff’s minimum settlement demand is
$10,000 and the defendant’s maximum offer is $12,000. Although the Priest–
Klein model predicts that both would accept a settlement between $10,000 and
$12,000, it never explains how the parties figure out that settlement is possible
or how they divide the surplus (Cooter, Marks, & Mnookin 1982, p. 228). For
24 See U.S. Courts of Appeals—Cases Commenced, Terminated, and Pending, by Circuit and Nature
of Proceeding During the 12-Month Period Ending March 31, 2017, http://www.uscourts.gov/file/
22606/download (documenting 9,294 pending criminal appeals out of 40,815 total as of March 31,
2017, representing 23 percent).
25 In principle, a court is not supposed to grant a stay unless the petitioner can demonstrate a likeli-
hood of success on the merits. See Nken v. Holder, 556 US 418, 434 (2009). In practice, however,
the standard for granting a stay is applied inconsistently (Marouf, Kagan, & Gill 2014; Pedro 2018).
Even with a weak claim, a petitioner may meet the threshold for a stay and then have a strong in-
centive to pursue an appeal.
the parties to know that settlement is feasible, each must know the range of set-
tlements that would be acceptable to the other. The parties, however, have no
incentive to disclose the very worst terms they would be willing to accept; to do
so would effectively concede the surplus to the other side. Thus, each party will
have an incentive to bargain aggressively to capture more of the surplus, even if
this increases the risk of going to trial (id., p. 226). In fact, Gross & Syverud
(1991) provide empirical evidence that bargaining breakdown is a substantial

impediment to settlement.
Bargaining breakdowns can occur in easy cases as well as hard ones.
Whenever settlement generates surplus, the parties will have to bargain over
how to divide it. Even if they can predict the outcome of litigation, they may
not know how much the other side is willing to concede. Although many such
litigants will eventually reveal enough information to generate a compromise,
some easy cases will end up going to trial, and possibly appeal, because of bar-
gaining breakdown.
4.2.5. Psychological Factors

All of the previous discussion about the selection of cases for appeal assumes
that litigants are rational economic actors who seek to maximize their expected
payouts. However, litigants are not always concerned solely about payouts, and
parties may depart from rationality when engaged in litigation. Some litigants,
for example, refuse to settle because they are more concerned about vindica-
tion than financial gain (Gross & Syverud 1991, p. 366; Korobkin & Guthrie
1994, pp. 142–143, 147–150; Epstein, Landes, & Posner 2013a, p. 55). Others
refuse to settle out of spite (Mnookin & Kornhauser 1979, p. 974; Gross &
Syverud 1991, p. 372). Litigants’ judgment may be influenced by framing
effects, so that they are more willing to accept settlement offers framed as gains
than losses, even if the probability of prevailing is unchanged (Korobkin &
Guthrie 1994, pp. 129–138).
4.3 Conclusion
The dominant models of settlement suggest that easy cases should be settled
prior to trial, and certainly prior to appeal, leaving only hard cases for appellate
courts. Nevertheless, there are several reasons why easy cases might not be set-
tled. Lumpy outcomes or bargaining breakdowns might prevent a comprom-
ise. Explicit or implicit subsidies for litigation might lead parties to litigate
cases with low probability of success. Parties might be motivated by nonpe-
cuniary factors such as vindication, or perhaps they are just stubborn. Criminal
cases involve different incentives than civil cases, so standard models of settle-
ment do not apply.
The preceding discussion provides reasons why easy cases could appear in appel-
late dockets, but it does not predict how many cases are easy or hard. Some basic
statistics on federal appeals suggest that many of these cases have features that vio-
late the assumptions of the Priest–Klein model. More than half of federal appeals
were filed by prisoners and pro se litigants,26 who are typically unsophisticated and
often motivated by nonpecuniary concerns (Greenwald & Schwarz 2002, p. 1158).
Immigration cases, which involve lumpy remedies, constitute 9 percent of the

docket in the circuit courts,27 while criminal cases constitute roughly 17 percent.28
5. EMPIRICAL EVIDENCE ABOUT EASY CASES
Many circuit judges have claimed that most appellate cases are easy, while
influential economic models predict that only hard cases will end up in ap-
pellate courts. These two perspectives are incompatible with each other and
there are reasons to be skeptical of both. Thus, it is not surprising that
many commentators have looked to empirical evidence to assess the pro-
portion of easy cases. For example, scholars routinely cite statistics such as
dissent rates, the proportion of unpublished dispositions, or different rates
at which Republican- and Democratic-dominated panels reach liberal or
conservative results. Unfortunately, these statistics do not provide much
clarity. Most decisions in circuit courts are unpublished and very few have
dissents; commentators often cite these statistics when claiming that most
cases are easy. Many studies, however, have found stark differences in vot-
ing patterns between panels dominated by Republican and Democratic
judges, suggesting that many more cases are hard. Clearly, these statistics
cannot all be reliable measures. In fact, they are all flawed in different ways,
as the following sections explain.
5.1 Judicial Disagreement: Within Panels vs. Between Panels

Commentators often cite statistics about judicial disagreement in discussions
about the proportion of easy and hard cases. Of course, evidence about judicial
26 http://www.uscourts.gov/statistics-reports/us-courts-appeals-judicial-business-2016
27 See U.S. Courts of Appeals—Cases Commenced, Terminated, and Pending, by Circuit and Nature
of Proceeding During the 12-Month Period Ending March 31, 2017, U.S. Cts., http://www.
uscourts.gov/file/22606/download (reporting that appeals from the Board of Immigration Appeals
constituted 5,215 out of 60,357 cases in the federal circuit courts in 2016 and 3,063 out of 11,473 in
the 9th Circuit).
28 See id. (reporting 10,355 criminal cases commenced in the federal circuit courts out of 58,951 total
cases).
agreement or disagreement does not correspond perfectly with the easy/hard

distinction. Judges may agree even in hard cases if they all have similar ideo-
logical views. Conversely, disagreement does not demonstrate that a case is
hard; one side might be flouting the law. Nevertheless, if judges have heteroge-
neous ideological views and act in good faith, there will be an intuitive connec-
tion between disagreement and hard cases.
Evidence about disagreement in the courts of appeals typically takes one

of two forms. The most common type of evidence is based on within-panel
disagreement, such as dissent rates or rates at which pairs of judges disagree
when participating in the same cases. Other studies examine between-panel
disagreement, which is inferred from rates at which different types of panels
reach different types of decisions. Because the panels are not deciding the
same cases, they are not disagreeing in a literal sense. However, if all-
Democratic panels would reach liberal decisions 30 percentage points more
often than all-Republican panels, then one can infer that these panels would
have disagreed at least 30 percent of the time if they had heard the same
cases.
As a general matter, estimates of within-panel disagreement are much
lower than estimates of between-panel disagreement. For this reason, com-
mentators claiming that most cases are easy often cite within-panel measures,
such as dissent rates. For example, Judge Jon Newman (1984, p. 204)
reported that fewer than 4 percent of federal appellate cases included a dissent
and concluded that most cases were easy. Other judges29 and scholars30 have
made similar claims.
Judge Harry Edwards (1985, p. 629) has similarly cited a 94 percent unanim-
ity rate to rebut the claim that judges are “mere political actors.” He has also
reported statistics on judges’ pairwise rates of disagreement in the D.C. Circuit
when sitting together on panels. He reported that during a one-year period, he
29 Judge Jeffrey Sutton (2010, p. 862) claimed that hard cases are rare in the federal courts of appeals,
observing that “judges dissent in just 3% of the cases.” Judge John Noonan (2002) claimed that
most cases were easy based on his own dissent rate of 3.4%. Judge Theodore McKee (2007, p. 1716)
cited the fact that “the vast majority of appeals result in unanimous decisions” and concluded that
“the vast majority of these cases are fairly clear cut” and “relatively easy to resolve.”
30 Daniel Farber (2000, p. 1430) noted that only 4.5 percent of cases in the 7th Circuit have dissenting
opinions. He observed: “Apparently, federal appellate courts hear many thousands of easy cases in
which there is little ground for disagreement. So much for legal indeterminacy!” Brian Tamanaha
(2009, p. 739) noted the high proportion of unanimous decisions and claimed that judges’ ideo-
logical values do not play much of a role in circuit courts. Kent Greenawalt (1990, p. 34) noted dis-
sent rates of around 4 percent, concluding that judges find “a high percentage” of appellate cases to
be “easy.” Kenneth Kress (1989, p. 324) similarly claimed that low dissent rates “confirm Cardozo’s
intuition” that “no more than ten percent of the cases” have “any indeterminacy or doubt about
the correct outcome.”
agreed with Judge Robert Bork 90 percent of the time and then-Judge Scalia 97
percent of the time (id., p. 630). Interestingly, Edwards reported that Justices
Scalia and Ruth Bader Ginsburg agreed 95 percent of the time while serving on
the D.C. Circuit during that same period (id., p. 644), a rate that coincides
with Obama’s subsequent claim on the Senate floor.31
Comparisons between panels, however, suggest far higher rates of dis-
agreement. In a series of studies, Sunstein et al. (2006, pp. 20–21) docu-

mented large differences in voting behavior between different types of
panels. For example, in federal appeals involving sex discrimination,
Democratic appointees voted in favor of plaintiffs 17 percentage points
more often than Republican appointees. But differences at the panel level
were much larger: all-Democratic panels (“DDD”) favored plaintiffs 46 per-
centage points more often than all-Republican panels (“RRR”).
Studies that examine differences in overall voting rates between Democratic
and Republican appointees also find smaller differences. Epstein et al. (2013a,
pp. 159–160) find that Republican appointees are 6 percentage points more
likely to reach conservative outcomes than Democratic appointees, with very
similar differences in civil criminal cases, constitutional cases, and economic
and labor cases. This difference is small enough that they question “why any
fuss is made in Senate confirmation hearings about the ideological leanings of
nominees for court of appeals judgeships” (id., p. 168).
As a general matter, average differences between Democratic and
Republican appointees will be larger than levels of within-panel disagreement
but smaller than between-panel differences.32 This does not mean, however,
that such statistics represent a happy medium. As the next two sections dem-
onstrate, both within- and between-panel measures of disagreement understate
actual levels of judicial disagreement, so average differences by party will simi-
larly understate such disagreement.
5.2 Problems with Within-Panel Disagreement

One advantage of analyzing within-panel disagreement is that it provides a
controlled comparison; when judges disagree in the same case, they are
responding differently to the same legal sources applied to the same facts. By
31 See supra Note 8 and accompanying text.

32 A well-established result in econometrics holds that a linear regression coefficient can be repre-
sented as a weighted average of between- and within-panel estimators. See Greene (2000, p. 564).
Because the average difference in voting behavior between Democratic and Republican appointees
can be represented as a regression coefficient of votes on party of appointment, the average differ-
ence by party will be a weighted average of within- and between-party differences.
contrast, between-panel comparisons are harder to interpret because different

panels must necessarily decide different cases. However, many studies have
shown that within-panel measures are unreliable indicators of actual disagree-
ment because judges are strongly influenced by their panel colleagues. In an in-
fluential study of cases challenging EPA regulations in the D.C. Circuit, Revesz
(1997) found that judges’ votes were strongly influenced by their panel col-
leagues. He argued that many prior studies underestimated the effect of judicial

ideology on voting because they failed to control for the influence of panel col-
leagues (id., pp. 1719–1720). Cross and Tiller (1998) similarly found that
“divided” panels (combining both Democratic and Republican appointees)
behaved more moderately than “unified” panels (consisting of judges from the
same political party.) They concluded that this moderation was due to strategic
behavior by judges in the panel minority; the threat of a dissent could discour-
age the majority from taking an extreme position. Others have attributed these
interaction effects to deliberation (Boyd, Epstein, & Martin 2010, pp. 391–
392), dissent aversion (Posner 2008, p. 32), or group polarization (Sunstein
et al. 2006, pp. 71–78).
For current purposes, the causal mechanism is irrelevant; it suffices to recog-
nize that judges’ votes within a panel are interdependent. If two judges were
examining the same case in isolation, they might reach different conclusions
about the correct decision. If these same two judges were serving on a panel, how-
ever, they might join a unanimous opinion. One judge might persuade the other,
they might strike a compromise, or one judge might decide that the case did not
warrant a dissent (Ginsburg 1990). Thus, it would be wrong to infer from a
unanimous opinion that all judges would have independently reached the same
decision. If the case had been assigned to an entirely different panel, the decision
could potentially have gone the other way. Thus, dissent rates underestimate the
rate of potential disagreement that would exist among the judges in a circuit.
The same logic applies to rates of disagreement between pairs of judges over
time. As Judge Edwards (1985, p. 644) observed, Justices Ginsburg and Scalia
voted together 95 percent of the time when serving on panels together on the
D.C. Circuit. However, Justice Ginsburg may have voted more conservatively
when she was on a panel with Justice Scalia, and he may have voted more liber-
ally when he was on a panel with Justice Ginsburg. They likely would have
agreed less frequently if they had reached their own conclusions about these
cases independently.
Some commentators who cite dissent rates have acknowledged that they
understate the rates of actual disagreement. Judge Newman (1984, pp. 204–205
n. 8), for example, “concede[d] that the percentage of dissenting opinions is
not necessarily a true reflection of all the occasions on which one member of a
panel disagrees with the panel decision,” because “[a] dissenting opinion and
vote may be withheld for a variety of reasons.” Kress (1989, p. 325) similarly
acknowledged that dissent rates understate true rates of disagreement because
of dissent aversion, but claimed that the effect must be small. He claimed that
“[i]t would be surprising to find that the dissent rate was four percent yet
judges disagreed in forty percent of cases.”
In fact, dissent rates may understate disagreement more severely than these
commentators recognized. Fischman (2015) reported differences in voting be-

havior between Democratic and Republican judges, both between and within
panels. In cases involving immigration, judicial review of administrative agen-
cies, sentencing, and sex discrimination, differences in voting rates between all-
Democratic and all-Republican panels were 6–8 times larger than disagree-
ments between Democratic and Republican judges within panels (id., p. 837).
In the immigration cases, for example, switching from an all-Republican panel
to an all-Democratic panel would increase the likelihood of relief for an asylum
petitioner by 29 percentage points (id.), yet only 5 percent of the cases had dis-
senting opinions (id., p. 823). As discussed in the next section, voting differen-
ces between different types of panels actually understate the extent of judicial
disagreement. Thus, dissent rates and other measures of within-panel disagree-
ment understate true disagreement far more severely.
5.3 Problems with Between-Panel Variation

As discussed above, comparisons between panels often show substantially
larger levels of disagreement than measures based on disagreement within pan-
els. Because judges strongly influence each other’s votes, much of the within-
panel disagreement is suppressed in the final decision. Because there are no
interaction effects between panels, between-panel measures have an important
advantage over within-panel measures.
However, there are two additional problems that arise when examining vot-
ing differences between panels. The first is that each panel will decide different
cases. If the cases are not randomly assigned, it is difficult to learn much from
the panels’ voting behavior. If cases are randomly assigned,33 however, then
each panel will encounter a mix of cases that is a random sample of the entire
docket. Thus, one can estimate the rates at which each panel would reach lib-
eral and conservative results. Using this information, it is possible to estimate
33 Although circuit court cases are often assumed to be randomly assigned, recent research has docu-
mented deviations from random assignment (Chilton & Levy 2015). For the remaining discussion,
it suffices to assume that case characteristics are uncorrelated with judicial characteristics, even if
panel composition is not fully random.
lower and upper bounds on their disagreement rate, but not the precise rate
(Fischman 2014).34
The second problem with comparisons between panels is that the same pan-
els may not appear frequently enough to generate precise estimates of their vot-
ing behavior. Although many judges appear often enough to analyze their
behavior individually, particular three-judge panels do not recur as frequently.
For this reason, many studies compare groups of panels based on the judges’

characteristics, such as the number of Democratic and Republican appointees
(e.g., Revesz 1997; Sunstein et al. 2006).
It is well understood, however, that these party variables are “crude measures
of ideology” (Fischman & Law 2009, p. 170). Justices Harry Blackmun, David
Souter, and Stevens were all Republican appointees, as are Justices Samuel
Alito, Clarence Thomas, and Kavanaugh. If these justices had hypothetically
served on three-judge panels, they would have been coded as all-Republican
panels. Yet there is no doubt that a panel consisting of the latter three justices
would be far more conservative than a panel of the former three.
Thus, the comparisons in such studies reveal the difference in voting propensity
between the average “DDD” panel and the average “RRR” panel. In fact, if the
propensities of the panels could be assessed individually, the differences would be
even greater (Fischman 2014, p. 68). Thus, the “DDD” and “RRR” panels in
Sunstein et al. (2006) would likely disagree even more than 46 percent of the time.
In sum, there are two problems with comparisons between panels, both of
which lead these measures to understate the true rates of disagreement. Because
the panels do not decide the same cases, the differences in liberal voting rate only
provide a lower bound for the rate of disagreement among panels. Measuring
panel ideology using crude proxies such as the number of Democratic and
Republican judges further understates the true rate of disagreement.
However, these statistics measure the rates at which panels would disagree,
not the rate at which they would reasonably disagree. In fact, ideologically
homogeneous panels (such as the “DDD” and “RRR” panels) might be the
most likely to reach extreme, and arguably unreasonable, decisions. Depending
on how one defines reasonable disagreement, the proportion of hard cases
could be lower than indicated by these disagreement statistics. In most circuits,
homogenous panels are also less common than mixed panels, so comparisons
between “DDD” and “RRR” panels might reflect differences between outliers
rather than typical panels.
34 See Section 6.1 for a detailed explanation.

5.4 Publication Rates

Some observers have cited the percentage of cases decided in unpublished dis-
positions as evidence about the proportion of easy cases. In principle, decisions
are supposed to be published when they establish new law or alter existing law.
Decisions do not need to be published when the outcome could have been
decided through a routine and uncontroversial application of existing law.
Although the standards for publication vary by circuit (Serfass & Cranford

2004, pp. 351–357), and there is some disagreement about what these stand-
ards require, many commentators have equated the distinction between pub-
lished and unpublished decisions with the distinction between hard cases and
easy ones.
For example, former District Judge David Levi (2009, p. 1800) wrote that
easy cases—the cases where “most appellate judges will come to the same con-
clusions”—are the ones that “end up in unpublished dispositions.” Citing sta-
tistics showing that more than 80 percent of circuit court decisions are
unpublished, he concluded that most cases in the circuit courts are easy. Judge
Kim McLane Wardlaw (2010, p. 1634) used Cardozo’s definition of an easy
case as the appropriate standard for withholding publication, observing that
87 percent of cases in her circuit were unpublished and therefore easy. Scholars
such as Epstein et al. (2013, p. 55), Sunstein et al. (2006, p. 18), and Solum
(1988, p. 1742) have drawn similar conclusions on the basis of the large pro-
portion of unpublished decisions.
Judge Alvin Rubin (1987) made a similar claim based on cases scheduled for
oral argument. He described the cases decided without oral argument as the
ones that Cardozo said “should not have been appealed” because they “were
destined to be decided one way” (id., pp. 363–364). Rubin reported that in
1986, 64 percent of cases in the 5th Circuit were decided without argument
(id., p. 363 & n. 30), although this proportion has increased in the federal
courts since then (Cleveland & Wisotsky 2012).
The use of publication rates as a measure of easy cases depends, however, on
the assumption that courts correctly identify which cases warrant publication
and which do not. Commentators such as Judge Richard Arnold (1999, p. 224)
and Vladeck & Gulati (2005, p. 1671) have expressed skepticism about this
process. Judge Patricia Wald (1995, p. 1376) has described the criteria for pub-
lication as “vague and infinitely maneuverable,” noting that different judges
apply sharply divergent standards for publication.35 She also reports that
judges sometimes agree not to publish for questionable reasons, such as to
avoid debate or to prevent the decision from being established as precedent
35 Levy (2011, pp. 361–364) also reports that different circuits have wide variation in publication
rates.
(id., p. 1374). Judge Danny Boggs dismisses the claim that unpublished cases
are easy as “self-evidently wrong,” noting that some unpublished decisions
have dissenting opinions (Boggs & Brooks 2000, pp. 21–22).
Empirical studies have also cast doubt on the notion that unpublished cases
are always easy. Merritt & Brudney (2001, p. 107) showed that 2 percent of un-
published decisions had concurring or dissenting opinions. Democratic judges
were also significantly more likely to vote in favor of unions than Republican

judges in unpublished cases, suggesting that political values still play a role (id.,
pp. 107–110). Similarly, Law (2005) found that majority-Democratic panels
were more than twice as likely to grant relief to asylum petitioners as majority-
Republican panels in unpublished decisions in the 9th Circuit. Songer (1990)
also found that Democratic-majority panels were significantly more likely to
support liberal outcomes than Republican-majority panels in unpublished
decisions in two circuits. Indeed, the party composition of the panel influences
the decision whether to publish, with “ideologically homogeneous panels pub-
lish[ing] opinions more frequently than diverse panels” (Grunwald 2018,
p. 766). Thus, empirical measures of disagreement, both within and between
panels, demonstrate that unpublished decisions are not necessarily routine.
5.5 Conclusion
Commentators recognize that empirical evidence is relevant for assessing the
proportion of easy cases. The three most commonly used indicators are dissent
rates, rates of unpublished dispositions, and voting differences between liberal
and conservative panels. Yet these statistics are inconsistent with each other
and each has different flaws. These statistics are easy to calculate and explain,
but they do not have a clear relationship to the proportion of easy cases.
6. EASY AND HARD CASES: A NEW EMPIRICAL FRAMEWORK
This section demonstrates how to derive inferences about the proportion of

easy and hard cases from judicial voting data. Because the analysis combines
objective empiricism with subjective judgment, it requires more careful theo-
rizing than crude measures such as dissent rates. At the heart of the inquiry are
two issues. Section 6.1 addresses what kind of inferences can be drawn from
data on judges’ or panels’ rates of liberal or conservative voting. When cases
are randomly assigned, it is possible to estimate lower bounds on rates of dis-
agreement, even when judges are not deciding the same cases. Although empir-
ical methods alone cannot determine how often disagreement is reasonable, it
can show how disagreement can be decomposed into instances of reasonable
disagreement and instances of clear errors. When coupled with assumptions

about the proportion of clear errors, this approach can estimate an upper
bound on the proportion of easy cases.
Section 6.2 addresses how to predict the behavior of circuit court panels
without relying on crude proxies such as the number of Democratic and
Republican appointees. Relying on a model of “consensus voting” in three-
judge panels, this section demonstrates that it is possible to derive more precise

predictions of panel behavior. Section 6.3 then combines the prior analyses to
demonstrate what can be inferred from panel voting data. First, it is possible to
estimate an upper bound on the proportion of cases that are easy in the sense
that all judges would agree. Similarly, there is a lower bound on the proportion
of cases that are hard, in the sense that some would disagree. It is also possible
to decompose evidence of disagreement into rates of hard cases and clear
errors. This decomposition derives from the fact that whenever judges disagree,
either the case is hard or one judge is clearly wrong. Although the proportion
of hard cases and clear errors cannot be estimated objectively, it is possible to
determine which combinations of these proportions are jointly feasible.
Finally, Section 6.3 illustrates the analysis using two datasets involving appeals
from decisions by the NLRB in the D.C. Circuit and asylum appeals in the 9th
Circuit. Technical details of the estimation procedure are provided in the
Appendix.
6.1 Inferences from Liberal and Conservative Voting Rates

As discussed in Section 5, statistics on voting differences between and within
panels are both unreliable measures of actual disagreement. Although both can
provide lower bounds on disagreement, within-panel comparisons provide
bounds that are too low to be useful. This section discusses how to derive valid
inferences from between-panel comparisons.36
Consider first a highly simplified example with two judges, A and B. A re-
searcher asks both judges how they would decide a sample of 100 cases.37 The
two judges consider the cases in isolation so that they cannot influence each
other’s votes. In this experimental setting, a researcher can see how often
Judges A and B would disagree by simply comparing their answers.
Now suppose that the judges’ answers on the individual cases are kept confi-
dential and the judges only report the number of cases in which they reached
36 The discussion in this section is simplified from Fischman (2014).

37 To keep this experiment properly controlled, we can assume that the judges decide on the basis of
the briefs alone, without oral argument.
liberal results.38 Suppose Judge A voted liberally 10 percent of the time and
Judge B voted liberally 40 percent of the time. Then there must be at least
30 percent of cases in which the two judges would disagree. This is only a lower
bound, however. Suppose that each judge would cast liberal votes in entirely
different cases, so that there was no overlap in their liberal decisions. Then the
two judges would disagree in 50 percent of the cases—those in which one or
the other would reach the liberal result. A researcher who only observes the

judges’ rates of liberal voting, and not their individual votes, could only con-
clude that they disagree between 30 and 50 percent of the time.
This experiment may seem peculiar, but in fact it closely resembles the data
that are available in observational studies of judicial behavior. When Judges A
and B decide different cases, it is impossible to know when they would disagree.
If cases are randomly assigned, however, it is possible to use each judge’s deci-
sions to estimate that judge’s rate of liberal voting in the entire universe of
cases. A comparison of these rates can yield lower and upper bounds on the
judges’ rate of disagreement.39
The next step is to relate evidence of disagreement to the proportion of easy
and hard cases. Suppose that 90 percent of cases are easy and 10 percent are
hard, as many judges have claimed. Some of the easy cases will require a liberal
result and some will require a conservative result. Suppose that x percent of all
cases require a liberal result. If judges always reach the correct results in easy
cases, then every judge should reach a liberal result between x and x þ 10 per-
cent of the time. Thus, the judges’ rates of reaching liberal decisions should not
vary by more than 10 percentage points. This would be a testable implication
of the claim that 90 percent of cases are easy and judges are always correct in
these cases.
Of course, it is too much to assume that judges are always correct in easy
cases. Thus, in order to infer a proportion of hard cases from voting data, one
must account for some proportion of clear errors. Suppose that Judges A and B
disagreed at least 30 percent of the time in the above example. Different observ-
ers could reasonably draw different conclusions from this disagreement. Anna
might believe that at least 30 percent of cases are hard. Brian might think that
Judge A was always correct and Judge B was clearly wrong 30 percent of the
time. Carla could believe that 10 percent of the cases are hard and each judge
38 The entire discussion will refer to “liberal” and “conservative” decisions without ever defining what
these terms mean. In many contexts, the meaning of these concepts will be contestable. For current
purposes, it is not necessary to flesh out a precise definition. The analysis will be valid as long the
definitions of “liberal” or “conservative” votes are applied consistently.
39 In mathematical terms, if Judges A and B vote in favor of plaintiffs at rates rA and rB , respectively, then
the lower bound on disagreement is rA rB and the upper bound is minfrA þ rB ; 2 rA rB g
(Fischman 2014, p. 49).
was clearly wrong 10 percent of the time. Although the data cannot distinguish
among these possible interpretations, it is possible to rule out some interpreta-
tions. It would be incorrect to claim that all cases are easy and both judges are
always following the law; if this were so, they would never disagree.
This reasoning can apply to any definition of “easy” as long as it is paired
with the appropriate notion of clear error. Consider first the individual con-
ception. Suppose that Daniel has a view about which cases are easy and hard. If

two judges disagree in a case that Daniel thinks is easy, then one of them must
be reaching a result that Daniel thinks is a clear error. Evelyn may disagree with
Daniel about which cases are easy, but the same reasoning applies to Evelyn’s
view of easy cases and clear errors.
Suppose that members of a community believe that a case is easy in the
sociological sense, so that all community members agree that the case is easy
and also agree on the correct outcome. Then if two judges disagree about the
outcome in that case, the community members would agree that one judge is
making a clear error. Different communities may have different perceptions of
easy cases and clear errors, but the reasoning still applies as long as both defini-
tions correspond to the same community. The same conclusions apply to pan-
els as well as to individual judges; if two panels disagree, then the appropriate
observer must believe that either the case is hard or that one of the panels has
reached a clearly wrong result.
This approach does not objectively measure the proportion of easy and hard
cases, but it allows observers to combine empirical data on judicial voting be-
havior with intuitive judgments about the proportion of clear errors. An obser-
ver who denies or minimizes the existence of hard cases must potentially
acknowledge a high proportion of clear errors. Conversely, one who claims
that judges rarely make clear errors (e.g., Edwards 1983–1984, p. 402) must ac-
knowledge a minimum proportion of hard cases.
Hard cases and clear errors are both central to debates about judicial legitim-
acy. A large proportion of hard cases could raise concerns about widespread
legal indeterminacy, suggesting that democratically unaccountable judges are
exercising excessive policymaking authority. Similarly, while judges may rea-
sonably disagree in hard cases, clear errors in easy cases constitute abuse of au-
thority (Fallon 2018, pp. 38–40). Of course, both hard cases and clear errors
must be evaluated in the context of the body of litigated cases.
The above discussion focused on individual judges, but the same reasoning
can be applied to judicial panels. A dataset of asylum cases from Law (2005)
provides a helpful illustration. These cases include 1892 asylum appeals in the
9th Circuit from 1995 to 2001, with decisions coded as liberal if they provided
some relief to the asylum petitioner.
Figure 1. Rates of liberal voting in 9th Circuit asylum cases by panel type, 1995–
2001

Source: Law (2005)
Table 1. Hypothetical minimum rates of clear error in 9th circuit asylum cases by
panel type, based on the assumption that “RRD” panels are always correct
Panel composition Proportion of Liberal voting Minimum error

panels (percent) rate (percent) rate (percent)
RRR 15 6 6
RRD 38 12 0
DDR 36 23 11
DDD 11 35 23
Weighted average 18 7.5
Figure 1 shows rates of liberal decisions for different types of panels based
on the number of Democratic and Republican appointees. Since each case was
only assigned to a single panel, these four types of panels were deciding differ-
ent subsets of cases. To simplify the example, assume that the liberal voting
rates reported in Figure 1 would also apply for each panel type to the entire set
of cases. All-Republican panels reach liberal results 6 percent of the time, while
all-Democratic panels reach liberal results 35 percent of the time, so these pan-
els would disagree at least 29 percent of the time. If the judges on these panels
are always competent and behaving reasonably, then at least 29 percent of these
cases must be hard, and at most 71 percent could be easy.
At the other extreme, if all of these cases are easy, then some of these panel
decisions would be clear errors. Although it is impossible to know exactly how
often this occurs, it is possible to construct a lower bound on such errors,
which is achieved when the median panel is always correct (Fischman 2014,
p. 63). In this example, the median panel is an “RRD” panel.40 Under the as-

sumption that “RRD” panels are always correct, the “RRR” panels would be
making clear errors at least 6 percent of the time. Similarly, “DDR” panels
would make clear errors at least 11 percent of the time and “DDD” panels at
least 23 percent of the time. These lower bounds for all panel types are shown
in Table 1. Taking the weighted average across groups, at least 7.5 percent of
the panel decisions must be clear errors.
Of course, there was no basis to believe that “RRD” panels were always cor-
rect; it was simply a convenient assumption for calculating the lower bound on
clear errors. If this assumption is violated—as is likely—then the rate of clear
errors will be higher. Suppose, for example, that the “DDD” panels were always
correct, so that 35 percent of cases warranted a liberal decision. By the same
logic, at least 16.8 percent of the panel decisions would be clear errors.41 In
practice, it is likely that every type of panel will be wrong at least some of the
time. But if all cases are easy, then no scenario can yield a rate of clear errors
below 7.5 percent.
The above analysis showed two ways of decomposing the evidence of dis-
agreement in the example taken from the NLRB cases. If judges are always
competent and acting in good faith, then at least 29 percent of the cases are
hard. If all cases are easy, then at least 7.5 percent of decisions are clear errors.
These examples are simple but also the least plausible; many commentators
would expect disagreement to result from some combination of hard cases and
clear errors. In fact, there is a continuum of ways that the evidence of disagree-
ment can be decomposed.
Fischman (2014) developed a model for estimating legal indeterminacy and
error from voting rates of individual judges. In the current context, the concept
of “indeterminacy” corresponds to “epistemic indeterminacy,” that is, hard
cases where the outcome is not knowable to an interpreter or interpretive com-
munity. Similarly, “error” corresponds to “clear error,” instances where judges
are defying the law in easy cases. Based on the simplified assumptions made
40 As shown in Table 1, 15% of all panels are RRR and 38% are RRD, so the 50th percentile panel
would be in the RRD group.
41 0:15ð0:35 0:06Þ þ 0:38ð0:35 0:12Þ þ 0:36ð0:35 0:23Þ þ :11ð0:35 0:35Þ ¼ 0:168
Figure 2. Feasible combinations for proportion of hard cases and clear errors in 9th
Circuit asylum cases, based on voting rates for RRR, RRD, DDR, and DDD panels

Note: Solid line represents point estimates for minimum level of clear errors for any given pro-
portion of hard cases.
above, it is possible to construct a curve that identifies the feasible combina-

tions of hard cases and clear errors, which is shown in Figure 2.
The intercepts of the curve match the examples discussed above. If all cases
are easy, then at least 7.5 percent of decisions are clear errors, corresponding to
the intersection of the curve with the vertical axis. If there are no clear errors,
then at least 29 percent of cases are hard, corresponding to the point where the
curve intersects the horizontal axis. The intermediate points are calculated by
considering any percentage of hard cases and finding parameters that minimize
the number of clear errors. To consider a simple example, suppose that exactly
11 percent of cases are hard. The number of clear errors will be minimized
under the assumption that “RRD” and “DDR” panels are always correct. Thus,
there would be exactly 12 percent of cases that clearly require a liberal vote and
77 percent that clearly require a conservative vote. The remaining 11 percent
would be hard cases where “RRD” and “DDR” panels could reasonably dis-
agree. Then “RRR” panels would be wrong at least 6 percent of the time, and
“DDD” panels would be wrong at least 12 percent of the time. Under these
assumptions, at least 2 percent of cases would be clearly wrong.42
This analysis compared panels based on the number of Republican and

Democratic appointees. This is common practice in research on circuit courts
because there are too many possible three-judge panels in a given circuit to
compare individually. The shortcoming of this approach is that these proxies
are crude predictors of judicial voting behavior.
A closer look at the data reveals substantial differences in voting behavior
among the judges in each party. Compare two Democratic appointees: Judge
Stephen Reinhardt and Judge Jerome Farris. Reinhardt favored asylum peti-
tioners 62 percent of the time (71 out of 115 cases), while Ferris favored peti-
tioners only 4 percent of the time (5 out of 131 cases). A “DDD” panel could
have a majority of judges like Reinhardt or like Farris. There are similar differ-
ences among Republican appointees: Judge Alfred Goodwin favored asylum
petitioners 23 percent of the time (36 out of 160 cases), whereas Judge
Diarmuid O’Scannlain did so only 4 percent of the time (8 out of 208 cases).
These differences in voting behavior are striking, but they still understate the
true differences among the 9th Circuit judges. Because the judges’ votes are mod-
erated by their panel colleagues, each judge’s rate of liberal voting will be pushed
toward the center. Each judge will interact with both liberal and conservative
judges over time, but these influences will not fully balance each other out. A lib-
eral judge such as Reinhardt, for example, would often sit with colleagues who
were more conservative and seldom (if ever) with colleagues who were more lib-
eral. His 62 percent liberal voting rate includes cases in which conservative col-
leagues pulled him in a more conservative direction. Thus, he would likely be
even more liberal if he were unconstrained on a “DDD” panel. Conservative
judges will similarly be pulled toward the center by panel colleagues.
6.2 Predicting Voting Behavior in Panels

Judges’ liberal voting rates may be useful to provide a rough comparison, but they
do not provide a way to predict how a particular three-judge panel will decide a
case. Accounting for the full variation in judges’ voting behavior will naturally pro-
vide more informative estimates of hard cases and clear errors. This section uses a
42 0:15ð0:06Þ þ 0:11ð0:12Þ ¼ 0:0222

Figure 3. Representation of three judges in one-dimensional spatial model

Figure 4. Unanimous decision when all judges agree
Figure 5. Unanimous decision when judge a suppresses dissent
model of “consensus voting” (Fischman 2011) based on spatial voting models

commonly used in political science, to estimate ideology scores for circuit judges
while accounting for the influence of colleagues in each case. This model can then
aggregate these ideology estimates to predict voting behavior for each panel.
The model maps the judges onto a one-dimensional issue space. Figure 3
depicts three judges, denoted A, B, and C, where Judge A is the most conservative
and Judge C is the most liberal. Each potential case would also be represented as a
“cut point” on the spectrum. All judges to the left of the cut point would prefer
the conservative outcome, while judges to the right would prefer the liberal out-
come. A judge’s location on the spectrum represents the judge’s “indifference
point.” Each judge will support the conservative outcome whenever the cut point
is to the right of the judge’s indifference point and the liberal outcome when the
cut point is to the left. The judge is precisely indifferent between the two out-
comes when the cut point coincides with the judge’s indifference point.
If a case had a cut point to the left of Judge A, as depicted in Figure 4, then
all three judges would vote unanimously in favor of the liberal outcome.
Similarly, a cut point to the right of Judge C would result in a unanimous con-
servative decision. If a case had a cut point between Judges A and B, then the
Figure 6. Dissent by Judge A

judges would disagree about the best outcome. Judge A would prefer the con-
servative side, while Judges B and C would prefer the liberal side. If the judges
did not influence each other, then this scenario would result in a 2–1 decision
in favor of the liberal side.
To account for the influence within panels, the model incorporates a utility
“cost” for issuing a dissent. In considering whether to dissent, a judge will im-
plicitly weigh the cost of dissenting against the cost of voting against the judge’s
preferred outcome. The costs of dissent reflect a variety of factors, such as the
effort to write a dissent and the possibility of impairing relationships with col-
leagues.43 The model assumes, for simplicity, that the cost of dissenting is the
same for all judges.
A judge who withholds a dissent also incurs a disutility from going along
with a decision the judge would prefer not to support. This disutility is repre-
sented by the distance between the judge’s indifference point and the case cut
point, so that a stronger disagreement generates a larger disutility. When this
distance is smaller than the cost of dissent, as depicted in Figure 5, the judge
will withhold the dissent. Thus, a judge will withhold a dissent when the judge
only weakly opposes the majority position. On the contrary, the judge will issue
a dissent when this distance exceeds the cost of dissent, as in Figure 6.
The judges’ indifference points are estimated using the method of maximum
likelihood; see Appendix A for details. Given the actual votes in circuit court
cases, the statistical algorithm finds the indifference points and cost of dissent
that best explain the voting coalitions that occurred in the data. As a general
matter, if a judge participates more often in panels that reach liberal decisions,
the the judge will have a more liberal indifference point. However, the model
will control for the influence of panel colleagues across cases. When a panel
with two extreme liberals reaches a liberal result, the inference on the third
judge will be weak, since that judge may be going along rather than sincerely
agreeing with the majority. On the contrary, a judge’s vote will be more in-
formative when the judge is most likely to be pivotal, such as when the judge is
43 See Section 5.2.

empaneled with one extremely liberal and one extremely conservative

colleague.
The cost of dissent is identified by the influence that judges have on their
colleagues’ votes. The model will estimate a larger cost of dissent when these
influences are stronger. The location of the cut points is estimated using case
characteristics, primarily the year of decision and type of case, where relevant.
Because of random assignment, any unobserved influences on the cut point

will be uncorrelated with the judges’ indifference points. Every panel will face
the same probability of encountering an easy case or a hard case or encounter-
ing a strong case for the plaintiff or defendant.
6.3 Inferences from Panel Voting Behavior

Section A demonstrated how to derive inferences about hard cases and clear
errors from voting data on panels, categorized by the number of Democratic
and Republican appointees. Section 6.2 described a more refined model for
analyzing panel voting behavior. This section combines the insights from these
sections. Using judicial indifference points estimated using the consensus vot-
ing model, we can generate more precise voting probabilities for individual
panels, which can in turn generate more informative bounds on the proportion
of hard cases and clear errors.
The analysis in Section 6.1 relies on few assumptions. As long as one has accur-
ate rates of liberal and conservative voting for each panel type, then the bounds for
hard cases and clear errors are correct. By contrast, the consensus voting model
discussed in Section 6.2 relies on many unverifiable assumptions. Most notably, it
assumes that a one-dimensional issue space can characterize the judges’ voting be-
havior. For ease of estimation, the case cut points follow a normal distribution and
the cost of dissent is the same for all judges and across all cases.
Although these assumptions allow the model to reasonably approximate panel
voting behavior, they are unlikely to be true in practice. However, the estimated
lower bounds may still be valid even if these assumptions are not precisely satis-
fied. It is only necessary for the consensus voting model to be well-calibrated, so
that it generates accurate estimates of the probabilities that particular panels will
reach liberal and conservative outcomes.44 This is difficult to verify for particular
panels; a hypothesis that a particular panel will reach liberal results 38 percent of
the time cannot be falsified if that panel is only observed deciding a single case. It
44 The lower bounds will be valid if the predicted probabilities are correct for each individual case, or
if the cases are partitioned into subsets and each case is assigned the average predicted probability
for the subset (Fischman 2014, p. 68). Thus, analysis based on cases divided into “RRR,” “RRD,”
“DDR,” and “DDD” panels will generate valid bounds, although they will be less informative than
the bounds generated using the consensus voting model.
Table 2. Summary statistics
Percent liberal decisions

Type of Circuit Dates Number Percent
cases of cases nonunanimous All R D RRR RRD DDR DDD
Asylum 9th 1995–2001 1892 5 18 12 25 6 12 23 35
NLRB D.C. 2003–2018 222 9 62 55 69 48 60 70 73

EPA D.C. 2003–2018 158 11 61 55 68 51 58 72 33
NLRB þ EPA D.C. 2003–2018 380 10 61 55 69 49 59 71 67
is easier to test calibration, however, for groups of panels. The examples below
show how to verify that the model is well calibrated for particular datasets.
6.3.1. Data
The following sections illustrate the approach discussed above using two data-
sets. The first dataset involves the 9th Circuit asylum cases discussed previously
(Law 2005). Prior studies of these cases also reported large ideological differen-
ces among the judges, despite the fact that most of them were unpublished
(Law 2005; Fischman and Law 2009; Fischman 2011, 2015). The data consist of
1,892 cases, of which 18 percent resulted in a liberal decision. There were dis-
sents in 5 percent of the cases, and only 8 percent generated a published
opinion.
The second example involves a set of appeals from the EPA and NLRB in the
D.C. Circuit Court of Appeals between 2003 and 2018.45 These cases tie closely
to the discussion of judicial perspectives because they include Judge Edwards,
Justice Kavanaugh, and Chief Justice Roberts, all of whom have made promin-
ent claims about neutral judging or the limited scope of judicial discretion.
These cases also provide a comparison with a pair of prominent studies by
Miles and Sunstein (2006, 2008a) that also analyzed appeals from the EPA and
NLRB.
The D.C. Circuit data include 222 NLRB cases and 158 EPA cases, for a total
of 380. Votes in EPA appeals were coded as liberal if they were in favor of an
environmental group or opposed to an industry group or business challenging
an environmental regulation. Votes in NLRB appeals were coded as liberal if
they were in favor of organized labor. Some of the cases involving multiple
45 These data were generated in an earlier study, joint with Kevin Cope, that evaluated the voting re-
cord of Justice Brett Kavanaugh when he served on the D.C. Circuit. See Kevin Cope & Joshua
Fischman, It’s Hard to Find a Federal Judge More Conservative than Brett Kavanaugh, WASH. POST
(Monkey Cage Blog), September 5, 2018.
Figure 7. Consensus voting model calibration tests

issues were coded as fractional outcomes, such as one-half liberal and one-half
conservative.
Table 2 provides summary statistics. In the D.C. Circuit data, there are just
over 60 percent liberal decisions in both NLRB and EPA cases, perhaps result-
ing from deference to liberal agency decisions challenged by conservative inter-
ests. By contrast, only 18 percent of the 9th Circuit asylum cases are liberal.
There were dissents in 5 percent of the 9th Circuit cases and in 10 percent of
the D.C. Circuit cases.
Differences in voting behavior by party were similar for all types of cases.
Democratic appointees were 13 percentage points more likely than Republican
appointees to reach liberal results in the asylum and EPA cases, and 14 percent-
age points more likely in the NLRB cases. Differences between “DDD” and
“RRR” panels were larger in the 9th Circuit cases, although this comparison
may be misleading due to the small number of “DDD” panels in the D.C.
Circuit data. Differences between “DDR” and “RRR” panels were slightly larger
in the D.C. Circuit cases.
6.3.2 Consensus Voting Model Calibration

The consensus voting model generated indifference point estimates for the
judges in both datasets from their voting behavior. The model used year
dummy variables as controls; for the D.C. Circuit data, it also included a
dummy variable indicating whether the appeal was from the EPA or the NLRB.
The model then generated predictions for how often each judge and each panel
would reach a liberal result. These predictions are more precise than using
crude categories such as party of appointment or “RRR” and “DDD” panels
Figure 8. Estimated liberal voting preference for 9th Circuit judges in asylum cases,
1995–2001

Notes: Black dots represent Republican appointees; white dots represent Democratic appoint-
ees. Error bars represent 95 percent confidence intervals, calculated using subsampling.
because they account for the proclivities of individual judges and the influences
of panel colleagues.
To test that the model is well-calibrated, the panels in each dataset were
sorted into bins based on the predicted probability of a liberal result. If the
model is well-calibrated, the average predicted probability in each bin should
be similar to the proportion of cases that resulted in a liberal outcome. Figure 7
displays these comparisons in two calibration plots with five bins.
In both datasets, there is a close correspondence between the proportion of
liberal decisions and the average predicted probability for every bin. In the 9th
Circuit asylum cases, 48.8 percent of cases in the top quintile have a liberal re-
sult, whereas the average predicted probability is 49.5 percent. Similarly, the
bottom quintile has 1.1 percent liberal results and a 2.1 percent average pre-
dicted probability. In the top decile of D.C. Circuit cases, the average predicted
probability is 73 percent, while 71 percent of these cases result in a liberal
Figure 9. Feasible combinations for proportion of hard cases and clear errors, based
on estimates from consensus voting model, asylum cases in 9th Circuit, 1995–2001

Notes: Solid line represents point estimates for minimum level of clear errors for any given pro-
portion of hard cases. Dashed line represents 95 percent confidence curve.
decision. In the bottom decile, the average predicted probability of a liberal de-
cision is 45 percent, whereas 42 percent actually result in a liberal decision.
The model calibration can be tested statistically using a chi-square test; see
Appendix B for details. For both datasets, the differences between the average
predicted probability and the proportion of liberal decisions are statistically in-
significant. The close fit in both datasets provides some assurance that the
probabilities predicted by the consensus voting model are reasonably accurate.
6.3.3 Asylum Cases in the 9th Circuit

Some of the judges in the 9th Circuit vote in the liberal direction far more often
than others. Nevertheless, the differences in voting rates understate the true
levels of disagreement because the extreme judges are still sometimes compro-
mising with more moderate colleagues. The consensus voting model predicts
how often each judge would prefer the liberal outcome if voting without the in-
fluence of panel colleagues. This provides estimates for the proportion of cases
in which judges would have genuinely different views about the correct result.
These probabilities for the 9th Circuit judges are displayed in Figure 8.
There are stark differences between the judges on the extremes of the 9th

Circuit. On the conservative side, Judges J. Clifford Wallace, Jerome Farris, and
Diarmuid O’Scannlain would favor asylum petitioners less than 2 percent of
the time if they were voting independently. On the liberal side, Judge Stephen
Reinhardt would favor asylum petitioners more than 90 percent of the time. If
all of these judges are reasonable, then at least 88 percent of these cases are
hard and at most 12 percent are easy.
Many observers may object to the premise that all 9th Circuit judges are al-
ways reasonable or the conclusion that 90 percent of the cases are hard.
Figure 9 provides a graph that interprets the data more flexibly, displaying
combinations of rates of hard cases and clear errors that are feasible given the
judges’ votes. The solid line is the point estimate of the lower bound on the
error rate, while the dashed line represents a 95-percent confidence curve for
the lower bound. Some panels would disagree in at least 85 percent of the cases,
corresponding to the point where the 95 percent confidence curve meets the
horizontal axis. If panels were always reasonable, then at least 85 percent of the
asylum cases would be hard. This level of disagreement is far higher than the 5
percent dissent rate in these cases or the 8 percent publication rate. These con-
clusions, however, are largely driven by extreme panels, which occur infre-
quently and may well be acting unreasonably. If 1 percent of decisions are clear
errors, then it is possible that only 41 percent of cases are hard. Note that these
lower bounds are much higher than those displayed in Figure 2, which used
the same data but only considered the partisan composition of the panel;
accounting for the individual proclivities of each judge generates more inform-
ative bounds. Analyzing the judges’ voting behavior individually yields much
sharper conclusions on hard cases and clear errors.
At least one judge has suggested that the conventional wisdom on easy cases
applies to asylum cases in the 9th Circuit. In a 2004 address at the Annual
Conference of the Board of Immigration Appeals, 9th Circuit Judge Michael
Hawkins claimed that “about 90% of the appeals from the BIA present few dif-
ficulties for him and his colleagues,” while “10% are more problematic.”
(quoted in Grant 2006, p. 961). If one interprets Hawkins’s claim to mean that
90 percent of cases were easy (either to Hawkins or to the entire community of
judges), then the curve in Figure 9 shows that at least 7 percent of these deci-
sions were clearly wrong (according to the same perspective).
Figure 10. Estimated liberal voting preference for D.C. circuit judges in EPA and
NLRB cases, 2003–2018

Notes: Black dots represent Republican appointees; white dots represent Democratic appoint-
ees. Error bars represent 95 percent confidence intervals, calculated using subsampling.
The asylum cases also illustrate an important point: unpublished decisions

were not necessarily easy cases. Ninety-two percent of these cases are unpub-
lished. If all of these were easy cases, then at least 7 percent of the asylum deci-
sions were clear errors. This finding is potentially troubling because
unpublished opinions are often justified on the ground that these are easy cases
that do not make new law. The relatively high rate of judicial disagreement sug-
gests that many of these cases warranted closer judicial attention, especially be-
cause many losing asylum petitioners were deported following this abbreviated
appellate review.
6.3.4 NLRB and EPA Cases in the D.C. Circuit

The indifference point estimates from the consensus voting model predict how
often each D.C. Circuit judge would prefer the liberal result in the NLRB and
EPA cases if they were unaffected by panel colleagues. These estimates are dis-
played in Figure 10. Most Democratic appointees have a higher propensity to
Figure 11. Feasible combinations for proportion of hard cases and clear errors, EPA
and NLRB cases in D.C. circuit, 2003–2018

Notes: Solid line represents point estimates for minimum level of clear errors for any given pro-
portion of hard cases. Dashed line represents 95 percent confidence curve, computed using
subsampling; see Appendix C. for details.
reach liberal outcomes than their Republican colleagues, but there is some
overlap in the middle of the spectrum. Edwards and Roberts are near the center
of the D.C. Circuit, while Kavanaugh is at the conservative end of the spec-
trum. Judges Cornelia Pillard and Sri Srinivasan are estimated to be the most
liberal; each would reach liberal results more than 80 percent of the time if un-
constrained by panel colleagues. On the conservative end, Kavanaugh and
Judge Laurence Silberman would cast liberal votes around 40 percent of the
time if unconstrained, so the most liberal and the most conservative judges
would disagree roughly 40 percent of the time. Thus, at most 60 percent of
cases are easy in the sociological sense, if the community is comprised of D.C.
Circuit panels and we assume that they always follow the law in easy cases.
However, these large differences among individual judges do not translate
into high levels of hard cases or clear errors. The most extreme judges are sel-
dom pivotal in panels. Occasionally, a panel may include two or more judges
from the same extreme, but even these panels may often decide correctly.
There may be up to 60 percent easy cases where even extreme panels would

agree and also some hard cases where either outcome could be considered
reasonable.
Figure 11 plots feasible combinations of hard cases and clear errors, given
the predicted voting behavior for all panels in the data. Despite the differences
between the most liberal and conservative judges, the lower bound on clear
errors is negligible when the proportion of hard cases exceeds 15 percent. If
90 percent of cases are easy, then one can conclude with 95 percent confidence
that at least 1 percent of decisions are clear errors.
The analysis in Figure 11 provides some perspective on contentious debates
about judicial ideology in the D.C. Circuit (Revesz 1997, 1999; Edwards 1998).
The results for NLRB and EPA cases are at least plausibly consistent with Judge
Edwards’s description of judging on the D.C. Circuit. Edwards (1983–1984,
p. 390) has estimated that 85–95 percent of cases are easy in the sense of having
a knowable answer. He has also observed that a “few judges occasionally may
be unwilling to abide by established law because of personal disagreements
with it” but that “the number of judges and cases affected by this affliction is
small” (id., p. 402). Edwards did not offer a precise estimate of the number of
clear errors, and he was not referring specifically to NLRB and EPA cases.
Nevertheless, a lower bound of 1 percent seems consistent with Edwards’s ac-
count. Of course, the proportions of hard cases and clear errors may be far
higher than the lower bound. Nevertheless, the voting data by itself does not re-
fute Edwards’s account of judging in the D.C. Circuit.
The claims by Chief Justice Roberts and Justice Kavanaugh to be “umpires”
and “neutral arbiters,”47 however, look very different in light of Figure 11.
Such assertions presuppose that all cases have a knowable right answer that
neutral judges could follow. If all cases are easy in this sense, then the D.C.
Circuit must have made clear errors in at least 4.4 percent of cases, as indicated
by the intersection of the vertical axis and the 95 percent confidence curve.
These rates of clear errors are merely lower bounds, which will be achieved
only if two conditions are satisfied. First, the voting behavior of the panels
must be monotonic, meaning that the panels must always be ideologically
47 See supra Notes 3,6 and accompanying text.

consistent (Fischman 2014, p. 65). Under monotonicity, if panel X would

sometimes reach a more liberal result than panel Y, then panel X would reach a
result at least as liberal as panel Y in every case (id., p. 50). Monotonicity can-
not be verified objectively because one can never observe panels X and Y decid-
ing the same case. Nevertheless, monotonicity may seem plausible in contexts
where hard cases are decided by reference to unidimensional political values,
so that the judges and cases fit perfectly on a left-right axis.48

On the contrary, disagreements in hard cases may also stem from a multipli-
city of factors, such as different approaches to interpreting texts or applying
deference. Evidence of two-dimensional voting in the Supreme Court
(Fischman 2020) suggests that monotonicity could also be violated in circuit
courts. Judge Edwards has challenged the assumption that judges occupy a left-
right axis, calling it “obviously questionable” (Edwards and Livermore 2009,
p. 1916). If so, then the true rate of clear errors would exceed the lower bound,
including Edwards’s own estimates regarding the D.C. Circuit.
The second condition for achieving the lower bound is harder to describe
precisely. If all cases are easy, then the median judge must always be correct
(Fischman 2014, p. 63). Otherwise, the judges in the middle of the ideological
spectrum must be correct in the easy cases. This assumption could be reason-
able from the perspective of Judge Edwards and Chief Justice Roberts, who are
both plausibly near the median. It is more problematic, however, for
Kavanaugh’s claims to be neutrally following the law. As Figure 10 shows,
Kavanaugh was close to the conservative end of the spectrum. The model esti-
mates that if all cases had knowable answers that coincided with Kavanaugh’s
ideological views, then at least 16.6 percent of the D.C. Circuit decisions were
clear errors.49 Claiming to be neutral cannot eliminate the fact of judicial dis-
agreement. If judicial nominees claim that they are neutrally applying settled
law, then they are implying that their colleagues are flouting settled law, pos-
sibly in a substantial proportion of cases.
This analysis also provides a comparison with two prominent studies by
Thomas Miles and Cass Sunstein (2006, 2008a) that examined ideological
influences in judicial review of NLRB and EPA decisions. These studies, which
encompassed all circuits, reported similar differences between Democratic and
48 The consensus voting model imposes a weak form of monotonicity by assumption because it relies
on a one-dimensional spatial model. However, it allows each judge’s indifference point to vary by
an error term in each case, thus allowing for violations of monotonicity in particular cases.
Moreover, the model may still be well-calibrated even if the issue space is multidimensional. If so,
the lower bound on clear errors would be valid but the true rate of clear errors would exceed the
lower bound.
49 The point estimate for the lower bound is 21.1 percent; the 95 percent confidence lower bound is
16.6 percent. See Appendix C and D for details.
Republican appointees: 17 percentage points in cases reviewing agency inter-

pretations of law (Miles and Sunstein 2006, p. 859) and 13 percentage points in
cases reviewing agencies on arbitrariness grounds (Miles and Sunstein 2008a,
p. 794), compared to 14 percentage points in the D.C. Circuit data examined
here. However, Miles and Sunstein found much larger differences between
“DDD” and “RRR” panels: 42 percentage points in cases involving interpret-
ation of law (Miles and Sunstein 2006, p. 862) and 29 percentage points in ar-

bitrariness cases (Miles and Sunstein 2008a, p. 793).
Miles and Sunstein struggled to interpret their findings in part because parti-
san differences in voting behavior do not correspond to meaningful legal con-
cepts. On the basis of the large differences between “DDD” and “RRR” panels,
they concluded that “the rule of law is affronted” (2008b, p. 841). Yet they
interpreted the smaller differences at the individual level to show that “the law
. . . seems to be having a constraining effect” (id., p. 844). Ultimately, they
offered a muddled conclusion: “The glass is half empty, perhaps, but it is also
half full.” (id.)
By contrast, the methodology developed here relates the voting behavior to
more intuitive concepts: hard cases and clear errors. Figure 10 shows that
there are substantial differences between the judges at opposite ends of the
ideological spectrum; these judges might disagree in principle 40 percent of
the time or more. In practice, however, three-judge panels moderate the influ-
ence of these extreme judges. The results in Figure 11 show that the data are
consistent with 10 percent hard cases and 1 percent clear errors. Although
these are merely lower bounds, the evidence hardly establishes that “the rule
of law is affronted.”
6.3.5 Conclusion
This method of analysis calculates lower bounds but cannot tell us precisely
how many cases are easy. This should not be surprising given that the easiness
of a case is inherently subjective. Nevertheless, this approach clarifies which ob-
jective conclusions can be drawn from the voting data and which are based on
speculation or unverifiable assumptions.
The two datasets appear similar when analyzed using conventional methods
that compare the voting behavior of Democratic and Republican appointees.
However, the analysis here finds notable differences between the two datasets.
The D.C. Circuit data are consistent with the dominant narrative that most
cases are easy. It is plausible that 10 percent of the D.C. Circuit cases are hard
and only 1 percent are clear errors. By contrast, the 9th Circuit data show
much higher rates of hard cases or clear error. Judging appears to be much less
constrained in these cases, even though most decisions are unpublished.
7. IMPLICATIONS
The analyses of these two sets of cases were intended primarily as illustrations.
It would be premature to speculate about the proportion of easy cases and clear
errors in other circuits, types of cases, or time periods. Furthermore, cases that
are litigated and appealed are not representative of all disputes that arise. The
proportion of easy cases in appellate courts may be very different from trial

court decisions, lawsuits that are settled, or disputes that never lead to litiga-
tion. Surely, many unlitigated disputes would result in easy cases.
In the 9th Circuit asylum data, most cases were decided with unpublished
opinions treating the legal and factual issues as straightforward. However, the
analysis of the voting data revealed high rates of hard cases or clear errors, sug-
gesting that the panels were overconfident about the easiness of many of the
cases. To the extent that this phenomenon generalizes to other appellate con-
texts, there are several important implications. To the extent that judges over-
state the degree to which their opinions are determined by traditional legal
sources (Altman 1990, p. 296; Eskridge & Frickey 1990, p. 379; Posner 1995,
p. 1441), their opinions would conceal the role of contestable value judgments.
As a result, there may be a substantial divergence between the rationales stated
in judicial opinions and the actual reasons that motivated the decisions.
Of course, litigated cases are not representative of all disputes that may arise.
However, to the extent that unlitigated disputes are settled in the shadow of
the law, the divergence between the actual reasons for judicial decisions and
their stated rationales would extend to these unlitigated cases as well (Schauer
2013, p. 768). When litigants settle their disputes “in the shadow of the law”
(Mnookin & Kornhauser 1979), they are actually settling in the shadow of their
expectations about the court’s judgment. If litigants recognize that contestable
values play a large role in litigated cases, this will affect which cases go to trial
and how the remainder are settled. Thus, some areas of the law may be less
clear and more value-laden than many judges care to admit, and not just appel-
late law, but the law that guides everyday behavior.
As a matter of judicial administration, many circuits have responded to
growing caseloads by using “fast-track” procedures to quickly dispose of cases
that do not warrant full attention (Vladeck & Gulati 2005; Richman &
Reynolds 2012; Levy 2013). Cases designated as hard get the traditional treat-
ment, with full briefing, oral argument, and reasoned opinions. Cases deemed
to be easy are decided primarily by staff attorneys, with cursory judicial over-
sight and unpublished dispositions. One premise underlying this practice is
that these cases are actually easy (Serfass & Cranford 2004, pp. 351–357); they
are delegated to staff attorneys because they can be resolved through
uncontroversial application of the law to the facts. Another premise underlying

the practice is that staff attorneys can readily separate the easy cases from the
hard ones.
The analysis developed in this article provides one way to examine the prem-
ises underlying these fast-track procedures. For this purpose, it would make
sense to presume that panels only rarely commit clear errors; otherwise, it
would be hard to justify curtailing full procedures. If the upper bound on easy

cases is lower than the proportion of unpublished decisions, then at least some
contestable cases are not receiving adequate attention.
For example, if there were at most two percent clear errors in the 9th Circuit
asylum cases, then at most 60 percent of the cases were easy. Yet 92 percent
were decided using fast-track procedures without published opinions. This dis-
crepancy shows that many of these were in fact hard cases and suggests that
they could have benefitted from closer scrutiny.
Because this mode of analysis requires few assumptions, it can help build
consensus for the reform of fast-track procedures. However, more information
will typically be needed to identify solutions. Notably, the approach developed
here cannot identify which cases warranted full procedures and published
opinions. Additional information, such as preliminary assessments made by
staff attorneys, may be helpful for evaluating the efficacy of screening
procedures.
Despite the formal standards for oral argument and opinion publication,
pragmatic concerns might weigh against full procedures for every hard case. As
Mashaw (1983, pp. 79–100) argues, a legal system must inevitably strike a bal-
ance between error costs and the costs of adjudication. Although this article
does not attempt to engage in cost–benefit analysis, the framework developed
here could be extended to do so. It would be natural to incorporate Mashaw’s
intuition that hard cases—those closer to the decision threshold—generate
lower error costs while clear errors generate higher error costs. The social costs
of errors may also be asymmetrical. For example, deporting an asylum peti-
tioner who reasonably fears persecution may cause greater harm than granting
asylum to a one who does not meet the statutory standard.
It is also possible that administrative reforms may not reduce clear
errors. If errors are due to recalcitrant judges flouting the law, then add-
itional procedures may add no value. To investigate this possibility, one
would need data on judicial decisions before and after a change in screen-
ing procedures.
Finally, insights developed here about easy and hard cases also serve to illu-
minate “clarity doctrines” (Re 2019) that establish presumptions based on
whether particular legal sources are clear or ambiguous. While this article has
focused on cases as the unit of analysis, clarity doctrines apply to issues within
a case. Many principles of interpretation, such as Chevron50 deference, consti-

tutional avoidance, the rule of lenity, and the use of interpretive canons, re-
quire judges to first determine if a legal source is clear or ambiguous.
Just as there has been enormous disagreement about what makes cases easy
or hard, lawyers and judges have also struggled to articulate standards for
determining whether particular legal provisions are clear or ambiguous (id.;
Farnsworth, Guzior, & Malani 2010, p. 275; Kavanaugh 2016, p. 2118). To

push the analogy further, clarity and ambiguity can both be defined as inter-
pretive concepts to be applied by individual judges or as sociological concepts
determined by reference to a particular community.51 In the former case, a
provision is clear if a particular interpreter is confident about its meaning; in
the latter case, it is clear if most or all interpreters would find a common mean-
ing. There is some debate about which definitions should apply to clarity deter-
minations, but there are good arguments for applying the sociological
definition in at least some contexts (Re 2019, pp. 16–36).
To the extent that clarity depends on levels of agreement within a community,
it is an empirical determination rather than an interpretive one. Given that many
judges appear to be overconfident about the prevalence of easy cases, it is likely
that they are similarly overconfident about levels of agreement regarding particular
provisions. A careful understanding of such agreement and how to measure it
could help to bring some coherence to these notoriously challenging doctrines.
8. CONCLUSION
The title of this article, “How Many Cases Are Easy?,” is not itself an easy ques-
tion. It is in fact many questions, corresponding to different conceptions of
easiness. Yet even if easy cases, hard cases, and clear errors are inherently sub-
jective, it is possible to draw some objective conclusions from empirical evi-
dence on judicial voting behavior.
This article has considered the strengths and shortcomings of judicial accounts,
models of case selection, and commonly used forms of empirical evidence. It has
also presented a new empirical approach, which applies the consensus voting
model to predict the behavior of circuit court panels and decomposes evidence of
50 Chevron U.S.A. v. Natural Resources Defense Council, Inc., 467 US 837 (1984).
51 Others have used different terms to draw the same distinction. Farnsworth et al. (2010, p. 258) con-
trast a reader’s “internal judgment” that she is “unsure how best to read [a] text” with an “external
judgment” that “ordinary readers of English would disagree about its meaning.” Re (2019, p. 1509)
observes that “legal clarity can be used in two distinct ways: it can account for one’s own confidence
in the correct answer (certainty) or the likelihood that other actors will reach a common answer
(predictability).”
disagreement into hard cases and clear errors. Applied to asylum cases in the 9th
Circuit, there appear to be high levels of disagreement, corresponding to either a
large proportion of hard cases or an uncomfortable level of clear errors. Applied to
NLRB and EPA cases in the D.C. Circuit, the data are consistent with lower levels
of hard cases or clear errors. Further studies may reveal whether these conclusions
hold in other contexts. At a minimum, discourse about easy and hard cases should
be more closely connected to empirical evidence on judicial voting. The frame-

work developed here shows how to make that connection.
APPENDIX
A. ESTIMATING VOTING PROBABILITIES FOR PANELS52

Let xi ; xj ; xk be the indifference points for judges i; j; k; respectively, and let c be
the cost of dissent. Then in case t with cutpoint mt , the probability of a unani-
mous liberal panel decision is

pu xi ; xj ; xk jmt ¼ Uðxi mt þ c ÞUðxj mt ÞUðxk mt Þ
þ Uðxi mt ÞUðxj mt þ c ÞUðxk mt Þ
þ U ðx i m t ÞU ðx j m t ÞU ðx k m t þ c Þ
2Uðxi mt ÞUðxj mt ÞUðxk mt Þ;
where U is the normal cumulative distribution function. The probabilities of a

liberal panel decision with the first, second, and third judges dissenting are

p1 xi ; xj ; xk jmt ¼ Uðmt xi c ÞUðxj mt ÞUðxk mt Þ;

p2 xi ; xj ; xk jmt ¼ Uðxi mt ÞUðmt xj c Þ Uðxk mt Þ;

p3 xi ; xj ; xk jmt ¼ Uðxi mt ÞUðxj mt ÞUðmt xk c Þ:
The probabilities of conservative panel decisions are the same as above, ex-
cept with xi mt replaced by mt xi and similarly for j and k.
The cutpoint mt is modeled as mt ¼ czt þ gt , where zt is a vector of case
characteristics, c is a coefficient to be estimated, and gt is a normally distrib-
uted random effect with variance r2 . Because cases are randomly assigned, gt
will be independent of the judges’ indifference points.
The cutpoint mt is unknown, but the probabilities can be estimated by inte-
grating over the random effect, for example,
52 For derivations and more details, see Fischman (2011).

ð

pu ðxi ; xj ; xk ; zt Þ ¼ pu xi ; xj ; xk j czt þ ru /ðuÞdu;
where / is the normal density function. This integral can be approximated

using Gauss–Hermite quadrature. The unconditional probabilities p1 ; p2 ; p3
can be calculated similarly. The probability of a liberal panel decision is the
sum of the four possible liberal outcomes: p ¼ pu þ p1 þ p2 þ p3 .

Given the data on panel voting, one can then use maximum likelihood estima-
tion to derive estimates of the indifference points xbi , the cost of dissent b c , the im-
pact of case characteristics bc , and the standard deviation b r .53 For case t with
judges i; j; k; the probability of a liberal panel decision will be pðxi ; xj ; xk ; zt Þ. For
the calibration plots in Figure 7, the probabilities are based on the actual cases that
the panels decide. For the rest of the analysis, the probability of a liberal decision
for a particular panel is computed by averaging pðxi ; xj ; xk ; zt Þ across all case char-
acteristics zt while keeping i; j; k fixed; this represents the proportion of liberal
decisions if a particular panel decided all of the cases in the data.
The probability that judge i would prefer the liberal position in case t, if judge
i were deciding autonomously without the influence of panel colleagues, is
ð
pðxi jmt Þ ¼ pðxi czt ruÞ/ðuÞdu:
The estimated proportion of cases in which judge i would prefer the liberal
position is therefore
T ð
1X
qi ¼ pðxi czt ruÞ/ðuÞdu:
T t¼1
B. CALIBRATION TESTS
For case t, let yt ¼ 1 if the panel reaches a liberal decision and yt ¼ 0 if the
panel reaches a conservative decision. Let pbt be the predicted probability of a
liberal outcome based on the panel of judges and case characteristics. If the
probabilities predicted by the consensus voting model are accurate, then

E ½yt ¼ pbt and Varðyt Þ ¼ pbt 1 pbt . If the cases are partitioned into bins
hP i P P
B1 ; . . . ; Bk , then E yt 2Bi yt ¼
b
yt 2Bi p t and Var yt 2Bi yt ¼
P P
b b
yt 2Bi p t 1 p t . If the bins are large enough that yt 2Bi yt is approximately
normal, then we can construct a z-score for each bin:
53 See id. at 792 (discussing maximum likelihood estimation for the consensus voting model).
P P
yt 2Bi pbt
yt 2Bi yt
zi ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
P :
yt 2Bi t
b
p 1 pb t
Pk
The test statistic zi2 will follow the chi-square distribution with k degrees
i¼1
of freedom.
Using five bins, the test statistics are v2 ¼ 4:30 for the 9th Circuit data and

2
v ¼ 1:32 for the D.C. Circuit data. The significance threshold for 95 percent
confidence with five degrees of freedom is 11.07, so both fall short of statistical
significance. Thus, the rates of liberal decisions do not deviate significantly
from the predicted rates.
C. ESTIMATING FEASIBLE COMBINATIONS OF HARD CASES AND CLEAR ERRORS54

If H is the proportion of hard cases, then the lower bound on the proportion
of clear errors is given by
X
T
L ðH Þ ¼ min
max pbt w H; 0 þ max w pbt ; 0
0 w 1H
t¼1
This function separates the feasible combinations from the infeasible combi-
nations, as in Figures 2, 9, and 11.
If every case has a correct answer and judge i purports to be neutral, then the
lower bound on the error rate, according to judge i’s perspective, is
1X T
jpb qi j:
T t¼1 t
D. CONFIDENCE INTERVALS AND STANDARD ERRORS

Confidence intervals for all estimators are calculated using subsampling
(Politis, Romano, & Wolf 1999). Subsampling, like the better-known boot-
strap, is a resampling method for calculating confidence intervals. The boot-
strap would generate invalid confidence intervals because LðH Þ is
a nonsmooth function of the pbi ’s (Andrews 2000). Whereas the
bootstrap resamples the data with replacement, subsampling resamples the
data without replacement, generating subsamples that are smaller than the
original dataset. Smaller subsamples will generally have wider standard
54 For derivations and more details, see Fischman (2014, pp. 59–67).
errors, so it is necessary to extrapolate the estimated confidence intervals to

the full dataset.
To calculate the confidence intervals, I generated random subsamples of the
full data, estimated the parameters using maximum likelihood, and calculated
the function LðH Þ. For every H, the 95 percent confidence curve is calculated
by taking the 5th percentile of these estimates of LðH Þ: For subsample ratios r
ranging from 20 to 70 percent of the full sample, the size of the confidence

interval decreased at a roughly linear rate for the 9th Circuit data and at a rate
of roughly r 1:5 for the D.C. Circuit data. To account for the smaller sizes of the
subsamples, I multiplied the sizes of the estimated confidence intervals by r b ,
where b ¼ 1 for the 9th Circuit data and b ¼ 1:5 for the D.C. Circuit data. For
each judge’s probability of reaching a liberal result in Figures 8 and 10, I used
two-sided confidence intervals. For lower bounds, including the 95 percent
confidence curves in Figures 9 and 11, I used one-sided confidence intervals.
Because LðH Þ is a convex function of the pbi ’s, the estimates of LðH Þ will be
biased upward. I adjust for this bias using the correction proposed in Politis,
Romano, & Wolf (1999, p. 93). Let Ld ðH Þ be the estimate of LðH Þ based on the
full sample and Lkd ðH Þ be the estimate from the k-th subsample. Then the bias-
h i
ð Þ d
ð Þ b dð Þ d
ð
corrected estimate of L H is L H r mean Lk H L H and similar- Þ
ly for the other estimators.
After adjusting for subsample size, the results did not vary much for subsam-
pling ratios ranging from 0.2 to 0.7. I used a subsampling ratio of r ¼ 0:3 with
200 iterations for the 9th Circuit asylum cases. For the smaller dataset of D.C.
Circuit NLRB and EPA cases, I used a subsampling ratio of r ¼ 0:6 to ensure
an adequate number of observations per judge. I dropped a subsample if any
judge’s indifference point diverged to 61 due to insufficient observations.
These subsamples did not affect the estimates of the confidence intervals but
were removed because they distorted the bias correction.
SUPPLEMENTARY MATERIAL
Supplementary material (the replication archive) is available at JLA online.
REFERENCES
Aldisert, Ruggero J. 1987. Philosophy, Jurisprudence, and Jurisprudential

Temperament of Federal Judges. 20 Ind. L. Rev. 453–515.
Altman, Scott. 1990. Beyond Candor. 89 Mich. L. Rev. 296–351.
Andrews, Donald W. K. 2000. Inconsistency of the Bootstrap When a

Parameter is on the Boundary of the Parameter Space. 68 Econometrica
399–405.
Arnold, Richard S. 1999. Unpublished Opinions: A Comment. 1 J. App. Prac.
Process 219–226.
Baum, Lawrence. 1997. The Puzzle of Judicial Behavior. Ann Arbor, MI:
University of Michigan Press.

Bibas, Stephanos. 2004. Plea Bargaining outside the Shadow of Trial. 117
Harv. L. Rev. 2464–2547.
Bix, Brian. 1993. Law, Language, and Legal Determinacy. Oxford: Clarendon
Press.
Bix, Brian H. 2016. Doctrine, Data, and High Theory. 6 U. C. Irvine L. Rev.
137–148.
Boggs, Danny J. & Brian P. Brooks. 2000. Unpublished Opinions & the
Nature of Precedent. 4 Green Bag 2d 17–26.
Boyd, Christina L., Lee Epstein & Andrew D. Martin. 2010. Untangling the
Causal Effects of Sex on Judging. 54 Am. J. Pol. Sci. 389–411.
Brink, David O. 2001. Legal Interpretation, Objectivity, and Morality. In
Leiter Brian, ed., Objectivity in Law and Morals, 12–37. Cambridge:
Cambridge University Press.
Cardozo, Benjamin N. 1921. The Nature of the Judicial Process. New Haven,
CT: Yale University Press.
——— 1924. The Growth of the Law. New Haven, CT: Yale University Press.
Chemerinsky, Erwin. 2006. Seeing the Emperor’s Clothes: Recognizing the
Reality of Constitutional Decision Making. 86 B. U. L. Rev. 1069–1081.
Chilton, Adam S. & Marin K. Levy. 2015. Challenging the Randomness of
Panel Assignment in the Federal Courts of Appeals. 101 Cornell L. Rev
1–56.
Clark, Charles E. & David M. Trubek. 1961. The Creative Role of the Judge:
Restraint and Freedom in the Common Law Tradition. 71 Yale L. J.
255–276.
Cleveland, David R. & Steven Wisotsky. 2012. The Decline of Oral Argument
in the Federal Courts of Appeals: A Modest Proposal for Reform. 13 J.
App. Prac. Process 119–151.
Coffin, Frank M. 1994. On Appeal: Courts, Lawyering, and Judging. New
York: W. W. Norton & Co.
Colby, Thomas B. 2012. In Defense of Judicial Empathy. 96 Minn. L. Rev.
1944–2015.
Coleman, Jules L. & Brian Leiter. 1993. Determinacy, Objectivity, and
Authority. 142 U. Pa. L. Rev. 549–637.
Cooter, Robert, Stephen Marks & Robert Mnookin. 1982. Bargaining in the
Shadow of the Law: A Testable Model of Strategic Behavior. 11 J. Legal
Stud. 225–251.
Corbin, Arthur L. 1961. The Judicial Process Revisited: Introduction. 71 Yale
L. J. 195–201.
Cross, Frank B. & Emerson H. Tiller. 1998. Judicial Partisanship and
Obedience to Legal Doctrine: Whistleblowing on the Federal Courts of

Appeals. 107 Yale L. J. 2155–2176.
Cross, Frank B. 2003. Decisionmaking in the U.S. Circuit Courts of Appeals.
91 Cal. L. Rev. 1457–1515.
Dworkin, Ronald. 1975. Hard Cases. 88 Harv. L. Rev. 1057–1109.
——— 2009. Justice Sotomayor: The Unjust Hearings. 56 N. Y. Rev. Books
37–40.
Easterbrook, Frank H. 1994. Text, History, and Structure in Statutory
Interpretation. 17 Harv. J. L. Pub. Pol’y 61–70.
Edwards, Harry T. 1983–1984. The Role of a Judge in Modern Society: Some
Reflections on Current Practice in Federal Appellate Adjudication. 32
Clev. St. L. Rev. 385–430.
——— 1985. Public Misperceptions concerning the “Politics” of Judging:
Dispelling Some Myths about the D.C. Circuit. 56 Univ. Colo. L. Rev.
619–646.
——— 1991. The Judicial Function and the Elusive Goal of Principled
Decisionmaking. 1991 Wis. L. Rev. 837–865.
——— 1998. Collegiality and Decision Making on the D.C. Circuit. 84 Va.
L. Rev. 1335, 1364.
Edwards, Harry T. & Michael A. Livermore. 2009. Pitfalls of Empirical
Studies That Attempt to Understand the Factors Affecting Appellate
Decisionmaking. 58 Duke L. J. 1895–1989.
Eisgruber, Christopher L. 2007. The Next Justice: Repairing the Supreme Court
Appointments Process. Princeton, NJ: Princeton University Press.
Epstein, Lee, William M. Landes & Richard A. Posner. 2013. The Behavior of
Federal Judges: A Theoretical and Empirical Study of Rational Choice.
Cambridge, MA: Harvard University Press.
Eskridge, William N. Jr., and Philip P. Frickey. 1990. Statutory Interpretation
as Practical Reasoning. 42 Stan. L. Rev. 321–384.
Fallon, Richard H. Jr. 2008. Constitutional Precedent Viewed through the
Lens of Hartian Positivist Jurisprudence. 86 N. C. L. Rev. 1107.
——— 2018. Law and Legitimacy in the Supreme Court. Cambridge, MA:
Belknap Press of Harvard University Press.
Farber, Daniel A. 2000. Do Theories of Statutory Interpretation Matter? A
Case Study. 94 Nw. U. L. Rev. 1409–1444.
Farber, Daniel A. & Suzanna Sherry. 2009. Judgment Calls: Principle and
Politics in Constitutional Law. Oxford: Oxford University Press.
Farbman, Daniel. 2019. Resistance Lawyering. 107 Cal. L. Rev. 1877–1953.
Farnsworth, Ward, Dustin F. Guzior & Anup Malani. 2010. Ambiguity about
Ambiguity: An Empirical Inquiry into Legal Interpretation. 2 J. Leg. Anal.
257–300.
Fischman, Joshua B. & David S. Law. 2009. What Is Judicial Ideology, and

How Should We Measure It? 29 Wash. Univ. J. L. Pol’y 133–214.
Fischman, Joshua B. 2011. Estimating Preferences of Circuit Judges: A Model
of Consensus Voting. 54 J. L. Econ. 781–809.
——— 2014. Measuring Inconsistency, Indeterminacy, and Error in
Adjudication. 16 Am. L. Econ. Rev. 40–85.
——— 2015. Interpreting Circuit Court Voting Patterns: A Social
Interactions Framework. 31 J. L. Econ. Organ. 808–842.
——— 2020. Politics and Authority in the U.S. Supreme Court. 104 Cornell
L. Rev. 1513–1592.
Frank, Jerome. 1930. Law and the Modern Mind. New York, NY: Brentano’s.
——— 1931. Are Judges Human? Part One: The Effect on Legal Thinking
of the Assumption That Judges Behave like Human Beings. 80 U. Pa.
L. Rev. 17.
——— 1948. Cardozo and the Upper-Court Myth. 13 L. Contemp. Probl.
369–390.
Friendly, Henry J. 1961. Reactions of a Lawyer-Newly Become Judge. 71 Yale
L. J. 218–238.
Garth, Leonard I. 1995. Views from the Federal Bench: Past, Present &
Future. 47 Rutgers L. Rev. 1361–1370.
Gilmore, Grant. 1977. The Ages of American Law. New Haven, CT: Yale
University Press.
Ginsburg, Ruth Bader. 1990. Remarks on Writing Separately. 65 Wash. L.
Rev. 133–150.
Gordon, Robert W. 1993. Lawyers, Scholars, and the Middle Ground. 91
Michigan L. Rev. 2075–2112.
Grant, Edward R. 2006. Laws of Intended Consequences: IIRIRA and Other
Unsung Contributors to the Current State of Immigration Litigation. 55
Cath. U. L. Rev. 923–963.
Greenawalt, Kent. 1990. How Law Can Be Determinate. 38 UCLA L. Rev.
1–86.
Greene, William H. 2000. Econometric Analysis. 4th Edition. Upper Saddle
River, NJ: Prentice-Hall.
Greenwald, David & Frederick A. O. Schwarz, Jr. 2002. The Censorial
Judiciary. 35 U. C. Davis L. Rev. 1133–1174.
Gross, Samuel R. & Kent D. Syverud. 1991. Getting to No: A Study of

Settlement Negotiations and the Selection of Cases for Trial. 90 Mich. L.
Rev. 319–393.
Grunwald, Ben. 2018. Strategic Publication. 92 Tulane L. Rev. 745–776.
Hand, Learned. 1922. The Nature of the Judicial Process. By Benjamin N.
Cardozo. 35 Harv. L. Rev. 479–481.
Harris, J. W. 1979. Law and Legal Science: An Inquiry into the Concepts Legal

Rule and Legal System. Oxford: Clarendon Press.
Howard, J. Woodford Jr.1981. Courts of Appeals in the Federal Judicial
System: A Study of the Second, Fifth, and District of Columbia Circuits.
Princeton, NJ: Princeton University Press.
Jones, Harry W. 1961. Law and Morality in the Perspective of Legal Realism.
61 Colum. L. Rev. 799–809.
Kahan, Dan M. 2011. Foreword: Neutral Principles, Motivated Cognition,
and Some Problems for Constitutional Law. 125 Harv. L. Rev. 1–77.
Kavanaugh, Brett M. 2016. Fixing Statutory Interpretation. 129 Harv. L. Rev.
2118–2163.
Kaye, Judith S. 1995. State Courts at the Dawn of a New Century: Common
Law Courts Reading Statutes and Constitutions. 70 N. Y. U. L. Rev. 1–35.
Kennedy, Duncan. 1997. A Critique of Adjudication: Fin de Siècle. Cambridge,
MA: Harvard University Press.
Keeton, Robert E. 1969. Venturing to Do Justice: Reforming Private Law.
Cambridge, MA: Harvard University Press.
King, Carolyn Dineen. 2007. Challenges to Judicial Independence and the
Rule of Law: A Perspective from the Circuit Courts. 90 Marq. L. Rev.
765–788.
Klein, David E. 2002. Making Law in the United States Courts of Appeals.
Cambridge: Cambridge University Press.
Korobkin, Russell & Chris Guthrie. 1994. Psychological Barriers to Litigation
Settlement: An Experimental Approach. 93 Mich. L. Rev. 107–192.
Kozinski, Alex. 2004. The Real Issues of Judicial Ethics. 32 Hofstra L. Rev.
1095–1106.
Kress, Ken. 1989. Legal Indeterminacy. 77 Cal. L. Rev. 283–338.
——— 1990. A Preface to Epistemological Indeterminacy. 85 Nw. Univ. L.
Rev. 134–147.
Law, David S. 2005. Strategic Judicial Lawmaking: Ideology, Publication, and
Asylum Law in the Ninth Circuit. 73 U. Cincinnati L. Rev. 817–866.
Leflar, Robert A. 1979. Honest Judicial Opinions. 74 Nw. U. L. Rev. 721–741.
Leiter, Brian. 1995. Legal Indeterminacy. 1 Legal Theory 481–492.
Levi, David F. 2009. Autocrat of the Armchair. 58 Duke Law Journal
1791–1805.
Levy, Marin K. 2011. The Mechanics of Federal Appeals: Uniformity and

Case Management in the Circuit Courts. 61 Duke L. J. 315–392.
——— 2013. Judicial Attention as a Scarce Resource: A Preliminary Defense
of How Judges Allocate Time across Cases in the Federal Courts of
Appeals. 81 Geo. Wash. L. Rev. 401–447.
Llewellyn, Karl N. 1931. Some Realism about Realism—Responding to Dean
Pound. 44 Harv. L. Rev. 1222–1264.

——— 1950. Remarks on the Theory of Appellate Decision and the Rules or
Canons about How Statutes Are to Be Construed. 3 Vand. L. Rev.
395–406.
——— 1960. The Common Law Tradition: Deciding Appeals. Boston, MA:
Little, Brown & Co.
Lumbard, J. Edward. 1968. Current Problems of the Federal Courts of
Appeals. 54 Cornell L. Rev. 29–44.
Lyons, David B. 1984. Justification and Judicial Responsibility. 72 Cal. L.
Rev. 178–199.
Mackie, J. L. 1977. The Third Theory of Law. 7 Philos. Public Aff. 3–16.
Marmor, Andrei. 1990. No Easy Cases? 3 Can. J. L. Jur. 61–79.
Marouf, Fatma, Michael Kagan & Rebecca Gill. 2014. Justice on the Fly: The
Danger of Errant Deportations. 75 Ohio St. L. J. 337–404.
Martineau, Robert J. 1986. The Value of Appellate Oral Argument: A
Challenge to the Conventional Wisdom. 72 Iowa L. Rev. 1–34.
Mashaw, Jerry L. 1983. Bureaucratic Justice: Managing Social Security
Disability Claims. New Haven: Yale University Press.
McKee, Theodore A. 2007. Judges as Umpires. 35 Hofstra L. Rev. 1709–1724.
Merritt, Deborah J. & James J. Brudney. 2001. Stalking Secret Law: What
Predicts Publication in the United States Courts of Appeals. 54 Vand. L.
Rev. 71–121.
Miles, Thomas J. & Cass R. Sunstein. 2006. Do Judges Make Regulatory
Policy? An Empirical Investigation of Chevron. 73 U. Chi. L. Rev.
823–882.
———. 2008a. The Real World of Arbitrariness Review. 75 U. Chi. L. Rev.
761–814.
———. 2008b. The New Legal Realism. 75 U. Chi. L. Rev. 831–851.
Miner, Roger J. 2012–2013. “Dealing with the Appellate Caseload Crisis”:
The Report of the Federal Courts Study Committee Revisited. 57 N. Y. L.
School L. Rev. 517–554.
Mishkin, Paul J. 1965. Foreword: The High Court, the Great Writ, and the
Due Process of Time and Law. 79 Harv. L. Rev. 56–102.
Mnookin, Robert H. & Lewis Kornhauser. 1979. Bargaining in the Shadow of
the Law: The Case of Divorce. 88 Yale L. J. 950–997.
Neely, Richard. 1981. How Courts Govern America. New Haven, CT: Yale
University Press.
Nelson, David A. 1995. The Nature of the Judicial Process Revisited. 22 N.
Ky. L. Rev. 563–578.
Newman, Jon O. 1984. Between Legal Realism and Neutral Principles: The
Legitimacy of Institutional Values. 72 Cal. L. Rev. 200–216.
Noonan, John T. Jr. 2002. Persons and Masks of the Law. 2nd Edition.

Berkeley, CA: University of California Press.
Patterson, Edwin W. 1947. Foreword. In Hall Margaret E., ed., Selected
Writings of Benjamin Nathan Cardozo v–x. New York: Fallon
Publications.
Pedro, Portia. 2018. Stays: The Law-Free Zone of Civil Procedure. 106 Cal. L.
Rev. 869–928.
Polenberg, Richard. 1997. The World of Benjamin Cardozo: Personal Values
and the Judicial Process. Cambridge, MA: Harvard University Press.
Politis, Dimitris N., Joseph P. Romano, and Michael Wolf. 1999.
Subsampling. New York: Springer.
Posner, Richard A. 1988. The Jurisprudence of Skepticism. 86 Mich. L. Rev.
827–891.
——— 1995. Judges’ Writing Styles (and Do They Matter?) 62 U. Chi. L.
Rev. 1421–1450.
——— 2008. How Judges Think. Cambridge, MA: Harvard University Press.
——— 2010. Some Realism about Judges: A Reply to Edwards and
Livermore. 59 Duke L. J. 1177–1186.
——— 2013. Reflections on Judging. Cambridge, MA: Harvard University
Press.
Pozen, David E. 2010. Judicial Elections as Popular Constitutionalism. 110
Columb. L. Rev. 2047–2134.
Priest, George L. & Benjamin Klein. 1984. The Selection of Disputes for
Litigation. 13 J. Leg. Stud. 1–55.
Radin, Max. 1942. In Defense of an Unsystematic Science of Law. 51 Yale L.
J. 1269–1279.
Re, Richard M. 2019. Clarity Doctrines. 86 U. Chi. L. Rev. 1497–1562.
Reavley, Thomas S. 1997. The Magnificent Profession. 60 Tex. Bar J.
1037–1041.
Revesz, Richard L. 1997. Environmental Regulation, Ideology, and the D.C.
Circuit. 83 Va. L. Rev. 1717–1772.
——— 1999. Ideology, Collegiality, and the D.C. Circuit: A Reply to Chief
Judge Edwards. 85 Va. L. Rev. 805–851.
Richman, William M. & William L. Reynolds. 2012. Injustice on Appeal: The
United States Courts of Appeals in Crisis. Oxford: Oxford University Press.
Rollert, John Paul. 2010. Reversed on Appeal: The Uncertain Future of

President Obama’s “Empathy Standard.” 120 Yale L. J. 89–107.
Rosati, Connie S. 2004. Some Puzzles about the Objectivity of Law. 23 L.
Philos. 273–323.
Rubin, Alvin B. 1987. Doctrine in Decision-Making: Rationale or
Rationalization. 1987 Utah L. Rev. 357–372.
Rubin, Edward. 2011. The Real Formalists, the Real Realists, and What They

Tell Us about Judicial Decision Making and Legal Education. 109 Mich.
L. Rev. 863–882.
Sachs, Stephen E. 2019. Finding Law. 107 Cal. L. Rev. 527–581.
Schaefer, Walter V. 1966. Precedent and Policy. 34 U. Chi. L. Rev. 3–25.
Schauer, Frederick. 1985. Easy Cases. 58 South. Cal. L. Rev. 399–440.
——— 1988. Formalism. 97 Yale L. J. 509–548.
——— 1988. Judging in the Corner of the Law. 61 South. Cal. L. Rev.
1717–33.
——— 2013. Legal Realism Untamed. 91 Tex. L. Rev. 749–780.
Schubert, Glendon. 1968. Behavioral Jurisprudence. 2 L. Soc. Rev. 407–428.
Schwartz, Gary T. 1999. Cardozo as Tort Lawmaker. 49 DePaul L. Rev.
305–318.
Segal, Jeffrey A. & Harold J. Spaeth. 2002. The Supreme Court and the
Attitudinal Model Revisited. New York, NY: Cambridge University Press.
Serfass, Melissa M. & Jessie Wallace Cranford. 2004. Federal and State Court
Rules Governing Publication and Citation of Opinions: An Update. 6 J.
App. Pract. Process 349–394.
Shapiro, Martin. 1994. Judges as Liars. 17 Harv. J. L. Pub. Pol’y 155–156.
Shapiro, Scott J. 2011. Legality. Cambridge, MA: Harvard University Press.
Simon, Dan. 2000. The Double-Consciousness of Judging: The Problematic
Legacy of Cardozo. 79 Or. L. Rev. 1033–1080.
Solum, Lawrence B. 1987. On the Indeterminacy Crisis: Critiquing Critical
Dogma. 54 U. Chi. L. Rev. 462–503.
——— 1988. The Virtues and Vices of a Judge: An Aristotelian Guide to
Judicial Selection. 61 South. Cal. L. Rev. 1735–1756.
Songer, Donald R. 1990. Criteria for Publication of Opinions in the U.S.
Courts of Appeals: Formal Rules versus Empirical Reality. 73 Judicature
307–313.
Stevens, John Paul. 2006. Learning on the Job. 74 Fordham L. Rev.
1561–1567.
Stone, Harlan F. 1922. Review of the Nature of the Judicial Process, by
Benjamin N. Cardozo. 22 Colum. L. Rev. 382–385.
Stuntz, William J. 2004. Plea Bargaining and Criminal Law’s Disappearing
Shadow. 117 Harv. L. Rev. 2548–2569.
Sunstein, Cass R., David Schkade, Lisa M. Ellman & Andres Sawicki. 2006.
Are Judges Political? An Empirical Analysis of the Federal Judiciary.
Washington, DC: Brookings Institution Press.
Sutton, Jeffrey S. 2010. A Review of Richard A. Posner, How Judges Think
(2008). 108 Mich. L. Rev. 859–876.
Tamanaha, Brian Z. 2010. Beyond the Formalist-Realist Divide: The Role of
Politics in Judging. Princeton, NJ: Princeton University Press.

——— 2009. The Distorting Slant in Quantitative Studies of Judging. 50 B.
C. L. Rev. 685–758.
Tate, Albert. Jr. 1959. “Policy” in Judicial Decisions. 20 La. L. Rev. 62–75.
Tushnet, Mark. 1990. A Critical Legal Studies Perspective. 38 Clev. St. L. Rev.
137–151.
Tyler, Tom R. & Gregory Mitchell. 1994. Legitimacy and the Empowerment
of Discretionary Legal Authority: The United States Supreme Court and
Abortion Rights. 43 Duke L. J. 703–815.
Vladeck, David C. & Mitu Gulati. 2005. Judicial Triage: Reflections on the
Debate over Unpublished Opinions. 62 Wash. Lee L. Rev. 1667–1708.
Wald, Patricia M. 1992. Some Real-Life Observations about Judging. 26 Ind.
L. Rev. 173–186.
——— 1995. The Rhetoric of Results and the Results of Rhetoric: Judicial
Writings. 62 U. Chi. L. Rev. 1371–1419.
Walker, John M. Jr. 2001. Judicial Tendencies in Statutory Construction:
Differing Views on the Role of the Judge. 58 N. Y. U. Annu. Surv. Am. L.
203–239.
Wallace, J. Clifford. 2005. Improving the Appellate Process Worldwide
through Maximizing Judicial Resources. 38 Vanderbilt J. Transnat’l L.
187–213.
Wardlaw, Kim McLane. 2010. Umpires, Empathy, and Activism: Lessons
from Judge Cardozo. 85 Notre Dame L. Rev. 1629–1662.
Weisberg, Richard H. 1979. Law, Literature and Cardozo’s Judicial Poetics. 1
Cardozo L. Rev. 283–342.
Wold, John T. 1978. Going through the Motions: The Monotony of
Appellate Court Decisionmaking. 62 Judicature 58–65.
Wood, Diane P. 2012. When to Hold, When to Fold, and When to
Reshuffle: The Art of Decisionmaking on a Multi-Member Court. 100
Cal. L. Rev. 1445–1477.

How Many Cases Are Easy

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

How Many Cases Are Easy

Uploaded by

Copyright:

Available Formats

HOW MANY CASES ARE EASY?

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

* Professor of Law, University of Virginia School of Law, E-mail: jfischman@virginia.edu.

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

decision making. When judges introduce controversial value judgments in easy

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

some panels would disagree in at least 60 percent of the asylum cases. If

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

2. WHAT MAKES CASES EASY OR HARD?

Although legal commentators often describe cases as “easy” or “hard,” there is

2.1 Metaphysical Determinacy

account of judicial discretion” in which easy cases have “unique, determinate

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

2.2 Epistemic Determinacy

2.3 Individual vs. Sociological

is nearly inevitable that there will be disagreement about the correct

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

2.4 Disagreement vs. Reasonable Disagreement

determined by a numerical threshold, such as “nine out of ten” lawyers (Neely

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

3. JUDICIAL PERSPECTIVES ON THE PROPORTION OF EASY

3.1 Judicial and Scholarly Perspectives on Easy and Hard Cases

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

p. 828), even suggesting that Judge Edwards’s estimate—that 5–15 percent of

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

3.2 Limitations of Judicial and Scholarly Perspectives

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

4. THE SELECTION OF DISPUTES FOR APPEAL

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

4.1 Divergent Expectations Models of Settlement

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

4.2 Why Hard Cases Are Litigated

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

4.2.2 Criminal Cases

4.2.3. Litigation Subsidies

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

4.2.4. Bargaining Breakdown

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

4.2.5. Psychological Factors

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

5. EMPIRICAL EVIDENCE ABOUT EASY CASES

5.1 Judicial Disagreement: Within Panels vs. Between Panels

agreement or disagreement does not correspond perfectly with the easy/hard

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

5.2 Problems with Within-Panel Disagreement

31 See supra Note 8 and accompanying text.

contrast, between-panel comparisons are harder to interpret because different

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

5.3 Problems with Between-Panel Variation

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

34 See Section 6.1 for a detailed explanation.

5.4 Publication Rates

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

Downloaded from https://academic.oup.com/jla/article/13/1/595/6460433 by guest on 26 November 2022

6. EASY AND HARD CASES: A NEW EMPIRICAL FRAMEWORK

This section demonstrates how to derive inferences about the proportion of

disagreement and instances of clear errors. When coupled with assumptions