AI Algorithms and Awful Humans - Daniel Solove

AI, ALGORITHMS,
AND AWFUL HUMANS
By
Daniel J. Solove
&
Hideyuki Matsumi
96 FORDHAM LAW REVIEW __

(forthcoming symposium article)
Draft: October 16, 2023
Electronic copy available at: https://ssrn.com/abstract=4603992

AI, Algorithms, and Awful Humans Daniel J. Solove & Hideyuki Matsumi
ABSTRACT
This Essay critiques a set of arguments often made to justify the use of AI
and algorithmic decision-making technologies. These arguments all share
a common premise – that human decision-making is so deeply flawed that
augmenting it or replacing it with machines will be an improvement.
In this Essay, we argue that these arguments fail to account for the full
complexity of human and machine decision-making when it comes to
deciding about humans. Making decisions about humans involves special
emotional and moral considerations that algorithms are not yet prepared
to make – and might never be able to make.
It is wrong to view machines as deciding like humans do, but better because
they are supposedly cleansed of bias. Machines decide fundamentally
differently, and bias often persists. These differences are especially
pronounced when decisions have a moral or value judgment or involve
human lives and behavior. Some of the human dimensions to decision-
making that cause great problems also have great virtues. Additionally,
algorithms often rely too much on quantifiable data to the exclusion of
qualitative data. Whereas certain matters might be readily reducible to
quantifiable data, such as the weather, human lives are far more complex.
Having humans oversee machines is not a cure; humans often perform
badly when reviewing algorithmic output.
We contend that algorithmic decision-making is being relied upon too

eagerly and with insufficient skepticism. For decisions about humans, there
are important considerations that must be better appreciated before these
decisions are delegated in whole or in part to machines.

AI, ALGORITHMS, AND AWFUL HUMANS

By Daniel J. Solove1 and Hideyuki Matsumi2
Introduction .................................................................................................. 4
I. Arguments About AI and Humans ............................................................ 6
A. The Awful Human Argument................................................................ 6
B. The Better Together Argument ............................................................. 6
C. The Humans in the Loop Argument ...................................................... 7
D. The Brighter Future Response ............................................................... 7
II. The Problems of Humans + Machines ..................................................... 8
A. The Limits of Quantification ................................................................. 8
1. Quantification Is Transformative ....................................................... 9
2. Emotion, Morality, and Value Judgments........................................ 10
3. The Challenge of Blending Machine and Human Decision-making 11
4. Mesmerized by Math and Machines ................................................. 12
B. Accurate – But on What Basis? ............................................................ 14
1. Is Accuracy the Right Goal? .............................................................. 14
2. The Conflict Between Accurate and Unbiased ................................. 15
III. Rethinking the Relationship Between Algorithms and Humans .......... 17
A. The Importance of Structure ............................................................... 17
B. The Virtues of Good Decisions ............................................................. 18
1Eugene L. and Barbara A. Bernard Professor of Intellectual Property and Technology Law,
George Washington University Law School. Thanks to JP Schnabel for excellent research
assistance.
2 Hideyuki (“Yuki”) Matsumi. PhD candidate/researcher at the Research Group on Law
Science, Technology and Society (LSTS) as well as at the Health and Ageing Law Lab (HALL)
of the Vrije Universiteit Brussel (VUB). Member of the New York Bar.

INTRODUCTION
Increasingly, sophisticated algorithms are being used in decisions about
people’s lives. Algorithms are deployed to help determine whether to hire
people or to lend them money—and if so, what interest rate to charge.
Algorithms are used in the criminal justice system for pretrial detention and
sentencing decisions. Algorithms are implemented in education and other
domains to make interventions and key determinations affecting people’s
opportunities and futures.3 These algorithms, often referred to as “artificial
intelligence” (AI) in popular discourse, are raising many concerns.
In the law and policy debate regarding algorithmic decision-making, a series

of arguments are frequently made involving the relative virtues and vices of
human and machine decision-making:
• The Awful Human Argument. Humans are very bad

decision-makers. Although using algorithms to make
decisions may have flaws, they are better than humans.
• The Better Together Argument. Algorithms can assist

humans in making better decisions. Algorithms are merely
providing additional information for human decision-
makers to consider.
• The Humans in the Loop Argument. Flaws in automated

decision-making can be cured by adding humans in the
process.
• The Brighter Future Argument. Any flaws in algorithms

can readily be fixed. Algorithms are the path to a brighter
future with better decisions.
These arguments exert a powerful influence on law and policy. The

arguments rest on the assumptions that machines can readily replace human
decision-making or that humans and machines can work well together to
produce better decisions.
Further underpinning these arguments is the recognition that human

decision-making is highly flawed. Humans are infected with bias; they have
a limited range of experience to draw upon; they often are slow and
inefficient when making decisions; they are swayed by emotion and
irrational factors; they can be impulsive; and they exhibit various cognitive
biases and heuristics that can lead them astray. The arguments advancing
algorithmic decision-making point to the many flaws in human decision-
making to make the case that algorithms are better. Indeed, the promise of
algorithms is that they can cleanse away the stain of bias; they are consistent,
fast, and efficient.
3See Hideyuki Matsumi & Daniel J. Solove, The Prediction Society: Algorithms and the
Problems of Forecasting the Future, https://ssrn.com/abstract=4453869.

In this Essay, we critique these arguments when they are applied to decisions
made about humans.4 We argue that these arguments fail to account for the
full complexity of human and machine decision. Although these arguments
have some dimension of truth, they are missing some key considerations that
make the reality far more complicated.
It is wrong to view machines as deciding like humans do, but better because
they are supposedly cleansed of bias. Machines decide fundamentally
differently, and bias often persists. These differences are especially
pronounced when decisions have a moral component or involve human lives
and behavior. Making decisions about humans involves special emotional
and moral considerations that algorithms are not yet prepared to make – and
might never be able to make.
Some of the human dimensions to decision-making that cause great

problems also have great virtues. Additionally, algorithms often rely too
much on quantifiable data to the exclusion of qualitative data. Whereas
certain matters might be readily reducible to quantifiable data, such as the
weather, human lives are far more complex. Having humans oversee
machines is not a cure; humans often perform badly when reviewing
algorithmic output.
Ultimately, we contend that algorithmic decision-making is being relied

upon too eagerly and with insufficient skepticism. For decisions about
humans, there are important considerations that must be better appreciated
before these decisions are delegated in whole or in part to machines.
Policymakers must be careful in allowing or requiring machines to become
involved in decision-making processes, whether as a replacement for
humans or in conjunction with humans. Although it is possible that using
more algorithmic decision-making could lead to better outcomes, many
policymakers and commentators fail to appreciate what is lost when
machines replace human decision-makers as well as the complexity of
mixing human and machine decision-making.
4 We are not addressing the decision-making involving non-humans, such as use of

algorithms for autonomous vehicles, manufacturing, agriculture, and so on. Our focus is on
decisions producing legal or similarly significant effects on individuals, a distinction made in
the GDPR to determine when the law should intervene with automated decisions. See GDPR
art. 20.

I. ARGUMENTS ABOUT AI AND HUMANS

A. THE AWFUL HUMAN ARGUMENT
A common set of arguments are made to justify the use of AI and algorithmic
decision-making technologies – that they are better than human decision-
making. In what we call the “Awful Human Argument,” these technologies
are justified by comparing them to humans, who are often biased, opaque,
and inconsistent in their decision-making. When compared to how badly
humans decide, the Awful Human Argument contends that algorithms are
clearly preferable.
Human decision-making is fraught with problems. Humans are undeniably

biased, overly influenced by their narrow range of life experiences, limited in
the amount of data they can process, and prone to being swayed by emotion
and irrational impulses.
Proponents of AI tout algorithms as superior to humans. According to Cary

Coglianese and Lavi Ben Dor, “new digital technologies that rely on machine-
learning algorithms to process vast quantities of data are making highly
accurate predictions that often outperform humans in executing important
tasks.”5 Cass Sunstein contends that algorithms can eliminate “noise” –
“unwanted variability in judgments.”6 Algorithms “prevent unequal
treatment and reduce errors.” Unlike humans, algorithms “do not use
mental shortcuts; they rely on statistical predictors, which means that they
can counteract or even eliminate cognitive biases.”7
B. THE BETTER TOGETHER ARGUMENT

A related argument to the Awful Human Argument is the “Better Together
Argument,” which contends that humans and machines are better than
human decision-makers alone. Algorithms provide additional information to
aid human decisions, and more information is better than less information.
In Wisconsin v. Loomis, a criminal defendant challenged a trial court’s use

of a recidivism risk assessment algorithm called COMPAS.8 The defendant
contended that the algorithm violated his Right to Due Process because it
was based not on data about him but of the recidivism statistics of others and
he thus wasn’t sentenced as a unique individual.9 The Wisconsin Supreme
Court rejected the defendant’s arguments and held that the algorithm was
merely providing the sentencing judge with “more complete information.”
The court noted: “[W]e expect that circuit courts will exercise discretion
5 Cary Coglianese & Lavi M. Ben Dor, AI in Adjudication and Administration, 86 Brook. L.
Rev. 791 (2021).
6 Id. at 1177-78
7 Cass R. Sunstein, Governing by Algorithm? No Noise and (Potentially) Less Bias, 71 Duke
L.J. 1175, 1177 (2022).
8 Wisconsin v. Loomis, 881 N.W.2d 749 (Wis. 2016).
9 Id. at 764.

when assessing a COMPAS risk score with respect to each individual

defendant.”10
Other courts have made similar arguments – that algorithmic tools are just
helping humans by giving them more information. For example, in
Malenchik v. State, the Indiana Supreme Court held that sentencing
assessment tools “enable a sentencing judge to more effectively evaluate and
weigh several express statutory sentencing considerations.”11
C. THE HUMANS IN THE LOOP ARGUMENT

A close cousin of the Better Together Argument is the “Humans in the Loop
Argument” – that flaws in automated decision-making can be cured by
adding humans in the process. Machines and humans will complement each
other, and the overall decisions will be improved.
Many laws are premised on this argument, such as the European Union’s
General Data Protection Regulation (GDPR), which provides that “[t]he data
subject shall have the right not to be subject to a decision based solely on
automated processing, including profiling, which produces legal effects
concerning him or her or similarly significantly affects him or her.”12 When
a person challenges a decision based solely on automated processing, the
data controller must implement protections for the person’s “rights and
freedoms and legitimate interests,” and provide for “the right to obtain
human intervention on the part of the controller, to express his or her point
of view and to contest the decision.”13
Both the Better Together Argument and the Humans in the Loop Argument
posit that the combination of humans and algorithms will address the
deficiencies of each. These arguments are essentially the same, but from
opposite sides. The Better Together Argument posits that algorithms can be
brought in to help human decision-makers. The Humans in the Loop
Argument postulates that humans can be brought in to address any problems
with algorithms.
D. THE BRIGHTER FUTURE RESPONSE

In response to critiques of the deficiencies of algorithms, such as the fact that
they can perpetuate bias from their training data, proponents of AI respond
with the “Brighter Future Argument” – that technologies can be adjusted to
overcome any problems. If AI isn’t better than humans now, it can readily be
improved to surpass humans. As Orly Lobel argues: “[D]espite its risks and
flaws, digitization can and must become a powerful force for societal good—
for fairness, inclusion, economic growth, expanded opportunities,
innovation, and above all else, equality. AI can be a force for debiasing, and
10 Id.
11 Malenchik v. State, 928 N.E.2d 564, 574 (Ind. 2010).
12 GDPR art. 22.
13 GDPR art. 22.

it can provide early detection of discrimination and abuse.”14 Cass Sunstein

notes that “Algorithms can encode or perpetuate discrimination, perhaps
because their inputs are based on discrimination or because what they
(accurately) predict is infected by discrimination.” However, he contends
that “properly constructed algorithms nonetheless hold a great deal of
promise for the administrative state.”15 In a similar fashion, as Sam Corbett-
Davies, Sharad Goel and Sandra González-Bailón argue: “Poorly designed
algorithms can indeed exacerbate historical inequalities, but well-designed
algorithms can mitigate pernicious problems with unaided human
decisions.”16 Many other commenters note the potential for algorithms to be
created in ways to reduce bias.17
II. THE PROBLEMS OF

HUMANS + MACHINES
The arguments we discuss above are quite powerful. On the surface, they
seem to be relatively uncontroversial. Of course, human decision-making is
besieged with problems – and it can be quite awful. Of course, algorithms
can avoid many of these problems. Of course, algorithms can augment
human decision-making and humans can help address deficiencies in
algorithms. And, of course, algorithms can be improved as technology
advances.
Although these arguments appear unassailable on the surface, they often

ignore considerations that should give us pause. Humans are bad, but this
fact shouldn’t lure us into concluding that algorithms are better. We argue
that the reality is far more complicated than these arguments let on.
A. THE LIMITS OF QUANTIFICATION

Algorithmic decisions are not simply the same as human decisions without
the human awfulness. Algorithmic decisions are fundamentally different
from human ones. Comparing human to machine decision-making is akin
to comparing apples and oranges, not rotten apples to fresh ones.
14 ORLY LOBEL, THE EQUALITY MACHINE: HARNESSING DIGITAL TECHNOLOGY FOR A BRIGHTER,
MORE INCLUSIVE FUTURE (2022).
15 Cass R. Sunstein, Governing by Algorithm? No Noise and (Potentially) Less Bias, 71 Duke
L.J. 1175, 1177 (2022).
16 Sam Corbett-Davies, Sharad Goel and Sandra González-Bailón, Even Imperfect Algorithms
Can Improve the Criminal Justice System, N.Y. Times, Dec. 20, 2017.
17 Carolin Kemper, Kafkaesque AI? Legal Decision-Making in the Era of Machine Learning,
24 Intell. Prop. & Tech. L.J. 251, 285 (2020) (“Algorithms can be designed to alleviate
biases.”); Mirko Bagaric, Dan Hunter, and Nigel Stobbs, Erasing the Bias Against Using
Artificial Intelligence to Predict Future Criminality: Algorithms Are Color Blind and Never
Tire, 88 U. Cin. L. Rev. 1037, 1040 (2019) (positing that it is “achievable” to ensure that
algorithms “do not discriminate directly or indirectly”).

1. Quantification Is Transformative
Algorithms focus on data that can readily be quantified, and this type of data
skews the output. Problems emerge when too much quantitative data is
relied upon to the exclusion of qualitative data. Not everything is readily
quantifiable.
In Seeing Like a State, James Scott argues that making things “legible”
(more capable of being understood and analyzed) involves simplification – a
“narrowing of vision” that “brings into sharp focus certain limited aspects of
an otherwise far more complex and unwieldy reality.” The simplification can
make things “more susceptible to careful measurement and calculation.”18
When modern institutions seek to “see” things by gathering quantifiable
data, they are seeing through a particular lens, and much is excluded from
the picture. Scott warns that “thin, formulaic simplifications” can lead to
tragic failures.19
Quantification can certainly lead to insights we might not otherwise

recognize. But the fact that we can see certain things through quantification
doesn’t mean that quantification is a superior way of knowing or that it
should be the only way of examining things. Unfortunately, it is easy to
become beguiled by large quantified datasets because they help us see in new
exciting ways. Lambert Adolphe Jacques Quetelet (1796-1874), an early
pioneer in statistics, enthusiastically proclaimed in 1835: “The ‘greater the
number of individuals observed, the more do individual particularities,
whether physical or moral, become effaced, and leave in a prominent point
of view the general facts, by virtue of which society exists and is preserved.”20
Quetelet believed that statistics was a more refined way of understanding
humankind than to consider the irregularities of specific individuals. But
focusing on “general facts” omits the rich tapestry of peculiarities of
individuals and human nature. As Laurence Tribe warns, “readily
quantifiable variables” could “dwarf those that are harder to measure.”21
Certainly, statistics can be quite useful, and particular attempts to rank,

score, or infer based on aggregated standardized data can be valuable. But
these practices can be fraught with danger because algorithmic systems do
not simply see the world; they simplify it. As Chris Wiggins and Matthew
Jones contend, “Statistics doesn’t simply represent the world. It transforms
how we categorize and view the world. It transforms how we categorize
others and ourselves. It changes the world.”22
18 JAMES C. SCOTT, SEEING LIKE A STATE: HOW CERTAIN SCHEMES TO IMPROVE THE HUMAN
CONDITION HAVE FAILED 11 (1998).
19 Id. at 309.
20 Quetelet quoted in CHRIS WIGGINS AND MATTHEW L. JONES, HOW DATA HAPPENED: A HISTORY
FROM THE AGE OF REASON TO THE AGE OF ALGORITHMS 26 (2023).
21 Laurence H. Tribe, Trial by Mathematics: Precision and Ritual in the Legal Process, 84
Harv. L. Rev. 1329, 1393 (1971).
22 CHRIS WIGGINS AND MATTHEW L. JONES, HOW DATA HAPPENED: A HISTORY FROM THE AGE OF
REASON TO THE AGE OF ALGORITHMS 57 (2023).

For example, ranking systems and scoring systems are designed to make
things more readily comparable. But they are reductive. Writing about the
IQ test, Professor Neil Postman argues that it “assume[s] that intelligence is
not only a thing, but a single thing, located in the brain, and accessible to the
assignment of a number.”23 Some have posited that we have multiple types
of intelligence, not just a singular type.24
When data is ingested into AI systems, the data must be shucked of

extraneous information that isn’t readily digestible for the algorithm. It must
be stripped down, edited, and sometimes translated into the right format.25
When the rich details of life are often turned into quantifiable data, essential
information can be lost. As one of us has argued “the information in
databases often fails to capture the texture of our lives. Rather than provide
a nuanced portrait of our personalities, compilations of data capture the
brute facts of what we do without the reasons. . . . In short, we are
reconstituted in databases as a digital person composed of data.”26
2. Emotion, Morality, and Value Judgments
The human mind is remarkably complicated. Some might be tempted to

equate thinking to rationality, with emotions clouding lucid thought, but
there are many dimensions to human decision-making beyond rationality,
and these non-rational elements are often underappreciated. Philosopher
Martha Nussbaum aptly argues that “emotions are suffused with intelligence
and discernment” and involve “an awareness of value or importance.”
Emotions “are “part and parcel of the system of ethical reasoning.”27
Algorithms do not experience emotions. Algorithms might be able to mimic

what people might say or do, but they do not understand emotions or feel
emotions, and it is questionable how well algorithms will be able to
incorporate emotion into their output.
Based on where the technology currently is, algorithms shouldn’t be seen as

akin to better-thinking humans. They don’t think, and they don’t feel. They
are based on models designed by humans that are simplifications of the vast
complexity of the world. As Aziz Huq notes, “machine decisions are not
presently appropriate for decisions with ethical or normative components.”28
Although emotion can certainly make decisions worse, it can also make them
better. The amount of desirable involvement of non-rational elements in
decision-making is a challenging question, one that is contextual and lacks a
23 NEIL POSTMAN, TECHNOPOLY: THE SURRENDER OF CULTURE TO TECHNOLOGY 130 (1992).

24 HOWARD GARDNER, FRAMES OF MIND: THE THEORY OF MULTIPLE INTELLIGENCES (1983).
25 Dan L. Burk, Algorithmic Legal Metrics, 96 Notre Dame L. Rev. 1147 (2020) (“To
reconfigure otherwise incompatible data, analytical processing imposes a radical
decontextualization on the data, paring away extraneous information and meanings”).
26 DANIEL J. SOLOVE, THE DIGITAL PERSON: TECHNOLOGY AND PRIVACY IN THE INFORMATION AGE
49 (2004).
27 MARTHA C. NUSSBAUM, UPHEAVALS OF THOUGHT: THE INTELLIGENCE OF EMOTIONS 1 (2001).
28 Aziz Z. Huq, A Right to a Human Decision, 106 Va. L. Rev. 611, 685 (2020).
10

definitive answer. But we posit that non-rational elements can play a positive
role when decisions are made about humans.
Humans are also capable of making decisions about moral and normative
issues in ways that machines cannot. Making normative choices can involve
tradeoffs and compromises between different values, which can be
incommensurate and conflicting.
Consider, for example, college admissions decisions. There is a value in

making decisions consistently, as this is fair to all applicants. The most
consistent way to decide might be to look only at test scores and grade point
averages. But some applicants might have unique experiences or skills.
Suppose an applicant is a prodigy pianist with low grades and test scores.
Should he be admitted? One might value applicants with unique
accomplishments and want to include them, even at a sacrifice of not
admitting some applicants with high grades and test scores.
Other values in the admissions process involve establishing a balance in the

overall class – gender balance, diversity, and a variety of interests, among
other things. Negotiating all these considerations is complicated because no
one consideration might dominate. There is no right or wrong answer as to
whether to admit the pianist. In selecting students, we might want a little bit
of everything. The ideal might be some degree of consistency, but also some
swerves for special cases, as well as some skewing to navigate conflicting
values. This is a difficult recipe for machines to follow. Machines are most
adept at consistency; they struggle when told to do inconsistent things.
It is easy to point to grades and scores as more objective, even though these
more quantifiable things can also be influenced by privilege and bias. In the
end, college admissions decisions, as well as many other types of decisions,
involve value judgments and tradeoffs that machines are currently unable to
make.
3. The Challenge of Blending Machine and Human Decision-

making
Algorithmic decision-making is not necessarily better than human decision-

making. It is quite different, and it can be better or worse depending upon
the circumstances. Moreover, it is complicated to determine how to measure
better or worse. Since antiquity, a debate has raged in jurisprudence
regarding whether to follow the rule of law rigidly or whether to make
exceptions out of compassion or justice. What is better or worse depends
upon one’s philosophical views.
But couldn’t human and machine decision-making be combined to produce

the perfect mix? Maybe, but it is far from simple. Merely stirring human and
machines together will not readily make an inspired blend of the best each
has to offer. In many cases, human and machine decision-making don’t mix
very well. At the very least, blending requires a complicated recipe, and such
recipes are lacking in law and policy and have been elusive thus far.
11

Discussions of combining human and machine decision-making often focus

on the virtues and vices of each, and assume the virtues of one will counteract
the vices of the other. But this is mere speculation. What is lacking is an
approach for how humans and machines might work together.
Merely allowing a human to overrule a machine is far too simplistic an

answer, as it remains unclear whether humans will overrule machines
appropriately. As Crootof, Kaminski, and Price observe, a “hybrid system”
consisting of human and machines could “all too easily foster the worst of
both worlds, where human slowness roadblocks algorithmic speed, human
bias undermines algorithmic consistency, or algorithmic speed and
inflexibility impair humans’ ability to make informed, contextual
decisions.”29
Ben Green points out an even more fundamental conflict between

algorithmic and human decision-making – algorithms offer “consistency
and rule-following” whereas humans offer “flexibility and discretion.” When
policymakers call for humans to oversee algorithms, they often don’t
recognize the “inherent tension” between these things and fail to provide
sufficient guidance about how to resolve this tension.30
Policymakers and laws thus lack an effective plan for how humans and
machines will work together beyond vague hopes that somehow they’ll just
get along.
4. Mesmerized by Math and Machines
The hope that humans and machines can decide better together is not just
vague and unsubstantiated; in fact, strong evidence shows that there are
significant problems with combining humans and machines in making
decisions. Humans can perform poorly when using algorithmic output
because of certain biases and flaws in human decision-making. Far from
serving to augment or correct human decision-making, algorithms can
exacerbate existing weaknesses in human thinking, making the decisions
worse rather than better.
Many commentators have pointed out that people are prone to “automation
bias” – to trust algorithms without sufficient skepticism.31 As one court
declared in 1968: “Mathematics, a veritable sorcerer in our computerized
society, while assisting the trier of fact in the search for truth, must not cast
29 Rebecca Crootof, Margot E. Kaminski & W. Nicholson Price II, Humans in the Loop, 76
Vand. L. Rev. 429, 468 (2023).
30 Ben Green, The Flaws of Policies Requiring Human Oversight of Government Algorithms,
45 Computer Law & Security Rev. 1, 12 (2022).
31 Dan L. Burk, Algorithmic Legal Metrics, 96 Notre Dame L. Rev. 1147, 1161 (2020); Katrina
Geddes, The Death of the Legal Subject, 25 Vand. J. Ent. & Tech. L. 1, 36 (2023) ( “automation
bias” is “the tendency of human decision makers to accept, rather than challenge, quantitative
assessments”); Margot E. Kaminski & Jennifer R. Urban, The Right to Contest AI, 121 Colum.
L. Rev. 1957, 1961 (2021) (“Humans may exhibit an “automation bias” that creates
overconfidence in machine decisions, and an ensuing bias against challenges to those
decisions.”).
12

a spell over him.”32 Laurence Tribe contends that “the very mystery that
surrounds mathematical arguments - the relative obscurity that makes them
at once impenetrable by the layman and impressive to him - creates a
continuing risk that he will give such arguments a credence they may not
deserve and a weight they cannot logically claim.”33
Various methods of quantification and automation create an illusion that

humans are not involved in shaping the output, which will then be free of the
taint of human bias or animus. But this is far from the case; machines merely
hide the human element but don’t eliminate it.34 AI is far from human-free;
humans are involved in nearly every aspect of AI at every stage of
development.35
Empirical studies show that people readily defer to automated systems,

overlook errors in algorithms, and deviate from algorithmic output in ways
that render a less accurate result.36 Moreover, Green notes, “people cannot
reliably balance an algorithm’s advice with other factors, as they often
overrely on automated advice and place greater weight on the factors that
algorithms emphasize.”37 Studies show that [a]utomation can also create a
diminished sense of control, responsibility, and moral agency among human
operators.”38
The unfortunate result, then, of having humans use algorithmic output for
decisions is that humans might fail to appreciate the human influence within
that output. Of course, humans can be educated and warned about these
problems, but it is not clear that this can save the day. Humans can become
“complacent” and “overreliant” when working with an autonomous system—
a tendency that “seems to afflict experts as much as novices, and is largely
resistant to training.”39
Moreover, there are significant pressures for humans to accept algorithmic

output rather than deviate from it and use their discretion, emotion, or
intangible considerations. Deviations from the algorithm will likely be
viewed quite suspiciously, require extensive justification, and take much
effort and moxie for human decisionmakers. Not only are humans internally
more inclined to rubber stamp algorithms, but external pressures will also
32 People v. Collins, 438 P.2d 33, 33 (1968).

33 Laurence H. Tribe, Trial by Mathematics: Precision and Ritual in the Legal Process, 84
Harv. L. Rev. 1329, 1334 (1971).
34 Kaminski & Urban, supra note X, at 1969 (noting that AI “obfuscates bias with layers of
ostensibly objective mathematical authority”); Burk Algorithmic Legal Metrics, supra note
X, at 1161 (“Scored data thus impart an illusion of objectivity because they are rendered by
machines, and their human initiators appear distantly removed from the ultimate output.”).
35 KATE CRAWFORD, ATLAS OF AI: POWER, POLITICS, AND THE PLANETARY COSTS OF ARTIFICIAL
INTELLIGENCE (2021).
36 Ben Green, The Flaws of Policies Requiring Human Oversight of Government Algorithms,
45 Computer Law & Security Rev. 1, 7 (2022).
37 Id. at 9.
38 Id. at 7.
39 John Zerilli, Alistair Knott, James Maclaurin, Colin Gavaghan, Algorithmic
Decision‑Making and the Control Problem, 29 Minds and Machines 555, 556 (2019).
13

push them toward doing so.
Ultimately, much more study must be devoted to figuring out the proper
ways for humans and machines to interact. The mere combination of
humans and machines is far from a solution – it is a problem itself yet to be
solved.
B. ACCURATE – BUT ON WHAT BASIS?

Algorithmic decisions are often touted as more “accurate” than human ones.
For example, as several commentators contend regarding sentencing
algorithms that predict future recidivism: “The most important computer
sentencing tools that have been used relate to predictions of reoffending.
These instruments have been demonstrated to be more accurate than
judicial assessments.”40
It remains questionable whether algorithms are really as accurate as their

evangelists contend.41 Moreover, we should not assume that accuracy is the
key goal, and it is unclear what an “accurate” decision even means.
1. Is Accuracy the Right Goal?
Regarding algorithmic predictions about recidivism, we have argued

elsewhere that such predictions can’t be verified because the future hasn’t
occurred yet, and the predictions lead to interventions that alter the future.
Algorithmic predictions can become a self-fulfilling prophecy.42 Thus, with
many algorithms, it is not possible to assess accuracy. There often is no one
right answer out there in some cosmic answer key. Even if accuracy could be
assessed, it is not clear what “accuracy” means for many decisions or if it is
the right goal for these decisions.
If the decision is whether a person is guilty of a crime, then there can be a

correct answer. If the decision is how long a criminal sentence should be,
there is not necessarily a correct answer. The answer depends upon one’s
theory of punishment as well as other goals in the criminal justice system. It
is far from clear that we can achieve an “accurate” punishment. Instead, we
should strive for a just punishment, one that is fair and that achieves valuable
social ends. There is no clear right or wrong answer when it comes to an
appropriate punishment as it involves balancing many considerations,
including fairness to individuals, societal interests, and other things.
Important values might be in conflict; tradeoffs must be made. The decision
teems with normative issues.
Algorithms may attempt to predict recidivism, but recidivism is dynamic and
40 Mirko Bagaric, Dan Hunter, and Nigel Stobbs, Erasing the Bias Against Using Artificial
Intelligence to Predict Future Criminality: Algorithms Are Color Blind and Never Tire, 88
U. Cin. L. Rev. 1037, 1038 (2019).
41 Matsumi & Solove, Prediction Society, supra note X, at X.
42 Matsumi & Solove, Prediction Society, supra note X, at X.
14

influenced by the sentence itself and how it is carried out. Recidivism is a

product of many factors, not something inherent in a person. It is shaped by
a person’s prison experience, by education and training during
incarceration, and by opportunities and social acceptance upon release. Of
course, the person plays a role too, but recidivism is produced by an
interaction of many different factors on an individual. There is no static
answer as to whether a person will recidivate, and it depends upon the very
sentencing decision in which recidivism is a factor.
2. The Conflict Between Accurate and Unbiased
Quantifiable data does not consist of neutral facts; such data is created and
curated by humans, which introduces bias into the algorithm.
Hypothetically, AI optimists propose, the data could be scrubbed of all bias.
But even if the most obvious biases are removed from the data, the data still
comes from a society of humans where bias infects nearly everything.
Often, trying to design an algorithm to produce an accurate and unbiased

result is asking it to do two contradictory things. If accuracy means reflecting
society, then the accurate output should be biased because society is riddled
with bias. Algorithmic output without bias is creating a false picture of
reality, like a history with all the murdering and plundering edited out. For
example, if we ask an algorithm to predict recidivism, it is difficult to imagine
how it can avoid reflecting bias because recidivism is affected by bias.
Algorithms used for hiring decisions and credit scoring will also be drawn to
bias because these things are biased, and the data is biased. Thus, an
algorithm often can’t produce an accurate and unbiased decision; it might be
able to produce one or the other. For unbiased decisions, it will be very
difficult because bias lives in every crevice and corner of society and
correlates with so many different types of data. A bias-free society is a fiction
that doesn’t follow from the biased data algorithms are trained on.
We might value bias-free decisions rather than ones that accurately follow
from the training data. After all, one purpose of using algorithms is to
improve upon human decision-making. But this demonstrates that we might
not really want an “accurate” decision; we want an ideal decision.
3. The Flaws of Scoring and Ranking
Of course, human judgment can be helped by machines, and machine

judgment might surpass human judgment in certain cases. But this is far
from certain, and there are many reasons why turning to machines might
make decisions worse
Consider law school rankings. A magazine, US News and World Report, has
ranked law schools using a formula that has been vehemently criticized by
law schools as being reductive, inaccurate, and unfair. Schools have
attempted to boycott the rankings by withholding data, but the magazine still
continues to rank schools by using an even more simplistic algorithm.
Can law schools be reduced to a singular ranking? People crave rankings,
15

partly because they simplify complex things and appear to make choices
easier by ignoring incommensurate dimensions. But this simplicity is just a
mirage; rankings distort in rather crude ways. One can reduce a
Shakespearean play to a simple plot summary, but this removes the most
important part – the artistry and beauty of the language. History can be
reduced to dates and events, but that does not make it useful or insightful.
Of course, people want the easy button, but life is not easy. The easy button
is a delusion. Unfortunately, many algorithmic tools are being used in this
way. They function to reduce the richness of life to certain quantifiable
elements.
Many decisions about humans are fraught with complex moral and policy
issues that elude an easy answer. For example, in sentencing, is recidivism
an appropriate factor to focus on? If so, how much weight should it be given?
Algorithms can try to predict recidivism. But they can’t answer the normative
question of how much recidivism should factor into the sentencing
calculation.
Far too often, though, the use of algorithms results in a lack of questioning
about these normative issues. The focus is solely on whether the algorithm is
accurately predicting recidivism rather than on how much recidivism should
be weighed in a sentencing decision. And the studies show that humans
might rely too much on the factors the algorithm computes because
algorithmic output is overly trusted.
4. Algorithms Cloak Problematic Decisions in Pseudoscience
Algorithms can make decisions appear more scientific, more “accurate,” and
more reliable and trustworthy. Even if the algorithms be used to make
decisions better (assuming this can be determined), they might be an overall
negative because they are dressing up a fraught activity in scientific-looking
vestments. During the heyday of phrenology – the false idea that physical
characteristics were linked to behavior – there was an attempt to dress it up
with science. This thin veneer of science was based on faulty studies.
Proponents raced forward with hope and enthusiasm; any contrary evidence
was ignored in their zeal to succeed.43
We see a similar story unfolding with algorithms. Algorithmic decision-

making is far from “accurate.” But we must continue! proponents proclaim.
Humans are awful decision-makers and algorithms have such great
promise. The future will be brighter! But these jubilant declarations are just
hype and hope.
Today, algorithms are widely being used to make many decisions about
people, and it is far too soon to be doing so on such a largescale basis. This
is, in essence, a grand experiment involving millions of people—probably
hundreds of millions and perhaps even more—without the controls and
43 Pierre Schlag, Law and Phrenology, 110 Harv. L. Rev. 877 (1997).
16

protections such human experiments typically require.
III. RETHINKING THE RELATIONSHIP

BETWEEN ALGORITHMS AND HUMANS
A. THE IMPORTANCE OF STRUCTURE
As Crootof, Kaminski, and Price aptly observe, “policymakers often assume
that adding a human to a machine system will result in the best of both
worlds. There’s a seductive simplicity to this “slap a human in it”
approach.”44 They argue that the law often fails to address the problems that
emerge when humans interact with machines or do much to define the role
that humans should play.45
When the law seeks to add a human in the loop, what does this mean?
Humans are already in the loop. Behind each machine, there are humans
who decide what data to use to train the model, what kind of output is
expected from the model, and how the algorithmic model will be used in the
decision-making process. Thus, what the law appears to mean by adding a
human is to have a human review algorithmic output at the end stage. But
this point might not be the ideal point for human involvement. Humans lack
the ability to process large datasets or understand specifically everything
algorithms are doing.46 Humans might thus not have the expertise or
cognitive ability to thoughtfully oversee algorithms. Perhaps the best place
for human oversight is in the creation and testing of algorithmic models.
Perhaps the humans involved in oversight should not be the humans at the
endpoint but somewhere at the beginning or middle. These humans are
already there, but the law often doesn’t hold them accountable.
If additional humans are to be added into the loop, the law must address
when humans should be added and what role these humans should have.
Regarding the role of additional humans in the loop, the law should identify
the goals that adding the humans will achieve and how the humans will
achieve these goals. In other words, there must be a meaningful gameplan
for how and why humans are to be added.
Structuring human-machine combinations involves more than just the

quality of the decisions that are produced. In some situations, adding
automated tools to assist and improve human decisions (and human
performance more generally) can lead to harm to humans. Automated
systems can function as surveillance mechanisms that micromanage humans
in tyrannical ways. For example, in the context of truck driving, Karen Levy
explores how AI is used “to address human ‘weakness’ through constant,
44Crootof, Kaminksi, & Price, Humans in the Loop, supra note X, at 437.
45Id.
46 John Zerilli, Alistair Knott, James Maclaurin, Colin Gavaghan, Algorithmic
Decision‑Making and the Control Problem, 29 Minds and Machines 555, 560 (2019)
(“Humans are often at a severe epistemic disadvantage vis-à-vis the systems they are tasked
with supervising.”).
17

intimate, visceral monitoring.” She notes how various automation

technologies used to augment truck drivers adversely affect the drivers by
creating an “intimate invasion into their work and bodies,” and resulting in
an “uneasy, confrontational” relationship.47 Certainly, automation can help
improve human decision-making, but it can come at a high cost to the
humans it purportedly helps. Human-machine unions are not necessarily
happy ones, and automation can become a tool of oppression. Perhaps there
might be ways to restructure human-machine relationships to minimize
these negative effects. Perhaps any unavoidable negative effects are a
justifiable price to pay for improved decision-making quality. But perhaps in
some cases—maybe many cases—the benefits aren’t worth the costs.
We cannot resolve the ideal structure here, but we want to emphasize

emphatically that structure matters. Merely uniting humans and machines
is naïve and simplistic. Extensive thought must be given to the roles humans
do play and should play in the process as well as where they should be added
and how they should perform their roles.
B. THE VIRTUES OF GOOD DECISIONS

Ultimately, we must never forget the overarching goal – the law should aim
to produce good decisions. Good qualities in decision-making include a
commitment to the scientific method, humility, feedback loops, fairness,
morality, lack of bias, consideration of as much relevant information as
possible, empathy, due process, listening to all stakeholders, diversity,
practicality, accuracy as to facts, critical reflection, philosophical depth,
open-mindedness, awareness of context, and much more.
Some decisions might call for more accuracy, but others less so. For landing
a plane, we want high accuracy, but for decisions about school admissions,
credit scoring, or criminal sentencing, other values are also quite important.
There is no one-size-fits-all approach to regulating AI, as the decisions it will
be employed to help make are quite different and demand different
considerations.
Algorithms focus on correlations, not causation, and rarely do those

designing the algorithm or using the algorithm ask why the correlations
exist.48 They often don’t care about causation; correlation gets the job done.
Human overreliance on algorithms can deter critical reflection, further
study, and deeper thought about causal relationships. For example, in a
decision to admit a student into law school, the algorithm might be
programmed to look for students who will pass the bar and obtain a lucrative
job. But this can overlook students who might struggle with the bar but will
47 KAREN LEVY, DATA DRIVEN: TRUCKERS, TECHNOLOGY, AND THE NEW WORKPLACE SURVEILLANCE
148 (2023).
48 As Mireille Hildebrandt aptly notes, “profilers are not very interested in causes or reasons,
their interest lies in a reliable prediction, to allow adequate decision making.” Mireille
Hildebrandt, Defining Profiling: A New Type of Knowledge?, in PROFILING THE EUROPEAN
CITIZEN: CROSS-DISCIPLINARY PERSPECTIVES 17, 18 (Mirielle Hildebrandt & Seth Gutwirth eds.
2008).
18

eventually pass. It can also overlook students who work in public interest
jobs. It can dismiss students who could succeed at the bar if given the right
training or opportunities. It might fail to consider students who use the
degree to go into a non-legal career (politics, writing, journalism) or who go
into a low-paying public interest position.
Ultimately, the goal should be good decisions, and such decisions are quite
varied and contextual. As a general matter, the law does a very poor job in
guiding good decision-making, whether it be by human or machine. In the
end, there is no substitute for good judgment. Awful humans produce awful
results, whether alone or combined with a machine.
When we focus on how awful human decision-making can be, it’s tempting
to rush to embrace the machines. But we might be shortchanging the good
human elements in decisions. AI decides differently than humans. Machines
often standardize decisions, which brings about good as well as bad
consequences. Algorithms might bring greater uniformity but might also
lead to rigidity when flexible decisions might be preferable. There are
certainly times when we want a standardized decision but there are other
times when we want a Solomonic one.
In the end, good decisions depend upon good humans. We need much more
thought and study about when and how machine decision-making can
replace, help, or be mixed with human decision-making. Ultimately, the
success of the integration of AI into important decisions about people will
depend quite heavily on humans and how they think about algorithmic
decisions and their own decisions.
19

AI Algorithms and Awful Humans - Daniel Solove

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

AI Algorithms and Awful Humans - Daniel Solove

Uploaded by

Copyright:

Available Formats

AI, ALGORITHMS,

AND AWFUL HUMANS

96 FORDHAM LAW REVIEW __

Draft: October 16, 2023

Electronic copy available at: https://ssrn.com/abstract=4603992

We contend that algorithmic decision-making is being relied upon too

Electronic copy available at: https://ssrn.com/abstract=4603992