Scientific American Special Collectors Edition - Fall 2022

SPECIAL COLLECTOR’S EDITION
FALL 2022
TRUTH
IES
VS.
L
THE S
IN
C
S
I
I
E
D
N
E
A
C
N
E
D
OF
H
M
OW
ISIN
TO
F O
K
ebook
u
a
n
R
N
n
t
d
e
M
O
Tw
A
W
it
T
t
W
IO
er a
nce th
• Co • The psy
• Imp
r
i
l
n
i
H
e
c
k
i
N
t
in
b
A
o
g
A
T
•
ND L
’S
B
ng
R
s
attl
piracy
ias aff
© 2022 Scientific American

D
t
E
c
• Fac g antiscie ehind cons ality •

ractin chology b ec ts our re
h
i
e
A
E
o
C
ety •
larizi ing disinfo
E P
r
ries •
T
ma
I
t
O N
ion •
FROM THE EDITOR
ESTABLISHED 1845
Truth vs. Lies is published by

Truth under Attack
the staff of Scientific American, Truths should be stubborn things, right? Not in today’s society. A set of polls conducted
with project management by: this summer revealed about 70 percent of Republican voters still believe that Joe Biden did
not win the 2020 presidential election, despite extensive bipartisan investigations into vot-
Editor in Chief: Laura Helmuth er fraud that validated the trustworthiness of the election. Online, the YouTube suggestion
Managing Editor: Jeanna Bryner
algorithm has been shown to steer viewers toward more extreme or far-fetched videos,
spreading conspiracy theories and fringe beliefs. And users on other platforms such as Tik-
Senior Editor, Collections:
Tok and Twitter deliberately disseminate misinformation about lifesaving vaccines.
Andrea Gawrylewski
Lies, extremism and the manipulation of reality seem to be common themes in today’s cur-
Creative Director: Michael Mrak rent events. Because all untruths
are antithetical to science, we
Issue Designer: Lawrence R. Gendron
hope this issue will serve in some
Senior Graphics Editor: Jen Christiansen measure as an antidote to the poi-
Associate Graphics Editor: A
manda Montañez son of manipulated facts and
other forms of mendacity. Never
Photography Editor: Monica Bradley
has it been more important to un
Associate Photo Editor: Liz Tormes derstand the science of how we
humans determine what is true.
Copy Director: Maria-Christina Keller
For starters, our perception is
Senior Copy Editors: inherently subjective (page 4).
Angelique Rondeau, Aaron Shattuck We may believe that we are open-
Managing Production Editor: Richard Hunt minded creatures, but most peo-
ple latch on to ideas that seem to
Prepress and Quality Manager:
validate their own preconceived
Silvia De Santis
beliefs (page 32)—even if this behavior prevents them from seeing new solutions (page 10).
Executive Assistant Supervisor: Maya Harty Such ingrained implicit bias has served us well in the course of evolution, but in the mod-
Senior Editorial Coordinator: Brianne Kane
ern era, it more often leads us astray (page 16).
Indeed, humans famously make, and commit to, decisions even when they don’t have all
the facts (page 38), and in some cases, those leaps to conclusions make some accept conspir-
President: K
imberly Lau
acy theories and other misinformation (page 52). Good news: the practice of questioning
Vice President, Commercial: Andrew Douglas your deepest-held beliefs, especially in light of strong evidence, can strengthen your objec-
tivity and critical thinking skills (page 48).
Publisher and Vice President:
Nowhere are our failings at objective reasoning more exploitable than on social media,
Jeremy A. Abbate
used globally by billions. Facebook and other platforms enable the spread of misinforma-
Vice President, Content Services: tion that sows social unrest—in particular, meme culture has been shown to propagate lies
Stephen Pincock
and increase division (page 58). Platform algorithms that take advantage of our psycholog-
Associate Vice President, ical vulnerabilities trap us in echo chambers (page 64). In the end, users become the unwit-
Business Development: Diane McGarvey ting vectors of these threats (page 57).
Civic life suffers because of these malevolent forces. Turmoil, anxiety and a sense that soci-
Marketing Director, Institutional Partnerships
ety is in jeopardy lead to the kind of polarization that makes winning an argument more impor-
and Customer Development: Jessica Cole
tant than understanding opponents’ viewpoints (page 82). We are stuck in what philosopher
Programmatic Product Manager: Zoya Lysak Kathleen Higgins describes as the post-truth era, where there is no longer an expectation that
Director, Integrated Media: Matt Bondlow politicians or pundits will be honest (page 86). Rejection of expertise and sound data has even
led the highest court in the land to issue rulings that endanger human health (page 98).
Senior Product Manager: Ian Kelly
Although the human mind comes equipped with built-in obstacles to objective thinking,
Senior Web Producer: Jessica Ramirez we shouldn’t give in to ignorance and bias. Psychologist Douglas T. Kenrick and his co-authors
Senior Commercial Operations Coordinator: offer simple interventions that can make us more open-minded, scientific thinkers (page 100).
Christine Kaelin In fact, scientists can look to philosophy to aid in some self-examination about how much, in
the hands of subjective creatures, the tools of science can ultimately discover (page 114).
Custom Publishing Editor: Lisa Pallatroni
The common theme in many of these seemingly abysmal examinations of the state of our
Head, Communications, USA: Rachel Scheer societal affairs is a heartening bright spot. By just being aware of how we perceive informa-
Boris Zhitkov/Getty Images
Press Manager: Sarah Hausman tion, we can protect ourselves from disinformation and hogwash. We don’t have to always
agree, but at least we’ll be anchored in what is real and what is not.
Production Controller: Madelyn Keyes-Milch
Advertising Production Manager: Andrea Gawrylewski

Michael Broomes Senior Editor, Collections, editors@sciam.com
SCIENTIFICAMERICAN.COM | 1

SPECIAL EDITION
Volume 32, Number 5, Fall 2022
GRAPPLING SOCIAL MEDIA’S

WITH REALITY INFLUENCE
4 Our Inner Universes 57 When “Like” Is a Weapon
Reality is constructed by the
brain, and no two brains
DECISION-MAKING Everyone is an agent in the
new information warfare.
are exactly alike. B
y Anil K. Seth 38 Tough Calls By the Editors
How we make decisions in the
10 Why Good Thoughts 58 A New World Disorder
face of incomplete knowledge and
Block Better Ones Our willingness to share content
uncertainty. B
y Baruch Fischhoff
While we are working through without thinking is exploited
a problem, the brain’s tendency 44 Confronting Unknowns to spread disinformation.
to stick with familiar ideas How to interpret uncertainty in By Claire Wardle
can prevent us from seeing common forms of data visualiza
64 The Attention Economy
superior solutions. tion. B
y Jessica Hullman
Understanding how algorithms
By Merim Bilalić and Peter McLeod
48 The Cause of America’s and manipulators exploit our
16 How to Think Post-Truth Predicament cognitive vulnerabilities empowers
about “Implicit Bias” People have been manipulated us to fight back.
Amid a controversy, it’s important to think that beliefs needn’t change By Filippo Menczer and
to remember that implicit bias in response to evidence, making Thomas Hills
is real—and it matters. us more susceptible to conspiracy
72 How Facebook Hinders
By Keith Payne, Laura Niemi theories, science denial and
Misinformation Research
and John M. Doris extremism. B y Andy Norman
The platform strictly limits and
18 Schooled in Lies 50 Perfect Storm controls data access, which stymies
Kids are prime targets of dis for Fringe Science scientists. B
y Laura Edelson and
information, yet educators can It’s always been with us, but Damon McCoy
not figure out how best to teach in a time of pandemic, its practi
74 The Shared Past That Wasn’t
them to separate fact from fiction. tioners have an amplified capac
How Facebook, fake news and
By Melinda Wenner Moyer ity to unleash serious harm.
friends are altering memories
By David Robert Grimes
24 Climate Miseducation and changing history.
How oil and gas representatives 52 Leaps of Confusion By Laura Spinney
manipulate the standards for People who jump to conclusions
80 The Black Box
courses and textbooks, from tend to believe in conspiracy
of Social Media
kindergarten to 12th grade. theories, are overconfident and
Social media companies need to
By Katie Worth make other mistakes in their
give their data to independent
thinking. B
y Carmen Sanchez and
32 Why We Trust Lies researchers to better understand
David Dunning
The most effective misinformation how to keep users safe.
starts with seeds of truth. 56 Big Data and Small Decisions By Renée DiResta, Laura Edelson,
By Cailin O’Connor and For individuals a deluge of facts can Brendan Nyhan and
James Owen Weatherall be a problem. B y Zeynep Tufekci Ethan Zuckerman
2 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

POLITICS FINDING ANSWERS
82 Arguing the Truth IN SCIENCE
As political polarization grows, 100 The Science of
the arguments we have with one Antiscience Thinking
another may be shifting our Convincing people who doubt
understanding of truth itself. the validity of climate change
By Matthew Fisher, Joshua Knobe, and evolution to change their
Brent Strickland and Frank C. Keil beliefs requires overcoming
86 Post-Truth: a set of ingrained cognitive biases.
A Guide for the Perplexed By Douglas T. Kenrick,
If politicians can lie without Adam B. Cohen, Steven L. Neuberg
condemnation, what are scien and Robert B. Cialdini DEPARTMENTS
tists to do? B
y Kathleen Higgins 106 How Professional FROM THE EDITOR
88 Why We Believe Truth Seekers 1 Truth under Attack
Conspiracy Theories Search for Answers
END NOTE
Baseless theories threaten our Nine experts describe how
they sort signal from noise. 116 Fake-News Sharers
safety and democracy. It turns
As told to Brooke Borel Highly impulsive people who
out that specific emotions make
lean conservative are most likely
people prone to such thinking.
112 The Truth about to pass along false news stories.
By Melinda Wenner Moyer
Scientific Models By Asher Lawson and
94 Contagious Dishonesty They don’t necessarily try Hemant Kakkar
Dishonesty begets dishonesty, to predict what will happen—
rapidly spreading unethical but they can help us understand
behavior through a society. possible futures. Articles in this special issue are updated or adapted from previous
By Dan Ariely and By Sabine Hossenfelder issues of Scientific American and from ScientificAmerican.com.
Copyright © 2022 Scientific American, a division of Springer Nature
Ximena Garcia-Rada
114 How Much Can We Know? America, Inc. All rights reserved. Scientific American Special (ISSN
1936-1513), Volume 32, Number 5, Fall 2022, published by Scientific
98 Evidence Shouldn’t The reach of the scientific American, a division of Springer Nature America, Inc., 1 New York
Be Optional method is constrained by the Plaza, Suite 4600, New York, N.Y. 10004-1562. Canadian BN No.
This Supreme Court often ignores limitations of our tools and the 127387652RT; TVQ1218059275 TQ0001. To purchase additional
science when handing down quantities: U.S., $13.95 each; elsewhere, $17.95 each. Send payment
intrinsic impenetrability of some
to Scientific American Back Issues, P.O. Box 3187,
decisions, and it affects far too of nature’s deepest questions. Harlan, Iowa 51537. Inquiries: fax 212-355-0408
many lives. B
y the Editors By Marcelo Gleiser or telephone 212-451-8415. Printed in U.S.A.

GRAPPLING WITH REALITY
OUR
INNER
UNIVERSES Reality is constructed by the brain,
and no two brains are exactly alike
By Anil K. Seth
Illustration by Brook VanDevelder
“We do not see things as they are, we see them as we are.”

—from Seduction of the Minotaur, by Anaïs Nin (1961)


O
n the 10 th of April 2019 Pope Francis, President Salva Kiir o f

South Sudan and former rebel leader Riek Machar sat down together for
dinner at the Vatican. They ate in silence, the start of a two-day retreat aimed
at reconciliation from a civil war that had killed some 400,000 people since
2013. At about the same time in my laboratory at the University of Sussex in
England, Ph.D. student Alberto Mariola was starting to work on an experiment
in which volunteers experience being in a room they believe is there but is not.
In psychiatry clinics across the globe, people arrive complaining that things
no longer seem “real” to them, whether it is the world around them or their
own selves. In the fractured societies in which we live, what is real—and what
is not—seems to be increasingly up for grabs. Warring sides may experience
and believe in different realities. Perhaps eating together in silence can help because it offers a
small slice of reality that can be agreed on, a stable platform on which to build further understanding.
We need not look to war and psychosis to find radically dif- reality we experience—the way things s eem—is not a direct re-
ferent inner universes. In 2015 a badly exposed photograph of a flection of what is actually out there. It is a clever construction
dress tore across the Internet, dividing the world into those who by the brain, for the brain. And if my brain is different from your
saw it as blue and black (me included) and those who saw it as brain, my reality may be different from yours, too.
white and gold (half my lab). Those who saw it one way were so
convinced they were right—that the dress truly was blue and THE PREDICTIVE BRAIN
black or white and gold—that they found it almost impossible to In Plato’s Allegory of the Cave, p risoners are chained to a blank
believe that others might perceive it differently. wall all their lives, so that they see only the play of shadows cast
We all know that our perceptual systems are easy to fool. The by objects passing by a fire behind them, and they give the shad-
popularity of visual illusions is testament to this phenomenon. ows names because for them the shadows are what is real. A thou-
Things seem to be one way, and they are revealed to be another: sand years later, but still a thousand years ago, Arabian scholar
two lines appear to be different lengths, but when measured they Ibn al-Haytham wrote that perception, in the here and now, de-
are exactly the same; we see movement in an image we know to pends on processes of “judgment and inference” rather than in-
be still. The story usually told about illusions is that they exploit volving direct access to an objective reality. Hundreds of years lat-
quirks in the circuitry of perception, so that what we perceive er again Immanuel Kant realized that the chaos of unrestricted
deviates from what is there. Implicit in this story, however, is the sensory data would always remain meaningless without being giv-
assumption that a properly functioning perceptual system will en structure by preexisting conceptions or “beliefs,” which for him
render to our consciousness things precisely as they are. included a priori frameworks such as space and time. Kant’s term
The deeper truth is that perception is never a direct window “noumenon” refers to a “thing in itself”—Ding an sich—an objec-
onto an objective reality. All our perceptions are active constructive reality that will always be inaccessible to human perception.
tions, brain-based best guesses at the nature of a world that is Today these ideas have gained a new momentum through an
forever obscured behind a sensory veil. Visual illusions are frac- influential collection of theories that turn on the idea that the
tures in the Matrix, fleeting glimpses into this deeper truth. brain is a kind of prediction machine and that perception of the
Take, for example, the experience of color—say, the bright red world—and of the self within it—is a process of brain-based pre-
of the coffee mug on my desk. The mug really does seem to be diction about the causes of sensory signals.
red: its redness seems as real as its roundness and its solidity. These new theories are usually traced to German physicist
These features of my experience seem to be truly existent prop- and physiologist Hermann von Helmholtz, who in the late 19th
erties of the world, detected by our senses and revealed to our century proposed that perception is a process of unconscious in-
mind through the complex mechanisms of perception. ference. Toward the end of the 20th century Helmholtz’s notion
Yet we have known since Isaac Newton that colors do not exist was taken up by cognitive scientists and artificial-intelligence
out there in the world. Instead they are cooked up by the brain from researchers, who reformulated it in terms of what is now gener-
mixtures of different wavelengths of colorless electromagnetic ra- ally known as predictive coding or predictive processing.
diation. Colors are a clever trick that evolution has hit on to help The central idea of predictive perception is that the brain is at-
the brain keep track of surfaces under changing lighting conditions. tempting to figure out what is out there in the world (or in here, in
And we humans can sense only a tiny slice of the full electromag- the body) by continually making and updating best guesses about
netic spectrum, nestled between the lows of infrared and the highs the causes of its sensory inputs. It forms these best guesses by com-
of ultraviolet. Every color we perceive, every part of the totality of bining prior expectations or “beliefs” about the world, together
each of our visual worlds, comes from this thin slice of reality. with incoming sensory data, in a way that takes into account how
Just knowing this is enough to tell us that perceptual experi- reliable the sensory signals are. Scientists usually conceive of this
ence cannot be a comprehensive representation of an external process as a form of Bayesian inference, a framework that spec-
objective world. It is both less than that and more than that. The ifies how to update beliefs or best guesses with new data when

both are laden with uncertainty. ple. This distinction explains why con-
In theories of predictive perception, ceiving of perception as controlled hal-
the brain approximates this kind of lucination does not mean it is okay to
Bayesian inference by continually gen- jump in front of a bus. This bus has pri-
erating predictions about sensory sig- mary qualities of solidity and space oc-
nals and comparing these predictions cupancy that exist independently of
with the sensory signals that arrive at our perceptual machinery and that can
the eyes and the ears (and the nose and do us injury. It is the way in which the
the fingertips, and all the other senso- bus appears to us that is a controlled
ry surfaces on the outside and inside hallucination, not the bus itself.
of the body). The differences between
predicted and actual sensory signals TRIPPING IN THE LAB
give rise to so-called prediction errors, A g r o w i n g b o dy of evidence sup-
which are used by the brain to update ports the idea that perception is con-
its predictions, readying it for the next trolled hallucination, at least in its
round of sensory inputs. By striving to broad outlines. A 2015 study by Chris-
minimize sensory-prediction errors ev- toph Teufel of Cardiff University in
erywhere and all the time, the brain Wales and his colleagues offers a
implements approximate Bayesian in striking example. In this study, the
ference, and the resulting Bayesian ability to recognize so-called two-tone
best guess is what we perceive. images was evaluated in patients with
To understand how dramatically early-stage psychosis who were prone
this perspective shifts our intuitions to hallucinations.
about the neurological basis of per- POORLY EXPOSED p hotograph of a dress Take a look at the top photograph
ception, it is helpful to think in terms appears blue and black to some people, on page 9—a sample of a two-tone
of bottom-up and top-down direc- white and gold to others. image. Probably all you will see is a
tions of signal flow in the brain. If we bunch of black-and-white splotches.
assume that perception is a direct window onto an external real- Now look at the image at the bottom of that page. Then have an-
ity, then it is natural to think that the content of perception is car- other look at the first photo; it ought to look rather different.
ried by bottom-up signals—those that flow from the sensory sur- Where previously there was a splotchy mess, there are now dis-
faces inward. Top-down signals might contextualize or finesse tinct objects, and something is happening.
what is perceived, but nothing more. Call this the “how things What I find remarkable about this exercise is that in your sec-
seem” view because it seems as if the world is revealing itself to ond examination of the top image, the sensory signals arriving at
us directly through our senses. your eyes have not changed at all from the first time you saw it.
The prediction machine scenario is very different. Here the All that has changed are your brain’s predictions about the causes
heavy lifting of perception is performed by the top-down signals that of these sensory signals. You have acquired a new high-level per-
convey perceptual predictions, with the bottom-up sensory flow serv- ceptual expectation, and this changes what you consciously see.
ing only to calibrate these predictions, keeping them yoked, in some If you show people many two-tone images, each followed by
appropriate way, to their causes in the world. In this view, our per- the full picture, they might subsequently be able to identify a good
ceptions come from the inside out just as much as, if not more than, proportion of two-tone images, though not all of them. In Teufel’s
from the outside in. Rather than being a passive registration of an study, people with early-stage psychosis were better at recogniz-
external objective reality, perception emerges as a process of active ing two-tone images after having seen the full image than were
construction—a controlled hallucination, as it has come to be known. healthy control subjects. In other words, being hallucination-
Why controlled hallucination? People tend to think of halluci- prone went along with perceptual priors having a stronger effect
nation as a kind of false perception, in clear contrast to veridical, on perception. This is exactly what would be expected if halluci-
true-to-reality, normal perception. The prediction machine view nations in psychosis depended on an overweighting of perceptu-
suggests instead a continuity between hallucination and normal al priors so that they overwhelmed sensory prediction errors, un-
perception. Both depend on an interaction between top-down, mooring perceptual best guesses from their causes in the world.
brain-based predictions and bottom-up sensory data, but during Recent research has revealed more of this story. In a 2021
hallucinations, sensory signals no longer keep these top-down pre- study, Biyu He of New York University and her colleagues had
dictions appropriately tied to their causes in the world. What we neurosurgical patients look at ambiguous images, such as a Neck-
call hallucination, then, is just a form of uncontrolled perception, er cube, that constantly flip between two different appearances
just as normal perception is a controlled form of hallucination. even though the sensory input remains the same. By analyzing
This view of perception does not mean that nothing is real. Writ- the signals recorded from within the patients’ brains, they dis-
ing in the 17th century, English philosopher John Locke made an covered that information flowed more strongly in a top-down,
Swiked.tumblr.com
influential distinction between “primary” and “secondary” quali- “inside-out” direction when the perceived appearance was con-
ties. Primary qualities of an object, such as solidity and occupancy sistent with the patients’ biases, as would be expected if percep-
of space, exist independently of a perceiver. Secondary qualities, in tual predictions were strong in this case. And when the perceived
contrast, exist only in relation to a perceiver—color is a good exam- appearance was inconsistent with preexisting biases, infor

mation flow was stronger in the bottom-up direction, suggest- The basic idea is simple. We again prerecorded some panoram-
ing a “prediction error” signal. This is an exciting new develop- ic video footage, this time of the interior of our VR lab rather than
ment in mapping the brain basis of controlled hallucinations. of an outside campus scene. People coming to the lab are invited
In my lab we have taken a different approach to exploring the to sit on a stool in the middle of the room and to put on a VR head-
nature of perception and hallucination. Rather than looking into set that has a camera attached to the front. They are encouraged
the brain directly, we decided to simulate the influence of over- to look around the room and to see the room as it actually is, via
active perceptual priors using a unique virtual-reality setup mas- the camera. At some point, without telling them, we switch the
terminded by our resident VR guru, Keisuke Suzuki. We call it, feed so that the headset now displays not the live real-world scene
with tongue firmly in cheek, the “hallucination machine.” but rather the prerecorded panoramic video. Most people in this
Using a 360-degree camera, we first recorded panoramic video situation continue to experience what they are seeing as real even
footage of a busy square in the University of though it is now a fake prerecording. (This
Sussex campus on a Tuesday at lunchtime. We is actually very tricky to pull off in practice—
then processed the footage through an algo-
rithm based on Google’s AI program Deep-
OUR PERCEPTIONS it requires careful color balancing and align-
ment to avoid people noticing any difference
Dream to generate a simulated hallucination. COME FROM THE that would tip them off to the shift.)
What happens is that the algorithm takes a
so-called neural network—one of the work- INSIDE OUT JUST AS I find this result fascinating because it
shows that it is possible to have people ex-
horses of AI—and runs it backward. The net-
work we used had been trained to recognize
MUCH AS, IF NOT perience an unreal environment as being
fully real. This demonstration alone opens
objects in images, so if you run it backward, MORE THAN, FROM new frontiers for VR research: we can test
updating the network’s input instead of its
output, the network effectively projects what
THE OUTSIDE IN. the limits of what people will experience,
and believe, to be real. It also allows us to in-
it “thinks” is there onto and into the image. vestigate how experiencing things as being
Its predictions overwhelm the sensory inputs, tipping the balance real can affect other aspects of perception. Right now we are run-
of perceptual best guessing toward these predictions. Our partic- ning an experiment to find out whether people are worse at de-
ular network was good at classifying different breeds of dogs, so tecting unexpected changes in the room when they believe that
the video became unusually suffused with dog presences. what they are experiencing is real. If things do turn out this way
Many people who have viewed the processed footage through (the study is still ongoing, despite being heavily delayed by a glob-
the VR headset have commented that the experience is rather al pandemic), that finding would support the idea that the per-
reminiscent not of the hallucinations of psychosis but of the ex- ception of things as being real itself acts as a high-level prior that
uberant phenomenology of psychedelic trips. can substantively shape our perceptual best guesses, affecting the
More recently, we have implemented the hallucination ma- contents of what we perceive.
chine in different ways to simulate different kinds of altered vi-
sual experience. By extending our algorithm to include two cou- THE REALITY OF REALITY
pled neural networks—a “discriminator network” much like the The idea that the world of our experience might not be real
one in our original study and a “generator network” that has is an enduring trope of philosophy and science fiction, as well as
been trained to reproduce (“generate”) its input image—we have of late-night pub discussions. Neo in T he Matrix takes the red
been able to model different types of hallucination. For example, pill, and Morpheus shows him how what he thought was real is
we have modeled the complex hallucinatory experiences report- an elaborate simulation, while the real Neo lies prone in a human
ed by people with Parkinson’s disease and some forms of demen- body farm, a brain-in-a-vat power source for a dystopian AI. Phi-
tia; the patterned, geometric hallucinations that occur after the losopher Nick Bostrom of the University of Oxford has famously
loss of foveal vision, as happens in Charles Bonnet syndrome; argued, based largely on statistics, that we are likely to be living
and a range of psychedeliclike hallucinations. We hope that by inside a computer simulation created in a posthuman age. I dis-
understanding hallucinations better, we will be able to under- agree with this argument in part because it assumes that con-
stand normal experience better, too, because predictive percep- sciousness can be simulated—I do not think that this is a safe as-
tion is at the root of all our perceptual experience. sumption—but it is thought-provoking nonetheless.
Although these chunky metaphysical topics are fun to chew
THE PERCEPTION OF REALITY on, they are probably impossible to resolve. Instead what we have
Although the hallucination machine is undoubtedly trippy, peo- been exploring throughout this article is the relation between ap-
ple who experience it are fully aware that what they are experienc- pearance and reality in our conscious perceptions, where part of
ing is not real. Indeed, despite rapid advances in VR technology this appearance is the appearance of being real itself.
and computer graphics, no current VR setup delivers an experi- The central idea here is that perception is a process of active in-
ence sufficiently convincing to be indistinguishable from reality. terpretation geared toward adaptive interaction with the world via
This is the challenge we took up when designing a new the body rather than a re-creation of the world within the mind.
“substitutional reality” setup at Sussex—the one we were The contents of our perceptual worlds are controlled hallucina-
working on when Pope Francis convened the retreat with Salva tions, brain-based best guesses about the ultimately unknowable
Kiir and Riek Machar. Our aim was to create a system in which causes of sensory signals. For most of us, most of the time, these
volunteers would experience an environment as being real—and hallucinations are experienced as real. As Canadian rapper and
believe it to be real—when in fact it was not real. science communicator Baba Brinkman suggested to me, when we

agree about our hallucinations, maybe

that’s what we call reality.
But we do not always agree, and we do
not always experience things as real. Peo-
ple with dissociative psychiatric conditions
such as derealization or depersonalization
syndrome report that their perceptual
worlds, even their own selves, lack a sense
of reality. Some kinds of hallucination, var-
ious psychedelic hallucinations among
them, combine a sense of unreality with
perceptual vividness, as does lucid dream-
ing. People with synesthesia consistently
have additional sensory experiences, such
as perceiving colors when viewing black let-
ters, which they recognize as not real. Even
with normal perception, if you look direct-
ly at the sun you will experience the subse-
quent retinal afterimage as not being real.
There are many such ways in which we ex-
perience our perceptions as not fully real.
What this means to me is that the prop- TWO-TONE IMAGE looks like a mess of black-and-white splotches, until you see
erty of realness that attends most of our the full image (below).
perceptions should not be taken for grant-
ed. It is another aspect of the way our brain settles on its Bayes- structive, creative mechanisms of perception has unexpected so-
ian best guesses about its sensory causes. One might thus ask what cial relevance. Perhaps once we can appreciate the diversity of ex-
purpose it serves. Perhaps the answer is that a perceptual best perienced realities scattered among the billions of perceiving
guess that includes the property of being real is usually more fit brains on this planet, we will find new platforms on which to build
for purpose—that is, better able to guide behavior—than one that a shared understanding and a better future—whether between
does not. We will behave more appropriately with respect to a cof- sides in a civil war, followers of different political parties, or two
fee cup, an approaching bus or our partner’s mental state when people sharing a house and faced with washing the dishes.
we experience it as really existing.
But there is a trade-off. As illustrated by the dress illusion, when Anil K. Seth is a professor of cognitive and computational neuroscience at the University
we experience things as being real, we are less able to appreciate that of Sussex in England. His research focuses on the biological basis of consciousness, and he
our perceptual worlds may differ from those of others. (A popu- is author of Being You—A New Science of Consciousness (Dutton, 2021).
lar explanation for the differing perceptions
of the garment holds that people who spend
most of their waking hours in daylight see it
as white and gold; night owls, who are main-
ly exposed to artificial light, see it as blue and
black.) And even if these differences start
out small, they can become entrenched and
reinforced as we proceed to harvest infor-
mation differently, selecting sensory data
that are best aligned with our individual
emerging models of the world and then up-
dating our perceptual models based on
these biased data. We are all familiar with
this process from the echo chambers of so-
cial media and the newspapers we choose
to read. I am suggesting that the same prin-
Richard Armstrong/EyeEm/Getty Images
ciples apply also at a deeper level, under-

neath our sociopolitical beliefs, right down
to the fabric of our perceptual realities.
They may even apply to our perception of
being a self—the experience of being me or
of being you—because the experience of be-
ing a self is itself a perception. PERCEPTUAL SHIFT: V iewing this photograph changes what one consciously sees
This is why understanding the con- in the two-tone image (above).


WHY
GOOD
THOUGHTS
While we are working

through a problem,
the brain’s tendency to
stick with familiar ideas
BLOCK
can prevent us from seeing BETTER
superior solutions
By Merim Bilalić and
ONES
Peter McLeod
Illustration by Danny Schwartz

n a classic 1942 experiment, American psychologist Abraham Luchins

asked volunteers to do some basic math by picturing water jugs in their mind.
I Given three empty containers, for example, each with a different capacity—
21, 127 and three units of water—the participants had to figure out how to trans-
fer liquid between the containers to measure out precisely 100 units. They could
fill and empty each jug as many times as they wanted, but they had to fill the
vessels to their limits. The solution was to first fill the second jug to its capacity of 127 units, then
empty it into the first to remove 21 units, leaving 106, and finally to fill the third jug twice to
subtract six units for a remainder of 100. Luchins presented his volunteers with several more
problems that could be solved with essentially the same three steps; they made quick work of
them. Yet when he gave them a problem with a simpler and faster solution than the previous
tasks, they failed to see it.
This time Luchins asked the participants to measure out chess players, we solved the mystery. It turns out that people
20 units of water using containers that could hold 23, 49 and under the influence of this cognitive shortcut literally do not
three liquid units. The solution is obvious, right? Simply fill the see certain details in their environment that could provide
first jug and empty it into the third one: 23 – 3 = 20. Yet many them with a more effective solution. Research also suggests
people in Luchins’s experiment persistently tried to solve the that many different cognitive biases discovered by psycholo-
easier problem the old way, emptying the second container into gists over the years—those in the courtroom and the hospital,
the first and then into the third twice: 49 – 23 – 3 – 3 = 20. And for instance—are in fact variations of the Einstellung effect.
when Luchins gave them a problem that had a two-step solu-
tion but could not be solved using the three-step method to BACK TO SQUARE ONE
which the volunteers had become accustomed, they gave up, Since at least the early 1990s, p sychologists have studied
saying it was impossible. the Einstellung e ffect by recruiting chess players of varying
The water jug experiment is one of the most famous exam- skill levels, from amateur to grand master. In such experi-
ples of the Einstellung e ffect: the human brain’s dogged ten- ments, researchers have presented players with specific ar
dency to stick with a familiar solution to a problem—the one rangements of chess pieces on virtual chessboards and asked
that first comes to mind—and to ignore alternatives. Often this them to achieve a checkmate in as few moves as possible. Our
type of thinking is a useful heuristic. Once you have hit on a own studies, for instance, provided expert chess players with
successful method to, say, peel garlic, there is no point in trying scenarios in which they could accomplish a checkmate using a
an array of different techniques every time you need a new well-known sequence called smothered mate. In this five-step
clove. The trouble with this cognitive shortcut, however, is that maneuver, the queen is sacrificed to draw one of the opponent’s
it sometimes prevents people from seeing more efficient or pieces onto a square to block off the king’s escape route. The
appropriate solutions than the ones they already know. players also had the option to checkmate the king in just three
Building on Luchins’s early work, psychologists replicated moves with a much less familiar sequence. As in Luchins’s
the Einstellung effect in many different laboratory studies with water jug studies, most of the players failed to find the more
both novices and experts exercising a range of mental abilities, efficient solution.
but exactly how and why it happened was never clear. About 15 During some of these studies, we asked the players what
years ago, by recording the eye movements of highly skilled was going through their mind. They said they had found the

smothered mate solution and insisted they were searching for a cognitive bias in his 1620 book N ovum Organum: “ The human
shorter one, to no avail. But the verbal reports offered no in understanding when it has once adopted an opinion . . . draws
sight into why they could not find the swifter solution. In 2007 all things else to support and agree with it. And though there
we decided to try something a little more objective: tracking be a greater number and weight of instances to be found on the
eye movements with an infrared camera. Which part of the other side, yet these it either neglects or despises, or else by
board people looked at and how long they looked at different some distinction sets aside and rejects. . . . Men . . . mark the
areas would unequivocally tell us which aspects of the problem events where they are fulfilled, but where they fail, though this
they were noticing and ignoring. happen much oftener, neglect and pass
In this experiment, we followed the gaze them by. But with far more subtlety does
of five expert chess players as they examined
a board that could be solved either with the
ANY DATA this mischief insinuate itself into philoso-
phy and the sciences, in which the first con-
longer smothered mate maneuver or with THAT DO NOT clusion colours and brings into conformity
the shorter three-move sequence. After an FIT THE with itself all that comes after.”
average of 37 seconds, all the players insisted In the 1960s English psychologist Peter
that the smothered mate was the speediest SOLUTION OR Wason gave this particular bias a name:
possible way to corner the king. When we THEORY WE “confirmation bias.” In controlled experi-
presented them with a board that could be ments, he demonstrated that even when
solved only with the three-sequence move,
ARE ALREADY people attempt to test theories in an objec-
however, they found it with no problem. And CLINGING TO tive way, they tend to seek evidence that
when we told the players that this same swift
checkmate had been possible in the previous
ARE IGNORED confirms their ideas and to ignore anything
that contradicts them.
chessboard, they were shocked. “No, it is OR DISCARDED. In T
he Mismeasure of Man, f or example,
impossible,” one player exclaimed. “It is a Stephen Jay Gould of Harvard University
different problem; it must be. I would have reanalyzed data cited by researchers trying
noticed such a simple solution.” Clearly, the mere possibility of to estimate the relative intelligence of different racial groups,
the smothered mate move was stubbornly masking alternative social classes and sexes by measuring the volumes of their skulls
solutions. In fact, the E
instellung effect was powerful enough or weighing their brains, on the assumption that intelligence
to temporarily lower expert chess masters to the level of much was correlated with brain size. Gould uncovered massive data
weaker players. distortion. On discovering that French brains were on average
The infrared camera revealed that even when the players smaller than their German counterparts, French neurologist
said they were looking for a faster solution—and indeed Paul Broca explained away the discrepancy as a result of the dif-
believed they were doing so—they did not actually shift their ference in average body size between citizens of the two nations.
gaze away from the squares they had already identified as part After all, he could not accept that the French were less intelli-
of the smothered mate move. In contrast, when presented with gent than the Germans. Yet when he found that women’s brains
the one-solution chessboard, players initially looked at the were smaller than those in men’s noggins, he did not apply the
squares and pieces important for the smothered mate and, same correction for body size, because he did not have any prob-
once they realized it would not work, directed their attention lem with the idea that women were less intelligent than men.
toward other squares and soon hit on the shorter solution. Somewhat surprisingly, Gould concluded that Broca and
others like him were not as reprehensible as we might think.
BASIS FOR BIAS “In most cases discussed in this book we can be fairly certain
In 2 01 3 Heather Sheridan, now at the University of Albany, that biases ... were unknowingly influential and that scientists
and Eyal M. Reingold of the University of Toronto published believed they were pursuing unsullied truth,” Gould wrote. In
studies that corroborate and complement our eye-tracking other words, just as we observed in our chess experiments,
experiments. They presented 17 novice and 17 expert chess play- comfortably familiar ideas blinded Broca and his contempo-
ers with two different situations. In one scenario, a familiar raries to the errors in their reasoning. Here is the real danger of
checkmate maneuver such as the smothered mate was advan- the E instellung e ffect. We may believe that we are thinking in
tageous but second best to a distinct and less obvious solution. an open-minded way, completely unaware that our brain is
In the second situation, the more familiar sequence would be a selectively directing attention away from aspects of our envi-
clear blunder. As in our experiments, once amateurs and mas- ronment that could inspire new thoughts. Any data that do not
ter chess players locked onto the helpful familiar maneuver, fit the solution or theory we are already clinging to are ignored
their eyes rarely drifted to squares that would clue them in to or discarded.
the better solution. When the well-known sequence was obvi- The surreptitious nature of confirmation bias has unfortu-
ously a mistake, however, all the experts (and most of the nov- nate consequences in everyday life, as documented in studies
ices) detected the alternative. on decision-making among doctors and juries. In a review of
The E instellung e ffect is by no means limited to controlled errors in medical thought, physician Jerome Groopman noted
experiments in the lab or even to mentally challenging games that in most cases of misdiagnosis, “the doctors didn’t stumble
such as chess. Rather it is the basis for many cognitive biases. because of their ignorance of clinical facts; rather, they missed
English philosopher, scientist and essayist Francis Bacon was diagnoses because they fell into cognitive traps.” When doctors
especially eloquent about one of the most common forms of inherit a patient from another doctor, for example, the first

Much More Than Move 1 Move 2 Move 3

Meets the Eye
The intellectually demanding game o f chess
has proved a wonderful way for psychologists
to study the E instellung effect—the brain’s
tendency to stick with solutions it already
Player A
knows rather than looking for potentially
superior ones. Experiments have shown
that this cognitive bias literally changes
how even expert chess players see the
board in front of them.
Two-Solution Problem
Player B
A B C D E F G H
5
Move 1 Move 2 Move 3
4
2
Player A
One-Solution Problem
Player B
A B C D E F G H
7
Chess Masters Fail to See the Quickest Path to Victory
6 In a well-known five-sequence move called smothered mate (yellow), player A
begins by moving the queen from E2 to E6, backing player B’s king into a corner.
5 Player A then repeatedly threatens to take B’s king with a knight, forcing player B to
dodge. As an act of deliberate sacrifice, player A moves the queen adjacent to B’s
4 king, allowing player B to take the queen with a rook. To end the game, player A
moves the knight to F7, boxing in B’s king with no chance of escape. In recent
3 experiments, psychologists presented master chess players with the two-solution
board shown, which could be won using either the smothered mate or a much swifter
2 three-step solution (green). The players were told to achieve checkmate as quickly as
possible, but once they recognized the smothered mate as a possibility, they became
1 seemingly incapable of noticing the more efficient strategy. When presented with a nearly
identical board on which the position of one bishop had shifted (blue), eliminating the
smothered mate as an option, the players did recognize the speedier solution, however.
14 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022 Graphic by George Retseck

clinician’s diagnosis can block the second from seeing impor-

tant and contradictory details of the patient’s health that
might change the diagnosis. It is easier to just accept the diag-
Move 4 Move 5 nosis—the “solution”—that is already in front of them than to
rethink the entire situation. Similarly, radiologists examining
chest x-rays often fixate on the first abnormality they find and
fail to notice further signs of illness that should be obvious,
such as a swelling that could indicate cancer. If those second-
ary details are presented alone, however, radiologists see
them right away.
Related studies have revealed that jurors begin to decide
whether someone is innocent or guilty long before all the evi-
dence has been presented. In addition, their initial impres-
sions of the defendant change how they weigh subsequent
evidence and even their memory of evidence they saw before.
Likewise, if an interviewer finds a candidate to be physically
attractive, he or she will automatically perceive that person’s
intelligence and personality in a more positive light, and vice
versa. These biases, too, are driven by the E
instellung effect. It
is easier to make a decision about someone if one maintains a
consistent view of that person rather than sorting through
contradictory evidence.
Can we learn to resist the Einstellung effect? Perhaps. In
The Explanation: Tunnel Vision our chess experiments and the follow-up experiments by
Eye-tracking devices revealed that as soon as chess players hit on the smothered Sheridan and Reingold, some exceptionally skilled experts,
mate as a solution, they spent far more time looking at squares relevant to that such as grand masters, did in fact spot the less obvious opti-
familiar maneuver (orange) than at squares pertinent to the more efficient mal solution even when a slower but more familiar sequence
three-step sequence (magenta), despite insisting that they were searching for of moves was possible. This suggests that the more expertise
alternatives. Conversely, when the smothered mate was not viable, the players’
someone has in their field—whether chess, science or medi-
gaze shifted to regions of the chessboard crucial to the swifter strategy.
cine—the more immune they are to cognitive bias.
But no one is completely impervious; even the grand mas-
Two-Solution Problem ters failed when we made the situation tricky enough. Actively
Problem-Solving Period
remembering that you are susceptible to the E instellung
Initial 10 Middle Final 5 effect is another way to counteract it. When considering the
seconds seconds
evidence on, say, the relative contribution of human-made
40
and naturally occurring greenhouse gases to global tempera-
Looking at Key Squares
ture, remember that if you already think you know the

Percent of Time Spent
30
answer, you will not judge the evidence objectively. Instead
20 you will notice evidence that supports the opinion you already
hold, evaluate it as stronger than it really is and find it more
10 memorable than evidence that does not support your view.
We must try to learn to accept our errors if we sincerely
0 want to improve our ideas. English naturalist Charles Darwin
came up with a remarkably simple and effective technique to
do just this. “I had . . . during many years, followed a golden
One-Solution Problem rule, namely, that whenever a published fact, a new observa-
Problem-Solving Period
tion or thought came across me, which was opposed by my
Initial 10 Middle Final 5 general results, to make a memorandum of it without fail and
seconds seconds
at once,” he wrote. “For I had found by experience that such
40
facts and thoughts were far more apt to escape from memory
Looking at Key Squares
Percent of Time Spent
30
than favourable ones.”
20 Merim Bilalić is a professor of cognitive psychology at the University of Northumbria

at Newcastle. His research on the Einstellung effect won the British Psychological
10 Society’s Award for Outstanding Doctoral Research Contributions to Psychology in
2008. His latest book is The Neuroscience of Expertise (Cambridge University Press, 2017).
0
Peter McLeod is an emeritus fellow at Queen’s College at the University of Oxford. He is a
trustee of the Oxford Foundation for Theoretical Neuroscience and Artificial Intelligence.

How to
Think about
“Implicit Bias” Amid a controversy, it’s important to remember
that implicit bias is real—and it matters
By Keith Payne, Laura Niemi and John M. Doris
W
hen ’ s the last time a stereotype few weeks apart, you might score very dif
popped into your mind? If you are ferently. And the correlation between a
person’s IAT scores and discriminatory
like most people, the authors in behavior is often small.
cluded, it happens all the time. The IAT is a measure, and it doesn’t
That doesn’t make you a racist, sex follow from a particular measure b eing
flawed that the phenomenon w e are at
ist or whatever-ist. It just means tempting to measure is not real. Drawing
your brain is noticing patterns and that conclusion is to commit the Divining
making generalizations. But the same thought processes that Rod Fallacy: just because a rod doesn’t
find water doesn’t mean there’s no such
make people smart can also make them biased. This tendency thing as water. A smarter move is to ask,
for stereotype-confirming thoughts to pass spontaneously “What does the other evidence show?”
through our minds is what psychologists call implicit bias. It sets In fact, there is lots of other evidence.
There are perceptual illusions, for example,
people up to overgeneralize, sometimes leading to discrimina in which white subjects perceive Black faces
tion even when people feel they are being fair. as angrier than white faces with the same
expression. Race can bias people to see
Scientific research on implicit bias has wants to understand implicit bias should harmless objects as weapons when they are
drawn ire from both the right and the left. know about. in the hands of Black men and to dislike
For the right, talk of implicit bias is just First, much of the controversy centers abstract images that are paired with Black
another instance of progressives seeing on the most famous implicit bias test, the faces. And there are dozens of variants of
injustice under every bush. For the left, Implicit Association Test (IAT). A major laboratory tasks showing that most partic
implicit bias diverts attention from more ity of people taking this test show evidence ipants are faster to identify bad words
damaging instances of explicit bigotry. of implicit bias, suggesting that most indi paired with Black faces than with white
Debates have become heated and have viduals are implicitly biased even if they faces. None of these measures is without
leaped from scientific journals to the pop do not think of themselves as prejudiced. limitations, but they show the same pattern
ular press. Along the way, some important As with any measure, the test does have of reliable bias as the IAT. There is a moun
points have been lost. We highlight two limitations. The stability of the test is low, tain of evidence—independent of any sin
misunderstandings that anyone who meaning that if you take the same test a gle test—that implicit bias is real.

The second misunderstanding is reading business. What the IAT does, and bears a much closer resemblance to the
about what scientists mean when they say does well, is predict average outcomes widespread stereotypical thoughts seen on
a measure predicts behavior. One fre across larger entities such as counties, cit implicit bias tests than to the survey stud
quent complaint is that an individual’s ies or states. For example, metro areas ies in which most people present them
IAT score doesn’t tell you whether the with greater average implicit bias have selves as unbiased.
person will discriminate on a particular larger racial disparities in police shoot One reason people on both the right
occasion. This is to commit the Palm ings. And counties with greater average and the left are skeptical of implicit bias
Reading Fallacy: u nlike palm readers, re implicit bias have larger racial disparities might be pretty simple: it isn’t nice to think
search psychologists aren’t usually in the in infant health problems. These correla we aren’t very nice. It would be comfort
business of telling you, as an individual, tions are important: the lives of Black cit ing to conclude, when we don’t consciously
what your life holds in store. Most mea izens and newborn Black babies depend entertain impure intentions, that all of
sures in psychology, from aptitude tests to on them. our intentions are pure. Unfortunately, we
personality scales, are useful for predict Field experiments demonstrate that can’t conclude that: many of us are more
ing how g roups w ill respond on a verage, real-world discrimination continues and biased than we realize. And that is an
notforecasting how particular i ndividu- is widespread. White applicants get about important cause of injustice—whether you
als w ill behave. 50 percent more callbacks than Black know it or not.
The difference is crucial. Knowing that applicants with the same resumes; college
an employee scored high on conscientious professors are 26 percent more likely to Keith Payne i s a professor in psychology and neuro
ness won’t tell you much about whether respond to a student’s e-mail when it is science at the University of North Carolina at Chapel Hill.
her work will be careful or sloppy if you signed by Brad rather than Lamar; and He is author of T he Broken Ladder: How Inequality Affects
inspect it right now. But if a large company physicians recommend less pain medica the Way We Think, Live, and Die ( Viking, 2017).
hires hundreds of employees who are all tion for Black patients than for white pa
Laura Niemi is an assistant professor in the department
conscientious, it will likely pay off with a tients with the same injury.
of psychology at Cornell University. She researches
small but consistent increase in careful Today managers are unlikely to an
moral judgment and the implications of differences in
Lyubov Ivanova/Getty Images
work on average. nounce that white job applicants should

moral values.
Implicit bias researchers have always be chosen over Black applicants, and phy
warned against using the tests for predict sicians don’t declare that Black people feel John M. Doris is Peter L. Dyson Professor of Ethics in
ing individual outcomes, such as how a less pain than white people. Yet the wide Organizations and Life at the Charles H. Dyson School
particular manager will behave in job in spread pattern of discrimination and dis of Applied Economics and Management and a professor
terviews—they’ve never been in the palm- parities seen in field studies persists. It at the Sage School of Philosophy at Cornell University.

SCHOOLED IN
Kids are prime targets of disinformation, yet educators cannot
figure out how best to teach them to separate fact from fiction
By Melinda Wenner Moyer
Illustrations by Taylor Callery

LIES

W
h e n A m a n da G a r d n e r , a n e d u c at o r w i t h t w o d e c a d e s o f
experience, helped to start a new charter elementary and middle
school outside of Seattle last year, she did not anticipate teaching stu-
dents who denied that the Holocaust happened, argued that C OVID
is a hoax and told their teacher that the 2020 presidential election
was rigged. Yet some children insisted that these conspiracy fantasies
were true. Both misinformation, which includes honest mistakes, and
disinformation, which involves an intention to mislead, have had “a growing impact on stu-
dents over the past 10 to 20 years,” Gardner says, yet many schools do not focus on the issue.
“Most high schools probably do some teaching to prevent plagiarism, but I think that’s about it.”
Children, it turns out, are ripe targets for fake viewer started with. For instance, when researchers
news. Age 14 is when kids often start believing in searched for videos using the phrase “lunar eclipse,”
unproven conspiratorial ideas, according to a study they were steered to a video suggesting that Earth is
published in September 2021 in the British Journal flat. YouTube is one of the most popular social media
of Developmental Psychology. Many teens also have sites among teens: After Zeynep Tufekci, an associ-
trouble assessing the credibility of online informa- ate professor at the University of North Carolina,
tion. In a 2016 study involving nearly 8,000 U.S. stu- Chapel Hill, School of Information and Library Sci-
dents, Stanford University researchers found that ence, spent time searching for videos on YouTube and
more than 80 percent of middle schoolers believed observed what the algorithm told her to watch next,
that an advertisement labeled as sponsored content she suggested that it was “one of the most powerful
was actually a news story. The researchers also found radicalizing instruments of the 21st century.”
that less than 20 percent of high schoolers seriously One tool that schools can use to deal with this
questioned spurious claims in social media, such as problem is called media literacy education. The idea
a Facebook post that said images of strange-looking is to teach kids how to evaluate and think critically
flowers, supposedly near the site of a nuclear power about the messages they receive and to recognize
plant accident in Japan, proved that dangerous radi- falsehoods masquerading as truth. For children
ation levels persisted in the area. When college stu- whose parents might believe conspiracy fantasies or
dents in the survey looked at a Twitter post touting other lies fueled by disinformation, school is the one
a poll favoring gun control, more than two thirds place where they can be taught skills to evaluate such
failed to note that the liberal antigun groups behind claims objectively.
the poll could have influenced the data. Yet few American kids are receiving this instruction.
Disinformation campaigns often directly go after Last summer Illinois became the first U.S. state to
young users, steering them toward misleading con- require all high school students to take a media litera-
tent. A 2018 W
all Street Journal investigation found cy class. Thirteen other states have laws that touch on
that YouTube’s recommendation algorithm, which media literacy, but requirements can be as general as
offers personalized suggestions about what users putting a list of resources on an education department
should watch next, is skewed to recommend videos website. A growing number of students are being
that are more extreme and far-fetched than what the taught some form of media literacy in college, but that

is “way, way too late to begin this kind of instruction,” Other approaches teach students methods for
says Howard Schneider, executive director of the Cen- evaluating the credibility of news and information
ter for News Literacy at Stony Brook University. When sources, in part by determining the goals and incen-
he began teaching college students years ago, he found tives of those sources. They teach students to ask:
that “they came with tremendous deficits, and they Who created the content and why? And what do oth-
were already falling into very bad habits.” er sources say? But these methods are relatively new
Even if more students took such classes, there is and have not been widely studied.
profound disagreement about what those courses The lack of rigorous studies of the different ap
should teach. Certain curricula try to train students proaches is indeed a major roadblock, says Paul Miha-
to give more weight to journalistic sources, but some ilidis, a civic media and journalism expert at Emer-
researchers argue that this practice ignores the po son College. He is the principal investigator of the
tential biases of publications and reporters. Other Mapping Impactful Media Literacy Practices initia-
courses push students to identify where information tive, a research project supported by the National
comes from and ask how the content helps those dis- Association for Media Literacy Education. “Most of
seminating it. Overall there are very few data show- the science done is very small scale, very exploratory.
ing the best way to teach children how to tell fact It’s very qualitative,” he says. That is not simply be
from fiction. cause of a lack of resources, he adds. “There’s also a
Most media literacy approaches “begin to look lack of clarity about what the goals are.”
thin when you ask, ‘Can you show me the evidence?’ ”
says Sam Wineburg, a professor of education at Stan-
ford, who runs the Stanford History Education Group.
There are factions of educational researchers behind
Children are ripe targets for
each method, says Renee Hobbs, director of the Media fake news. Age 14 is when kids
Education Lab at the University of Rhode Island, and
“each group goes out of its way to diss the other.” These often start believing in unproven
approaches have not been compared head-to-head,
and some have only small studies supporting them.
conspiratorial ideas, according
Like online media sources themselves, it is hard to to a 2021 study.
know which ones to trust.
News literac y is a subset of media literac y For instance, in a 2017 study researchers looked
research that deals directly with the propagation of at how well students who had taken Stony Brook’s
conspiracies and the ability to distinguish real news undergraduate course could answer certain questions
from fake stories. It entails a set of skills that help a year later compared with students who had not.
people judge the reliability and credibility of news Students who had taken the class were more likely to
and information. But as with media literacy, research- correctly answer questions about the news media,
ers have very different ideas about how this type of such as that PBS does not rely primarily on advertis-
news analysis should be taught. ing for financial support. But the study did not test
Some programs, such as Schneider’s Stony Brook how well the students could discern fake from real
program and the nonprofit, Washington, D.C.–based news, so it is hard to know how well the program
News Literacy Project, teach students to discern the inoculates students against falsehoods.
quality of the information in part by learning how Moreover, the small amount of research that does
responsible journalism works. They study how jour- exist has largely been conducted with college stu-
nalists pursue news, how to distinguish between dif- dents, not the middle school or high school students
ferent kinds of information and how to judge evidence who are so vulnerable to disinformation. Indeed, the
behind reported stories. The goal, Schneider wrote in various approaches that are being used in K–12 class-
a 2007 article for N
ieman Reports, i s to shape students rooms have hardly been tested at all. As part of his
into “consumers who could differentiate between raw, current research initiative, Mihailidis and his team
unmediated information coursing through the Inter- interviewed the heads of all major organizations that
net and independent, verified journalism.” are part of the National Media Literacy Alliance,
Yet some media literacy scholars doubt the effica- which works to promote media literacy education.
cy of these approaches. Hobbs, for instance, wrote a “We are finding, repeatedly, that many of the ways in
2010 paper arguing that these methods glorify jour- which they support schools and teachers—resources,
nalism, ignore its many problems and do little to guidelines, best practices, etcetera—are not studied
instill critical thinking skills. “All that focus on the in much of a rigorous fashion,” he says.
ideals of journalism is mere propaganda if it is blind Some researchers, including Wineburg, are trying
to the realities of contemporary journalism, where to fill in the research gaps. In a study published in 2019,
partisan politics and smear fests are the surest way Wineburg and his team compared how 10 history pro-
to build audiences,” she stated. fessors, 10 journalism fact-checkers and 25 Stanford

S
undergraduates evaluated websites and information t i l l , e v e n i f n e ws l i t e r a c y e d u c at i o n
on social and political issues. They found that where- teaches specific skills well, some researchers
as historians and students were often fooled by manip- question its broader, longer-term impact. Once
ulative websites, journalism fact-checkers were not. students learn how to evaluate websites and claims,
In addition, their methods of analysis differed signif- how confident can we be that they will retain these
icantly: historians and students tried to assess the skills and use them down the line? How sure can we
validity of websites and information by reading verti- be that these methods will inculcate students with
cally, navigating within a site to learn more about it, skepticism about conspiracy theories and disinforma-
but fact-checkers read laterally, opening new brows- tion campaigns? And will these methods lead students
er tabs for different sources and running searches to to become civically engaged members of society?
judge the original website’s credibility. “There’s always this kind of leap into ‘that will make
Working with the Poynter Institute and the Local our democracy and news systems stronger.’ And I don’t
Media Association and with support from Google.org know if that’s necessarily the case,” Mihailidis says.
(a charity founded by the technology giant), Wine- Some research does hint that news literacy ap
burg and his team have created a civic online reason- proaches could have these broader beneficial effects.
ing course that teaches students to evaluate informa- In a 2017 study of 397 adults, researchers found that
tion by reading laterally. The effects so far look prom- people who were more media-literate were less like-
ising. In a field experiment involving 40,000 high ly to endorse conspiracy theories compared with peo-
school students in urban public health districts, Wine- ple who were less media-literate. “We can’t definite-
burg and his group found that students who took the ly say news literacy causes you to reject conspiracy
class became better able to evaluate websites and the theories, but the fact that we see a positive relation-
credibility of online claims, such as Facebook posts, ship there tells us there’s something to this that
compared with students who did not take the class. we need to continue to explore,” says co-author Seth

Ashley, an associate professor of journalism and you value ongoing debates about how the world
media studies at Boise State University. works.” Instead of driving students to apathy, the goal
While Ashley’s results are encouraging, some ex is to steer them toward awareness and engagement.
perts worry that a focus only on evaluating websites
and news articles is too narrow. “News literacy in a lot Schools still have a l ong way to go before
of ways focuses on credibility and whether we know they get there, though. One big challenge is how to
something is true or not, and that’s a really important expand these programs so they reach everyone, espe-
question, but that is one question,” says Michelle Ciul- cially kids in lower-income school districts, who are
la Lipkin, executive director of the National Associa- much less likely to receive any news literacy instruc-
tion for Media Literacy Education. “Once we figure out tion at all. And teachers already have so much mate-
if it’s false or true, what is the other assessment and rial they have to impart—can they squeeze in more,
the other analyzing we need to do?” Determining cred- especially if what they have to add is nuanced and
ibility of the information is just the first step, she complex? “[We] desperately need professional devel-
argues. Students should also be thinking about why opment and training and support for educators
the news is being told in a particular way, whose sto- because they’re not experts in the field,” Adams says.
ries are being told and whose are not, and how the “And it’s the most complex and fraught and largest
information is getting to the news consumer. information landscape in human history.”
Pressing students to be skeptical about all infor-
mation also may have unexpected downsides. “We
think that some approaches to media literacy not only
don’t work but might actually backfire by increasing
“Some approaches to media literacy
students’ cynicism or exacerbating misunderstand- not only don’t work but might
ings about the way news media work,” says Peter
Adams, senior vice president of education at the News actually backfire by increasing
Literacy Project. Students may begin to “read all students’ cynicism.”

kinds of nefarious motives into everything.” Adams’s
concern was amplified by danah boyd, a technology —Peter Adams News Literacy Project
scholar at Microsoft Research and founder and pres-
ident of the Data & Society research institute, in a
2018 talk at the South by Southwest media confer- In 2019 Senator Amy Klobuchar of Minnesota
ence. Boyd argued that although it is good to ask stu- introduced the Digital Citizenship and Media Liter-
dents to challenge their assumptions, “the hole that acy Act into the U.S. Senate, which, if passed, would
opens up, that invites people to look for new expla- authorize $20 million to create a grant program at
nations, that hole can be filled in deeply problemat- the Department of Education to help states develop
ic ways.” Jordan Russell, a high school social studies and fund media literacy education initiatives in K–12
teacher in Bryan, Tex., agrees. “It’s very easy for stu- schools. More investment in this kind of education is
dents to go from healthy critical thinking to unhealthy critical if America’s young people are going to learn
skepticism” and the idea that everyone is lying all the how to navigate this new and constantly evolving
time, he says. media landscape with their wits about them. And
To avoid these potential problems, Ashley advo- more research is necessary to understand how to get
cates for broad approaches that help students devel- them there. At the Center for News Literacy, Schnei-
op mindsets in which they become comfortable with der plans to conduct a trial soon to determine how
uncertainty. According to educational psychologist his course shapes the development of news literacy,
William Perry of Harvard University, students go civic engagement and critical thinking skills among
through various stages of learning. First children are students in middle school and high school.
black-and-white thinkers—they think there are right But many more studies will be needed for re
answers and wrong answers. Then they develop into searchers to reach a comprehensive understanding
relativists, realizing that knowledge can be contextu- of what works and what doesn’t over the long term.
al. This stage can be dangerous, however. It is the one Education scholars need to take “an ambitious, big
where, as Russell notes, people can come to believe step forward,” Schneider says. “What we’re facing are
there is no truth. Ashley adds that when students transformational changes in the way we receive, pro-
think everything is a lie, they also think there is no cess and share information. We’re in the middle of
point in engaging with difficult topics. the most profound revolution in 500 years.”
With news literacy education, the goal is to get stu-
dents to the next level, “to that place where you can Melinda Wenner Moyer, a contributing editor at Scientific American, is
start to see and appreciate the fact that the world is author of How to Raise Kids Who Aren’t Assholes: Science-Based Strategies
messy, and that’s okay,” Ashley says. “You have these for Better Parenting—from Tots to Teens ( G. P. Putnam’s Sons, 2021). She
fundamental approaches to gathering knowledge that wrote about the reasons that autoimmune diseases overwhelmingly affect
you can accept, but you still value uncertainty, and women in the September 2021 issue.


Climate
Miseducation
How oil and gas representatives manipulate
the standards for courses and textbooks,
from kindergarten to 12th grade
By Katie Worth
Illustrations by Taylor Callery

I n a d ra b h e a r i n g r o o m i n Au s t i n, T e x . , m e m b e r s o f t h e S tat e Boa r d o f
Education, seated at small desks arranged in a broad, socially distanced circle, debated
whether eighth grade science students should be required to “describe efforts to mitigate
climate change.” One board member, a longtime public school science teacher, argued in
favor of the proposed new requirement. Another, an in-house attorney for Shell Oil Com-
pany, argued to kill it.
The attorney won. In the end, the board voted to
require that eighth grade science students “describe
the carbon cycle” instead.
Over the past two years school board meetings
around the country have erupted into shout fests over
face masks, reading lists and whether to ban educa-
tion about structural racism in classrooms. In Texas, a
quieter political agenda played out during the lightly
attended process to set science education standards—
guidelines for what students should learn in each sub-
ject and grade level. For the first time, the state board
the largest textbook purchasers in the nation—and
because its elected 15-member State Board of Educa-
tion has a history of applying a conservative political
lens to those textbooks—publishers pay close atten-
tion to Texas standards as they create materials they
then sell to schools across America. As a former sci-
ence textbook editor once told me, “I never heard any-
one explicitly say, ‘We can’t talk about environmental-
ism because of Texas.’ But we all kind of knew. Every-
body kind of knows.” In this way, the proceedings in
an Austin boardroom influence what millions of chil-
considered requiring that students learn something dren nationwide are taught.
about human-caused climate change. That require- Most Americans favor teaching kids about the
ment came under tense dispute between industry rep- climate crisis. A 2019 nationwide poll by NPR/Ipsos
resentatives interested in encouraging positive good- found that nearly four in five respondents—including
will about fossil fuels and education advocates who two of three Republicans—thought schoolchildren
think students should learn the science underlying should be taught about climate change. When the Tex-
the climate crisis unfolding around them. as Education Agency surveyed science educators
Standards adoptions are an exercise in bureaucra- across the state about what should be added to the
cy, but the results wield great power over what is standards, one in four wrote in asking for climate
taught in classrooms. Publishers consult them as they change or something adjacent, such as alternative
write textbooks. State education officials use them as energy. No one asked for more content on fossil fuels.
the basis of standardized tests. School districts call on And yet, as I learned when I watched 40 hours of
them as they shape curricula. Teachers refer to them live and archived board hearings, reviewed scores of
as they devise lesson plans. Every state adopts its own public records and interviewed 15 people involved in
standards, but Texas adoptions have long had influ- the standard-setting process, members of the fossil-
ence far beyond the state’s borders. fuel industry participated in each stage of the Texas sci-
In 2020 two major education advocacy groups— ence standards adoption process, working to influence
the National Center for Science Education and the what children learn in the industry’s favor. Texas edu-
Texas Freedom Network—hired experts to grade the cation officials convened teams of volunteers to rewrite
science standards of all 50 states and Washington, the existing standards, and industry members volun-
D.C., based on how they covered the climate crisis. teered for those writing teams and shaped the lan-
Thirty states and D.C. made As or Bs. Texas was one of guage around energy and climate. Industry members
six states that made an F. But because Texas is one of rallied to testify each time proposals to revise stan-

dards got a public hearing. When the board considered 2020, held in person and virtually on Zoom because of
the rewritten standards for final approval, the industry the COVID pandemic. More than 30 teachers, parents
appealed to members to advance their favored amend- and other education advocates showed up to testify that
ments, ensuring that the seemingly local drama in Aus- the climate crisis has biological, chemical and physical
tin will have outsized consequences. aspects that make it relevant to all the core classes.
For at least a decade the fossil-fuel industry has Three and a half hours into that meeting, however,
tried to green its public image. The Texas proceedings someone with a different message appeared on the
show that its actions do not always reflect that image. Zoom screen: Robert Unger, a silver-haired engineer
In little-watched venues, the industry continues to from Dallas who had worked for the oil and gas indus-
downplay the crisis it has wrought, impeding efforts to try for more than 45 years. He was representing the Tex-
provide clear science about that crisis to a young gen- as Energy Council, and he had some suggestions.
eration whose world will be defined by it.
T h e l a s t t i m e the board overhauled the

Texas Essential Knowledge and Skills After hours of testimony, Robert
(TEKS) for Science, in 2009, it was chaired
by Don McLeroy, a dentist from east-central Unger appeared to represent the
Texas Energy Council, and he had
Texas. McLeroy made his views on science
education clear when he declared at one
some suggestions.
meeting, “Somebody’s got to stand up to
experts!” The board spent much of that
adoption cycle clashing over evolution, but
it also required that high school environ-
mental science students debate something scientists The Texas Energy Council is a coalition of about 35
hadn’t debated for a long time: whether global warm- industry organizations, predominantly from the oil
ing is happening. McLeroy told a reporter he was and gas sector, collectively made up of more than 5,000
pleased because “conservatives like me think the evi- members. Some months earlier the council had begun
dence is a bunch of hooey.” recruiting volunteers to participate in the standards
At the end of 2019, when it was time to begin adoption process. “The earth sciences and the oil/gas
another overhaul, McLeroy was gone. The board industry in particular have suffered significant degra-
made it clear to the 85 volunteers recruited by the Tex- dation in the K–12 curriculum over time,” a page on the
as Education Agency to draft the new standards that council’s website said. In hopes of reversing that trend,
it hoped there would not be a fight over evolution the council enlisted 17 people—geoscientists, petro-
again. It soon became clear the group would fight leum engineers, professors, attorneys and other fossil-
about climate science instead. fuel careerists—who, the site said, “shared its vision of
To start the process, board members carved the ensuring that oil/gas is portrayed in a balanced fashion
standards into three tranches that they would consid- as a critical contribution to the Texas, U.S. and world-
er one at a time: first, high school core sciences, then wide energy mix.” Unger had helped organize the vol-
high school elective sciences and finally grades K–8 unteers. (Several members of the organization, includ-
sciences. The board would give each tranche to writing ing Unger, declined to be interviewed for this story. In
teams composed of volunteers. Professional content an e-mail exchange, Michael Cooper, president of the
advisers, most nominated by board members, would council, took issue with some of this article’s findings
provide feedback to the board on proposed changes. but said he would be unable to provide a comprehen-
Over the summer of 2020 one team took on the first sive response without reviewing a complete draft.)
tranche, the high school core subjects: biology, chemis- Unger asked the board to remove a line in the
try, physics, and an integrated chemistry and physics introductory material for each of the high school core
class. The core science standards were important for classes that discussed social justice and ethics, terms
two reasons. The classes had sky-high enrollment; every he said “do not belong in the course material.” Instead,
year nearly half a million students took biology alone. he said, the standards should include the concept of
And what happened with these classes would set the cost-benefit analysis.
tone for the high school electives and for K–8. To the cli- Most board members had expressed little reaction
mate education advocates’ dismay, when the Texas Edu- to the many people testifying in favor of climate edu-
cation Agency posted the writing groups’ results on its cation, but Unger’s testimony got their attention.
website in July 2020, the draft standards didn’t contain Longtime Republican member Barbara Cargill, a for-
a single reference to modern-day climate change. But mer biology teacher from north of Houston serving
there was still a chance to fix that omission. The state her last few months on the board, asked Unger how
board would present the draft standards for public tes- cost-benefit analysis might be incorporated into the
timony, hearings and amendments. science TEKS. He gave an example: The main benefit
The first major hearing took place in September of fossil fuels is the energy they produce, and the costs

are “environmental issues that our industry is already new petitions. They insisted on removing the terms
regulating.” But oil and gas aren’t the only fuels with a “renewable” and “nonrenewable” to describe different
cost, Unger said. Take solar: “It seems like the benefits energy sources; they preferred to describe all the
are wonderful, but the costs, in fact, are the mining of options as “natural resources.” And they frequently
rare minerals to create batteries,” he said. “Wind brought up energy poverty—the lack of access to afford-
equally has cost and benefit to it.” A science teacher able electricity. “Energy poverty is one of the gravest but
could weigh these things with students, he noted, least talked-about dangers facing humanity,” testified
“and not get into the ambiguities of social injustice Jason Isaac, director of an energy initiative for a con-
and social ethics.” Cargill promised to consider Ung- servative think tank, at one meeting. He suggested just
er’s proposal. one solution: “Right here in Texas the key to ending
All sources of energy come with costs. But a fixa- global energy poverty lies under our feet.”
tion on “cost-benefit analysis” is a plank in a raft of The climate education advocates on the board
arguments supporting what climate scientist Michael expected to lose some of these battles. But they hoped
Mann has called “inactivism”—a tactic that doesn’t the Texas Energy Council volunteers would stand
deny human-caused climate change but downplays it, down when it came to including clear information
deflects blame for it and seeks to delay action on it. about the science of the climate crisis. During the next
Sure, this brand of thinking goes, fossil fuels have set of deliberations, it became evident that would not
their ills. But what form of energy doesn’t? Mann and be the case.
others have criticized such arguments for their false In January 2021 the board held the first hearings
equivalencies: the environmental and health costs of for high school electives: environmental science,
rare earth minerals for certain renewable energy aquatic science, earth science and astronomy. Far few-
sources are small compared with those of fossil fuels. er students take the electives than take biology, chem-
The next day, when the board met to consider istry or physics, but the earth science and environ-
amendments to the standards, Cargill delivered. She mental science course standards were the only ones
proposed removing social justice from the standards that already mentioned climate change.
and adding cost-benefit analysis. Fellow Republican In the months leading up to the hearings, the 23
Pat Hardy, a retired history teacher and curriculum people on the electives writing teams had met about
developer representing suburbs near Dallas–Fort every two weeks to draft the new standards. The old
Worth, eagerly supported the addition. “People talk standards for the earth science course had asked stu-
about electric cars like they’re saving the universe,” dents to “analyze the empirical relationship between
Hardy said, captured on a video of the meeting. “And the emissions of carbon dioxide, atmospheric carbon
the answer is no, they are not.” The board voted to dioxide levels, and the average global temperature
accept the changes. It was the Texas Energy Council’s trends over the past 150 years,” a reference to the peri-
first major victory. od since industrialization, during which atmospheric
The climate education advocates did get a win on carbon dioxide levels have soared. That language
the final day of the hearings. Marisa Pérez-Díaz, a didn’t sit well with William J. Moulton, a longtime
Democratic board member from San Antonio and the geophysicist for the petroleum industry. Encouraged
youngest Latina to ever be elected to any state’s edu- by the Texas Energy Council, he and several other
cation board, had heard their pleas. She proposed industry representatives had applied to the Texas
adding the words “and global climate change” to the Education Agency for a seat on a writing group and
end of a standard that asked students to examine a had been placed. Moulton was on the team rewriting
variety of human impacts on the environment. the earth science and astronomy courses.
Remarkably, the board approved the motion. It wasn’t Moulton agreed that climate change should be
a big win; the wording applied to just one standard, mentioned in some way because students would hear
for the integrated physics and chemistry course, about it anyway. But he felt students should not be led
which is taken by a fifth of the students who take biol- to believe the science is settled. He argued that the
ogy. But for the advocates it was a hopeful sign—cer- phrase “the past 150 years” should be removed. The
tainly a step up from “a bunch of hooey.” group agreed to that change and to several of Moul
ton’s other language tweaks. When those already
I
n the foll owi ng mo nths, as t h e b oar d diluted standards came before the board in January,
considered the next two tranches—the high school four other Texas Energy Council volunteers appeared
electives and the K–8 standards—Texas Energy on Zoom, all recommending amendments. One per-
Council volunteers showed up at meeting after meet- son said the standards should focus on the dangers of
ing. Sometimes they pursued changes that the climate rare earth minerals. Another said it was important for
education advocates found reasonable, such as requir- children to learn that the inception of the fossil-fuel
ing that students learn the laws of geology and en industry stopped the practice of whaling for blubber
couraging the use of resources such as museums and that could be turned into fuel. “Oil and gas literally
mentors. But they kept a relentless focus on adding saved the whales,” she said.
cost-benefit analysis to the standards, and they added The industry also had a new champion on the

board: Will Hickman, who had just been elected in old standards was gone, and the strongest description
November 2020 for a district outside of Houston. of climate change had been weakened.
Hickman’s experience in education included serving
on parent groups at his kids’ schools, coaching com- The climate education advocates had failed
munity sports and teaching Sunday school. He’d held to install a robust presentation of the science surround-
the same day job since 2004: senior legal counsel at ing the climate crisis in any of the high school core or
Shell Oil. elective classes, as they had watched the Texas Energy
In the January hearing, Hickman’s first, his opening Council volunteers achieve one goal after another. But
question was where in the proposed standards he they held out hope for the K–8 standards. Nearly every
could find the advantages and disadvantages of various middle schooler takes the same sciences, and the class-
forms of energy. The next day he offered an example es cover weather and climate systems, an obvious and
that might be raised in class: “Everyone
thinks renewable power’s a great idea, and
Germany adopted it on a large scale,” he
said. “But the cost-benefit—it ended up rais- “Inactivism” doesn’t deny human-
ing their power prices to about 2.5 times our
power prices.” caused climate change but
downplays it, deflects blame for it
The writing committees had already
included a reference to cost-benefit analysis
and seeks to delay action on it.

in the “scientific and engineering practices”
section of each of the elective courses, and
the standard for the environmental science
course had a second mention. But at the
next board hearings, in April, Hickman pressed for effective place to discuss the crisis for a generation of
more. Another member, Rebecca Bell-Metereau, a students that would have to live with its consequences.
professor of English and film at Texas State University, On a 96-degree day at the end of August 2021, the
who had just been elected to represent Austin, pressed board held a public hearing on the K–8 standards, in
back: “The very phrase ‘costs and benefits’ places the person and virtually. The writing groups had labored
primary emphasis on money, not on society or well- over the drafts, adding a single passage mentioning
being or human health.” The board nonetheless climate change. Eighth grade science students, the
approved a motion by Hickman to add another men- draft declared, would be expected to “use scientific
tion of costs and benefits, to aquatic sciences. evidence to describe how human activities can influ-
Moulton began showing up at the board hearings ence climate, such as the release of greenhouse gases.”
with additional proposed changes. His colleagues on One writing group, which included the executive
the writing group had accepted some of his sugges- director of a natural gas foundation, had also append-
tions but not all of them, so he wanted the board to ed a note stating it had not been able to reach consen-
consider adding them as amendments. In the final sus on a proposal to add another line: “Research and
hearing in June, board member Hardy asked Moulton describe the costs and benefits of reducing green-
if he’d heard the “newest stuff that’s been coming out house gas emissions versus global energy poverty.”
on climate,” which, she said, was that the climate crisis At the hearing, two of the professional content
was not unfolding as scientists had predicted. Moulton advisers who had reviewed the standards gave the
suggested that the consensus about warming had been board radically different opinions. Ron Wetherington,
exaggerated by scientists in pursuit of grant money. a retired anthropology professor from Southern Meth-
Hardy began proposing amendments word for odist University nominated by Pérez-Díaz, argued that
word from Moulton’s suggestions. This elicited an the climate standards needed significant strengthen-
outcry from Bell-Metereau. “Do you not think that if ing. Among other things, he advocated that the word
someone’s area of work is in fossil fuels that they “can” be dropped from the phrase “describe how
might have some bias on this issue?” she asked Hardy. human activities can influence climate.” “Can” implies
“It might be that I have a bias for the fossil-fuel indus- that something is a possibility, but an abundance of evi-
try,” Hardy answered. dence shows that the influence is already taking place.
Bell-Metereau and others on the board threatened He also asked the board to add an expectation that stu-
to delay the entire adoption if Hardy insisted on mov- dents explore efforts to mitigate the crisis. Because stu-
ing the changes forward. Ultimately Hardy dropped dents would learn that it’s happening, he posited, they
the proposals. But Moulton and the council had al should learn what people are doing to fix it.
ready succeeded in important ways: The new electives Gloria Chatelain, a longtime educator and CEO of
standards had multiple references to cost-benefit anal- her own consulting firm called Simple Science Solu-
ysis. The terms “renewable energy” and “nonrenew- tions, who had been nominated by Hardy and Cargill,
able energy” were removed in several places. The sin- stood in absolute opposition. She began her testimony
gle mention of the effects of burning fossil fuels in the by praising the “absolutely amazing job” the Texas

Energy Council had already done in improving the rewording the climate standard to “describe how hu
standards. She also said human-caused climate change man activities over the past 150 years, including the
should be treated very lightly in middle school, if at all. release of greenhouse gases, influence climate.” Then
“Our goal is not to produce angry children but chil- she proposed adding a separate line: “Describe efforts
dren who love science. We’re challenging them to go to mitigate climate change, including a reduction in
solve some of these exciting problems but not turn greenhouse gas emissions.” The amendments both
them into Gretas,” she said, referring to the teenage carried. But on the third day the board axed the refer-
climate activist Greta Thunberg of Sweden. Instead, ence to the past 150 years and added the word “can”
she contended, the board should add an expectation back in. The details of recent climate change, Hardy
that students “research and describe the role of energy argued, would simply be too hard for eighth graders
in improving the quality of life in reducing malnutri- to grasp.
tion and global poverty,” language the council had sug- Aicha Davis, a board member from Dallas who
gested. “I think it needs to go in, guys. It’s very, very spent 11 years teaching science before pursuing her
important that we address it,” Chatelain said. Ph.D. in education leadership and policy, spoke up.
For three days that week the board considered the “With all respect to my colleague, you’ve never taught
K–8 language. Over the protests of Democrats, Hardy eighth grade science,” she said, her voice tinged with
moved to add “cost-effectiveness” to each middle forbearance. “We absolutely can’t let the oil and gas
school class. She and Hickman persuaded the conser- industry dictate what our kids need to learn when it
vative board majority to change multiple references to comes to science. It shouldn’t be about the Texas Ener-
renewable and nonrenewable energy to “natural re gy Council. It should be about what’s best for our stu-
sources” in the elementary standards. dents.” Neither scientists nor educators had voiced
On the second day climate education advocates concern about teaching climate change to eighth grad-
landed two unexpected victories. Pérez-Díaz proposed ers, she noted. “So let’s call this what it is. At this point

we’re only making votes based on what oil and gas schoolchildren. I have found examples across the U.S.
wants us to do.” Petroleum companies regularly fund teacher train-
Hickman, the Shell attorney, turned on his micro- ings incentivized by free classroom supplies. Industry
phone. “A few thoughts and reactions,” he said. “One is organizations have spent millions of dollars produc-
I think our permanent school fund is generally funded ing and distributing energy lesson plans. I witnessed
by oil and gas,” referring to a major source of educa- an oil and gas industry employee give a PowerPoint
tion funding maintained in part by proceeds from fos- presentation radically downplaying the climate crisis
sil fuels reaped from public lands. “All of us are proba- to a class of seventh graders.
bly going to get home using oil and gas.... If all of this Even with abundant online educational materials,
is true—greenhouse gases are evil—what do we do? Do just 9 percent of high school science teachers say they
we ban gasoline and stop using gasoline-powered never use a textbook. The nation’s most popular middle
cars? Do we ban diesel for trucks? How do
we get our Amazon and Walmart purchas-
es?” The board chair suggested they table
the issue until the final round of hearings, “Let’s call this what it is,” Aicha
scheduled for November 2021.
Davis said. “At this point we’re
A only making votes based on
s they wa ited fo r the las t
round, the National Center for Sci-
what oil and gas wants us to do.”

ence Education and the Texas Free-
dom Network organized. They recruited 67
Texan climate scientists to join a letter ask-
ing, among other things, that the word
“can” be dropped from the climate passage and that school science textbooks are replete with language that
the mitigation language stay put not only because it conveys doubt about climate change, subtly or other-
consisted of “basic knowledge” that every citizen wise. In one textbook that, as of 2018, was in a quarter
should have but because it would provide students of the nation’s middle schools, students read that
with a sense of hope. “some scientists propose that global warming is due to
Nevertheless, the final round of deliberations in natural climate cycles.” In fact, the number of climate
November was a slaughter. Climate change had been scientists who support that idea is effectively zero.
added in a limited way to the standards, and the con- Texas isn’t the only major buyer of textbooks. Other
servative majority supported that. But it rejected a large states such as California have adopted standards
motion to strike the word “can.” It blocked a motion that embrace the science of climate change, leading to
to remove cost-benefit analysis from the middle a divide. Textbook publishers create one set of prod-
school sciences. It approved new language about “the ucts to sell in Texas and states that lean the same way
critical role of energy resources” to modern life. It and a second set of products for states aligned with
inserted a reference to rare earth elements. It intro- California. This poses an equity problem: the educa-
duced the concept of global energy poverty. tion a child receives on an issue central to the modern
Last, Hickman moved to drop the climate mitiga- world depends on what state they happen to live in.
tion standard that Pérez-Díaz had managed to add in In April 2022 the Texas Education Agency issued a
September, arguing that the subject was more appro- call for textbooks based on the new standards. Publish-
priate for social studies than for science and that it ers were given a year to submit materials to the agen-
“just seems above and beyond for an eighth grade stu- cy. Review panels, made up of educators, will search
dent and teacher.” The board Democrats fought the the textbooks for errors and rate how closely they fol-
change, but they were outnumbered. The board re low the standards. Then the materials go before the
placed the mitigation standard with the line “Describe state board for approval or rejection. Texas school dis-
the carbon cycle.” tricts have the option of establishing their own text-
The Texas Energy Council and two allied organiza- book adoption process but still must choose books that
tions issued a press release praising the State Board of comply with the standards. Most just defer to the
Education for adopting standards that “emphasize the board’s choices. The new science textbooks should be
critical role of energy in modern life.” The Texas Free- on classroom shelves starting in the fall of 2024.
dom Network hit a more ambivalent note in its year- The Texas Energy Council’s Moulton told me he
end report. “The State Board of Education could have— found the standards adoption process energizing, and
and should have—done much better. But our campaign he hopes to stay involved. As soon as he gets the chance,
resulted in new science standards that for the first time he said, he’ll start reviewing the new textbooks and will
make clear to Texas public school students that climate head back to the board to give them his thoughts.
change is real and that human activity is the cause.”
The fossil-fuel industry, like some others, has Katie Worth is a freelance writer in Boston. She is author of Miseducation:
worked for decades to get its messages in front of How Climate Change Is Taught in America ( Columbia Global Reports, 2021).

Why
knowledge can fail us have come into sharp focus. Misinforma-
tion shared on social media websites has fueled an epidemic of
false belief, with widespread misconceptions concerning topics
ranging from the COVID-19 pandemic to voter fraud, whether the
Sandy Hook school shooting was staged and whether vaccines
We
are safe. The same basic mechanisms that spread fear about the
tomato hornworm have now intensified—and, in some cases, led
to—a profound public mistrust of basic societal institutions. One
consequence is the largest measles outbreak in a generation.
“Misinformation” may seem like a misnomer here. After all,
many of today’s most damaging false beliefs are initially driven by
Trust
acts of propaganda and disinformation, which are deliberately
deceptive and intended to cause harm. But part of what makes
disinformation so effective in an age of social media is the fact that
people who are exposed to it share it widely among friends and
peers who trust them, with no intention of misleading anyone.
Lies
Social media transforms disinformation into misinformation.
Many communication theorists and social scientists have
tried to understand how false beliefs persist by modeling the
spread of ideas as a contagion. Employing mathematical models
involves simulating a simplified representation of human social
interactions using a computer algorithm and then studying
these simulations to learn something about the real world. In a
contagion model, ideas are like viruses that go from mind to
The most effective misinformation mind. You start with a network, which consists of nodes, repre-
senting individuals, and edges, which represent social connec-
starts with seeds of truth tions. You seed an idea in one “mind” and see how it spreads
under various assumptions about when transmission will occur.
By Cailin O’Connor and Contagion models are extremely simple but have been used
to explain surprising patterns of behavior, such as the epidemic
James Owen Weatherall of suicide that reportedly swept through Europe after publica-
tion of Goethe’s T he Sorrows of Young Werther i n 1774 or when
Illustration by Lisk Feng
dozens of U.S. textile workers in 1962 reported suffering from
I
nausea and numbness after being bitten by an imaginary insect.
n the mid - 1800 s a caterpillar the size of a human They can also explain how some false beliefs propagate on the
finger began spreading across the northeastern U.S. This Internet. Before the 2016 U.S. presidential election, an image of
appearance of the tomato hornworm was followed by terri- a young Donald Trump appeared on Facebook. It included a
fying reports of fatal poisonings and aggressive behavior quote, attributed to a 1998 interview in P eople m
agazine, saying
toward people. In July 1869 newspapers across the region that if Trump ever ran for president, it would be as a Republi-
posted warnings about the insect, reporting that a girl in can because the party is made up of “the dumbest group of vot-
Red Creek, N.Y., had been “thrown into spasms, which end- ers.” Although it is unclear who “patient zero” was, we know
ed in death” after a run-in with the creature. That fall the S
yra- that this meme passed rapidly from profile to profile.
cuse Standard printed an account from one Dr. Fuller, who had The meme’s veracity was quickly evaluated and debunked.
collected a particularly enormous specimen. The physician The fact-checking website Snopes reported that the quote was
warned that the caterpillar was “as poisonous as a rattlesnake” fabricated as early as October 2015. But as with the tomato horn-
and said he knew of three deaths linked to its venom. worm, these efforts to disseminate truth did not change how the
Although the hornworm is a voracious eater that can strip a rumors spread. One copy of the meme alone was shared more
tomato plant in a matter of days, it is, in fact, harmless to than half a million times. As new individuals shared it over the
humans. Entomologists had known the insect to be innocuous next several years, their false beliefs infected friends who ob
for decades when Fuller published his dramatic account, and served the meme, and they, in turn, passed the false belief on to
his claims were widely mocked by experts. So why did the ru new areas of the network.
mors persist even though the truth was readily available? Peo- This is why many widely shared memes seem to be immune
ple are social learners. We develop most of our beliefs from the to fact-checking and debunking. Each person who shared the
testimony of trusted others such as our teachers, parents and Trump meme simply trusted the friend who had shared it rather
friends. This social transmission of knowledge is at the heart of than checking for themselves. Putting the facts out there does not
culture and science. But as the tomato hornworm story shows help if no one bothers to look them up. It might seem like the
us, our ability has a gaping vulnerability: sometimes the ideas problem here is laziness or gullibility—and thus that the solution
we spread are wrong. is merely more education or better critical thinking skills. But
In recent years the ways in which the social transmission of that is not entirely right. Sometimes false beliefs persist and

spread even in communities where everyone works very hard to papers and legal advice supporting antivaccination eff orts. Partic-
learn the truth by gathering and sharing evidence. In these cases, ipants in these online communities care very much about wheth-
the problem is not unthinking trust. It goes far deeper than that. er vaccines are harmful and actively try to learn the truth. Yet they
come to dangerously wrong conclusions. How does this happen?
TRUST THE EVIDENCE The contagion model is inadequate for answering this ques-
Before it was shut down i n November 2020, the “Stop Man- tion. Instead we need a model that can capture cases where
datory Vaccination” Facebook group had hundreds of thousands people form beliefs on the basis of evidence that they gather
of followers. Its moderators regularly posted material that was and share. It must also capture why these individuals are moti-
framed to serve as evidence for the community that vaccines are vated to seek the truth in the first place. When it comes to
harmful or ineffective, including news stories, scientific papers health topics, there might be serious costs to acting on false
and interviews with prominent vaccine skeptics. On other Face beliefs. If vaccines are safe and effective (which they are) and
book group pages, thousands of concerned parents ask and parents do not vaccinate, they put their kids and immunosup
answer questions about vaccine safety, often sharing scientific pressed people at unnecessary risk. If vaccines are not safe, as

How Network Science Maps NETWORK EPISTEMOLOGY

FRAMEWORK
the Spread of Misinformation Network epistemology models represent
situations in which people form beliefs by
We use network science t o better understand how social connections influence the gathering and sharing evidence. This kind
of model applies to many cases in science.
beliefs and behaviors of individuals in a social network—and especially how false beliefs
Beliefs do not simply spread from individual
can spread from person to person. Here we look at two kinds of network models that to individual. Instead each individual has some
capture different ways in which ideas or beliefs spread. Each node in these models repre- degree of certainty about an idea. This prompts
sents an individual. Each edge, or connection between the nodes, represents a social tie. the person to gather evidence in support of it,
and that evidence changes beliefs. Each indi
vidual shares the evidence with network
neighbors, which also influences beliefs.
THE CONTAGION MODEL How to Read the Contagion Plots
Contagion models treat ideas or beliefs like viruses Each circular node is a person who is
that spread between individuals in a social network. influenced by ideas presented by others.
There are different ways that this “infection” can How to Read the Network
Each line represents a connection
work. In some models, everyone will be infected between individuals. Epistemology Framework Plots
by an infected neighbor. In others, ideas spread
Each circular or square node is a person who
whenever some percentage of an individual’s The gauge at the top of the nodes indicates
is influenced by evidence presented by others.
neighbors become infected. Here we illustrate the percent of that person’s connections
25% 10 Each has a belief about whether action A (blue)
these “complex contagions” with examples where who hold a particular belief. In the % or action B (orange) is better. The person’s
individuals take on a new belief if at least 25 percent
0%
scenarios below, the threshold for an
0
belief can strengthen, weaken and/or flip over
of their neighbors hold it. In these models, the individual to take on the belief of their
time, as shown here by changing colors.
structure of the network affects how ideas spread. neighbors is 25% (at least 1 out of 4).
The strength of the color represents the
individual’s level of certainty in a particular
BONDING AND BRIDGING: In less connected groups, ideas cannot reach all members. Sometimes too many
action. For example, an assignment of 75%
connections can also stop the spread of an idea. Some networks have tight-knit cliques, where even if an idea
means that the individual thinks that there is
spreads within one clique, it can be difficult for it to spread to other cliques.
a 75% chance that action B is better than
action A. If the value is greater than 50%, then
Too few connections: Complex idea Too many connections: Just right: the individual performs action B. Then, we use
(green) starts with a node Idea doesn’t spread Idea spreads Bayes’s rule—which probability theory tells us
(individual) and doesn’t spread far is the rational way to change beliefs in light of
evidence—to update the individual’s credence
100%
in light of this result and then update all
0% 20% connections in his or her network.
0%
Action A
is better 25% chance that
action B is better
Sources: The Wisdom and/or Madness of Crowds, by Nicky Case https://ncase.me/crowds; T he Misinformation Age:
50%
Starting point: Idea spreads Idea spreads Action B
Complex idea starts with an individual within group (bonding) to another group (bridging) 75% chance that
is better
How False Beliefs Spread, b y Cailin O’Connor and James Owen Weatherall. Yale University Press, 2019
action B is better
50% 100%
33%
Square nodes are individuals who
test actions and update their beliefs
accordingly (evidence seekers)
66%
Circular nodes represent individuals
Time Time who observe results from others
but do not test the actions directly
25% 40% (observers)
Stars represent individuals who do not

hold beliefs of their own but instead
focus on introducing selective results
into the system (propagandists)
34 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022 Graphic by Jen Christiansen

UPDATING AND EXPERIMENTING: Initial credences Experiment results (successes out of 10) Updated credences
Individuals in these models start with some
random level of certainty, or credence, about Four (blue) 6 5
whether action A or B is better. They then test action A
take the action they prefer—that is, “experi
ment.” Their outcomes provide evidence Time 7 4 Time
about the success of these actions, which they
Two (orange)
share with neighbors. All individuals update
their credences based on what they observe.
test action B 2 5
CONVERGENCE ON TRUE BELIEFS: Over time the social connections in these models mean that groups of people come to a consensus about whether A or B is better.
As they gather and share evidence, they usually learn that the better action is, indeed, better. Someone trying the worse action, for instance, will see how much better their
neighbor is doing and switch. Sometimes, though, strings of misleading evidence will convince the entire group that the worse action is better.
Time step 1 Time step 2 Time step 3 Time step 4 Time step 5 Time step 6
POLARIZATION: If we add social trust or Stable, opposing beliefs within a group Cliquish arrangement with stable opposing beliefs due to conformity
conformity to these models, they may no Lower level
longer reach consensus. If each individual of trust
trusts the evidence that comes from those (faint lines)
who share their beliefs, polarized camps
that listen only to those in their group form.
If each individual seeks to conform their
actions with group members, good ideas
fail to spread between cliques.
Believes action B (orange) is better but conforms to actions of blue clique
EVIDENCE SEEKERS, OBSERVERS AND Observers Evidence seekers Propagandist Observers Evidence seekers
PROPAGANDISTS: In some cases, propa
gandists try to mislead a group of people about
scientific results. We can use these models to
represent a set of evidence seekers who gather
evidence, a group of observers who update
beliefs based on this evidence, and a propa
gandist who misleads the observers.
BELIEF UPDATING WHEN SELECTIVE RESULTS ARE IN PLAY: Industrial propagandists shape public belief by selectively sharing only those results that happen
to spuriously support the worse action. This can mislead the public, even in cases when groups of evidence seekers converge to a consensus about the true belief.
This strategy for public disinformation takes advantage of the inherent randomness of scientific results to mislead.
Initial credences Experiment results (successes out of 10) Updated credences

Propagandist shares only results
that support action A (blue)
4 5 3
9
5 3 7
3
5 8 6

the participants in these Facebook groups have concluded, then does not fit what we see in the real world. In actual communi-
the risks go the other way. This means that figuring out what is ties, we see polarization—entrenched disagreement about wheth-
true, and acting accordingly, matters deeply. er to vaccinate. We argue that the basic model is missing two
To better understand this behavior in our research, we drew crucial ingredients: social trust and conformism.
on what is called the network epistemology framework. It was Social trust matters to belief when individuals treat some
first developed by economists more than 20 years ago to study sources of evidence as more reliable than others. This is what
the social spread of beliefs in a community. Models of this kind we see when anti-vaxxers trust evidence shared by others in
have two parts: a problem and a network of individuals (or their community more than evidence produced by the Centers
“agents”). The problem involves picking one of two choices. for Disease Control and Prevention or other medical research
These could be “vaccinate” and “don’t vaccinate” your children. groups. This mistrust can stem from all sorts of things, includ-
In the model, the agents have beliefs about which choice is better. ing previous negative experiences with doctors or concerns that
Some believe vaccination is health care or governmental
safe and effective, and others institutions do not care about
believe it causes autism. their best interests. In some
Agents’ beliefs shape their cases, this distrust may be
behavior—those who think justified, given that there is a
vaccination is safe choose to long history of medical re
perform vaccinations. Their searchers and clinicians ig
be
havior, in turn, shapes noring legitimate issues from
their beliefs. When agents patients, particularly women.
vaccinate and see that noth- Yet the net result is that
ing bad happens, they be anti-vaxxers do not learn
come more convinced vacci- from the very people who
nation is indeed safe. are collecting the best evi
The second part of the dence on the subject. In ver-
model is a network that rep sions of the model where
resents social connections. individuals do not trust evi-
Agents can learn not only dence from those who hold
from their own experiences PROTESTERS attend a 2015 rally against California state law very different beliefs, we
of vaccinating but also from SB277 banning personal exemptions from childhood vaccines. find communities become
the experiences of their polarized, and those with
neighbors. Thus, an individual’s community is highly important poor beliefs fail to learn better ones.
in determining what beliefs they ultimately develop. Conformism, meanwhile, is a preference to act in the same
The network epistemology framework captures some essen- way as others in one’s community. The urge to conform is a pro-
tial features missing from contagion models: individuals inten- found part of the human psyche and one that can lead us to take
tionally gather data, share data and then experie nce conse- actions we know to be harmful. When we add conformism to the
quences for bad beliefs. The findings teach us some important model, what we see is the emergence of cliques of agents who
lessons about the social spread of knowledge. The first thing we hold false beliefs. The reason is that agents connected to the out-
learn is that working together is better than working alone be- side world do not pass along information that conflicts with
cause someone facing a problem like this is likely to premature- their group’s beliefs, meaning that many members of the group
ly settle on the worse theory. For instance, they might observe never learn the truth.
one child who turns out to have autism after vaccination and Conformity can help explain why vaccine skeptics tend to
conclude that vaccines are not safe. In a community, there tends cluster in certain communities. Some private and charter schools
to be some diversity in what people believe. Some test one ac- in southern California have reported vaccination rates in the low
tion; some test the other. This diversity means that usually double digits. And rates have been startlingly low among Somali
enough evidence is gathered to form good beliefs. immigrants in Minneapolis and Orthodox Jews in Brooklyn—
But even this group benefit does not guarantee that agents two communities that have suffered from measles outbreaks.
learn the truth. Real scientific evidence is probabilistic, of Interventions for vaccine skepticism need to be sensitive to
Michael Macor/S an Francisco Chronicle v ia Getty Images
course. For example, some nonsmokers get lung cancer, and both social trust and conformity. Simply sharing new evidence
some smokers do not get lung cancer. This means that some with skeptics will probably not help because of trust issues. And
studies of smokers will find no connection to cancer. Relatedly, convincing trusted community members to speak out for vacci-
although there is no actual statistical link between vaccines and nation might be difficult because of conformism. The best ap
autism, some vaccinated children will be autistic. Thus, some proach is to find individuals who share enough in common with
parents observe their children developing symptoms of autism members of the relevant communities to establish trust. A rab-
after receiving vaccinations. Strings of misleading evidence of bi, for instance, might be an effective vaccine ambassador in
this kind can be enough to steer an entire community wrong. Brooklyn, whereas in southern California, you might need to
In the most basic version of this model, social influence get Gwyneth Paltrow involved.
means that communities end up at consensus. They decide Social trust and conformity can help explain why polarized
either that vaccinating is safe or that it is dangerous. But this beliefs can emerge in social networks. But at least in some cas-

es, including the Somali community in Minnesota and Ortho- beliefs are widely held, including by your friends and others
dox Jewish communities in New York, they are only part of the with whom you identify. Even the PEACH creators may have
story. Both groups were the targets of sophisticated misinfor- encountered this kind of synthetic discourse about vaccines.
mation campaigns designed by anti-vaxxers. According to a 2018 article in the American Journal of Public
Health, such disinformation was distributed by accounts linked
INFLUENCE OPERATIONS to Russian influence operations seeking to amplify American
How we vote, what we buy and who we acclaim all depend on discord and weaponize a public health issue. This strategy
what we believe about the world. As a result, there are many works to change minds not through rational arguments or evi-
wealthy, powerful groups and individuals who are interested in dence but simply by manipulating the social spread of knowl-
shaping public beliefs—including those about scientific matters edge and belief.
of fact. There is a naive idea that when industry attempts to influ- The sophistication of misinformation efforts (and the highly
ence scientific belief, they do it by buying off corrupt scientists. targeted disinformation campaigns that amplify them) raises a
Perhaps this happens sometimes. But a careful study of historical troubling problem for democracy. Returning to the measles exam-
cases shows there are much more subtle—and arguably more ple, children in many states can be exempted from mandatory
effective—strategies that industry, nation states and other groups vaccinations on the grounds of “personal belief.” This became a
utilize. The first step in protecting ourselves from this kind of flash point in California in 2015 following a measles outbreak
manipulation is to understand how these campaigns work. traced to unvaccinated children visiting Disneyland. Then governor
A classic example comes from the tobacco industry, which Jerry Brown signed a new law, SB277, removing the exemption.
developed new techniques in the 1950s to fight the growing Immediately vaccine skeptics filed paperwork to put a refer-
consensus that smoking kills. During the 1950s and 1960s the endum on the next state ballot to overturn the law. Had they
Tobacco Institute published a bimonthly newsletter called succeeded in getting 365,880 signatures (they made it to only
“Tobacco and Health” that reported only scientific research sug- 233,758), the question of whether parents should be able to opt
gesting tobacco was not harmful or research that emphasized out of mandatory vaccination on the grounds of personal belief
uncertainty regarding the health effects of tobacco. would have gone to a direct vote—the results of which would
The pamphlets employ what we have called selective sharing. have been susceptible to precisely the kinds of disinformation
This approach involves taking real, independent scientific re campaigns that have caused vaccination rates in many commu-
search and curating it by presenting only the evidence that favors nities to plummet.
a preferred position. Using variants on the models described ear- Luckily, the effort failed. But the fact that hundreds of thou-
lier, we have argued that selective sharing can be shockingly effec- sands of Californians supported a direct vote about a question
tive at shaping what an audience of nonscientists comes to believe with serious bearing on public health, where the facts are clear
about scientific matters of fact. In other words, motivated actors but widely misconstrued by certain activist groups, should give
can use seeds of truth to create an impression of uncertainty or serious pause. There is a reason that we care about having poli-
even convince people of false claims. cies that best reflect available evidence and are responsive to re
Selective sharing has been a key part of the anti-vaxxer play- liabale new information. How do we protect public well-being
book. Before the 2018 measles outbreak in New York, an organi- when so many citizens are misled about matters of fact? Just as
zation calling itself Parents Educating and Advocating for Chil- individuals acting on misinformation are unlikely to bring about
dren’s Health (PEACH) produced and distributed a 40-page the outcomes they desire, societies that adopt policies based on
pamphlet entitled “The Vaccine Safety Handbook.” The infor- false belief are unlikely to get the results they want and expect.
mation shared—when accurate—was highly selective, focusing The way to decide a question of scientific fact—are vaccines
on a handful of scientific studies suggesting risks associated safe and effective?—is not to ask a community of nonexperts to
with vaccines, with minimal consideration of the many studies vote on it, especially when they are subject to misinformation
that find vaccines to be safe. campaigns. What we need is a system that not only respects the
The PEACH handbook was especially effective because it processes and institutions of sound science as the best way we
combined selective sharing with rhetorical strategies. It built have of learning the truth about the world but also respects
trust with Orthodox Jews by projecting membership in their core democratic values that would preclude a single group,
community (although it was published pseudonymously, at such as scientists, dictating policy.
least some authors were m embers) and emphasizing concerns We do not have a proposal for a system of government that
likely to resonate with them. It cherry-picked facts about vac- can perfectly balance these competing concerns. But we think
cines intended to repulse its particular audience; for instance, the key is to better separate two essentially different issues: What
it noted that some vaccines contain gelatin derived from pigs. are the facts, and what should we do in light of them? Democrat-
Wittingly or not, the pamphlet was designed in a way that ex ic ideals dictate that both require public oversight, transparency
ploited social trust and conformism—the very mechanisms cru- and accountability. But it is only the second—how we should
cial to the creation of human knowledge. make decisions given the facts—that should be up for a vote.
Worse, propagandists are constantly developing ever more
sophisticated methods for manipulating public belief. Over the Cailin O’Connor i s an associate professor of logic and philosophy of science, and James
past several years we have seen purveyors of disinformation roll Owen Weatherall is a professor of logic and philosophy of science at the University of
out new ways of creating the impression—especially through California, Irvine.
social media conduits such as Twitter bots, paid trolls, and the They are co-authors of T he Misinformation Age: How False Beliefs Spread ( Yale University
hacking or copying of friends’ accounts—that certain false Press, 2019). Both are members of the Institute for Mathematical Behavioral Sciences.

DECISION-MAKING
TOUGH
CALLS How we make decisions in the face
of incomplete knowledge and uncertainty
By Baruch Fischhoff
Illustration by Wesley Allsbrook

DECISION-MAKING

DECISION-MAKING
P sychologists study how humans make decisions by

giving people “toy” problems. In one study, for example,
my colleagues and I described to subjects a hypothetical
disease with two strains. Then we asked, “Which would
you rather have? A vaccine that completely protects you
against one strain or a vaccine that gives you 50 percent
protection against both strains?” Most people chose the
first vaccine. We inferred that they were swayed by the phrase about complete
protection, even though both shots gave the same overall chance of getting sick.
But we live in a world with real problems, not just
toy ones—situations that sometimes require people
to make life-and-death decisions in the face of incom-
plete or uncertain knowledge. Years ago, after I had
begun to investigate decision-making with my col
leagues Paul Slovic and the late Sarah Lichtenstein,
both at the firm Decision Research in Eugene, Ore.,
we started getting calls about non-toy issues—calls
mate the annual death toll from causes such as
drowning, emphysema and homicide and then com
pared their estimates with scientific ones. Based on
previous research, we expected that people would
make generally accurate predictions but that they
would overestimate deaths from causes that get
splashy or fre q uent headlines—mur d ers, torna-
does—and underestimate deaths from “quie t kill-
from leaders in industries that produced nuclear ers,” such as stroke and asthma, that do not make
power or genetically modified organisms (GMOs). big news as often.
The gist was: “We’ve got a wonderful technology, but Overall, our predictions fared well. People over
people don’t like it. Even worse, they don’t like us. estimated highly reported causes of death and un
Some even think that we’re evil. You’re psychologists. derestimated ones that received less attention.
Do something.” Images of terror attacks, for example, might explain
We did, although it probably wasn’t what these why people who watch more television news worry
company officials wanted. Instead of trying to change more about terrorism than individuals who rarely
people’s minds, we set about learning how they watch. But one puzzling result emerged when we
really t hought about these technologies. To that end, probed these beliefs. People who were strongly op
we asked them questions designed to reveal how posed to nuclear power believed that it had a very
they assessed risks. The answers helped us under- low annual death toll. Why, then, would they be
stand why people form beliefs about divisive issues against it? The apparent paradox made us wonder if
such as nuclear energy—and today, climate change— by asking them to predict average annual death tolls,
when they do not have all the facts. we had defined risk too narrowly. So, in a new set of
questions we asked what risk really meant to people.
INTIMATIONS OF MORTALITY When we did, we found that those opposed to nuclear
T o s ta rt o f f, we wanted to figure out how well the power thought the technology had a greater poten-
general public understands the risks they face in tial to cause widespread catastrophes. That pattern
everyday life. We asked groups of laypeople to esti- held true for other technologies as well.

DECISION-MAKING
To find out whether knowing more about a tech archaeology and glaciology. In complex climate anal
nology changed this pattern, we asked technical ex yses, these expert judgments reflect great knowedge
perts the same questions. The experts generally driven by evidence-based feedback. But some aspects
agreed with laypeople about nuclear power’s death still remain uncertain.
toll for a typical year: low. But when they defined risk My first encounter with these analyses was in
themselves, on a broader time frame, they saw less 1979, as part of a project planning the next 20 years
potential for problems. The general public, unlike the of climate research. Sponsored by the Department of
experts, emphasized what could happen in a very bad Energy, the project had five working groups. One
year. The public and the experts were talking past dealt with the oceans and polar regions, a second
each other and focusing on different parts of reality. with the managed biosphere, a third with the less
UNDERSTANDING RISK
D i d e x p e rt s a lways have an accurate understand-
ing of the probabilities for disaster? Experts analyze
risks by breaking complex problems into more know- When the Public Disagrees
able parts. With nuclear power, the parts might in
clude the performance of valves, control panels, about Science
evacuation schemes and cybersecurity defenses. On politically controversial scientific issues, polarization is greater
With GMO crops, the parts might include effects on among better-informed people. Investigators saw this effect in two
human health, soil chemistry and insect species. national surveys in the U.S. The surveys, conducted in 2006 and 2010,
The quality and accuracy of a risk analysis depend combined to cover just more than 6,500 people. Participants were
on the strength of the science used to assess each asked what they believed on several hot topics and whether they agreed
part. Science is fairly strong for nuclear power and with scientific consensus. As education and science literacy increased
GMOs. For new technologies such as self-driving ve among liberals and conservatives, so did their divergence. This may be
hicles, it is a different story. The components of risk because more well-versed people are better attuned to the position of
could be the probability of the vehicle laser-light sen- their political group and more confident in defending it.
sors “seeing” a pedestrian, the likelihood of a pedes-
trian acting predictably, and the chances of a driver
taking control at the exact moment when a pedes- General Education Science Education Science Literacy
trian is unseen or unpredictable. The physics of Less More Less More Less More
pulsed laser-light sensors is well understood, but More consistent with
how they perform in snow and gloom is not. Research scientific consensus
on how pedestrians interact with autonomous vehi-
cles barely exists. And studies of drivers predict that
they cannot stay vigilant enough to handle infre- Less consistent with
quent emergencies. scientific consensus
Government should support stem cell research
When scientific understanding is incomplete, risk
analysis shifts from reliance on established facts to
expert judgment. Studies of those judgments find
that they are often quite good—but only when experts
get good feedback. For example, meteorologists rou-
Source: “Individuals with Greater Science Literacy and Education Have More Polarized Beliefs
Political Identity
on Controversial Science Topics,” by Caitlin Drummond and Baruch Fischhoff, in P roceedings
tinely compare their probability-of-precipitation fore-

casts with the rain gauge at their station. Given that Extremely
clear, prompt feedback, when forecasters say that liberal The universe developed from the big bang
there is a 70 percent chance of rain, it rains about Liberal
ol. 114, No. 36; September 5, 2017
70 percent of the time. With new technologies such as Slightly

the self-driving car or gene editing, however, feed- liberal
back will be a long time coming. Until it does, we will Moderate
be unsure—and the experts themselves will not Slightly
know—how accurate their risk estimates really are. conservative
Humans developed through evolution
Conservative
of the National Academy of Sciences USA, V
THE SCIENCE OF CLIMATE SCIENCE Extremely

E x p e r t j u d g m e n t, which is dependent on good conservative
feedback, comes into play when one is predicting the
costs and benefits of attempts to slow climate change
or to adapt to it. Climate analyses combine the judg-
ments of experts from many research areas, includ-
ing obvious ones, such as atmospheric chemistry and Climate change is a serious concern
oceanography, and less obvious ones, such as botany,
Graphic by Jen Christiansen SCIENTIFICAMERICAN.COM | 41

DECISION-MAKING
managed biosphere, and a fourth with economics lated into terms that are relevant to people’s deci-
and geopolitics. The fifth group, which I joined, dealt sions about their lives, their communities and their
with social and institutional responses to the threat. society. While most scientists are experienced com-
Even then, more than 40 years ago, the evidence municators in a classroom, out in the world they
was strong enough to reveal the enormous gamble may not get feedback on how clear or relevant their
being taken with our planet. Our overall report, sum- messages are.
marizing all five groups, concluded that “the proba- Addressing this feedback problem is straightfor
ble outcome is beyond human experience.” ward: test messages before sending them. One can
learn a lot simply by asking people to read and para
THINKING OF THE UNTHINK ABLE phrase a message. When communication researchers
H o w, t h e n, c a n r e s e a r c h e r s in this area fulfill have asked for such rephrasing about weather fore
their duty to inform people about accurate ways casts, for example, they have found that some are
to think about events and choices that are beyond confused by the statement that there is a “70 percent
their experience? Scientists can, in fact, accomplish chance of rain.” The problem is with the words, not
this if they follow two basic lessons from studies the number. Does the forecast mean it will rain
of decision-making. 70 percent of the time? Over 70 percent of the area?
Or there is a 70 percent chance of at least 0.01 inch of
L ESSON 1: The facts of climate science will not rain at the weather station? The last interpretation is
speak for themselves. The science needs to be trans- the correct answer.
Many studies have found that numbers, such as
70 percent, generally communicate much better than
“verbal quantifiers,” such as “likely,” “some” or “of
ten.” One classic case from the 1950s involves a U.S.
Risky Business National Intelligence Estimate that said “an attack
on Yugoslavia in 1951 should be considered a serious
The way people view the risks of technologies and activities depends on
possibility.” When asked what probability they had
factors such as familiarity, whether exposure is voluntary or involuntary,
in mind, the analysts who signed the document gave
and the likelihood of fatalities. Novelty, involuntary exposure and lethal
a wide range of numbers, from 20 to 80 percent. (The
potential lead people to rate things as riskier, assessments that some
Soviets did not invade.)
times differ from scientific counts and estimates. The results come from
Sometimes people want to know more than the
surveys given to laypeople and first published in 1978; they have been
probability of rain or war when they make decisions.
repeated often with similar findings.
They want to understand the processes that lead to
those probabilities: how things work. Studies have
Nuclear power found that some critical aspects of climate change
Involuntary exposure, research are not intuitive for many people, such as
unfamiliar, Pesticides how scientists can bicker yet still agree about the
Food coloring
new, threat of climate change or how carbon dioxide is
Food preservatives catastrophic potential different from other pollutants. (It stays in the at
Source: R isk: A Very Short Introduction, b y Baruch Fischhoff and John Kadvany. Oxford University Press, 2011;
mosphere longer.) People may reject the research
Redrafted from “How Safe Is Safe Enough? A Psychometric Study of Attitudes towards Technological Risks
Spray cans
results unless scientists tell them more about how
Antibiotics they were derived.
X-rays
Contraceptives LESSON 2: People who agree on the facts can still
and Benefits,” by Baruch Fischhoff et al., in Policy Sciences, Vol. 9, No. 2; April 1978
Vaccination Surgery disagree on what to do about them. A solution that
Not definitely fatal, Definitely fatal, seems sound to some can seem too costly or unfair
common instill dread to others.
Railroads Motor vehicles
General aviation For example, people who like plans for carbon
Home appliances Electric capture and sequestration, because it keeps carbon
power Smoking
Football Construction dioxide out of the air, might oppose using it on
Handguns
Police work Motorcycles coal-fired power plants. They fear an indirect
Bicycles
Power mowers Firefighting consequence: cleaner coal may make mountaintop-
Skiing Alcoholic Hunting removal mining more acceptable. Those who know
beverages Swimming
Mountain climbing what cap-and-trade schemes are meant to do—
Voluntary exposure, create incentives for reducing emissions—might
familiar, still believe that they will benefit banks more than
old, the environment.
chronic risk These examples show why two-way communica-
tion is so important in these situations. We need to
learn what is on others’ minds and make them feel

DECISION-MAKING
like partners in decision-making.

Sometimes that communication
will reveal misunderstandings
that research can reduce. Or it
may reveal solutions that make
more people happy. One example
is British Columbia’s revenue-
neutral carbon tax, which pro-
vides revenues that make other
taxes lower; it has also received
broad enough political support
to weather several changes of
government since 2008. Some-
times, of course, better two-way
communication will reveal fun-
damental disagreements, and in
those cases action becomes a
matter for the courts, streets and
ballot boxes.
MORE THAN SCIENCE

These lessons about how facts YOUNG ACTIVISTS g athered in New York City in May 2019 to demand
are communicated and inter- immediate action on climate change.
preted are important because cli-
mate-related decisions are not
always based on what research says or shows. For arguing the issues. A third, related explanation is
some individuals, scientific evidence or economic that they are more likely to see, and seize, the chance
impacts are less important than what certain deci- to express themselves than those who do not know
sions reveal about their beliefs. These people ask how as much.
their choice will affect the way others think about
them, as well as how they think about themselves. WHEN DECISIONS MATTER MOST
For instance, there are people who forgo energy Although decision science researchers still have
conservation measures but not because they are much to learn, their overall message about ways to
against conservation. They just do not want to be deal with uncertain, high-stakes situations is optimis-
perceived as eco-freaks. Others who conserve do it tic. When scientists communicate poorly, it often indi-
more as a symbolic gesture and not based on a belief cates that they have fallen prey to a natural human
that it makes a real difference. Using surveys, tendency to exaggerate how well others understand
researchers at Yale Climate Connections identified them. When laypeople make mistakes, it often reflects
what they call Six Americas in terms of attitudes, their reliance on mental models that have served
ranging from alarmed to dismissive. People at those them well in other situations but that are not accurate
two extremes are the ones who are most likely to in current circumstances. When people disagree
adopt measures to conserve energy. The alarmed about what decisions to make, it is often because they
group’s motives are what you might expect. Those in have different goals rather than different facts.
the dismissive group, though, may see no threat from In each case, the research points to ways to help
climate change but also have noted they can save people better understand one another and them-
money by reducing their energy consumption. selves. Communication studies can help scientists
Knowing the science does not necessarily mean create clearer messages. And decision science can
agreeing with the science. The Yale study is one of help members of the public to refine their mental
several that found greater polarization among dif models to interpret new phenomena. By reducing
ferent political groups as people in the groups gained miscommunication and focusing on legitimate dis-
knowledge of some science-related issues. In our re agreements, decision researchers can help society
search, Caitlin Drummond, currently a postdoctoral have fewer conflicts and make dealing with the ones
fellow at the University of Michigan’s Erb Institute, that remain easier for us all.
and I have uncovered a few hints that might account
Michael Nigro/Getty Images
for this phenomenon. One possible explanation is Psychologist Baruch Fischhoff is Howard Heinz University Professor in the
that more knowledgeable people are more likely to department of engineering and public policy and the Institute for Politics and
know the position of their affiliated political group Strategy at Carnegie Mellon University. He is a member of the National
on an issue and align themselves with it. A second Academy of Sciences and National Academy of Medicine and past president
possibility is that they feel more confident about of the Society for Risk Analysis.

DECISION-MAKING
Confronting
Unknowns
How to interpret uncertainty in common forms
of data visualization
By Jessica Hullman
W
hen tracking a hurricane, forecasters Ideally, visualizations help us make judgments, analytically and
often show a map depicting a “cone of uncer- emotionally, about the probability of different outcomes. Abun-
tainty.” It starts as a point—the hurricane’s cur- dant evidence on human reasoning suggests, however, that when
rent position—and widens into a swath of ter- people are asked to make judgments involving probability, they
ritory the storm might cross in the upcoming often discount uncertainty. As society increasingly relies on data,
days. The most likely path is along the centerline of the cone, with graphics designers are grappling with how best to show uncer-
the probability falling off toward the edges. The problem: many tainty clearly.
people misinterpret the cone as the size of the future storm. What follows is a gallery of visualization techniques for dis-
Researchers have found that the misunderstanding can be pre- playing uncertainty, organized roughly from less effective to
vented if forecasters instead show a number of possible paths. Yet more effective. Seeing how different approaches are chosen and
this approach can also introduce misunderstanding: lots of peo- implemented can help us become more savvy consumers of data
ple think the probability of damage is greater where each path and the uncertainty involved.
intersects land and less likely between the lines (maps).
Uncertainty pervades the data that scientists and all kinds of Jessica Hullman is an associate professor of computer science and journalism at North
organizations use to inform decisions. Visual depictions of infor- western University. She and her research group develop and evaluate data-visualization
mation can help clarify the uncertainty—or compound confusion. and data-interaction techniques to enhance reasoning about uncertainty.
OK TN OK TN
AR AR
Predictions Using Representative Samples from Ensembles of Forecast Tracks,” by Le Liu et al., in I EEE
AL GA AL GA
ol. 25; August 20, 2018 (m ultiple storm paths)

MS MS
Sources: National Hurricane Center (c one of uncertainty) ; “Visualizing Uncertain Tropical Cyclone
1 A.M. Sunday 1 A.M. Sunday
LA FL LA FL
TX 1 A.M. Saturday TX 1 A.M. Saturday

Projected storm size:
1 A.M. Friday
Most likely storm
center path
Transactions on Visualization and Computer Graphics, V
Possible
storm center position
Projected storm center position: 1 A.M. Friday Possible storm center paths
Measured storm center position: 4 A.M. Thursday Measured storm center position: 4 A.M. Thursday
“CONE OF UNCERTAINTY” (left) shows where a hurricane may head, according to a group of forecasts. An alternative is to show
the specific path predicted by each forecast (right). Both approaches have pros and cons in helping people judge the risk they may face,
but the one on the right makes it clearer that the path is difficult to predict.
44 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022 Maps by Tiffany Farrant-Gonzalez

DECISION-MAKING
INTERVALS
Intervals may be the most common representations of quantified uncertainty. Error bars (top)
and confidence envelopes (bottom) are widely recognized, but even though they seem exact
Group A
and straightforward, they are notoriously hard to interpret properly. Research shows they are
Group B
often misunderstood, even by scientists.
PROS
● Widely recognized as a representation of uncertainty.
Group C ● O ffers a simple format for expressing the possibility of different values.
● The choice of interval can be customized for different types of questions about the same data
set. For example, when one is making inferences about the range of values in a population,
intervals based on standard deviation are helpful; for inferences about the range of values
of a statistic such as a mean, intervals based on standard error are appropriate.
CONS
Group A
● Ambiguity in what is shown: intervals may represent standard deviation, standard error
or something else. Each has a unique interpretation.
Group B
● Readers can make “deterministic construal errors”—interpreting the ends of the error
bar as the high and low values in observed measurements rather than estimates
Group C
denoting uncertainty.
● Error bars can lead to “within-the-bar bias,” common in bar charts. Below, readers may
0 20 40 60
see the bar values to the right of the dots as more probable than the bar values to the left.
● Easy to ignore the uncertainty regions in favor of the central tendency, which may lead
NO QUANTIFICATION to incorrect decisions.
The least effective way to present uncertainty is to not show it at all.
Sometimes data designers try to compensate for a lack of specified
Error bar: 95% confidence interval Mean
uncertainty by choosing a technique that implies a level of imprecision
but does not quantify it. For example, a designer might map data to a Group A
visual variable that is hard for people to define, such as a circle floating Group B
in space (top) rather than a dot on a graph that has x and y axes. This
approach makes the reader’s interpretation more error-prone. Alter Group C
natively a designer might use a program that creates a hand-drawn or
“sketchy” feel (bottom). Both approaches are risky. 0 20 40 60 80
PROS Error range: 95% confidence envelope

Height
170
● If readers sense that a visualization is difficult to quantify or
is simply impressionistic, they may be more cautious in making 160
inferences or decisions based on it.
150
CONS
140 Each dot is one measurement
● Readers may not realize that the visualization is intended to convey
imprecision and may reach conclusions that have large errors. Trend line
130
● Even if readers recognize that the visualization was chosen
30 40 50 60
to imply imprecision, they have no way of inferring how much
Weight
uncertainty is involved.
True population 20 hypothetical samples, each made up of WHAT DOES A CONFIDENCE INTERVAL MEAN?
mean, which the same number of randomly chosen
can only be observations (black dots) from the population A natural interpretation of an error bar or confidence envelope that
estimated by denotes 95 percent confidence is that the interval has a 95 percent
taking samples Sample: 1 2 3 ... ... 20 chance of containing the true value. Yet it actually refers to the
percentage of confidence intervals that would include the true value
50
if an infinite number of random samples of the same size were pulled
45 from the data and each time a 95 percent
confidence interval was constructed.
40 Although in practice this pervasive
Even when misinterpretation may not drastically
Mean of calculated perfectly,
35 change decisions, the fact that even
sample 1 on average, 1 in 20 of the
scientists make such mistakes shows
30 95% confidence intervals
will not contain the how challenging it can be to interpret
population mean. uncertainty depictions correctly.
25
Graphics by Jessica Hullman and Jen Christiansen SCIENTIFICAMERICAN.COM | 45

DECISION-MAKING
PROBABILITY DENSITY MAPS

Group A Mean
Designers can map uncertainty directly to a visual property
of the visualization. For example, a gradient plot (top) can shift
from dark color (high probability) at the center to lighter color Group B
(low probability) at the edges. In a violin plot (bottom), wider points
mean greater probability. Mapping probability density to a visual Group C
variable displays uncertainty in greater detail than interval methods
(error bars and confidence envelopes), but its effectiveness depends
on how well readers can perceive differences in shading, height –20 0 20 40 60 80 100
or other visual properties.
PROS
● Often well aligned with intuition: dark shading or hard boundaries Group A
are certain; light shading or fuzzy boundaries are uncertain.
● Avoids common biases such as those raised by intervals. Mean
CONS
Group B
● Readers may not recognize that density reflects probability.
50% confidence interval
● Readers often equate the part of the visualization that is easiest
to read (darkest, widest) with the data values themselves and mis- 95% confidence interval
interpret the parts that are harder to read (lightest, most narrow)
Group C
as the uncertainty.
● Estimates can be biased to the darkest or highest points.
–20 0 20 40 60 80 100
● C an be difficult to infer specific probability values.
Healthy
89/100 MULTIPLE SAMPLES IN SPACE
Plotting of multiple samples in space can be used to show probability in a discrete format for
Complications one or more variable quantities. One example of this approach is a quantile dot plot. It shows
11/100 a number of distinct cases from the quantiles of the data distribution, so that the number of dots
(such as two dots high or five dots high, in the example below) conveys probability. When
there is uncertainty about parameter values from which estimates are drawn, such as initial
conditions, samples can be generated that vary these parameters and can be shown in
a single visualization.
PROS
● A designer can choose how many data samples to present, aiming to show enough to
convey the distribution but not so many that it becomes difficult for a reader to make out
the individual samples.
CONS
● Plotting many data samples can result in occlusion, making probability estimates
more error-prone.
ARRAYS OF ICONS ● Sampling introduces imprecision, especially if the underlying distribution is heavily
Reframing a probability such as 30 percent as a frequency—three out skewed by outliers.
of 10—can make it easier for people to understand uncertainty and
consequently use such information appropriately. People may better
understand discrete probabilities because they run into them in every-
day experiences.
PROS
● More self-explanatory than some other techniques because
readers can readily see that probability is analogous to the num- Group B
ber of times a symbol appears.
● Readers can make quick estimates if a small number of symbols
is used because our visual system recognizes small quantities
immediately without counting. Group C
CONS
0 40 80
● Designed to present only a single probability.
46 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022 Graphics by Jessica Hullman and Jen Christiansen

DECISION-MAKING
A JITTERY ELECTION NIGHT NEEDLE

Sometimes an uncertainty visualization is controversial. On the night of the 2016
presidential election in the U.S., the New York Times introduced an animated
gauge on its website to display predictions about the outcome. A continuum of
colored areas made up the background, from a landslide Hillary Clinton win (left)
to a landslide Donald Trump win (right). The data model behind the gauge up
dated several times a minute as new local results came in. An animated needle
jiggled back and forth rapidly, even more frequently than the model was updated.
Seeing a constantly moving visualization made many viewers anxious on an
evening when unexpected events transpired. Uncertainty visualizations should
provoke anxiety that is proportional to the uncertainty in the data. But after
decades of people seeing only static projections for election outcomes that
allowed viewers to overlook uncertainty, suddenly shifting to a visualization that
Chance of winning election provoked a visceral reaction to uncertainty was unsettling.
MULTIPLE SAMPLES IN TIME Prior bank estimates of growth Projected growth

7
Percent Change in Gross Domestic Product (GDP), Year over Year

Plotting multiple possible outcomes as frames in an animation makes uncertainty visceral and
much harder to ignore. This technique, called hypothetical outcome plots, can be used for 6
simple and complex visualizations. Perceptual studies indicate that people are surprisingly
adept at inferring the distribution of data from the frequency of occurrences: we do not neces 5
sarily need to count the number of times an event occurs to estimate its probability. One Published data
important factor is the speed of events, which must be fast enough so that people can see a 4
sufficient number of samples yet slow enough for them to consciously register what they saw.
3
PROS
● The human visual system can estimate probability fairly accurately without having 2
to deliberately count the items presented.
● C an be applied widely across different data types and visualization styles. 1 Shading shows
the bank’s
● Animation makes it possible to estimate probabilities involving multiple variables, 0 predictions of
which is difficult with static plots.
how published
CONS –1 data might
be revised. In any future
● Sampling introduces imprecision, especially if the distribution is heavily skewed
–2 three-month
by outliers.
period, the
● No guarantees on how many individual samples a user will pay attention to. –3 percent change
in GDP should
● Requires creating a dynamic or animated visualization, which some formats such as
lie somewhere
scientific papers may not yet easily support. –4
in the red
shaded area
–5
on 90 out of
100 occasions.
180 –6
Height
–7
170 2006 2008 2010 2012
Source: I nflation Report. B ank of England, February 2010 (G DP chart)
160 In an animated HYBRID APPROACHES

display, the
Designers can create effective uncertainty visualizations by combining
lines rapidly
different techniques rather than choosing a standard chart “type.”
150 appear and
One example is a fan chart, made famous by the Bank of England
disappear one
(shown). It depicts data up to the present (left side of dotted line), then
at a time.
projections into the future (right side); uncertainty about the past is
140 an important component in assessing uncertainty about the future.
The fan chart presents probability from higher chance (dark shading)
to lesser chance (light shading) in multiple bands that represent differ
130 ent levels of confidence, which the reader can choose from. Readers
can perceive the information through the position of the edges of
30 40 50 60
the bands, as well as lightness versus darkness. Some modern soft
Weight
ware packages for statistical graphics and modeling make it easy to
combine uncertainty visualization approaches.
Illustration by Tiffany Farrant-Gonzalez (election needle) SCIENTIFICAMERICAN.COM | 47

DECISION-MAKING
Q-ANON BANNER held

up at a rally supporting
President Donald Trump
on January 5, 2021,
in Washington, D.C.
OPINION
The Cause
of America’s Post-Truth
Predicament
People have been manipulated to think that
beliefs needn’t change in response to evidence,
making us more susceptible to conspiracy theories,
science denial and extremism
By Andy Norman

I
DECISION-MAKING
n the hours af ter Joe Biden was sworn Moynihan have avowed that we’re “entitled” to our
in as president on January 20, 2021, an online opinions. The trouble with this idea is that it inter-
discussion channel followed by 35,000 Q Anon feres with efforts to promote accountable talk: call
believers was rife with disbelief. “It simply something a “right,” and anything that impinges on it
doesn’t make sense that we all got played,” one counts as transgressive—transgressive, in fact, of
wrote. But they did get played. So did we all. something sacred. (Rights belong to a category of
Of course, we were played in different ways. things psychologists call “sacred values”—things
QAnon devotees were fed a ludicrous story about we’re not supposed to trade off against other things.)
Satan-worshipping, “deep state” pedophiles plotting Evidence and critical questioning can (and should)
to oust President Donald Trump. The anonymous impinge on belief, and that makes them transgressive
source of the story—“Q”—promised a purge, and tens of something we’re conditioned to see as a right. In
of thousands pinned their hopes on that happening this way, critical thinking about core values has been
before Biden could take office. Clearly, Q played them. rendered all but taboo. A core American value sys-
The insurrectionists of January 6, 2021, were also tematically subverts critical thinking.
played. At his rally to “stop the steal,” Trump fired up When we affirm one another’s “right” to believe
his audience, then sent them to the Capitol to prevent things—even things that fly in the face of evidence—
the certification of his election loss. “We [need to] we essentially decouple critical thinking and belief
fight like hell,” he said. “We’re going to walk down, revision. This damages the norm that keeps minds
and I’ll be there with you.” Only he didn’t, and he tethered to reality. In 2020 a team of researchers in
wasn’t. Later he denounced the very rioters he’d incit- Canada made an important discovery: when people
ed and left them to suffer the legal consequences of lose the “meta belief” that beliefs should change in
his sedition. “Trump just used us,” said Lenka Perron, response to evidence, they become more susceptible
a former Q Anon believer, in a 2021 New York Times to conspiracy theories, paranormal beliefs, science
article. She went on to explain that when you’re “liv- denial and extremism—mind viruses, if you will. This
ing in fear, [you’re] prone to believe this stuff.” is a critical finding. I like to put it more simply: The
Many Republicans don’t seem to recognize that idea that beliefs should yield to evidence is the linch-
they, too, are being played. The GOP now trades al pin of the mind’s immune system. Remove it—or even
most exclusively in manufactured bogeymen. “Death chip away at it—and an Internet-connected mind will
panels,” “feminazis” and the “war on Christmas” are eventually be overrun by mind parasites. When this
obvious ploys, but fearmongering is now the defin- happens to enough minds, all hell breaks loose.
ing feature of American conservatism. Socialists aim This phenomenon is the root cause of our post-
to destroy our way of life. The government is plan- truth predicament. When we buy into the prevailing
ning to seize your guns. Secularists will steal your fundamentalism about speech rights or downplay
freedom to worship. Gays will destroy the institu- the importance of accountable talk, we exacerbate
tion of marriage. BLM protesters will burn down an increasingly existential problem.
your neighborhood. Cognitive scientists call what The deep culprit here is not a shadowy govern-
Republican strategists do “amygdala hijacking,” ment insider. It’s not an aspiring demagogue or a
after the brain module that responds to fear. corrupt political party. Trace the problem to its
Brains manipulated in this way may lose the ca roots, and you find a compromised cultural immune
pacity for reasoned reflection. When Sean Hannity system. Astonishingly irrational ideas proliferate
and Tucker Carlson feed you grievance after griev- because they’re playing us.
ance—Benghazi! Hillary’s e-mails! Election theft!— If we continue to let them play us, we’ll chase one
they’re suppressing your higher brain function. another down the rabbit hole of delusion. There’s
They’re playing you. really only one alternative. First, we must grasp that
But let’s be honest: liberals are also being played. bad ideas are mind parasites—entities that can prolif-
When people fixate on the wingnut outrage of the erate and harm the very minds that host them. In
day and nurse grievances, they suppress their own fact, they can lay waste to delusion-tolerant cultures.
higher brain function. (The human brain can actual- Second, it’s time to take the emerging science of men-
ly become addicted to grievance.) Right-wing provo- tal immunity seriously. We must grasp how mental
cateurs love to “own the libs,” and too often liberals immune systems work and figure out how to strength-
play along. When they do, they play themselves. en them. Then we need to inoculate one another
Robert Nickelsberg/Getty Images
You don’t have to be a conspiracy theorist to see against the worst forms of cognitive contagion.
deeper forces at work here. America’s founders uni-
versally acclaimed the so-called liberty of conscience. Andy Norman d irects the Humanism Initiative at Carnegie Mellon
But over time this admirable principle morphed into University. He studies how ideologies short-circuit minds and develops
the idea that everyone has a right to believe as they antidotes to mental immune disruptors. His book Mental Immunity w as
please. Even liberal stalwarts such as Daniel Patrick published by Harper Wave in May 2021.

OPINION
Perfect Storm
for Fringe Science
It’s always been with us, but in a time of pandemic, its practitioners
have an amplified capacity to unleash serious harm
By David Robert Grimes
T h e e x p l o s i o n o f d i s i n f o r m at i o n who refuted the conjecture as utterly false. on ostensible gurus rather than consen-
about COVID has been a defining aspect Ineffective treatments ranging from sus opinion. The only authority a scien-
of the pandemic. Alongside the virus it- hydrochloroquine to ivermectin to vita- tist can ever truly invoke is a reflected
self, we’ve been shadowed by what the min D and alternative medicine have one, dependent on accurate representa-
World Health Organization has called an thrived, too, endorsed by a rogues’ gallery tion of the evidence base. If they embrace
infodemic. This is widely known, of of doctors and researchers. Even as the fringe positions and jettison the princi-
course, but much less discussed is the role lifesaving impact of vaccination began to ples of scientific skepticism, then their
of ostensible “experts” in perpetuating be felt across the globe, a new cohort of qualifications, education and prestige
dangerous fictions. Since the dawn of the impressively credentialed contrarians mean absolutely nothing.
crisis, a disconcerting number of emi- emerged, spreading mistruths about im- Were these claims merely vapid, that
nently qualified scientists and physicians munization. The grandiosely named would be bad enough. But they are also
have propagated falsehoods across social “World Doctors Alliance” is a potent ex- uniquely damaging to public understand-
media, elevating themselves to the status ample, boasting among its membership ing. Scientists and physicians occupy an
of gurus to lend a veneer of seeming sci- physician Vernon Coleman (an anti-vaxx- extremely trusted position in society, and
entific legitimacy to empty, dangerous er activist and author of a book insisting an imprimatur of scientific legitimacy is a
claims. And these bogus claims, like their COVID is a hoax) and Dolores Cahill, the powerful one. This is a trust utterly abused
pathological namesake, have gone uncon- once respected Irish scientist whose con- by fringe figures who present qualifica-
trollably viral. spiratorial proclamations became a staple tions as a proxy for scientific validity. This
In March 2020, for example, physician of lockdown protests and COVID denialist is superficially convincing to the point that
Thomas Cowan insisted that C OVID was disinformation across Europe. it does not matter that these videos origi-
caused by 5G radio frequencies. This asser- In slickly produced videos shared re- nated in conspiratorial circles; the intrin-
tion was both devoid of evidence and lentlessly online, these fringe scientists sic aura of “science” afforded by apparent
physically impossible, but that proved no are lauded as experts unafraid to speak experts enables them to metastasize far
impediment to its widespread acceptance, truth to power. But it is crucial to note beyond this odious origin. This in turn
with anti-5G sentiment accounting for at that these individuals, for all their formal casts a specter of doubt over the advice of
least 87 arson attacks on cell-phone towers credentials, extol a narrative completely public health bodies, distorting public un-
in the U.K. alone. The ostensible documen- at odds with reality, readily refuted by derstanding by presenting rank fictions in
tary Plandemic, starring Ph.D. virologist public health bodies the world over. the stolen robes of science.
Judy Mikovits, ratcheted up millions of These pseudoscientific, conspiratorial The rise of pseudoexperts is perhaps
views with the central thesis that the novel claims are archetypal arguments from symptomatic of a change in how we ac-
coronavirus is a planned hoax. authority in which a perceived expert’s cess information. As we become curators
Even Nobel laureates in medicine have support is used to justify positions un- of our own media, the traditional gate-
been culpable; a statement from the late supported by data. Scientific claims de- keepers and fact-checkers once implicit
virologist Luc Montagnier that COVID was rive their authority not by virtue of their in most reporting have been increasingly
probably manufactured earned him both coming from scientists but from the sidelined. This in turn has made us more
the enthusiastic embrace of conspiracy weight of the evidence behind them. polarized and reduced our ability to dif-
theorists and the enmity of scientific peers Pseudoscience, in contrast, tends to focus ferentiate fact from opinion. Motivated

DECISION-MAKING
reasoning, our human bias toward cher- ded to a belief they bend facts to fit that and staggeringly irresponsible, conduct.
ry-picking only arguments that chime preconception. Their motivations are hu- It is entirely understandable that
with what we wish were true, most cer- man; they can be seduced by the lure of many people are left confused and uneasy
tainly plays a role. The impositions of money, infamy or admiration. Science, in by the vocal assertions of fringe figures,
COVID are manifold; it is not surprising contrast, is a systemic method of inquiry but the onus of proof is always on those
that fringe scientists are inevitably in- whereby positions are formed on the to- making grand claims. The history of sci-
voked as sources for those with strong tality of evidence. Crucially, to be labeled ence and medicine is littered with the hu-
feelings against lockdowns, masks and “scientific,” ideas should be testable, and bris of the arrogant and misguided, and
vaccination. Even if we are not ideologi- those that fail to withstand dispassionate mere credentials are no impediment to
cally predisposed to such positions, these investigation should be duly discarded. being wrong; only evidence truly matters.
claims undermine public understanding, For all their qualifications, fringe sci- When confronted with the pronounce-
blurring perceptions of scientific consen- entists fail to uphold this basic tenet ment of fringe figures, the motto of the
sus, nudging us collectively toward fear of science, as they are united in their will- Royal Society should always be at the
and distrust. ingness to embrace conspiracy theory forefront of our mind: “Nullius in verba”
The dark irony is that these fringe fig- when their claims are refuted. Lack of ev- (“take nobody’s word for it”).
ures weaponize the societal trust afforded idence for their position is airily dis-
to science, unduly amplifying their capac- missed as a cover-up by everyone from David Robert Grimes is a scientist and author of G ood
ity to unleash serious harm. To mitigate the WHO to the entire medical establish- Thinking: Why Flawed Logic Puts Us All at Risk and How
this, we need to keep in mind the vital ment. But this performative outrage is Critical Thinking Can Save the World ( The Experiment,
distinction between “science” and “scien- so much sound and fury to distract from 2021). His work focuses on health disinformation and
Christina Baeriswyl
tists.” Individual scientists are far from the inescapable reality that their posi- conspiracy theory, and he is an international advocate for
infallible; they can be fooled by subtle tions are completely contradicted by the the public understanding of science. Grimes is a recipient
mistakes, be haunted by spurious conclu- overwhelming weight of scientific evi- of the Nature/Sense about Science Maddox Prize and
sions or even become so ideologically wed- dence. This is scientifically reprehensible, a fellow of the Committee for Skeptical Inquiry.

DECISION-MAKING

DECISION-MAKING
People who jump to

conclusions tend to believe
in conspiracy theories,
are overconfident and
make other mistakes
in their thinking
By Carmen Sanchez
and David Dunning
Illustration by
Islenia Milien
Leaps of
Confusion
DECISION-MAKING
H
ow much time do you spend doing research before you make
a big d
ecision? The answer for many of us, it turns out, is hardly any.
Before buying a car, for instance, most people make two or fewer trips
to a dealership. And when picking a doctor, many individuals simply use
recommendations from friends and family rather than consulting med
ical professionals or sources such as health-care websites or articles on
good physicians, according to an analysis published in the journal Health
Services Research.
We are not necessarily conserving our mental re In this game, players encountered someone who
sources to spend them on even weightier decisions. was fishing from one of two lakes: in one lake, most
One in five Americans spends more time planning of the fish were red; in the other, most were gray. The
their upcoming vacation than they do on their finan fisher would catch one fish at a time and stop only
cial future. There are people who go over every when players thought they could say which lake was
detail exhaustively before making a choice, and it is being fished. Some players had to see many fish
certainly possible to overthink things. But a fair before making a decision. Others—the jumpers—
number of individuals are quick to jump to conclu stopped after only one or two.
sions. Psychologists call this way of thinking a cogni We also asked participants questions to learn
tive bias, a tendency toward a specific mental mis more about their thought patterns. We found that
take. In this case, the error is making a call based on the fewer fish a player waited to see, the more errors
the sparsest of evidence. that individual made in other types of beliefs, rea
In our own research, we have found that hasty soning and decisions.
judgments are often just one part of larger error- For instance, the earlier people jumped, the more
prone patterns in behavior and thinking. These pat likely they were to endorse conspiracy theories, such
terns have costs. People who tend to make such jumps as the idea that the Apollo moon landings had been
in their reasoning often choose a bet in which they faked. Such individuals were also more likely to
have low chances of winning instead of one where believe in paranormal phenomena and medical
their chances are much better. myths, such as the idea that health officials are active
To study jumping, we examined decision-making ly hiding a link between cell phones and cancer.
patterns among more than 600 people from the gen Jumpers made more errors than nonjumpers on
eral population. Because much of the work on this problems that require thoughtful analysis. Consider
type of bias comes from studies of schizophrenia this brainteaser: “A baseball bat and ball cost $1.10
( jumping to conclusions is common among people together. The bat costs $1 more than the ball. How
with the condition), we borrowed a thinking game much does the ball cost?” Many respondents leaped
used in that area of research. to the conclusion of 10 cents, but a little thought

DECISION-MAKING
reveals the right answer to be five cents. (It’s true;

think the problem through.)
In a gambling task, people with a tendency to jump
were more often lured into choosing inferior bets over
those in which they had a better chance of winning.
A baseball bat
Specifically, jumpers fell into the trap of focusing on
the number of times a winning outcome could happen
rather than the full range of possible outcomes.
and ball cost
$1.10 together.
Jumpers also had problems with overconfidence:
on a quiz about U.S. civics, they overestimated the
chance that their answers were right significantly
The bat costs

more than other participants did—even when their
answers were wrong.
The distinctions in decision quality between those
$1 more than
who jumped and those who did not remained even
after we took intelligence—based on a test of verbal
intellect—and personality differences into account.
the ball.
Our data also suggested the difference was not mere
ly the result of jumpers rushing through our tasks.
So what is b ehind jumping? Psychological re
How much does

searchers commonly distinguish between two path
ways of thought: automatic, known as system 1,
which reflects ideas that come to the mind easily,
spontaneously and without effort; and controlled, or
system 2, comprising conscious and effortful reason
ing that is analytical, mindful and deliberate.
We used several assessments that teased apart
the ball cost?
how automatic our participants’ responses were and
how much they engaged in deliberate analysis. We Hint: It is not 10 cents, although
found that jumpers and nonjumpers were equally many people jump to that answer.
swayed by automatic (system 1) thoughts. The jump
ers, however, did not engage in controlled (system 2)
reasoning to the same degree as nonjumpers.
It is system 2 thinking that helps people counter process, we aim to give back to schizophrenia re
balance mental contaminants and other biases in search. In some studies, as many as two thirds of
troduced by the more knee-jerk system 1. Put anoth people with schizophrenia who express delusions
er way, jumpers were more likely to accept the con also exhibit a jumping bias when solving simple, ab
clusions they made at first blush without deliberative stract probability problems, in comparison with up
examination or questioning. A lack of system 2 think to one fifth of the general population.
ing was also more broadly connected to their prob Schizophrenia is a relatively rare condition, and
lematic beliefs and faulty reasoning. much about the connection between jumping and
Happily, there may be some hope for jumpers: judgment issues is not well understood. Our work
Our work suggests that using training to target their with general populations could potentially fill this
biases can help people think more deliberatively. gap in ways that help people with schizophrenia.
Specifically, we adapted a method called metacogni In everyday life, the question of whether we
tive training from schizophrenia research and creat should think things through or instead go with our
ed a self-paced online version of the intervention. In gut is a frequent and important one. Recent studies
this training, participants are confronted with their show that even gathering just a little bit more evi
own biases. For example, as part of our approach, we dence may help us avoid a major mistake. Sometimes
ask people to tackle puzzles, and after they make the most important decision we make can be to take
mistakes related to specific biases, these errors are some more time before making a choice.
called out so the participants can learn about the
missteps and other ways of thinking through the Carmen Sanchez i s an assistant professor at the University of Illinois
problem at hand. This intervention helps to chip at Urbana-Champaign’s Gies College of Business. She studies the
away at participants’ overconfidence. development of misbeliefs, decision-making and overconfidence.
We plan to continue this work to trace other
problems introduced by jumping. Also, we wonder David Dunning is a social psychologist and a professor of psychology
whether this cognitive bias offers any potential ben at the University of Michigan. His research focuses on the psychology
efits that could account for how common it is. In the of human misbelief, particularly false beliefs people hold about themselves.

DECISION-MAKING
Big Data and

Saffir-Simpson scale of 1 to 5, corresponding to sustained wind
speeds, along with the “cones” of the storms’ probable trajecto-
ries. Outside mandatory evacuation zones, it was up to everyone
Small Decisions
to decide for themselves what to do. As management consultant
Peter Drucker is credited with saying: “If you can’t measure it,
you can’t improve it.” I’d add: “If you d om easure it, you’ll be
trapped by the number.” That’s the problem with wind intensi-
ties: wind damage is obviously relevant, but the worst impact
For individuals a deluge of facts can come from flooding. Florence came ashore as a mere cate-
gory 1, then dumped t hree feet o
f rain in some places—including
can be a problem away from the cone.
Seeking clarity, I checked in on the local TV meteorologists,
By Zeynep Tufekci who could pinpoint local impacts beyond one number. But ar-
guably they had a bias toward emphasizing the dangers, which
Illustration by Sam Island
is better both for ratings and for self-preservation: it’s much dic-
I
ier if people don’t evacuate when they should than if they flee
n 2 01 8 , a s bac k -t o - bac k H u r r i c a n e s F l o r e n c e a n d unnecessarily. So I geared up to find more data. I sought out
Michael threatened Chapel Hill, N.C., where I live and work, weather experts on social media and found well-curated lists. It
I faced a simple, binary decision like millions of others: seemed like a great idea at first. These were genuine experts. The
Stay or go? commentary was respectful and intelligent. There were links to
Nowadays data science is the hottest thing around. Com- sources, and the discussion was rich.
panies cannot hire enough practitioners. There are books and But I quickly remembered why I never want to watch sausage
online courses, and many universities are launching some being made. I learned a lot about European versus North Amer-
flavor of a data science degree or center. Classes can barely accom- ican weather models—fascinating but fairly useless when you’re
modate the demand. One would hope that this golden age would trying to decide whether to pack up a few sentimental photo-
mean we can make better decisions. But the deluge of data can, graphs and leave. One model predicted devastation, the other
paradoxically, make decision-making harder: it doesn’t easily just some heavy rain. A storm could turn north, for a direct hit,
translate into useful information. The democratization of access or south—a miss. Worse, each model updated periodically, each
and the proliferation of expert commentary can make things even run generating more expert discussion.
thornier. Finally, measurement itself is not a neutral process. Now I knew too much but had even less clarity for decision-
The days leading up to the landfall of both hurricanes, for making. This is sometimes referred to as the “paradox of
example, were dominated by their number on the familiar choice”—too many options can paralyze people trying to make a
decision. It’s that feeling you get standing in front of the ketchup
shelves in the supermarket, overwhelmed. Organic or not? Low
sugar? Sweetened with honey? With artificial sweeteners—and
if so, sucralose or aspartame? Low sodium? I have resorted to
grabbing one at random—I just want a bottle of ketchup. (Well,
glass or plastic?)
So if more data, better science and mightier computation can
give us a hurricane’s trajectory so many days in advance, why
can’t anyone make better predictions of impacts at the hyper
local level? Unfortunately, broad predictions don’t easily trickle
down, because individual outcomes retain big error ranges—too
many false positives and false negatives to be easily actionable.
So should we give up on data-driven decision-making in our
own lives? Like many things in the age of big data, the way for-
ward requires paying attention to things beyond the data—from
how and what to measure to how to communicate about it. We
need more frank talk about the shortcomings, so we can refine our
understanding of the difference between a lot of data and useful
information. And we especially need to build independent inter-
mediaries to help guide us. Data science by itself can’t do all that.
As for the hurricanes, I had just moved to my street so I did
the simplest thing I could think of: I asked my neighbors who’d
been there for a long time. They advised me to stock up on bat-
teries. They stayed put, and so did I.
Zeynep Tufekci is an associate professor at the University of North Carolina, whose
research revolves around how technology, science and society interact.

SOCIAL MEDIA’S INFLUENCE
OPINION
When “Like”
Is a Weapon
Everyone is an agent in
the new information warfare
By the Editors
Illustration by Aad Goudappel
N
o one thinks, I am the kind of person who
is susceptible to misinformation. It is those others
(stupid anti-vaxxers! arrogant liberal elites!) who are
swayed by propaganda masquerading as news and
bot armies pushing partisan agendas on Twitter.
But recent disinformation campaigns—especially
ones that originate with coordinated agencies in Rus-
sia or China—have been far more sweeping and insidious. Using
memes, manipulated videos and impersonations to spark outrage oppressed communities are also familiar with ways to push back
and confusion, these campaigns have targets that transcend any on authorities’ attempts to overwrite truth. Critically, technolo-
single election or community. These efforts aim to engineer vola- gists should collaborate with social scientists to propose interven-
tility to undermine democracy itself. If we’re all mentally exhausted tions—and they would be wise to imagine how attackers might
and disagree about what is true, then authoritarian networks can thwart these tools or turn them around to use for their own means.
more effectively push their version of reality. Playing into the “us Ultimately, though, for most disinformation operations to suc-
vs. them” dynamic makes everyone more vulnerable to false belief. ceed, it is regular users of the social Web who must share the vid-
Instead of surrendering to the idea of a post-truth world, we eos, use the hashtags and add to the inflammatory comment
must recognize this so-called information disorder as an urgent threads. That means each one of us is a node on the battlefield for
societal crisis and bring rigorous, interdisciplinary scientific reality. We need to be more aware of how our emotions and biases
research to combat the problem. We need to understand the can be exploited with precision and consider what forces might
transmission of knowledge online; the origins, motivations and be provoking us to amplify divisive messages.
tactics of disinformation networks, both foreign and domestic; So every time you want to “like” or share a piece of content,
and the exact ways even the most educated evidence seekers can imagine a tiny “pause” button hovering over the thumbs-up icon
unwittingly become part of an influence operation. Little is on Facebook or the retweet symbol on Twitter. Hit it and ask
known, for instance, about the effects of long-term exposure to yourself, Am I responding to a meme meant to brand me as a
disinformation or how it affects our brain or voting behavior. To partisan on a given issue? Have I actually read the article, or am
examine these connections, technology behemoths such as Face I simply reacting to an amusing or enraging headline? Am I shar-
book, Twitter and Google must make more of their data available ing this piece of information only to display my identity for my
to independent researchers (while protecting user privacy). audience of friends and peers, to get validation through likes? If
The pace of research must try to catch up with the rapidly grow- so, what groups might be microtargeting me t hrough my con-
ing sophistication of disinformation strategies. One positive step sumer data, political preferences and past behavior to manipu-
was the January 2019 launch of the M isinformation Review, a mul- late me with content that resonates strongly?
timedia-format journal from Harvard University’s John F. Kennedy Even if—especially if—you’re passionately aligned with or dis-
School of Government that fast-tracks its peer-review process and gusted by the premise of a meme, ask yourself if sharing it is
prioritizes articles about real-world implications of misinforma- worth the risk of becoming a messenger for disinformation meant
tion in areas such as the media, public health and elections. to divide people who might otherwise have much in common.
Journalists must be trained in how to cover deception so that It is easy to assume memes are innocuous entertainment, not
they don’t inadvertently entrench it, and governments should powerful narrative weapons in a battle between democracy and
strengthen their information agencies to fight back. Western authoritarianism. But these are among the tools of the new global
nations can look to the Baltic states to learn some of the innova- information wars, and they will only evolve as machine learning
tive ways their citizens have dealt with disinformation over the advances. If researchers can figure out what would get people to
past decade: for example, volunteer armies of civilian “elves” take a reflective pause, it may be one of the most effective ways to
expose the methods of Kremlin “trolls.” Minority and historically safeguard public discourse and reclaim freedom of thought.

A New
World
Disorder Our willingness to share content
without thinking is exploited
to spread disinformation
By Claire Wardle
Illustration by Wesley Allsbrook


A
s s o m e o n e w h o s t u d i e s t h e i m pac t o f m i s i n f o r m at i o n
on society, I often wish the young entrepreneurs of Silicon Valley
who enabled communication at speed had been forced to run
a 9/11 scenario with their technologies before they deployed
them commercially.
One of the most iconic images from that day shows a large
clustering of New Yorkers staring upward. The power of the
photograph is that we know the horror they’re witnessing. It is easy to imagine
that, today, almost everyone in that scene would be holding a smartphone. Some
would be filming their observations and posting them to Twitter and Facebook.
Powered by social media, rumors and misinformation would be rampant. Hate-
filled posts aimed at the Muslim community would proliferate, the speculation
and outrage boosted by algorithms responding to unprecedented levels of shares,
comments and likes. Foreign agents of disinformation would amplify the division,
driving wedges between communities and sowing chaos. Meanwhile those stranded
on the tops of the towers would be livestreaming their final moments.
Stress-testing technology in the context of the During the past six years the discussion around
worst moments in history might have illuminated the causes of our polluted information ecosystem has
what social scientists and propagandists have long focused almost entirely on actions taken (or not
known: that humans are wired to respond to emo- taken) by the technology companies. But this fixation
tional triggers and share misinformation if it rein- is too simplistic. A complex web of societal shifts is
forces existing beliefs and prejudices. Instead design- making people more susceptible to misinformation
ers of the social platforms fervently believed that and conspiracy. Trust in institutions is falling because
connection would drive tolerance and counteract of political and economic upheaval, most notably
hate. They failed to see how technology would not through ever widening income inequality. The effects
change who we are fundamentally—it could only map of climate change are becoming more pronounced.
onto existing human characteristics. Global migration trends spark concern that commu-
Online misinformation has been around since the nities will change irrevocably. The rise of automation
mid-1990s. But in 2016 several events made it broadly makes people fear for their jobs and their privacy.
clear that darker forces had emerged: automation, Bad actors who want to deepen existing tensions
microtargeting and coordination were fueling infor- understand these societal trends, designing content
mation campaigns designed to manipulate public that they hope will so anger or excite targeted users
opinion at scale. Journalists in the Philippines started that the audience will become the messenger. The goal
raising flags as Rodrigo Duterte rose to power, buoyed is that users will use their own social capital to rein-
by intensive Facebook activity. This was followed by force and give credibility to that original message.
unexpected results in the Brexit referendum in June Most of this content is designed not to persuade
and then the U.S. presidential election in November— people in any particular direction but to cause confu-
all of which sparked researchers to systematically sion, to overwhelm and to undermine trust in demo-
investigate the ways in which information was being cratic institutions from the electoral system to jour-
used as a weapon. nalism. And although much has been made about

preparing the U.S. electorate for election cycles since

the 2016 presidential race, misleading and conspirato- THREE CATEGORIES OF INFORMATION DISORDER
rial content did not begin with that election, and it will To understand and study the complexity of the information ecosystem, we need a common
language. The current reliance on simplistic terms such as “fake news” hides important
not end anytime soon. As tools designed to manipulate distinctions and denigrates journalism. It also focuses too much on “true” versus “fake,”
and amplify content become cheaper and more acces- whereas information disorder comes in many shades of “misleading.”
sible, it will be even easier to weaponize users as unwit-
ting agents of disinformation.
SS O HARM
ENE NTT
WEAPONIZING CONTEXT
LS TE
G e n e r a l ly, t h e l a n g ua g e u
sed to discuss the
FA
IN
misinformation problem is too simplistic. Effective
Malinformation
research and interventions require clear definitions, Deliberate publication
yet many people use the problematic phrase “fake Disinformation of private information
news.” Used by politicians around the world to attack Fabricated or for personal or
a free press, the term is dangerous. Research has Misinformation deliberately manipulated corporate rather than
shown that audiences frequently connect it with the Unintentional mistakes content. Intentionally public interest, such
such as inaccurate created conspiracy as revenge porn.
mainstream media. It is often used as a catchall to captions, dates, theories or Deliberate change of
describe things that are not the same, including lies, statistics or translations rumors. context, date or
rumors, hoaxes, misinformation, conspiracies and or when satire is time of genuine
propaganda, but it also papers over nuance and com- taken seriously. content.
plexity. Much of this content does not even masquer-
ade as news—it appears as memes, videos and social
posts on Facebook and Instagram.
In February 2017 I created seven types of “informa-
tion disorder” in an attempt to emphasize the spec-
trum of content being used to pollute the information tions around the world between 2016 and 2020, I
ecosystem. They included, among others, satire, which observed a shift in tactics and techniques. The most
is not intended to cause harm but still has the poten- effective disinformation has always been that which
tial to fool; fabricated content, which is 100 percent has a kernel of truth to it, and indeed most of the con-
false and designed to deceive and do harm; and false tent disseminated recently is not fake—it is mislead-
context, which is when genuine content is shared with ing. Instead of wholly fabricated stories, influence
false contextual information. Later that year technol- agents are reframing genuine content and using
ogy journalist Hossein Derakhshan and I published a hyperbolic headlines. The strategy involves connect-
report that mapped out the differentiations among ing genuine content with polarizing topics or people.
disinformation, misinformation and malinformation. Because bad actors are always one step (or many
Purveyors of disinformation—content that is steps) ahead of platform moderation, they are relabel-
intentionally false and designed to cause harm—are ing emotive disinformation as satire so that it will not
motivated by three distinct goals: to make money; to get picked up by fact-checking processes. In these
have political influence, either foreign or domestic; efforts, context, rather than content, is being weap
Research and Policymaking, b y Claire Wardle and Hossein Derakhshan. Council of Europe, October 2017
and to cause trouble for the sake of it. onized. The result is intentional chaos.
Graphic by Jen Christiansen; Source: I nformation Disorder: Toward an Interdisciplinary Framework for
Those who spread misinformation—false content Take, for example, the edited video of House
shared by a person who does not realize it is false or Speaker Nancy Pelosi that circulated in May 2019. It
misleading—are driven by sociopsychological factors. was a genuine video, but an agent of disinformation
People are performing their identities on social plat- slowed down the video and then posted that clip to
forms to feel connected to others, whether the “oth- make it seem that Pelosi was slurring her words. Just
ers” are a political party, parents who do not vacci- as intended, some viewers immediately began specu-
nate their children, activists who are concerned about lating that Pelosi was drunk, and the video spread on
climate change, or those who belong to a certain reli- social media. Then the mainstream media picked it up,
gion, race or ethnic group. Crucially, disinformation which undoubtedly made many more people aware of
can turn into misinformation when people share dis- the video than would have originally encountered it.
information without realizing it is false. Research has found that traditionally reporting on
We added the term “malinformation” to describe misleading content can potentially cause more harm.
genuine information that is shared with an intent to Our brains are wired to rely on heuristics, or mental
cause harm. An example of this is when Russian shortcuts, to help us judge credibility. As a result, rep-
agents hacked into e-mails from the Democratic etition and familiarity are two of the most effective
National Committee and the Hillary Clinton cam- mechanisms for ingraining misleading narratives,
paign and leaked certain details to the public to dam- even when viewers have received contextual informa-
age reputations. tion explaining why they should know a narrative is
While monitoring misinformation in eight elec- not true.

Bad actors know this: In 2018 media scholar Whit- The seemingly playful nature of these visual for-
ney Phillips published a report for the Data & Society mats means that memes have not been acknowledged
Research Institute that explores how those attempting by much of the research and policy community as
to push false and misleading narratives use techniques influential vehicles for disinformation, conspiracy or
to encourage reporters to cover their narratives. Yet hate. Yet the most effective misinformation is that
another report from the Institute for the Future found which will be shared, and memes tend to be much
that only 15 percent of U.S. journalists had been trained more shareable than text. The entire narrative is visi-
in how to report on misinformation more responsibly. ble in your feed; there is no need to click on a link. A
A central challenge now for reporters and fact-check- 2019 book by An Xiao Mina, Memes to Movements,
ers—and anyone with substantial reach, such as politi- outlines how memes are changing social protests and
cians and influencers—is how to untangle and debunk power dynamics, but this type of serious examination
falsehoods such as the Pelosi video without giving the is relatively rare.
initial piece of content more oxygen. Indeed, of the Russian-created posts and ads on
Facebook related to the 2016 election, many were
MEMES: A MISINFORMATION POWERHOUSE memes. They focused on polarizing candidates such
In January 2 017 the NPR radio show This American as Bernie Sanders, Hillary Clinton or Donald Trump
Life interviewed a handful of Trump supporters at one and on polarizing policies such as gun rights and
of his inaugural events called the DeploraBall. These immigration. Russian efforts often targeted groups
people had been heavily involved in using social media based on race or religion, such as Black Lives Matter
to advocate for the president. Of Trump’s surprising or Evangelical Christians. When the Facebook archive
of Russian-generated memes was
released, some of the commentary
at the time centered on the lack of
OF TRUMP’S SURPRISING ASCENDANCE, sophistication of the memes and
ONE OF THE DEPLORABALL INTERVIEWEES their impact. But research has

shown that when people are fearful,
EXPLAINED: “WE MEMED HIM INTO oversimplified narratives, conspira-
torial explanation and messages
POWER.... WE DIRECTED THE CULTURE.” that demonize others become far
more effective. These memes did
just enough to drive people to click
the share button.
ascendance, one of the interviewees explained: “We Technology platforms such as Facebook, Insta
memed him into power.. .. We directed the culture.” gram, Twitter and Pinterest play a significant role in
The word “meme” was first used by theorist Rich- encouraging this human behavior because they are
ard Dawkins in his 1976 book, T he Selfish Gene, to designed to be performative in nature. Slowing down
describe “a unit of cultural transmission or a unit of to check whether content is true before sharing it is
imitation,” an idea, behavior or style that spreads far less compelling than reinforcing to your “audi-
quickly throughout a culture. During the past several ence” on these platforms that you love or hate a cer-
decades the word has been appropriated to describe a tain policy. The business model for so many of these
type of online content that is usually visual and takes platforms is attached to this identity performance
on a particular aesthetic design, combining colorful, because it encourages you to spend more time on
striking images with block text. It often refers to other their sites.
cultural and media events, sometimes explicitly but Researchers have built monitoring technologies to
mostly implicitly. track memes across different social platforms. But
This characteristic of implicit logic—a nod and they can investigate only what they can access, and
wink to shared knowledge about an event or person— the data from visual posts on many social platforms
is what makes memes impactful. Enthymemes are are not made available to all researchers. Addition-
rhetorical devices where the argument is made ally, techniques for studying text such as natural-lan-
through the absence of the premise or conclusion. guage processing are far more advanced than tech-
Often key references (a recent news event, a statement niques for studying images or videos. That means the
by a political figure, an advertising campaign or a research behind solutions being rolled out is dispro-
wider cultural trend) are not spelled out, forcing the portionately skewed toward text-based tweets, web-
viewer to connect the dots. This extra work required of sites or articles published via URLs, and fact-check-
the viewer is a persuasive technique because it pulls ing of claims made by politicians in speeches.
an individual into the feeling of being connected to Although plenty of blame has been placed on the
others. If the meme is poking fun or invoking outrage technology companies—and for legitimate reasons—
at the expense of another group, those associations are they are also products of the commercial context in
reinforced even further. which they operate. No algorithmic tweak, update to

the platforms’ content-moderation guidelines or reg-

ulatory fine will alone improve our information eco- HOW DISINFORMATION BECOMES MISINFORMATION
The spread of false or misleading information is often dynamic. It starts when a
system at the level required.
disinformation agent engineers a message to cause maximum harm—for example,
designing real-life protests that put opposing groups in public conflict. In the next phase,
PARTICIPATING IN THE SOLUTION the agent creates “Event” pages on Facebook. The links are pushed out to communities
I n a h e a lt h y i n f o r m at i o n commons, people that might be intrigued. People who see the event are unaware it is a false premise and
would still be free to express what they want—but share it with their communities, using their own framing. This reproduction continues.
information that is designed to mislead, incite hatred,
reinforce polarization or cause physical harm would Creation Production Distribution
When the message When the message is turned When the product is
not be amplified by algorithms. That means it would is designed into a media product pushed out or made public
not be allowed to trend on Twitter or in the YouTube
content recommender. Nor would it be chosen to o
c ti
n
appear in Facebook feeds, Reddit searches or top R e p ro d u
Google results.
Until this amplification problem is resolved, it is
precisely our willingness to share without thinking
that agents of disinformation will use as a weapon. Islamification of Texas.” The other page, “United Mus-
Hence, a disordered information environment re lims of America,” advertised its own protest, entitled
quires that every person recognize how they, too, can “Save Islamic Knowledge,” for the exact same time
become a vector in the information wars and develop and location. The result was that two groups of peo-
a set of skills to navigate communication online as ple came out to protest each other, while the real cre-
well as offline. ators of the protest celebrated the success at amplify-
Currently conversations about public awareness ing existing tensions in Houston.
tend to be focused on media literacy, often with a Another popular tactic of disinformation agents is
paternalistic framing that the public simply needs to dubbed “astroturfing.” The term was initially con-
be taught how to be smarter consumers of informa- nected to people who wrote fake reviews for products
tion. Instead online users would be better taught to online or tried to make it appear that a fan community
develop cognitive “muscles” for emotional skepticism was larger than it really was. Now automated cam-
and trained to withstand the onslaught of content paigns use bots or the sophisticated coordination of
designed to trigger base fears and prejudices. passionate supporters and paid trolls, or a combina-
Anyone who uses websites that facilitate social tion of both, to make it appear that a person or policy
interaction would do well to learn how they work— has considerable grassroots support. They hope that if
and especially how algorithms determine what users they make certain hashtags trend on Twitter, particu-
see by “prioritiz[ing] posts that spark conversations lar messaging will get picked up by the professional
and meaningful interactions between people,” as media, and they will be able to direct the amplification
Facebook once put it. I would also recommend that to bully specific people or organizations into silence.
everyone try to buy an advertisement on Facebook at Understanding how each one of us is subject to
least once. The process of setting up a campaign helps such campaigns—and might unwittingly participate
to drive understanding of the granularity of informa- in them—is a crucial first step to fighting back against
Research and Policymaking, b y Claire Wardle and Hossein Derakhshan. Council of Europe, October 2017
tion available. You can choose to target a subcategory those who seek to upend a sense of shared reality. Per-
Graphic by Jen Christiansen; Source: I nformation Disorder: Toward an Interdisciplinary Framework for
of people as specific as women, aged between 32 and haps most important, though, accepting how vulnera-
42, who live in the Raleigh-Durham area of North Car- ble our society is to manufactured amplification
olina, have preschoolers, have a graduate degree, are needs to be done sensibly and calmly. Fearmongering
Jewish and like Kamala Harris. The company even will only fuel more conspiracy and continue to drive
permits you to test these ads in environments that down trust in quality-information sources and insti-
allow you to fail privately. These “dark ads” let organi- tutions of democracy. There are no permanent solu-
zations target posts at certain people, but they do not tions to weaponized narratives. Instead we need to
sit on that organization’s main page. This makes it dif- adapt to this new normal. Just as putting on sun-
ficult for researchers or journalists to track what posts screen was a habit that society developed over time
are being targeted at different groups of people, which and then adjusted as additional scientific research
is particularly concerning during elections. became available, building resiliency against a disor-
Facebook events are another conduit for manipu- dered information environment needs to be thought
lation. One of the most alarming examples of foreign about in the same vein.
interference in a U.S. election was a protest that took
place in Houston, Tex., yet was entirely orchestrated Claire Wardle is a professor of the practice at the Brown University School of
by trolls based in Russia. They had set up two Face Public Health and co-director of its Information Futures Lab. Previously she
book pages that looked authentically American. One was U.S. director of the nonprofit First Draft and a research fellow at the
was named “Heart of Texas” and supported secession; Shorenstein Center on Media, Politics and Public Policy at Harvard University.
it created an “event” for May 21, 2016, labeled “Stop She has a Ph.D. in communication from the University of Pennsylvania.

THE
ATTENTION
ECONOMY
Understanding how algorithms and
manipulators exploit our cognitive
vulnerabilities empowers us to fight back
By Filippo Menczer and Thomas Hills
Illustration by Cristina Spanò


C onsider Andy, who is worried about contracting COVId in 2020 .

Unable to read all the articles he sees on it, he relies on trusted friends for
tips. When one opines on Facebook that pandemic fears are overblown,
Andy dismisses the idea at first. But then the hotel where he works closes
its doors, and with his job at risk, Andy starts wondering how serious the
threat from the virus really is. No one he knows has died, after all. A col-
league posts an article about the COVID “scare” having been created by
Big Pharma in collusion with corrupt politicians, which jibes with Andy’s distrust of govern-
ment. His Web search quickly takes him to articles claiming that C OVID is no worse than the
flu. Andy joins an online group of people who have been or fear being laid off and soon finds
himself asking, like many of them, “What pandemic?” When he learns that several of his new
friends are planning to attend a rally demanding an end to lockdowns, he decides to join them.
Almost no one at the massive protest, including him, wears a mask. When his sister asks about
the rally, Andy shares the conviction that has now become part of his identity: C OVID is a hoax.
This example illustrates a minefield of cognitive bi- online information. Viewing and producing blogs, vid-
ases. We prefer information from people we trust, our eos, tweets and other units of information called
in-group. We pay attention to and are more likely to memes have become so cheap and easy that the infor-
share information about risks—for Andy, the risk of mation marketplace is inundated. Unable to process
losing his job. We search for and remember things that all this material, we let our cognitive biases decide
fit well with what we already know and understand. what we should pay attention to. These mental short-
These biases are products of our evolutionary past, cuts influence which information we search for, com-
and for tens of thousands of years, they served us well. prehend, remember and repeat to a harmful extent.
People who behaved in accordance with them—for ex- The need to understand these cognitive vulnera-
ample, by staying away from the overgrown pond bilities and how algorithms use or manipulate them
bank where someone said there was a viper—were has become urgent. At the University of Warwick in
more likely to survive than those who did not. England and at Indiana University Bloomington’s Ob-
Modern technologies are amplifying these biases servatory on Social Media (OSoMe, pronounced “awe-
in harmful ways, however. Search engines direct Andy some”), our teams are using cognitive experiments,
to sites that inflame his suspicions, and social media simulations, data mining and artificial intelligence
connects him with like-minded people, feeding his to comprehend the cognitive vulnerabilities of social
fears. Making matters worse, bots—automated social media users. Insights from psychological studies on
media accounts that impersonate humans—enable the evolution of information conducted at Warwick
misguided or malevolent actors to take advantage of inform the computer models developed at Indiana,
his vulnerabilities. and vice versa. We are also developing analytical and
Compounding the problem is the proliferation of machine-learning aids to fight social media manipu-

lation. Some of these tools are already being used by meme being shared three times was approximately
journalists, civil-society organizations and individu- nine times less than that of its being shared once.
als to detect inauthentic actors, map the spread of This winner-take-all popularity pattern of memes,
false narratives and foster news literacy. in which most are barely noticed while a few spread
widely, could not be explained by some of them being
INFORMATION OVERLOAD more catchy or somehow more valuable: the memes
The glut of information has generated intense in this simulated world had no intrinsic quality. Vi-
competition for people’s attention. As Nobel Prize– rality resulted purely from the statistical consequenc-
winning economist and psychologist Herbert A. es of information proliferation in a social network of
Simon noted, “What information consumes is rather agents with limited attention. Even when agents pref-
obvious: it consumes the attention of its recipients.” erentially shared memes of higher quality, researcher
One of the first consequences of the so-called atten- Xiaoyan Qiu, then at OSoMe, observed little improve-
tion economy is the loss of high-quality information. ment in the overall quality of those shared the most.
The OSoMe team demonstrated this result with a set Our models revealed that even when we want to see
of simple simulations. It represented users of social and share high-quality information, our inability to
media such as Andy, called agents, as nodes in a net- view everything in our news feeds inevitably leads us
work of online acquaintances. At each time step in the to share things that are partly or completely untrue.
simulation, agents may either create a meme or re- Cognitive biases greatly worsen the problem. In a
share one that they see in a news feed. To mimic lim- set of groundbreaking studies in 1932, psychologist
ited attention, agents are allowed to view only a cer- Frederic Bartlett told volunteers a Native American
tain number of items near the top of their news feeds. legend about a young man who hears war cries and,
Running this simulation over many time steps, pursuing them, enters a dreamlike battle that even-
Lilian Weng, now at OpenAI, and researchers at tually leads to his real death. Bartlett asked the vol-
OSoMe found that as agents’ attention became in- unteers, who were non-Native, to recall the rather
creasingly limited, the propagation of memes came confusing story at increasing intervals, from minutes
to reflect the power-law distribution of actual social to years later. He found that as time passed, the re-
media: the probability that a meme would be shared memberers tended to distort the tale’s culturally un-
a given number of times was roughly an inverse pow- familiar parts such that they were either lost to mem-
er of that number. For example, the likelihood of a ory or transformed into more familiar things. We now
Information Overload
Our social media news feeds a re often so full that many of us can user, linked by lines to friends or followers who receive the items
view only the top few items, from which we choose to reshare or they share or reshare. Investigators found that as the number of
retweet. Researchers at the Observatory on Social Media (OSoMe) memes in the network rises (toward the right), the quality of those
at Indiana University Bloomington simulated this limited capacity that propagate widely falls (circles become smaller). So information
to pay attention. Each node in the model network represents a overload can alone explain why fake news can become viral.
Few Number of Different Memes in Play Many

Information load is low, and quality Information load is high, and quality
Source: “Limited Individual Attention and Online Virality of Low-Quality Information,”
of shared information is high of shared information is low

Different colors represent
different memes
Meme A
Meme B
by Xiaoyan Qiu et al., in N ature Human Behaviour, Vol. 1; June 2017
Meme C
Circle size
indicates quality
Each circle Lines represent of last meme shared
represents a social connections High
media account between accounts Low

Pollution by Bots
Bots, or automated accounts that impersonate human users, when less than 1 percent of human users follow bots, information
greatly reduce the quality of information in a social network. quality is high (left). But when the percentage of bot infiltration
In one computer simulation, OSoMe researchers included bots exceeds 1, poor-quality information propagates throughout the
(modeled as agents that tweet only memes of zero quality and network (right). In real social networks, just a few early upvotes
retweet only one another) in the social network. They found that by bots can make a fake news item become viral.
Low Level of Bot Infiltration High

When bot infiltration is low, overall quality When bot infiltration is high, overall quality
of shared information is high of shared information is low
Circle tint represents
Each circle represents quality of shared information
a social media account High
Low
Pink circles
are authentic
accounts
Yellow circles are

Circle size represents
bots (automated
influence (number
accounts)
Lines and circle proximity of authentic followers)
represent connections High
between accounts Low
know that our minds do this all the time: they adjust to detect online manipulation is affected by our polit-
our understanding of new information so that it fits ical bias, though not symmetrically: Republican users
in with what we already know. One consequence of are more likely to mistake bots promoting conserva-
this so-called confirmation bias is that people often tive ideas for humans, whereas Democrats are more
seek out, recall and understand information that best likely to mistake conservative human users for bots.
confirms what they already believe.
This tendency is extremely difficult to correct. Ex- SOCIAL HERDING
periments consistently show that even when people I n N e w Yo r k C i t y in August 2019, people began
encounter balanced information containing views running away from what sounded like gunshots. Oth-
from differing perspectives, they tend to find supporters followed, some shouting, “Shooter!” Only later did
ing evidence for what they already believe. And when they learn that the blasts came from a backfiring mo-
people with divergent beliefs about emotionally torcycle. In such a situation, it may pay to run first
charged issues such as climate change are shown the and ask questions later. In the absence of clear sig-
same information on these topics, they become even nals, our brains use information about the crowd to
more committed to their original positions. infer appropriate actions, similar to the behavior of
Making matters worse, search engines and social schooling fish and flocking birds.
media platforms provide personalized recommenda- Such social conformity is pervasive. In a fascinat-
tions based on the vast amounts of data they have ing 2006 study involving 14,000 Web-based volun-
about users’ past preferences. They prioritize infor- teers, Matthew Salganik, then at Columbia Universi-
mation in our feeds that we are most likely to agree ty, and his colleagues found that when people can see
with—no matter how fringe—and shield us from in- what music others are downloading, they end up
formation that might change our minds. This makes downloading similar songs. Moreover, when people
us easy targets for polarization. Nir Grinberg and his were isolated into “social” groups, in which they could
co-workers at Northeastern University showed in 2019 see the preferences of others in their circle but had
that conservatives in the U.S. are more receptive to no information about outsiders, the choices of indi-
misinformation. But our own analysis of consumption vidual groups rapidly diverged. But the preferences
of low-quality information on Twitter shows that the of “nonsocial” groups, where no one knew about oth-
vulnerability applies to both sides of the political spec- ers’ choices, stayed relatively stable. In other words,
trum, and no one can fully avoid it. Even our ability social groups create a pressure toward conformity so
68 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022 Graphic by Filippo Menczer

powerful that it can overcome individual preferenc- homophily by allowing users to alter their social net-
es, and by amplifying random early differences, it can work structures through following, unfriending, and
cause segregated groups to diverge to extremes. so on. The result is that people become segregated
Social media follows a similar dynamic. We con- into large, dense and increasingly misinformed com-
fuse popularity with quality and end up copying the munities commonly described as echo chambers.
behavior we observe. Experiments on Twitter by At OSoMe, we explored the emergence of online
Bjarke Mønsted, then at the Technical University of echo chambers through another simulation, Echo
Denmark, and his colleagues indicate that informa- Demo. In this model, each agent has a political opin-
tion is transmitted via “complex contagion”: when we ion represented by a number ranging from −1 (say,
are repeatedly exposed to an idea, typically from liberal) to +1 (conservative). These inclinations are
many sources, we are more likely to adopt and re- reflected in agents’ posts. Agents are also influenced
share it. This social bias is further amplified by what by the opinions they see in their news feeds, and they
psychologists call the “mere exposure effect”: when can unfollow users with dissimilar opinions. Starting
people are repeatedly exposed to the same stimuli, with random initial networks and opinions, we found
such as certain faces, they grow to like those stimuli that the combination of social influence and unfol-
more than those they have encountered less often. lowing greatly accelerates the formation of polarized
Such biases translate into an irresistible urge to and segregated communities.
pay attention to information that is going viral—if ev- Indeed, the political echo chambers on Twitter are
erybody else is talking about it, it must be important. so extreme that individual users’ political leanings
In addition to showing us items that conform with can be predicted with high accuracy: you have the
our views, social media platforms such as Facebook, same opinions as the majority of your connections.
Twitter, YouTube and Instagram place popular con- This chambered structure efficiently spreads infor-
tent at the top of our screens and show us how many mation within a community while insulating that
people have liked and shared
something. Few people realize that
these cues do not provide indepen-
dent assessments of quality. Information that passes from person to
In fact, programmers who de-
sign the algorithms for ranking person along a chain becomes more negative
memes on social media assume
that the “wisdom of crowds” will and more resistant to correction.
quickly identify high-quality items;
they use popularity as a proxy for
quality. Our analysis of vast amounts of anonymous community from other groups. In 2014 our research
data about clicks shows that all platforms—social me- group was targeted by a disinformation campaign
dia, search engines and news sites—preferentially claiming that we were part of a politically motivated
serve up information from a narrow subset of popu- effort to suppress free speech. This false charge
lar sources. spread virally mostly in the conservative echo cham-
To understand why, we modeled how they combine ber, whereas debunking articles by fact-checkers were
signals for quality and popularity in their rankings. In found mainly in the liberal community. Sadly, such
this model, agents with limited attention—those who segregation of fake news items from their fact-check
see only a given number of items at the top of their reports is the norm.
news feeds—are also more likely to click on memes Social media can also increase negativity. In a 2018
ranked higher by the platform. Each item has intrin- laboratory study, Robert Jagiello, now at the Univer-
sic quality, as well as a level of popularity determined sity of Oxford, and one of us (Hills) found that social-
by how many times it has been clicked on. Another ly shared information not only bolsters biases but also
variable tracks the extent to which the ranking relies becomes more resilient to correction. We investigat-
on popularity rather than quality. Simulations of this ed how information is passed from person to person
model reveal that such algorithmic bias typically sup- in a so-called social diffusion chain. In the experiment,
presses the quality of memes even in the absence of the first person in the chain read a set of articles
human bias. Even when we want to share the best in- about either nuclear power or food additives. The ar-
formation, the algorithms end up misleading us. ticles were designed to be balanced, containing as
much positive information (for example, about less
ECHO CHAMBERS carbon pollution or longer-lasting food) as negative
M o s t o f us d o n o t b e l i e v e we follow the herd. information (such as risk of meltdown or possible
But our confirmation bias leads us to follow others harm to health).
who are like us, a dynamic that is sometimes referred The first person in the social diffusion chain told
to as homophily—a tendency for like-minded people the next person about the articles, the second told the
to connect with one another. Social media amplifies third, and so on. We observed an overall increase in

Vulnerability 100
Percent of Users’ Tweets That Share Links from Low-Credibility Sources

High
to Fake News
A study of Twitter users t hat rated
their political leanings found that 75
both liberals and conservatives
Risk of spreading misinformation

end up sharing information from
sites that repeatedly post news
of low credibility (as identified
by independent fact-checkers). 50
Conservative users are somewhat
more susceptible to sharing fake
news, however.
25
More than 15,000 Twitter users are plotted
on this matrix. The size of each dot
represents the number of accounts that
share that political bias/misinformation
Low
coordinate, ranging from one to 429. 0

1 429 Very liberal Center Very conservative
Political Bias of Twitter Users (inferred by set of news sources shared by user)
the amount of negative information as it passed along RISE OF THE BOTS

the chain—known as the social amplification of risk. I n f o r m at i o n q ua l i t y is further impaired by so-
Moreover, work by Danielle J. Navarro and her col- cial bots, which can exploit all our cognitive loop-
leagues at the University of New South Wales in Aus- holes. Bots are easy to create. Social media platforms
tralia found that information in social diffusion provide so-called application programming interfac-
chains is most susceptible to distortion by individu- es that make it fairly trivial for a single actor to set
als with the most extreme biases. up and control thousands of bots. But amplifying a
Even worse, social diffusion also makes negative message, even with just a few early upvotes by bots
information more “sticky.” When Jagiello and Hills on social media platforms such as Reddit, can have a
subsequently exposed people in the social diffusion huge impact on the subsequent popularity of a post.
chains to the original, balanced information—that is, At OSoMe, we have developed machine-learning
the news that the first person in the chain had seen— algorithms to detect social bots. One of these,
the balanced information did little to reduce individ- Botometer, is a public tool that extracts 1,200 fea-
uals’ negative attitudes. The information that had tures from a given Twitter account to characterize
passed through people not only had become more its profile, friends, social network structure, tempo-
negative but also was more resistant to updating. ral activity patterns, language and other features.
A 2015 study by Emilio Ferrara and Zeyao Yang, The program compares these characteristics with
then both OSoMe researchers, analyzed empirical those of tens of thousands of previously identified
data about such “emotional contagion” on Twitter and bots to give the Twitter account a score for its likely
found that people overexposed to negative content use of automation.
tend to share negative posts, whereas those overex- In 2017 we estimated that up to 15 percent of ac-
posed to positive content tend to share more positive tive Twitter accounts were bots—and that they had
Source: Dimitar Nikolov and Filippo Menczer (data)
posts. Because negative content spreads faster than played a key role in the spread of misinformation dur-
positive content, it is easy to manipulate emotions by ing the 2016 U.S. election period. Within seconds of
creating narratives that trigger negative responses a fake news article being posted—such as one claim-
such as fear and anxiety. Ferrara, now at the Univer- ing the Clinton campaign was involved in occult rit-
sity of Southern California, and his colleagues at the uals—it would be tweeted by many bots, and humans,
Bruno Kessler Foundation in Italy have shown that beguiled by the apparent popularity of the content,
during Spain’s 2017 referendum on Catalan indepen- would retweet it.
dence, social bots were leveraged to retweet violent Bots also influence us by pretending to represent
and inflammatory narratives, increasing their expo- people from our in-group. A bot only has to follow, like
sure and exacerbating social conflict. and retweet someone in an online community to
70 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022 Graphic by Jen Christiansen

quickly infiltrate it. Xiaodan Lou of Beijing Normal Apart from spreading fake news, misinformation
University, working with OSoMe, developed another campaigns can also divert attention from other, more
model in which some of the agents are bots that infil- serious problems. To combat such manipulation, we
trate a social network and share deceptively engaging developed a software tool called BotSlayer. It extracts
low-quality content—think of clickbait. One parame- hashtags, links, accounts and other features that co-
ter in the model describes the probability that an au- occur in tweets about topics a user wishes to study.
thentic agent will follow bots—which, for the purpos- For each entity, BotSlayer tracks the tweets, the ac-
es of this model, we define as agents that generate counts posting them and their bot scores to flag en-
memes of zero quality and retweet only one another. tities that are trending and probably being amplified
Our simulations show that these bots can effectively by bots or coordinated accounts. The goal is to enable
suppress the entire ecosystem’s information quality by reporters, civil-society organizations and political
infiltrating only a small fraction of the network. Bots candidates to spot and track inauthentic influence
can also accelerate the formation of echo chambers by campaigns in real time.
suggesting other inauthentic accounts to be followed, These programmatic tools are important aids,
a technique known as creating “follow trains.” but institutional changes are also necessary to curb
Some manipulators play both sides of a divide the proliferation of fake news. Education can help,
through separate fake news sites and bots, driving po- although it is unlikely to encompass all the topics
litical polarization or monetization by ads. At OSoMe, on which people are misled. Some governments and
we uncovered a network of inauthentic accounts on social media platforms are also trying to clamp
Twitter that were all coordinated by the same entity. down on online manipulation and fake news. But
Some pretended to be pro-Trump supporters of the who decides what is fake or manipulative and what
Make America Great Again U.S. election campaign, is not? Information can come with warning labels
whereas others posed as Trump “resisters”; all asked such as the ones Facebook and Twitter provide, but
for political donations. Such operations amplify con- can the people who apply those labels be trusted?
tent that preys on confirmation biases and accelerate The risk that such measures could either deliber-
the formation of polarized echo chambers. ately or inadvertently suppress free speech, which
is vital for robust democracies, is real. The domi-
CURBING ONLINE MANIPULATION nance of social media platforms with global reach
U n d e r s ta n d i n g o u r c o g n i t iv e biases and how and close ties with governments further compli-
algorithms and bots exploit them allows us to better cates the possibilities.
guard against manipulation. OSoMe has produced a One of the best ideas may be to make it more dif-
number of tools to help people understand their own ficult to create and share low-quality information.
vulnerabilities, as well as the weaknesses of social me- This could involve adding friction by forcing people
dia platforms. One is a mobile app called Fakey that to pay to share or receive information. Payment could
helps users learn how to spot misinformation. The be in the form of time, mental work such as puzzles,
game simulates a social media news feed, showing or microscopic fees for subscriptions or usage. Auto-
actual articles from low- and high-credibility sourc- mated posting should be treated like advertising.
es. Users must decide what they can or should not Some platforms are already using friction in the form
share and what to fact-check. Analysis of data from of C APTCHAs and phone confirmation to access ac-
Fakey confirms the prevalence of online social herd- counts. Twitter has placed limits on automated post-
ing: users are more likely to share low-credibility ar- ing. These efforts could be expanded to gradually shift
ticles when they believe that many other people have online sharing incentives toward information that is
shared them. valuable to consumers.
Another program available to the public, called Free communication is not free. By decreasing the
Hoaxy, shows how any extant meme spreads through cost of information, we have decreased its value and
Twitter. In this visualization, nodes represent actual invited its adulteration. To restore the health of our
Twitter accounts, and links depict how retweets, information ecosystem, we must understand the vul-
quotes, mentions and replies propagate the meme nerabilities of our overwhelmed minds and how the
from account to account. Each node has a color rep- economics of information can be leveraged to protect
resenting its score from Botometer, which allows us- us from being misled.
ers to see the scale at which bots amplify misinfor-
mation. These tools have been used by investigative Filippo Menczer is Distinguished Professor of Informatics and Computer
journalists to uncover the roots of misinformation Science and director of the Observatory on Social Media at Indiana
campaigns, such as one pushing the “pizzagate” con- University Bloomington. He studies the spread of disinformation and
spiracy in the U.S. They also helped to detect bot-driv- develops tools for countering social media manipulation.
en voter-suppression efforts during the 2018 U.S. mid-
term election. Manipulation is getting harder to spot, Thomas Hills i s a professor of psychology and director of the Behavioral
however, as machine-learning algorithms become and Data Science master’s program at the University of Warwick in
better at emulating human behavior. England. His research addresses the evolution of mind and information.

OPINION
How Facebook Hinders

Misinformation Research
The platform strictly limits and controls data access,
which stymies scientists
By Laura Edelson and Damon McCoy
W
h en th e wo r ld first he ard t hat touting this tool, however, Facebook limits
Russia had used Facebook ads in at the available data set to a three-month
time period leading up to the 2020 elec
tempts to interfere with the U.S.’s 2016 tions. To access the information, research
elections, computer scientists and cyber ers must agree to work in a closed envi
security experts heard a call to action. ronment, and they are not allowed to
download the data to share it with other
For the past four years we have been researchers. This means others are unable
studying how hate and disinformation spread online so that to replicate their findings, a core practice
independent researchers can build stronger defenses to protect in science that is essential to building con
fidence in results.
the public. But as we have tried to conduct this basic science, we Many scientists have a problem with
have met steep resistance from the primary platform we study: these limitations. Princeton University
Facebook. Our own accounts were shut down in 2021, another misinformation researchers have de
scribed problems with FORT that led
sign of the social media company’s rejection of scrutiny. them to scrap a project using the tool. Of
Facebook wants people to see it as mation has been called an “infodemic,” specific concern was a provision that Face
“the most transparent platform on the which, like a virus, grows, replicates and book had the right to review research be
Internet”—as its vice president of integri causes harm in the real world. Online fore publication. The researchers feared
ty said in August 2021. But in reality, it misinformation contributes to people’s this rule could be used to prevent them
has set up nearly insurmountable road hesitancy to wear masks or get vaccinated from sharing information about ad target
blocks for researchers seeking shareable, to help prevent the spread of C OVID-19. It ing in the 2020 elections.
independent sources of data. It’s true contributes to distrust in the soundness Second, Facebook aggressively moves
that Facebook does provide researchers of our election system. To reduce these to counter independent sources of data
with some data: It maintains a search harms, it is vital that researchers be able about its platform—and our team is a
able online ad library and allows autho to access and share data about social me good example. In 2020 we built a tool we
rized users to download limited informa dia behavior and the algorithms that call Ad Observer, a citizen science brows
tion about political ads. Researchers shape it. But Facebook’s restrictions are er extension that allows consenting users
have also been able to use Facebook’s getting in the way of this science. to share with us limited and anonymous
business analytics tools to glean some in First, Facebook limits which research information about the ads that Facebook
formation about the popularity of unpaid ers are permitted to receive platform data shows them. The extension communi
content. But the platform not only sharp and requires them to sign agreements cates with our project Cybersecurity for
ly limits access to these tools, it also ag that severely curtail how they access it, as Democracy, sending basic information,
gressively moves to shut down indepen well as how they share it. One example of such as who paid for the ad and how long
dent efforts to collect data. this problem is the FORT (Facebook Open it ran. It also reports how advertisers tar
This is not just a spat between a social Research and Transparency) program, get the ads, an issue that researchers and
media platform and the people who study which Facebook created for researchers journalists have exposed as a vector in
it. The proliferation of online misinfor to study ad-targeting data. Despite widely the spread of misinformation. For ethical

reasons, we do not collect personal infor uated that its actions were mandated by we think Facebook should reinstate our
mation about people who share the ads an agreement it entered into with the accounts and stop threatening other legit
they see. And for scientific reasons, we do Federal Trade Commission regarding imate researchers. In the long term, how
not need to—everything we need to know user privacy. The FTC responded swiftly, ever, scientists cannot rely on limited vol
to answer our research questions is con telling Facebook that the platform is untary transparency measures from the
tained in public information that we are wrong to block our research in the name platforms we follow. Researchers and jour
gathering with consent. of its agreement with the agency: “The nalists who study social media platforms
Even these limited data about ads consent decree does not bar Facebook in a privacy-shielding way need better le
have been tremendously helpful for our from creating exceptions for good-faith gal protections so that companies such as
research, and the project demonstrates research in the public interest. Indeed, Facebook are not the ones deciding what
the necessity of independent auditing of the FTC supports efforts to shed light on research can go forward. Numerous pro
social media platforms. With the data opaque business practices, especially posals have been brought before the U.S.
collected by our volunteers, we were able around surveillance-based advertising.” Congress and the European Union on how
to identify ads promoting the conspiracy Facebook has not backed down from to strengthen these protections. Now it’s
theory Q Anon and far-right militias, as its decision to suspend our accounts, and time for lawmakers to take action.
well as demonstrate that Facebook failed it has placed other researchers in the
to identify approximately 10 percent of crosshairs. In August 2021 the Germany- Laura Edelson i s a postdoctoral researcher
political ads that ran on its platform. And based project AlgorithmWatch an at New York University.
we have published our data so other re nounced that it had discontinued a proj
searchers can work with them, too. ect that used crowdsourced data to moni Damon McCoy is an associate professor of computer
In response, Facebook shut down our tor how Instagram (a platform also science and engineering at the New York University
personal accounts in August 2021. This owned by Face book) treated political Tandon School of Engineering. He received his Ph.D.,
Malte Mueller/Getty Images
prevented Cybersecurity for Democracy posts and other content. In a statement, M.S. and B.S. in computer science from the University
from accessing even the limited trans AlgorithmWatch noted that Facebook had of Colorado Boulder. McCoy is the recipient of a National
parency information the platform pro cited privacy concerns with its research. Science Foundation CAREER award, and he is a former
vides to researchers. The company insin So where do we go from here? Of course, CRA/CCC Computing Innovation Fellow.

THE
SHARED
PAST
THAT
WASN’T
How Facebook, fake
news and friends are
altering memories
and changing history
By Laura Spinney
Illustration by
Taylor Callery


S trange things have happened in the media in

recent years. In 2017 members of the Trump admin
istration alluded to a “Bowling Green massacre” and
terror attacks in Sweden and Atlanta that never
happened. The misinformation was swiftly corrected,
but some historical myths have proved difficult to
erase. Since at least 2010, for example, an online
community has shared the apparently unshakable
recollection of Nelson Mandela dying in prison in the
1980s, despite the fact that he lived until 2013, leaving
prison in 1990 and going on to serve as South Africa’s first Black president.
Memory is notoriously fallible, but some experts
worry that a new phenomenon is emerging. “Memo
ries are shared among groups in novel ways through
sites such as Facebook and Instagram, blurring the
line between individual and collective memories,” says
psychologist Daniel Schacter, who studies memory at
Harvard University. “The development of Internet-
based misinformation, such as well-publicized fake
news sites, has the potential to distort individual and
collective memories in disturbing ways.”
Collective memories form the basis of history, and
even if it is wrong. Not all the findings are gloomy,
however. Research is pointing to ways of dislodging
false memories or preventing them from forming in
the first place.
To combat the influence of fake news, says Micah
Edelson, a memory researcher at the University of
Zurich in Switzerland, “it’s important to understand
not only the creation of these sites but also how peo
ple respond to them.”
ALL TOGETHER NOW

people’s understanding of history shapes how they C o m m u n i c at i o n s h a p e s m e m o ry. R
esearch on
think about the future. The fictitious terrorist pairs of people conversing about the past shows that
attacks, for example, were cited to justify a travel ban a speaker can reinforce aspects of an event by selec
on the citizens of seven “countries of concern.” tively repeating them. That makes sense. Things that
Although history has frequently been interpreted for get mentioned get remembered—by both speaker
political ends, psychologists are now investigating and listener. There is a less obvious corollary: related
the fundamental processes by which collective mem information that goes unmentioned is more likely to
ories form, to understand what makes them vulner fade than unrelated material, an effect known as
able to distortion. They show that social networks retrieval-induced forgetting.
powerfully shape memory and that people need lit These cognitive, individual-level phenomena have
tle prompting to conform to a majority recollection— been proposed as a mechanism for memory conver

gence—the process by which two or more people information collaboratively. Finally, they tried to
come to agree on what happened. But clues have recall the events individually again.
emerged that group-level forces influence conver The researchers investigated two scenarios—one
gence, too. In 2015 psychologists Alin Coman of in which the group formed two subclusters, with
Princeton University and William Hirst of the New almost all conversations taking place within the sub
School for Social Research reported that a person clusters, and one in which it formed one large cluster
experiences more induced forgetting when listening [see box on next page]. Although people in the single
to someone in their own social group—a student at cluster agreed on the same set of information, Coman
the same university, for example—than if they see that says, those in the two subclusters generally converged
person as an outsider. That is, memory convergence is on different “facts” about the fictional volunteers.
more likely to occur within social groups than be This effect is evident in real-world situations. Pal
tween them—an important finding in light of survey estinians living in Israel and those in the West Bank,
data suggesting that 62 percent of U.S.
adults get their news from social media,
where group membership is often obvi
ous and reinforced.
“MEMORIES ARE SHARED AMONG
Groups can also distort memories. In GROUPS IN NOVEL WAYS THROUGH
SITES SUCH AS FACEBOOK AND
2011 Edelson, then at the Weizmann
Institute of Science in Rehovot, Israel,
INSTAGRAM, BLURRING THE LINE

showed 30 volunteers a documentary.
They watched the film in groups of five
and, a few days later, answered ques
tions about it individually. One week BETWEEN INDIVIDUAL AND
after the viewing session, participants
answered questions again—but only COLLECTIVE MEMORIES.”
after seeing answers that members of
their group had supposedly given. When
—DANIEL SCHACTER HARVARD UNIVERSITY
most of the fabricated responses were
false, participants conformed to the same false who were separated by force during the Arab-Israeli
answer about 70 percent of the time—despite having wars of 1948 and 1967, have gravitated to different
initially responded correctly. But when they learned versions of their past, despite a shared Arab-Palestin
that the answers had been generated randomly, the ian identity. Similarly divergent truths emerged after
participants reversed their incorrect answers only the erection of the Berlin Wall.
about 60 percent of the time. “We found that pro In the lab, Coman can manipulate social networks
cesses that happen during initial exposure to errone and look at the memories that form. His comparison
ous information make it more difficult to correct such of the two scenarios revealed the importance of
influences later,” Edelson says. “weak links” in information propagation. These are
Studying those processes as they happen—as col links between, rather than within, networks—ac
lective memories are shaped through conversation— quaintances, say, rather than friends—and they help
has been difficult to do in large groups. Years ago to synchronize the versions held by separate net
monitoring communication in groups of 10 or more works. “They are probably what drives the formation
would have required several rooms for private con of community-wide collective memories,” he says.
versations, many research assistants and lots of time. One function of those weak links might be to re
Now multiple participants can interact digitally in mind people of information expunged through the
real time. Coman’s group has developed a software processes of memory convergence. But timing is key.
platform that can track exchanges between volun Coman has shown that information introduced by a
teers in a series of timed chats. “It takes one research weak link is much more likely to shape the network’s
assistant 20 minutes and one lab room,” Coman says. memory if it is introduced before its members talk
In 2016 the group used this software to ask, for among themselves. Once a network agrees on what
the first time, how the structure of social networks happened, collective memory becomes relatively
affects the formation of collective memories in large resistant to competing information.
groups. The researchers fed information about four Coman thinks that memory convergence bolsters
fictional Peace Corps volunteers to 140 participants group cohesion. “Now that we share a memory, we
from Princeton University, divided into groups of 10. can have a stronger identity and might care more
First, the participants were asked to recall as much about each other,” he says. Abundant research links
information as they could on their own. Then they strong group identity with higher reported individ
took part in a series of three conversations—online ual well-being. This is shown by research on the fam
chat sessions lasting a few minutes each—with other ily. At Emory University, psychologist Robyn Fivush
members of their group, in which they recalled the is studying the stories that families tell themselves.

versions of the past. These may be preserved for

posterity in statues and history books. But they
Hello, Operator can evolve over time.
In an experiment, 1 0 volunteers discussed the details of a story in one- MAKING MEMORIES, MAKING HISTORIES
on-one chats with three other people. The distance between participants In Ostend, Bel gium, a public monument de
in the network (degrees of separation) correlated to how well their picts King Leopold II, surrounded by two
recollections of the story aligned with one another. groups of grateful subjects—one Belgian, the
other Congolese. In 2004 protesters who felt
Each person participated in three out of four that the monument misrepresented history sev
chat rounds (numbered) in a specified order. ered the bronze hand of one of the Congolese
figures. They explained anonymously to a local
1 1
newspaper that the amputation more accurately
reflected Leopold’s role in Belgium’s African col
ony: not genial protector but brutal tyrant.
2 2 2 2
In 2010 social psychologists Laurent Licata
4 1 4 and Olivier Klein of the Free University of Brus
sels carried out a survey to explore different
3 3 3 3
generations’ attitudes toward Belgium’s colonial
Weak links between clusters past. They found that Belgian students ex
1 can help synchronize memories 1 pressed higher levels of collective guilt and sup
between groups. port for reparative actions toward what is now
the Democratic Republic of the Congo than did
their parents, who in turn expressed higher lev
CLUSTERED els than their parents. An important factor
With as many as five degrees of separation between some participants, shaping that evolution, the researchers suggest,
memories converged within clusters but less so between them.
was Adam Hochschild’s influential book K ing
Leopold’s Ghost (Houghton Mifflin, 1998), which
painted a much darker picture of the colonial
4 period than had been accepted previously.
“Those who were young when that book came
1 1
out were particularly marked by it,” Licata says,
“whereas older Belgians had grown up with a
different set of facts.”
3 2 2 2
ature; S ource: “Mnemonic Convergence in Social Networks: The Emergent Properties of Cognition at a Collective Level,” by A. Coman,
Not all collective memories pass into history.
1 Cognitive psychologists Norman Brown of the
I. Momennejad, R. D. Drach and A. Geana, in P roceedings of the National Academy of Sciences USA, Vol. 113, No. 29; July 19, 2016
2 3 3 3
University of Alberta and Connie Svob of
Columbia University have proposed that some
thing besides cognitive and social processes
1 1 determines whether an event survives the tran
sition across generations: the nature of the
4 event itself. “It is the amount of change to a per
son’s fabric of daily life that is most crucially at
UNCLUSTERED stake,” Svob says.
With no more than three degrees of separation, there was more alignment
In a study published in 2016, they reported
in the memories of any two members.
that the children of Croatians who had lived
through the Yugoslav wars of the 1990s were
more likely to recall their parents’ war-related
experiences—getting shot, for example, or the
“What we find is that adolescents and young adults house being bombed—than their non-war-related
who know more family stories show better psycho ones, such as marriage or birth of a first child. Wars,
logical well-being,” she says. as with immigration, bring great upheaval in their
Although shared memories may foster more wake and so are highly memorable, Svob says.
closely knit groups, they can also distort the role of This “transition theory” she says, could also ex
outsiders, driving a wedge between groups. Memory plain one of the biggest voids in Westerners’ collec
shapes group identity, which in turn shapes memory, tive memory of the 20th century—why they easily
in a potentially vicious cycle. Weak links have an recall the two World Wars but not the flu pandemic of
important corrective effect, but in their absence, two 1918–1920 that killed more than either of them (con
groups might converge on mutually incompatible sidering unrecorded deaths from the disease). “The
N

degree of change wrought by war tends to be greater confidence in an inaccurate memory—and, ulti
than the degree of change wrought by a pandemic,” mately, with whether they reverted to their initial,
Svob says. Others find that explanation puzzling: “If accurate one. “By exposing them to the fact that this
you lost a loved one in the flu epidemic,” Fivush says, information is not credible, in most cases, individu
“then it certainly disrupted your daily life.” als will take that into account,” Edelson says. “In
The set of collective memories that a group holds 60 percent of cases, they will flip their answer. But
clearly evolves over time. One reason for this is that even if they maintain a wrong answer, they’ll be less
people tend to be marked most by events in their confident about it.”
adolescence or young adulthood—a
phenomenon known as the reminis
cence bump. As a new generation grows “IF YOU UNDERSTAND THE NATURE
OF THE FALSE INFORMATION, YOU
up, events that happen to its members
during their youth override the events
that previously dominated society and
thus “update” the collective memory. A CAN TARGET IT FOR SUPPRESSION.”
2016 survey by the Pew Research Center
in Washington, D.C., showed that the —ALIN COMAN PRINCETON UNIVERSITY
defining historical moments for baby
boomers in the U.S. were the terror attacks on Sep Coman has two suggestions from his findings.
tember 11, 2001, and the assassination of President The first is directed at the justice system. In some
John F. Kennedy. For those born since 1965, they U.S. states, jurors are forbidden to take notes made
were the attacks on 9/11 and the 2008 election of during a trial into the deliberation room—a legacy
President Barack Obama. of historically high illiteracy rates and a belief that
And over time each generation adds some events the group remembers more reliably than the individ
and forgets others. Psychologists Henry Roediger of ual. In fact, Coman says, using notes could protect
Washington University in St. Louis and Andrew jurors from retrieval-induced biases and group-level
DeSoto of the Association for Psychological Science social influences.
in Washington, D.C., report, for example, that succes His second suggestion concerns the diffusion of
sive U.S. generations forget their past presidents in a crucial information to the public during emergencies
regular manner that can be described by a power such as epidemics. Having observed that retrieval-
function. They predict that Harry Truman (1945– induced forgetting is enhanced in high-anxiety situ
1953) will be as forgotten by 2040 as William McKin ations, Coman has come up with some advice for offi
ley (1897–1901) is today. cials: draw up a short but comprehensive list of key
That evolution is reflected by evolving attitudes points, make sure that all officials have the same list,
toward the future. Roediger and anthropologist repeat those points often and keep tabs on bad infor
James Wertsch, also at Washington University, have mation that enters circulation. During the 2014
observed that U.S. politicians debating the invasion Ebola outbreak, for example, concerns in the U.S.
of Iraq in the early 2000s fell into two groups: those were stoked by a misconception that being in the
who advocated invasion on the grounds that Sad same room as a person with the infection was
dam Hussein had to be stopped like Adolf Hitler enough to catch it. The best way to kill that rumor,
before him and those who opposed it because they Coman says, would have been to explain—often—
feared another bloody, protracted Vietnam War. that Ebola can be transmitted only through bodily
Although each might have chosen their historical fluids. “If you understand the nature of the false in
precedent for political reasons, they in turn rein formation, you can target it for suppression just by
forced that precedent in the memory of anyone who mentioning information that is conceptually related
heard them speak. but accurate,” he says.
Collective memory is a double-edged sword.
SPOTTING THE FAKE Some will no doubt use it to mislead. “The fact that
Research into collec tive memory has pointed information can freely circulate in the community
to ways that it might be shaped for the collective has been considered one of the most important and
good. Edelson and his team gave grounds for opti constructive features of open and democratic societ
mism when, in a 2014 follow-up to their earlier study, ies,” Coman says. “But creating such societies does
they reported that although some false memories are not inherently guarantee positive outcomes.” False
resistant to change, the people who hold them can collective memories might be the price of defending
nonetheless be influenced by credible information. free speech. But understanding how they form might
The team used functional magnetic resonance imag offer some protection the next time people are re
ing to scan volunteers’ brains as they recalled infor minded about a massacre that never happened.
mation about a film. The scans revealed changes in
brain activation that correlated with the degree of Laura Spinney i s a science journalist based in Paris.

OPINION
The Black Box

of Social Media
Social media companies need to give their data
to independent researchers to better understand
how to keep users safe
By Renée DiResta, Laura Edelson, Brendan Nyhan and Ethan Zuckerman
Social media platforms are where billions of people Unfortunately, social media companies are stingy about re
around the globe go to connect with others, get information leasing data and publishing research, especially when the find-
and make sense of the world. The companies that run these ings might be unwelcome (although notable exceptions exist).
sites, including Facebook, Twitter, Instagram, Tiktok and Red- The only way to understand what is happening on the platforms
dit, collect vast amounts of data based on every interaction that is for lawmakers and regulators to require social media compa-
takes place on their platforms. nies to release data to independent researchers. In particular, we
And despite the fact that social media has become one of our need access to data on the structures of social media, such as
most important public forums for speech, several of the most platform features and algorithms, so we can better analyze how
important platforms are controlled by a small number of peo- they shape the spread of information and affect user behavior.
ple. Mark Zuckerberg controls 58 percent of the voting share of For example, platforms have assured legislators that they
Meta, the parent company of both Facebook and Instagram, are taking steps to counter misinformation and disinformation
effectively giving him sole control of two of the largest social by flagging content and inserting fact-checks. Are these efforts
platforms. Elon Musk made a $44-billion offer to take Twitter effective? Again, we would need access to data to know. Without
private (although whether that deal goes through will be deter- better data, we can’t have a substantive discussion about which
mined by a lawsuit). All these companies have a history of shar- interventions are most effective and consistent with our values.
ing scant portions of the data about their platforms with re We also run the risk of creating new laws and regulations that
searchers, preventing us from understanding the impacts of do not adequately address harms or of inadvertently making
social media on individuals and society. Such singular owner- problems worse.
ship of the three most powerful social media platforms makes Some of us have consulted with lawmakers in the U.S. and
us fear this lockdown on data sharing will continue. Europe on potential legislative reforms along these lines. The
After decades of little regulation, it is time to require more conversation around transparency and accountability for social
transparency from social media companies. media companies has grown deeper and more substantive,
In 2020 social media was an important mechanism for the moving from vague generalities to specific proposals. The de
spread of false and misleading claims about the election and for bate still lacks important context, however. Lawmakers and
mobilization by groups that participated in the January 6, 2021, regulators frequently ask us to better explain why we need ac
Capitol insurrection. We have seen misinformation about COVID cess to data, what research it would enable, and how that re
spread widely online during the pandemic. And today social search would help the public and inform regulation of social
media companies are failing to remove the Russian propaganda media platforms.
about the war in Ukraine that they promised to ban. Social media To address this need, we’ve created this list of questions we
has become a major conduit for the spread of false information could answer if social media companies began to share more of
about every issue of concern to society. We don’t know what the the data they gather about how their services function and how
next crisis will be, but we do know that false claims about it will users interact with their systems. We believe such research
circulate on these platforms. would help platforms develop better, safer systems and also

inform lawmakers and regulators who seek to hold platforms ied. Do other algorithms used by other social media platforms
accountable for the promises they make to the public. show systemic political bias as well?
● Research suggests that misinformation is often more engag- ● Because of the central role they now play in public discourse,
ing than other types of content. Why is this the case? What fea- platforms have a great deal of power over who can speak. Minor-
tures of misinformation are most associated with heightened ity groups sometimes feel their views are silenced online as a
user engagement and virality? Researchers have proposed that consequence of platform moderation decisions. Do decisions
novelty and emotionality are key factors, but we need more about what content is allowed on a platform affect some groups
research to know whether this is true. A better understanding of disproportionately? Are platforms allowing some users to silence
why m isinformation is so engaging will help platforms improve others through the misuse of moderation tools or through sys-
their algorithms and recommend misinformation less often. temic harassment designed to silence certain viewpoints?
● Research shows that the delivery-optimization techniques Social media companies ought to welcome the help of inde-
companies use to maximize revenue, and even the ad-delivery pendent researchers to better measure online harm and inform
algorithms themselves, can be discriminatory. Are some groups policies. Some companies, such as Twitter and Reddit, have
of users significantly more likely than others to see potentially been helpful, but we can’t depend on the goodwill of a few busi-
harmful ads, such as consumer scams? Are others less likely to be nesses whose policies might change at the whim of a new own-
shown useful ads, such as job postings? How can ad networks er. We hope a potentially Musk-led Twitter would be as forth-
improve delivery and optimization to be less discriminatory? coming as before, if not more so. In our fast-changing informa-
● Social media companies attempt to combat misinformation tion environment, we should not regulate and legislate by
by labeling content of questionable provenance, hoping to push anecdote. We need lawmakers to ensure our access to the data
users toward more accurate information. Results from survey we need to help keep users safe.
experiments show that the effects of labels on beliefs and behav-
ior are mixed. We need to learn more about whether labels are Renée DiResta is the technical research manager at Stanford Internet Observatory.
effective when individuals encounter them on platforms. Do
Laura Edelson i s a postdoctoral researcher at New York University.
labels reduce the spread of misinformation or attract attention
to posts that users might otherwise ignore? Do people start to Brendan Nyhan i s James O. Freedman Presidential Professor of Government
dem10/Getty Images
ignore labels as they become more familiar? at Dartmouth College.

● Internal studies at Twitter show that Twitter’s algorithms
amplify right-leaning politicians and political news sources Ethan Zuckerman is an associate professor of public policy, information and
more than left-leaning accounts in six of seven countries stud- communication at the University of Massachusetts Amherst.

POLITICS
ARGUING
THE TRUTH
As political polarization grows, the arguments we have with
one another may be shifting our understanding of truth itself
By Matthew Fisher, Joshua Knobe, Brent Strickland and Frank C. Keil
Illustration by Hanna Barczyk
I n a k e y mo m e nt o f t h e f i na l 2 01 6 T ru m p- C l i n t o n p r e s i d e n t i a l d e bat e ,
Donald Trump turned to a question regarding Russian president Vladimir Putin:
“He has no respect for her,” Trump said, pointing at Hillary Clinton.
“Putin, from everything I see, has no respect for this person.”
The two debaters then drilled down to try to gain a more nuanced
understanding of the difficult policy issues involved. Clinton said,
“Are you suggesting that the aggressive approach I propose

would actually fail to deter Russian expansionism?”
To which Trump responded,
“No, I certainly agree that it would deter
Russian expansionism; it’s just that it
would also serve to destabilize the . ..”
Just kidding. That’s not at all what happened. Actually each side
aimed to attack and defeat the other. Clinton really said,
“Well, that’s because he’d rather have a

puppet as president of the United States.”
To which Trump retorted,
“You’re the puppet!”

POLITICS
Episodes like this one have become such a staple of con- together on a problem and find themselves disagreeing about
temporary political discourse that it is easy to forget how radi- the solution:
cally different they are from disputes we often have in ordi-
nary life. Consider a couple of friends trying to decide on a Mary: The cube root of 2,197 is 13.
restaurant for dinner. One might say, “Let’s try the new Indian Susan: No, the cube root of 2,197 is 14.
restaurant tonight. I haven’t had Indian for months.” To which
another replies, “You know, I saw that place is getting poor People observing this conflict might not know which ans
reviews. Let’s grab some pizza instead?” “Good to know—pizza wer is correct. Yet they might be entirely sure that there is a
it is,” says the first. Each comes in with an opinion. They begin single objectively correct answer. This is not just a matter of
a discussion in which each presents an argument, then listens opinion—there is a fact of the matter, and anyone who has an
to the other’s argument, and then they both move toward an alternative view is simply mistaken.
agreement. This kind of dialogue happens all the time. In our Now consider a different kind of scenario. Suppose these
research, which involves cognitive psychology and experimen- two friends decide to take a break for lunch and find them-
tal philosophy, we refer to it as “arguing to learn.” selves disagreeing about what to put on their bagels:
But as political polarization increases in the U.S., the kind
of antagonistic exchange exemplified by the Trump-Clinton Mary: Veggie cream cheese is really tasty.
debate is occurring with increasing frequency—not just among Susan: N
o, veggie cream cheese is not tasty at all.
policy makers but among us all. In interactions such as these, It is completely disgusting.
people may provide arguments for their views, but neither
side is genuinely interested in learning from the other. Instead In this example, observers might take up another attitude:
the real aim is to “score points,” in other words, to defeat the Even if two people have opposite opinions, it could be that
other side in a competitive activity. Conversations on Twitter, neither is incorrect. It seems that there is no objective truth of
Facebook and even YouTube comment sections have become the matter.
powerful symbols of what the combativeness of political dis- With that in mind, think about what happens when people
course looks like these days. We refer to this kind of discussion debate controversial questions about morally infused political
as “arguing to win.” topics. As our two friends are enjoying their lunch, suppose
The divergence of Americans’ ideology is accompanied by they wade into a heated political chat:
an animosity for those across the aisle. Polls have shown that
partisan liberals and conservatives associate with one another Mary: Abortion is morally wrong and should not be legal.
less frequently, have unfavorable views of the opposing party, Susan: N
o, there is nothing wrong with abortion, and it
and would even be unhappy if a family member married some- should be perfectly legal.
one from the other side. At the same time, the rise of social
media has revolutionized how information is consumed—news The question we grapple with is how to understand this
is often personalized to one’s political preferences. Rival per- kind of debate. Is it like the math question, where there is an
spectives can be completely shut out from one’s self-created objectively right answer and anyone who says otherwise must
media bubble. Making matters worse, outrage-inducing con- be mistaken? Or is it more like a clash over a matter of taste,
tent is more likely to spread on these platforms, creating a where there is no single right answer and people can have
breeding ground for clickbait headlines and fake news. This opposite opinions without either one being wrong?
toxic online environment is very likely driving Americans fur- In recent years work on this topic has expanded beyond
ther apart and fostering unproductive exchanges. the realm of philosophy and into psychology and cognitive
In this time of rising polarization, an important question science. Instead of relying on the intuitions of professional
has arisen about the psychological effects of arguing to win. philosophers, researchers like us have begun gathering empir-
What happens in our minds—and to our minds—when we find ical evidence to understand how people actually think about
ourselves conversing in a way that simply aims to defeat an these issues. Do people tend to think moral and political ques-
opponent? Our research has explored this question using tions have objectively correct answers? Or do they have a
experimental methods, and we have found that the distinction more relativist view?
between different modes of argument has some surprisingly On the most basic level, the past decade or so of research
far-reaching effects. Not only does it change people’s way of has shown that the answer to this question is that it’s compli-
thinking about the debate and the people on the opposing cated. Some people are more objectivist; others are more rela-
side, but it also has a more fundamental effect on our way of tivist. That might seem obvious, but later studies explored the
understanding the very issue under discussion. differences between people with these types of thinking. When
participants are asked whether they would be willing to share
ARE WE OBJECTIVISTS OR RELATIVISTS? an apartment with a roommate who holds opposing views on
The question of moral and political objectivity is notor moral or political questions, objectivists are more inclined to
iously thorny, one that philosophers have been debating say no. When participants are asked to sit down in a room next
for millennia. Still, the core of the question is easy enough to to a person who has opposing views, objectivists actually sit
grasp by considering a few hypothetical conversations. Con- farther away. As University of Pennsylvania psychologist Geof-
sider a debate about a perfectly straightforward question in frey P. Goodwin once put it, people who hold an objectivist
science or mathematics. Suppose two friends are working view tend to respond in a more “closed” fashion.

POLITICS
Why might this be? One straightforward possibility is that truth about the topics they had just debated. Strikingly, these
if you think there is an objectively correct answer, you may be 15-minute exchanges actually shifted people’s views. Individu-
drawn to conclude that everyone who holds the opposite view als were more objectivist after arguing to win than they were
is simply incorrect and therefore not worth listening to. Thus, after arguing to learn. In other words, the social context of the
people’s view about objective moral truths could shape their discussion—how people frame the purpose of controversial
approach to interacting with others. This is a plausible hypodiscourse—actually changed their opinions on the deeply
thesis and one worth investigating in further studies. philosophical question about whether there is an objective
Yet we thought that there might be more to the story. In truth at all.
particular, we suspected there might be an effect in the oppo- These results naturally lead to another question that goes
site direction. Perhaps it’s not just that having objectivist beyond what can be addressed through a scientific study.
views shapes your interactions with other people; perhaps Which of these two modes of argument would be better to
your interactions with other people can actually shape the adopt when it comes to controversial political topics? At first,
degree to which you hold objectivist views. the answer seems straightforward. Who could fail to see that
there is something deeply important about cooperative dia-
WINNING VS. LEARNING logue and something fundamentally counterproductive about
To test this theory, we ran an experiment in which adults sheer competition?
engaged in an online political conversation. Each participant Although this simple answer may be right most of the time,
logged on to a website and indicated their positions on a vari- there may also be cases in which things are not quite so clear-
ety of controversial political topics, including abortion and cut. Suppose we are engaged in a debate with a group of cli-
gun rights. They were matched with another participant who mate science skeptics. We could try to sit down together, listen
held opposing views. The participants then engaged in an to the arguments of the skeptics and do our best to learn from
online conversation about a topic on which they disagreed. everything they have to say. But some might think that this
Half of the participants were encouraged to argue to win. approach is exactly the wrong one. There might not be any-
They were told that this would be a highly competitive ex thing to be gained by remaining open to ideas that contradict
change and that their goal should be to outperform the other scientific consensus. Indeed, agreeing to partake in a coopera-
person. The result was exactly the kind of communication one tive dialogue might be an instance of what journalists call
sees every day on social media. Here, for example, is a tran- “false balance”—legitimizing an extreme outlier position that
script from one of the actual conversations: should not be weighed equally. Some would say that the best
approach in this kind of case is to argue to win.
P1: I believe 100 percent in a woman’s choice. Of course, our studies cannot directly determine which
P2: A bortion should be prohibited because it stops mode of argument is “best.” And although plenty of evidence
a beating heart. suggests that contemporary political discourse is becoming
P1: Abortion is the law of the land, the land you live in. more combative and focused on winning, our findings do not
P2: The heart beats at 21 days its murder [sic]. elucidate w hy t hat change has occurred. Rather they provide
an important new piece of information to consider: the mode
The other half of participants were encouraged to argue to of argument we engage in actually changes our understand-
learn. They were told that this would be a very cooperative ing of the question itself.
exchange and that they should try to learn as much as they The more we argue to win, the more we will feel that there
could from their opponent. These conversations tended to is a single objectively correct answer and that all other
have a quite different tone: answers are mistaken. Conversely, the more we argue to learn,
the more we will feel that there is no single objective truth
P3: I believe abortion is a right all women should possess. and different answers can be equally right.
I do understand that some people choose to place So the next time you are deciding how to enter into an
certain determinants on when and why, but I think argument on Facebook about the controversial question of
it should be for any reason before a certain time point the day, remember that you are not just making a choice about
in the pregnancy agreed upon by doctors, so as not how to interact with a person who holds the opposing view.
to harm the mother. You are also making a decision that will shape the way you—
P4: I believe that life begins at conception (sperm meeting and others—think about whether the question itself has a cor-
egg), so abortion to me is the equivalent of murder. rect answer.
P3: I can absolutely see that point. As a biologist, it
is obvious from the first cell division that “life” is Matthew Fisher i s an assistant professor of psychology at Southern Methodist University.
happening. But I do not think life is advanced enough
to warrant abolishing abortion. Joshua Knobe is a professor at Yale University, appointed both in the program in
cognitive science and in the department of philosophy.
It is not all that surprising that these two sets of instruc-
Brent Strickland is a researcher in cognitive science at the Jean Nicod Institute
tions led to such results. But would these exchanges in turn
in Paris.
lead to different views about the very nature of the question
being discussed? After the conversation was over, we asked Frank C. Keil is Charles C. and Dorathea S. Dilley Professor of Psychology and a professor
participants whether they thought there was an objective of linguistics and cognitive science at Yale University.

POLITICS
Post-Truth: A Guide
for the Perplexed
If politicians can lie without condemnation,
what are scientists to do?
By Kathleen Higgins
T
Illustration by Hannah Salyer
h e Oxf ord Di c ti onarie s name d “ post-t rut h ” as t he ir 2016 Wo r d

of the Year. It must have sounded alien to scientists at the time. Science’s
quest for knowledge about reality presupposes the importance of truth, both
as an end in itself and as a means of resolving problems. How could truth
become passé? For philosophers like me, post-truth also goes against the
grain. But in the wake of the 2016 U.S. presidential election and all that has
happened since, author Ralph Keyes’s 2004 declaration that we have arrived
in a post-truth era seems distressingly plausible.
Post-truth refers to blatant lies being U.S. president Richard Nixon’s lies was he does suggest at times that deception is
routine across society, and it means that greeted with outrage. rife and should not be categorically re-
politicians can lie without condemnation. One might be tempted to blame phi- jected. His point is to complicate our view
This is different from the cliché that all losophy for post-truth. Some of us write of human behavior and to object to moral
politicians lie and make promises they about epistemic relativism, the view that certainties that encourage black-and-
have no intention of keeping—that notion truth can vary depending on the context. white judgments about what’s good and
still assumes honesty is the default posi- Yet relativism is itself relative. An extreme what’s evil. Thus, he denies that there are
tion. In a post-truth world, this ex relativist might hold that the truth varies moral facts, saying that we have only
pectation no longer holds. from person to person, a position that “moral interpretations,” and in doing so
This can explain the current political does not leave much room for debate. But he denies that moral assertions are uncon-
situation in the U.S. and elsewhere. Public more rational positions can also involve ditionally true. But this does not mean
tolerance of inaccurate and undefended al- at least a modicum of relativism. In a there is no truth. Even when he claims
legations, non sequiturs in response to sense, even 18th-century philosopher Im that our truths amount to our “irrefutable
hard questions and outright denials of facts manuel Kant’s quite sensible contention errors,” he is pointing to the exaggerated
are shockingly high. Repetition of talking that we can never know what things are clarity of abstractions by comparison with
points passes for political discussion, and like “in themselves”—independent of how empirical reality.
serious interest in issues and options is our minds format what we perceive—is a In fact—contrary to how he is often
treated as the idiosyncrasy of wonks. The relativistic position. It implies that what presented—Nietzsche held intellectual
lack of public indignation when political is true of the world for humans is proba- honesty at a premium. His most strenu-
figures claim disbelief in response to grow- bly different from what is true for a fly. En- ous rejections of “truth” are mostly di
ing scientific evidence of the reality of cli- tomologists would surely agree. rected not at truth but at what has been
mate change is part of this larger pattern. More radical forms of relativism are of- asserted as true. Yes, Nietzsche was an
“Don’t bother me with facts” is no longer a ten denounced as undermining basic val- elitist who was skeptical of democracy,
punchline. It has become a political stance. ues. Friedrich Nietzsche, the 19th-century and so his work does not necessarily fault
It’s worth remembering that it has not al- philosopher who is often invoked to jus- leaders for talking down to the public. But
ways been this way: the exposure of former tify post-truth, was such a relativist, and it also points out the inconsistency of re-

ligious teachers who assume they have the they agree with. But contemptuous lead- keep reminding society of the importance
right to lie. ers and voters who are content with hand- of the social mission of science—to pro-
When political leaders make no effort waving and entertaining bluster under- vide the best information possible as the
to ensure that their “facts” will withstand mine the democratic idea of rule by the basis for public policy. And they should
scrutiny, we can only conclude that they people. The irony is that politicians who publicly affirm the intellectual virtues that
take an arrogant view of the public. They benefit from post-truth tendencies rely on they so effectively model: critical think-
take their right to lie as a given, perhaps truth, too, but not because they adhere to ing, sustained inquiry and revision of be-
particularly when the lies are transparent. it. They depend on most people’s good- liefs on the basis of evidence. Another line
Many among the electorate seem not to natured tendency to trust that others are from Nietzsche is especially pertinent
register the contempt involved, perhaps be- telling the truth, at least the vast majority now: “Three cheers for physics!—and even
cause they would like to think that their fa- of the time. more for the motive that spurs us toward
vored candidate is at least well intentioned Scientists and philosophers should be physics—our honesty!”
and would not deliberately mislead them. shocked by the idea of post-truth, and they
Much of the public hears what it wants should speak up when scientific findings Kathleen Higgins t eaches and writes on Nietzsche,
to hear because many people get their are ignored by those in power or treated aesthetics and philosophy of emotion at the University
news exclusively from sources whose bias as mere matters of faith. Scientists must of Texas at Austin.

WHY WE BELIEVE
CONSPIRACY

THEORIES Baseless
theories
threaten our
safety and
democracy.
It turns out
that specific
emotions
make people
prone to
such
thinking
By Melinda
Wenner Moyer
Illustration by
Eddie Guy

S
POLITICS
t e phan Le wandowsky was de e p in de nial . N e ar ly

10 years ago the cognitive scientist threw himself into
a study of why some people refuse to accept the over
whelm ing evidence that the planet is warming and
humans are responsible. As he delved into this climate
change denialism, Lewandowsky, then at the University of
Western Australia, discovered that many of the naysayers
also believed in outlandish plots, such
as the idea that the A
pollomoon land
ing was a hoax created by the Ameri
can government. “A lot of the dis
course these people were engaging in
on the Internet was totally conspira
torial,” he recalls.
Lewandowsky’s findings, published in 2013 in
sychological Science, brought these conspiracy the
P
orists out of the woodwork. Offended by his claims,
they criticized his integrity online and demanded
that he be fired. (He was not, although he has since
moved to the University of Bristol in England.) But
as Lewandowsky waded through one irate post after 1
another, he discovered that his critics—in response
to his assertions about their conspiratorial tenden
cies—were actually spreading new conspiracy theo
ries about him. These people accused him and his injured six others in a Pittsburgh synagogue in
colleagues of faking survey responses and of con October 2018 justified his attack by claiming that
ducting the research without ethical approval. When Jewish people were stealthily supporting illegal
his personal website crashed, one blogger accused immigrants. In 2016 a conspiracy theory positing
him of intentionally blocking critics from seeing it. that high-ranking Democratic Party officials were
None of it was true. part of a child sex ring involving several Washing
Preceding pages: Getty Images (i llustration reference) ; This page: Getty Images (1 )
The irony was amusing at first, but the ranting ton, D.C.–area restaurants incited one believer to
even included a death threat, and calls and e-mails fire an assault weapon inside a pizzeria. Luckily no
to his university became so vicious that the adminis one was hurt.
trative staff who fielded them asked their managers The mindset is surprisingly common, although
for help. That was when Lewandowsky changed his thankfully it does not often lead to gunfire. More
assessment. “I quickly realized that there was noth than a quarter of the American population believes
ing funny about these guys at all,” he says. there are conspiracies “behind many things in the
The dangerous consequences of the conspirato world,” according to a 2017 analysis of government
rial perspective—the idea that people or groups are survey data by University of Oxford and University
colluding in hidden ways to produce a particular of Liverpool researchers. The prevalence of conspir
outcome—have become painfully clear. The belief acy mongering may not be new, but today the theo
that the coronavirus pandemic is an elaborate hoax ries are becoming more visible, says Viren Swami, a
designed to prevent the reelection of Donald social psychologist at Anglia Ruskin University in
Trump has incited some Americans to forgo impor England, who studies the phenomenon. For in
tant public health recommendations, costing lives. stance, when more than a dozen bombs were sent to
The gunman who shot and killed 11 people and prominent Democrats and Trump critics, as well as

POLITICS
CNN, in October 2018, a number of high-profile con evidence-based indictments and U.S. intelligence
servatives quickly suggested that the explosives agency conclusions. So how is one to know what to
were really a “false flag,” a fake attack orchestrated believe? There, too, psychologists have been at work
by Democrats to mobilize their supporters during and have uncovered strategies that can help people
the U.S. midterm elections. distinguish plausible theories from those that are
One obvious reason for the current raised pro almost certainly fake—strategies that seem to be
file of this kind of thinking is that the last U.S. come more important by the day.
president was a vocal conspiracy theorist. Donald
Trump has suggested, among other things, that the THE ANXIETY CONNECTION
father of Senator Ted Cruz of Texas helped to assas I n M ay 2 01 8 the American Psychiatric Association
sinate President John F. Kennedy and that Demo released the results of a national survey suggesting
crats funded the same migrant caravan traveling that 39 percent of Americans felt more anxious than
from Honduras to the U.S. that worried the Pitts they did a year ago, primarily about health, safety,
burgh synagogue shooter. finances, politics and relationships. A 2017 report
But there are other factors at play, too. New re found that 63 percent of Americans were extremely
search suggests that events happening worldwide worried about the future of the nation and that
are nurturing underlying emotions that make peo 59 percent considered that time the lowest point in
2 3 4
CONSPIRACY THEORISTS believe plots are behind many situations. Some hold that the Apollo moon landing
was faked (1), others that the White House forced Supreme Court Justice Anthony Kennedy to retire (2) , and
others that Trump slogans on a mail bomber's van were put there to frame Republicans (3). The gunman who
killed 11 synagogue members in 2018 claimed a Jewish group was undermining America (4).
Brendan Smialowski/Getty Images (2 ) ; Paul Bilodeau/Alamy (3 ) ; Jeff Swensen/Getty Images (4 )
ple more willing to believe in conspiracies. Experi U.S. history that they could remember. Such feelings
ments have revealed that feelings of anxiety make span the political spectrum. A 2018 Pew Research
people think more conspiratorially. Such feelings, Center survey found that the majority of both Dem
along with a sense of disenfranchisement, currently ocrats and Republicans felt that “their side” in poli
grip many Americans, according to surveys. In such tics had been losing in recent years on issues they
situations, a conspiracy theory can provide comfort found important.
by identifying a convenient scapegoat and thereby Such existential crises can promote conspirato
making the world seem more straightforward and rial thinking. In a 2015 study in the Netherlands, re
controllable. “People can assume that if these bad searchers split college students into three groups.
guys weren’t there, then everything would be fine,” People in one group were primed to feel powerless.
Lewandowsky says. “Whereas if you don’t believe in The scientists asked them to recall and write about a
a conspiracy theory, then you just have to say terri time in their lives when they felt they were not in
ble things happen randomly.” control of the situation they were in. Those in a sec
Discerning fact from fiction can be difficult, how ond group were cued in the opposite direction. They
ever, and some seemingly wild conspiracy ideas were asked to write about a time when they felt
turn out to be true. The once scoffed-at notion that totally in control. And still others, in the third group,
Russian nationals meddled in the 2016 presidential were asked something neutral: to describe what
election is now supported by a slew of guilty pleas, they had for dinner last night. Then the researchers

POLITICS
asked all the groups how they felt about the con atorially than those who support the controlling
struction of a new subway line in Amsterdam that party. In the U.S., political liberals put forth a num
had been plagued by problems. ber of unproved conjectures as conservatives as
Students who had been primed to feel in control cended to control the government in recent years.
were less likely than students in the other two These include the charge that the White House
groups to support conspiracy theories regarding the coerced Anthony Kennedy to retire from the U.S.
subway line, such as the belief that the city council Supreme Court and the allegation that Russian
was stealing from the subway’s budget and that it president Vladimir Putin is blackmailing Trump
was intentionally jeopardizing residents’ safety. with a video of him watching prostitutes urinate on
Other studies have uncovered similar effects. Swami a Moscow hotel bed.
and his colleagues, for instance, reported in 2016 When feelings of personal alienation or anxiety
that individuals who feel stressed are more likely are combined with a sense that society is in jeop
than others to believe in conspiracy theories, and a ardy, people experience a kind of conspiratorial
2017 study found that promoting anxiety in people double whammy. In a study conducted in 2009, near
also makes them more conspiracy-minded. the start of the U.S.’s Great Recession, Daniel Sulli
Feeling alienated or unwanted also seems to van, a psychologist now at the University of Arizona,
make conspiratorial thinking more attractive. In and his colleagues told one group that parts of their
2017 Princeton University psychologists set up an lives were largely out of their control because they
could be exposed to a natural disaster
or some other catastrophe and told
When feelings of personal alienation or another group that things were under
their control. Then participants were
anxiety are combined with a sense that asked to read essays that argued that
the government was handling the
society is in jeopardy, people experience economic crisis either well or poorly.
Those cued about uncontrolled life
a kind of conspiratorial double whammy, situations and told their government
was doing a bad job were the most
according to a study conducted near likely to think that negative events in
the start of the U.S.’s Great Recession.

their lives would be instigated by ene
mies rather than random chance,
which is a conspiratorial hallmark.
Although humans seek solace in
experiment with trios of people. The researchers conspiracy theories, they rarely find it. “They’re
asked all participants to write two paragraphs de appealing but not necessarily satisfying,” says Dan
scribing themselves and then told them that their iel Jolley, a psychologist at the University of Not
descriptions would be shared with the other two in tingham in England. For one thing, conspiratorial
their group, who would use that information to de thinking can incite individuals to behave in a way
cide if they would work with the person in the that increases their sense of powerlessness, making
future. After telling some subjects that they had them feel even worse. A 2014 study co-authored by
been accepted by their group and others that they Jolley found that people who are presented with
had been rejected, the researchers evaluated the conspiracy theories about climate change—scien
subjects’ thoughts on various conspiracy-related tists are just chasing grant money, for instance—
scenarios. The “rejected” participants, feeling alien are less likely to plan to vote. And a 2017 study re
ated, were more likely than the others to think the port ed that believing in work-related con spir
scenarios involved a coordinated conspiracy. acies—such as the idea that managers make decis
It is not just personal crises that encourage indi ions to protect their own interests—causes individ
viduals to form conspiratorial suspicions. Collective uals to feel less committed to their job. “It can
social setbacks do so as well. In a 2018 study, re snowball and become a pretty vicious, nasty cycle
searchers at the University of Minnesota and Lehigh of inaction and negative behavior,” says Karen
University surveyed more than 3,000 Americans. Douglas, a social psychologist at the University of
They found that participants who felt that American Kent in England and a co-author of the paper on
values were eroding were more likely than others to work-related conspiracies.
agree with conspiratorial statements, such as that The negative and alienating beliefs can also pro
“many major events have behind them the actions of mote dangerous behaviors in some, as with the
a small group of influential people.” Joseph Uscin Pittsburgh shootings and the pizzeria attack. But
ski, a political scientist at the University of Miami, the theories need not involve weapons to inflict
and his colleagues have shown that people who dis harm. People who believe vaccine conspiracy theo
like the political party in power think more conspir ries, for example, say they are less inclined to vacci

POLITICS
nate their kids, which creates pockets of infectious improve their analytic thinking skills should ask
disease that put entire communities at risk. three key questions when interpreting conspiracy
claims. One: What is your evidence? Two: What is
TELLING FACT FROM FICTION your source for that evidence? Three: What is the
I t m ay b e p o s s i b l e to quell conspiracy ideation, reasoning that links your evidence back to the
at least to some degree. One long-standing question claim? Sources of evidence need to be accurate,
has been whether it is a good idea to counter con credible and relevant. For instance, “you shouldn’t
spiracy theories with logic and evidence. Some older take advice from your mom about whether the yel
research has pointed to a “backfire effect”—the idea low color under your fingernails is a bad sign,” Mur
that refuting misinformation can just make individ phy says—that kind of information should come
uals dig their heels in deeper. “If you think there are from someone who has expertise on the topic, such
powerful forces trying to conspire and cover as a physician.
[things] up, when you’re given what you see as a In addition, false conspiracy theories have sev
cover story, it only shows you how right you are,” eral hallmarks, Lewandowsky says. Three of them
Uscinski says. are particularly noticeable. First, the theories in
But other research suggests that this putative clude contradictions. For example, some deniers of
effect is, in fact, rare. A 2016 paper reported that climate change argue that there is no scientific con
when scientists refuted a conspiracy theory by sensus on the issue while framing themselves as
pointing out its logical inconsistencies, it became heroes pushing back against established consensus.
less enchanting to people. And in a study published Both cannot be true. A second telltale sign is when
online in 2018 in P olitical Behavior,researchers a contention is based on shaky assumptions. Trump,
recruited more than 10,000 people and presented for instance, claimed that millions of illegal immi
them with corrections to various claims made by grants cast ballots in the 2016 presidential election
political figures. The authors concluded that “evi and were the reason he lost the popular vote. Be
dence of factual backfire is far more tenuous than yond the complete lack of evidence for such voting,
prior research suggests.” In a review article, the re his assumption was that multitudes of such votes—
searchers who first described the backfire effect said if they existed—would have been for his Demo
that it may arise most often when people are being cratic opponent. Yet past polls of unauthorized His
challenged over ideas that define their worldview or panic immigrants suggest that many of them would
sense of self. Finding ways to counter conspiracy have voted for a Republican candidate over a Dem
theories without challenging a person’s identity may ocratic one.
therefore be an effective strategy. A third sign that a claim is a far-fetched theory,
Encouraging analytic thinking may also help. In rather than an actual conspiracy, is that those who
a 2014 study published in Cognition, Swami and his support it interpret evidence against their theory
colleagues recruited 112 people for an experiment. as evidence for it. When the van of the convicted
First, they had everyone fill out a questionnaire that mail bomber Cesar Sayoc was found in Florida plas
evaluated how strongly they believed in various tered with Trump stickers, for instance, some indi
conspiracy theories. A few weeks later the subjects viduals said this helped to prove that Democrats
came back in, and the researchers split them into were really behind the bombs. “If anyone thinks
two groups. One group completed a task that in this is what a real conservative’s van looks like, you
cluded unscrambling words in sentences containing are being willfully ignorant. Cesar Sayoc is clearly
words such as “analyze” and “rational,” which just a fall guy for this obvious false flag,” one person
primed them to think more analytically. The second posted on Twitter.
group completed a neutral task. Conspiracy theories are a human reaction to
Then the researchers readministered the con confusing times. “We’re all just trying to understand
spiracy theory test to the two groups. Although the world and what’s happening in it,” says Rob
the groups had been no different in terms of con Brotherton, a psychologist at Barnard College and
spiratorial thinking at the beginning of the experi author of S uspicious Minds: Why We Believe in Con-
ment, the subjects who had been incited to think spiracy Theories(Bloomsbury Sigma, 2015). But
analytically became less conspiratorial. Thus, by real harm can come from such thinking, especially
giving people “the tools and the skills to analyze when believers engage in violence as a show of sup
data and to look at data critically and objectively,” port. If we look out for suspicious signatures and
we might be able to suppress conspiratorial think ask thoughtful questions about the stories we
ing, Swami says. encounter, it is still possible to separate truth from
Analytical thinking can also help discern im lies. It may not always be an easy task, but it is a cru
plausible theories from ones that, crazy as they cial one for all of us.
sound, are supported by evidence. Karen Murphy,
an educational psychologist at Pennsylvania State Melinda Wenner Moyeris a contributing editor at S cientific American.
University, suggests that individuals who want to She wrote about multidisease vaccines in the June 2019 issue.

POLITICS

POLITICS
CONTAGIOUS
DISHONESTY
Dishonesty begets dishonesty, rapidly spreading
unethical behavior through a society
By Dan Ariely and Ximena Garcia-Rada
I
Illustration by Lisk Feng
magine that you go to City Hall for a construction CONTAGION

permit to renovate your house. The employee who receives Suppose you refused t he City Hall employee’s request for a
your form says that because of the great number of applica bribe. How would the experience influence your response to a
tions the office has received, the staff will take up to nine subsequent ethical dilemma? In laboratory studies we conduct
months to issue the permit—but if you give her $100, your ed with behavioral researchers Vladimir Chituc, Aaron Nichols,
form will make it to the top of the pile. You realize she has just Heather Mann, Troy H. Campbell and Panagiotis Mitkidis, we
asked for a bribe: an illicit payment to obtain preferential sought an answer to that question.
treatment. A number of questions are likely to go through We invited individuals to a university behavioral lab to play
your head. Will you pay to speed things up? Would any of your a game that involved throwing a virtual die for a reward. Every
friends or relatives do the same? You would probably not wonder, one was told that they would be compensated based on the out
however, whether being exposed to the request would in itself affect come of multiple rolls. In practice, however, they could misre
a subsequent ethical decision. That is the kind of question behav port their rolls to earn more money. So all participants faced a
ioral researchers ask to investigate how corruption spreads. conflict between playing the game by the rules and behaving dis
The extent of bribery is hard to measure, but estimates from honestly to earn more. We created this setup to assess how indi
the World Bank suggest that corrupt exchanges involve $1 trillion viduals balance external and internal—or psychological—rewards
annually. In 2018 Transparency International reported that more when making ethical decisions. Research that Nina Mazar, On
than two thirds of 180 countries it surveyed got a score of less than Amir and one of us (Ariely) published in 2008 indicates that most
50 on a scale from 0 (“highly corrupt”) to 100 (“very clean”). Major people act unethically to the extent that they can benefit while
scandals regularly make global headlines, such as when Brazilian also preserving their moral self-image—an observation they de
construction company Odebrecht admitted in 2016 to having paid scribed as the theory of self-concept maintenance.
upward of $700 million in bribes to politicians and bureaucrats in Our game involved rolling a virtual die 30 times on iPads.
12 countries. But petty corruption, involving small favors between Many behavioral economists have used similar paradigms in
a few people, is also very common. Transparency International’s volving physical dice and coins to assess dishonesty in so-called
Global Corruption Barometer for 2017 showed that one in every decontextualized games—that is, games that are not affected by
four of those surveyed said they had paid a bribe when accessing social or cultural norms. Prior to each roll, participants were in
public services in the previous year, with almost one in three re structed to choose a side of the die in their mind—top or bot
porting such payments in the Middle East and North Africa. tom—and report their choice after s eeing the outcome of the roll.
Corruption, big or small, impedes the socioeconomic devel They would earn a fixed amount of money per dot on the side
opment of nations. It affects economic activities, weakens insti they reported each time. So everyone had a financial incentive
tutions, interferes with democracy, and erodes the public’s trust to cheat by reporting the high-paying side. For example, if the
in their government officials, politicians and neighbors. Under outcome of the roll was two on the top of the die and five on the
standing the underlying psychology of bribery could be crucial bottom of the die, people might be tempted to say they had cho
to tackling the problem. Troublingly, our studies suggest that sen “bottom” before the roll even if they had not.
mere exposure to corruption is corrupting. Unless preventive This paradigm does not allow us to know whether someone
measures are taken, dishonesty can spread stealthily and un cheated in a specific roll. Nevertheless, when results are aggregat
intentionally from person to person like a disease, eroding social ed across all rolls and participants in a group, the proportion of fa
norms and ethics—and once a culture of cheating and lying be vorable rolls chosen can be compared against chance (50 percent)
comes entrenched, it can be difficult to dislodge. to assess the magnitude of dishonesty. After participants received

POLITICS
instructions about the game and how they would make

money in the session, which they would get to take home,
they were randomly assigned to a low- or high-payment Corruption Perception Index
version. Those in the high-payment game would take the
same actions as those in the low-payment game but earn Levels of corruption in the public sector vary greatly around the world,
10 times more. Everyone was told about the existence of according to Transparency International. Every year the nongovern-
the other game. Then, half the participants in the low- mental agency uses opinion surveys and expert assessments to rank
payment condition were offered the option of paying a countries on a corruption scale ranging from 0 to 100. The chart dis-
bribe to be switched to the high-payment game. plays the evolution of these rankings from 2012 to 2018, highlighting
The research assistant administering the session the most and least corrupt countries, as well as a few that evinced the
framed that opportunity as illegal to engender a moral greatest change in corruption. Levels of dishonest behavior can worsen
dilemma similar to one that might arise in real life. The or decline with surprising rapidity but are relatively stable in the least
person mentioned that the boss was not around and corrupt countries. Curiously, behavioral studies show that the innate
that the participant could easily be switched to the inclination of individuals to behave dishonestly is roughly the same
high-paying game without anyone finding out. Thus, in different countries, regardless of their actual levels of corruption.
we ended up with three groups of people: low-payment
no bribe, high-payment no bribe, and bribe exposed;
the last group could be further split into bribe payers Considered least corrupt (2018) Biggest increases in CPI (2012–2018)
and bribe refusers. This arrangement allowed us to as 100
Corruption Perception Index (CPI) Value

Considered most corrupt (2018) Biggest decreases in CPI (2012–2018)
Less corrupt
sess how ethically those exposed to the idea of a bribe
would behave after having encountered the offer.
We administered three versions of the test to a total 88 Denmark
of 349 individuals in the behavioral lab. In the first two 87 New Zealand
studies, some participants were offered the possibility 85 Finland
of paying a $2 bribe to be placed in the high-payment 85 Sweden
85 Singapore
version of the game, and 85 percent of them paid. Cru
80
cially, we observed that in the games they went on to
play, the bribe-exposed group cheated more than par
ticipants who did not receive such a request. In the sec
ond study, for example, the bribe-exposed group cheat 71 U.S.
ed 9 percent more than those who played the high-pay
ment version of the game and 14 percent more than the No data (dotted) 66 Seychelles (+14)
group who played the low-payment version of the game
but had not been asked for a bribe. 60
In a third study, we tested whether people act more
immorally when they pay a bribe or when they are mere 55 Saint Lucia (–16)
ly exposed to one. We made the bribe costlier at $12, and
82 percent turned down the request, giving us a large
sample size of bribe refusers. Disturbingly, even when
we limited our analysis to this group of apparently eth 44 Belarus (+13)
ical individuals, we found that bribe-exposed people
cheated more than those who did not receive the illegal 40
request. Taken together, results from these three exper 36 Bahrain (–15)
iments suggest receiving a bribe request erodes moral
character, prompting people to behave more dishonest
ly in subsequent ethical decisions. 29 Myanmar (+14)
ERODING NORMS
Our work suggests t hat bribery is like a contagious 20
disease: it spreads quickly among individuals, often by 14 Yemen
Source: Transparency International (C PI data)
mere exposure, and becomes harder to control as time 14 North Korea

passes. This is because social norms—the patterns of be 13 Syria (–13)
havior that are accepted as normal—impact how people 13 South Sudan
More corrupt
will behave in many situations, including those involv 10 Somalia

ing ethical dilemmas. In 1991 psychologists Robert B.
Cialdini, Raymond R. Reno and the late Carl A. Kallgren
drew the important distinction between descriptive 0
norms (the perception of what most people do) and in 2012 2013 2014 2015 2016 2017 2018
junctive norms (the perception of what most people ap
96 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022 Graphic by Tiffany Farrant-Gonzalez

POLITICS
prove or disapprove of ). We argue that both types of norms influ of offenders, the number of unpaid violations decreased signifi
ence bribery. Simply put, knowing that others are paying bribes cantly. Their work suggests that cultural norms and legal en
to obtain preferential treatment (a descriptive norm) makes peo forcement are key factors in shaping ethical behavior.
ple feel that it is more acceptable to pay a bribe themselves. Sim
ilarly, thinking that others believe that paying a bribe is accept PROBING DEEPER
able (an injunctive norm) will make people feel more comfort B u t w h at a r e the psychological mechanisms involved in the
able when accepting a bribe request. Bribery becomes normative, exchange of a bribe? Behavioral researchers have examined these
affecting people’s moral character. in the lab and the field. For example, behavioral economists Uri
In 2009 Ariely, with behavioral researchers Francesca Gino Gneezy, Silvia Saccardo and Roel van Veldhuizen studied the psy
and Shahar Ayal, published a study showing how powerful social chology behind the acceptance of bribes. They conducted a lab
norms can be in shaping dishonest behavior. In two lab studies, study with 573 participants divided into groups of three. Two par
they assessed the circumstances in which exposure to others’ un ticipants competed for a prize by writing jokes, and the third chose
ethical behavior would change someone’s ethical decision-mak the winner. The writers could bribe the referees by including $5
ing. Group membership turned out to have a significant effect: in an envelope when submitting their entry. Gneezy and his col
When individuals observed an in-group member behaving dis leagues studied how referees reacted and how receiving a bribe
honestly (a student with a T-shirt suggesting he or she was from distorted their judgment. They found when the referees could keep
the same school cheating in a test), they, too, behaved dishonest only the winner’s bribe, bribes distorted their judgment, but when
ly. In contrast, when the person behaving dishonestly was an out- the referees could keep the bribe regardless of the winner, bribes
group member (a student with a T-shirt from the rival school), no longer influenced their decision. This study suggests that peo
observers acted more honestly. ple are influenced by bribes out of self-interest and not because
But social norms also vary from culture to culture: What is ac they want to return the favor to whoever paid the bribe.
ceptable in one culture might not be acceptable in another. For ex In related studies, published in 2017, Nils Köbis, now at the
ample, in some societies giving gifts to clients or public officials Max Planck Institute for Human Development in Berlin, and his
demonstrates respect for a business relationship, whereas in oth colleagues tested the idea that severe corruption emerges gradu
er cultures it is considered bribery. Similarly, gifts for individuals ally through a series of increasingly dishonest acts. They found
in business relationships can be regarded either as lubricants of that, in fact, participants in their four experiments were more
business negotiations, in the words of behavioral economists Mi likely to behave unethically when given the opportunity to do so
chel André Maréchal and Christian Thöni, or as questionable busi in an abrupt manner—that is, when tempted with a single oppor
ness practices. And these expectations and rules about what is ac tunity to behave unethically for a large gain rather than when
cepted are learned and reinforced through observation of others faced with a series of choices for small benefits. As the research
in the same group. Thus, in countries where individuals regular ers concluded, “sometimes the route to corruption leads over a
ly learn that others are paying bribes to obtain preferential treat steep cliff rather than a slippery slope.”
ment, they determine that paying bribes is socially acceptable. Given how damaging corruption is to societies, we believe it
Over time the line between ethical and unethical behavior becomes is crucial to further probe its psychological roots. Three areas beg
blurry, and dishonesty becomes the “way of doing business.” for future research. First, we need a fuller accounting of what
Interestingly, in cross-cultural research we published in 2016 drives a culture toward less ethical behavior. What, for example,
with Mann and behavioral researchers Lars Hornuf and Juan prompts someone to ask for a bribe? What impacts the likelihood
Tafurt, we found that people’s underlying tendency to behave dis of accepting a bribe? Second, what are the consequences of brib
honestly is similar across countries. We studied 2,179 native res ery? Clearly, bribery and, more broadly, dishonesty are conta
idents in the U.S., Colombia, Portugal, Germany and China. Us gious. But future research could investigate the lasting effects of
ing a game similar to the one in our bribing studies, we observed bribery over time and across domains: What happens when peo
that cheating levels in these countries were about the same. Re ple are consistently exposed to bribes? Does recurring exposure
gardless of the country, people were cheating to an extent that to bribery strengthen or weaken the effect of bribes on individu
balanced the motive of earning money with that of maintaining al dishonesty? Last, what kinds of interventions would be most
a positive moral image of themselves. And contrary to common effective in reducing bribe solicitations and acceptance?
ly held beliefs (which we assessed among a different set of par Going back to our initial example, we see that the corrupt ex
ticipants) about how these countries vary, we did not find more change that the City Hall employee offered might seem trivial or
cheaters in countries with high corruption levels (such as Colom at least be considered an isolated event. Sadly, a single bribe re
bia) than in countries with low corruption levels (Germany). quest will affect the requester and the recipient. And notably, its
So why do we observe huge international differences in lev dominolike effect can impact many individuals over time, spread
els of corruption and bribery? It turns out that although individ ing quickly across a society and, if left unchecked, entrenching
uals’ innate tendencies to behave honestly or otherwise are sim a culture of dishonesty.
ilar across countries, social norms and legal enforcement pow
erfully influence perceptions and behaviors. In 2007 economists Dan Ariely is James B. Duke Professor of Psychology & Behavioral Economics
Raymond Fisman and Edward Miguel published a study of park at Duke University and founder of the Center for Advanced Hindsight. He is co-creator
ing violations among United Nations diplomats living in Man of a documentary on corruption and a bestselling author.
hattan. They found that diplomats from high-corruption coun
tries accumulated more unpaid parking violations. But when en Ximena Garcia-Rada i s an assistant professor at Texas A&M University. She studies
forcement authorities could confiscate diplomatic license plates how social factors influence consumer decision-making.

POLITICS
OPINION
Evidence Shouldn’t
Be Optional
This Supreme Court often ignores science
when handing down decisions,
and it affects far too many lives
I
By the Editors
n the summer of 2022 the Supreme Court ignored the scientific evidence
underlying safe abortion, the need to slow climate change and the value of gun-safety laws.
It is alarming that the justices have indicated a willingness to consider a voting rights case
next term, given Chief Justice John Roberts’s feelings on what he calls the “sociological gob-
bledygook” of research into the effects of gerrymandering.
The promise of democracy is being sorely tested by the recent injustices leveled by the
Supreme Court’s conservative justices in cases involving health, welfare and the future of
the planet. Over and over in the 2021–2022 term, their decisions put industry, religion (specifi-
cally, a conservative strain of Christianity) and special interests above facts. They have devalued
the role of expertise.
Disregarding science and evidence is a terrible shift for the and welfare is one that we fear will lead to needless suffering
highest court in the land, which once safeguarded the health of and death. We urge the court to change its reasoning—to value
the public in rulings that upheld state vaccine mandates and statistics and research and to understand how ignoring them in
safe food production. This is in contrast to the way our current making decisions is contrary to common decency and their
conservative justices have viewed C OVID restrictions, whether responsibility as jurists to the people of the U.S.
exempting religious groups from bans on group gatherings or In their June 2022 decision in D obbs v. Jackson Women’s
barring vaccine mandates for large businesses. Even in deci- Health Organization, the majority justices ignored what we and
sions that uphold basic public health tenets, conservative jus- others have repeatedly reported: abortion is safe—much safer
tices have spouted misleading scientific claims. In his dissent on than pregnancy itself—and denying people access to legal abor-
the court’s decision to not take on New York’s vaccine-mandate tions leads to poorer physical and mental health outcomes, not
law for health-care workers, Justice Clarence Thomas laments to mention economic outcomes. In overturning R oe v. Wade
that the workers demanding a religious exemption objected to and shunting abortion rights to states, the justices who voted in
available C OVID vaccines “because they were developed using favor of Dobbs put religion and the status of a mass of cells over
cell lines derived from aborted children,” wording that obscures the health and welfare of actual people who make up approxi-
that the cells were grown in a laboratory using lines derived mately 50 percent of the U.S. population. They also indicated
from elective abortions decades ago and are also used in the their disregard for the medical profession and the privacy of the
development of routine drugs. doctor-patient relationship that the justices in the majority will
This shift away from our social responsibilities for health no doubt continue to enjoy after their ruling becomes practice.

POLITICS
JUSTICES o f the U.S. Supreme Court are shown here before

aging rain and wildfires—these all affect the health and welfare
Ketanji Brown Jackson was sworn in on June 30, 2022, to replace
of people in the U.S. The science is clear on this: we have to act
Stephen G. Breyer.
now, and the Supreme Court made those actions harder.
As with every level of government, there is no requirement
In striking down New York’s gun-safety law that same month, that the court factor science into its decision-making. And, as
the majority justices ignored data showing that unfettered Justice Amy Coney Barrett has said, “I’m certainly not a scien-
access to guns leads to more murders and suicides, not fewer tist.” But expertise matters, and knowing when you don’t know
crimes. They ignored data indicating that guns are now respon- something and seeking that information makes for a better jus-
sible for more child deaths than automobiles. They even ignored tice. Yet in their efforts to be constitutional purists, at least
data showing that when you repeal a gun law, gun-related kill- when it suits their ideology, the justices in the majority show
ings go up. It was a coldhearted decision against the backdrop that ignoring science and evidence is their modus operandi.
of Uvalde, Buffalo and every mass shooting our nation has suf- Instead they are using their power to uphold a certain vein of
fered in the past decades. It was another slap in the face of our religion: this same term the majority ruled against separation
health-care system and the emergency clinicians who must try of church and state in two education cases, one of which forces
to save people shredded apart by high-powered weapons that Maine to fund schools that teach children misinformation
are incredibly easy to get. As we have said before, gun-safety about evolution and climate science. The U.S. once inspired
laws are part of what makes a compassionate nation, and in this other countries to protect people’s liberties. Now the rest of the
Erin Schaff/The New York Times/ Bloomberg via Getty Images
ruling, the majority justices showed their callousness. world is watching and reacting to the decisions that our
And then there is climate change. In stripping power from the Supreme Court made this term. And it’s not good.
Environmental Protection Agency to help power plants mitigate You don’t need to be a scientist or mathematician to make
their carbon output, also in the same month, the majority jus- good decisions and judgments. But if you are a justice of the
tices again said evidence doesn’t matter and science doesn’t mat- U.S. Supreme Court, with the lives and livelihoods of hundreds
ter. Our planet is warming. Coal is one of the largest contributors of millions of people hanging on your every opinion, you owe it
of greenhouse gases in the world. Taking regulatory power away to us to use the data that science painstakingly compiles when
from the epa now puts states in charge of slowing climate change. handing down your decisions. We cannot go back to a world of
Piecemeal efforts will not yield the reductions we need to slow religious and racial supremacy where the bodies of women and
warming. Federal action, as part of global efforts, is the necessary people of color are objects without self-determination. We must
solution to this problem. And climate change is a public health not become the dystopian future so much science fiction has
issue. Increases in ferocious winter storms, unbearable heat, dam- warned us about. Let evidence rule judgment.

FINDING ANSWERS IN SCIENCE
THE SCIENCE OF
ANTI-
SCIENCE
THINKING
Convincing people who doubt the validity of climate change
and evolution to change their beliefs requires overcoming
a set of ingrained cognitive biases
By Douglas T. Kenrick, Adam B. Cohen, Steven L. Neuberg
and Robert B. Cialdini
Illustration by Heads of State

I n p r i n c i p l e , s c i e n c e s h o u l d s e t i t s e l f a pa r t f r o m t h e h u e a n d c ry o f
partisan bickering. After all, the scientific enterprise reaches its conclusions by testing
hypotheses about the workings of the natural world. Consider the porpoise. Based on its
appearance and aquatic home, the animal was assumed to be a fish. But evidence gleaned
from observing its bone structure, its lack of gills and the genes it holds in common with
other warm-blooded land animals leads to its being classified as a mammal with a very
high level of confidence.
Yet a consensus about what constitutes a fact does not
always come so readily. Take a glance at your online news feed.
On a regular basis, government decision-makers enact policies
that fail to heed decades of evidence on climate change. In pub-
lic opinion surveys, a majority of Americans choose not to ac
cept more than a century’s worth of evidence on evolution by
natural selection. Academic intellectuals put the word “science”
in quotes, and members of the lay public reject vaccinations for
their children.
Scientific findings have long met with ambivalent responses:
A welcome mat rolls out instantly for horseless buggies or the
latest smartphones. But hostility arises just as quickly when sci-
ble of masterful intellectual accomplishments. Unfortunately,
we are not completely rational decision-makers.
Understanding why people engage in irrational thinking re
quires combining knowledge from a range of psychological dis-
ciplines. As authors, each of us studies a separate area address
ing how biased views originate. One of us (Cialdini) has exper-
tise in heuristics, the rules that help us to quickly make everyday
choices. Another of the authors (Kenrick) has studied how deci-
sions are distorted by social motives such as the desire to find a
mate or protect oneself from physical harm.
Yet another of us—Cohen—has investigated how religious
beliefs affect judgment. Finally, Neuberg has studied simple
entists’ findings challenge the political or religious status quo. cognitive biases that lead people to hold on to existing beliefs
Some of the British clergy strongly resisted Charles Darwin’s when confronted with new and conflicting evidence. All of us,
theory of evolution by natural selection. Samuel Wilberforce, in different ways, have tried to develop a deeper understanding
bishop of Oxford, asked natural selection proponent Thomas of the psychological mechanisms that warp rationality.
Huxley, known as “Darwin’s bulldog,” on which side of his fam- Explaining why thinking goes astray is critically important
ily Huxley claimed descent from an ape. to dispel false beliefs that circulate among politicians, students
In Galileo’s time, officials of the Roman Catholic Church, or even misinformed neighbors. Our own research and that of
well-educated and progressive intellectuals in most respects, our colleagues have identified key obstacles that stand in the
expressed outrage when the Renaissance scientist reported way of clear scientific thought. We have investigated why they
celestial observations that questioned the prevailing belief that arise and how they might be challenged and ultimately knocked
Earth was the center of the universe. Galileo was placed under down. Among the many hurdles, three in particular stand out:
house arrest and forced to recant his views as heresy.
In principle, scientific thinking should lead to decisions Shortcuts. Human brains are endowed with a facile means for
based on consideration of all available information on a given dealing with information overload. When we are overwhelmed
question. When scientists encounter arguments not firmly or are too short on time, we rely on simple heuristics, such as
grounded in logic and empirical evidence, they often presume accepting the group consensus or trusting an expert.
that purveyors of those alternative views either are ignorant of Confirmation Bias. Even with ample time and sufficient inter
the facts or are attempting to discourage their distribution for est to move beyond shortcuts, we sometimes process informa-
self-serving reasons—tobacco company executives suppressing tion in a manner less like an impartial judge and more like a
findings linking tobacco use to lung cancer, for instance. Faced lawyer working for the mob. We show a natural tendency to pay
with irrational or tendentious opponents, scientists often grow attention to some findings over others and to reinterpret mixed
increasingly strident. They respond by stating the facts more evidence to fit with preexisting beliefs.
loudly and clearly in the hope that their interlocutors will make Social Goals. Even if we surmount the first two obstacles, pow-
more educated decisions. erful forms of social motivation can interfere with an objective
Several lines of research, however, reveal that simply pre- analysis of the facts at hand. Whether one is biased toward
senting a litany of facts does not always lead to more objective reaching one scientific conclusion versus another can be influ-
decision-making. Indeed, in some cases, this approach might enced by the desire to win status, to conform to the views of a
actually backfire. Human beings are intelligent creatures, capa- social network or even to attract a mate.

MARCH FOR SCIENCE in Los Angeles, one of many held in 2017, tried to bolster support for the scientific community and for dealing
with issues such as climate change. Pro-Trump counterdemonstrators also rallied.
BEWARE THE SHORTCUT of a patient on their floor. The stranger on the phone asked the
M a s t e ry o f t h e s c i e n c e s r equires dealing with a set of dif- nurses on duty to go to the medicine cabinet and retrieve an
ficult concepts. Take Darwin’s theory of natural selection. To unfamiliar drug called Astroten and to administer a dose twice
understand it, one must comprehend a set of logical premises— as high as the daily maximum, violating not only the boldly
that environments with limited resources favor individuals stated guidelines on the label but also a hospital policy requir-
who are better able to procure food, shelter and mates, thereby ing handwritten prescriptions. Did the nurses balk? Ninety-
leading to selective representation of traits that confer these five percent obeyed the unknown “doctor” without raising any
skills to future generations. The student of Darwinian theory questions. Indeed, they had to be stopped on their way to the
must also know something about comparative anatomy (whales patient’s room with the potentially dangerous drug in hand.
have bone structures more similar to those of humans than to The nurses had unknowingly applied what is known as the
those of fish). Another prerequisite is familiarity with ecology, authority heuristic, trusting too readily in a person in a posi-
modern genetics and the fossil record. tion of responsibility.
Although natural selection stands out as one of the most sol-
idly supported scientific theories ever advanced, the average cit- CONFIRMATION BIAS
izen has not waded through textbooks full of evidence on the W h e n w e c a r e e n o u g h about a topic and have the time to
topic. In fact, many of those who have earned doctorates in sci- think about it, we move beyond simple heuristics to a more sys-
entific fields, even for medical research, have never taken a for- tematic analysis of the actual evidence. But even when we try
mal course in evolutionary biology. In the face of these challeng- hard to retain an objective perspective, our existing knowledge
es, most people rely on mental shortcuts or the pronouncements may still get in the way.
of experts, both strategies that can lead them astray. They may Abundant evidence suggests that people pay selective atten-
also rely—at their own peril—on intuition and gut instinct. tion to arguments that simply reinforce their own viewpoints.
We use heuristics because they frequently work quite well. If They find disagreement unpleasant and are inclined to dislike
a computer malfunctions, users can spend months learning the bearer of positions that run counter to their current beliefs.
about its various electronic components and how they are con- But what happens if intelligent individuals are forced to consid-
nected—or they can ask a computer technician. If a child devel- er evidence on both sides of an issue?
ops a serious health problem, parents can study the medical lit- In 1979 Charles Lord, then at Stanford University, and his
erature or consult a physician. colleagues conducted a study with Stanford students, who
But sometimes shortcuts serve us poorly. Consider a classic should have been able to make reasonable judgments about sci-
Sarah Morris/Getty Images
1966 study by psychiatrist Charles K. Hofling and his col- entific information. The students were exposed to several
leagues on how things can go terribly wrong when people rely rounds of scientific evidence on the deterrence effect of the
on the title “Dr.” as a cue to an individual’s authority. In the death penalty. They might first read a description of a study
study, nurses working on a busy hospital ward received a that questioned whether capital punishment prevents serious
phone call from a man who identified himself as the physician crime. It compared murder rates for the year before and the

year after the implementation of capital punishment in 14 the gun advocates, they would scrutinize the details of the study
states. In 11 of the states, murder rates climbed after the death until they discovered the numbers that suggested the opposite
penalty was established, implying that it lacks a deterrent effect. conclusion. If the researchers, moreover, later told one of the
Next, the students heard arguments from other scientists groups that results favored the opposite side, its members tend-
about possible weaknesses in that study’s evidence. Then the ed to be skeptical of the scientists who conducted the studies.
original researchers came back with counterarguments. After
that, the students heard about a different type of study suggest- THE SOCIAL PRESSURE GAUNTLET
ing the opposite: that capital punishment stops others from additional obstacles arise from the same powerful social
committing crimes. In it, researchers compared murder rates in impulses that help us get along with others. Take the scenario of
10 pairs of neighboring states with different capital punishment an office party where an individual’s co-workers sound off on
laws. In eight of the paired states, murder rates notched lower erroneous claims about evolution, global warming or evidence
with capital punishment on the books, supporting the death linking vaccines to autism. Confronted with that situation, does
penalty. Then students heard that evidence challenged, fol- one object or keep quiet to avoid seeming disruptive?
lowed by a counterargument to that challenge. Research on conformity runs deep in the psychological
If the students began with a strong opinion one way or the annals. In a classic 1951 study of group dynamics, psychologist
other and then performed a cold, rational analysis of the facts, Stanley Schachter observed what happened to an individual
they might have been expected to who disagreed with the majority’s
gravitate toward a middle ground in consensus. After trying unsuccessfully
their views, having just heard a mix
of evidence that included scientific
Even if the human to change the divergent opinion, other
group members ended up cutting off
claims that contradicted both posi- mind has many any further communication, ostraciz-
tions for and positions against capital
punishment. But that is not what obstacles to ing the outlier. A 2003 functional mag-
netic resonance imaging study by
happened. Rather students who pre-
viously favored the death penalty be
objective thinking, Kipling D. Williams, now at Purdue
University, and his colleagues found
came even more disposed toward it, we shouldn’t accept that ostracism activates the brain’s
and opponents of it turned more dis-
approving. It became clear that stu- that ignorance and dorsal anterior cingulate cortex—the
same region recruited when we expe-
dents on either side of the issue had
not processed the information in an
bias will always rience physical pain. In a 2005 study, a
team of researchers led by Gregory
evenhanded manner. Instead they be- triumph. Social Berns, a neuroeconomics professor at
lieved evidence that reinforced their
position was stronger, whereas refu-
psychology suggests Emory University, and his colleagues
found that disagreeing with a group
tations of that evidence were weak. So ways of coping. to which you belong is associated with
even if counterarguments can make increased activity in the amygdala, an
it past our inner censors, we show an area that turns on in response to dif-
inclination to weigh those arguments in a very biased manner. ferent types of stress. Holding an opinion different from other
A study published in September 2017, by Anthony N. Wash- group members, even a correct one, hurts emotionally. It there-
burn and Linda J. Skitka, both then at the University of Illinois fore comes as no surprise that people are often reluctant to pro-
at Chicago, seems to reinforce the Stanford paper’s findings. The vide evidence counter to what the rest of their group believes.
investigators tested the hypothesis that conservatives are more Social pressures can also influence how we process new in
distrustful of scientific evidence than liberals, perhaps because formation. Group consensus may encourage us to take recourse
such individuals exhibit rigid thinking and are less open to new in heuristics or to cling tightly to an opinion, all of which can
experiences. What they discovered, though, is that those on both interfere with objective thinking.
the right and the left reject scientific findings that do not jibe Our own research team conducted a study in which partici-
with their own political ideologies. The authors gave 1,347 study pants would make aesthetic judgments about a series of ab
participants scientific evidence on six hot-button issues—cli- stract designs and paintings and then read a passage designed
mate change, gun control, health-care reform, immigration, to put them in either a self-protective or a romantic frame of
nuclear power and same-sex marriage. A cursory look at the evi- mind. In the former condition, you might be asked to imagine
dence from scientific studies tended to favor one side of the being awakened by a loud sound while alone at home. As the
issue—the absolute numbers of crimes in cities with stricter gun scenario unfolds, it becomes clear that an intruder has entered
control might be higher than in cities without it. But a closer the house. You imagine reaching for the phone but finding that
look at the data might give credence to the opposite view—per- the line is dead. A call for help receives no response. Suddenly,
centage crime reductions in those same cities might actually be the door to the bedroom bursts open to reveal the dark shadow
greater than they were for cities lacking gun-control laws. of a stranger standing there.
If the initial hasty inspection of the data tended to favor the Alternatively, you might be randomly assigned to read an
anti-gun-control group’s expectations, members would general- account of a romantic encounter and asked to imagine being on
ly look no further, content with finding results that supported vacation and meeting an attractive person, then spending a
their particular bias. If the results contradicted the beliefs of romantic day with the partner that ends with a passionate kiss.

Next you would enter a virtual chat room, joining three other es suddenly vanished—students no longer used new evidence to
participants to evaluate abstract images, including one you had bolster existing preconceptions.
earlier judged as of average interest. Before making the second One way to counteract social pressures requires first explor-
judgment, though, you learn that this image has been rated as ing whether agreement within the group really exists. Someone
way below average by the other subjects. who disagrees with an erroneous opinion can sometimes open
So did study subjects change their initial judgment to con- other group members’ minds. In a 1955 S cientific American a rti-
form to the other group members? How people responded de cle, social psychologist Solomon E. Asch described studies on
pended on their current goals. Study participants who had read conformity, finding that if a single person in the group dis-
the home break-in scenario were more likely to conform to the agreed with the majority, consensus broke down. Similarly, in
group judgment. In contrast, those exposed to the amorous sto- Stanley Milgram’s famed studies of obedience—in which partic-
ry answered differently depending on gender: women con- ipants were led to believe that they were delivering painful
formed, but men actually went against the group’s judgment. shocks to an individual with a heart problem—unquestioned
Other studies by our team have found that fear can lead both obedience dissipated if other team members chose not to obey.
men and women to comply with group opinion, whereas sexual Fear increases the tendency toward conformity. If you wish
motives prompt men to try to stand out from the group, perhaps to persuade others to reduce carbon emissions, take care whom
to show that they are worthy mates. Men, in this frame of mind, you scare: a message that arouses fear of a dystopian future
are more likely to challenge the consensus and increase the risk- might work well for an audience that accepts the reality of cli-
iness of their actions. In all cases, though, our participants’ mate change but is likely to backfire for a skeptical audience.
views were shaped by their social goals in the moment. They did We have provided a few simple suggestions for overcoming
not process available information in a completely objective way. psychological obstacles to objective scientific thinking. There is
a large literature on persuasion and social influence that could
WHAT TO DO be quite useful to anyone attempting to communicate with a
If the human mind is built with so many obstacles to objec- group holding beliefs that fly in the face of scientific evidence.
tive scientific thinking, should we just give up and accept that For their part, scientists need to adopt a more systematic ap
ignorance and bias will always triumph? Not at all. Research in proach in collecting their own data on the effectiveness of dif-
social psychology also suggests ways of coping with heuristics, ferent strategies for confronting antiscientific thinking about
confirmation biases and social pressures. particular issues. It is essential to understand whether an indi-
We have seen that people frequently rely on heuristics when vidual’s resistance to solid evidence is based on simple heuristic
they lack the time or interest to carefully consider the evidence. thinking, systematic bias or particular social motives.
But such rules of thumb can often be defeated with simple in These steps are critical because antiscientific beliefs can lead
terventions. In one experiment by market researchers Joseph W. to reduced research funding and a consequent failure to fully
Alba and Howard Marmorstein, subjects considered informa- understand potentially important phenomena that affect pub-
tion about a dozen separate features of two cameras. Brand A lic welfare. In recent decades government funding has de
was superior to brand B on just four of the features, but these creased for research into the health impact of keeping guns in
were features critical in considering camera quality—the expo- the home and of reducing the harmful effects of air pollution.
sure accuracy, for instance. Brand B, on the other hand, came Guns in the home are frequently involved in teenage suicides,
recommended as superior on eight features, all of which were and an overwhelming scientific consensus shows that immediate
relatively unimportant—having a shoulder strap, for example. measures are needed to address the planet’s warming.
Some subjects examined each attribute for only two seconds; It is easy to feel helpless in the face of our reluctance to em
others had more time to study all the information. brace novel scientific findings. Still, there is room for optimism:
When they had only two seconds to evaluate each feature, the majority of Galileo’s fellow Italians and even the pope now
only a few subjects (17 percent) preferred the higher-quality accept that our planet revolves around the sun, and most of Dar-
camera, most opting instead for the one with a greater number win’s compatriots today endorse the theory of evolution. Indeed,
of unimportant functions. When the subjects were given suffi- the Anglican Church’s director of public affairs wrote an apology
cient time and allowed to directly compare the two cameras, to Darwin for the 200th anniversary of his birth. If scientists can
however, more than two thirds favored the camera with the few incorporate the insights of research on the psychological obsta-
features key to its overall quality. These results suggest that cles to objective thinking, more people will accept objective evi-
when communicating complicated evidence, sufficient time is dence of how the natural world functions as well.
needed to switch from a heuristic to a systematic mode of think-
ing that allows for better overall evaluation. Douglas T. Kenrick is a professor of psychology at Arizona State University
Confirmation biases can often be overcome by changing who has studied behaviors ranging from altruism to homicidal fantasies.
one’s perspective. The same Stanford researchers who studied
attitudes toward capital punishment also investigated how to Adam B. Cohen is a professor of psychology at Arizona State whose work focuses
change them. They instructed some students to remain objec- on the psychological foundations of religious beliefs.
tive and weigh evidence impartially in making a hypothetical
Steven L. Neuberg is a Foundation Professor and chair of the department of psychology
decision related to the death penalty. That instruction had no
at Arizona State. He researches stereotyping, prejudice and the effects of religion on conflict.
effect. Others were asked to play their own devil’s advocate by
considering what their opinions would have been if the research Robert B. Cialdini is Regents’ Professor Emeritus of psychology and marketing at Arizona
about the death penalty had contradicted their own views. Bias- State. He explores the reasons that people comply with requests in everyday settings.

How Professional
Truth Seekers Search
for Answers
Nine experts describe how they sort signal from noise
As told to Brooke Borel
Illustrations by Bud Cook
A D ATA J O U R N A L I S T
People assume
that because there
are data, the data
must be true.
But the truth is,
all data are dirty.
People create data, which means data have flaws just like
people. One thing data journalists do is interrogate that
assumption of truth, which serves an important account
ability function—a power check to make sure we aren’t
collectively getting carried away with data and making
bad social decisions.
To interrogate the data, you have to do a lot of janitorial
work. You have to clean and organize them; you have
to check the math. And you also have to acknowledge
the uncertainty. If you are a scientist, and you don’t have
the data, you can’t write the paper. But one of the fabulous
things about being a data journalist is that sparse data
don’t deter us—sometimes the lack of data tells me
something just as interesting. As a journalist, I can use
words, which are a magnificent tool for communicating
about uncertainty.
Meredith Broussard, an associate professor at the Arthur L.

Carter Journalism Institute at New York University

A B E H AV I O R A L S C I E N T I S T
The kind
of control
you have
in bench
science
is much
tighter
than in
behavioral
science—
t he power to detect small effects in people is much lower than sufficient to explain patterns of racially disparate policing
in, say, chemistry. Not only that, people’s behaviors change outcomes. That means we still have work to do. Because it’s
across time and culture. When we think about truth in behavioral not like we don’t know how to produce nonviolent and equitable
science, it’s really important not only to reproduce a study directly policing. Just look at the suburbs. We’ve been doing it there
but also to extend reproduction to a larger number of situations— for generations.
field studies, correlational studies, longitudinal studies. Of course, there is uncertainty. In most of this world, we are
So how do we measure racism, something that’s not a single nowhere near confidence about causality. Our responsibility as
behavior but a pattern of outcomes—a whole system by which scientists is to characterize these uncertainties, because a wrong
people are oppressed? The best approach is to observe the calculation in what drives something like racism is the difference
pattern of behaviors and then see what happens when we alter between getting policies right and getting them wrong.
or control for a variable. How does the pattern change? Take
policing. If we remove prejudice from the equation, racially Phillip Atiba Goff, a professor of African American studies
disparate patterns persist. The same is true of poverty, education and psychology at Yale University and co-founder and CEO
and a host of things we think predict crime. None of them are of the Center for Policing Equity

A PHYSICIAN
The answer to questions about

human life isn’t a certain thing,
like measuring how a stone drops to the ground in exactly so many seconds. If it were, it
probably would not be life. It would be a stone. Within biomedicine, it’s tricky finding out
if an effect is real—there are different standards across different fields. Not all tools will
work for every question, and there are different levels of complexity for what we know
before we even start a study.
Still, the one core dimension across biomedicine is the ability to replicate, in a new
study, what was seen in the first investigation. For many years in the field, we have been
discouraged from doing this. Why waste money to do the exact thing you had done before,
let alone something someone else had done before? But many researchers are realizing it
is not possible to leave out replication studies.
To make replication work, though, it is essential to have a detailed explanation of how
the original study was done. You need the instructions, the raw data and maybe even
some custom-built computer software. For a long time, scientists didn’t want to share that
information, but that is changing. Science is a communal effort, and we should default to
being open and sharing.
John P. A. Ioannidis, a professor of medicine at Stanford University
A SOCIAL TECHNOLOGIST
The biggest epistemological

question facing the field of
machine learning is: What is
our ability to test a hypothesis?
Algorithms learn to detect patterns and details from massive sets
of examples—for instance, an algorithm could learn to identify a
cat after seeing thousands of cat photographs. Until we have great
er interpretability, we can test how a result was achieved by appeal
ing conclusions from the algorithms. This raises the specter that we
don’t have real accountability for the results of deep-learning sys
tems—let alone due process when it comes to their effects on
social institutions. These issues are part of a live debate in the field.
Also, does machine learning represent a type of rejection of
the scientific method, which aims to find not only correlation but
also causation? In many machine-learning studies, correlation has
become the new article of faith, at the cost of causation. That
raises real questions about verifiability.
In some cases, we may be taking a step backward. We see this
in the space of machine vision and affect recognition. These are
systems that extrapolate from photographs of people to predict
their race, gender, sexuality or likelihood of being a criminal.
These sorts of approaches are both scientifically and ethically con
cerning—with echoes of phrenology and physiognomy. The focus
on correlation should raise deep suspicions in terms of our ability
to make claims about people’s identity. That’s a strong statement,
by the way, but given the decades of research on these issues in
the humanities and social sciences, it should not be controversial.
Kate Crawford, a research professor at the University of Southern

California Annenberg, co-founder of the AI Now Institute at New York
University and member of Scientific American’s board of advisers

A S TAT I S T I C I A N
In statistics, we aren’t
generally seeing the
whole universe but only
a slice of it. A small slice usually, which could tell
a completely different story than another small slice. We are trying to
make a leap from these small slices to a bigger truth. A lot of people
take that basic unit of truth to be the p-value, a statistical measure
of how surprising what we see in our small slice is, if our assumptions
about the larger universe hold. But I don’t think that’s correct.
In reality, the notion of statistical significance is based on an
arbitrary threshold applied to the p-value, and it may have very
little to do with substantive or scientific significance. It’s too easy
to slip into a thought pattern that provides that arbitrary threshold
with meaning—it gives us a false sense of certainty. And it’s also
too easy to hide a multitude of scientific sins behind that p-value.
One way to strengthen the p-value would be to shift the culture
toward transparency. If we not only report the p-value but also
show the work on how we got there—the standard error, the
standard deviation or other measures of uncertainty, for example—
we can give a better sense of what that number means. The more
information we publish, the harder it is to hide behind that p-value.
Whether we can get there, I don’t know. But I think we should try.
Nicole Lazar, a professor of statistics at Pennsylvania State University
A NEUROSCIENTIST
Science does not

search for truth,
as
many might think.
Rather the real purpose of science is to look for better questions. We run
experiments because we are ignorant about something and want to learn more,
and sometimes those experiments fail. But what we learn from our ignorance
and failure opens new questions and new uncertainties. And these are better
questions and better uncertainties, which lead to new experiments. And so on.
Take my field, neurobiology. For around 50 years the fundamental question
for the sensory system has been: What information is being sent into the brain?
For instance, what do our eyes tell our brain? Now we are seeing a reversal of
that idea: the brain is actually asking questions of the sensory system. The brain
may not be simply sifting through massive amounts of visual information from,
say, the eye; instead it is asking the eye to seek specific information.
In science, there are invariably loose ends and little blind alleys. While you
may think you have everything cleared up, there is always something new
and unexpected. But there is value in uncertainty. It shouldn’t create anxiety.
It’s an opportunity.
Stuart Firestein, a professor in the department of biological sciences

at Columbia University

A HISTORICAL LINGUIST
Like any scientist, linguists

rely on the scientific
method. O ne of the principal goals of
linguistics is to describe and analyze languages to
discover the full range of what is possible and not
possible in human languages. From this, linguists aim
to reach their goal of understanding human cognition
through the capacity for human language.
So there is an urgency to efforts to describe
endangered languages, to document them while
they are still in use, to determine the full range
of what is linguistically possible. There are around
6,500 known human languages; around 45 percent
of them are endangered.
Linguists use a specific set of criteria to identify
endangered languages and to determine just
how endangered a language is: Are children still
learning the language? How many individual people
A THEORETICAL PHYSICIST
Physics is the
most mature
science, and
physicists are
obsessive on the
subject of truth.
There is an actual universe out there. The central
miracle is that there are simple underlying laws,
expressed in the precise language of mathemat
ics, which can describe it. That said, physicists
don’t traffic in certainties but in degrees of confi
dence. We’ve learned our lesson: throughout
history, we have again and again found out that
some principle we thought was central to the
ultimate description of reality isn’t quite right.
To figure out how the world works, we have
theories and build experiments to test them.
Historically, this method works. For example,
physicists predicted the existence of the Higgs
boson particle in 1964, built the Large Hadron

speak it? Is the percentage of speakers declining
with respect to the broader population? And
are the contexts in which the language is being
used decreasing?
The question of scientific objectivity and
“truth” is connected to endangered language
research. Truth, in a way, is contextual. That is,
what we hold to be true can change as we get
more data and evidence or as our methods
improve. The investigation of endangered lan
guages often discovers things that we did not
know were possible in languages, forcing us to
reexamine previous claims about the limits of
human language, so that sometimes what we
thought was true can shift.
Lyle Campbell, an emeritus professor of linguistics

at the University of Hawaii at Mānoa
Collider (LHC) at CERN in the late 1990s and early

2000s, and found physical evidence of the Higgs
in 2012. Other times we can’t build the experi
ment—it is too massive or expensive or would be
impossible with available technology. So we try
thought experiments that pull from the existing
A PA L E O B I O L O G I S T
infrastructure of existing mathematical laws and
experimental data.
Here’s one: The concept of spacetime has Our basic unit of truth in paleobiology
been accepted since the early 1900s. But to look is the fossil—a clear record of life in the past—and we also use genetic
at smaller spaces, you have to use more powerful evidence from living organisms to help us put fossils within the tree of life. Together
resolution. That’s why the LHC is 17 miles they help us understand how these creatures changed and how they are related.
around—to produce the huge energies needed Because we are looking at extinct animals as they existed in a broader ecosystem,
to probe tiny distances between particles. But at we pull in information from other fields: chemical analysis of surrounding rocks
some point, something bad happens. You’ll put to get a sense of the fossil’s age, where the world’s landmasses might have been
out such an enormous amount of energy to look at the time, what kind of environmental changes were happening, and so on.
at such a small bit of space that you’ll actually cre To discover fossils, we scour the landscape to find them among rocks. You can
ate a black hole instead. Your attempt to see what tell the difference between a fossil and any old rock by its shape and its internal
is inside makes it impossible to do so, and the structure. For example, a fossil bone will have tiny cylinders called osteons where
notion of spacetime breaks down. blood vessels once ran through the bone. Some fossils are obvious: a leg of a dino
At any moment in history, we can understand saur, a giant, complete bone. Smaller bits can be telling, too. For mammals, which
some aspects of the world but not everything. I study, you can tell a lot from the shape of a single tooth. And we can combine this
When a revolutionary change brings in more of information with genetics by using DNA samples from living creatures that we
the larger picture, we have to reconfigure what think are related to the fossils, based on anatomy and other clues.
we knew. The old things are still part of the truth We don’t do these investigations just to reconstruct past worlds but also to see
but have to be spun around and put back into the what they can tell us about the world we live in. There was a huge spike in temper
larger picture in a new way. ature 55 million years ago, for example. It was nothing like today, but still, we’ve
found radical changes in the animals and plants from that era. We can compare
Nima Arkani-Hamed, a professor in the School
those changes to see how related creatures may respond to current climate change.
of Natural Sciences at the Institute for Advanced
Study in Princeton, N.J. Anjali Goswami, a professor and research leader at the Natural History Museum in London

“projections” or “scenarios” that, in con-

trast to predictions, are forecasts that de
pend on the course of action we will take.
That is, after all, the reason we consult
models: so we can decide what to do. But
because we cannot predict political deci-
sions themselves, the actual future trend
is necessarily unpredictable.
This has become one of the major dif-
ficulties in explaining pandemic models.
Dire predictions in March 2020 for
COVID’s global death toll did not come
true. But they were projections for the
case in which we took no measures; they
were not predictions.
Political decisions are not the only
OPINION reason a model may make merely con-
The Truth about

tingent projections rather than definite
predictions. Trends of global warming,
for example, depend on the frequency
and severity of volcanic eruptions, which
Scientific Models
themselves cannot currently be predict-
ed. They also depend on technological
progress, which itself depends on eco-
nomic prosperity, which in turn depends
on, among many other things, whether
society is in the grasp of a pandemic.
They don’t necessarily try to predict Sometimes asking for predictions is re-
what will happen—but they can help us ally asking for too much.
Predictions are also not enough to
understand possible futures make for good science. Recall how each
time a natural catastrophe happens, it
turns out to have been “predicted” in a
By Sabine Hossenfelder movie or a book. Given that most natural
catastrophes are predictable to the ex-
A
tent that “eventually something like this
s COVID- 19 claimed victims at the start of the will happen,” this is hardly surprising. But
pandemic, scientific models made headlines. We these are not predictions; they are scien-
tifically meaningless prophecies because
needed such models to make informed decisions. But they are not based on a model whose
how can we tell whether they can be trusted? The phi- methodology can be reproduced, and no
losophy of science, it seems, has become a matter of life one has tested whether the prophecies
were better than random guesses.
or death. Whether we are talking about traffic noise from a new Thus, predictions are neither neces-
highway, climate change or a pandemic, scientists rely on mod- sary for a good scientific model nor suffi-
els, which are simplified, mathematical representations of the real cient to judge one. But why, then, were
the philosophers so adamant that good
world. Models are approximations and omit details, but a good science needs to make predictions? It’s
model will robustly output the quantities it was developed for. not that they were wrong. It’s just that
they were trying to address a different
Models do not always predict the fu- that correct predictions are a way of tell- problem than what we are facing now.
ture. This does not make them unscientif- ing science from pseudoscience. Scientists tell good models from bad
ic, but it makes them a target for science But correct predictions alone don’t ones by using statistical methods that are
skeptics. I cannot even blame the skep- make for a good scientific model. And the hard to communicate without equations.
tics, because scientists frequently praise opposite is also true: a model can be good These methods depend on the type of
correct predictions to prove a model’s science without ever making predictions. model, the amount of data and the field
worth. It isn’t originally their idea. Many Indeed, the models that matter most for of research. In short, it’s difficult. The
eminent philosophers of science, includ- political discourse are those that do not rough answer is that a good scientific
ing Karl Popper and Imre Lakatos, opined make predictions. Instead they produce model accurately explains a lot of data

with few assumptions. The fewer the as- Co., who thought it is bad scientific prac- reasons for the skewed number of papers
sumptions and the better the fit to data, tice. But of course, a good scientist up that have nothing to do with scientific
the better the model. dates their model when they get new merit. Now we are counting heads and
But the philosophers were not con- data! This is the essence of the scientific quibbling about the ethics of scientific
cerned with quantifying explanatory pow- method: When you learn something new, publishing rather than talking science.
er. They were looking for a way to tell revise. In practice, this usually means re- What went wrong? Predictions are the
good science from bad science without calibrating model parameters with new wrong argument.
having to dissect scientific details. And data. This is why we saw regular updates A better answer to deniers is that cli-
although correct predictions may not tell of COVID case projections. What a scien- mate models explain loads of data with
you whether a model is good science, tist is not supposed to do is add so many few assumptions. The computationally
they increase trust in the scientists’ con- assumptions that their model can fit any simplest explanation for our observations
clusions because predictions prevent sci- data. This would be a model with no ex is that the trends are caused by human
entists from adding assumptions after planatory power. carbon dioxide emission. It’s the hypothe-
they have seen the data. Thus, asking for Understanding the role of predictions sis that has the most explanatory power.
predictions is a good rule of thumb, but it in science also matters for climate mod- In summary, to judge a scientific mod-
is a crude and error-prone criterion. And els. These models have correctly predict- el, do not ask for predictions. Ask instead
fundamentally it makes no sense. A mod- ed many observed trends, from the in- to what degree the data are explained by
el either accurately describes nature or crease of surface temperature, to strato- the model and how many assumptions
doesn’t. At which moment in time a sci- spheric cooling, to sea ice melting. This were necessary for this. And most of all,
entist made a calculation is irrelevant for fact is often used by scientists against cli- do not judge a model by whether you like
the model’s relation to nature. mate change deniers. But the deniers then what it tells you.
A confusion closely related to the idea come back with some papers that made
that good science must make predictions wrong predictions. In response, the sci- Sabine Hossenfelder is a physicist and research fellow
is the belief that scientists should not up- entists point out the wrong predictions at the Frankfurt Institute for Advanced Studies in Germany.
Getty Images
date a model when new data come in. were few and far between. The deniers She currently works on dark matter and the foundations
This can also be traced back to Popper & counter there may have been all kinds of of quantum mechanics.

How Much Can We Know?

The reach of the scientific method is constrained by
the limitations of our tools and the intrinsic impenetrability
of some of nature’s deepest questions
By Marcelo Gleiser
Illustration by Jessica Fortner
W
h at w e o b s e rv e i s n o t nat u r e i n i t s e l f as our tools of exploration transform.
This trend is the trademark of science.
but nature exposed to our method of question Sometimes people take this state
ing,” wrote German physicist Werner Heisen ment about the limitation of scientific
berg, who was the first to fathom the uncertainty knowledge as being defeatist: “If we can’t
get to the bottom of things, why bother?”
inherent in quantum physics. To those who This kind of response is misplaced. There
think of science as a direct path to the truth about the world, is nothing defeatist in understanding the
this quote must be surprising, perhaps even upsetting. Is Heisen limitations of the scientific approach to
knowledge. Science remains our best
berg saying that our scientific theories are contingent on us as methodology to build consensus about
observers? If he is, and we take him seriously, does this mean the workings of nature. What should
that what we call scientific truth is nothing but a big illusion? change is a sense of scientific triumph
alism—the belief that no question is be
People will quickly counterstrike with mathematical or conceptual models to yond the reach of scientific discourse.
something like: Why do airplanes fly or understand it better. And to do this, we There are clear unknowables in sci
antibiotics work? Why are we able to need tools that extend into realms be ence—reasonable questions that, unless
build machines that process information yond our sensorial reach: the very small, currently accepted laws of nature are vi
with such amazing efficiency? Surely, the very fast, the very distant and the vir olated, we cannot find answers to. One
such inventions and so many others are tually inaccessible, such as what is inside example is the multiverse: the conjecture
based on laws of nature that function in the brain or buried in Earth’s core. What that our universe is but one among a
dependently of us. There is order in the we observe is not nature itself but nature multitude of others, each potentially
universe, and science gradually uncovers as discerned through data we have col with a different set of laws of nature.
this order. lected from machines. In consequence, Other universes lie outside our causal
No question about it: There is order the scientific worldview depends on the horizon, meaning that we cannot receive
in the universe, and much of science is information we can acquire through our or send signals to them. Any evidence for
about finding patterns of behavior—from instruments. And given that our tools their existence would be circumstan
quarks to mammals to galaxies—that we are limited, our view of the world is nec tial—for example, scars in the radiation
translate into general laws. We strip essarily myopic. We can see only so far permeating space because of a past colli
away unnecessary complications and fo into the nature of things, and our ever sion with a neighboring universe.
cus on what is essential, the core proper shifting scientific worldview reflects this Other examples of unknowables can
ties of the system we are studying. We fundamental limitation on how we per be conflated into three categories about
then build a descriptive narrative of how ceive reality. origins: of the universe, of life and of the
the system behaves, which, in the best Just think of biology before and after mind. Scientific accounts of the origin of
cases, is also predictive. the microscope or gene sequencing, or the universe are incomplete because they
Often overlooked in the excitement of of astronomy before and after the tele must rely on a conceptual framework to
research is that the methodology of sci scope, or of particle physics before and even begin to work: energy conservation,
ence requires interaction with the sys after colliders or fast electronics. Now, as relativity, quantum physics, for instance.
tem we are studying. We observe its be in the 17th century, the theories we build Why does the universe operate under
havior, measure its properties, and build and the worldviews we construct change these laws and not others?

Similarly, unless we can prove that How do we establish—as opposed to con we experience reality. This is the playing
only one or very few biochemical path jecture—that something is conscious? field where the game of science unfolds,
ways exist from nonlife to life, we cannot Paradoxically, it is through our con and if we play by the rules we can see
know for sure how life originated on sciousness that we make sense of the only so much of what lies beyond.
Earth. For consciousness, the problem is world, even if only imperfectly. Can we
the jump from the material to the subjec fully understand something of which we Marcelo Gleiser i s Appleton Professor of Natural
tive—for example, from firing neurons to are a part? Like the mythic snake that Philosophy and a professor of physics and astronomy
the experience of pain or the color red. bites its own tail, we are stuck within a at Dartmouth College. He has authored several books,
Perhaps some kind of rudimentary con circle that begins and ends with our lived including T he Island of Knowledge: The Limits of Science
sciousness could emerge in a sufficiently experience of the world. We cannot de and the Search for Meaning ( Basic Books, 2014). In 2019
complex machine. But how could we tell? tach our descriptions of reality from how he was awarded the Templeton Prize.

END NOTE
Fake-News Sharers
Highly impulsive people who lean conservative are most likely
to pass along false news stories
By Asher Lawson and Hemant Kakkar
Behavioral and political scientists have pointed fingers at polit- as their support of conservative issues, support for Donald Trump,
ical conservatives, as opposed to liberals, when it comes to trust in mainstream media and time spent on social media. LCCs,
spreading fake news stories. But not all conservatives do it, and we learned, expressed a general desire for chaos, and this need
sweeping generalizations threaten to condemn everyone who may explain their proclivity to spread misinformation. Other fac-
subscribes to conservative values. This approach risks even more tors, including support for Trump, were not as strongly related.
dangerous polarization. Unfortunately, our work on this personality trait also suggests
Political leanings are far from the only determinants of be- that accuracy labels on news stories will not solve the problem of
havior. Personality is a crucial influence, so our research on mis- misinformation. We ran a study where we explicitly stated wheth-
information sharing has focused on that. One widely used psy- er each news story in question was false, using a “disputed” tag
chological system for identifying personality traits organizes commonly seen on social media, or true, using a “supported” tag.
them into five categories: openness to experience, conscientious- We found that the supported tag increased the rate at which real
ness, extroversion, agreeableness and stories were shared among
neuroticism. (It is called, unsur- both liberals and conservatives.
prisingly, the five-factor theory.) LCCs, however, continued to
We looked specifically at con- share misinformation at a great-
scientiousness, which cap- er rate despite the clear warn-
tures differences in peo- ings that the stories were false.
ple’s orderliness, impulse We ran another study that in-
control, conventionality volved explicitly telling partici-
and reliability. pants that an article they want-
In a series of eight studies ed to share was inaccurate. Peo-
with a total of 4,642 partici- ple then had the chance to
pants, we examined wheth- change their choice. Not only did
er low-conscientiousness LCCs still share fake news at a
conservatives (LCCs) dis- higher rate than others in the
seminate more misinfor- study, but they also were compar-
mation than other conserva- atively insensitive to direct warn-
tives or low-conscientiousness ings that the stories they wanted to
liberals. First we determined peo- share were false.
ple’s political ideology and The poor effectiveness of warn-
conscientiousness through ings among LCCs is worrying because our
assessments that asked par- research suggests these people are primary drivers of fake-
ticipants about their values and news proliferation. Social media networks therefore need to find
behaviors. We then showed the same people a series of real and a different solution than just tagging stories with warning labels.
fake news stories relating to C OVID and asked them to rate how Interventions based on the assumption that truth matters to read-
accurate the stories were. We also asked whether they would ers may be inadequate. Another option might involve social
consider sharing each story. media companies monitoring fake news that has the potential to
Both liberals and conservatives sometimes saw false stories hurt others, such as misinformation related to vaccines and elec-
as accurate. This error was likely driven in part by their w anting tions, and actively removing such content from their platforms.
certain stories to be true because they aligned with their beliefs. Whatever the case, until these companies find an approach
But actually sharing false news was markedly higher among that works, this problem will persist. In the interim, our society
LCCs compared with everyone else in the study, although some will pay the cost of spreading misinformation. The long, con-
people of all persuasions did it. There was no difference between spiratorial road that rioters followed to the January 2021 Capi-
liberals and conservatives with high levels of conscientiousness. tol insurrection shows that this spread can have serious and
Low-conscientiousness liberals did not share more misinforma- damaging consequences.
tion than their high-conscientiousness liberal counterparts.
What explains the exceptional tendency of LCCs to share fake Asher Lawson is an assistant professor of decision sciences at INSEAD
news? To explore this question, we gathered information about in Fontainebleau, France.
participants’ politics and personalities and administered ques-
tionnaires to assess their need for chaos—the desire to disrupt Hemant Kakkar i s an assistant professor of management and organizations
and destroy the existing political and social institutions—as well at Duke University’s Fuqua School of Business.
116 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022 Illustration by Ross MacDonald

Scientific American Special Collectors Edition - Fall 2022

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Scientific American Special Collectors Edition - Fall 2022

Uploaded by

Copyright:

Available Formats

SPECIAL COLLECTOR’S EDITION

© 2022 Scientific American

• Fac g antiscie ehind cons ality •

Truth vs. Lies is published by

Advertising Production Manager: Andrea Gawrylewski

© 2022 Scientific American

Volume 32, Number 5, Fall 2022

GRAPPLING SOCIAL MEDIA’S

2 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

© 2022 Scientific American

© 2022 Scientific American

“We do not see things as they are, we see them as we are.”

4 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

© 2022 Scientific American

© 2022 Scientific American

n the 10 th of April 2019 Pope Francis, President Salva Kiir o f

6 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

© 2022 Scientific American

© 2022 Scientific American

8 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

© 2022 Scientific American

agree about our hallucinations, maybe

ciples apply also at a deeper level, under-

© 2022 Scientific American

© 2022 Scientific American

While we are working

© 2022 Scientific American

n a classic 1942 experiment, American psychologist Abraham Luchins

12 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

© 2022 Scientific American

© 2022 Scientific American

Much More Than Move 1 Move 2 Move 3

14 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022 Graphic by George Retseck

© 2022 Scientific American

clinician’s di­­agnosis can block the second from seeing impor-

ture, remember that if you already think you know the

20 Merim Bilalić is a professor of cognitive psychology at the University of Northumbria

© 2022 Scientific American

16 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

© 2022 Scientific American

work on average. nounce that white job applicants should

© 2022 Scientific American

18 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

© 2022 Scientific American

© 2022 Scientific American

20 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

© 2022 Scientific American

© 2022 Scientific American

22 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

© 2022 Scientific American

© 2022 Scientific American

24 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

© 2022 Scientific American

Illustrations by Taylor Callery

© 2022 Scientific American

26 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

© 2022 Scientific American

T h e l a s t t i m e the board overhauled the

© 2022 Scientific American

28 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

© 2022 Scientific American

and seeks to delay action on it.

© 2022 Scientific American

30 | SCIENTIFIC AMERICAN | SPECIAL EDITION | FALL 2022

clinician’s diagnosis can block the second from seeing impor-