Scary AI Is More “Fantasia” Than “Terminator”

When Nate Soares psychoanalyzes himself, he sounds less Freudian than Spockian. As a boy, he’d see people acting in ways he never would “unless I was acting maliciously,” the former Google software engineer, who now heads the non-profit Machine Intelligence Research Institute, reflected in a blog post last year. “I would automatically, on a gut level, assume that the other person must be malicious.” It’s a habit anyone who’s read or heard David Foster Wallace’s “This is Water” speech will recognize.

Later Soares realized this folly when his “models of other people” became “sufficiently diverse”—which isn’t to say they’re foolproof, he wrote in the same post. “I’m probably still prone to occasionally believing someone is malicious when they’re merely different than me, especially in cases where they act similarly to me most of the time (thereby fooling my gut-level person-modeler into modeling them too much like myself).” He suspected that this “failure mode” links up with the “typical mind” fallacy, and is “therefore,” he concluded, “difficult to beat in general.”

Beating biases is one of Soares’ core concerns—“I care a lot about having accurate beliefs,” he says—and one of Neil deGrasse Tyson that keeping a super intelligent machine “in a box,” disconnected from the Internet, was a lousy safeguard for a misaligned AI and Steven Pinker’s misconceptions about AI safety. Donors find the Institute’s mission worthwhile: In December 2017, 341 of them it $2.5 million.

You're reading a preview, sign up to read more.

More from Nautilus

Nautilus9 min read
Consciousness Doesn’t Depend on Language: We share the basic experience of life with all mammals.
The contrast could not have been starker—here was one of the world’s most revered figures, His Holiness the Fourteenth Dalai Lama, expressing his belief that all life is sentient, while I, as a card-carrying neuroscientist, presented the contemporary
Nautilus10 min readTech
Would You Survive a Merger with AI?: The cost of brain enhancement may be your identity.
The idea that humans should merge with AI is very much in the air these days. It is offered both as a way for humans to avoid being outmoded by AI in the workplace, and as a path to superintelligence and immortality. For instance, Elon Musk recently
Nautilus7 min read
Why Symbols Aren’t Forever: The removal of cultural emblems is not the erasure of history but part of it.
In November 2016, a swastika was painted on an elementary school in my Denver, Colorado, neighborhood of Stapleton. As an archaeologist who specializes in identifying the remains of animals hunted by early humans, my work doesn’t often involve symbol