The Ascents and Collapses of Artificial Intelligence

Melanie Mitchell
Portland State University
and Santa Fe Institute
“A year from now, we’ll have over a million cars with full self-driving,
software, everything.” — Elon Musk, 2019
“Perhaps expectations are too high, and... this will eventually result in
disaster…. [S]uppose that five years from now [funding] collapses
miserably as autonomous vehicles fail to roll. Every startup company
fails. And there's a big backlash so that you can't get money for
anything connected with AI. Everybody hurriedly changes the names
of their research projects to something else.

This condition [is] called the ‘AI Winter.’”

“Perhaps expectations are too high, and... this will eventually result in
disaster…. [S]uppose that five years from now [funding] collapses
miserably as autonomous vehicles fail to roll. Every startup company
fails. And there's a big backlash so that you can't get money for
anything connected with AI. Everybody hurriedly changes the names
of their research projects to something else.

This condition [is] called the ‘AI Winter.’”

—Drew McDermott, 1984

Outlook for AI

AI Winter
AI Winter

1960 1970 1980 1990 2000 2010 2020

Outlook for AI ?
AI Winter
AI Winter

1960 1970 1980 1990 2000 2010 2020

Frank Rosenblatt
The Mark I Perceptron,
1957

Outlook for AI

1960 1970 1980 1990 2000 2010 2020

 Frank Rosenblatt
The Mark I Perceptron,
1957
“The Navy revealed the embryo
of an electronic computer today
that it expects will be able to
walk, talk, see, write, reproduce Outlook for AI
itself, and be conscious of its
existence.”

— New York Times, July, 1958 1960 1970 1980 1990 2000 2010 2020
I confidently expect that
within a matter of 10 or 15
years, something will Outlook for AI
emerge from the
laboratory which is not too
far from the robot of
science fiction fame.
— Claude Shannon, 1961
1960 1970 1980 1990 2000 2010 2020
 Machines will be
capable, within
twenty years, of doing
any work that
a man can do.
— Herbert Simon, 1965
I confidently expect that
within a matter of 10 or 15
years, something will Outlook for AI
emerge from the
laboratory which is not too
far from the robot of
science fiction fame.
— Claude Shannon, 1961
1960 1970 1980 1990 2000 2010 2020
 Machines will be
capable, within Within a generation...the
twenty years, of doing problem of creating ‘artificial
any work that intelligence’ will be
a man can do. substantially solved.
— Herbert Simon, 1965 — Marvin Minsky, 1967
I confidently expect that
within a matter of 10 or 15
years, something will Outlook for AI
emerge from the
laboratory which is not too
far from the robot of
science fiction fame.
— Claude Shannon, 1961
1960 1970 1980 1990 2000 2010 2020
[The results are] wholly discouraging about general-
purpose programs seeking to mimic the problem-
solving aspects of human [brain] activity over a rather
wide field. Such a general-purpose program, the
coveted long-term goal of AI activity, seems as remote
as ever.

— James Lighthill, 1973

Outlook for AI

1960 1970 1980 1990 2000 2010 2020

 [The results are] wholly discouraging about general-
purpose programs seeking to mimic the problem-
solving aspects of human [brain] activity over a rather
wide field. Such a general-purpose program, the
coveted long-term goal of AI activity, seems as remote
as ever.

— James Lighthill, 1973

“AI research is unlikely to produce military applications in

the foreseeable future.”

— American Study Group, 1973

Outlook for AI

1960 1970 1980 1990 2000 2010 2020

 Outlook for AI

1960 1970 1980 1990 2000 2010 2020

 The Rise of Expert Systems

From XCON (1982):

If:
• the most current active context is distributing
massbus devices, and
• there is a single-port disk drive that has not been
assigned to a massbus, and
• there are no unassigned dual-port disk drives, and
• the number of devices that each massbus should support is known, and
• there is a massbus that has been assigned at least one disk drive that
should support additional disk drives, and
• the type of cable needed to connect the disk drive to the previous
device on the massbus is known
Then: assign the disk drive to the massbus.
Outlook for AI

Adapted from
http://www.cs.utexas.edu/users/ea
r/ugs302/Lectures/L10-
1960 1970 1980 1990 2000 2010 2020
ExpertSystems.ppt
The Rise of Neural Networks and Machine Learning

Published 1986

Outlook for AI

1960 1970 1980 1990 2000 2010 2020

, 1987

Outlook for AI

1960 1970 1980 1990 2000 2010 2020

, 1987

I believe that this [neural network]

technology which we are about to
embark upon is more important than the
atom bomb.

— Jasper Lupo, director of DARPAs

Tactical Technology Office, 1988

Outlook for AI

1960 1970 1980 1990 2000 2010 2020

Expert systems are brittle and unreliable

Neural networks are hard to train, don’t seem

to scale to complex problems

Outlook for AI

1960 1970 1980 1990 2000 2010 2020

 Outlook for AI

1960 1970 1980 1990 2000 2010 2020

1990: I get out of graduate school.

Outlook for AI

1960 1970 1980 1990 2000 2010 2020

1990: I get out of graduate school.

I’m advised not to use the term “artificial intelligence” on my job

applications.

Outlook for AI

1960 1970 1980 1990 2000 2010 2020

 Outlook for AI

1960 1970 1980 1990 2000 2010 2020

 Outlook for AI

1960 1970 1980 1990 2000 2010 2020

 The Deep Learning Revolution

All knowledge is learned from examples/experience,

and is encoded as weights.

Outlook for AI

https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/
1960 1970 1980 1990 2000 2010 2020
 Outlook for AI

1960 1970 1980 1990 2000 2010 2020

Speech Recognition Error Rate

Outlook for AI

1960 1970 1980 1990 2000 2010 2020

 Performance in “Reading Comprehension”

(Stanford Question-Answering Dataset)

“human level”

Outlook for AI

2018 2019 1960 1970 1980 1990 2000 2010 2020

 Outlook for AI

1960 1970 1980 1990 2000 2010 2020

 Outlook for AI

1960 1970 1980 1990 2000 2010 2020

 Outlook for AI

1960 1970 1980 1990 2000 2010 2020

Google’s new service translates
languages almost as well as
humans can.

Outlook for AI

1960 1970 1980 1990 2000 2010 2020

DeepMind’s
Deep Q-Learning

Atari video games

Human Level

Mnih, V. et al. "Human-level control through deep reinforcement learning." Nature 518, no. 7540
(2015): 529.
Deep Q-Learning discovers “tunneling”
Human-level AI will be passed in
the mid-2020s.

— Shane Legg, 2016

(Co-Founder and Chief Scientist,
Google DeepMind)
Human-level AI will be passed in
the mid-2020s.
— Shane Legg, 2016
(Co-Founder and Chief Scientist,
Google DeepMind)
I set the date for the Singularity . . .
as 2045. The nonbiological
intelligence created in that year will
be one billion times more powerful
than all human intelligence today.
— Ray Kurzweil
 ?
Outlook for AI

1960 1970 1980 1990 2000 2010 2020

[In the quest to create human-level
AI] there has been almost no
progress.
?
Outlook for AI
— Gary Marcus, 2016
(Professor, NYU and Founder,
Geometric Intelligence)

1960 1970 1980 1990 2000 2010 2020

 Some cracks in the foundation
of deep learning

• Brittleness and vulnerability to attacks

• Lack of generalization, abstraction, “transfer learning”

• Lack of “common sense”

 Deep neural networks make unhuman-like errors
if inputs differ from training data
Alcorn, Michael A., et al. "Strike (with) a Pose: Neural Networks Are Easily
Fooled by Strange Poses of Familiar Objects." arXiv preprint arXiv:1811.11553
(2018).
Automated captioning systems are unreliable

“A group of people sitting at a bus stop”

— Microsoft captionbot
Translation systems can ignore important context
Attacks on Image Classification Systems

From http://cs231n.stanford.edu/
Attacks on Autonomous Driving Systems
Target: “Speed Limit 80”

Evtimov et al., “Robust Physical-World Attacks on Deep

Learning Models”, 2017
 Attacks on Speech Recognition Systems

2016

“Okay Google, browse to evil.com”

 Attacks on Question-Answering Systems

Jia & Liang, “Adversarial Examples for Evaluating Reading Comprehension

Systems”, 2017
Deep reinforcement learning sometimes is not able to transfer to
slightly different scenarios

Breakout with
Standard Breakout Paddle shifted up

Kansky, K. et al., 2017. Schema networks: Zero-shot transfer with a generative causal model of intuitive
physics. arXiv preprint arXiv:1706.04317.
Deep neural networks have trouble with abstraction

#18 #19

Bongard Problems (1967)

Deep neural networks have trouble with abstraction

#84 #91

Bongard Problems (1967)

 Deep neural networks have trouble with abstraction

For each class, network is trained on 20,000 examples. Then

tested on 10,000 new examples.

Stabinger, S., Rodríguez-Sánchez, A., & Piater, J. 25 years of CNNs: Can we compare to
human abstraction capabilities?. In International Conference on Artificial Neural Networks
(pp. 380-387). Springer, 2016.
 Deep neural networks have trouble with abstraction

For each class, network is trained on 20,000 examples. Then

tested on 10,000 new examples.

Method Accuracy
LeNet .57
Google LeNet .5
Humans .98

Stabinger, S., Rodríguez-Sánchez, A., & Piater, J. 25 years of CNNs: Can we compare to human abstraction capabilities?.
In International Conference on Artificial Neural Networks (pp. 380-387). Springer, 2016.
https://blog.piekniewski.info/2018/05/28/ai-winter-is-well-on-its-way/
Four AI Fallacies
 Four AI Fallacies

Narrow AI is on a continuum with general AI

 Four AI Fallacies

Narrow AI is on a continuum with general AI

— Related to Hubert Dreyfus’s “First step fallacies"

 Four AI Fallacies
Easy things are easy and hard things are hard
 Four AI Fallacies
Easy things are easy and hard things are hard

— Herbert Simon: “Everything of interest in cognition

happens above the 100-millisecond level.”
 Four AI Fallacies
Easy things are easy and hard things are hard

— Herbert Simon: “Everything of interest in cognition

happens above the 100-millisecond level.”

— Andrew Ng: “If a typical person can do a mental task

with less than one second of thought, we can probably
automate it using AI either now or in the near future.”
 Four AI Fallacies
Easy things are easy and hard things are hard

— Herbert Simon: “Everything of interest in cognition

happens above the 100-millisecond level.”

— Andrew Ng: “If a typical person can do a mental task

with less than one second of thought, we can probably
automate it using AI either now or in the near future.”

— Demis Hassibis et al.: Go is one of “the most

challenging of domains”
 Four AI Fallacies
Names confer abilities
 Four AI Fallacies
Names confer abilities

— Benchmark datasets called

“object recognition”
“reading comprehension”
“question answering”
“common sense understanding”.
 Four AI Fallacies
Names confer abilities

— Benchmark datasets called

“object recognition”
“reading comprehension”
“question answering”
“common sense understanding”.

— Methods called “deep learning”, “neural networks”

 Four AI Fallacies
Names confer abilities

— Benchmark datasets called

“object recognition”
“reading comprehension”
“question answering”
“common sense understanding”.

— Methods called “deep learning”, “neural networks”

— “Overattributions” in explanations of what networks

learned (e.g., “tunneling”)
 Four AI Fallacies

Intelligence is all in the brain

 Four AI Fallacies

Intelligence is all in the brain

— Allen Newell and Herbert Simon: “A physical symbol

system has the necessary and sufficient means for
general intelligent action.”
 Four AI Fallacies

Intelligence is all in the brain

— Allen Newell and Herbert Simon: “A physical symbol

system has the necessary and sufficient means for
general intelligent action.”

— Geoffrey Hinton: “I think you can capture a thought by

a vector.”
Some steps to more robust and human-like AI

• Generative, human-like concepts

• Active, dynamic perception

• Predictive coding and causal models

• Abstraction, analogy, metaphor

• Developmental foundations of common sense

Learning generative, human-like concepts
Lake et al. (2015). Human-level concept learning through probabilistic program
induction. Science, 350(6266), 1332-1338.
Active Perception and Analogies for
Situation Recognition

Quinn et al. (2018). Semantic image retrieval via

active grounding of visual situations. In 2018
IEEE 12th International Conference on Semantic
Computing (ICSC) (pp. 172-179). IEEE.
Active Perception and Analogies for
Situation Recognition

Quinn et al. (2018). Semantic image retrieval via

active grounding of visual situations. In 2018
IEEE 12th International Conference on Semantic
Computing (ICSC) (pp. 172-179). IEEE.
Active Perception and Analogies for
Situation Recognition

Quinn et al. (2018). Semantic image retrieval via

active grounding of visual situations. In 2018
IEEE 12th International Conference on Semantic
Computing (ICSC) (pp. 172-179). IEEE.
Active Perception and Analogies for
Situation Recognition

Quinn et al. (2018). Semantic image retrieval via

active grounding of visual situations. In 2018
IEEE 12th International Conference on Semantic
Computing (ICSC) (pp. 172-179). IEEE.
Active Perception and Analogies for
Situation Recognition

Quinn et al. (2018). Semantic image retrieval via

active grounding of visual situations. In 2018
IEEE 12th International Conference on Semantic
Computing (ICSC) (pp. 172-179). IEEE.
 Foundations of Common Sense

Winograd Schema “Common Sense” Challenge

Levesque, Hector, Ernest Davis, and Leora Morgenstern. "The Winograd Schema
challenge." Thirteenth International Conference on the Principles of Knowledge
Representation and Reasoning. 2012.

I poured water from the bottle I poured water from the bottle into
into the cup until it was full. the cup until it was empty.

What was full? What was empty?

The steel ball hit the glass table The glass ball hit the steel table
and it shattered. and it shattered.

What shattered? What shattered?

State-of-the-art AI: ~60% (vs. 50% with random
guessing)

Humans: 100% (if paying attention)

State-of-the-art AI: ~60% (vs. 50% with random
guessing)

Humans: 100% (if paying attention)

“When AI can’t determine what ‘it’ refers to in a sentence,

it’s hard to believe that it will take over the world.”

— Oren Etzioni, Allen Institute for AI

https://www.seattletimes.com/business/technology/paul-allen-invests-125-million-to-teach-computers-common-sense/

https://allenai.org/alexandria/
 DARPA “Foundations of Common Sense” Challenge

https://www.fbo.gov/index.php?s=opportunity&mode=form&id=f98476244ba0c06de9e0b38bfe75f54d&tab=cor
e&_cview=0
Summary
 Summary

• AI has a history of ascents and collapses: optimistic predictions

followed by disappointments
 Summary

• AI has a history of ascents and collapses: optimistic predictions

followed by disappointments

• Will our current “AI Spring”, full of optimistic predictions, be

followed by another “AI Winter”?
 Summary

• AI has a history of ascents and collapses: optimistic predictions

followed by disappointments

• Will our current “AI Spring”, full of optimistic predictions, be

followed by another “AI Winter”?

• Deep learning has had huge success in narrow domains, but

fails at some fundamental aspects of general intelligence.
 Summary

• AI has a history of ascents and collapses: optimistic predictions

followed by disappointments

• Will our current “AI Spring”, full of optimistic predictions, be

followed by another “AI Winter”?

• Deep learning has had huge success in narrow domains, but

fails at some fundamental aspects of general intelligence.

• Will these struggles be solved via more layers and more data, or
is something fundamentally different needed?
Coming this Fall!