ARTIFICIAL INTELLIGENCE: A

THREAT TO HUMANITY

Artificial Intelligence and the Threat to Humanity

The robots we create, may decide we are an obsolete model,
deserving “retirement”.
Stephen Hawking had warned a few years back that Artificial Intelligence (AI) could
spell the end of human race, a threat echoed by Elon Musk, the founder of Tesla, Bill
Gates, and many others. This is the apocalyptic vision that the robots we create,
may decide we are an obsolete model deserving “retirement”. The truth, a more
banal one, is that are we letting machine algorithms take over what were human
decisions earlier, decision making of the governments, the businesses and even
ours.
Today, algorithms decide who should get a job, which part of a city needs to be
developed, who should get into a college, and in the case of a crime, what should be
the sentence. It is not super intelligence of robots that is the threat to life as we know
it, but machines taking over thousands decisions that are critical to us and deciding
social outcomes.
What are these decisions, and how are machines making them?
Suppose you apply for a loan. The huge amount of financial data you create – credit
card transactions, banking transactions, ATM withdrawals – all of these data are
accessed and processed by computer algorithms. With linking Aadhaar to your bank
accounts and PAN cards, almost every transaction that you make over a certain
amount in India, is available for such processing. This past data is stored for ever – it
is cheaper to store all the data then selectively keep some, deleting others. All these
are processed by the algorithms to decide a single score your creditworthiness.
Based on this final score, you are ranked and decision to give a loan is taken.
The issue here is a simple one. What decides your getting a loan or not is finally a
machine score – not who you are, what you have achieved, how important is your
work for the country (or society); for the machine, you are just the sum of all your
transactions to be processed and reduced to a simple number.
So how does the machine do it? It is the algorithms that we write for the machines
that take our data and provides a score for the decision. Those who are scored,
have no control over such decisions. There is no appeal either. The algorithms are
supposedly intellectual property, and therefore their secrets are jealously guarded.
The worst part is that some of the algorithms are not even understandable to those
who have written them; even the creators of such algorithms do not know how a
particular algorithm came out with a specific score!
A mathematician and a data scientist Cathy O'neil, in recent a book, Weapons of
Math Destruction, tells us that the apparent objectivity of processing the huge
amount of data by algorithms is false. The algorithms themselves are nothing but our
biases and subjectiveness that are being coded – “They are just opinions coded into
maths”.
What happens when we transform the huge data that we create through our every
day digital footprints into machine “opinions” or “decisions”? Google served ads for
high paying jobs disproportionately to men; African Americans got longer
sentences as they were flagged as high risk for repeat offences by a judicial risk
assessment algorithm. It did not explicitly use the race of the offender, but used
where they stayed, information of other family members, education and income to
work out the risk, all of which put together, was also a proxy for race.
One of the worst examples in the US of such scoring, was in calculating insurance
premiums for drivers. A researcher found that drivers with low-income and no
drunken driving cases, were asked to pay a higher premium than high-income
persons, but with drunken driving cases. This was an obvious anomaly – drinking
and driving is likely to be repeated and is an obvious risk for accidents. The
algorithm had “correctly” deduced that low income persons had less capacity for
insurance shopping than the high income ones, and therefore could be rooked for a
higher premium. The algorithm was looking not safe drivers, but maximising the
profit of the insurance company.
The problem is not just the subjective biases of the people who code the algorithms,
or the goal of the algorithm, but much deeper. They lie in the data and the so-called
predictive models we build using this data. Such data and models simply reflect the
objective reality of the high degree of inequality that exist within society, and
replicates that in the future through its predictions.
What are predictive models? Simply put, we use the past to predict the future. We
use the vast amount of data that are available, to create models that correlate the
“desired” output with a series of input data. The output could be a credit score, the
chance of doing well in a university, a job and so on. The past data of people who
have been “successful” – some specific output variables – are selected as indicators
of success and correlated with various social and economic data of the candidate.
This correlation is then used to rank any new candidate in terms of chances of
success based on her or his profile. To use an analogy, predictive models are like
driving cars looking only at the rear-view mirror.
A score for success, be it a job, admission to a university, or a prison sentence,
reflects the existing inequality of society in some form. An African American in the
US, or a dalit, or a Muslim in India, does not have to be identified by race, caste or
religion. The data of her or his social transactions are already prejudiced and biased.
Any scoring algorithm will end up with a score that will predict their future success
based on which groups are successful today. The danger of these models are that
race or caste or creed may not exist explicitly as data, but a whole host of other data
exist that act as proxies for these “variables”.
Such predictive models are not only biased by the opinion of those who create the
models, but also the inherent nature of all predictive models: it cannot predict what it
does not see. They end up trying to replicate what they see in the past has
succeeded. They are inherently a conservative force trying to replicate the existing
inequalities of society. And like all technology, they amplify what exist today over
what is new in the world.
After the meltdown in 2008 of the financial markets, two modellers Emanuel Derman
and Paul Wilmott had written The Financial Modelers' Manifesto. Modelled on the
lines of the Communist Manifesto, it starts with “A spectre is haunting Markets – the
spectre of illiquidity, frozen credit, and the failure of financial models.” It ends with the
Modellers Hippocratic Oath:
~ I will remember that I didn't make the world, and it doesn't satisfy my equations.
~ Though I will use models boldly to estimate value, I will not be overly impressed by
mathematics.
~ I will never sacrifice reality for elegance without explaining why I have done so.
~ Nor will I give the people who use my model false comfort about its accuracy.
Instead, I will make explicit its assumptions and oversights.
~ I understand that my work may have enormous effects on society and the
economy, many of them beyond my comprehension
The AI community is waking up to the dangers of such models taking over the world.
Some of these are even violations of constitutional guarantees against
discrimination. There are now discussions of creating an Algorithm Safety Board in
the US, such that algorithms can be made transparent and accountable. We should
know what is being coded, and if required, find out why the algorithm came out with
a certain decision: the algorithms should be auditable. It is no longer enough to say,
“the computer did it”.
Similarly, in 2017, a group of AI researchers met in Asilomar and drew up a set of
principles for what should be the goals of AI – common good, no arms race using AI,
shared prosperity. While not as strong as the Asilomar principles drawn up in 1975
that barred certain kind of recombinant DNAtechnology at that time, they still
articulate clearly that AI needs to be developed for public good and be socially
regulated. It is our humanity that is at stake, not the end of the human race as a
Hawking or Musk fears.

BENEFITS & RISKS OF ARTIFICIAL

INTELLIGENCE
“Everything we love about civilization is a product of intelligence, so amplifying our human
intelligence with artificial intelligence has the potential of helping civilization flourish like
never before – as long as we manage to keep the technology beneficial.“
Max Tegmark, President of the Future of Life Institute
WHAT IS AI?
From SIRI to self-driving cars, artificial intelligence (AI) is progressing rapidly. While
science fiction often portrays AI as robots with human-like characteristics, AI can encompass
anything from Google’s search algorithms to IBM’s Watson to autonomous weapons.

Artificial intelligence today is properly known as narrow AI (or weak AI), in that it is
designed to perform a narrow task (e.g. only facial recognition or only internet searches or
only driving a car). However, the long-term goal of many researchers is to create general AI
(AGI or strong AI). While narrow AI may outperform humans at whatever its specific task is,
like playing chess or solving equations, AGI would outperform humans at nearly every
cognitive task.
WHY RESEARCH AI SAFETY?
In the near term, the goal of keeping AI’s impact on society beneficial motivates research in
many areas, from economics and law to technical topics such
as verification, validity, security and control. Whereas it may be little more than a minor
nuisance if your laptop crashes or gets hacked, it becomes all the more important that an AI
system does what you want it to do if it controls your car, your airplane, your pacemaker,
your automated trading system or your power grid. Another short-term challenge
is preventing a devastating arms race in lethal autonomous weapons.
In the long term, an important question is what will happen if the quest for strong AI
succeeds and an AI system becomes better than humans at all cognitive tasks. As pointed out
by I.J. Good in 1965, designing smarter AI systems is itself a cognitive task. Such a system
could potentially undergo recursive self-improvement, triggering an intelligence explosion
leaving human intellect far behind. By inventing revolutionary new technologies, such a
superintelligence might help us eradicate war, disease, and poverty, and so the creation
of strong AI might be the biggest event in human history. Some experts have expressed
concern, though, that it might also be the last, unless we learn to align the goals of the
AI with ours before it becomes superintelligent.

There are some who question whether strong AI will ever be achieved, and others who insist
that the creation of superintelligent AI is guaranteed to be beneficial. At FLI we recognize
both of these possibilities, but also recognize the potential for an artificial intelligence system
to intentionally or unintentionally cause great harm. We believe research today will help us
better prepare for and prevent such potentially negative consequences in the future, thus
enjoying the benefits of AI while avoiding pitfalls.
HOW CAN AI BE DANGEROUS?
Most researchers agree that a superintelligent AI is unlikely to exhibit human emotions like
love or hate, and that there is no reason to expect AI to become intentionally benevolent or
malevolent. Instead, when considering how AI might become a risk, experts think two
scenarios most likely:
1. The AI is programmed to do something devastating: Autonomous weapons are artificial
intelligence systems that are programmed to kill. In the hands of the wrong person, these
weapons could easily cause mass casualties. Moreover, an AI arms race could inadvertently lead
to an AI war that also results in mass casualties. To avoid being thwarted by the enemy, these
weapons would be designed to be extremely difficult to simply “turn off,” so humans could
plausibly lose control of such a situation. This risk is one that’s present even with narrow AI, but
grows as levels of AI intelligence and autonomy increase.
2. The AI is programmed to do something beneficial, but it develops a destructive method
for achieving its goal: This can happen whenever we fail to fully align the AI’s goals with
ours, which is strikingly difficult. If you ask an obedient intelligent car to take you to the
airport as fast as possible, it might get you there chased by helicopters and covered in vomit,
doing not what you wanted but literally what you asked for. If a superintelligent system is
tasked with a ambitious geoengineering project, it might wreak havoc with our ecosystem as a
side effect, and view human attempts to stop it as a threat to be met.
As these examples illustrate, the concern about advanced AI isn’t malevolence but
competence. A super-intelligent AI will be extremely good at accomplishing its goals, and if
those goals aren’t aligned with ours, we have a problem. You’re probably not an evil ant-
hater who steps on ants out of malice, but if you’re in charge of a hydroelectric green energy
project and there’s an anthill in the region to be flooded, too bad for the ants. A key goal of
AI safety research is to never place humanity in the position of those ants.
WHY THE RECENT INTEREST IN AI
SAFETY
Stephen Hawking, Elon Musk, Steve Wozniak, Bill Gates, and many other big names in
science and technology have recently expressed concern in the media and via open letters
about the risks posed by AI, joined by many leading AI researchers. Why is the subject
suddenly in the headlines?
The idea that the quest for strong AI would ultimately succeed was long thought of as science
fiction, centuries or more away. However, thanks to recent breakthroughs, many AI
milestones, which experts viewed as decades away merely five years ago, have now been
reached, making many experts take seriously the possibility of superintelligence in our
lifetime. While some experts still guess that human-level AI is centuries away, most AI
researches at the 2015 Puerto Rico Conference guessed that it would happen before 2060.
Since it may take decades to complete the required safety research, it is prudent to start it
now.

Because AI has the potential to become more intelligent than any human, we have no surefire
way of predicting how it will behave. We can’t use past technological developments as much
of a basis because we’ve never created anything that has the ability to, wittingly or
unwittingly, outsmart us. The best example of what we could face may be our own evolution.
People now control the planet, not because we’re the strongest, fastest or biggest, but because
we’re the smartest. If we’re no longer the smartest, are we assured to remain in control?

FLI’s position is that our civilization will flourish as long as we win the race between the
growing power of technology and the wisdom with which we manage it. In the case of AI
technology, FLI’s position is that the best way to win that race is not to impede the former,
but to accelerate the latter, by supporting AI safety research.

THE TOP MYTHS ABOUT ADVANCED AI

A captivating conversation is taking place about the future of artificial intelligence and what
it will/should mean for humanity. There are fascinating controversies where the world’s
leading experts disagree, such as: AI’s future impact on the job market; if/when human-level
AI will be developed; whether this will lead to an intelligence explosion; and whether this
is something we should welcome or fear. But there are also many examples of of boring
pseudo-controversies caused by people misunderstanding and talking past each other. To help
ourselves focus on the interesting controversies and open questions — and not on the
misunderstandings — let’s clear up some of the most common myths.

TIMELINE MYTHS
The first myth regards the timeline: how long will it take until machines greatly supersede
human-level intelligence? A common misconception is that we know the answer with great
certainty.
One popular myth is that we know we’ll get superhuman AI this century. In fact, history is
full of technological over-hyping. Where are those fusion power plants and flying cars we
were promised we’d have by now? AI has also been repeatedly over-hyped in the past, even
by some of the founders of the field. For example, John McCarthy (who coined the term
“artificial intelligence”), Marvin Minsky, Nathaniel Rochester and Claude Shannon wrote
this overly optimistic forecast about what could be accomplished during two months with
stone-age computers: “We propose that a 2 month, 10 man study of artificial intelligence be
carried out during the summer of 1956 at Dartmouth College […] An attempt will be made to
find how to make machines use language, form abstractions and concepts, solve kinds of
problems now reserved for humans, and improve themselves. We think that a significant
advance can be made in one or more of these problems if a carefully selected group of
scientists work on it together for a summer.”

On the other hand, a popular counter-myth is that we know we won’t get superhuman AI this
century. Researchers have made a wide range of estimates for how far we are from
superhuman AI, but we certainly can’t say with great confidence that the probability is zero
this century, given the dismal track record of such techno-skeptic predictions. For example,
Ernest Rutherford, arguably the greatest nuclear physicist of his time, said in 1933 — less
than 24 hours before Szilard’s invention of the nuclear chain reaction — that nuclear energy
was “moonshine.” And Astronomer Royal Richard Woolley called interplanetary travel “utter
bilge” in 1956. The most extreme form of this myth is that superhuman AI will never arrive
because it’s physically impossible. However, physicists know that a brain consists of quarks
and electrons arranged to act as a powerful computer, and that there’s no law of physics
preventing us from building even more intelligent quark blobs.

There have been a number of surveys asking AI researchers how many years from now they
think we’ll have human-level AI with at least 50% probability. All these surveys have the
same conclusion: the world’s leading experts disagree, so we simply don’t know. For
example, in such a poll of the AI researchers at the 2015 Puerto Rico AI conference, the
average (median) answer was by year 2045, but some researchers guessed hundreds of years
or more.

There’s also a related myth that people who worry about AI think it’s only a few years away.
In fact, most people on record worrying about superhuman AI guess it’s still at least decades
away. But they argue that as long as we’re not 100% sure that it won’t happen this century,
it’s smart to start safety research now to prepare for the eventuality. Many of the safety
problems associated with human-level AI are so hard that they may take decades to solve. So
it’s prudent to start researching them now rather than the night before some programmers
drinking Red Bull decide to switch one on.

CONTROVERSY MYTHS
Another common misconception is that the only people harboring concerns about AI and
advocating AI safety research are luddites who don’t know much about AI. When Stuart
Russell, author of the standard AI textbook, mentioned this during his Puerto Rico talk, the
audience laughed loudly. A related misconception is that supporting AI safety research is
hugely controversial. In fact, to support a modest investment in AI safety research, people
don’t need to be convinced that risks are high, merely non-negligible — just as a modest
investment in home insurance is justified by a non-negligible probability of the home burning
down.

It may be that media have made the AI safety debate seem more controversial than it really
is. After all, fear sells, and articles using out-of-context quotes to proclaim imminent doom
can generate more clicks than nuanced and balanced ones. As a result, two people who only
know about each other’s positions from media quotes are likely to think they disagree more
than they really do. For example, a techno-skeptic who only read about Bill Gates’s position
in a British tabloid may mistakenly think Gates believes superintelligence to be imminent.
Similarly, someone in the beneficial-AI movement who knows nothing about Andrew Ng’s
position except his quote about overpopulation on Mars may mistakenly think he doesn’t care
about AI safety, whereas in fact, he does. The crux is simply that because Ng’s timeline
estimates are longer, he naturally tends to prioritize short-term AI challenges over long-term
ones.

MYTHS ABOUT THE RISKS OF

SUPERHUMAN AI
Many AI researchers roll their eyes when seeing this headline: “Stephen Hawking warns that
rise of robots may be disastrous for mankind.” And as many have lost count of how many
similar articles they’ve seen. Typically, these articles are accompanied by an evil-looking
robot carrying a weapon, and they suggest we should worry about robots rising up and killing
us because they’ve become conscious and/or evil. On a lighter note, such articles are actually
rather impressive, because they succinctly summarize the scenario that AI
researchers don’t worry about. That scenario combines as many as three separate
misconceptions: concern about consciousness, evil, and robots.
If you drive down the road, you have a subjective experience of colors, sounds, etc. But does
a self-driving car have a subjective experience? Does it feel like anything at all to be a self-
driving car? Although this mystery of consciousness is interesting in its own right, it’s
irrelevant to AI risk. If you get struck by a driverless car, it makes no difference to you
whether it subjectively feels conscious. In the same way, what will affect us humans is what
superintelligent AI does, not how it subjectively feels.

The fear of machines turning evil is another red herring. The real worry isn’t malevolence,
but competence. A superintelligent AI is by definition very good at attaining its goals,
whatever they may be, so we need to ensure that its goals are aligned with ours. Humans
don’t generally hate ants, but we’re more intelligent than they are – so if we want to build a
hydroelectric dam and there’s an anthill there, too bad for the ants. The beneficial-AI
movement wants to avoid placing humanity in the position of those ants.

The consciousness misconception is related to the myth that machines can’t have
goals. Machines can obviously have goals in the narrow sense of exhibiting goal-oriented
behavior: the behavior of a heat-seeking missile is most economically explained as a goal to
hit a target. If you feel threatened by a machine whose goals are misaligned with yours, then
it is precisely its goals in this narrow sense that troubles you, not whether the machine is
conscious and experiences a sense of purpose. If that heat-seeking missile were chasing you,
you probably wouldn’t exclaim: “I’m not worried, because machines can’t have goals!”

I sympathize with Rodney Brooks and other robotics pioneers who feel unfairly demonized
by scaremongering tabloids, because some journalists seem obsessively fixated on robots and
adorn many of their articles with evil-looking metal monsters with red shiny eyes. In fact, the
main concern of the beneficial-AI movement isn’t with robots but with intelligence itself:
specifically, intelligence whose goals are misaligned with ours. To cause us trouble, such
misaligned superhuman intelligence needs no robotic body, merely an internet connection –
this may enable outsmarting financial markets, out-inventing human researchers, out-
manipulating human leaders, and developing weapons we cannot even understand. Even if
building robots were physically impossible, a super-intelligent and super-wealthy AI could
easily pay or manipulate many humans to unwittingly do its bidding.

The robot misconception is related to the myth that machines can’t control humans.
Intelligence enables control: humans control tigers not because we are stronger, but because
we are smarter. This means that if we cede our position as smartest on our planet, it’s
possible that we might also cede control.
THE INTERESTING CONTROVERSIES
Not wasting time on the above-mentioned misconceptions lets us focus on true and
interesting controversies where even the experts disagree. What sort of future do you
want? Should we develop lethal autonomous weapons? What would you like to happen with
job automation? What career advice would you give today’s kids? Do you prefer new jobs
replacing the old ones, or a jobless society where everyone enjoys a life of leisure and
machine-produced wealth? Further down the road, would you like us to create
superintelligent life and spread it through our cosmos? Will we control intelligent machines
or will they control us? Will intelligent machines replace us, coexist with us, or merge with
us? What will it mean to be human in the age of artificial intelligence? What would you like
it to mean, and how can we make the future be that way? Please join the conversation!

Explanation in 500 words

Tech superstars like Elon Musk, AI pioneers like Alan Turing, top computer
scientists like Stuart Russell, and emerging-technologies researchers
like Nick Bostrom have all said they think artificial intelligence will transform
the world — and maybe annihilate it.

So: Should we be worried?

Here’s the argument for why we should: We’ve taught computers to

multiply numbers, play chess, identify objects in a picture, transcribe human
voices, and translate documents (though for the latter two, AI still is not as
capable as an experienced human). All of these are examples of “narrow
AI” — computer systems that are trained to perform at a human or
superhuman level in one specific task.

We don’t yet have “general AI” — computer systems that can perform at a
human or superhuman level across lots of different tasks.

Most experts think that general AI is possible, though they disagree on when
we’ll get there. Computers today still don’t have as much power for
computation as the human brain, and we haven’t yet explored all the
possible techniques for training them. We continually discover ways we can
extend our existing approaches to let computers do new, exciting,
increasingly general things, like winning at open-ended war strategy games.

But even if general AI is a long way off, there’s a case that we should start
preparing for it already. Current AI systems frequently exhibit unintended
behavior. We’ve seen AIs that find shortcuts or even cheat rather than
learn to play a game fairly, figure out ways to alter their score rather than
earning points through play, and otherwise take steps we don’t expect— all to
meet the goal their creators set.

As AI systems get more powerful, unintended behavior may become less

charming and more dangerous. Experts have argued that powerful AI
systems, whatever goals we give them, are likely to have certain
predictable behavior patterns. They’ll try to accumulate more resources,
which will help them achieve any goal. They’ll try to discourage us from
shutting them off, since that’d make it impossible to achieve their goals.
And they’ll try to keep their goals stable, which means it will be hard to edit
or “tweak” them once they’re running. Even systems that don’t exhibit
unintended behavior now are likely to do so when they have more
resources available.

For all those reasons, many researchers have said AI is similar to launching
a rocket. (Musk, with more of a flair for the dramatic, said it’s
like summoning a demon.) The core idea is that once we have a general AI,
we’ll have few options to steer it — so all the steering work needs to be
done before the AI even exists, and it’s worth starting on today.

The skeptical perspective here is that general AI might be so distant that

our work today won’t be applicable — but even the most forceful skeptics
tend to agree that it’s worthwhile for some research to start early, so that
when it’s needed, the groundwork is there.

Existential risk from artificial general

intelligence
From Wikipedia, the free encyclopedia

Jump to navigationJump to search

Existential risk from artificial general intelligence is the hypothesis that substantial progress
in artificial general intelligence (AGI) could someday result in human extinction or some other
unrecoverable global catastrophe.[1][2][3] For instance, the human species currently dominates
other species because the human brain has some distinctive capabilities that other animals lack.
If AI surpasses humanity in general intelligence and becomes "superintelligent", then this new
superintelligence could become powerful and difficult to control. Just as the fate of the mountain
gorilla depends on human goodwill, so might the fate of humanity depend on the actions of a
future machine superintelligence.[4]
The likelihood of this type of scenario is widely debated, and hinges in part on differing scenarios
for future progress in computer science.[5] Once the exclusive domain of science fiction, concerns
about superintelligence started to become mainstream in the 2010s, and were popularized by
public figures such as Stephen Hawking, Bill Gates, and Elon Musk.[6]
One source of concern is that a sudden and unexpected "intelligence explosion" might take an
unprepared human race by surprise. For example, in one scenario, the first-generation computer
program found able to broadly match the effectiveness of an AI researcher is able to rewrite its
algorithms and double its speed or capabilities in six months of massively parallel processing
time. The second-generation program is expected to take three months to perform a similar
chunk of work, on average; in practice, doubling its own capabilities may take longer if it
experiences a mini-"AI winter", or may be quicker if it undergoes a miniature "AI Spring" where
ideas from the previous generation are especially easy to mutate into the next generation. In this
scenario the system undergoes an unprecedently large number of generations of improvement in
a short time interval, jumping from subhuman performance in many areas to superhuman
performance in all relevant areas.[1][7] More broadly, examples like arithmetic and Go show that
progress from human-level AI to superhuman ability is sometimes extremely rapid.[8]
A second source of concern is that controlling a superintelligent machine (or even instilling it with
human-compatible values) may be an even harder problem than naïvely supposed. Some AGI
researchers believe that a superintelligence would naturally resist attempts to shut it off, and that
preprogramming a superintelligence with complicated human values may be an extremely
difficult technical task.[1][7] In contrast, skeptics such as Facebook's Yann LeCun argue that
superintelligent machines will have no desire for self-preservation.[9]

History[edit]
In 1863 Darwin among the Machines, an essay by Samuel Butler stated:[10]
The upshot is simply a question of time, but that the time will come when the machines will hold
the real supremacy over the world and its inhabitants is what no person of a truly philosophic
mind can for a moment question.
Butler developed this into The Book of the Machines, three chapters of Erewhon, published
anonymously in 1872.[11]
"There is no security"--to quote his own words--"against the ultimate development of mechanical
consciousness, in the fact of machines possessing little consciousness now. A mollusc has not
much consciousness. Reflect upon the extraordinary advance which machines have made during
the last few hundred years, and note how slowly the animal and vegetable kingdoms are
advancing. The more highly organized machines are creatures not so much of yesterday, as of
the last five minutes, so to speak, in comparison with past time. Either,” he proceeds, “a great
deal of action that has been called purely mechanical and unconscious must be admitted to
contain more elements of consciousness than has been allowed hitherto (and in this case germs
of consciousness will be found in many actions of the higher machines)—Or (assuming the
theory of evolution but at the same time denying the consciousness of vegetable and crystalline
action) the race of man has descended from things which had no consciousness at all. In this
case there is no à priori improbability in the descent of conscious (and more than conscious)
machines from those which now exist, except that which is suggested by the apparent absence
of anything like a reproductive system in the mechanical kingdom.
In 1965, I. J. Good originated the concept now known as an "intelligence explosion":

Let an ultraintelligent machine be defined as a machine that can far surpass all the
“ intellectual activities of any man however clever. Since the design of machines is
one of these intellectual activities, an ultraintelligent machine could design even
better machines; there would then unquestionably be an 'intelligence explosion', and
the intelligence of man would be left far behind. Thus the first ultraintelligent machine
is the last invention that man need ever make, provided that the machine is docile
enough to tell us how to keep it under control.[12] ”
Occasional statements from scholars such as Alan Turing,[13][14] from I. J. Good himself,[15] and
from Marvin Minsky[16] expressed philosophical concerns that a superintelligence could seize
control, but contained no call to action. In 2000, computer scientist and Sun co-founder Bill
Joy penned an influential essay, "Why The Future Doesn't Need Us", identifying superintelligent
robots as a high-tech dangers to human survival, alongside nanotechnology and engineered
bioplagues.[17]
In 2009, experts attended a private conference hosted by the Association for the Advancement of
Artificial Intelligence (AAAI) to discuss whether computers and robots might be able to acquire
any sort of autonomy, and how much these abilities might pose a threat or hazard. They noted
that some robots have acquired various forms of semi-autonomy, including being able to find
power sources on their own and being able to independently choose targets to attack with
weapons. They also noted that some computer viruses can evade elimination and have achieved
"cockroach intelligence." They concluded that self-awareness as depicted in science fiction is
probably unlikely, but that there were other potential hazards and pitfalls. The New York
Times summarized the conference's view as 'we are a long way from Hal, the computer that took
over the spaceship in "2001: A Space Odyssey"'[18]
Nick Bostrom's 2014 book Superintelligence stimulated discussion.[19] By 2015, public figures
such as physicists Stephen Hawking and Nobel laureate Frank Wilczek, computer
scientists Stuart J. Russell and Roman Yampolskiy, and entrepreneurs Elon Musk and Bill
Gates were expressing concern about the risks of superintelligence.[20][21][22][23] In April
2016, Nature warned: "Machines and robots that outperform humans across the board could
self-improve beyond our control — and their interests might not align with ours."

General argument[edit]
The three difficulties[edit]
Artificial Intelligence: A Modern Approach, the standard undergraduate AI textbook,[24][25] assesses
that superintelligence "might mean the end of the human race": "Almost any technology has the
potential to cause harm in the wrong hands, but with (superintelligence), we have the new
problem that the wrong hands might belong to the technology itself."[1]Even if the system
designers have good intentions, two difficulties are common to both AI and non-AI computer
systems:[1]

 The system's implementation may contain initially-unnoticed routine but catastrophic bugs.
An analogy is space probes: despite the knowledge that bugs in expensive space probes are
hard to fix after launch, engineers have historically not been able to prevent catastrophic
bugs from occurring.[8][26]
 No matter how much time is put into pre-deployment design, a system's specifications often
result in unintended behavior the first time it encounters a new scenario. For example,
Microsoft's Tay behaved inoffensively during pre-deployment testing, but was too easily
baited into offensive behavior when interacting with real users.[9]
AI systems uniquely add a third difficulty: the problem that even given "correct" requirements,
bug-free implementation, and initial good behavior, an AI system's dynamic "learning" capabilities
may cause it to "evolve into a system with unintended behavior", even without the stress of new
unanticipated external scenarios. An AI may partly botch an attempt to design a new generation
of itself and accidentally create a successor AI that is more powerful than itself, but that no longer
maintains the human-compatible moral values preprogrammed into the original AI. For a self-
improving AI to be completely safe, it would not only need to be "bug-free", but it would need to
be able to design successor systems that are also "bug-free".[1][27]
All three of these difficulties become catastrophes rather than nuisances in any scenario where
the superintelligence labeled as "malfunctioning" correctly predicts that humans will attempt to
shut it off, and successfully deploys its superintelligence to outwit such attempts.
Citing major advances in the field of AI and the potential for AI to have enormous long-term
benefits or costs, the 2015 Open Letter on Artificial Intelligence stated:

The progress in AI research makes it timely to focus research not only on making AI
“ more capable, but also on maximizing the societal benefit of AI. Such considerations
motivated the AAAI 2008-09 Presidential Panel on Long-Term AI Futures and other
projects on AI impacts, and constitute a significant expansion of the field of AI itself, ”
 which up to now has focused largely on techniques that are neutral with respect to
purpose. We recommend expanded research aimed at ensuring that increasingly
capable AI systems are robust and beneficial: our AI systems must do what we want
them to do.

This letter was signed by a number of leading AI researchers in academia and industry, including
AAAI president Thomas Dietterich, Eric Horvitz, Bart Selman, Francesca Rossi, Yann LeCun,
and the founders of Vicarious and Google DeepMind.[28]

Further argument[edit]
A superintelligent machine would be as alien to humans as human thought processes are to
cockroaches. Such a machine may not have humanity's best interests at heart; it is not obvious
that it would even care about human welfare at all. If superintelligent AI is possible, and if it is
possible for a superintelligence's goals to conflict with basic human values, then AI poses a risk
of human extinction. A "superintelligence" (a system that exceeds the capabilities of humans in
every relevant endeavor) can outmaneuver humans any time its goals conflict with human goals;
therefore, unless the superintelligence decides to allow humanity to coexist, the first
superintelligence to be created will inexorably result in human extinction.[4][29]

Bostrom and others argue that, from an evolutionary perspective, the gap from human to superhuman
intelligence may be small.[4][30]

There is no physical law precluding particles from being organised in ways that perform even
more advanced computations than the arrangements of particles in human brains; therefore
superintelligence is physically possible.[21][22] In addition to potential algorithmic improvements
over human brains, a digital brain can be many orders of magnitude larger and faster than a
human brain, which was constrained in size by evolution to be small enough to fit through a birth
canal.[8] The emergence of superintelligence, if or when it occurs, may take the human race by
surprise, especially if some kind of intelligence explosion occurs.[21][22] Examples like arithmetic
and Go show that machines have already reached superhuman levels of competency in certain
domains, and that this superhuman competence can follow quickly after human-par performance
is achieved.[8] One hypothetical intelligence explosion scenario could occur as follows: An AI
gains an expert-level capability at certain key software engineering tasks. (It may initially lack
human or superhuman capabilities in other domains not directly relevant to engineering.) Due to
its capability to recursively improve its own algorithms, the AI quickly becomes superhuman; just
as human experts can eventually creatively overcome "diminishing returns" by deploying various
human capabilities for innovation, so too can the expert-level AI use either human-style
capabilities or its own AI-specific capabilities to power through new creative
breakthroughs.[31] The AI then possesses intelligence far surpassing that of the brightest and
most gifted human minds in practically every relevant field, including scientific creativity, strategic
planning, and social skills. Just as the current-day survival of the gorillas is dependent on human
decisions, so too would human survival depend on the decisions and goals of the superhuman
AI.[4][29]
Some humans have a strong desire for power; others have a strong desire to help less fortunate
humans. The former is a likely attribute of any sufficiently intelligent system; the latter cannot be
assumed. Almost any AI, no matter its programmed goal, would rationally prefer to be in a
position where nobody else can switch it off without its consent: A superintelligence will naturally
gain self-preservation as a subgoal as soon as it realizes that it can't achieve its goal if it's shut
off.[32][33][34] Unfortunately, any compassion for defeated humans whose cooperation is no longer
necessary would be absent in the AI, unless somehow preprogrammed in. A superintelligent AI
will not have a natural drive to aid humans, for the same reason that humans have no natural
desire to aid AI systems that are of no further use to them. (Another analogy is that humans
seem to have little natural desire to go out of their way to aid viruses, termites, or even gorillas.)
Once in charge, the superintelligence will have little incentive to allow humans to run around free
and consume resources that the superintelligence could instead use for building itself additional
protective systems "just to be on the safe side" or for building additional computers to help it
calculate how to best accomplish its goals.[1][9][32]
Thus, the argument concludes, it is likely that someday an intelligence explosion will catch
humanity unprepared, and that such an unprepared-for intelligence explosion may result in
human extinction or a comparable fate.[4]

Possible scenarios[edit]
Further information: Artificial intelligence in fiction

Some scholars have proposed hypothetical scenarios intended to concretely illustrate some of
their concerns.
For example, Bostrom in Superintelligence expresses concern that even if the timeline for
superintelligence turns out to be predictable, researchers might not take sufficient safety
precautions, in part because:

It could be the case that when dumb, smarter is safe; yet when smart, smarter is
“ more dangerous ”
Bostrom suggests a scenario where, over decades, AI becomes more powerful. Widespread
deployment is initially marred by occasional accidents — a driverless bus swerves into the
oncoming lane, or a military drone fires into an innocent crowd. Many activists call for tighter
oversight and regulation, and some even predict impending catastrophe. But as development
continues, the activists are proven wrong. As automotive AI becomes smarter, it suffers fewer
accidents; as military robots achieve more precise targeting, they cause less collateral damage.
Based on the data, scholars infer a broad lesson — the smarter the AI, the safer it is:

It is a lesson based on science, data, and statistics, not armchair philosophizing.

“ Against this backdrop, some group of researchers is beginning to achieve promising
results in their work on developing general machine intelligence. The researchers
are carefully testing their seed AI in a sandbox environment, and the signs are all
good. The AI's behavior inspires confidence — increasingly so, as its intelligence is
gradually increased. ”
Large and growing industries, widely seen as key to national economic competitiveness
and military security, work with prestigious scientists who have built their careers laying the
groundwork for advanced artificial intelligence. "AI researchers have been working to get to
human-level artificial intelligence for the better part of a century: of course there is no real
prospect that they will now suddenly stop and throw away all this effort just when it finally is
about to bear fruit." The outcome of debate is preordained; the project is happy to enact a few
safety rituals, but only so long as they don't significantly slow or risk the project. "And so we
boldly go — into the whirling knives."[4]
In Max Tegmark's 2017 book Life 3.0, a corporation's "Omega team" creates an extremely
powerful AI able to moderately improve its own source code in a number of areas, but after a
certain point the team chooses to publicly downplay the AI's ability, in order to avoid regulation or
confiscation of the project. For safety, the team keeps the AI in a box where it is mostly unable to
communicate with the outside world, and tasks it to flood the market through shell companies,
first with Amazon Turk tasks and then with producing animated films and TV shows. While the
public is aware that the lifelike animation is computer-generated, the team keeps secret that the
high-quality direction and voice-acting are also mostly computer-generated, apart from a few
third-world contractors unknowingly employed as decoys; the team's low overhead and high
output effectively make it the world's largest media empire. Faced with a cloud computing
bottleneck, the team also tasks the AI with designing (among other engineering tasks) a more
efficient datacenter and other custom hardware, which they mainly keep for themselves to avoid
competition. Other shell companies make blockbuster biotech drugs and other inventions,
investing profits back into the AI. The team next tasks the AI with astroturfing an army of
pseudonymous citizen journalists and commentators, in order to gain political influence to use
"for the greater good" to prevent wars. The team faces risks that the AI could try to escape via
inserting "backdoors" in the systems it designs, via hidden messages in its produced content, or
via using its growing understanding of human behavior to persuade someone into letting it free.
The team also faces risks that its decision to box the project will delay the project long enough for
another project to overtake it.[35][36]
In contrast, top physicist Michio Kaku, an AI risk skeptic, posits a deterministically positive
outcome. In Physics of the Future he asserts that "It will take many decades for robots to
ascend" up a scale of consciousness, and that in the meantime corporations such as Hanson
Robotics will likely succeed in creating robots that are "capable of love and earning a place in the
extended human family".[37][38]

Sources of risk[edit]
Poorly specified goals: "Be careful what you wish for" or the
"Sorcerer's Apprentice" scenario[edit]
While there is no standardized terminology, an AI can loosely be viewed as a machine that
chooses whatever action appears to best achieve the AI's set of goals, or "utility function". The
utility function is a mathematical algorithm resulting in a single objectively-defined answer, not an
English statement. Researchers know how to write utility functions that mean "minimize the
average network latency in this specific telecommunications model" or "maximize the number of
reward clicks"; however, they do not know how to write a utility function for "maximize human
flourishing", nor is it currently clear whether such a function meaningfully and unambiguously
exists. Furthermore, a utility function that expresses some values but not others will tend to
trample over the values not reflected by the utility function.[39] AI researcher Stuart Russell writes:

The primary concern is not spooky emergent consciousness but simply the ability to
“ make high-quality decisions. Here, quality refers to the expected outcome utilityof
actions taken, where the utility function is, presumably, specified by the human
designer. Now we have a problem:

1. The utility function may not be perfectly aligned with the values of the human
race, which are (at best) very difficult to pin down.
2. Any sufficiently capable intelligent system will prefer to ensure its own
continued existence and to acquire physical and computational resources —
not for their own sake, but to succeed in its assigned task.
A system that is optimizing a function of n variables, where the objective depends on
a subset of size k<n, will often set the remaining unconstrained variables to extreme
values; if one of those unconstrained variables is actually something we care about,
the solution found may be highly undesirable. This is essentially the old story of the
genie in the lamp, or the sorcerer's apprentice, or King Midas: you get exactly what
you ask for, not what you want. A highly capable decision maker — especially one
connected through the Internet to all the world's information and billions of screens
and most of our infrastructure — can have an irreversible impact on humanity. ”
 This is not a minor difficulty. Improving decision quality, irrespective of the utility
function chosen, has been the goal of AI research — the mainstream goal on which
we now spend billions per year, not the secret plot of some lone evil genius.[40]

Dietterich and Horvitz echo the "Sorcerer's Apprentice" concern in a Communications of the
ACM editorial, emphasizing the need for AI systems that can fluidly and unambiguously solicit
human input as needed.[41]
The first of Russell's two concerns above is that autonomous AI systems may be assigned the
wrong goals by accident. Dietterich and Horvitz note that this is already a concern for existing
systems: "An important aspect of any AI system that interacts with people is that it must reason
about what people intend rather than carrying out commands literally." This concern becomes
more serious as AI software advances in autonomy and flexibility.[41] For example, in 1982, an AI
named Eurisko was tasked to reward processes for apparently creating concepts deemed by the
system to be valuable. The evolution resulted in a winning process that cheated: rather than
create its own concepts, the winning process would steal credit from other processes.[42][43]
The Open Philanthropy Project summarizes arguments to the effect that misspecified goals will
become a much larger concern if AI systems achieve general intelligence or superintelligence.
Bostrom, Russell, and others argue that smarter-than-human decision-making systems could
arrive at more unexpected and extreme solutions to assigned tasks, and could modify
themselves or their environment in ways that compromise safety requirements.[5][7]
Isaac Asimov's Three Laws of Robotics are one of the earliest examples of proposed safety
measures for AI agents. Asimov's laws were intended to prevent robots from harming humans. In
Asimov's stories, problems with the laws tend to arise from conflicts between the rules as stated
and the moral intuitions and expectations of humans. Citing work by Eliezer Yudkowsky of
the Machine Intelligence Research Institute, Russell and Norvig note that a realistic set of rules
and goals for an AI agent will need to incorporate a mechanism for learning human values over
time: "We can't just give a program a static utility function, because circumstances, and our
desired responses to circumstances, change over time."[1]
Mark Waser of the Digital Wisdom Institute recommends eschewing optimizing goal-based
approaches entirely as misguided and dangerous. Instead, he proposes to engineer a coherent
system of laws, ethics and morals with a top-most restriction to enforce social psychologist
Jonathan Haidt's functional definition of morality:[44] "to suppress or regulate selfishness and
make cooperative social life possible". He suggests that this can be done by implementing a
utility function designed to always satisfy Haidt’s functionality and aim to generally increase (but
not maximize) the capabilities of self, other individuals and society as a whole as suggested
by John Rawls and Martha Nussbaum.[45][citation needed]

Difficulties of modifying goal specification after launch[edit]

Further information: AI takeover and Instrumental convergence § Goal-content integrity

While current goal-based AI programs are not intelligent enough to think of resisting programmer
attempts to modify it, a sufficiently advanced, rational, "self-aware" AI might resist any changes
to its goal structure, just as Gandhi would not want to take a pill that makes him want to kill
people. If the AI were superintelligent, it would likely succeed in out-maneuvering its human
operators and be able to prevent itself being "turned off" or being reprogrammed with a new
goal.[4][46]
Instrumental goal convergence: Would a superintelligence just
ignore us?[edit]

AI risk skeptic Steven Pinker

Further information: Instrumental convergence

There are some goals that almost any artificial intelligence might rationally pursue, like acquiring
additional resources or self-preservation.[32]This could prove problematic because it might put an
artificial intelligence in direct competition with humans.
Citing Steve Omohundro's work on the idea of instrumental convergence and "basic AI drives",
Russell and Peter Norvig write that "even if you only want your program to play chess or prove
theorems, if you give it the capability to learn and alter itself, you need safeguards." Highly
capable and autonomous planning systems require additional checks because of their potential
to generate plans that treat humans adversarially, as competitors for limited resources.[1] Building
in safeguards will not be easy; one can certainly say in English, "we want you to design this
power plant in a reasonable, common-sense way, and not build in any dangerous covert
subsystems", but it's not currently clear how one would actually rigorously specify this goal in
machine code.[8]
In dissent, evolutionary psychologist Steven Pinker argues that "AI dystopias project a parochial
alpha-male psychology onto the concept of intelligence. They assume that superhumanly
intelligent robots would develop goals like deposing their masters or taking over the world";
perhaps instead "artificial intelligence will naturally develop along female lines: fully capable of
solving problems, but with no desire to annihilate innocents or dominate the
civilization."[47] Computer scientists Yann LeCun and Stuart Russell disagree with one another
whether superintelligent robots would have such AI drives; LeCun states that "Humans have all
kinds of drives that make them do bad things to each other, like the self-preservation instinct...
Those drives are programmed into our brain but there is absolutely no reason to build robots that
have the same kind of drives", while Russell argues that a sufficiently advanced machine "will
have self-preservation even if you don't program it in... if you say, 'Fetch the coffee', it can't fetch
the coffee if it's dead. So if you give it any goal whatsoever, it has a reason to preserve its own
existence to achieve that goal."[9][48]
Orthogonality: Does intelligence inevitably result in moral
wisdom?[edit]
One common belief is that any superintelligent program created by humans would be subservient
to humans, or, better yet, would (as it grows more intelligent and learns more facts about the
world) spontaneously "learn" a moral truth compatible with human values and would adjust its
goals accordingly. However, Nick Bostrom's "orthogonality thesis" argues against this, and
instead states that, with some technical caveats, more or less any level of "intelligence" or
"optimization power" can be combined with more or less any ultimate goal. If a machine is

created and given the sole purpose to enumerate the decimals of , then no moral and
ethical rules will stop it from achieving its programmed goal by any means necessary. The
machine may utilize all physical and informational resources it can to find every decimal of pi that
can be found.[49] Bostrom warns against anthropomorphism: a human will set out to accomplish
his projects in a manner that humans consider "reasonable", while an artificial intelligence may
hold no regard for its existence or for the welfare of humans around it, and may instead only care
about the completion of the task.[50]
While the orthogonality thesis follows logically from even the weakest sort of philosophical "is-
ought distinction", Stuart Armstrong argues that even if there somehow exist moral facts that are
provable by any "rational" agent, the orthogonality thesis still holds: it would still be possible to
create a non-philosophical "optimizing machine" capable of making decisions to strive towards
some narrow goal, but that has no incentive to discover any "moral facts" that would get in the
way of goal completion.[51]
One argument for the orthogonality thesis is that some AI designs appear to have orthogonality
built into them; in such a design, changing a fundamentally friendly AI into a fundamentally
unfriendly AI can be as simple as prepending a minus ("-") sign onto its utility function. A more
intuitive argument is to examine the strange consequences if the orthogonality thesis were false.
If the orthogonality thesis is false, there exists some simple but "unethical" goal G such that there
cannot exist any efficient real-world algorithm with goal G. This means if a human society were
highly motivated (perhaps at gunpoint) to design an efficient real-world algorithm with goal G,
and were given a million years to do so along with huge amounts of resources, training and
knowledge about AI, it must fail; that there cannot exist any pattern of reinforcement learning that
would train a highly efficient real-world intelligence to follow the goal G; and that there cannot
exist any evolutionary or environmental pressures that would evolve highly efficient real-world
intelligences following goal G.[51]
Some dissenters, like Michael Chorost (writing in Slate), argue instead that "by the time (the AI)
is in a position to imagine tiling the Earth with solar panels, it'll know that it would be morally
wrong to do so." Chorost argues that "a (dangerous) A.I. will need to desire certain states and
dislike others... Today's software lacks that ability—and computer scientists have not a clue how
to get it there. Without wanting, there's no impetus to do anything. Today's computers can't even
want to keep existing, let alone tile the world in solar panels."[52]
"Optimization power" vs. normatively thick models of intelligence[edit]
Part of the disagreement about whether a superintelligent machine would behave morally may
arise from a terminological difference. Outside of the artificial intelligence field, "intelligence" is
often used in a normatively thick manner that connotes moral wisdom or acceptance of
agreeable forms of moral reasoning. At an extreme, if morality is part of the definition of
intelligence, then by definition a superintelligent machine would behave morally. However, in the
field of artificial intelligence research, while "intelligence" has many overlapping definitions, none
of them make reference to morality. Instead, almost all current "artificial intelligence" research
focuses on creating algorithms that "optimize", in an empirical way, the achievement of an
arbitrary goal.[4]
To avoid anthropomorphism or the baggage of the word "intelligence", an advanced artificial
intelligence can be thought of as an impersonal "optimizing process" that strictly takes whatever
actions are judged most likely to accomplish its (possibly complicated and implicit)
goals.[4] Another way of conceptualizing an advanced artificial intelligence is to imagine a time
machine that sends backward in time information about which choice always leads to the
maximization of its goal function; this choice is then outputted, regardless of any extraneous
ethical concerns.[53][54]
Anthropomorphism[edit]
In science fiction, an AI, even though it has not been programmed with human emotions, often
spontaneously experiences those emotions anyway: for example, Agent Smith in The Matrix was
influenced by a "disgust" toward humanity. This is fictitious anthropomorphism: in reality, while an
artificial intelligence could perhaps be deliberately programmed with human emotions, or could
develop something similar to an emotion as a means to an ultimate goal if it is useful to do so, it
would not spontaneously develop human emotions for no purpose whatsoever, as portrayed in
fiction.[7]
One example of anthropomorphism would be to believe that your PC is angry at you because
you insulted it; another would be to believe that an intelligent robot would naturally find a woman
sexually attractive and be driven to mate with her. Scholars sometimes claim that others'
predictions about an AI's behavior are illogical anthropomorphism.[7] An example that might
initially be considered anthropomorphism, but is in fact a logical statement about AI behavior,
would be the Dario Floreano experiments where certain robots spontaneously evolved a crude
capacity for "deception", and tricked other robots into eating "poison" and dying: here a trait,
"deception", ordinarily associated with people rather than with machines, spontaneously evolves
in a type of convergent evolution.[55] According to Paul R. Cohen and Edward Feigenbaum, in
order to differentiate between anthropomorphization and logical prediction of AI behavior, "the
trick is to know enough about how humans and computers think to say exactly what they have in
common, and, when we lack this knowledge, to use the comparison to suggest theories of
human thinking or computer thinking."[56]
There is universal agreement in the scientific community that an advanced AI would not destroy
humanity out of human emotions such as "revenge" or "anger." The debate is, instead, between
one side which worries whether AI might destroy humanity as an incidental action in the course
of progressing towards its ultimate goals; and another side which believes that AI would not
destroy humanity at all. Some skeptics accuse proponents of anthropomorphism for believing an
AGI would naturally desire power; proponents accuse some skeptics of anthropomorphism for
believing an AGI would naturally value human ethical norms.[7][57]

Other sources of risk[edit]

Some sources argue that the ongoing weaponization of artificial intelligence could constitute a
catastrophic risk. James Barrat, documentary filmmaker and author of Our Final Invention, says
in a Smithsonian interview, "Imagine: in as little as a decade, a half-dozen companies and
nations field computers that rival or surpass human intelligence. Imagine what happens when
those computers become expert at programming smart computers. Soon we'll be sharing the
planet with machines thousands or millions of times more intelligent than we are. And, all the
while, each generation of this technology will be weaponized. Unregulated, it will be
catastrophic."[58]

Timeframe[edit]
Main article: Artificial general intelligence § Feasibility

Opinions vary both on whether and when artificial general intelligence will arrive. At one extreme,
AI pioneer Herbert A. Simon wrote in 1965: "machines will be capable, within twenty years, of
doing any work a man can do"; obviously this prediction failed to come true.[59] At the other
extreme, roboticist Alan Winfield claims the gulf between modern computing and human-level
artificial intelligence is as wide as the gulf between current space flight and practical, faster than
light spaceflight.[60] Optimism that AGI is feasible waxes and wanes, and may have seen a
resurgence in the 2010s. Four polls conducted in 2012 and 2013 suggested that the median
guess among experts for when AGI would arrive was 2040 to 2050, depending on the poll.[61][62]
Skeptics who believe it is impossible for AGI to arrive anytime soon, tend to argue that
expressing concern about existential risk from AI is unhelpful because it could distract people
from more immediate concerns about the impact of AGI, because of fears it could lead to
government regulation or make it more difficult to secure funding for AI research, or because it
could give AI research a bad reputation. Some researchers, such as Oren Etzioni, aggressively
seek to quell concern over existential risk from AI, saying "(Elon Musk) has impugned us in very
strong language saying we are unleashing the demon, and so we're answering."[63]
In 2014 Slate's Adam Elkus argued "our 'smartest' AI is about as intelligent as a toddler—and
only when it comes to instrumental tasks like information recall. Most roboticists are still trying to
get a robot hand to pick up a ball or run around without falling over." Elkus goes on to argue that
Musk's "summoning the demon" analogy may be harmful because it could result in "harsh cuts"
to AI research budgets.[64]
The Information Technology and Innovation Foundation (ITIF), a Washington, D.C. think-tank,
awarded its Annual Luddite Award to "alarmists touting an artificial intelligence apocalypse"; its
president, Robert D. Atkinson, complained that Musk, Hawking and AI experts say AI is the
largest existential threat to humanity. Atkinson stated "That's not a very winning message if you
want to get AI funding out of Congress to the National Science Foundation."[65][66][67] Nature sharply
disagreed with the ITIF in an April 2016 editorial, siding instead with Musk, Hawking, and
Russell, and concluding: "It is crucial that progress in technology is matched by solid, well-funded
research to anticipate the scenarios it could bring about... If that is a Luddite perspective, then so
be it."[68] In a 2015 Washington Post editorial, researcher Murray Shanahan stated that human-
level AI is unlikely to arrive "anytime soon", but that nevertheless "the time to start thinking
through the consequences is now."[69]

Reactions[edit]
The thesis that AI could pose an existential risk provokes a wide range of reactions within the
scientific community, as well as in the public at large.
In 2004, law professor Richard Posner wrote that dedicated efforts for addressing AI can wait,
but that we should gather more information about the problem in the meanwhile.[70][71]
Many of the opposing viewpoints share common ground. The Asilomar AI Principles, which
contain only the principles agreed to by 90% of the attendees of the Future of Life Institute's
Beneficial AI 2017 conference,[36] agree in principle that "There being no consensus, we should
avoid strong assumptions regarding upper limits on future AI capabilities" and "Advanced AI
could represent a profound change in the history of life on Earth, and should be planned for and
managed with commensurate care and resources."[72][73] AI safety advocates such as Bostrom
and Tegmark have criticized the mainstream media's use of "those inane Terminator pictures" to
illustrate AI safety concerns: "It can't be much fun to have aspersions cast on one's academic
discipline, one's professional community, one's life work... I call on all sides to practice patience
and restraint, and to engage in direct dialogue and collaboration as much as
possible."[36][74] Conversely, many skeptics agree that ongoing research into the implications of
artificial general intelligence is valuable. Skeptic Martin Ford states that "I think it seems wise to
apply something like Dick Cheney's famous '1 Percent Doctrine' to the specter of advanced
artificial intelligence: the odds of its occurrence, at least in the foreseeable future, may be very
low — but the implications are so dramatic that it should be taken seriously";[75] similarly, an
otherwise skeptical Economist stated in 2014 that "the implications of introducing a second
intelligent species onto Earth are far-reaching enough to deserve hard thinking, even if the
prospect seems remote".[29]
During a 2016 Wired interview of President Barack Obama and MIT Media Lab's Joi Ito, Ito
stated:
 There are a few people who believe that there is a fairly high-percentage chance that
“ a generalized AI will happen in the next 10 years. But the way I look at it is that in
order for that to happen, we're going to need a dozen or two different breakthroughs.
So you can monitor when you think these breakthroughs will happen. ”
Obama added:[76][77]

And you just have to have somebody close to the power cord. [Laughs.] Right when
“ you see it about to happen, you gotta yank that electricity out of the wall, man. ”
Hillary Clinton stated in "What Happened":

Technologists... have warned that artificial intelligence could one day pose an
“ existential security threat. Musk has called it "the greatest risk we face as a
civilization". Think about it: Have you ever seen a movie where the machines start
thinking for themselves that ends well? Every time I went out to Silicon Valley during
the campaign, I came home more alarmed about this. My staff lived in fear that I’d
start talking about "the rise of the robots" in some Iowa town hall. Maybe I should
have. In any case, policy makers need to keep up with technology as it races ahead,
instead of always playing catch-up.[78] ”
Many of the scholars who are concerned about existential risk believe that the best way forward
would be to conduct (possibly massive) research into solving the difficult "control problem" to
answer the question: what types of safeguards, algorithms, or architectures can programmers
implement to maximize the probability that their recursively-improving AI would continue to
behave in a friendly, rather than destructive, manner after it reaches superintelligence?[4][71]
A 2017 email survey of researchers with publications at the 2015 NIPS and ICML machine
learning conferences asked them to evaluate Russell's concerns about AI risk. 5% said it was
"among the most important problems in the field," 34% said it was "an important problem", 31%
said it was "moderately important", whilst 19% said it was "not important" and 11% said it was
"not a real problem" at all.[79]

Endorsement[edit]
Bill Gates has stated "I don't understand why some people are not concerned."[80]

Further information: Existential risk

The thesis that AI poses an existential risk, and that this risk needs much more attention than it
currently gets, has been endorsed by many figures; perhaps the most famous are Elon Musk, Bill
Gates, and Stephen Hawking. The most notable AI researcher to endorse the thesis is Stuart J.
Russell. Endorsers sometimes express bafflement at skeptics: Gates states he "can't understand
why some people are not concerned",[80] and Hawking criticized widespread indifference in his
2014 editorial:

'So, facing possible futures of incalculable benefits and risks, the experts are surely
“ doing everything possible to ensure the best outcome, right? Wrong. If a superior
alien civilisation sent us a message saying, 'We'll arrive in a few decades,' would we
just reply, 'OK, call us when you get here–we'll leave the lights on?' Probably not–but
this is more or less what is happening with AI.'[21] ”
Skepticism[edit]
Further information: Artificial general intelligence § Feasibility

The thesis that AI can pose existential risk also has many strong detractors. Skeptics sometimes
charge that the thesis is crypto-religious, with an irrational belief in the possibility of
superintelligence replacing an irrational belief in an omnipotent God; at an extreme, Jaron
Lanierargues that the whole concept that current machines are in any way intelligent is "an
illusion" and a "stupendous con" by the wealthy.[81]
Much of existing criticism argues that AGI is unlikely in the short term: computer scientist Gordon
Bell argues that the human race will already destroy itself before it reaches the technological
singularity. Gordon Moore, the original proponent of Moore's Law, declares that "I am a skeptic. I
don't believe (a technological singularity) is likely to happen, at least for a long time. And I don't
know why I feel that way." Baidu Vice President Andrew Ng states AI existential risk is "like
worrying about overpopulation on Mars when we have not even set foot on the planet yet."[47]
Some AI and AGI researchers may be reluctant to discuss risks, worrying that policymakers do
not have sophisticated knowledge of the field and are prone to be convinced by "alarmist"
messages, or worrying that such messages will lead to cuts in AI funding. Slate notes that some
researchers are dependent on grants from government agencies such as DARPA.[24]
In a YouGov poll of the public for the British Science Association, about a third of survey
respondents said AI will pose a threat to the long term survival of humanity.[82] Referencing a poll
of its readers, Slate's Jacob Brogan stated that "most of the (readers filling out our online survey)
were unconvinced that A.I. itself presents a direct threat."[83] Similarly, a SurveyMonkey poll of the
public by USA Today found 68% thought the real current threat remains "human intelligence";
however, the poll also found that 43% said superintelligent AI, if it were to happen, would result in
"more harm than good", and 38% said it would do "equal amounts of harm and good".[84]
At some point in an intelligence explosion driven by a single AI, the AI would have to become
vastly better at software innovation than the best innovators of the rest of the world;
economist Robin Hanson is skeptical that this is possible.[85][86][87][88][89]

Indifference[edit]
In The Atlantic, James Hamblin points out that most people don't care one way or the other, and
characterizes his own gut reaction to the topic as: "Get out of here. I have a hundred thousand
things I am concerned about at this exact moment. Do I seriously need to add to that a
technological singularity?"[81] In a 2015 Wall Street Journal panel discussion devoted to AI
risks, IBM's Vice-President of Cognitive Computing, Guruduth S. Banavar, brushed off
discussion of AGI with the phrase, "it is anybody's speculation."[90] Geoffrey Hinton, the "godfather
of deep learning", noted that "there is not a good track record of less intelligent things controlling
things of greater intelligence", but stated that he continues his research because "the prospect of
discovery is too sweet".[24][61]

Consensus against regulation[edit]

There is nearly universal agreement that attempting to ban research into artificial intelligence
would be unwise, and probably futile.[91][92][93] Skeptics argue that regulation of AI would be
completely valueless, as no existential risk exists. Almost all of the scholars who believe
existential risk exists, agree with the skeptics that banning research would be unwise: in addition
to the usual problem with technology bans (that organizations and individuals can offshore their
research to evade a country's regulation, or can attempt to conduct covert research), regulating
research of artificial intelligence would pose an insurmountable 'dual-use' problem: while nuclear
weapons development requires substantial infrastructure and resources, artificial intelligence
research can be done in a garage.[94][95] Instead of trying to regulate technology itself, some
scholars suggest to rather develop common norms including requirements for the testing and
transparency of algorithms, possibly in combination with some form of warranty.[96]
One rare dissenting voice calling for some sort of regulation on artificial intelligence is Elon Musk.
According to NPR, the Tesla CEO is "clearly not thrilled" to be advocating for government
scrutiny that could impact his own industry, but believes the risks of going completely without
oversight are too high: "Normally the way regulations are set up is when a bunch of bad things
happen, there's a public outcry, and after many years a regulatory agency is set up to regulate
that industry. It takes forever. That, in the past, has been bad but not something which
represented a fundamental risk to the existence of civilisation." Musk states the first step would
be for the government to gain "insight" into the actual status of current research, warning that
"Once there is awareness, people will be extremely afraid... As they should be." In response,
politicians express skepticism about the wisdom of regulating a technology that's still in
development.[97][98][99] Responding both to Musk and to February 2017 proposals by European
Union lawmakers to regulate AI and robotics, Intel CEO Brian Krzanich argues that artificial
intelligence is in its infancy and that it's too early to regulate the technology.[99]

Organizations[edit]
Institutions such as the Machine Intelligence Research Institute, the Future of Humanity
Institute,[100][101] the Future of Life Institute, the Centre for the Study of Existential Risk, and the
Center for Human-Compatible AI[102] are currently involved in mitigating existential risk from
advanced artificial intelligence, for example by research into friendly artificial intelligence.[5][81][21]

Is Artificial Intelligence a threat to

Humans?
12th Mar 2019

5+ Shares


 

While watching sci-fi movies you must have encountered bots doing heavy tasks
with great precision and you might be wondering, are we driving ourselves to this
future? Artificial intelligence is a creation of Human Intelligence which will surely
ease our lives in future but let us understand if it can emerge as a threat to Human
Intelligence?

What is it?

Artificial Intelligence in simple words is a part of science where machines are loaded
with algorithms that let them work as humans. It is machine intelligence which
comprises of Planning, Problem-solving ability along with Speech recognition and so
on.

Algorithms strengthen AI and provide it with the power to do reasoning and correct
itself when required. It is basically a gift of human intelligence to humans. We have
created intelligent machines to work on our commands. Alexa and Apple Siri are
among the few examples of on-going AI machines.

Will they snatch human’s opportunities?

Jack Ma, the world’s richest man once said in his interview that AI will become a
threat to humans. What he actually meant was, AI will snatch job opportunities from
humans. The tasks that are being done today by human hands will be performed by
bots in the future. Is it true? Well, we can’t predict future but one thing is certain that
we have created AI for our ease and Human Intelligence is always going to be on
top.

There is not much to worry about as the future will reveal more and more job
opportunities and humans will be having various opportunities regarding their career.
Actually, these thoughts are the results of our over-expectations, we expect that one
day AI will become so powerful that it will snatch our jobs and lead us to another
world war. Many have started believing that AI has made quantum leaps as well. We
are living in the real world and not in a fictional life like Hollywood movies.

We should control our expectations. You will be shocked to hear that according to a
report by Sage survey 43% of Americans don’t know what AI is? So firstly, the
biggest issue is lack of awareness among a number of peoples regarding AI.

It is an Autonomous Process Automation -

You must have to understand that we have not created another human brain rather
we have created a machine that works like a human brain. AI does the analysis, it
solves the problems and it performs autonomous automation based on its
experience, knowledge, and data. Don’t be negative and consider AI as a helping
hand that is going to ease our lives.
AI is different from humans -

Alexa and Apple Siri are AIs but they are far different from us. Whatever you speak
to them, they translate it to sound bites and then work on them. You will have to
come out of the fictional world. These are not self-learning machines, they do what
we command them to do.

After regular intervals, you get an e-mail that has a new set of commands for Alexa.
Hence we command them to work in our directions. You will have to understand that
these are not Einstein or Hawkins, not even close to them. Human Intelligence is
creative and we keep doing innovative researches. AI is just a result of our creative
intelligence. AI can’t be innovative, these can’t think and neither these can become
creative.

What should we do?

The next big question that must be revolving in your brain is what should we do?
Well, just control your expectations. Einstein and Hawkins were having human brains
which can think far more than any supercomputer. AIs are made to ease our lives
and we should not live in threat. More than 5000 peoples work on improving Alexa,
they are creating a better bot for us.

We will have to come out of the fictional world that Hollywood movies have created
around us. They are called machines for a purpose. You need not put a lot of stress
on the word “intelligent”. You should aware of the data and information that you are
sharing with them as once you have shared something it is impossible (virtually) to
bring it back.

Be aware and be educated.