AI

"A great design appears at first insane;

But chance will soon seem quaint and blind,
And such an exemplary thinking brain
Will soon by thinkers be designed"
- Goethe, ‘Faust’

pretty comprehensive list by Future of Life Institute on AI readings and orgs

Read

Intro
● The Artificial Intelligence Revolution - Wait But Why. Introductory post summarizing
the potential dangers of Artificial General Intelligence
● Life 3.0 - book by Max Tegmark.
● Society ofZ Mind - e-book by Marvin Minsky. Alternative definition of AI that highlights
a broader understanding of intelligence according to which societies can be
intelligent, too
● EFF AI Metrics - a useful guide to AI metrics that will be updated regularly
● AI 100 report - 100 year study on AI by Stanford
● A Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy - Seth
Baum. Survey of major AGI safety projects, comparison between academic vs.
corporate, different nationalities, and other factors.
● Singularity - Could artificial intelligences really outthink us? And would we want them
to? - book with chapters by Daniel Dennett, Ben Goertzel, Robin Hanson, Roman
Yampolskiy, Ray Kurzweil, David Chalmers

Risks
● Superintelligence - e-book by Nick Bostrom. Classic on Superintelligence, that
explores different take-off scenarios of AI, why they’re dangerous and urges for
preparatory action now.
● 12 AGI Aftermath Scenarios - summary scenarios of Tetlock’s Life 3.0
● That Alien Message - Yudkowsky. Challenges mainstream human intuitions on speed
of AI.
● Orthogonality Thesis - Bostrom, Sandberg, Douglas. Why Intelligence and goals are
orthogonal
● AI as positive and negative factor in Global Risk - Eliezer Yudkowsky
 ● There is no fire alarm for AI safety - Eliezer Yudkowsky.
● Slow Takeoff vs Fast Takeoff - Paul Christiano
● The AI Foom Debate between Yudkowsky & Hanson.Hard or slow take off?
● Likelihood around discontinued progress in AGI - AI Impacts. Rose McDermott.
● AGI Hype & Hope - notes on strategy session at 2017 Vision Weekend held by Peter
Voss.
● Intelligence Explosion Microeconomics - Yudkowsky. suggests a methods to test
whether a sufficiently advanced machine intelligence could build a smarter
version of itself, which could in turn build an even smarter version, and that
this process could continue to the point of vastly surpassing human capability.
● Basic AI drives - Steve Omohundro. sufficiently advanced AI systems of any
design would, by default, have incentives to pursue a number of
instrumentally useful subgoals, such as acquiring more computing power and
amassing many resources, which are harmful even if the overarching goals
are not.
● Research Priorities for Robust AI - Stuart Russell, Max Tegmark, Daniel
Dewey
● Roko’s Basilisk - Eliezer Yudkowsky

Definitions of Intelligence
● What is intelligence? Luke Muehlhauser on intelligence as optimization power divided
by resources used.
● Decentralized Approaches to Reducing Existential Risks - Mark Miller, Christine
Peterson, Allison Duettmann on intelligence as problem-solving ability of ecosystems
● The Singularity Is Near - Kurzweil on narrow intelligence.
● How Long Before Superintelligence? - Bostrom on superhuman intelligence
as smarter than the best human brains in practically every field, including
scientific creativity, general wisdom and social skills.
● AGI & Corporations Literature Review - Peter Scheyer on self-improvement as
necessary condition for superintelligence
● The Myth of A Superhuman AI - Kevin Kelly on different types of intelligence, arguing
humans have narrow intelligence.

Human - AI Hybrids
● Whole Brain Emulation Roadmap 2008
● Age of Em 2016 Book by Robin Hanson imaging future of human emulations without
strong AI
● FAQ on Simulation Argument - Nick Bostrom answers questions on his often wildly
misinterpreted simulation argument
 ● Machine Ethics Interfaces - Melanie Swan. Conceptualizes cognitive nanorobots, an
ethics of perception, and machine ethics interfaces.
● Carboncopies workshop with video on reverse engineering the brain
● From Brain Preservation to Reconstruction - workshop with video on brain
reconstruction
● The Ultimate Brain Interface - Ramez Naam
● Human Connectome Project
● Intelligence: Human, AI, Hybrids - panel I moderated for Vision Weekend with Randal
Koene, David Eagleman, Robert McIntyre, Nell Watson
● Neuralink - WBW post on Elon Musk’s Wizard Hat
● Is Brain Emulation Dangerous? - Peter Eckersley,

I give an overview of AI safety and why the field can be broken down into the four focus
areas below (Ethics, Technical Alignment, Cybersecurity, Social Coordination), including a
brief overview of the readings here: AI Safety: Why It’s Hard & State of the Art - Allison
Duettmann, workshop video for SXSW 2018

Ethics
● Value Alignment Landscape - Lucas Perry. Interactive, comprehensive overview of
the field of value alignment.
● OpenEth - crowdsourcing ethics to make it computable. Hit play!
● EthicsNet - evolution of OpenEth: org that seeks to build a framework for analysis of
various ethical situations, in a way that can be easily understood by both humans and
machines. 
● Machine Ethics index - Nell Watson. Literature index on machine ethics
● Humans Consulting CHC - Paul Christiano. Improving human decision making via AI
● Ought - new non-profit, relying on Humans Consulting CHC principles
● Coherent Extrapolated Volition - Eliezer Yudkowsky. Good paper to understand why
AI safety is hard. His solution CEV is still often cited, even if it is hard to implement in
practice.
● Objections to Coherent Extrapolated Volition -Lesswrong. Inter alia argues for
rationality as instrumental, rather than as end in itself
● Supermoral AI - Nell Watson. What if superintelligence comes with supermorality?
Against Orthogonality Thesis
● Moral Trade - Toby Ord. Just like trading resources, agents may trade morals.
Morality of agents would be more a matter of negotiation than normativity. Could be
applied to AI, too?
● Value Drift - Robin Hanson. Why values will drift and the values of future generations
might be incomprehensible for us now - and what this entails for AI safety
Technical Alignment
● AI Safety Research Landscape - FLI project on mapping AI safety research.
● AI Alignment: Why it’s hard and where to start - Eliezer Yudkowsky. Even if we knew
what we would like an AGI to do, how could we communicate this reliably? Includes a
nice overview of AI alignment strategies, including corrigibility, low-impact agents, the
value learning problem, act-based agents
● Directions and desiderata for AI alignment - Paul Christiano
● Ensuring Smarter-Than-Human Intelligence Has a Positive Outcome talk by Nate
Soares @Google
● Prosaic AI alignment - Paul Christiano
● Learning From Human Preferences - OpenAI
● Evolution Strategies - OpenAI. On the surprising success of evolutionary strategies
compared to known RL strategies
● Apprenticeship Learning via Inverse Reinforcement Learning - Pieter Abbeel, Andrew
Ng - Learning by observing human experts
● Formally Stating the Alignment Problem - Gordon Worley
● Intelligence Distillation - Eric Drexler. Instead of building a more or less unitary
superintelligence, we could attempt to distill advanced intelligence so as to reap the
benefits of superintelligence in certain problem domains without having one unitary
intelligent entity.

Social Coordination
● Reading Guide for the Global Politics of Artificial Intelligence - comprehensive
reading list by Allan Dafoe.

Near-term opportunities to influence AI policy

● The Malicious Use Of Artificial Intelligence: Forecasting, Prevention, and Mitigation -
strongly recommend this paper, published by FHI, written by Miles Brundage Shahar
Avin Jack Clark Helen Toner Peter Eckersley Ben Garfinkel Allan Dafoe Paul Scharre
Thomas Zeitzoff Bobby Filar Hyrum Anderson Heather Roff Gregory C. Allen Jacob
Steinhardt Carrick Flynn Seán Ó hÉigeartaigh Simon Beard Haydn Belfield Sebastian
Farquhar Clare Lyle Rebecca Crootof Owain Evans Michael Page Joanna Bryson
Roman Yampolskiy Dario Amode
● Guide to working in AI policy - Miles Brundage. Lists good orgs one could work for to
make a positive difference in AI policy
● Policy Desiderata in the Development of Machine Superintelligence - Nick Bostrom,
Allan Dafoe, Carrick Flynn. Birds-eye view of policy areas that would be radically
affected by AI and recommendations on how to approach policy development
● Smart Policies for Artificial Intelligence - Miles Brundage, Joanna Bryson
China
● Alibaba: The House that Jack Ma built- Duncan Clark. Duncan gave a talk at the Asia
Society on Big Tech in China. The talk gave a useful overview of tech giants in China
but was off the record. The next closest option is this Google talk

Emergent race dynamics between great powers, e.g., between the US & China
● Deciphering China’s AI Dream - Jeffrey Ding.

Historic strategies for achieving cooperation between different players in games

● Doomsday Machine - Daniel Ellsberg

General concerns regarding open vs. closed research

● For more on how openly sharing technology with extreme destructive potential might be
extremely harmful, see Scott Alexander’s Should AI be Open?, and Nick Bostrom’s Strategic
Implications of Openness in AI Development.
● Openness Considerations for AI - Nick Bostrom.

Potential for increased coordination between AI safety organizations

● Unilateralist’s Curse: The Case for a Principle of Conformity by Anders Sandberg,
Nick Bostrom, Tom Douglas

Strategies for Coordination

● Submission to OSTP on AI Outcomes MIRI’s submission to OSTP’s request for information -
final part focuses on social coordination, discourages strong regulation and encourages
multinational and multicorporate collaborations and more interdisciplinary reseach into
technical challenges outlined in the text
● Long-term strategies to end existential risk from hard take-off - Daniel Dewey. Covers 4
scenarios: International coordination, sovereign AI, AI-empowered project, Other decisive
technological advantage
● AGI: Timeframes & Policy White Paper - Allison Duettmann. Paper based on meeting
that Foresight held on AGI timeframes. The main part of the report focuses on three
high-risk scenarios, (1) cyber security, (2) near-term AI concerns, and (3) cooperation
leading up to Artificial General Intelligence.

Corporate Artificial General Intelligence

● Livestream of seminar on Artificial General Intelligences & Corporations @Internet
Archive
● AGI & Corporations Literature Review - Peter Scheyer’s literature review that is the
base of this seminar
● Decentralized Approaches to Reducing Existential Risks - Mark Miller, Christine
Peterson, Allison Duettmann on civilization as superintelligence
● Corporations are 1st generation AIs - Brewster Kahle.
● Personhood - A Game for Two or More Players - Kevin Simler
 ● The First Machine Intelligences - Danny Hillis.

Listen & Watch

● Beneficial AI Conference - strongly recommended Youtube channel by FLI on their
2017 conference
● 3 Principles for Safe AGI - TED talk by Stuart Russell
● Concerning AI - Podcast on AI by Ted Sarvata & Brandon Sanders
● Talking Machines - Podcast on AI
● Sam Harris & Eliezer Yudkowsky - good intro on common problems in AI safety
● Yudkowsky vs Hanson: Singularity Debate
● AI Ethics Panel: Yudkowsky, Tegmark, Chalmers, Russel, Wallach, Petersen
● The Three Schools of the Singularity - Eliezer Yudkowsky
● AI Safety Overview & Definitions - Allison Duettmann. I give a very quick overview of
the field of AI safety and distinctions between AI, AGI, Superintelligence.

Do

AI safety orgs
● Partnership on AI - Established to study and formulate best practices on AI
technologies, to advance the public’s understanding of AI, and to serve as an open
platform for discussion and engagement about AI and its influences on people and
society.Established to study and formulate best practices on AI technologies, to
advance the public’s understanding of AI, and to serve as an open platform for
discussion and engagement
● Machine Intelligence Research Institute - MIRI's artificial intelligence research is
focused on developing the mathematical theory of trustworthy reasoning for
advanced autonomous AI systems.
● OpenAI - OpenAI is a non-profit artificial intelligence research company that aims to
promote and develop friendly AI in such a way as to benefit humanity as a whole
● DeepMind - AI company, bought by Google. Has strong safety focus
● Center for Human-Compatible AI - CHAI's goal is to develop the conceptual and
technical wherewithal to reorient the general thrust of AI research towards provably
beneficial systems.
● Leverhulme Centre for the Future of Artificial Intelligence - A global community to
ensure that AI benefits all of humanity
● Center for the Safety and Reliability of Autonomous Systems - SARAS is an
interdisciplinary organization focusing on autonomous system safety and reliability.
● AI Council on AI & Robotics - World Economic Forum
 ● SFPL - Make sure all of San Francisco has access to resources that aren’t free by
searching for them in our public library’s catalog and suggesting titles if they aren’t
available
● Future of Humanity Institute Future of Humanity Institute (FHI) is multidisciplinary research 
institute working on Existential Risk at the University of Oxford.
● Future of Life Institute - a volunteer-run research and outreach organization in the
Boston area that works to mitigate existential risks facing humanity, particularly
existential risk from advanced artificial intelligence
● Foresight Institute - Foresight Institute is a leading non-profit research organization
focused on technologies of fundamental importance for the human future, focusing
on molecular machine nanotechnology, cybersecurity, and artificial intelligence.

Websites
● AI Alignment - really good blog by Paul Christiano on anything alignment.
● AIsafety.com - virtual reading list on AI safety, meets every Wednesday.
● Superintelligence Survey - survey on people’s opinion toward superintelligence

Cyberspace
Governments of the Industrial World, you weary giants of flesh and steel, I come from
Cyberspace, the new home of Mind. On behalf of the future, I ask you of the past to leave us
alone. You are not welcome among us. You have no sovereignty where we gather.
- JPB, Declaration of Independence of Cyberspace

Intro
● Declaration of Independence of Cyberspace - John Perry Barlow. Speech given by
JPB at the World Economic Forum on the properties of cyberspace that make it
unlike any other invention.
● Transparent Society - David Brin
● A Hacker Manifesto - McKenzie Wark. criticizes the commodification of information in
the age of digital culture and globalization
● The Stack - Benjamin Bratton. He proposes that different genres of planetary scale
computation -smart grids, cloud platforms, mobile apps, smart cities, the Internet of
Things, automation- can be seen as forming a coherent whole: an accidental
megastructure that is both a computational infrastructure and a new governing
architecture.
● Symbiot Papers - Three papers on cybersecurity offense and evolutionary inspired
risk models and architectures
Cybersecurity & AI
● AGI Timelines & Policy - go to section on cybersecurity - Allison Duettmann.
Whitepaper based on private meeting on AGI safety in 2017
● Cyber, Nano, and AGI Risk - and Decentralized Approaches to Reduce Risk -
Christine Peterson, Mark Miller, Allison Duettmann. Cybersecurity as near-term,
undervalued catastrophic risk that is more imminent than risk from AGI and potential
solutions.
● AI and Cyber security - Roman Yampolskiy. Cybersecurity approaches for AI safety
● AI Alignment & Security - Paul Christiano
● The Malicious Use Of Artificial Intelligence: Forecasting, Prevention, and Mitigation -
go to section on cybersecurity. Strongly recommend this paper, published by FHI,
written by Miles Brundage Shahar Avin Jack Clark Helen Toner Peter Eckersley Ben
Garfinkel Allan Dafoe Paul Scharre Thomas Zeitzoff Bobby Filar Hyrum Anderson
Heather Roff Gregory C. Allen Jacob Steinhardt Carrick Flynn Seán Ó hÉigeartaigh
Simon Beard Haydn Belfield Sebastian Farquhar Clare Lyle Rebecca Crootof Owain
Evans Michael Page Joanna Bryson Roman Yampolskiy Dario Amodei
● Agoric Papers - Eric Drexler, Mark Miller. Three papers that became classics for their
treatment of computation as an economic system.
● Symbiot Papers - Three papers on cybersecurity offense as defense and evolutionary
inspired risk models and architectures
● Opensource Hardware Design Principles - guarding privacy, avoiding trapdoors
● Purism - open-source hardware design company
● Cyber Insurance - Sandberg, Petratos, Zhou. Book chapter to be published soon on
which cyber risks could be insured.
● Cyberinsecurity As Catastrophic Risk - notes on strategy session at 2017 Vision
Weekend, held by Mark S. Miller

Do
● Electronic Frontier Foundation - The Electronic Frontier Foundation (EFF) is an
international non-profit digital rights group based in San Francisco, California.
● Data & Society - Data & Society is a research institute focused on the social and
cultural issues arising from data-centric technological development.
● Internet Archive - Internet Archive is a non-profit library of millions of free books,
movies, software, music, websites, and more
● Center for Long-term Cybersecurity - is developing and shaping cybersecurity
research and practice based on a long-term vision of the internet and its future
Computing
● Steve Jurvetson on Quantum Computing - After-On interview with Steve Jurvetson
on Quantum Computing