Alright, so can we start with just you introducing yourself—your name and your position?

My name is David Danks, I’m officially the L.L. Thurston Professor of Philosophy and
Psychology and affiliated faculty in the Heinz College of Information Systems and Public Policy
here at Carnegie Mellon.

So, David, can you tell me a little bit about how your work in relation to AI began?

So, it began in a certain sense when I was about 8 years old and I was one of those little
wanted to be a white hat hacker kind of kids, who just loved playing with computers and loved
trying to figure out how to get systems to behave in sophisticated ways. Even though I had done
all the programming, it was sort of fascinating to me as a kid. Where it really took a turn was in
graduate school when I started, principally for intellectual reasons, to think about the ways that
all of the work that was happening in the late 90s in AI could inform the work I was trying to do
in cognitive science. So, I was at that time in philosophy graduate school working on problems
of how humans understand the causal structure of the world, and it seemed to me that what we
knew, or didn’t know at the time about how machines could do this, might be informative to the
cognitive science. And when I shifted and started to look at that sort of AI, it became very clear
that there were a lot of interesting puzzles over there. There were a lot of things that the human
mind was and still is able to do that we actually don’t even know how to start to get a machine to
do it. It isn’t that we know and we just have to finetune the algorithm. We aren’t quite sure what
it is the human’s doing that we would try and get a machine to replicate. So that was a very
interesting way for me I think to come into AI. It wasn’t coming into it trying to build things, it was
coming and trying to actually use AI to understand human cognition. And that was actually much
of the way that I was using AI for quite a while. It was developing algorithms that could
complement the way that humans were understanding the world, so, for example, the idea that’s
very popular nowadays of having explainable AI and AI that a human can understand why it
made the prediction it did. That’s actually a constraint I’ve had on my own work in AI since the
late 90s. Because it just seemed to me, I want something that fits with a human mind, so it
better be able to interface appropriately with it. So that was much of the work in sort of
developing AI and then in the last 5 years, I’ve been turning much more to ethical and policy
issues around AI and a bit less in the actual building of the systems.

So, sounds like a lot of your work was squarely within the academic realm and quite pragmatic,
but were you ever influenced, do you think, by elements of popular culture, science fiction,
television?

On the ethics and policy side yes, but the influence is perhaps not the one people would expect.
It was an influence of deep annoyance. I looked at what was being put out there in popular
culture and as somebody who at that point, so say around 2013 when I started to really work on
the ethics and policy side, at that point I had been doing a certain kind of AI and machine
learning for 15 years and it suddenly became something that everyone was worried about, and I
kept not seeing in the public presentations, the kind of thing that I was doing. I would look and
say that’s just not what we do. That’s not how we think about these things. So actually one of
the reasons that I started to do work in this space was exactly because the popular, and by
popular I want to include sort of the intellectual blogs, right so it isn’t just the mass market film,
it’s also the way it was written about in the OpEd page in the New York Times, became deeply
frustrating to me. I just got annoyed and in true academic fashion when you get annoyed, you
say well I’m going to do something about this. I’m going to start writing papers. I’m going to start
to think about what’s the right way to think about these technologies. Because it seemed to me
that much of what was happening in popular culture was not being driven by people deliberately
misrepresenting, it was the AI technologies, it was that they truly didn’t know how to think about
them or how to talk about them. And so, they were using sort of the stories and schemata that
they had available to them which turned out not to work in my view for a lot of things with AI.

Yeah, so precision in communicating about these systems is actually something we’ve been
really concerned about and have been analyzing. So, in regards to your developing work and
your policy, and also attending to this practice of communication, can you offer an example of a
particularly accurate, precise or useful communication in regards to AI for broad publics?

It’s very difficult, I think. I think that the vast majority of discussions of the technology are in part
skewed by the very way that the technologies themselves are being presented by the
companies. And the way the companies present the technology is, depending on how cynical
you are, either accidentally or deliberately designed to trigger in people ways of thinking about
the technology that are in fact not appropriate. I remember having a discussion when we got an
Alexa, Echo, you know, in our house, having a discussion with my, at that time 14-year-old
daughter about how it is that it seems like Alexa is producing these complete sentences but that
there’s no real content behind there in the way that she’s thinking about it. And I tried to use it
as a teachable moment, but it also highlighted for me the way that the natural language
interface, it can’t help but mislead people into thinking about the technology in a different way.
And I think you see this over and over, you see there was at one point, I don’t know when this
will come out right, but there was a YouTube clip that went viral of a Boston Dynamics robot that
comes, can’t open a door, and so go gets another robot that can open it. But all the mannerisms
of the robot are those of a dog. So, it sends all of these signals that you can’t help but think of it
as a dog, and it’s not. It’s not as intelligent as my dog who’s you know nice but not always the
smartest dog. These robots are not even that level. And yet the presentation, the performative
aspect of all the technologies consistently triggers useful, perhaps, for the company, but not
accurate renditions. And then I think that spills over to the popular conceptions. So, I think it’s
actually hard to find in the popular press, even in OpEd pieces, clear careful accurate depictions
of what these technologies really are and what they’re capable of. So sorry I can’t give you a
good example. I think that’s actually one of the persistent problems we’ve had over the last five
years. Actually more, but especially over the last five years, is the ways in which people think
about these technologies and then they’re surprised at the “failures”, when if you know how the
technology works, almost all the things that are being held as “failures” you look and go well of
course it wasn’t going to be able to do that. Why would you have thought it was? And you would
have been like oh, because you thought there was actually reasoning going on. You thought it
was actually logically thinking through what the right thing to do is. So, I think it’s a big problem
we have on the policy side, and certainly if you go down to Washington D.C. you find a lot of this
happening.

Yeah so on some level it comes back to some of those early questions it sounds like you’re
attending to which were basically using these engineered systems that can control the variables
in terms of input, really paying attention to those logical progressions, and then compare that to
the complexity of variables that play in a human mind.

That’s absolutely right, I’m a very big believer in the idea that humans are incredibly good at
certain things and incredibly bad at others. We are great at recognizing that there’s been a
subtle contextual shift, something’s just not quite right or not quite the same. We’re very bad at
multiplying 10-digit numbers. We’re very bad at remembering exact details, I mean the literature
on human memory is filled with all of the ways in which we are fallible in entirely predictable
ways at remembering very precise details. We’re very bad at vigilance. It’s hard to sustain
attention over a long period of time. What are machines good at? Vigilance, memory, numeric
computation. What are machines bad at? Subtle contextual changes, adapting to new
environments. So, it seems to me that we actually in some ways have this beautiful possibility to
let the human do what the human is good at, let the machine do what its good at, and view it all
through the lens of partnerships and teaming and pairing and whatever term one wants to use.
The problem is that that requires the technology developers to think really carefully about what it
is that the user of the technology, this human, wants. What are they trying to achieve and how
do we play, how does the technology play well with that? With the human’s knowledge and the
human’s abilities. That’s hard to do and it isn’t always financially sensible to do if you’re trying to
build a big company. So, I think actually a lot of the failures that have occurred, or a lot of the
scandals or crises that have occurred in industry over the last year, year-and-a-half, many of
them are really centered around the technology is designed not for the human user. It’s
designed for some other purpose. And so, when people try to team with it, they’re continually
frustrated by it because it isn’t actually designed to support their values.

And they’re also ill-informed on what its actual capability or functions may be.

Right, so the frustration comes through in the sense of, I keep expecting it do something and it
doesn’t, or it can’t, and I don’t know why. You know, the psychological literature is pretty clear
that people are willing to tolerate many failures in a system if they know why the system failed. If
it’s explainable, predictable and avoidable, then people will go okay, yeah that didn’t work but I
see why, and I’ll just make sure next time I don’t rely on it. It’s when things happen for
seemingly no reason or no reason that the person can extract, that’s where you start to get loss
of trust, loss of faith both in the technology and the companies that produce it.

Yeah, I want to talk more about trust in a few minutes, but I’m wondering in your work in terms
of policy recommendations in terms of drawing on your work in ethics, but also as an educator,
as a head of a department at a university with deep roots in technological advancement. What
do you see as your responsibility in communicating on AI to the public?

I think it’s a pretty massive responsibility. I think that we in academia have tremendous freedom
to ask questions, to teach the next generation of thinkers, to lead the way in pushing the
frontiers of human knowledge, without very much in the way of external control. And I think that
that great power carries with it an obligation to engage with the public, to help to share all the
knowledge we have with the public so that they’re better able to understand what’s happening. I
want to make sure to be careful, I think there are many ways in which that public engage and
outreach can happen. I think that can happen in the kind of obvious ways: YouTube videos,
OpEd pieces, going and giving public lectures. I think it can actually happen in subtler ways. I
think it can happen by going and talking to a Congress person and trying to help the Congress
person or his or her aide understand here’s what the technology is, so when you’re writing laws
or you’re helping to shape policy around the technology, here are the questions you need to be
asking. That can sometimes have an even bigger impact, even if nobody knows that you’ve
done that. I prefer not to think of it in terms of outreach and more in terms of public service.
Serving the interests of the public, it in fact can even sometimes be working with the companies
themselves. More and more tech companies are bringing in people to serve as consultants to
varying degrees of success of course around issues of ethics, public good, supporting the
interests of their users and their clients. And I think that helping the companies do better is itself
a kind of public service and so I think one of the things that is very important is for academics, at
least many of us, to engage in that kind of public service exactly because there’s so many ways
to do it. I think there isn’t just one way to help educate the public. There isn’t just one way to
help the public be better supported by technology, so it’s all the more reason that people I think
ought to be taking on one or another of these roles.
And it sounds like you’re also trying to equip students as well with better information, but also
ideally the tools to become more inquisitive in their consideration of technology’s impact on
society.

And this is a somewhat controversial opinion, I don’t think that the student in the new AI major
that we have here at Carnegie Mellon, I don’t think that that student has to become an ethicist. I
think that student needs to understand when the app they want to develop has significant ethical
concerns or implications and then be able to call and talk to somebody who can help them think
through their problem. We don’t ask our doctors to be able to handle every ethical dilemma
they’ll confront. We do ask our doctors to recognize the ethical dimension of treatment of a
patient and recognize when they might be in over their heads and to call and ask for help. And I
think we should have exactly the same expectation of the tech developers. I think we should
have the same expectation actually in the other direction of the policy people. That policy people
have an obligation to know enough technology to recognize I don’t know all the answers here
and I have to be able to collaborate. So, a history major here at Carnegie Mellon I think ought to
know enough about AI, if they’re intending to go into policy, to be able to talk to somebody
who’s a tech developer, to be able to recognize when I need help, this is not something I
understand right now. And so that’s been, that is, has been and is moving forward one of my
real goals personally as an educator here at Carnegie Mellon, is to find ways to help the
students understand that there are many different perspectives that one can take to understand
the impacts of a technology: ethical, economic, philosophical, sociological, historical, political, to
be sufficiently familiar with these perspectives to see what each one could bring and then to
know when to call and ask for help from somebody who is trained in one of those perspectives.

So is it fair to say your goal is ensure that students who are studying at the undergraduate level
are conversant, but also understand the depth of the expertise that can be drawn upon to really
attend to the questions that might pop up in their actual practice?

I think that’s a great way to put it. I mean selfishly, or not selfishly, but egocentrically I find in my
own work I know just enough about certain areas to know when I need to go talk to somebody
else, when I can say, you know what this has run up against the limits of my understanding, I’m
going to go get help. And I think that that requires a willingness to pause and think, not very
long, but you do have to think long enough to think through what are the impacts potentially of
the technology that I’m developing. So in certain ways I recognize that this is a bit antithetical to
a lot of the zeitgeist around a place like Silicon Valley where it’s, you know, develop as fast as
you possibly can, don’t worry if you break things along the way, there will be time to go back
and clean it up, even though no one ever does go back and clean it up. And so I recognize that,
so it’s actually a research interest of mine is how can you develop ways to help people quickly
figure out whether something is problematic and ask for help, rather than having to step back
and do a two week analysis on every single idea they might have, because I think if that’s what
we as academics, as educators, are telling people it will never make an impact on the practices
in industry.

So, my understanding in reading some of your work is it’s not necessarily squarely in position in
regards to labor but I’m still curious about the ways in which you think that AI systems have
changed the way that people have worked up until now?

I think part of it depends on how broadly we want to scope AI system. I think that computational
technologies have been an enormous boom to many people’s work lives by taking tasks that
were very mindless and enabling them to be done by a machine so that people could focus on a
more, the more interesting parts, and I don’t think that’s just a creative vs. non-creative class
kind of thing, I think the fact that you don’t have to throw away a piece of paper if you make a
typo halfway through on a typewriter has made everyone’s life in corporate America better even
the people whose job is principally transcribing dictation. I think their lives are better as a result.
So, I think there are ways in which technology has improved the lives of workers. There’s also a
lot of ways in which technology has replaced workers and I think we often focus on the
economic arguments around that, often to the neglect of psychological arguments. If a machine
could suddenly do my job, I wouldn’t just be out a paycheck, there would be real psychological
harm done to me, part of my identity is to be a professor. And I think we often underestimate
how important that is. I also think that technology has made most people’s lives, maybe not
most, many people’s lives even in highly corporate settings, not just on a manufacturing floor,
much more structured and much more constrained. The machines now dictate what people are
able to do. The machines will say no, and then somebody says how do I override that, and they
say you can’t. Now why can’t they, well because from the perspective of the company often
times they in fact have found that the humans don’t override it when they should, the humans
override it more often than they should. So, from the company’s perspective they’re better off
not allowing the human to override, at least in the short run. What that does though is that it
changes the relationship between the worker and their understanding of the role they play in the
company. They’re now just a cognitive machine. They have lost a part of the autonomy, a part
of the authority that they previously had and so they’re going to lose identification with the
company as one would naturally expect. It changes the interaction of outside consumers’ clients
with the company because now they’re being told no for an opaque reason and there’s nothing
anyone can do about it, so one would expect a loss of trust. And I think in general, the ways at
least in most of the western industrialized nations that technologies have been deployed has
been in service of a kind of rationalism about companies and economic engagements that
misses out on the psychological and emotional aspects of economic engagement. And so, I
think, you know, in the early days that was not a problem, you were getting rid of the emotional
engagement of anger that you had a typo and had to redo it. Now it’s getting rid of the emotional
engagement even I think we can see it coming in places like the emotional engagement of a
patient with a doctor which we might not think of as an economic interaction, but it clearly is.
You know, imagine the machine, the computer says no but now it’s the computer says here’s
your diagnosis and the doctor can’t change that. There are plenty of medical conditions now
where AI systems are better than the average doctor. How long is it going to take before the
insurance company says to a hospital you have to use this system because overall it’s going to
save more lives, overall the diagnoses will be more accurate? But now your doctor is just a
middle person, they’re just an information broker that passes information from the machine to
you. What harm does that do to the relationships of trust? How much are we winning in the
short run with efficiency gains and losing in the long run with exactly the kinds of human
connections that are, well the economists don’t like to think of it this way, actually at the heart of
most economic exchanges. So that’s the way I tend to think about the economy. I don’t think
about labor per se, I’m not trained as an economist I can’t do the big work-force analyses. I’m
much more interested in the individual human element within the economic space. How does it
change my relationship with a company that I can’t talk to a human who can understand my
particular weird situation? In that regard, I think there’s a real risk right now that there’s short run
gains with long run risk. Especially given, for example, here in the United States, the incredible
pressure on companies for short run performance. Every incentive is there to do things in such
a way that its short run, right, consultants go to companies and say use our AI technology and
you can fire people. It’s never use our AI technology and look at all the extra things your
employees will have time to do. I mean if you ever look at the pitch-decks, that these companies
consultants come forwards with, or tech companies, it’s always about reducing the work force,
reducing cost. It's never about this is a way to increase your income. And I think those kinds of
framing issues have pushed us down a path that I’m not sure is going to end well for a lot of
these companies in the long run.

Okay, I want to take a deep dive into some of the power negotiations that you allude to and also
some of your work in regards to autonomy in a second, but you started to touch on the potential
infringements on human dignity and also some of the psychological implications in the shifting
work environment as individuals are working with these AI systems. What do you think might be
some of the, the biggest challenges 20 years from now? We’ve talked a little bit about this
boundary space, but I’m wondering if you can touch on that a little.

I’m always hesitant to do predictions because I think we often don’t know, I think that the
trajectory that our societies and our technologies are going to be taking are going to be. These
aren’t things that just evolve in some undisturbed way. One of the things that actually that
bothers me about a lot of the efforts to predict is that invariably you find either a utopian or a
dystopian vision, and I think that those are typically driven by a kind of background implicit
assumption that technology is like this hurricane and it’s just bearing down on us and we will
either make it through into the, you know, blissful sunshine or we’re going to be devastated and
have flooding everywhere. But in practice who builds the tech? We do. Who makes the policies
around the tech? We do. If it’s hurricane, it’s a hurricane with a whole bunch of people standing
there with fans, which is a terrible extension of the metaphor, but the idea is we actually have a
lot more control over this trajectory. And the we here by which I mean is humanity. We have
much more control over this trajectory than we think we do, or at least than is commonly
presented in predictions. So, if you ask me to predict where we’re going to be in twenty years, I
think for example, an enormous factor is how much technological shift continues to contribute to
income and social inequality in most of the countries around the world. If there’s a political
upheaval such that it’s made clear that technology needs to help reduce inequality, I think we’ll
end up in a very different place with regards to human dignity than we are right now. If
companies are provided with incentives perhaps because of the market, perhaps because of
policy or law changes to focus on longer run performance as for example one frequently finds in
Germany. Then I think you’re going to find much more interest in developing technology that
supports and helps people and so therefore can help to preserve dignity. If there is the next
generation of developers coming out of places like Carnegie Mellon come out with an
understanding that they shouldn’t just think about the cost-benefit analysis of the technology,
they also should think about how it shapes people’s lives, whether notions of dignity or identity
or trust. Then I think we’re going to be in a much better place. And I don’t really know how to
predict you know which of those is most likely. I will say I think I can imagine every possible
future from 20 years from now we’ve had a series of upheavals that have led to us to
understand that all technology development should be guided by how it supports the interests
and values of the people engaging with it. You know I can also see a future where the 1%
becomes the 0.1% becomes the 0.01% and the technology gets skewed to support the interests
of an incredibly small minority of people and is continuing to do real harms to broad swaths of
the population and especially, one would expect, to already marginalized and disadvantaged
parts of the population. And everything in between where some disadvantaged groups figure out
how to leverage technology to advance their interests and help support their values to ones
where you have sort of the upper half of the socioeconomic sort of strata is able to benefit, and I
can sort of imagine all of this, and I think one of the questions that we have to be asking
ourselves right now is what kind of a future do we want? And to not passively accept that these
are events taking place in Silicon Valley and Washington D.C. and New York City and Boston
and Pittsburgh. But to recognize, we get to play a role in this by using our voices, by speaking
up about what it is about technology that we want or don’t want, by getting off of Facebook if
you actually are bothered by it. I think that that’s a role that we as citizens have to be willing to
play, to try and move these trajectories around.

Yeah, I think that notion of the agency and responsibility associated with shaping that future is
something that I am curious about also because the scenario with the notion of hang ringing and
the idea of being subject to technology as though it’s somehow just being bestowed upon
society rather than actually developed by societies. I wonder if, if on some level those are some
of the latent effect of people’s interactions with these systems. Right, the ways in which there’s
deference to certain tools or seeming deference to the powers that be. I know that your work
has really delved into trust and I’m wondering if you could talk a little bit about a particular AI
tool or system that you’ve worked with in your research or that you’ve been paying attention to
in particular where the power has been transferred from the human user to that system.

I’m actually going to go back 20+ years for this to a system that I think most people don’t even
realize has been running for 20 years which is credit card approval. So, for 20+ years 99-plus
percent of credit card applications, no human ever looks at them, even back in the days when
you had to handwrite out the application, it would come in, a machine opens it, runs it through
optical character recognition, gets pushed against a model and its decided whether you get a
credit card, if so what the credit limit would be. Now you fill out an online form. No human looks
at this. There’s no human decision that says, “Oh you have a stable job and income of this kind
so therefore we can trust you with a higher credit limit”. There’s nothing that looks and says
you’re the kind of person who has, is just about to graduate from college so you’re going to
need a higher credit limit, so we’re going to give that to you. There’s nothing that says you’ve
had the following experiences as a child that’s made it difficult for you to manage money, so in
fact it would be better for you not to have as high a credit limit, even though it would be in the
interest of the company to give you the higher credit limit. The agency has been moved from the
human. The granting of credit and this now has become more widely known because you’re
seeing it with the larger loan applications, its happening quite a lot, but the granting of credit,
which in a modern Western nation is incredibly important for any sort of social and economic
mobility. The granting of credit has shifted away from an engagement with the person and
understanding whether they are trustworthy to a machine that makes its decisions solely on the
basis of measurable factors that somebody thought to include and was hopefully able to
accurately measure and encoded correctly in the system and programmed it correctly and its
wired up correctly on the backend to be able to be used correctly and make the right actions in
the world. The human’s out of the loop and has been out of the loop for 20 years. And I think
we’ve seen a shift because people have learned how to game the system, they’ve learned how
to play tricks with the system. Companies have put in new rules to say you can only apply for a
certain number of credit cards at a time. And then people don’t realize the impacts these are
having on their own credit and I think that we’ve seen the process of getting access to resources
shift from human agency, human engagement, human decisions to the construction of an ever
more Byzantine system of rules and criteria that people are attempting to game on both sides.
And that’s transformed I think, our ability—people’s ability to be mobile: physically, socially,
economically, in ways that are disproportionately falling on the already disadvantaged among
us.

So, in regards to the scenario that you suggested at the very beginning of our conversation of a
relatively symbiotic relationship between these AI systems and the human users, is there an
example of this system that you think is actually empowering an individual or group of people?

So, I think that there actually are a number of rapidly developing systems in the educational
sphere where there seems to be a better recognition that, there was the boom of MOOCs,
massive online open courseware, where the idea was to replace, by and large, the teacher with
the machine. I think there are now a number of instances of educational technology
development where the idea is to partner the technology with a teacher, to enable the teacher to
provide assistance to the students in the way that the student needs to where the technology
can assume some of the responsibility from the teacher and thereby free him or her up to
engage one-on-one with students. That’s in place, what sometimes gets called blended
classrooms. And I think that that’s actually perhaps the place where we’re seeing some of the
best instances, now I want to be clear we’re also seeing some horrible instances, it’s being done
very badly in some cases well, but if I had to look for some of the very best examples of that
kind of partnering, it’s coming out of actually educational technology. Now a worry I have about
these systems, is that they, the wrong lessons will be drawn from them, that people will say “Oh
the technology makes it so that the teacher doesn’t have to do certain kinds of things and has
more time, therefore we can give the teacher more students.” Right, so the worry is going back
to the issue of do you use the technology to enable people to do more, do better, or to you use it
to reduce the number of bodies. I worry that we’re already seeing this sort of pattern play out in
school districts around the United States where the technology comes in and people say great,
now we only need one teacher for every 45 students or they already had one teacher for 45
students, everyone said this is appalling, and they said no now it’s okay. Rather than using
these to enable us to do better educational delivery.

So, in regards to autonomy, I’m wondering if you can share a little bit on what you perceive as
valuable in the prospect of machine autonomy?

So the first thing I would say is I don’t like talking about autonomy, I like talking about
autonomous capabilities, and I realize that sounds like such an academic thing to do. Right,
take a work and change it to a denser, more jargony word. But the reason I think that that’s
important is because I think it’s important to think about the senses in which a system can
function autonomously. There are systems that can autonomously perceive, but actually can’t
do anything else beyond that, they can’t plan, they can’t act in the world, but they’re great at
autonomously perceiving what’s happening around them. And actually that’s one of the areas
that AI has done the best over the last decade. So, I think it’s important to think about
autonomous capabilities, and so why do you want systems, well why might we want systems
with autonomous capabilities? I think a key is that we should aim for systems that have
autonomous capabilities that can do the things that we either don’t want to or we ought not be
doing. Whether that is because it’s going into hazardous environments, whether it’s because
these are simple tasks that the machine is better at than we are, whether it’s because the
system is able to free me up to do other things. So, let me take a simple example, consider the,
you know, now overused example of self-driving cars. My own view about self-driving cars is
that they are incredibly good in certain cases already, there are certain contexts and situations
where they are, in my view, arguably better than human drivers. If I’m driving across Oklahoma
or Texas on a freeway on a sunny, summer day, I would rather have a machine driving me right
now, the best self-driving cars, I would rather have them drive than me, because they will be
better at it than I will. You’re dealing good lighting conditions, low-likelihood of anything
anomalous happening, and a really tough situation actually for the human driver because you’re
just doing the same thing over and over, there’s loss of situational awareness, loss of vigilance.
If it’s snowing here in Pittsburgh on a back alley, I don’t want to go anywhere near a self-driving
car. Right, so it’s capabilities that these vehicles have right now are ones that can really
complement my abilities and they can do better than me, they can free me to be able to engage
in other activities when used appropriately, but that last part is critical, the “when used
appropriately”. The machines right now do not have, and by and large, I don’t know of any
instances right now of AI that I would say has the autonomous capability to understand the
context. They can respond to signals from the environment, but to understand it, in the sense
that we as humans are able to adaptively understand a complex context. They aren’t yet able to
do that. And so, what I really want is I want AI technologies that are designed in a way that I can
say this is an appropriate context to use it, go do your thing. But where I have had to, I think that
that’s one of the things that at least for the moment, on purely pragmatic grounds, be clear I’m
not necessarily making a moral argument, just on purely pragmatic grounds I think right now,
these decisions about when to deploy the capabilities, by and large need to remain with the
humans, because we’re just much better at making those kinds of decisions than the machines
are right now. So, when you ask why do I want AI, it’s because the machine’s just better at
things than I am. I’m glad that I’ve got a computer that can do lots of tasks on my behalf. I’m
glad that I have adaptive cruise control in my car, if you think that that’s a kind of early semi-
autonomous capability. Even if I had a Tesla, I would only use autopilot in very specific
circumstances, and I’m glad that Tesla’s can’t just declare “Oh it seems like you’re driving on
the freeway, I’m going to turn on autopilot now”. I think it’s good that Tesla isn’t able to do that.
So, what I want, I want AI because it’s helpful, I’m a techno-optimist in that regard. I think
technology can make our lives much better in many ways, but it has to be designed and
deployed correctly.

And it sounds like the terms of engagement need to also be very clear to the user to actually
engender the level of trust to allow for those autonomous capabilities to actually even be
utilized.

Exactly. I mean a lot of the work that I’ve done on trust has focused on the importance with
these kind of technologies on going beyond sort of mere predictability that the kind of trust we
need in these technologies is a trust that enables us to understand how the system would
respond in unusual or anomalous context. And I think, by and large, we don’t have that with
most of these technologies, we don’t know what would happen if they were suddenly put in a
snowstorm, or if suddenly one of the sensors failed. And so, if the technology is not going to be
designed for us to be able to figure that out, then it had better be designed for us to be able to
say, you know what we’re about to go outside of normal operating conditions, no I’m going to
turn this off because I don’t know how the system will function in these contexts. We have to
have, not an off button, but we need the ability to say this is not the kind of situation that the
technology was designed for which requires us to know what it was designed for of course. But
then also to have the ability to say, “You know what, we’re not going to do this.” Just as one
might do to a sixteen-year-old learning to drive, say “You know what, this is getting to be kind of
tricky, you aren’t there yet. Let’s pull over to the side and let’s swap and let me do the driving on
this occasion.” So, I mean, I think we do this flexibly all the time, with other people. And with
non-autonomous technologies. The problem with the autonomous ones that people are tricked
into thinking they’re as good as other humans, when they’re actually a lot more like the simpler
non-autonomous ones.

Yeah, and I think that brings us back to the communication, and it also brings back to the
influence of popular culture, storytelling, and narrative, often times metaphor for the use of
particular troths that capture our imagination rather than bringing us down to the actual discrete
responsibilities of these systems might they be able to take on.

Though I’ll just mention, part of the challenge also of course is that those metaphors are a
wonderful way to build trust when used appropriately. So, I’m not opposed to story-telling
metaphors, I think that those are crucial, but you better anthropomorphize in the right way for
example, rather than in the way that a lot of people do.
Yeah, and this might be opening up a chasm of conversations, but I’m wondering if you can talk
a little bit about one or two, maybe just one scenario that could illustrate how you think
responsibility might be ascribed if an autonomous system does cause harm?

So, I think that when we think about autonomous systems causing harm we need to think about
why we ascribe responsibility. So, we don’t ascribe responsibility to use an example say from
warfare, we don’t ascribe responsibility to the bullet. We ascribe responsibility to the human who
decided that a bullet was needed here. So, with autonomous technologies I think we’re going to
be able to need to draw distinction between ones that are transparently carrying out the
intentions of the person who directed them in a way that is very predictable, could’ve been
foreseen, versus those that do things in surprising or unusual ways. Because I think that if it
carries it out in a way that the user who directed it could’ve foreseen it, it seems to me there it
would be pretty straightforward, that the responsibility still inheres in the person who made the
original decision to deploy. Where it becomes interesting of course, is exactly when they have
autonomous capabilities and can do things that might surprise you, or that you didn’t fully
specify. And there I think it comes back to questions of trust, I think that where we allocate
responsibility depends on whether the person who deployed it had appropriate and justified trust
in the system to behave in the “right” way or in the way they “intended”. If they did, and then the
system did something completely bizarre, then that strikes me as starting to approach my dog
has always been well-behaved and been fine and all the sudden does something weird. Where
we can look and say yes, you have a little bit of responsibility, but you took appropriate
precautions. It becomes the kind of ordinary accident kind of thing where maybe we want to
change the system, maybe we want to improve the technology, give it a better interface so it
becomes even more predictable, or more explainable, but the responsibility I think really sort of
inheres not in any deep moral way, but in the sense of what we should be trying to fix to do
better next time. Inheres much more on the machine. If the person deployed it without
appropriate justified trust, then I say, well that’s equivalent to somebody brings a brand-new dog
I’ve never interacted with and goes “Oh just run around the neighborhood”. And I have no idea.
There it is important push back on me, but it’s because we’re trying to have these failures not
happen again in the future. For me responsibility has a very deep forward-looking component.
We assign responsibility in part because we want to do better next time, we want to shape the
way that things happen in the future. It isn’t just a backwards looking thing, which I realize is
somewhat odd for some people, or different from some people’s conception. And in that regard,
I think we really have to think about where the sort of surface is for improvement, where the
ways that we can actually do better next time and use that to help guide some of the ways that
we think about responsibility.

Yeah, and I suppose that becomes incredibly complicated for the systems that are actually
designed to cause particular harm within a particular mode of ...

Autonomous weapons are a very deep, tricky issue. It’s one I’ve thought a lot about and done
some work on, and I think they’re very hard. It’s very hard even just to get our understanding of
the problem into the right space because there’s so many complexities that go on there.
Warfare is a very, understatement of the century, but warfare is a difficult problem. Right, and
then you add autonomy to it, and you start adding morality and policy and political constraints
and technological constraints and the immediacy of decision making in warfare, where you often
don’t have the time to sit and reflect. I think it’s a hugely important, but also hugely difficult
problem.

And the uneven access to the technology to make it more.

That’s a whole other set of issues. Yeah, I agree.

So, lastly, I’m wondering if you can share any of your thought on the development of general
artificial intelligence?

So, I think that the people who aspire to build artificial general intelligence, I think that they
should look to the example of psychology. So, in psychology, starts around 1860 as a modern
science and for 150 years, most of cognitive psychology is pursued the strategy of narrow
down, figure out this part of the mind, that part of the mind, this other part of the mind, and then
at some point we’re going to start to stitch it all back together. So, we’ll figure out perception,
we’ll figure out decision making, and then we’ll link them up later. We’ve got 150 years of trying
to link things back together, trying to put the building blocks back together, with basically not a
whole lot of progress. I think the exact same thing is what we’re looking at with regards to
artificial general intelligence. We have the AI winter in the 1980s where we just aren’t making
progress on AGI, so what do people do in the 90s you get the shift to say “Alright, I’m going to
find some narrow-focused problem: chess, go, supply chain management, give it to the
machine, let the machine solve it. Image recognition. Give it to the machine, let the machine
solve it. And we’ve been spectacularly successful now for 25 years at local problem, local
context, and the thinking has always been, and you see this the way people talk about self-
driven cars, “Oh we just have to put the pieces back together”. Speaking as somebody with a
background in cognitive science and knowing the history of psychology, it’s a whole lot harder
than you think. It’s not just putting the pieces back together. And in fact, I would argue we don’t
even quite know what it is we need to be doing beyond looking at these individual components.
It's not a simple matter of stitching, we don’t know how to stitch them back together, and I think
we’re going to find exactly the same problem with AGI. I think my prediction is actually that it’s
quite a ways off, at least using the current kinds of strategies, which is break things down into
their component pieces and then once you want AGI, you just put it all back together. I don’t
think that’s going to be a particularly successful strategy in the long run, or if it is it’s going to be
a very long run. Just as we’ve been finding in psychology and cognitive science.