HOW TO BUILD AI

THAT PEOPLE TRUST

Table of contents
03... Introduction

04... Why is trust a unique issue for AI?

05... Why trust matters: examples of good and bad AI

07... A relative concept: how much trust do I need?

08... The points in the AI life cycle where trust can be undermined

10... A framework for building and deploying trusted AI

2
Introduction
Many assume that AI success hinges on getting
the model to accurately interpret data. This is
important, but accuracy is one element of a
more nuanced challenge: trust.
Take a hypothetical AI that looks at medical
scans and identifies whether they contain
signs of an early-stage tumor. The model is
tested using medical data and shown to be
80% accurate in spotting early-stage tumors,
10% higher than a human specialist (figures
for illustration only).
But the AI has learned based on making
complex correlations between the images in
its training data. And while the data shows it is
accurate, it is not entirely clear how it reached
that decision.
Do we trust it? Is 80% accuracy enough? Should
we believe the claim of 80% accuracy? Are we
sure the scans were interpreted correctly? Are
we sure the test data was right? If a specialist
disagrees, should she go with the AI or her
own diagnosis, neither of which are perfect?
What should she do about that instinctive
feeling that something is wrong that she can’t
quite put her finger on?
These are complex questions. For many, the
straightforward answer is – if an AI is shown to
perform better than a human, we should trust
the AI. But such relative concepts as “better
than” are hard to be sure of in the complex
world of AI. This raises very difficult questions
when it comes to trust. Lots of companies
have implemented AI but have issues trusting
the results.
AI is a technical concept, but trust is a
human one
Trust is certainly undermined by low accuracy;
but high accuracy alone does not guarantee
trust. An AI can be 99% accurate, but so
complex and confusing – or new – that no one
trusts it. We need comprehensible evidence
it works, and explainability of how and why it
works, before we can trust it.
Even with all that, trust may take time to earn.
Would you get on the maiden flight of a plane
piloted only by AI? If you were told it had been
shown in trials to be safer than human pilots,
would that be enough?
It means managing expectations around what
AI can and can’t do
AI must be planned and designed for human
trust and understanding. That means
managing expectations around what AI
can and can’t do, and rolling out a complex
technology in a way that aligns with how
people learn.
This whitepaper will explain the importance
of trusted AI and discuss how to develop it.
3
Why is trust a
unique issue
for AI?
Most people are familiar with software in their
everyday life and workplace, which operates
based on rules. They may be complex rules, but
they are (bugs aside) predictable. They have
been explicitly programmed to follow a set of
instructions which turn inputs into outputs.
AI works differently. It ingests data and learns
how to interpret it by establishing connections
between different datasets. So, an English-
Spanish translation AI is not explicitly told
word-by-word that perro means dog or gato
means cat, etc., alongside fixed grammar rules.
It is fed texts that have been translated and is
told to learn what pattern links to the other –
with guidance from language and data experts.
This allows it to learn complex tasks such
as translation or image recognition quickly.
Many tasks performed with AI would not be
possible with traditional software, or would
take decades of programming.
However, this approach brings unpredictability
because the input data is complex and
imperfect, not a set of binary options. To learn
a language, an AI needs huge amounts of text
and there is not enough time to manually check
it all. Some translations may be poor, contain
mistakes, or deliberately misuse language. Even
correct ones contain nuance, where experts
disagree on the precise translation. A phrase
can be translated in several ways, depending on
the context. Anyone who has used a translation
app will know they are good, but not perfect.
Translation is usually low stakes, and we
can trust a language translation AI for many
applications, even if we can see it makes some
mistakes. But for AIs which diagnose disease,
spot when a plane engine component needs
replacing, or predict drug formulations, we
need to be very confident that it has reached
the right answer before we can trust it.
Added to this complexity is that AI conclusions
may be confusing, but still correct. NASA used
AI to design an antenna against a defined set of
criteria. The result would never have occurred
to a human, but it was better aligned to their
needs than anything a human came up with.
What does one do when an AI recommends
something completely counter-intuitive? It
could be a breakthrough (as in NASA’s case),
or it could be a spectacular oversight in the AI
design or training. How do we know?
All of this raises questions of trust. If we
know it is not 100% accurate, we need to
reach a decision about how much we trust
its recommendation. This comes down to
multiple factors including how accurate we
are told it is, how much we believe that claim,
how much control we had over the inputs, how
well we understand its decision-making, what
supplementary information it has provided to
back up its recommendations, its past record,
the consequences of it being wrong, and the
user’s own knowledge of the problem.
4
Why trust matters: examples
of good and bad AI
High-profile examples of AI failure have impact on organizations. We look at two cases
undermined trust. But this is not the full story; that got trust wrong, and two that got it right.
plenty of AIs are also having a hugely positive

The good

The project The goal What went well The outcome

AI drug Use AI to Algorithms were used to sift through A drug
design identify drug potential compounds for an OCD treatment, molecule
molecules for checking them against a huge database of ready to go
Link treating OCD parameters. There was a dedicated focus on into clinical
high-quality data acquisition and checking, trials was
and tailoring algorithms to the specific task developed in
at hand. This was achieved through close 12 months,
collaboration between AI and drug chemistry where the
experts, who checked inputs and outputs industry
throughout to ensure that results could be average is 4.5
trusted. years.
Google’s Develop The tutor app encourages, explains, and The app has
Bolo a speech corrects the child as they read aloud. It applied since been
reading- recognition existing speech recognition and text-to-speech rolled out
tutor app to help technology to a specific application, which widely.
children in was developed with a clear purpose in mind
Link rural India with and carefully tested in 200 Indian villages. The
reading skills pilot was verified by ongoing research in the
field, showing 64% of children with significant
improvements in reading proficiency.

5
The bad

The project The goal What went wrong The outcome

An AI to Identify the The same datasets were used in both training A model that
predict link between and validation. So, it was trained to learn looked like it
premature non-invasive about a correlation, then tested with the should go into
births electro- same data to see if that correlation existed. clinical practice
hysterography Given that the initial dataset was quite small was shown to
Link readings and anyway, this gave it a very high accuracy rate. be one that
premature When other researchers reproduced the definitely
births. An models, accuracy dropped to 50%. Trust in should not.
initial project the original accuracy claims was destroyed in
suggested one stroke.
up to 99%
accuracy.
Amazon’s AI Look at Models were trained by observing patterns The model
recruitment applicant CVs in resumes from over ten years. Most came was scrapped.
tool and predict from men due to the tech industry bias. Not Considerable
the best understanding this, the system taught itself negative
Link candidates that male candidates were preferable and coverage
based on started rejecting applicants for being female. ensued.
similarities As a result, it could not be trusted to make
with previous unbiased decisions.
successful
hires.

6
How much
trust do I need?
The level of confidence in an AI output needed, far from perfect, but high-risk AI decisions need
before the user will trust it, depends on the much greater levels of confidence in order to
seriousness of the consequences of failure. create trust.
Users will trust a useful low-risk AI even if it is

Examples of AI applications and levels of trust

Application Potential negative Level of confidence needed for

consequence trusted use

Disease diagnosis Preventable

death Higher
Drug design Missed
opportunities,
expensive mistakes
Oil well drilling Major financial loss

Predictive Unnecessary
maintenance downtime
Mortgage Harm to customer,
recommendation legal challenges
Targeted adverts Missed sales
opportunities
Translation Miscommunication

Film Occasionally
recommendation frustrated
customers
Photo tagging Wrong photo
tagged Lower
AI-created artwork Probably none

7
The points in the AI life
cycle where trust can be
undermined
There are various stages where trust can
be undermined in the AI development and
deployment process. In this section, we discuss
the main risk factors.
Bias in training data
Unconscious gender or racial bias has often hit
the headlines, usually created by applying AI to
process automation without understanding the
limits of the data. Amazon’s Rekognition, for
example, misidentified women and people of
color, likely due to the use of smaller training
datasets for these groups. Such stories
undermine the credibility of commercially
available technology.
The AI doesn’t learn incorrectly; it learns to
reflect bias in its training, which reflects bias in
the real world. Prejudice is the nasty face of this,
but bias in data can also extend to misplaced
assumptions by scientists, doctors recording
incorrect diagnoses, and even people’s choice
of written or recorded language.
Badly curated data
Data can also be mislabeled, or poorly curated,
such that the AI struggles to make sense of it.
If data is not appropriately selected, then the
model will not learn how to reach the right
conclusions. And if conclusions seem suspect,
people won’t trust it (or worse, they will trust
it and take bad decisions as a result).
User interface and explainability
Trust is not just about how good the model is,
but how easy it is to use and interact with and
how clear the answers are represented to the
user. If the user does not feel they can input
the information they want, they are likely to be
suspicious of the result. If the interface is overly
complex or the results presented in a confusing
way, or with no explanation as to how they were
reached (even if they are correct), it will quickly
be abandoned. Even something as simple as a
film recommendation is much more trustworthy
if you can see what aspects of viewing history
led to it.
Bias in the real world
Many AIs continue to learn post deployment,
but they are not necessarily well prepared for
the complexities of real-world data. Famously,
Microsoft’s Tay, an artificially intelligent
chatbot, was designed to learn from interactions
it had with real people on Twitter. Some users
decided to feed it offensive information,
which it had not been designed to deal with
appropriately. Within 24 hours, Tay had to
be deactivated and withdrawn for spreading
deeply upsetting opinions.
Malicious attacks
AI is susceptible to new malicious attacks in
ways that are poorly understood by users. AIs
that appear to take human decisions can be
fooled in ways that humans cannot.
In a test case, an AI was trained to recognize
images. By changing just one pixel in an image,
researchers fooled the AI, causing it to wrongly
label what it saw – sometimes very wide off the
mark (one thought a stealth bomber was a dog).
Tesla’s self-driving image recognition systems
have been tricked by placing stickers on roads
and signs, causing them to suddenly accelerate
or change lanes.
8
Lack of transparency
Sitting above all these issues is a fear fed by AI’s
lack of transparency. Not only do the end-users
not understand how AIs make their decisions,
in many cases nor do their makers.
Apple’s credit card, backed by Goldman Sachs,
was investigated by regulators after customers
complained that the card’s lending algorithms
discriminated against women. No one from
Apple or Goldman was able to justify the
output or describe how the algorithm worked.
The apparent correlation between gender and
credit doesn’t necessarily mean one is causing
the other, but it creates suspicion that bias has
crept in. Without transparency, it’s impossible
to know, and that makes it hard to trust.
9
A framework for building
and deploying trusted AI
Despite the risks discussed in this paper, AI
delivers huge value when done well. And aside
from the negative headlines, it is often done
very well.
The problems usually come when poor
process and lack of experience lead to poor
choices: such as wrong algorithms, bad data,
inadequate verification, poor interfaces, or
the lack of post-deployment support. These
errors are often baked in from the outset with
fundamental mistakes in initial scoping and
design, caused by a lack of understanding about
AI and the real-world problem it solves. These
all undermine trust.
As AI plays an ever-increasingly important role
in our lives, we need to design it to be trusted.
This goes beyond data scientists designing an
algorithm that learns about correlations and
works on test data. AI must be designed as a
whole product, with a set of support services
around it that allow the user to trust its
outputs. Doing so needs a rigorous approach
to AI development.
In this final section, we outline five key
parameters for creating trusted AI.
1. Assured
A data-driven decision is only as trusted as the
data that underpins it.
The most obvious aspect of trusted AI is
ensuring it does what it is supposed to. Because
AI learns from data, that data must be reliable.
You can train an AI to recognize cats and dogs
by feeding it lots of labeled images of each. But
if some cats are labeled as dogs, some are not
labeled, or some show a completely different
animal, the AI will learn incorrectly and make
incorrect decisions. If all images of dogs are
in the snow, the AI may learn to detect snow
rather than dogs.
As soon as it makes mistakes, users will stop
trusting it. This may not matter too much
when classifying cats and dogs, but it matters
a lot when classifying images of healthy vs.
precancerous cells.
Trusted AIs must use a well-designed model,
and be trained and tested on data that is
proven to be accurate, complete, from trusted
sources, and free from bias. Capturing that
data requires rigorous processes around data
collection and curation.
Those designing an assured AI model should
examine AI inputs and ask:
• Does this data accurately represent the
system to be modeled?
• Does the data contain confounding or
irrelevant information?
• How will I know that the data is of
sufficient quality?
• Is the underlying data biased – and how
would I tell?
• Are my assumptions about data
collection biased?
2. Explainable
A functioning model is not enough; users need
to understand how it works. The AI earns
trust by backing up its recommendations with
transparent explanations and further details.
If a bank turned you down for a mortgage, you’d
expect to know why. Was it past or existing
debt? Was it an error? Did they confuse you
with someone else? Knowing the reason allows
10
you to move forward in the most constructive
manner. For the bank, it allows them to spot
faults, retain customers, and improve processes.
It’s the same for AI. A recommendation is much
more useful if you understand how and why it
was made. Explainability allows the user to see
if the AI supports their own intuition (e.g., about
a disease diagnosis or the best way to make a
new material), or helps them question it. And
it allows developers to spot errors in the AI’s
learning and remedy them.
A responsibly designed AI will have tools to
analyze what data was used, its provenance,
and how the model weighted different inputs,
then report on that conclusion in clear language
appropriate to the user’s expertise.
Explainability may also involve some trade-off
between raw predictive power and transparency
of interpretation. If AI cannot fully explain its
outcome, trust may still be built in some cases
through rigorous validation to show it has a
high success rate, and by ensuring the user has
the information they need to understand that
validation.
Those designing AI to be explainable should ask:
• What could be known in principle about
the working of this AI?
• Does the model need to be fully
i n t e r p r e t a b l e o r a r e p o s t- h o c
explanations sufficient?
• Can the AI rationalize why it decided to
offer the user this piece of clarifying
information, rather than another?
• How consistent is the given answer with
previous examples?
• Does too much transparency make the AI
vulnerable to attack?
• Does the information on offer meet the
accountability needs of different human
users?
3. Human
A trusted AI is intuitive to use. Netflix would be
less successful if users had to enter a complex
set of parameters to get film recommendations.
But instead, it automatically presents films you
may like based on your history of search terms
in an easy to navigate interface, and sometimes
even tells you why it recommended them (the
Because You Watched feature).
An intuitive interface, consistently good
recommendations, and easy-to-understand
decisions, all help the user come to trust it
over time.
Intuitive doesn’t always mean simple. A simple
smartphone app may use very intuitive, guided
decision-making. A drug property prediction
platform can expect advanced chemistry
knowledge from its user and display complex
information in a manner appropriate for an
expert to understand and interact with.
The complexity of these guided decisions must
be matched to the user’s knowledge. Equally,
the time it takes the user to fully trust the AI will
be relative to the complexity and risk of failure.
Making AI usable for humans means
understanding the end-user and how they
interact and learn over time. Those designing
AIs should ask:
• What does each user need to understand
about why the AI did what it did?
• How should we communicate with users
and collect feedback?
• Why would users not trust this AI?
• What reassurances are they likely to need?
• What training and support is needed for
different users?
• Should the system allow users to ask for
more details?
11
• How can we retain confidence if the AI gets
it wrong?
• How do we make human users feel the AI
is accountable?
4. Legal and ethical
A trusted AI should reach decisions that are fair
and impartial, with privacy and ethical concerns
given equal weight to accuracy.
An AI may conclude that certain groups are
more likely to reoffend or default on loans.
While this may be true at a group level (for
broader socio-economic reasons), it does not
mean an individual from that group is more likely
to do so. An AI using this as a decision-making
factor creates undue prejudice and opens its
user up to legal challenges.
Those designing AI to be ethical and legally
compliant should ask:
• Why are we building this AI at all?
• Are we aligned with prevailing
ethical standards?
• Is it fair and impartial?
• Is it proportionate in its decisions?
• How is it governed?
• Are we honest about what we claim it
can do?
• Are we transparent about what it’s
doing, or is it doing something else in
the background?
5. Performant
Finally, a trusted AI continues to work
after deployment.
Too many AIs work well in a controlled
environment but fall over once deployed, either
because they are not ready for the complexities
of real-world data or because they have not
been designed to integrate into the users’
working life (either technically or practically).
Users will quickly lose trust in an AI they see
making less and less reliable decisions.
A truly performant AI is future-proofed for
throughput, accuracy, robustness, security,
and balancing raw predictive power with
transparent interpretation while remaining
aligned to genuine business needs.
Those designing AI to perform in the real world
should examine AI outputs, and ask:
• Does this AI actually solve the intended
business problem?
• Do we understand the required levels of
throughput and robustness?
• Do we understand the required output
quality i.e., accuracy and precision?
• What safety barriers are needed if the AI
makes a mistake?
• How robust is my testing, validation, and
verification policy?
• Is the in-service AI protected against
adversarial attacks?
• Do we have a plan to continuously assess
and improve in-service performance?
• Do we know when and how the in-service
model could become invalid?
12
Authors
Matt Jones
Lead analytics strategist
LinkedIn

David Hughes
Analytics solutions lead

Sam Genway
Analytics consultant
LinkedIn

John Godfree
Head of consulting
LinkedIn

13
About Capgemini Engineering
Capgemini Engineering combines, under one brand, a unique set of strengths from across
the Capgemini Group: the world leading engineering and R&D services of Altran – acquired
by Capgemini in 2020 – and Capgemini’s digital manufacturing expertise. With broad industry
knowledge and cutting-edge technologies in digital and software, Capgemini Engineering
supports the convergence of the physical and digital worlds. Combined with the capabilities of
the rest of the Group, it helps clients to accelerate their journey towards Intelligent Industry.
Capgemini Engineering has more than 52,000 engineer and scientist team members in over 30
countries across sectors including aeronautics, automotive, railways, communications, energy,
life sciences, semiconductors, software & internet, space & defence, and consumer products.

For more details, contact us:

www.capgemini-engineering.com

Write to us at:
engineering@capgemini.com

© Capgemini Engineering 2021. All rights reserved

14