SPECIAL SECTION ON ADVANCED SOFTWARE AND

DATA ENGINEERING FOR SECURE SOCIETIES

Received July 18, 2019, accepted August 4, 2019, date of publication August 9, 2019, date of current version August 23, 2019.
Digital Object Identifier 10.1109/ACCESS.2019.2934226

Visual Surveillance Within the EU General Data

Protection Regulation: A Technology Perspective
MAMOONA N. ASGHAR 1,2 , NADIA KANWAL 1,3 , BRIAN LEE1 , MARTIN FLEURY4 ,
MARCO HERBST5 , AND YUANSONG QIAO 1
1 Software Research Institute, Athlone Institute of Technology, Athlone, N37 HD68 Ireland
2 Department of Computer Science and IT, The Islamia University of Bahawalpur, Bahawalpur 63100, Pakistan
3 Department of Computer Science, Lahore College for Women University, Lahore 54000, Pakistan
4 School of Science, Technology and Engineering, University of Suffolk, Ipswich IP4 1QJ, U.K.
5 Evercam Ltd., Dublin 1, Ireland

Corresponding author: Mamoona N. Asghar (masghar@ait.ie)

This work was supported in part by the Marie Skłodowska-Curie (MSC) Career-FIT Postdoc Fellowship (Project ID: MF 2018-0179)
funded by the European Union’s Horizon2020 Research and Innovation Programme under the MSC under Grant 713654, in part by the
Science Foundation Ireland (SFI) under Grant SFI 16/RC/3918, and in part by the European Regional Development Fund.

ABSTRACT From an individual’s perspective, technological advancement has merits and demerits. Video
captured by surveillance cameras while a person goes about their daily life may improve their personal
safety but the images collected may also represent an invasion of their privacy. Because of the ease of digital
information sharing, there exists a need to protect that visual information from illegal utilization by untrusted
parties. The European parliament has ratified the General Data Protection Regulation (GDPR), which has
been effective since May 2018 with a view to ensuring the privacy of European Union (EU) citizens’
and visitors’ personal data. The regulation has introduced data safeguards through Pseudonymisation,
Encryption, and Data protection-by-design. However, the regulation does not assist with technical and
implementation procedures, such as video redaction, to establish those safeguards. This paper refers to the
GDPR term ‘‘personal data’’ as ‘‘visual personal data’’ and aims to discuss regulatory safeguards of visual
privacy, such as reversible protection, from the technological point-of-view. In the context of GDPR, the roles
of machine learning (i.e. within computer vision), image processing, cryptography, and blockchain are
explored as a way of deploying Data Protection-by-Design solutions for visual surveillance data. The paper
surveys the existing market-based data protection solutions and provides suggestions for the development
of GDPR-compliant Data Protection-by-Design video surveillance systems. The paper is also relevant for
those entities interacting with EU citizens from outside the EU and for those regions not currently covered
by such regulation that may soon implement similar provisions.

INDEX TERMS Blockchain, CCTV, cryptography, data protection-by-design, gdpr, pseudonymisation,

reversible protection, visual privacy, video redaction.

I. INTRODUCTION gas, and water services. Companies frequently collect this

As technology advances, the ways of collecting and pro-
cessing personal data are now almost completely digitized.
Companies are established to collect and process the identifi-
able personal data of their customers [1]. Banks, collect the
most confidential financial details of individuals. However,
there are many other entities that have access to personal
data, amongst which are: communication providers for broad-
band, landline, television, and mobile services; insurance
companies; retail shops; and utility providers for electricity,
The associate editor coordinating the review of this article and approving
it for publication was Feng Shao.
information, ostensibly to ensure a better customer expe-
rience. Customers may often trust them by providing this
information without a second thought and without bothering
to even ask how this identifiable data is processed by a com-
pany. Many people are aware that those companies may have
complete and identifiable personal data information (such as
their name, home address, credit/debit card numbers, bank
details, and other service-specific information). However,
beyond those companies, there are some service providers
who covertly monitor the day-to-day activities of people and
process their personal data. These are security companies,
which monitor people, often using surveillance cameras for

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/
VOLUME 7, 2019 111709
 M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
the purpose of safety. Visual surveillance is the principal
context of the current paper.
Security is an overarching concept that is also tied to
an individual’s personal data security. It is not simply the
security offered to companies, other organizations, and state
players as a service to protect their interests. For an indi-
vidual, during each working day, the thought of being safe
starts at the time of leaving home and ends on the return
home. Video surveillance cameras, such as those that are
part of a Closed Circuit Television (CCTV) system, are
utilized as safety tools by recording individuals’ activities.
People are very much exposed through video surveillance in
the name of safety or security. Examples of video surveil-
lance applications include: the monitoring of sensitive loca-
tions (such as embassies, airports, nuclear plants, military
zones, and at border controls), intrusion detection (residential
and retail monitoring); public safety installations (such as
for traffic control, and at car parks and ATM machines);
vehicle detection systems through license plate recognition;
event detection (such as during child care and the care of
the elderly); and marketing and statistics-gathering systems
(such as for discovering customers’ habits and behaviours,
and recording the number of visitors). Illicit activities such
as theft, assault, and shooting incidents are captured by
the widespread deployment of surveillance cameras, which
clearly benefits the majority of citizens. Nevertheless, with
the advent of enhanced CCTV based security, more chal-
lenges are presented in terms of people’s privacy protection,
and their dignity and free will, even when they are being
monitored.
The invasion of privacy, due to the extensive use of surveil-
lance cameras, has been widely and frequently reported. Par-
ticularly in the UK, some may say that there is an excessive
number of CCTV cameras. For example, any citizen, as a
daily average in London, is caught on about 300 CCTV
cameras [2]. In fact, there are about 4 million CCTV cameras
in the country as a whole. In respect to the abuse and misuse
of surveillance data, there are reports [3] of various disturbing
incidents. In 2005, according to the British Broadcasting
Corporation (BBC) news [4], a woman in Liverpool was spied
on by four council workers misusing a street CCTV pan-tilt-
zoom camera. The extremely intrusive use of CCTV surveil-
lance by Nahid Akbar (in 2017) [6] in her neighbourhood was
penalised by the Scottish court. The Scottish court found that
Ms. Akbar was in breach of the fifth Principle of Data Protec-
tion, resulting in the £17,000 damages awarded to the affected
Mr. and Mrs. Wooley. Outside the UK, an instance of illegal
spying on celebrities and government officials occurred [5]
when a museum’s CCTV camera was misused by a secu-
rity guard to spy on the Chancellor of Germany’s (Angela
Merkel’s) private flat. Such incidents can occur in numerous
sensitive situations: such as when people are identified in
surveillance video captured during a demonstration or rally,
or when, in oppressive regimes, political opponents active
against a sitting government can be threatened or harmed
later.
111710
The following quotation of Benjamin Franklin is well
suited to today’s secure society:
‘‘Any society that will give up a little liberty to gain a little
security will deserve neither and lose both.’’
In response, it may be said that the European Union (EU)
parliament has introduced a General Data Protection Regu-
lation (GDPR) [7] into European law in respect to the pro-
tection of individuals’ rights to freedom, while their personal
data, in the form of texts, audio, and videos, are processed.
GDPR seeks to ensure data protection and privacy during
the processing of personal data of all individuals within the
EU and the European Economic Area (EEA). GDPR also
covers the transfer of personal data outside the EU and
EEA area. In GDPR law, the data subject (GDPR term for
a natural person/individual) has complete control over the
processing of their personal data. Without the consent of
the data subject, the data cannot be processed and exported
outside the EU. GDPR provides extensive guidelines to data
collectors and processors in order to maintain the confi-
dentiality of the data subject’s information. The law covers
the rights of individuals and instructs data controllers and
supervisory authorities accordingly. If personal data collec-
tion companies do not provide GDPR-compliant solutions
for their data subjects (customers and employees), heavy
penalties could well result. A survey [8] shows that 80%
of businesses in Europe know minimal details or almost
nothing about the implementation of GDPR within their com-
panies. In fact, GDPR is a global data protection regime,
which according to the authors’ analysis can be summarized
as:
• Harmonization of privacy protection regulation in the
EU.
• Freedom rights for EU citizens/visitors by providing
safeguards for their identifiable data.
• Notification: Meaningful symbols must be placed by
controllers/processors to notify people whenever CCTV
recording is in progress.
• Access rights of individuals: Personal data should be
collected by initiating a consent form. The regulation
gives the right to individuals to gain access and request
the erasure of their personal visual data in a reasonable
time-frame.
• Privacy Safeguards: Data should not be stored and
exported openly; there should be privacy safeguards to
hide the originality and identification of data through
automated techniques.
• Retention Period: Personal visual data should be kept
for a specific time-frame.
This paper overviews the relevant articles within the first
four chapters of the GDPR regulation (see Section II.A)
in terms of re-defining the concepts of personal data,
pseudonymisation, encryption and Data Protection-by-
Design in accordance with the visual protection of data
subjects. Thus, the following research questions are posed
for a GDPR-compliant video surveillance industry:
VOLUME 7, 2019
M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
TABLE 1. GDPR descriptions for data and the authors’ definitions for visual data.
Q1: What is the visual personal data of the data sub-
ject? (Descriptions of data relevant to visual data are given
in Table 1 and Section III describes visual data).
Q2: Do all recorded visual data need to be protected?
(Section II (specifically II.A))
Q3: In what circumstances are GDPR-compliant surveil-
lance cameras required? (Section III.A)
Q4: What are the pseudonymisation vs. anonymisation
strategies for visual data protection? (Section IV.A)
Q5: What is Data Protection-by-Design in respect to video
surveillance companies? (Sections IV and V)
The purpose of this paper is to clarify the aforementioned
questions with the following contributory points:
• Brief overview of the GDPR relevant articles requir-
ing clarification for visual data collection compa-
nies/controllers/processors.
• The role of technology in the implementation of Data
Protection-by-Design solutions for CCTV data besides
the implementation of policies.
VOLUME 7, 2019
• A review of the existing solutions for GDPR-compliant
video-redaction methods for surveillance applications;
and
• Forthcoming real-time technologies for implementing
Data Protection-by-Design solutions.
The remainder of the paper is organized as follows.
Section II is an overview of the pertinent articles of the
GDPR regulation. That section provides a clear picture of
GDPR for the CCTV surveillance industry, along with GDPR
safeguards and distinguishable concepts of pseudonymi-
sation vs. anonymisation. Section III presents identifiable
visual personal data, identified premises and use-cases for
mandatory GDPR-compliant surveillance. Section IV dis-
cusses the role of technology in GDPR-compliant Data
Protection-by-Design solutions with market-based solu-
tions. Section V presents future Data Protection-by-Design
solutions by incorporating other technologies and finally,
Section V1 concludes the paper with several observations and
comments.
111711
 M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
II. GENERAL DATA PROTECTION REGULATION
The Data Protection Directive 95/46/EC (Directive 95/46/EC,
1995) was introduced in October 1995 by the EU and came
into force in December 1995. The purpose of this directive
was to protect the individual’s personal data. The directive
was implemented in October 1998 as an important part of
EU privacy and human rights law. That directive was super-
seded with the European General Data Protection Regulation
(GDPR), adopted in April 2016 and becoming enforceable
on May 25, 2018 in all EU member states. The aim of GDPR
is to harmonize the degree of data protection across the EU
countries. By introducing a single law across all EU states,
the intention was that it will bring better transparency to the
processing of an individual’s data and boost the collective
digital economy of the 28 participating EU states (possibly
27 states if the UK leaves the EU). Personal data-collecting
organizations in these EU countries, will probably be most
affected by the GDPR.
Data protection, as defined by the Joe Meade (Ireland’s
Data Protection Commissioner) [9] is as follows:
‘‘Personal data protection applies to all our interactions
with public and private sector organizations and thus applies
to applications, purchases and transactions in State services,
business and economic matters, in the social and medical
areas, in the workplace and in the globalized technological
arena.’’
The Data Protection Acts 1988 and 2003 confer rights
onto individuals and place responsibilities on the shoulders
of those who process personal data. GDPR applies globally;
thus companies outside the EU processing personal data of
an EU citizen with the aim of providing services, selling
goods or monitoring the behavior of any EU citizen, e.g.
any video surveillance companies [10] or health centers, are
bound to the regulation. A Data Protection Officer (DPO)
should be appointed by these companies. The appointed DPO
will be in charge of the GDPR compliance of that organi-
zation. The stakes have been set high, as failure to comply
will result in a substantial fine of 4% of the company’s
global revenue or of 20 million Euros, whichever is higher.
Due to the recent misuse of Facebook’s customer data [11],
all eyes are on the proper protection of customers’ private
information.
The resulting harmonization in law makes it easier for EU
citizens to understand how their data is being used and how
they can raise complaints, even if they are not located in
an EU member country. GDPR is not applied to IT-related
data alone; it has broad-sweeping implications for a whole
company, such as for the handling of sales, marketing and
human resourced data.
A. OVERVIEW OF FOCUSED GDPR ARTICLES
The EU GDPR aims to increase the privacy of the EU’s
citizens and allows regulatory authorities to use maximum
powers to take actions against any businesses/companies that
breach the law. The regulation (GDPR) has 173 recitals
(representing GDPR goals), and 11 chapters which cover
111712
99 articles [7]. However, for the implementation of GDPR-
compliant surveillance systems, the first four chapters cover
all necessary articles for secure processing of personal data by
the controllers/processors. For the ease of readers, a flowchart
of focussed GDPR articles in this paper is presented in Fig. 1.
Notice that as the main focus of this paper is technological
aspects of GDPR. Though the authors have applied due care
and consideration to legal aspects of GDPR, readers should
bear in mind the technological focus and perform due dili-
gence in respect to any legal aspects, which are added for
the reader’s convenience and are not intended to be legally
definitive.
1) GDPR DESCRIPTION and DEFINITIONS
Article 4 of GDPR [7] covers the definitions of terms used
throughout the regulation. Table 1 defines the term data. For
CCTV controllers, this current paper also discusses some
other important definitions to clarify the concepts. Overall,
GDPR considers seven types for the term ‘data’ through-
out the regulation and all are relevant for surveillance data
collection companies. However, notice that according to
GDPR (article 9) ‘‘Genetic Data’’ and ‘‘Biometric Data’’ are
included in the ‘sensitive’ data definition or equally ‘‘spe-
cial categories’ of personal data definition. Thus, in terms
of GDPR definitions, as opposed to descriptions, ‘‘Genetic
Data’’ and ‘‘Biometric Data’’ appear together in the same
definitions.
2) GDPR PRINCIPLES & LAWFULNESS OF PROCESSING
Article 5, clause-1 describes six (6) GDPR principles, that
is lettered a–f, for ensuring privacy during processing of
personal data.
[Art. 5-1(a)] Lawfulness, fairness and transparency
Lawfulness: Processing must be in accordance with the
GDPR criteria.
Fairness: Personal data should be processed in ways that
the subjects would reasonably expect and not use it in ways
that have unjustified adverse effects on them.
Transparency: Explicit reasons for the collection and pro-
cessing of personal data are required.
[Art. 5-1(b)] Purpose limitations
Personal data can only be obtained for ‘‘specified, explicit
and legitimate purposes’’. The data subject must be aware
of the reason for collecting data. There should not be any
processing without further consent, although, the data related
to public interest, research or statistical purposes have no
obligations in respect to purpose limitation.
[Art. 5-1(c)] Data minimisation
Data collected specific to a subject should be ‘‘adequate,
relevant and limited to what is necessary in relation to the
purposes for which they are processed’’, which means that the
minimum amount of data should be collected and retained for
specific processing.
[Art. 5-1(d)] Accuracy
The collected data must be ‘‘accurate and where necessary
kept up to date’’. There should not be any alterations by
VOLUME 7, 2019
M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
FIGURE 1. Flowchart of reviewed GDPR Articles.
way of ensuring good protection against identity theft. Data
controllers must build editable data management systems to
allow the subject to update their data.
[Art. 5-1(e)] Storage limitations
The regulator expects that personal data are ‘‘kept in a form
which permits identification of data subjects for no longer
than necessary’’. With proper safeguards, data related to
public interest, research or statistical purposes can be stored
for a long time. The data that are no longer required should
be erased/deleted from the repositories of a controller.
[Art. 5-1(f)] Integrity and confidentiality
During processing, the collected data must maintain its
integrity and confidentiality. The controllers should use
appropriate technical or organizational measures to provide
‘‘appropriate security of the personal data including protec-
tion against unlawful processing or accidental loss, destruc-
tion or damage’’.
[Art. 5-2] Accountability
Controllers shall be accountable if the data processing is
not compliant with clause 5.1(a-f) principles.
Article 6 concerns the lawfulness of processing by pro-
viding a proper ‘‘consent form’’ for the data subject for the
processing of their personal data. The processing should be
done while keeping the legal obligations in mind. Article 6
(4)(e) mentions the appropriate safeguards for personal data
by using the procedures of encryption or pseudonymisation
(Section II.C). Article 6(f) clarifies that legitimate processing
of data by the controllers is allowed, except that this process-
ing should not override the rights and freedom of the data
subject, especially if the data subject is a child.
Article 7 is about the conditions for getting consent about
the processing of personal data from the subject. Articles 8 to
10 discuss the processing of personal data related to children,
special categories and criminal convictions and offences,
VOLUME 7, 2019
which are out of the scope of this paper. Article 11 indicates
that if the controller is processing personal data, which do no
longer identify that particular data subject, then the controller
is no longer compliant with the GDPR.
3) RIGHTS OF THE DATA SUBJECT
Chapter III-GDPR is an important chapter, which demands a
proper focus from all companies dealing with the processing
of personal data. This is a detailed chapter with five sub-
sections. Section 1, Article 12 transcribes the transparency
and modalities of the rights of data subjects. Section 2,
including Articles 13-15, is about information and access
to personal data. Article 13 deals with the information that
should be provided to the data subject when their personal
data is being acquired or processed. Article 13(2) defines
the additional information provided by the controller to the
data subject regarding personal data retention time, the right
to request an erasure of personal data, withdraw consent,
the right to lodge a complaint with the supervisory authority,
the statutory or contractual requirements, and the existence
of automated decision making for profiling. Article 14 is
about personal data that has not been directly obtained from
the data subject. Article 15 has information about the right
of access by the data subject. The controller should provide
a copy of the personal data undergoing processing, which
shall not adversely affect the rights of freedom of others.
Section 3 ranging from Articles 16 to 20 are fundamentals
that should be known by EU citizens. Articles 16 to 18 are
about their rights of rectification, erasure, and restrictions on
the processing of their personal data. In short, the following
eight (8) rights are given to data subjects under the umbrella
of GDPR:
1. Right to access (Article 15): Data subjects have the
right to request access to their collected personal data
111713
 M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
and can inquire of a company about the use of their
data. The company is bound to provide a cost free copy
of personal data in an electronic format upon request.
2. Right to rectification (Article 16): This ensures that
individuals can have their data updated, in case of
changes and if the data are incomplete or incorrect.
3. Right to erasure (Article 17): Data collection com-
panies must delete the data of subjects if they are no
longer their customers, or if they withdraw their con-
sent from a company to use their personal data.
4. Right to restrict processing (Article 18): Individuals
can request that their data should not be used for pro-
cessing. Their record can remain in place, without any
use.
5. Right to be informed (Article 19): This specifies that
before collection of any data by companies, individuals
must be informed. Consumers have to opt in for their
data to be collected, and consent must be freely given
rather than implied.
6. Right to data portability (Article 20): Individuals have
a right to transfer their data from one service provider
to another.
7. Right to object and stop the processing of their data for
direct marketing (Article 21).
8. Right to be notified within 72 hours of a data breach
(Article 34).
4) GDPR PRIVACY PRINCIPLES
Article 25 describes the key privacy principle for all busi-
nesses, i.e. Data Protection-by-Design and –by-default. This
article serves as the backbone for GDPR-compliant busi-
nesses by implementing privacy protection for the sensitive
data of a subject.
• Data Protection-by-Design: This means that ‘‘appro-
priate organizational and technical measures to ensure
personal data security and privacy are embedded
into the complete lifecycle of an organization’s prod-
ucts, services, applications, and business and techni-
cal procedures’’. The proposed technical measures are
pseudonymisation and data minimization.
• Data Protection-by-Default: This means that: (i) only
necessary personal data are collected, stored, or pro-
cessed; and (ii) personal data are not accessible by
un-authorized people.
• Certification: Compliance with Data Protection-by-
Design and -by-Default requirements should be demon-
strated through an approved certification (as stated in
Article 42).
5) SECURITY OF PROCESSING
Article 32 is an important article in respect to providing
secure processing for personal data. This article emphasizes
that the controller and processors should implement appropri-
ate technical and organizational measures to ensure an appro-
priate level of security to preserve the rights and freedoms of
111714
a data subject. Article 32(1) describes the following security
measures:
(a) Pseudonymisation and encryption of personal data.
(b) Ensuring the procedures apply across the range of secu-
rity services i.e. ‘‘confidentiality, integrity, availability
and resilience’’.
(c) In the event of an emergency, quickly restoring the
availability and access to personal data.
(d) Ensuring the regular evaluation and validation testing
of the adopted security procedures for effective privacy
protection.
Article 32(2) points out the risks for processing, i.e. acci-
dental and unlawful destruction, loss, alteration, unauthorized
disclosure of processed and stored data. These risks should
also be assessed along with assessment of the appropriateness
of security levels.
6) DATA PROTECTION IMPACT ASSESSMENT
Article 35 describes the procedure of Data Protection Impact
Assessment (DPIS), whereby, if processing is done automat-
ically through technologically-advanced means, then a DPIS
should be performed. DPIS should be carried out by the
controller upon the advice of a designated Data Protection
Officer (DPO). DPIS must be performed on the automated
procedures, i.e. profiling in which legal impacts upon a data
subject are assessed, whether there exists a special category
of data or personal data relating to the criminal convictions
and offences. DPIS must contain a systematic description
of the envisaged processing operation in a legitimate way.
The assessment of the risks to the rights and freedoms of
data subjects should be given. The measures to address the
risks, i.e. safeguards, security measures and mechanisms to
ensure the protection of personal data in compliance with the
regulation should also be present in the DPIS. Article 36 is
about prior consultation of controllers with the supervisory
authority prior to the DPIS under Article 35, if processing
would result in a high risk in the absence of measures taken
by the controller to mitigate the risk. Article 37 ensures
the designation of a DPO with professional qualities and,
in particular, expert knowledge of data protection law and
practices and the ability to fulfil the tasks referred in Article
39. Article 37(2) facilitates that by stating that, for this group
of undertakings, controllers may appoint a vigilant and con-
stantly available DPO. A DPO can also be a staff member of
a controller’s team. Articles 38 and 39 describe the position
and duties of a DPO in detail.
B. GDPR SAFEGUARDS
From the previous section’s overview of GDPR, it can
be clearly perceived that EU citizens have control over
their personal data and without their will, companies can-
not collect and process their personal information. GDPR
relaxes the controllers/processors’ need to retain the per-
sonal data of the subject by providing safeguards in the
form of pseudonymised information and/or encryption.
VOLUME 7, 2019
M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
FIGURE 2. Safeguards designated by GDPR.
(Article 6(4) (e) and Article 32(1) (a)). Recitals 26, 78 clarify
the processes of pseudonymisation. Recital 83 explains that
encryption should not be weak and should ensure confiden-
tiality. The regulation just uses the terms but does not assist
with technologies, methods and technical information for
Data Protection-by-Design. This current paper now provides
an insight into GDPR-compliant technical solutions for confi-
dential video redaction of data subjects of that video. To facil-
itate reading, the targeted terms from GDPR are presented in
Fig. 2.
C. PSEUDONYMISATION VS. ANONYMISATION
GDPR came into force in May 2018, with the result that
surveillance-data collection companies now must understand
the main terms, as well as the procedures, needed to be
adopted according to the regulation. However, the following
two safeguard terms are still somewhat ambiguous and might
be used interchangeably. All the same, these two terms are
clearly different and, hence, need to be understood so as to
provide GDPR-compliant safeguards for the captured visual
personal data of a subject.
Anonymisation: is the safeguarding of data by adopting
an irreversible automated method. The term irreversible is
important to understand, as the anonymous data cannot be
reverted back to its original form or cannot be used for the
identification of the data subject by deploying any practical
methods.
Pseudonymisation: is the safeguarding of data by replacing
the actual personal information with a pseudonym through an
automated method, so that the data subject cannot be directly
accessible through a pseudonym.
The previous definitions indicate a distinction between
anonymized and pseudonymised data. The distinguishable
definition by the Data Protection Commission [12] of Ireland
is as follows:
‘‘Although pseudonymisation has many uses, it should
be distinguished from anonymisation, as it only provides a
limited protection for the identity of data subjects in many
cases as it still allows identification using indirect means.
Where a pseudonym is used, it is often possible to iden-
tify the data subject by analysing the underlying or related
data.’’
VOLUME 7, 2019
The above pseudonymisation definition shows that the data
can be reversible/identified in some cases. Here, the term
reversible shows that the data subject can be recognized by
using any other related information or automated method
even when the data is not converted back to its original form.
So, in general, one can distinguish both terms by saying that:
pseudonymisation can be a safeguarding technique that may
be reversible but anonymisation is an irreversible technique
to provide protection to data. Irreversible anonymised data
is out of the scope of the regulation [13] and controllers
can keep this form of data for an indefinite period of time
(for future statistical analysis) without any compliance issues
arising with the GDPR.
Now the question arises that if the controllers retain the
anonymised form of data, then what would be the purpose of
retaining that data? Controllers cannot revert the data back
to its actual form for any kind of processing, even if it was
legal. Hence, it may be considered merely wasteful of storage
space at the CCTV controllers end to keep this form of data
which is meant for future statistical analysis, should that
processing at some distant, future date be permissible, even
though currently it is not permissible.
This scenario might be like a scenario in which somebody
locks a data file in an unbreakable box and throws the lock’s
key into the river. Consequently, now the file (in a box)
becomes a life-long secret and cannot be taken out by any
means, even for a useful purpose. To avoid that situation,
as anonymized data evidently has potential beneficial uses,
such as the analysis of correlations between the health sta-
tus and the condition of patients, another solution might be
possible, as now described:
If the box’s opening key is hidden separately and can
feasibly be eventually (at some distant, future date in time)
used by controllers to open the box then this procedure is
called pseudonymisation and the file is still considered to
be personal data, and, hence, needs to be compliant with the
GDPR.
The GDPR emphasizes pseudonymisation OR encryp-
tion (Article 6(4)(e)) and pseudonymisation AND encryption
(Article 32(1)(a)). In Article 6, the use of the word OR in
between these two procedures and in Article 32, the use of
the word AND are important and, thus, are highlighted in
this current paper. Controllers may consider that these two
procedures are equally appropriate as a means of securing
video personal data. However, it is worth mentioning that
encryption is always reversible but that pseudonymisation can
be or cannot be. Although encryption and pseudonymisation
can be used simultaneously or separately, they are not alter-
natives to each other that can be assessed according to their
merits and demerits.
III. RECOGNITION OF VISUAL DATA
In videos, personal data are the person’s physical information
and their behaviour. Physical information includes looks,
clothing information, skin, eye, hair color, health and so
on, while behavior includes gait, physical actions, and facial
111715
 M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
FIGURE 3. Obscuring of faces for non-identification of person
(Gilad-Gutnick et al., 2018).
gestures. The recognition of humans through faces, clothes,
gaits, and actions within surveillance videos is not difficult
for a normal human. Interesting research on facial recog-
nition [14] was performed by MIT neuroscientists. This
revealed that humans are remarkably good at recognizing
faces even if they are highly distorted. That paper [14]
includes numerous experiments to significantly deform faces
by horizontal and vertical compression (also known as thin-
ning and flattening). Gilad-Gutnick et al. [14] found that
people are very good at recognizing thinned and flattened
celebrity faces. In fact, a thinning or flattening of as much as
80% has almost zero impact on recognition accuracy. Beyond
80%, performance starts to fall off but even at a distortion
level of 90% – in which the face is reduced to a mere
‘sliver’ – volunteers were still able to recognize about half
of the celebrities. After those experiments, Gilad-Gutnick
et al. wanted to know which parts of the face were most
important in terms of recognition. This led them to create
some – well – amusing stimuli. They performed selective
compression of the ‘internal features’ of Tom Cruise’s face,
while the ‘external features’ are left untouched (Fig. 3).
Gilad-Gutnick et al. interpret this finding as suggesting that
vertical ‘within-axis distance ratios’ are key to facial recogni-
tion. If one compresses the whole face, the relative distances
between the eyes, nose, and hairline do not change. However,
if one only compresses the internal features, these ratios are
altered.
There are some studies in which people are detected
and recognized through gaits. Bouchrika and Nixon [15]
detected and recognized people through their gaits by
using shape-based feature correspondence between consec-
utive frames. They derived a gait signature by means of a
model-based method, with the result that their system was
92% successful in recognizing people. In another example,
Semwal et al. [16] presented periodic cellular automata
rules for different gait state classification and prediction
through Extreme Machine Learning (EML). They formulated
sixteen rules for cellular automata and eight rules for each
111716
human leg, achieving 60% accuracy in prediction during their
experiments.
A. PREMISES FOR GDPR-COMPLIANT SURVEILLANCE
As the previous discussion indicates, facial and gait recog-
nition are easy tasks for a normal human. Therefore, pri-
vacy protection should be affected through methods in which
a sufficient level of distortion is applied across the whole
of a video. This section describes the boundaries/premises
whereby GDPR compliance is achieved when installing
surveillance cameras.
1) PERSONAL SPACE VISUAL DATA
Personal recorded data occur in a domestic setting or within
a household, as per Article 8 of the European Convention on
Human Rights [17] and, hence, is within the personal space
of visual data. For example, suppose that a CCTV system is
installed in one’s home, such as in a living room or in a bed-
room for monitoring an elderly person for safety purposes,
or in a playground for monitoring a child, again for safety
purposes. It could also be installed in a garage for vehicle
safety and in other similar circumstances. Video footages
obtained in those scenarios are not required to be protected,
as they are not forwarded to any third person, unless, that is,
a serious incident related to theft, any damage to property, or a
health issue related to a child or elderly person occurs and is
captured by a camera. Thus, these CCTV footages, in general,
do not have a value for local authorities, city planners or other
third parties. Such footages can be called ‘private data’ and
GDPR compliance is then not mandatory for such domestic
usage.
2) IMPERSONAL SPACE VISUAL DATA
Impersonal space data is not linked to an individual and,
being unsuitable, cannot be used for surveillance and con-
trol purposes. Examples of such data are the monitoring of
traffic flows within public transport, shopping malls, crowds
in roads/public parks, and construction sites and for the
purposes of sports and event management. Usually public
authorities collect this data by means of infrared video or
by CCTV. Such data may not be automatically considered as
sensitive, as it does not measure individuals but rather flows
of vehicles or impersonal crowds. Similarly, impersonal data
for surveillance purposes comes from the aggregation and
combination of survey and registration data in a city [18].
Such kinds of data may also be known as Public data. In this
setting, GDPR-compliant CCTV cameras are not necessary
but the protection of recorded videos from misuse is definitely
essential.
3) PROTECTED SPACE VISUAL DATA
Another important space lies in a category lying between Pri-
vate and Public spaces. This is the semi-personal-impersonal
space, and, herein, it is called ‘protected space’. Fig. 4 clar-
ifies the concept of protected space. People usually install
cameras for their safety in protected spaces, e.g. in gardens
VOLUME 7, 2019
M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
FIGURE 4. Concept of Premises for CCTV monitoring, after [19].
or out of doors (for an external viewpoint). The following are
some identified protected spaces:
i. Outside road view from a home/office.
ii. Outside footpaths/walkways external to a home/office.
iii. Monitored gardens and gates of neighbours.
iv. Monitored roads, neighbours, personal interactions
through Dash Cams.
People do have rights to monitor protected spaces through
CCTV systems. Protected space data also does have a value to
third parties such as the police or city planners or even persons
passing by an installed camera. A GDPR solution is certainly
required for these spaces.
4) USE-CASES
The following are some use-cases to help establish the form of
GDPR compliance required. These use-cases are based upon
those given by the Information Commissioners’ Office of the
U.K. [20].
UC1: Police Monitoring - If the suspected movements of
drug dealers are being monitored and recorded by the police
with covert surveillance equipment in order to identify if
they are committing any related offences then the level of
privacy impact can be higher (No mandatory GDPR compli-
ance is necessary). However, notice that overall the police are
expected to be GDPR-compliant. For example, the police are
not allowed to install covert cameras in subjects’ homes or in
private places like bathrooms.
UC2: Parking - If cars parked in a car park or on roads (in
front of homes) get frequently damaged and broken into at
night, or whether improved lighting affects the performance
of CCTV for spotting the criminals or not. (Protected +
Public Space − mandatory GDPR compliance is necessary.)
UC3: Neighbourhood - If the CCTV monitors the inside
of a hospital, or monitors a High street shopping area, then
it is subject to different privacy expectations. (Private +
Protected Space − mandatory GDPR compliance is needed.)
UC4: Physical Scanning - Footages obtained from cam-
eras covering the entrance to a drug rehabilitation centre will
VOLUME 7, 2019
require captured images of people moving in and out to be
obscured, as these images are considered sensitive personal
data. Otherwise, this type of monitoring will lead to an unfair
intrusion into the privacy of the individuals. Conversely,
footage of an individual’s movements in a bookshop is far
less likely to require obscuring, due to the non-sensitive
nature of the location. (Protected Space − mandatory GDPR
compliance is required.)
IV. DATA PROTECTION-BY-DESIGN THROUGH
TECHNOLOGY
Article 25 focusses on Data Protection-by-Design and -
by-default (refer also to Section II.A). Data Protection-by-
Default ensures data minimisation in the recording. That is
to say only necessary information related to a data subject is
collected and retained in the recording. If Data Protection-by-
Design is implemented with proper technical measures, it will
automatically ensure Data Protection-by-Default.
Data Protection-by-Design is an essential technical con-
cept that should be considered during an implementation of
a surveillance system. It means ‘‘safeguards provided to the
personal data through technological design’’. Currently, for
the management of Big Data, manual systems are replaced
by automatic systems. The processing and handling of data
procedures are best adhered [21] to when they are inte-
grated in the technology. Nevertheless, there is still uncer-
tainty about how Data Protection-by-Design is implemented
in GDPR-complaint solutions. The regulations leave pro-
tective measures for visual privacy completely open. In the
literature, numerous automated video redaction methods are
proposed and implemented for effective visual protection.
This current paper has classified and discussed them as
reversible (Pseudonymisation) and irreversible (Anonymisa-
tion) visual protection techniques (Section II). The paper now
highlights video redaction methods by incorporating the roles
of computer vision, image processing and cryptography in the
succeeding sections.
A. VIDEO REDACTION FOR VISUAL DATA PROTECTION
Video redaction is a method of obfuscating the personally
identifiable and sensitive information of a data subject within
any video. It is applied through different automated meth-
ods to ensure privacy is preserved at the time of capturing
or storing videos. In redacted videos, whole object (indi-
vidual), faces, background, and complete video frames can
be deformed automatically so as to make the data subject
unidentifiable and unrecognizable. In redaction-based soft-
ware [22], two components are important: (1) Object Detec-
tion/tracking; and (2) Object obfuscation methods. Video
redaction can be applied manually by detecting objects or
Regions-of-Interest (ROI) or automatically by tracking an
object.
1) ROLE OF COMPUTER VISION
Computer Vision (CV) is a sub-domain of Artificial Intelli-
gence (AI). Any expert/intelligent system which processes
visual information to recognize specific objects or ROIs
111717
 M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
within images [23] usually works through state-of-art com-
puter vision algorithms. Such computer vision algorithms are
increasingly robust when performing object detection and
tracking.
Object Detection is the procedure of finding real-world
instances, i.e. objects such as faces, bodies, bicycles, build-
ings, and licence plates, in images or videos. Object-
detection algorithms classically employ extracted fea-
tures and learning algorithms to recognize instances of
an object category. They then take advantage of object clas-
sification and localization techniques to achieve accuracy
in their results. Thus, to make a secure Data protection-by-
design solution for any CCTV system, CV works by ROI
selection (manually) or by detection/tracking (automatically)
through pattern recognition and learning techniques (includ-
ing Deep learning). Afterwards visual privacy protection
methods will be applied through video redaction.
Currently, there are frequently utilized, pre-trained mod-
els for object detection including: YOLO (You Only Look
Once) [24], Regional CNN (RCNN), Fast RCNN, Mask
RCNN [25], and Multibox [26].
Object Recognition is based on: template matching; color
matching; shape-based matching; or facial landmark detec-
tion (by locating the facial key points (15 to 68 unique
points)). In the case of faces, these key points mark important
areas of the face, such as the eyes, corners of the mouth, and
the nose.
2) ROLE OF IMAGE/VIDEO PROCESSING
Image/video processing is extensively used to provide visual
protection to CCTV surveillance videos. Image processing
is the manipulation of a digitized image to improve (or
deteriorate) the visual quality of given image. It comprises
of different types of manipulations. For example: (1) image
segmentation is employed to identify the pixel color-based
information from images; (2) geometric transformations
(enlargement, reduction, and rotation) may take place;
(3) there may be a combination or blending of images;
(4) image editing may take place; and (5) there could be
interpolation [27].
For quick safeguard methods, GDPR emphasizes the word
‘‘Pseudonymisation’’ fifteen (15) times in the regulation,
separate to the term Encryption. The term ‘anonymous data’
occurs once in Recital 26 [28]; otherwise, the term anonymi-
sation is not discussed within the whole regulation. For
the convenience of applying pseudonymisation as a GDPR-
compliant solution, companies provide masking to detected
objects in the visual data. Thus, most CCTV controllers
consider masking as the pseudonymised solution, which is
wrong in the light of the definition given in Section II.C.
It is worth mentioning here that irreversible schemes
can only technically revert back to an original or can be
identified/recognized again, if the originals are saved in
storage for re-access. Most AI-based tools accessed over
the Internet save originals for recovery of the manipulated
image. Widely-used video redaction methods are classified
111718
TABLE 2. Irreversible vs. reversible video redaction techniques.
in Table 2 and summarized after the Table. Fig. 5 shows the
visual effects (produced by in-house experiments) of these
methods applied to an image.
(a) Blurring: This redaction method [29] is applied
through image filters. The blurring filters can be
applied to a complete video frame or some specific
region/object, such as a face, person, licence plates,
or signage. Common blurring filters are: (1) Mean fil-
ter; (2) Weighted average filter; and (3) Gaussian filter.
For effective blurring, the Gaussian filter is utilized in
most privacy-protection applications.
(b) Pixelation: The process of enlarging pixels within
images to give them a blurred effect is known as pixe-
lation [30]. It is also performed through pixel interpo-
lation for high distortion effects. Interpolation works in
two directions by using known data points to estimate
the values at unknown points. Thus, image interpola-
tion occurs in the distortion of an image from one-pixel
grid to another grid.
(c) Mosaic: Small blocks of pixels from different regions
are combined to give an un-identifiable effect to the
picture [31], e.g. a face is obscured through pixel blocks
taken from different areas of an image
(d) Cartooning: This is an image distortion technique
which uses blurring filters and pixelation together 32]
to give robust privacy to videos.
(e) Masking: The process of hiding visual information by
replacing the original data with some unknown data.
(f) Warping: The process of distorting shapes in images
by digitally manipulating them to hide the original
shape. The warp is applied [33] by first transforming
the location of pixels with a vector and then linearly
interpolating the image.
(g) Morphing: This is a fluid transformation from one
image to another. Images are morphed [33] by cross
dissolving cuts in images.
Morphing is equivalent to warping (shape) plus cross-
dissolving (colors)
Morphing, warping, face swapping are done through
Facial Landmark detection.
VOLUME 7, 2019
M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
FIGURE 5. Privacy-protected video redaction techniques in image processing.
(h) Visual Abstraction: The process of visual abstrac-
tion [34] is the replacement of objects/persons appear-
ing in images by a visual model to protect the privacy
of the individual, while enabling that person’s activ-
ity. Abstraction can be obtained in a variety of ways:
Change the face of a person to some cartoon face;
apply a 3D avatar (Fig. 5) to the person’s whole body;
use a silhouette or blind box; and so on. Complete
removal of the object within an image is also consid-
ered to be visual abstraction. This abstraction can be
reversible if applied through tokenization (discussed
below).
(i) Scrambling: Scrambling [35] refers to the permuta-
tion of data within an image. The formal definition
of permutation is the re-ordering of data through spe-
cific patterns or at random. If scrambling is done
through defined patterns or with a seed value, then
this is called a reversible technique and data can be
retrieved by using pattern information and a seed
value. On the other hand, if scrambling is by a ran-
dom re-ordering then it is known as an irreversible
technique.
(j) Tokenization: Tokenization [36] is basically a
non-mathematical data security method and a reversible
technique. The process of tokenization is the
swapping-out of sensitive information and the replace-
ment of it with random numbers. The original and
mapped numbers are separately stored in tables. This
VOLUME 7, 2019
also works well in those code systems that rely
upon codebooks and which act to transform plaintext
into code-text. It is also widely used for securing
credit/debit card numbers. In visual data tokenization,
swapping of image pixels randomly with other values,
is known as scrambling [35]. While, for complete
object replacement, this can be implemented as a visual
abstraction [34] as discussed above.
(k) False Colors: In this method [37], image privacy pro-
tection is achieved through mapping the original image
to some other colour palette. It is a reversible tech-
nique and the original colours are reversed back to the
original.
(l) Hashing: This is an irreversible technique [38] for
converting arbitrary sized data to fixed size data. It is
used for image indexing or finding the same images in
a database. However, it cannot obfuscate images. Many
GDPR blogs discuss this as a data security process but,
in fact, it cannot be used for the redaction of visual
data [39].
3) ROLE OF CRYPTOGRAPHY
Cryptography is the practice and study of data securing
techniques [40]. Cryptography is a reversible process using
encryption and decryption methods. In fact, encryption is the
only reversible solution (through decryption) designated by
the GDPR in its Articles. In the context of CCTV video,
encryption can act on all or part of the video material with
111719
 M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
the intent of protecting the privacy of individuals captured by
cameras and appearing within the video footage.
Thus, encryption is the process to make the original data
(plaintext) into an unintelligible form (ciphertext) by means
of encryption algorithms known as ciphers, with some spe-
cific random secret value(s) known as a key. For visual data
protection, encryption is applied with the cipher and symmet-
ric or asymmetric key/s over the full or part of a CCTV video.
The cipher with the same or different key serves to decrypt the
ciphertext in symmetric or asymmetric cryptography respec-
tively. The regions or some important information such as
color pixels or motion in videos can be identified by image
processing and computer vision methods and then encryption
acts as a safeguard for reversible obfuscation of visual data.
There are many forms of encryption [41] which can provide
some measure of protection. For example, there are:
1) Naïve Encryption: Full video encryption with a suit-
able block cipher. The encrypted video is not playable
through video recorders and cannot be watched without
decryption.
2) Selective/Sufficient Encryption: Specific parts/objects
in the videos are detected and then encrypted. In the
literature, other words i.e. ROI-based, soft, lightweight
and partial encryption are also applied to this type of
encryption. The videos are obfuscated in a way that
they are viewable but not recognizable. The obfuscated
videos may remain format compliant, and playable
in the distorted form. The obfuscation is applied by
strong ciphers, so that the adversary is not able to
reconstruct the recognizable version through inference
attacks [42].
3) Transparent/perceptual encryption: This is applied
over multiple versions of a video [43]; a lower quality
version is free to view but to view a full-quality ver-
sion, the encryption key is required. Consequently, this
method of encryption is applicable to the pay-per-view
industry.
In cryptography, the robustness of the applied encryption is
estimated through the robustness of the cipher and the length
of secret key/s to avoid perceptual attacks on videos and brute
force attacks on the keys. Numerous state-of-art algorithms
have been proposed and tested in the last three decades to
provide confidentiality to the data. The prevailing ciphers are
DES, AES, RC4, and RSA, though DES has been largely
replaced by AES because it can easily be broken by a brute-
force attack (trying with all possible keys) with current PCs
and RC4 has recently been broken by cryptographers [44].
For a secure key, a key space greater than 2100 is resilient to
brute force attacks up to 2020 (by virtue of the time needed to
guess every key possibility) [45]. Selective encryption on the
carefully chosen video syntax elements within a large volume
of videos data has proven to be strong, and, hence, cannot be
easily cryptanalyzed [46].
It is noted here that there is always a trade-off
between security and computational complexity when
applying cryptography to visual data. The block-algorithm
111720
generally-considered to be the strongest, the Advanced
Encryption Standard (AES) [47], has a complex set of sixteen
rounds to compute using a minimal 64-bit block size (longer
key lengths are available), which takes almost four times
as much time than when the video is encrypted with a
lightweight cipher (see Fig. 11, [48]). Selective encryption
is the most adoptable way to obfuscate visual data because it
takes much less encryption time to secure the video than the
naïve or full form of encryption.
B. MARKET-BASED DATA PROTECTION-BY-DESIGN
SOLUTIONS
This section surveys current Data protection-by-design solu-
tions, operating globally within the commercial market place.
Some provide reversible solution with encryption while the
majority solutions only use video redaction through irre-
versible techniques.
StratoKey, [49] is a data protection-by-design solution for
cloud and Software-as-a-Service (SaaS) for EU clients. This
solution incorporates authentication and access controls for
accessing personal data over a third-party cloud. StratoKey
divides data into layers. This layering adds to the exist-
ing security, while providing a stringent form of additional
security. StratoKey also provides Rule-based Security and
Policy enforcement tools such as real-time Data Loss Preven-
tion, geographical access fencing, device profiling and other
measures to ensure security and data privacy. Specifically,
StratoKey supports ‘best-in-class’ Federal Information Pro-
cessing Standard (FIPS) 140-2 validated AES encryption
using a 256-bit key and Format-Preserving Encryption
algorithms. In addition, StratoKey has in-built support for
standard key rotations. StratoKey also supports locking
encryption keys to individual applications and even indi-
vidual groups of users. These encryption key management
features are native features within the StratoKey product.
StratoKey also supports third- party key management ser-
vices via the Key Management Interoperability Protocol
(KMIP). In fact, it provides both hardware and software
security.
Smartcrypt, [50] provides an on-the-spot encryption solu-
tion with Transparent Data Encryption (TDE) to the sensitive
data of EU citizens. Smartcrypt encrypts sensitive data at
its creation time and saves it in the encrypted form. This
Smartcrypt encryption stays with the data even when it is
moved and replicated to other user devices, file servers,
or external systems. Smartcrypt employs the AES cipher in
Cipher-Block-Chaining (CBC) mode with a 256-bit key. For
signing digital certificates, RSA 2048 Probabilistic Signature
Scheme (PSS) with preliminary SHA 512 hashing of data is
used. They (AES and RSA) have their own key storage and
retrieval mechanisms and are implemented on existing Intel
and IBM hardware accelerators.
Sighthound, [51] offers GDPR-compliant video redaction
software to EU clients for public surveillance. This software
is available as a plug-in for cloud services, or as a stand-alone
product for Windows- and Linux-based servers. The software
VOLUME 7, 2019
M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
is implemented through CV and Deep Learning models for
automatic detection of people, faces and licence plates in
real-time. It can be applied through manual selection to blur
specific objects such as street signs or animals. The software
is also able to detect people’s age and gender, as well as their
emotions. An irreversible blurring technique is applied to the
data subject.
An Intelligent Video Analytics services [52] for public
safety organizations provides advanced features of redaction
and facial recognition to help investigations by security agen-
cies. This is a complete security model to monitor suspi-
cious activities through cameras. In this software, an ROI is
extracted and blurring is applied to specific ROIs.
The GDPR-compliant securityRuntime (secRT) [53] is
implemented with the AES cryptographic algorithm along
with appropriate key management for the protection of tex-
tual data. This includes the process of tokenization of sen-
sitive data, e.g. on the digits of a credit card number. secRT
applies encryption to the original digits before tokenization
and it then stores them in a database only intended for
tokens.
Genetec software provides encryption and authentication
for videos by means of the Omnicast Internet Protocol (IP)
video management system [54]. Genetec offers cloud hosting,
storage and sharing of video, while Identity Cloak is a local
application of the software (see next).
Facit Data Systems Identity Cloak,
(https://ipvm.com/reports/cloak-identity) provides GDPR-
compliant solutions for CCTV videos. It provides two modes
of blurring, (a) standard, and (b) full body. In standard
blurring, faces along with the upper body and arms are
considered for blurring, while full-body blurring covers the
entire body of a data subject. Identity Cloak’s face detec-
tion performs more consistently than Genetec Clearance,
at higher angles of incidence, while also not losing faces as
persons move away from the camera or across the field of
view.
Pro Pixelated (https://www.propixelated.ie/) solutions use
irreversible pixelation for static CCTV images. They pro-
vide the services of full face, complete image, number plate
and other types of sensitive data blurring (pixelation) to EU
clients.
Kinesense [55], offers video redaction services for CCTV
videos. Irreversible masking and pixelation techniques are
provided through annotation. An ROI is masked and pixe-
lated through secure filters and areas outside the ROI can be
pixelated for the purpose of location hiding.
Redaction, uses the concept of blurring and pixela-
tion for ROI-based video redaction. The software pro-
vides GDPR-compliant solutions for body-worn cameras
and drone footage and also provides CCTV anonymization
(https://www.redaction.ie/).
Face404 software [56] performs the blurring of selected
ROIs, e.g. human faces in a video. The Face404 (based on AI
and CV) is a secure cloud-based solution, which meets the
requirements of GDPR compliance for stored videos.
VOLUME 7, 2019
C. LIMITATIONS OF EXISTING SOLUTIONS
The majority of market-based solutions discussed above pro-
vide irreversible solutions, such as pixelation and blur, mak-
ing that data no longer subject to GDPR. However, as dis-
cussed in Section II-C, this type of data storage is not useful
for future purposes, even for legal use by supervisory authori-
ties and stakeholders. Some surveyed market-based solutions
provide encryption-based reversible and robust solutions by
means of the AES cipher but mostly they are applied to
textual data, not videos. Thus, there is a need for reversible
and reliable visual Data protection-by-design solutions in the
future.
Surveillance data collection companies should understand
that the video redaction-based irreversible solutions are not
sufficient in themselves. There is a pressing role for computer
vision, image processing, and cryptography along with secure
key management solutions [57] to provide a complete privacy
protection model to visual surveillance data. Therefore, all
these information technology domains must be combined
together to provide a robust Data protection-by-design solu-
tion for the surveillance industry.
V. FORTHCOMING DATA PROTECTION-BY-DESIGN
SOLUTIONS
Data protection-by-design solutions must ensure the four
security services, i.e. confidentiality, integrity, availability
and resiliency (Article 32(1)). For effective GDPR-compliant
CCTV surveillance application, the following basic
requirements must be considered by future researchers and
developers:
1) Perceptual Security: The footage is effectively visually
distorted and proof against attacks to remove that distortion.
Here, the term ‘‘effectively’’ means that the distortion level
within the video is sufficient to make video unwatchable. The
video can be viewable but not understandable or pleasantly
watchable.
2) Reversibility: The implemented safeguard should be
reversible, so that the video can be used for legal purposes,
if required.
3) Intelligibility: The activity level in the footage should
not be obscured to prevent the identification of un-
ethical/unlawful behaviors. However, notice that this require-
ment may be very difficult to be implement because: (i) there
is no common definition of ‘‘unethical’’ behaviors; and (ii)
unlawful behaviors vary between different EU states.
4) Efficiency: Camera devices operated with reduced hard-
ware specifications (such as the Raspberry Pi and CMOS
sensors) need real-time and efficient privacy-protection solu-
tions.
5) Format compliance: Secured footage should be decod-
able within video players in an obscured/unintelligible
form.
Thus, Data protection-by-design encompasses a complete
technical solution, which may further include authentication,
access rights, digital evidence chain, video records storage,
and a method of generating successive secret keys from
111721
 M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
FIGURE 6. An example of a suggested multi-version encryption.
previous ones. The following are some suggestions for
implementing Data protection-by-design solutions for visual
data:
1) MULTI-VERSION ENCRYPTION
To provide privacy to visual data, multiple versions of the
video footage can be created with different forms of reversible
encryption. Naïve encryption is computationally expensive.
Therefore, ROI-based and selective/lightweight encryption
is desirable and it can be applied to the original video with
different levels of confidentiality. The lightweight encryption
stages can be utilized well for the purpose of anonymisation
at the times of video capturing, streaming, and storage. For
instance: (1) The foreground features of the footage can
be encrypted for individual/object secrecy; (2) The back-
ground of footage can be obfuscated for location hiding;
(3) Only motion in the footages can be encrypted to pro-
vided anonymity to activities in the footage; or (4) The pixel
information of the whole video can be scrambled for com-
plete confidentiality (see Fig. 6). These four usages can be
encrypted through four different keys and the keys can be gen-
erated through state-of-the-art key management protocols and
session keys may be managed by the blockchain technology
discussed in part 3 of this Section.
2) ACCESS-CONTROLS FOR AUTHENTICATION
The above four scenarios can provide access controls to spe-
cific users. Law enforcement authorities, defence companies,
and the police only accept original videos. They also have the
right to access the whole video for investigation, while other
stakeholders (third party/persons) do not need the original,
111722
whole videos. The specific key will be given to each user
and they can only view the data according to their access
rights.
Our proposed model will work almost in the same way as
given by the authors in [58]. In [58], the authors’ privacy con-
sole manages operator access to different versions of video
derived data according to access-control lists, but without
encryption.
3) BLOCKCHAIN
A Blockchain [59] is a system for ensuring the trust and
integrity of transactions made on different machines. It works
through hashing algorithms, which were first used for cryp-
tocurrencies [60]. Notice that in some circumstances, such as
when all participants are known to each other and several are
known or thought to be trustworthy, then a Blockchain may
be over-cumbersome. Classical signatures may then be more
than sufficient to protect data integrity. However, Blockchains
are a technologically innovative solution, which according to
circumstances, may be a good solution.
A Blockchain has three main properties, i.e. Decentraliza-
tion, Transparency, and Immutability, which are the factors
that have contributed to the success of this technology in
domains such as digital currencies. There are at least three
possible forms of blockchain:
1) Public blockchain: The data is shared with no access
restrictions on the participants and similarly any val-
idator can voluntarily validate the shared data. This
strategy is getting more attention due to inherent mon-
etary benefits for validators. Digital currencies such
as Bitcoin and Ethereum are well-known examples of
public blockchain technologies.
2) Private blockchain: Contrary to the public blockchain,
a private blockchain enforces a registration and an
approval mechanism, both for participants and val-
idators. This approach benefits the authentication and
authorization of shared data for the known participants,
maintaining a reliable chain of custody.
3) Hybrid blockchain: A combination of public and pri-
vate blockchain systems in which multiple companies
can exercise control. This hybrid approach by nature is
permissioned and semi-decentralized.
The following analysis of the Digital Evidence Chain,
includes two possible blockchain-based solutions for privacy
protection of visual data, using either a private blockchain or
a public blockchain. As is noted, the likely access pattern may
well determine which (if any) of these options is chosen.
Digital Evidence Chain: Blockchain technology can be
used for secure video evidence management in CCTV
systems [61], as it is an effective solution for securing
crime-scene evidence. In this solution, for each video footage
transfer (transaction) from the controller to the supervisory
authority, courts or other stakeholder, the cryptographic hash
through SHA (256) [62] can be created at the start or after
any modification by a third party within the chain. If any
modification is made to that video by the data controller or
VOLUME 7, 2019
M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
FIGURE 7. Suggested secure video evidence management chain.
the police, a new hash across the video footage, along with a
time-stamped value, will be created. The hash must be stored
in a separate table and also embedded within the transferred
footage.
An example of a blockchain with three known participants
i.e. controller, police and court is shown in Fig. 7, in which
the generated hashes will make an integrated chain on each
transfer, while maintaining trust for final verification by the
law enforcement authorities.
As per CCTV company requirements the controllers can
use private, public and hybrid blockchain networks for their
digital history management purpose [63]. If the identities
of all participants/stakeholders are known in advance (as
in Fig. 7) then a private blockchain network (such as Hyper-
ledger Fabric and R3 Corda) should be selected rather than a
non-private blockchain [64].
However, it is sometimes the case that all stakeholders are
not known in advance. For example, a passer-by, who has
been recorded by the CCTV system in protected space, may
also be interested in the footage for his/her own safety after
some serious incident. Notice that the passer-by may also be
an EU visitor.
Due to the use of a Decentralized Leger System (DLS),
this technology is presently tamper-proof. While implement-
ing a digital evidence chain through DLS, CCTV compa-
nies should consider the data subject’s Right to Erasure
(GDPR-Article 17). Therefore, to satisfy Article 17, existing
immutable blockchain technology is not sufficient. There
should be some allowance in the history management chain
such as stipulating that only calculated hashes over trans-
ferred data become part of the chain rather than the com-
plete data from the evidence chain. Thus, if the original
data is not transferred with the chain then there will be no
issues for controllers or processors in terms of Article 17.
GDPR Article 17 compliance within a blockchain is an open
challenge for all researchers, which merits a future in-depth
analysis.
VOLUME 7, 2019
In any case, when implementing any such DLS, Brewer’s
well-known Consistency, Availability, and Partition toler-
ance (CAP) theorem should be borne in mind because such
a blockchain-based DLS is distributed. The CAP theorem
states that only two of the three desirable CAPs can be
maintained. For example, if consistency within the evidence
chain is a priority, i.e. individual transaction consistency is
a priority rather than eventual consistency, then it may not be
possible to maintain high availability in terms of transactions
per second when updating the digital evidence chain. Equally,
it may not be possible to maintain network partition tolerance,
i.e. allow the distributed system to recover from a network
outage.
Keys Generation Evidence Chain: Likewise, blockchain
technology can be utilized for the creation and management
of cryptographic keys for data controllers, the police and
the supervisory authorities. The private secret keys for the
algorithm to decrypt the encrypted video for an authenticated
receiver are generated with a secure hash value. The hash
can be embedded within the key to track the originality
of key generation and be transferred to other stakeholders.
On every key transfer, the secure hash can be re-generated
to verify the originality of a key. Otherwise, a compromised
key will be discarded. Blockchain ledgers can operate along
with standard key management algorithms [57] to strengthen
the key-generation history management and the original key
verification process.
VI. CONCLUSIONS
Video surveillance is a pervasive phenomenon throughout the
world. Along with the self-evident benefits of surveillance
applications, protecting an individual’s privacy is a critical
task that controllers must perform. An individual, who is
being monitored through CCTV systems, has a fundamental
right to protect their identity. To ensure the freedom rights of
individuals, GDPR has come into force within the EU. GDPR
is a regulation, rather than a directive, because it is directly
and equally applicable in all EU member states.
The purpose of this current paper was to shed light on
GDPR-compliant strategies aimed at visual privacy pro-
tection in the presence of CCTV surveillance. This paper
covers GDPR Article 6 in relation to both methods for
pseudonymisation and encryption; Article 25 for Data
protection-by-design and also by default; Article 32 for
secure data processing; and Article 35 for DPIS as a founda-
tion for proposing visual protection solutions through tech-
nology. The paper gives insights into: The visual personal
data of an individual; the premises or circumstances for
GDPR-compliant surveillance; and the existing vs. future
technological solutions for assisting GDPR policies.
GDPR places a strong emphasis on encryption due to its
reversible nature and its robustness in resisting intrusion.
However, cryptographic solutions are expensive in terms of
implementation, software, hardware, and manpower. Com-
panies need to hire appropriately-trained IT staff to handle
and maintain the complex ciphers within their organizations,
111723
 M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
particularly when they export data to a third party. The third
party may need to hire trained cryptographers to manage and
edit that information.
Data protection-by-design is still at an immature stage and
requires reliable secure technologies for the privacy provision
of digitized visual data in the context of the regulation. Article
25 serves as a backbone for secure technology-based solu-
tions and Article 32 clarifies the security services provided
by those solutions. For instance, Confidentiality requires
that personal data whether in storage or transit can only be
accessed by authenticated persons/systems to avoid inter-
ception attacks. Integrity requires that any modification to
personal data whether in storage or transit can only be per-
formed by authenticated persons/systems with the consent of
the data subject. Availability requires that certain hardware
and software services can only be accessed by authorized
persons/systems, a failing in which can be due to an interrup-
tion attack by which a system is spoiled or made unusable.
Resilience ensures the robustness of the technology, which
should restore itself to its original form after attacks.
To conclude, as visual privacy is of great importance, thus,
there is a vital need to deploy complete Data protection-by-
design solutions for affected companies. The paper has dis-
cussed technological solutions provided by companies and it
further suggests improvements. In the opinion of this paper’s
authors, cryptography with video redaction, acting together
or in isolation are not sufficient. Multiple information tech-
nology domains will need to work hand-in-hand to imple-
ment effective Data protection-by-design solutions for video
surveillance systems. Clearly, an additional cost will ensue
in order to mitigate the privacy risks arising from the use of
visual data. However, this cost will result in a meaningful
implementation of GDPR-compliance by video surveillance
companies, which will have a positive impact upon EU busi-
nesses. Importantly, similar solutions will become necessary
outside the EU, if not already implemented and they are
already necessary for those companies interacting with EU
clients from outside the EU.
ACKNOWLEDGMENTS
We would like to thank the reviewers for their valuable com-
ments in improving the quality of the article and in enhancing
the readability and usefulness of our manuscript.
REFERENCES
[1] C. Tikkinen-Piri, A. Rohunen, and J. Markkula, ‘‘EU general data protec-
tion regulation: Changes and implications for personal data collecting com-
panies,’’ Comput. Law Secur. Rev., vol. 34, no. 1, pp. 134–153, Feb. 2018.
[2] G. Pillai. (2012). Caught on Camera: You are Filmed on CCTV
300 Times a Day in London. Accessed: May 2019. [Online]. Avail-
able: https://www.ibtimes.co.uk/britain-cctv-camera-surveillance-watch-
london-big-312382
[3] A. Cavallaro, ‘‘Privacy in video surveillance,’’ IEEE Signal Process. Mag.,
vol. 24, no. 2, pp. 166–168, Mar. 2007.
[4] Q. M. Rajpoot and C. D. Jensen, ‘‘Video surveillance: Privacy issues and
legal compliance,’’ in Promoting Social Change and Democracy Through
Information Technology, V. Kumar and J. Svenson, Eds. Hershey, PA,
USA: IGI Global, 2015, ch. 4, pp. 69–92.
[5] R. Furlong. (2006). BBC NEWS | Europe | Germans Probe Merkel
Spy Camera. Accessed: May 2019. [Online]. Available: http://news.bbc.
co.uk/2/hi/europe/4849806.stm
111724
[6] Scottish Court. (2017). £17,000 Damages Awarded by Scottish Court
in Neighbour CCTV Dispute | Brett Wilson LLP. Accessed: May 2019.
[Online]. Available: https://www.brettwilson.co.uk/blog/17000-damages-
awarded-scottish-courtwooley-wooley-v-nahid-akbar-akram-2017-sc-
edin-7-neighbour-cctv-dispute/
[7] EU GDPR. (2016). EUR-Lex–32016R0679–EN–EUR-Lex. Accessed:
May 2019. [Online]. Available: https://eur-lex.europa.eu/eli/reg/2016/
679/oj
[8] C. Babel. (2017). Privacy and the EU GDPR US and UK Privacy Pro-
fessionals. [Online]. Available: http://www.corporatecomplianceinsights.
com/wp-content/uploads/2017/11/TrustArc-Privacy-and-the-EU-GDPR-
Research-Report-09.26.17.pdf
[9] G. Mainard. (2018). Data Protection Issues. Accessed: Jun. 2019. [Online].
Available: https://www.localenterprise.ie/DublinCity/Start-or-Grow-your-
Business/Knowledge-Centre/eBusiness/Data-Protection-Issues/
[10] D. George, K. Reutimann, and A. Tamò-Larrieux, ‘‘GDPR bypass by
design? Transient processing of data under the GDPR,’’ SSRN Electron. J.,
Aug. 2018.
[11] G. Petro. (2018). Facebook’s Scandal and GDPR are Creating New
Opportunities for Retail. Accessed: Jul. 2019. [Online]. Available:
https://www.forbes.com/sites/gregpetro/2018/05/27/facebooks-scandal-
and-gdpr-are-creating-new-opportunities-for-retail#481601a6626c
[12] Data Protection Commission. (2018). Anonymisation and
pseudonymisation | Data Protection Commissioner. Accessed: Feb. 2019.
[Online]. Available: https://www.dataprotection.ie/en/guidance-landing/
anonymisation-and-pseudonymisation
[13] S. Stalla-Bourdillon and A. Knight, ‘‘Anonymous data v. personal data-
false debate: An EU perspective on anonymization, pseudonymization
and personal data,’’ Wisconsin Int. Law J., vol. 34, no. 2, pp. 284–322,
2017.
[14] S. Gilad-Gutnick, E. S. Harmatz, K. Tsourides, G. Yovel, and P. Sinha,
‘‘Recognizing facial slivers,’’ J. Cogn. Neurosci., vol. 30, no. 7,
pp. 951–962, Jul. 2018.
[15] I. Bouchrika and M. S. Nixon, ‘‘People detection and recognition using
gait for automated visual surveillance,’’ in Proc. IET Conf. Crime Secur.,
2006, pp. 576–581.
[16] V. B. Semwal, N. Gaud, and G. C. Nandi, ‘‘Human gait state prediction
using cellular automata and classification using ELM,’’ in Proc. Mach.
Intell. Signal Anal., 2019, pp. 135–145.
[17] Handbook No. 1: The Right to Respect for Private and Family Life.
A Guide to the Implementation of Article 8 of the European Convention
on Human Rights, European Court of Human Rights, Strasbourg, France,
Dec. 2001.
[18] L. van Zoonen, ‘‘Privacy concerns in smart cities,’’ Government Inf. Quart.,
vol. 33, no. 3, pp. 472–480, 2016.
[19] R. Buyya, S. T. Selvi, and X. Chu, Object Oriented Programming With
Java: Essentials and Applications. New York, NY, USA: McGraw-Hill,
2009.
[20] Information Commissioners’ Office (ICO). In the Picture: A Data
Protection Code of Practice for Surveillance Cameras and Personal
Information Version 1.2.. Accessed: Jul. 2019. [Online]. Available:
https://ico.org.uk/for-organisations/guide-to-data-protection-1998/
encryption/scenarios/cctv/
[21] A. Romanou, ‘‘The necessity of the implementation of Privacy by Design
in sectors where data protection concerns arise,’’ Comput. Law Secur. Rev.,
vol. 34, no. 1, pp. 99–110, 2018.
[22] S. Sah, A. Shringi, R. Ptucha, A. M. Burry, and R. P. Loce, ‘‘Video
redaction: A survey and comparison of enabling technologies,’’ J. Electron.
Imag., vol. 26, no. 5, 2017, Art. no. 051406.
[23] P. Sinha, B. Balas, Y. Ostrovsky, and R. Russell, ‘‘Face recognition by
humans: Nineteen results all computer vision researchers should know
about,’’ Proc. IEEE, vol. 94, no. 11, pp. 1948–1962, Nov. 2006.
[24] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, ‘‘You only look once:
Unified, real-time object detection,’’ in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit., Jun. 2016, pp. 779–788.
[25] K. He, G. Gkioxari, P. Dollár, and R. Girshick, ‘‘Object detection for
dummies part 3: R-CNN family,’’ Facebook AI Res., New York, NY, USA,
Tech. Rep., 2017.
[26] Z. Li and F. Zhou, ‘‘FSSD: Feature fusion single shot multibox detec-
tor,’’ Dec. 2017, arXiv:1712.00960. [Online]. Available: https://arxiv.org/
abs/1712.00960
[27] J. R. Padilla-López, A. A. Chaaraoui, and F. Flórez-Revuelta, ‘‘Visual
privacy protection methods: A survey,’’ Expert Syst. Appl., vol. 42, no. 9,
pp. 4177–4195, 2015.
VOLUME 7, 2019
M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
[28] M. Mourby, E. Mackey, M. Elliot, H. Gowans, S. E. Wallace, J. Bell,
H. Smith, S. Aidinlis, and J. Kaye, ‘‘Are ‘pseudonymised’ data always
personal data? Implications of the GDPR for administrative data research
in the UK,’’ Comput. Law Secur. Rev., vol. 34, no. 2, pp. 222–233,
2018.
[29] J. Flusser, S. Farokhi, C. Höschl, T. Suk, B. Zitová, and M. Pedone,
‘‘Recognition of images degraded by Gaussian blur,’’ IEEE Trans. Image
Process., vol. 25, no. 2, pp. 790–806, Feb. 2016.
[30] T. Gerstner, D. DeCarlo, M. Alexa, A. Finkelstein, Y. Gingold, and
A. Nealen, ‘‘Pixelated image abstraction with integrated user constraints,’’
Comput. Graph., vol. 37, no. 5, pp. 333–347, 2013.
[31] Y. Kusama, H. Kang, and K. Iwamura, ‘‘Mosaic-based privacy-protection
with reversible watermarking,’’ in Proc. 12th Int. Joint Conf. e-Bus.
Telecommun., Jul. 2015, pp. 98–103.
[32] A. Erdélyi, T. Barát, P. Valet, T. Winkler, and B. Rinner, ‘‘Adap-
tive cartooning for privacy protection in camera networks,’’ in Proc.
11th IEEE Int. Conf. Adv. Video Signal Based Surveill., Aug. 2014,
pp. 44–49.
[33] J. Gomes and L. Velho, ‘‘Warping and Morphing,’’ in Proc. Image Process.
Comput. Graph., 1997, pp. 271–296.
[34] K. Chinomi, N. Nitta, Y. Ito, and N. Babaguchi, ‘‘PriSurv: Privacy pro-
tected video surveillance system using adaptive visual abstraction,’’ in
Advances in Multimedia Modeling—MMM (Lecture Notes in Computer
Science), vol. 4903, S. Satoh, F. Nack, and M. Etoh, Eds. Berlin, Germany:
Springer, 2008, pp. 144–154.
[35] F. Dufaux, ‘‘Video scrambling for privacy protection in video surveil-
lance: Recent results and validation framework,’’ Proc. SPIE vol. 8063,
May 2011, Art. no. 806302.
[36] T. Spies and R. T. Minner, ‘‘System for protecting sensitive
data with distributed tokenization,’’ U.S. Patents 0 046 853 A1,
Feb. 13, 2014.
[37] S. Çiftçi, A. O. Akyüz, and T. Ebrahimi, ‘‘A reliable and reversible image
privacy protection based on false colors,’’ IEEE Trans. Multimedia, vol. 20,
no. 1, pp. 68–81, Jan. 2018.
[38] V. E. Liong, J. Lu, Y.-P. Tan, and J. Zhou, ‘‘Deep video hashing,’’ IEEE
Trans. Multimedia, vol. 19, no. 6, pp. 1209–1219, Jun. 2017.
[39] A. Ewerlöf, ‘‘GDPR pseudonymization techniques,’’ Medium Corp. (US),
San Francisco, CA, USA, Tech. Rep., May 2018.
[40] D. R. Stinson, Cryptography: Theory and Practice, 3rd ed. London, U.K.:
Chapman & Hall, 2005.
[41] H. Hofbauer and A. Uhl, ‘‘Identifying deficits of visual security metrics for
images,’’ Signal Process., Image Commun., vol. 46, pp. 60–75, Aug. 2016.
[42] M. N. Asghar, M. Ghanbari, M. Fleury, and M. J. Reed, ‘‘Sufficient encryp-
tion based on entropy coding syntax elements of H.264/SVC,’’ Multimedia
Tools Appl., vol. 74, no. 23, pp. 10215–10241, 2015.
[43] M. N. Asghar, R. Kousar, H. Majid, and M. Fleury, ‘‘Transparent encryp-
tion with scalable video communication: Lower-latency, CABAC-based
schemes,’’ J. Vis. Commun. Image Represent., vol. 45, pp. 122–136,
May 2017.
[44] A. Popov, Prohibiting RC4 Cipher Suites, Standard RFC 7465, IETF,
Fremont, CA, USA, Feb. 2015.
[45] G. Alvarez and S. Li, ‘‘Some basic cryptographic requirements for
chaos-based cryptosystems,’’ Int. J. Bifurcation Chaos, vol. 16, no. 8,
pp. 2129–2151, 2006.
[46] M. N. Asghar, M. Ghanbari, M. Fleury, and M. J. Reed, ‘‘Confidentiality
of a selectively encrypted H.264 coded video bit-stream,’’ J. Vis. Commun.
Image Represent., vol. 25, no. 2, pp. 487–498, 2014.
[47] J. Daemen and V. Rijmen, The Design of Rijndael: AES—The Advanced
Encryption Standard. Berlin, Germany: Springer, 2002.
[48] A. Shifa, M. N. Asghar, S. Noor, N. Gohar, and M. Fleury, ‘‘Lightweight
cipher for H.264 videos in the Internet of multimedia things with encryp-
tion space ratio diagnostics,’’ Sensors, vol. 19, no. 5, p. 1228, 2019.
[49] StratoKey. (2018). Technical White Paper. Accessed: Feb. 18, 2019.
[Online]. Available: https://www.stratokey.com/resources/stratokey-
white-paper
[50] Pkware. (2019). Smartcrypt Transparent Data Encryption (TDE) Reliable
Encryption for Structured and Unstructured Data [Internet]. Accessed:
May 15, 2019. [Online]. Available: https://www.pkware.com/smartcrypt-
tde
[51] Sighthound. Computer Vision Software With Deeply-Learned AI Vision
API & SDK’s | Technology. Accessed: Feb. 1, 2019. [Online]. Available:
https://www.sighthound.com/technology/
[52] IBM. IBM Intelligent Video Analytics—Overview—Ireland. Accessed:
Jul. 7, 2019. [Online]. Available: https://www.ibm.com/analytics/ie/en/
VOLUME 7, 2019
[53] EPeri. (2019). Open Source—EPeri [Internet]. Accessed: May 15, 2019.
[Online]. Available: https://eperi.com/open-source/
[54] Clearance. (2019). Omnicast | Genetec [Internet]. Accessed:
May 15, 2019. [Online]. Available: https://www.genetec.com/solutions/
all-products/omnicast
[55] Kinesense. Video Redaction for GDPR. Accessed: Feb. 1, 2019. [Online].
Available: https://www.kinesense-vca.com/2018/05/25/kinesense-tips-
tricks-18-gdpr-addition/
[56] SeekLayer. SeekLayer—Fast, Affordable, GDPR-Compliant
Video Redaction. Accessed: May 15, 2019. [Online]. Available:
https://www.seeklayer.com#cctv_redaction_section
[57] S. Bellovin and R. Housley, Guidelines for Cryptographic Key Manage-
ment, Standard RFC 4107, IETF, Fremont, CA, USA, Jun. 2005.
[58] A. Senior, S. Pankanti, A. Hampapur, L. Brown, Y.-L. Tian, A. Ekin,
J. Connell, C. F. Shu, and M. Lu, ‘‘Enabling video privacy through
computer vision,’’ IEEE Security Privacy, vol. 3, no. 3, pp. 50–57,
May 2005.
[59] T. Aste, P. Tasca, and T. D. Matteo, ‘‘Blockchain technologies: The foresee-
able impact on society and industry,’’ Computer, vol. 50, no. 9, pp. 18–28,
Jan. 2017.
[60] S. Nakamoto. (2008). Bitcoin: A Peer-to-Peer Electronic Cash System.
[Online]. Available: http://www.cryptovest.com
[61] M. Liu, J. Shang, P. Liu, Y. Shi, and M. Wang, ‘‘VideoChain: Trusted video
surveillance based on blockchain for campus,’’ in Cloud Computing and
Security—ICCS (Lecture Notes in Computer Science), vol. 11066, X. Sun,
Z. Pan, and E. Bertino, Eds. Cham, Switzerland: Springer, 2018, pp. 48–58.
[62] H. Vranken, ‘‘Sustainability of bitcoin and blockchains,’’ Current Opinion
Environ. Sustainability, vol. 28, pp. 1–9, Oct. 2017.
[63] H. Al-Khateeb, G. Epiphaniou, and H. Daly, ‘‘Blockchain for modern digi-
tal Forensics: The chain-of-custody as a distributed ledger,’’ in Blockchain
and Clinical Trial (Advanced Sciences and Technologies for Security
Applications), H. Jahankhani, S. Kendzierskyj, A. Jamal, G. Epiphaniou,
and H. Al-Khateeb, Eds. Springer-Verlag, 2019.
[64] K. Wüst and A. Gervais, ‘‘Do you need a Blockchain?’’ in Proc. Crypto
Valley Conf. Blockchain Technol., Jun. 2018, pp. 45–54.
MAMOONA N. ASGHAR received the Ph.D.
degree with the School of Computer Science
and Electronic Engineering, University of Essex,
Colchester, U.K., in 2013. She has been a Marie
Sklodowska-Curie (MSC) Career-Fit Research
Fellow with the Software Research Institute,
Athlone Institute of Technology (AIT), Ireland,
since June 2018. As an MSC Principal Investigator
(PI), her research targets the proposals and imple-
mentation of technological solutions for General
Data Protection Regulation (GDPR) compliant Surveillance systems. She is
also a regular Faculty Member with the Department of Computer Science and
Information Technology (DCS & IT), The Islamia University of Bahawalpur,
Punjab, Pakistan, where she is currently on postdoc leave. She has more
than 14 years of teaching and R&D experience. She has published several
ISI indexed journal articles along with numerous International conference
papers. She is also actively involved in reviewing for renowned journals and
conferences. Her research interests include security aspects of multimedia
(image, audio and video), compression, visual privacy, encryption, steganog-
raphy, secure transmission in future networks, video quality metrics, and key
management schemes.
111725
 M. N. Asghar et al.: Visual Surveillance Within the EU General Data Protection Regulation
NADIA KANWAL received the M.Sc. and Ph.D.
degrees in computer science from the University
of Essex, Essex, U.K., in 2009 and 2013, respec-
tively. She is currently a Marie Sklodowska-Curie
Career-Fit Postdoctoral Fellow with the Software
Research Institute, Athlone Institute of Technol-
ogy (AIT), Ireland. The primary objective of this
fellowship is to propose technological solutions
for privacy protection of humans as per GDPR
guidelines. She is also associated with the Lahore
College for Women University, Pakistan, where she is also an Associate
Professor and is also on leave to pursue Postdoctoral Fellowship. She is
applying deep learning methods to improve the performance of vision algo-
rithms for detection and matching tasks which can help to develop robust
solutions for different vision-related applications. Her research interests
include machine learning, image/video processing, medical imaging, and
privacy. She remained a student member of the IEEE Computer Society,
the Institution of Engineering and Technology, and the British Machine
Vision Association. She has been actively involved in reviewing for reputed
conferences and journals.
BRIAN LEE received the Ph.D. degree from the
Trinity College Dublin, Dublin, Ireland, in the
application of programmable networking for net-
work management. He has over 25 years R&D
experience in telecommunications network mon-
itoring, their systems and software design and
development for large telecommunications prod-
ucts with very high impact research publications.
He was the Director of research for LM Erics-
son, Ireland, with responsibility for overseeing all
research activities, including external collaborations and relationship man-
agement. He was an Engineering Manager with Duolog Ltd., where he was
responsible for strategic and operational management of all research and
development activities. He is currently the Director of the Software Research
Institute, Athlone Institute of Technology, Athlone, Ireland.
111726
MARTIN FLEURY received the degree in modern
history from Oxford University, U.K., the degree
in maths/physics from the Open University, Milton
Keynes, U.K., the M.Sc. degree in astrophysics
from the QMW College, University of London,
U.K., in 1990, the M.Sc. degree in parallel com-
puting systems from the University of South-West
England, Bristol, in 1991, and the Ph.D. degree in
parallel image-processing systems from the Uni-
versity of Essex, Colchester, U.K. He was a Senior
Lecturer with the University of Essex, after which he became a Visiting Fel-
low. He is currently associated with the School of Engineering, Arts, Science,
Technology and Engineering (EAST), University of Suffolk, Ipswich, U.K.
He is also a Free-Lance Consultant. He has authored or coauthored around
295 and book chapters on topics such as document and image compression
algorithms, performance prediction of parallel systems, software engineer-
ing, reconfigurable hardware, and vision systems. He has published or edited
books on high performance computing for image processing and peer-to-peer
streaming. His current research interests include video communication over
wireless networks.
MARCO HERBST received the degree in mechan-
ical engineering from the Trinity College Dublin,
in 1999. He has successfully leads several soft-
ware development teams. As CEO and CTO of
Jobs.ie and Evercam, he was responsible for lead-
ing a team of software developers to build Ireland’s
most popular recruitment website and CCTV sys-
tems. He worked on CCTV hardware, software and
large-scale camera deployments for seven years.
He is currently an Innovator in time-lapse and
monitoring software for construction projects and urban CCTV systems,
positioning his company Evercam Ltd., as a Market Leader in the sector.
YUANSONG QIAO received the B.Sc. and M.Sc.
degrees in solid mechanics from Beihang Univer-
sity, Beijing, China, in 1996 and 1999, respec-
tively, and the Ph.D. degree in computer applied
technology from the Institute of Software, Chinese
Academy of Sciences (ISCAS), Beijing, in 2008.
He is currently a Senior Research Fellow with the
Software Research Institute (SRI), Athlone Insti-
tute of Technology (AIT), Ireland. His research
interests include the future Internet architecture,
blockchain systems, the IoT systems, and edge intelligence and computing.
He is a member of the IEEE (Communications and Computer Societies and
Blockchain Community) and ACM (SIGCOMM and SIGMM).
VOLUME 7, 2019
 Prepared by: Group 3

NO. NAME STUDENT ID

1. 'IZZAH NUR 'AINA BINTI ZAINUDDIN 2019810782

2. AINA MELINA JOHAN ARIFF LIM 2019883806

3. NIK AQIL BAIG BIN KASHIF BAIG MUGHAL 2019810898

4. PUTERI SARAH KAIYISAH BINTI MEGAT 2019810776

SUHAIMI

5. SITI NURHUSNA BINTI ABDUL HALIM 2019847768

6. WAN MUHAMAD SYAHMI BIN WAN 2019811088

MUHAMAD SABRI

Title :
Visual Surveillance Within the EU General Data Protection Regulation: A Technology
Perspective

Journal : IEEE Access

Name of Authors :
Mamoona N. Asghar, Nadia Kanwal, Brian Lee, Martin Fleury, Marco Herbst and Yuansong
Qiao

Abstract :
Background info of the article : The article explains on need to protect that visual information
from illegal utilization by untrusted parties. This is because, though technology has given many
benefits to people, it also comes with disadvantages whereby it could invade their privacy. As an
example, a surveillance camera captures videos of a person could improve their personal
safety, however, if the videos is used by untrusted parties, it could be a breach of privacy and
brings harm to the particular person.

Objectives : Aims to discuss regulatory safeguards of visual privacy, such as reversible

protection, from the technological point-of-view. In the context of GDPR, the roles of machine
learning (i.e. within computer vision), image processing, cryptography, and blockchain are
explored as a way of deploying Data Protection-by-Design solutions for visual surveillance data.
Methodology : The paper surveys the existing market-based data protection solutions and
provides suggestions for the development of GDPR-compliant Data Protection-by-Design video
surveillance systems.

Findings :

PRINCIPLES OF LAW OR LEGISLATIONS

- European General Data Protection Regulation (GDPR) was adopted in April 2016 and
came into force in May 2018 in every EU member state.
- Purposes of GDPR are to ensure transparency to the act of processing one’s data and
improve the collective digital economy of the EU states
- Rights are conferred onto individuals in which the responsibilities are carried on by those
who process personal data.
- GDPR is globally adopted in which the companies outside the EU that process personal
data of any EU citizen with the purpose of selling goods, giving services, tracking the
actions of any EU citizens are subject to the regulation. A Data Protection Officer must
be appointed by the companies who bear the responsibility of ensuring GDPR
compliance. Disobedience will cause a substantial fine of 6% of the company’s global
revenue or of RM30 Million Euros (whichever is greater).

Article 5(1) provides the GDPR principles in protecting the privacy to the act of processing
personal data. The controllers of the collected data must be held accountable for the failure in
complying as provided under Article 5(2).
- Article 5(1)(a): Lawfulness, fairness and transparency.
1) Lawfulness: processing personal data must comply with the GDPR requirements.
2) Fairness: personal data shall be processed in means that the subjects are able to
reasonably expect. The data cannot be employed that brings any unjustified
adverse effects on the subjects.
3) Transparency: explicit grounds in collecting and processing of personal data
must be disclosed.
- Article 5(1)(b): Limitation of the personal data processing is that personal data can be
obtained subject to the specified, explicit and legitimate purposes. The subjects must be
informed of the reasons for the act of collecting personal data and the consent of the
subjects must be obtained.
- Article 5(1)(c): Personal data of a subject must be minimised to what is adequate,
relevant and limited to what is necessary for the purpose of data processing.
- Article 5(1)(d): The data must be accurate in which any alteration of the data is
prohibited. Editable data management systems must also be made in permitting the
subjects to update their personal data.
 - Article 5(1)(e): The personal data can only be kept for no longer than necessary. If the
data are no longer necessary, the data must be removed from the repositories of a
controller.
- Article 5(1)(f): The data must be protected by integrity and confidentiality in which the
controllers must employ any organizational measures in ensuring the security of the
personal data including protecting against unlawful processing or accidental loss,
destruction or damage to the data.

Article 6
- Article 6(4)(e): Proper safeguards must be employed such as the procedures of
encryption or pseudonymisation (replacing any information which could be used to
identify an individual with a pseudonym, or, in other words, a value which does not allow
the individual to be directly identified) to protect the personal data.
- Article 6(f): lawful processing of personal data is permitted provided that the act of
processing must not offend the rights and freedoms of the data subject (especially if the
subject is a minor).

RIGHTS OF THE DATA SUBJECTS

Article 13 provides that some information must be disclosed to the data subject when the
personal data is collected.
- Article 13(2): additional information must be provided to the data subject pertaining to
the retention period, the right to request for the removal of personal data, the right to
withdraw consent.

8 rights of data subjects under GDPR. These rights are conferred to the EU citizens.
1. Article 15: Right to access- data subjects have the right to request access for the
collected data. The companies are required to provide a cost free copy of collected
personal data in an electronic format.
2. Article 16: Right to rectification- the subjects have the right to alter and update their
collected data
3. Article 17: Right to erasure- the companies are required to remove or erase the
collected and processed personal data of subjects when it is found that the data is no
longer necessary or when the data subjects withdraw their consent.
4. Article 18: Right to restrain processing- the data subjects may request for the data to be
held back from any act of processing. The data can be kept by the controller without any
use.
5. Article 19: Right to be informed- the data subjects must be informed of the collection of
their personal data by the companies and thus free consent of the subjects must be
obtained for the collection.
6. Article 20: Right to data portability- the data subjects have the right to transfer their
personal data from one service provider to another service provider.
7. Article 21: Right to oppose the processing of their personal data for direct marketing of
the companies.
8. Article 34: Right to be given notice of any breach of the personal data collection. (must
 be notified within 72 hours.

3 Interesting Points

Point 1 – Recognition of Visual Data

-personal data may include the person’s physical information (facial features, clothing
information) and behaviour (facial gestures, physical actions).
-the recognition of humans through this information is an easy task for humans
-a research on facial recognition was conducted and humans were found to be good at
recognising highly distorted faces
-Gilad Gutnick et al. found that thinned and flattened celebrity faces could be recognised by
humans accurately up to 80% distortion. Beyond that, recognition starts to be less accurate
however half of the celebrities were still identified correctly by the volunteers

Premises for GDPR-Compliant Surveillance

-privacy protection should be affected through methods which will distort the video at a sufficient
level

1) Personal Space Visual Data

-a data is said to be recorded in a personal space when it occurs in a domestic setting or within
a household (Article 8 of the European Convention on Human Rights)
-examples: CCTV system installed in homes or in the playground to monitor children both for
safety reasons, CCTV installed in garages for vehicle safety
-video footages obtained in those circumstances do not require protection since they are not
recorded to be forwarded to any third party unless a serious incident relating to crime or tort
happens. Therefore, generally, these CCTV footages (private data) are deemed as invaluable
for local authorities, city planners or other third parties. GDPR compliance is not mandatory.

2) Impersonal Space Visual Data

-the data is not linked to an individual and cannot be used for surveillance purpose
-examples: monitoring of traffic flows within public transport, shopping malls, crowds in
road/public parks
-since the data is used for flows of vehicles or impersonal crowds instead of measuring
individuals, GDPR compliance is not necessary. However, it is necessary to protect such data
from being misused

3) Protected Space Visual Data

-lies between private and public spaces (semi-personal-impersonal space)

-example: cameras installed in protected spaces for safety (outside road view from a
home/office, monitored gardens and gates of neighbours, monitored roads, neighbours,
personal interaction through Dash Cams
-protected space data is valuable to third parties such as police or city planners thus, a GDPR
solution is required

-We find this point interesting and necessary to be known because our faces and actions are
being recorded by CCTVs installed all around the premises we enter or associate with.
Therefore, it is important for us to know our rights as the data subject and if our privacy is
protected by the law.
-In Malaysia, CCTV recordings in public or private places are subject to the Personal Data
Protection Act 2010 (Act 709).
-Section 8 states that subject to section 39, no personal data shall, without the consent of the
data subject, be disclosed for any purpose other than allowed by the Act.
-Personal data processed only for the purposes of that individual's personal, family, or
household affairs, including recreational purposes, are exempted from the PDPA. (similar to the
personal space visual data concept)

Point 2 – Data Protection By Design Through Protection

-Data Protection-by-Design is an essential technical concept that should be applied during an
implementation of a surveillance system
-It means ‘‘safeguards provided to the personal data through technological design’’.
-The processing and handling of data procedures are best adhered to when they are integrated
in the technology.
-Article 25 focusses on Data Protection-by-Design and - by-default (refer also to Section II.A).
Data Protection-by Default ensures data minimisation in the recording. That is to say only
necessary information related to a data subject is collected and retained in the recording. If Data
Protection-byDesign is implemented with proper technical measures, it will automatically ensure
Data Protection-by-Default

Point 3 – Keys Generation Evidence Chain

-Keys Generation Evidence Chain: Likewise, blockchain technology can be utilized for the
creation and management of cryptographic keys for data controllers, the police and the
supervisory authorities.
-The private secret keys for the algorithm to decrypt the encrypted video for an authenticated
receiver are generated with a secure hash value. The hash can be embedded within the key to
track the originality of key generation and be transferred to other stakeholders.
- On every key transfer, the secure hash can be re-generated to verify the originality of a key.
Otherwise, a compromised key will be discarded.
- Blockchain ledgers can operate along with standard key management algorithms to
strengthen the key-generation history management and the original key verification process.
Legal Development
-The introduction of GDPR into European law by the EU parliament caters to the main problem
of this issue by protecting individuals’ rights to freedom as all data processing or exporting
would be fully operated only with the data subject’s consent.
-The author had summarise the regime of GDPR in the context of visual personal data:
● Harmonization of privacy protection regulation in the EU
● Freedom rights where EU citizens/visitors are provided with safeguards concerning
their identifiable data
● Notification - A requirement to place meaningful symbols that could notify people that
CCTV is recording
● Access rights of individuals - Consent is required in the collection of personal data.
Data subjects have the rights to request access and erasure of their personal visual data
in a reasonable time-frame.
● Privacy safeguards - To keep the originality of data hidden and identify data through
automated techniques.
● Retention period - A given time-frame which allows personal visual data to be kept only
within that time.
-Article 5(1)(a)-(f) described all 6 principles that must be applied in ensuring privacy during
processing of personal data.

Recommendation

Around the world, video surveillance is a common occurrence.

● Protecting an individual's privacy is a key responsibility that controllers must fulfil, in
addition to the self-evident benefits of surveillance applications. A person whose identity
is being tracked by CCTV systems has a fundamental right to privacy. GDPR went into
effect in the EU in order to protect people's freedom rights. GDPR is a regulation rather
than a directive because it applies to all EU member states directly and equally.
● The goal of this article was to throw light on GDPR-compliant visual privacy protection
measures in the context of CCTV surveillance. This paper discusses GDPR Article 6 in
relation to both pseudonymization and encryption methods; GDPR Article 25 for data
protection-by-design and by default; GDPR Article 32 for secure data processing; and
GDPR Article 35 for DPIS as a foundation for proposing visual protection solutions via
technology. The article delves into the following topics: an individual's visual personal
data; the premises or circumstances for GDPR-compliant surveillance; and existing vs.
future technical solutions for helping GDPR laws.
● Because of its reversible nature and resilience in preventing infiltration, the GDPR places
a strong focus on encryption. Cryptographic solutions, on the other hand, are costly to
install in terms of software, hardware, and people. To handle and maintain the
complicated ciphers within their organisations, companies must hire adequately skilled IT
staff.
Visual Surveillance, M. N. Asghar et al. When they export data to a third party, they must comply
with the EU General Data Protection Regulation.
● To maintain and alter that information, the third party may need to hire professional
cryptographers.
● Data protection-by-design is still in its infancy, and therefore necessitates the use of
dependable, secure technology for the privacy supply of digitised visual data in the
framework of the regulation. Article 25 provides a foundation for safe technology-based
solutions, whereas Article 32 defines the security services offered by those solutions. To
avoid interception attempts, for example, Confidentiality ensures that personal data,
whether in storage or transit, can only be accessed by verified persons/systems. Integrity
dictates that any changes to personal data, whether in storage or transit, can only be
made with the consent of the data subject by authenticated persons/systems. Availability
necessitates that specific hardware and software services can only be accessed by
authorised persons/systems, which can be harmed or rendered unusable as a result of
an interruption attack. Resilience ensures the technology's robustness, allowing it to
recover from attacks and return to its original state.

Conclusion

● To summarise, because visual privacy is so important, it is critical for concerned firms to

provide robust Data protection-by-design solutions. The article discusses technology
solutions offered by businesses and makes recommendations for improvements. The
authors of this research believe that encryption combined with video redaction, whether
used together or separately, is insufficient.
● To implement effective Data protection-by-design solutions for video surveillance
systems, multiple information technology domains will need to cooperate together.
Clearly, there would be an additional cost to reduce the privacy problems associated with
the usage of visual data. However, this expense will result in video surveillance
companies implementing GDPR compliance in a serious way, which will benefit EU
businesses. Importantly, if not already in place, similar solutions will be required outside
of the EU, and they are now required for enterprises working with EU clients from
outside the EU.
 Volume 3, No. 1 | January–February 2020

R A I L The Journal of Robotics,

Artificial Intelligence & Law

Editor’s Note: The Future

Victoria Prussen Spears

Flying Cars: Are We Ready for Them?

Elaine D. Solomon

U.S. AI Regulation Guide: Legislative Overview and Practical Considerations

Yoon Chae

Artificial Intelligence in Healthcare: Can Regulation Catch Up with Innovation?

Alaap B. Shah

New York and New Jersey Make an Early Effort to Regulate Artificial Intelligence
Marc S. Martin, Charlyn L. Ho, and Michael A. Sherling

Artificial Intelligence and the Fair Housing Act: Algorithms Under Attack?
William T. Gordon, Katherine Kirkpatrick, and Katherine Mueller

Angel Investing Lessons: The First Mover Disadvantage

Paul A. Jones

The Convertible Debt Valuation Cap: The Trigger Financing Investor Perspective
Paul A. Jones

Big News for Small Mobility: Germany Opens Up to E-Scooters

Andreas Grünwald, Christoph Nüßing, and Theresa Oehm

Everything Is Not Terminator: AI Issues Raised by the California Consumer Privacy Act
John Frank Weaver

FULL
®

COURT
PRESS
RAIL The Journal of Robotics,
Artificial Intelligence & Law
Volume 3, No. 1 | January–February 2020

5 Editor’s Note: The Future

Victoria Prussen Spears

9 Flying Cars: Are We Ready for Them?

Elaine D. Solomon

17 U.S. AI Regulation Guide: Legislative Overview and Practical

Considerations
Yoon Chae

41 Artificial Intelligence in Healthcare: Can Regulation Catch Up with

Innovation?
Alaap B. Shah

47 New York and New Jersey Make an Early Effort to Regulate

Artificial Intelligence
Marc S. Martin, Charlyn L. Ho, and Michael A. Sherling

53 Artificial Intelligence and the Fair Housing Act: Algorithms Under

Attack?
William T. Gordon, Katherine Kirkpatrick, and Katherine Mueller

57 Angel Investing Lessons: The First Mover Disadvantage

Paul A. Jones

63 The Convertible Debt Valuation Cap: The Trigger Financing

Investor Perspective
Paul A. Jones

69 Big News for Small Mobility: Germany Opens Up to E-Scooters

Andreas Grünwald, Christoph Nüßing, and Theresa Oehm

73 Everything Is Not Terminator: AI Issues Raised by the California

Consumer Privacy Act
John Frank Weaver
 EDITOR-IN-CHIEF
Steven A. Meyerowitz
President, Meyerowitz Communications Inc.

EDITOR
Victoria Prussen Spears
Senior Vice President, Meyerowitz Communications Inc.

BOARD OF EDITORS
Miranda Cole
Partner, Covington & Burling LLP

Kathryn DeBord
Partner & Chief Innovation Officer, Bryan Cave LLP

Melody Drummond Hansen

Partner, O’Melveny & Myers LLP

Paul B. Keller
Partner, Norton Rose Fulbright US LLP

Garry G. Mathiason
Shareholder, Littler Mendelson P.C.

Elaine D. Solomon
Partner, Blank Rome LLP

Linda J. Thayer
Partner, Finnegan, Henderson, Farabow, Garrett & Dunner LLP

Mercedes K. Tunstall
Partner, Pillsbury Winthrop Shaw Pittman LLP

Edward J. Walters
Chief Executive Officer, Fastcase Inc.

John Frank Weaver

Attorney, McLane Middleton, Professional Association
THE JOURNAL OF ROBOTICS, ARTIFICIAL INTELLIGENCE & LAW (ISSN
2575-5633 (print) /ISSN 2575-5617 (online) at $495.00 annually is published
six times per year by Full Court Press, a Fastcase, Inc., imprint. Copyright
2020 Fastcase, Inc. No part of this journal may be reproduced in any form—by
microfilm, xerography, or otherwise—or incorporated into any information
retrieval system without the written permission of the copyright owner. For
customer support, please contact Fastcase, Inc., 711 D St. NW, Suite 200,
Washington, D.C. 20004, 202.999.4777 (phone), 202.521.3462 (fax), or email
customer service at support@fastcase.com.

Publishing Staff
Publisher: Morgan Morrissette Wright
Journal Designer: Sharon D. Ray
Cover Art Design: Juan Bustamante

Cite this publication as:

The Journal of Robotics, Artificial Intelligence & Law (Fastcase)

This publication is sold with the understanding that the publisher is not engaged
in rendering legal, accounting, or other professional services. If legal advice or
other expert assistance is required, the services of a competent professional should
be sought.

Copyright © 2020 Full Court Press, an imprint of Fastcase, Inc.

All Rights Reserved.

A Full Court Press, Fastcase, Inc., Publication

Editorial Office
711 D St. NW, Suite 200, Washington, D.C. 20004
https://www.fastcase.com/

POSTMASTER: Send address changes to THE JOURNAL OF ROBOTICS,

ARTIFICIAL INTELLIGENCE & LAW, 711 D St. NW, Suite 200, Washington,
D.C. 20004.
 Articles and Submissions

Direct editorial inquiries and send material for publication to:

Steven A. Meyerowitz, Editor-in-Chief, Meyerowitz Communications Inc.,

26910 Grand Central Parkway, #18R, Floral Park, NY 11005, smeyerowitz@
meyerowitzcommunications.com, 646.539.8300.

Material for publication is welcomed—articles, decisions, or other items of interest

to attorneys and law firms, in-house counsel, corporate compliance officers,
government agencies and their counsel, senior business executives, scientists,
engineers, and anyone interested in the law governing artificial intelligence and
robotics. This publication is designed to be accurate and authoritative, but neither
the publisher nor the authors are rendering legal, accounting, or other professional
services in this publication. If legal or other expert advice is desired, retain the
services of an appropriate professional. The articles and columns reflect only the
present considerations and views of the authors and do not necessarily reflect
those of the firms or organizations with which they are affiliated, any of the former
or present clients of the authors or their firms or organizations, or the editors or
publisher.

QUESTIONS ABOUT THIS PUBLICATION?

For questions about the Editorial Content appearing in these volumes or reprint
permission, please call:

Morgan Morrissette Wright, Publisher, Full Court Press at mwright@fastcase.com

or at 202.999.4878

For questions or Sales and Customer Service:

Customer Service
Available 8am–8pm Eastern Time
866.773.2782 (phone)
support@fastcase.com (email)

Sales
202.999.4777 (phone)
sales@fastcase.com (email)
ISSN 2575-5633 (print)
ISSN 2575-5617 (online)
U.S. AI Regulation Guide:
Legislative Overview and
Practical Considerations
Yoon Chae*

This article discusses the state of U.S. regulation of artificial intelligence

(“AI”), covering legislative instruments directed to algorithmic account-
ability, the use of facial recognition technology and associated data, and
promoting “transparency” when using AI, as well as pending federal bills
on general governance or research issues for AI, among other things. The
author also discusses practical considerations when using AI for businesses
and in-house counsel.

Regulation of artificial intelligence (“AI”) is still in its infancy,

perhaps with the exception of autonomous vehicles, which have
seen the most legislative activities worldwide.1 Many countries have
just issued their national plans, guidelines, or codes—which often
highlight essential principles for developing ethical AI—without
having passed much substantive law; notable examples include the
European Parliament’s resolution on Civil Law Rules on Robotics
(February 2017), the European Union’s Ethics Guidelines for Trust-
worthy AI (April 2019), and OECD’s Council Recommendation
on Artificial Intelligence (May 2019).2 The time may now be ripe,
according to certain experts, to tackle the associated challenges in
transitioning from these general principles to actual legislation in
order to provide the necessary regulatory frameworks that safe-
guard those principles.3
Much of the recent development in filling the legal void that
exists for AI governance is surprisingly coming from the United
States, which, with the exception of three AI reports issued by the
Obama administration,4 has been relatively inactive with AI regula-
tion until last year. In 2015–2016, for example, the 114th Congress
saw only two bills containing the term “artificial intelligence,” which
increased to 42 bills with the 115th Congress (2017–2018) and to
51 bills for the current, 116th, Congress, as of early November
2019. Similar trends are observed at the state and city levels as well.
California, for instance, had 0 bills on “artificial intelligence” two

Robotics, Artificial Intelligence & Law / January–February 2020, Vol. 3, No. 1, pp. 17–40.
© 2020 Full Court Press. All rights reserved.
ISSN 2575-5633 (print) / ISSN 2575-5617 (online)
18 The Journal of Robotics, Artificial Intelligence & Law [3:17

legislative sessions ago (2015–2016), five bills during the last term
(2017–2018), and 13 bills for the current legislature (2019–2020).
Other than regulatory instruments on autonomous vehicles 5—
which are beyond the scope of this article—most of the lawmaking
on AI, automated systems, and their underlying technologies have
been in the following areas:

The first area is what this article labels as “policy,” and it

includes documents like executive orders, resolutions, and plans
that reflect the U.S. government’s policies on AI regulation. The
second category covers legislative instruments directed to algorith-
mic accountability, likely reflecting the governments’ response to
recently publicized concerns of algorithmic bias and discrimina-
tion. The third is on the rapidly growing body of law that governs
the use of facial recognition technology and associated data. The
fourth area is labeled as “transparency,” and it includes those that
are primarily directed to promoting transparency when it comes to
the use of AI in different contexts. The last category is designated as
“other,” and comprises pending federal bills on general governance
or research issues for AI, among other things.6
2020] U.S. AI Regulation Guide 19

Policy
The Executive Order and the NIST Plan

President Trump issued the Executive Order on Maintaining

American Leadership in Artificial Intelligence (No. 13,859) on
February 11, 2019.7 The Order explains that the federal government
plays an important role in facilitating AI research and development
(“R&D”) and in promoting trust, training people for a changing
workforce, and protecting national interests, security, and values.8
It launched the “American AI Initiative,” guided by five principles:

(1) Driving technological breakthroughs;

(2) Driving the development of appropriate technical
standards;
(3) Training workers with the skills to develop and apply
AI technologies;
(4) Protecting American values, including civil liberties and
privacy, and fostering public trust and confidence in AI
technologies; and
(5) Protecting U.S. technological advantage in AI, while
promoting an international environment that supports
innovation.9

The Order requires all executive departments and agencies that

develop or deploy AI, among other things, to adhere to six strategic
objectives that generally align with the five principles listed above.10
An objective that is particularly worth noting is the one directed
to ensuring that “technical standards minimize vulnerability to
attacks from malicious actors and reflect Federal priorities for inno-
vation, public trust, and public confidence in systems that use AI
technologies” and highlighting the need to “develop international
standards to promote and protect those priorities.”11 The Order
tasks the National Institute of Standards and Technology (“NIST”)
of the U.S. Department of Commerce to create a plan for federal
engagement in the development of technical standards to support
reliable, robust, and trustworthy AI systems.12
In response, in August 2019, NIST issued its Plan for Federal
Engagement in Developing Technical Standards.13 It identifies nine
areas of technical AI standards that are available or being developed,
among which “[s]tandardization of AI safety, risk management, and
20 The Journal of Robotics, Artificial Intelligence & Law [3:17

some aspects of trustworthiness such as explainability or security,

are in their formative stages and especially would benefit from
research to provide a strong technical basis for development.”14
The Plan also identifies three areas of “non-technical” AI standards
that inform policy decisions, such as “societal and ethical consid-
erations,” “governance,” and “privacy.”15
The Executive Order and the release of the NIST Plan may be
indications that federal governance of AI under the Trump admin-
istration favors or values a standards-driven approach,16 and they
are significant in that they mark the federal government’s first major
efforts in providing clarity and guidance to agencies that are looking
to adopt AI.17 NIST also has plans for more guidance in the near
future, with plans to seek public comment on recommendations
on confronting adversarial machine learning, as well as to seek
public feedback on upcoming standards for explainable AI, which
can further inform businesses on how to address those issues.18

House Resolution 153

Shortly after the Executive Order, the House of Representatives

introduced House Resolution 153 on Supporting the Development
of Guidelines for Ethical Development of Artificial Intelligence on
February 27, 2019. The Resolution seeks to strike a balance between
promoting AI’s “potential to enhance wellbeing, foster economic
growth, and improve care and services for many people” with the
need for its “safe, responsible, and democratic development.”19 It
provides ten aims for achieving that balance, including “[t]rans-
parency and explainability,” “[i]nformation privacy and the pro-
tection of one’s personal data,” “[a]ccountability and oversight for
all automated decisionmaking,” “[a]ccess and fairness regarding
technological services and benefits,” and “[s]afety, security, and
control of AI systems now and in the future,” among other things.20
Although the Resolution has not yet been adopted, it highlights
some of the same principles that are emphasized by other govern-
ments and international organizations for developing ethical and
trustworthy AI.

California’s Assembly Concurrent Resolution 215

Several months prior to the federal House Resolution 153,

California unanimously adopted Assembly Concurrent Resolution
2020] U.S. AI Regulation Guide 21

215 on September 7, 2018.21 The Resolution expresses legislative

support for using the 23 Asilomar AI Principles as the “guiding
values for the development of artificial intelligence and of related
public policy.”22 The Principles were developed in conjunction with
a 2017 conference in Asilomar, California, where experts from vari-
ous disciplines met to establish the core principles for managing
responsible development of AI.23 As of November 12, 2019, the
Principles have been signed and endorsed by 5,030 of the world’s
leading AI researchers and leaders in government, industry, and
academia, including Stephen Hawking and Elon Musk.24
The Resolution identifies the need for developing AI in a
manner that “ensures security, reliability, and consonance with
human values,” and looks to the Principles for guidance, which
are categorized into “research issues,” “ethics and values,” and
“longer-term issues” that are designed to promote safe and ben-
eficial development of AI.25 The Principles listed under ethics and
values include safety, failure transparency, judicial transparency,
responsibility, value alignment, human values, personal privacy,
liberty and privacy, and human control, among other themes.26
Since the adoption of ACR 215, California has been significantly
more legislative active on governing AI.

Algorithmic Accountability
Algorithmic Accountability Act

Senate and House bills for the Algorithmic Accountability Act

(S.  1108, H.R. 2231) were introduced in Congress on April  10,
2019, 27 likely in response to recently publicized reports on the
risks of AI’s biased outcomes. In the words of one of the sponsors,
Senator Wyden, the intention behind the bill was to require “com-
panies to regularly evaluate their tools for accuracy, fairness, bias,
and discrimination.”28
If the bill were to get enacted, it would require covered entities
to conduct “impact assessments” on their “high-risk” automated
decision systems in order to evaluate the impacts of the system’s
design process and training data on “accuracy, fairness, bias, dis-
crimination, privacy, and security.”29 The Act also provides that the
impact assessments should be performed “with external third par-
ties, including independent auditors and independent technology
experts” when “reasonably” possible.30 Businesses would then be
22 The Journal of Robotics, Artificial Intelligence & Law [3:17

required to “reasonably address” any identified issues in a “timely

manner.”31
“Automated decision system” is defined broadly as “any compu-
tational process, including one derived from machine learning, sta-
tistics, or other data processing or artificial intelligence techniques,
that makes a decision or facilitates human decision making, that
impacts consumers.”32 Covered entities would include companies
that:

(1) Make $50 million or more per year;

(2) Hold data for over one million consumers or consumer
devices; or
(3) Act as data brokers that buy and sell personal information.33

The Algorithmic Accountability Act expressly provides that it

would not preempt any state law,34 which means that businesses
would need to remain vigilant on keeping up with any state law
development on similar subject matter. The bill also does not pro-
vide for a private right of action, nor extraterritorial jurisdiction.35
Instead, it would be enforced by the Federal Trade Commission
(“FTC”) under Section 5 of the Federal Trade Commission Act on
deceptive and unfair acts and practices, or via civil suits brought
by the affected state’s attorney general.36
Although the bill’s future is uncertain, it is significant in that it
is the first federal legislative effort to regulate AI across industries,37
with broad proposed protections that mirror the General Data
Protection Regulation (“GDPR”) in several ways.38 The bill is also
significant in that it may be a harbinger of the laws to come,39 and
in fact, some of this movement is already gaining momentum at
the state and city levels.

New Jersey’s Algorithmic Accountability Act

Just a month after the federal Algorithmic Accountability Act

was introduced, New Jersey introduced a similar bill, A.B. 5430,
titled “New Jersey Algorithmic Accountability Act,” on May  20,
2019. 40 Like the federal counterpart, the bill requires covered
entities to conduct impact assessments on “high-risk” automated
decisions systems and information systems.41
“Automated decision system impact assessment” requires an
evaluation of the system’s development process, including the
2020] U.S. AI Regulation Guide 23

design and training data, for impacts on “accuracy, fairness, bias,

discrimination, privacy, and security,” and must include “a detailed
description of the best practices used to minimize the risks” and a
“cost-benefit analysis,” among other things.42 The bill also requires
the covered entities to work with independent third parties, record
any bias or threat to the security of consumer’s personally identifi-
able information discovered through the impact assessments, and
provide any other information that is required by the Director of
the Division of Consumer Affairs in the Department of Law and
Public Safety.43
The bill is significant in that it mirrors the federal Algorithmic
Accountability Act, with a broad scope and large potential impact
on businesses.

Other States’ and Cities’ Regulation of AI Bias

Although New Jersey appears to be the only state so far to have

introduced legislation mirroring the federal Algorithmic Account-
ability Act, there is also a growing trend toward other state and
city governments considering laws that regulate algorithmic bias
in the context of AI procurement and use by government entities.44
In early 2018, for example, New York City enacted the first algo-
rithm accountability law—“A Local Law in relation to automated
decision systems used by agencies” (Int. No. 1696-2017)—in the
United States.45 In January 2019, Washington State introduced bills,
S.B. 5527 and H.B. 1655, with GDPR-like prohibitions against algo-
rithmic discriminations, although likewise limited to government’s
procurement and use.46 In February 2019, California introduced
a bill, S.B. 444, where businesses that rely on AI for delivery of a
product to a public entity would be required to make certain dis-
closures to the public entity regarding “the steps that it has taken
to reduce the bias inherent in the artificial intelligence system.”47

Facial Recognition Technology

Commercial Facial Recognition Privacy Act

The Commercial Facial Recognition Privacy Act (S. 847) was

introduced on March 14, 2019,48 to strengthen consumer protec-
tions and increase transparency, and in the words of one of the
24 The Journal of Robotics, Artificial Intelligence & Law [3:17

sponsors, Senator Blunt, to make sure “that people are given the
information and [] the control over how their data is shared” when
it comes to facial recognition.49
If enacted, the bill would generally prohibit covered entities
from using “facial recognition technology to collect facial recog-
nition data” of end users without providing notice and obtaining
their consent.50 “Covered entities” is broadly defined and would
include any non-governmental entity that “collects, stores, or pro-
cesses facial recognition data.”51 This definition would therefore
seemingly include entities like app operators that collect facial
data, businesses that use the technology on their premises, and
outside vendors that process such data for the original data collec-
tors.52 Facial recognition data is tied to personal information (i.e.,
“unique personal identification of a specific individual”), but may
also include pseudonymized information (i.e., where “a unique,
persistent identifier” is assigned).53
Covered entities would also be prohibited from using the tech-
nology to discriminate against the consumers, from repurposing the
facial data, or from sharing such data with third parties without
obtaining further consent from the end users.54 The covered entities
would also not be allowed to condition service of the technology
on the consumer’s consent to waive privacy rights when the use of
facial recognition technology is not necessary for that service.55 In
an effort to provide more oversight, the bill also requires the cov-
ered entities to conduct meaningful human review before making
any final decision based on the output of the technology if it can
result in a reasonably foreseeable harm or be “highly offensive”
to a reasonable end user.56 If the technology is made “available as
an online service,” then the covered entity would additionally be
required to make available an API to enable at least one third party
to conduct reasonable independent tests for accuracy and bias.57
The bill, however, contains some notable exceptions. For
example, the bill exempts “security applications” that use the
technology for “loss prevention” or “to detect or prevent criminal
activity.”58 The bill also exempts products or services “designed for
personal file management or photo or video sorting or storage if
the facial recognition technology is not used for unique personal
identification of a specific individual,” involves “identification of
public figures for journalistic media created for public interest,”
“involves identification of public figures in copyrighted material,”
or is used in an emergency.59
2020] U.S. AI Regulation Guide 25

The law would not preempt state laws, except to the extent such
regulations are “inconsistent” with the provisions of the bill.60 If
passed, the act would thus likely not preempt stricter state laws,
such as Illinois’s Biometric Information Privacy Act (“BIPA”) or
other state biometric information privacy laws in Texas and Wash-
ington.61 The bill does not provide for a private right of action, and
would instead be enforced by the FTC or via civil suits brought by
the affected state’s attorney general.62
This bill is worth monitoring as it makes its way through Con-
gress given that it is a bipartisan bill63 with a narrow scope that
has received early conceptual support from several technology
companies.64 Again, like the Algorithmic Accountability Act, the
bill mirrors certain aspects of the GDPR, such as the differentiation
between processors and controllers.65 Numerous commentators
believe that this bill signals the types of impending laws on facial
recognition technology. In fact, in 2019, lawmakers in at least 10
states have introduced bills to ban or delay the use of facial recog-
nition technology by government agencies or businesses.66

Other Federal Bills on Facial Recognition Technology

Since the introduction of the Commercial Facial Recognition

Privacy Act, Congress has seen an increasing number of bills on
facial recognition technology. For example, H.R. 3875 was intro-
duced on July 22, 2019, to “prohibit Federal funding from being
used for the purchase or use of facial recognition technology.”67
On July 25, 2019, the Facial, Analysis, Comparison, and Evalu-
ation (“FACE”) Protection Act of 2019 (H.R. 4021) was introduced
to prohibit a federal agency from applying “facial recognition tech-
nology to any photo identification issued by a State or the Federal
Government or any other photograph otherwise in the possession
of a State or the Federal Government unless the agency has obtained
a Federal court order determining that there is probable cause for
the application of such technology.”68
On the same day, another bill on the No Biometric Barriers
to Housing Act of 2019 (H.R. 4008) was introduced to “prohibit
the use of biometric recognition technology in certain federally
assisted dwelling units.”69 If enacted, it would also require the Sec-
retary of Housing and Urban Development (“HUD”) to submit a
report describing “the impact of such technology on the residents
26 The Journal of Robotics, Artificial Intelligence & Law [3:17

of such covered federally assisted rental dwelling units” and the

potential impacts on vulnerable communities of additional usage
of such technology in covered federally assisted rental dwelling
units, including impacts on “resident privacy, civil rights, and fair
housing,” among other required details.70 These trends may indicate
the federal government’s increasing willingness to regulate the use
of facial recognition technology.

State and City Regulation of Facial Recognition

Technology

There has been a wave of state and local laws and bills on the
procurement and use of facial recognition technology. For example,
California introduced a bill, A.B. 1215, on February 21, 2019, which
would prohibit law enforcement agencies and officials from using
any “biometric surveillance system,” including facial recognition
technology, in connection with an officer camera or data collected
by the camera. 71 On the same day, another bill, A.B. 1281, was
introduced in California, which would require California businesses
that use facial recognition technology to disclose such usage on a
physical sign that is “clear and conspicuous at the entrance of every
location that uses facial recognition technology.”72
Senate Bill 1385 was introduced in Massachusetts on January 22,
2019, to establish a moratorium on the use of face recognition sys-
tems by state and local law enforcement, and Senate Bill 5687 was
introduced in New York on May 13, 2019, to propose a temporary
stop to the use of facial recognition technology in public schools.73
Similarly, companion bills, S.B. 5528 and H.B. 1654, were intro-
duced in Washington in January 2019, concerning the procurement
and use of facial recognition technology by government entities
and privacy rights relating to facial recognition technology.74
As for cities, San Francisco and Oakland, California, and Somer-
ville, Massachusetts, passed ordinances this summer to ban the use
of facial recognition software by the police and other government
agencies.75
On a related note, there has also been a resurgence of biomet-
ric privacy bills being introduced in state legislatures. Although
the Illinois Biometric Information Privacy Act and Texas’s law
on the Capture or Use of Biometric Identifier have been around
since 2008 and 2009,76 other states have recently started enact-
ing or introducing privacy laws on biometric identifiers, such as
2020] U.S. AI Regulation Guide 27

Washington’s enactment of the bill on Biometric Identifiers in July

2017,77 California’s enactment of the California Consumer Privacy
Act (“CCPA”) in June 2018 (to become effective in January 2020),78
and Massachusetts’s introduction of a bill, S. 120, on An Act relative
to consumer data privacy in January 2019,79 among others.
The legislative trends on facial recognition technology and
associated data, combined with the resurgence of biometric privacy
laws, likely highlight the governments’ increasing attention to the
use of biometric and facial data.

Transparency
California’s Bolstering Online Transparency Act

Moving on to laws directed to improving transparency, Cali-

fornia’s Bolstering Online Transparency (“B.O.T.”) Act (S.B. 1001)
came into effect this July after being enacted last year.80 The law is
the first of its kind,81 and it prohibits the use of “a bot to commu-
nicate or interact with another person in California online, with
the intent to mislead the other person about its artificial identity”
for commercial or political purposes.82 The law defines a “bot” as
“an automated online account where all or substantially all of the
actions or posts of that account are not the result of a person.”83
There is a safe harbor exemption, however, if the bot first
discloses its identity in a “clear, conspicuous, and reasonably
designed” manner.84 Further, the B.O.T. Act applies only to bot
communications deployed via public-facing internet websites, web
applications, and digital applications, including social networks or
publications,85 and it expressly provides that no duty is imposed
on “service providers of online platforms.”86 The statute does not
specify the method of enforcement, but given that it is a part of
California’s Unfair Competition Law, it would likely be enforced by
the state attorney general via fines of up to $2,500 per violation. 87
The bill also went through significant amendments, which,
according to the Electronic Frontier Foundation (“EFF”), were the
results of its involvement.88 The bill’s initial definition of a “bot,”
for example, covered most or all bots, but was amended to focus
on bots used for a commercial or political purpose, and the bill’s
earlier requirement that platforms create a notice and takedown
system for suspected bots was also removed as a result of the
amendments.89
28 The Journal of Robotics, Artificial Intelligence & Law [3:17

California’s Anti-Eavesdropping Act

California’s Anti-Eavesdropping Act (A.B. 1395) recently passed

the Assembly and is now pending in the State Senate.90 If enacted,
it would prohibit manufacturers of smart speakers from installing
devices “without prominently informing” the user.91 It would also
prohibit any recording or transcription that “qualifies as personal
information or is not deidentified” from being used for advertising
purposes, shared or sold to third parties, or retained without the
user’s affirmative consent.92 Under the bill, a manufacturer would
“only be liable for functionality provided at the time of the original
sale of a connected television or smart speaker device and shall
not be liable for functionality provided by applications that the
user chooses to use in the cloud or are downloaded and installed
by a user.”93

Illinois’s Artificial Intelligence Video Interview Act

Illinois’s Artificial Intelligence Video Interview Act (H.B. 2557)

was signed on August 9, 2019, and will become effective in January
2020.94 It governs an employer’s ability to use AI analysis on video
interviews by requiring the employer to:

(1) Notify each applicant before the interview that AI may

be used to analyze the video interview;
(2) Provide the applicant with information on how the AI
works; and
(3) Obtain consent from the applicant to be evaluated by
the AI program before the interview.95

The Act also limits sharing of the videos—only “with persons

whose expertise or technology is necessary in order to evaluate
an applicant’s fitness for a position”—and requires deletion of an
applicant’s interview(s) within 30 days after receipt of any request
from the applicant.96 The Act does not include provisions on a
private right of action.97
The Act is significant in that it is the first U.S. law to specifically
regulate artificial intelligence as an evaluation tool on applicant
videos.98
2020] U.S. AI Regulation Guide 29

Other Federal Bills on AI

There are several other pending federal bills on AI, the earliest
being S. 2217 and H.R. 4625 on the FUTURE of Artificial Intel-
ligence Act of 2017, introduced on December 12, 2017.99 Although
never passed, it would have required the Department of Commerce
to establish a new committee to advise on topics related to the
development and implementation of AI.100
Further, H.R. 2202 on the Growing Artificial Intelligence
Through Research (“GrAITR”) Act, introduced on April 10, 2019,
and S. 1558 on the Artificial Intelligence Initiative Act (“AI-IA”),
introduced on May 21, 2019, are primarily directed to establishing
a coordinated federal initiative to accelerate research and develop-
ment on AI.101 Several components of the proposed initiative are
directed to eliminating AI bias. For example, the bills provide that
the president shall establish a National AI Research and Develop-
ment Initiative, where the responsibilities would include strength-
ening research and development with respect to AI by “identifying
and minimizing inappropriate bias in data sets algorithms, and
other aspects of artificial intelligence.” 102 The bills also seek to
“establish and implement a research and education program on” AI
and AI engineering, where the program would need to “continue to
support interdisciplinary research” on “algorithm accountability,”
“minimization of inappropriate bias in training data sets or algo-
rithmic feature selection,” and “societal and ethical implications”
of AI, among other things.103
Related bills, H.R. 2575 and S. 1363, on the AI in Government
Act were re-introduced on May 8, 2019, and are generally directed
to establishing an AI Center of Excellence.104 The Act also discusses
AI bias in some detail—for example, responsibilities of the AI
Center of Excellence would include studying “economic, policy,
legal, and ethical challenges and implications related to the use of
artificial intelligence by the Federal Government” and establishing
“best practices for identifying, assessing, and mitigating any bias
on the basis of any classification protected under Federal non-
discrimination laws or other negative unintended consequence
stemming from the use of artificial intelligence systems.”105
Many believe that these federal bills reflect the government’s
heightened awareness of the risks associated with AI bias.
30 The Journal of Robotics, Artificial Intelligence & Law [3:17

Practical Considerations for Businesses and

In-House Counsel
Internal Governance Structures and Measures

It would be helpful for businesses to establish and continu-

ally improve on its internal governance structures and measures
to promote robust oversight of AI.106 Having such infrastructure
reflect the key principles highlighted in the various regulatory
instruments, such as transparency, fairness (no bias/discrimina-
tion), human oversight, accountability, and safety, among other
things, will likely facilitate consistency and compliance with future
laws on AI.
For their internal governance structures, businesses may con-
sider establishing clear roles and responsibilities for the ethical
deployment of AI and having sound risk management and internal
controls.107 Such governance structures may also benefit from care-
ful consideration of the “susceptibility of data” and assessment of
whether using such data would promote fairness for the consum-
ers.108 Further, from a broader organizational perspective, helping
develop and establish diverse, multidisciplinary teams to build and
oversee such internal measures can provide an additional layer of
protection against potential biases that can permeate algorithms
and datasets.109
It would also be useful to provide sufficient training on the
various risks of AI and its underlying technologies, such as the
risks of inherent algorithmic bias or of discrimination stemming
from the use of facial recognition technology. Such training should
be provided not just for the legal or policy members, but also to
the engineers, coders, and product managers that are engaged in
AI development and data processing, so as to promote the culture
of fair and unbiased AI. Well-documented training processes
and internal guidelines can also provide more transparency to
regulators.

External Resources

In addition to employing any internally developed measures,

strategically considered use of effective external resources may help
safeguard against some of the risks that can arise from the use of
2020] U.S. AI Regulation Guide 31

AI, as well as bring objectivity and transparency to the businesses’

AI deployment.
When it comes to algorithmic bias, for example, businesses can
consider employing AI tools that generate metrics for evaluating
whether there are unintended biases in algorithmic models. They
can also consider using external open source AI frameworks, such
as IBM’s AI Fairness 360, which implements bias metrics and miti-
gation algorithms,110 or employing DARPA’s Explainable Artificial
Intelligence (“XAI”) program that provides a toolkit library for
developing future explainable AI systems.111 To minimize poten-
tial discrimination from the use of facial recognition technology,
businesses can consider using third-party data repositories, such as
Canadian Institute for Advanced Research’s CIFAR-10, that contain
images commonly used to train machine learning and computer
vision algorithms.112
It is also important for organizations to ensure data quality
in the form of “the accuracy of the dataset,” “completeness of the
dataset,” “veracity of the dataset,” “relevance of the dataset and the
context for data collection,” and integrity of the dataset, among
other factors. 113 It is also advisable to consider using different
datasets for “training, testing, and validation” of an AI model and
to periodically review and update the pertinent datasets.114
Keeping up with guidelines and best practices being developed
by governments, international and non-profit organizations, and
private entities can also help inform businesses on implement-
ing their own governance structures and measures. For example,
Canada’s Algorithmic Impact Assessment questionnaire or AI
Now Institute’s Algorithmic Accountability Policy Toolkit, which
are designed to help assess and mitigate the risks associated with
automated decision systems, may be useful resources.115 The Part-
nership on AI to Benefit People and Society, which was established
to study and formulate best practices on AI technologies, among
other things, may provide useful resources for businesses as well.116
Such practices may also help satisfy the Algorithmic Account-
ability Act’s and the New Jersey Algorithmic Accountability Act’s
respective provisions requiring that impact assessments evaluate
the relative benefits and costs of the system in light of its purpose,
where relevant factors include “data minimization practices.”117
Establishing best practices involving the use of such external tools
and datasets can also help satisfy the New Jersey Algorithmic
Accountability Act’s requirement that covered entities provide “a
32 The Journal of Robotics, Artificial Intelligence & Law [3:17

detailed description of the best practices used to minimize the risk

of the automated decision system” that impacts “accuracy, fairness,
bias, discrimination, privacy, and security,” among other things.118

Technical Standards

Using certain AI technical standards as benchmarks and stay-

ing up-to-date on their development can help inform businesses
on the direction of AI development and safeguard them against
future laws. For instance, the ISO (International Organization
for Standardization) and the IEC (International Electrotechnical
Commission) are developing numerous standards on Artificial
Intelligence, including standards on “Overview of trustworthiness
in Artificial Intelligence.”119 IEEE is also developing AI standards
under its P7000 series, such as P7001 on “Transparency of Autono-
mous Systems” and P7010 on “Wellbeing Metrics Standard for
Ethical Artificial Intelligence and Autonomous Systems,” which
could be helpful for businesses to consider when striving to design
trustworthy and transparent AI systems.120
For laws on algorithmic accountability (against bias), such as
the Algorithmic Accountability Act, reviewing ISO/IEC’s tech-
nical standard being developed on “Bias in AI systems and AI
aided decision making” and IEEE’s P7003 on “Algorithmic Bias
Considerations” could be informative.121 For laws on facial recog-
nition technology, it could be useful to stay up-to-date on ISO’s
standard, JTC 1 SC 37, on Biometrics and IEEE’s P7013 on “Inclu-
sion and Application Standards for Automated Facial Analysis
Technology.”122 For laws directed to promoting transparency, such
as California’s B.O.T. Act, some of the relevant technical standards
that are being developed include IEEE’s P7008, which governs
ethically driven methodologies for autonomous systems that make
overt or hidden suggestions or manipulations for influencing the
behavior or emotions of a user.123

Evaluation & Audit Systems

Implementing an evaluation or audit system for regularly check-

ing the input data and the generated results, then having appropri-
ate feedback channels for further improvements, can provide more
transparency and oversight, while reducing some of AI’s inherent
2020] U.S. AI Regulation Guide 33

risks.124 Organizations can also consider periodically conducting

risk impact assessments in order to “continually identify and review
risks relevant to their technology solutions, mitigate those risks,
and maintain a response plan should mitigation fail.”125 This would
also likely assist the businesses in satisfying the provisions of the
algorithmic accountability laws that require covered entities to
conduct impact assessments on their automated decision systems.
And having third-party algorithm audits can provide even a greater
level of transparency and explainability to the public and regulators.
To the extent feasible, providing general information on whether
and how an AI product is used by the organization can inspire trust
and transparency, which may help comply with legislations focused
on transparency, as well as contribute to building greater confidence
in and acceptance by the end users of the AI products.126

Notes
*  Yoon Chae is an associate in Baker McKenzie’s Dallas office, where he
advises on IP and technology matters. He also served as the firm’s first fellow
at the World Economic Forum’s Centre for the Fourth Industrial Revolution,
where he researched diverse policy issues arising from AI. This article reflects
his personal views and not the positions of the firm. He may be reached at
yoon.chae@bakermckenzie.com.
1.  See The Law Library of Cong., Regulation of Artificial Intel-
ligence in Selected Jurisdictions 1 (2019).
2.  Other examples include South Korea’s Mid- to Long-Term Master
Plan in Preparation for the Intelligent Information Society (December 2016),
the United Kingdom’s National Robotics and Autonomous Systems Strategy
(January 2017) and AI in the UK: ready, willing and able? report (April 2018),
Canada’s Pan-Canadian Artificial Intelligence Strategy (March 2017), France’s
National Artificial Intelligence Research Strategy (March 2017) and National
Strategy for AI (March 2019), China’s National AI Strategy (July 2017), United
Arab Emirates’ Strategy for Artificial Intelligence (October 2017), Germany’s
National Strategy for AI (November 2018), Singapore’s Proposed Model
Artificial Intelligence Governance Framework (January 2019), and Japan’s
AI Strategies 2019 (June 2019), among many others. Non-governmental
organizations have also been active in issuing guidelines and frameworks
on ethical development of AI. See, e.g., IEEE, Ethically Aligned Design
(2017); Microsoft, The Future Computed (2018).
3.  Mina Hanna, We Don’t Need More Guidelines or Frameworks on
Ethical AI Use. It’s Time for Regulatory Action, BRINK News (July 25, 2019),
https://www.brinknews.com/we-dont-need-more-guidelines-or-frameworks
34 The Journal of Robotics, Artificial Intelligence & Law [3:17

-on-ethical-ai-use-its-time-for-regulatory-action; cf. World Econ. Forum,

AI Governance—A Holistic Approach to Implement Ethics into AI
9 (2019) (discussing disadvantages of legislation as regulatory instrument
for AI).
4.  Exec. Office of the President, Artificial Intelligence,
Automation, and the Economy 8 (2016); Exec. Office of the Presi-
dent Nat’l Sci. & Tech. Council Comm. on Tech., Preparing for the
Future of Artificial Intelligence (2016); Networking & Info. Tech.
Research & Dev. Subcomm., Exec. Office of the President Nat’l Sci.
and Tech. Council, The National Artificial Intelligence Research
and Development Strategic Plan (2016).
5.  This article does not cover trends in autonomous driving legislation,
which merit preparation of a separate article. It also does not cover the Filter
Bubble Transparency Act due to timing of the article’s submission.
6.  The chart is not provided as an exhaustive list of laws and bills under
the different categories, but is meant to merely provide a substantive sampling
of the different legislative instruments. A red star on the chart indicates that
the bill has been enacted.
7.  Exec. Order No. 13,859, 3 C.F.R. 3967 [hereinafter “Executive Order”].
8.  Executive Order, supra note 7, at § 1.
9.  Id.
10.  AI Policy—United States, Future of Life, https://futureoflife.org/
ai-policy-united-states/ (last visited Sept. 23, 2019).
11.  Executive Order, supra note 7, at § 2. The other five objectives are:
(a) promote sustained investment in AI R&D; (b) enhance access to high-
quality and fully traceable Federal data, models, and computing resources to
increase the value of such resources for AI R&D, “while maintaining safety,
security, privacy, and confidentiality protections consistent with applicable
laws and policies;” (c) reduce barriers to the use of AI technologies while
protecting American technology, economic and national security, civil liber-
ties, privacy, and values; (d) train the next generation of AI researchers and
users; and (e) develop and implement an action plan, in accordance with the
National Security Presidential Memorandum of February 11, 2019. Id.
12.  Id. at § 6(d).
13.  Nat’l Inst. of Standards & Tech., U.S. Leadership in AI: A Plan
for Federal Engagement in Developing Technical Standards and
Related Tools (2019) [hereinafter “NIST Plan”].
14.  Id. at 12.
15.  Id.
16.  See Brian Higgins, Government Plans to Issue Technical Standards for
Artificial Intelligence Technologies, Artificial Intelligence Technology
and the Law (Feb. 20, 2019), http://aitechnologylaw.com/2019/02/govern
ment-plans-to-issue-technical-standards-governing-artificial-intelligence-
technologies/ (“[F]ederal governance of AI under the Trump Administration
2020] U.S. AI Regulation Guide 35

will favor a policy and standards governance approach over a more onerous
command-and-control-type regulatory agency rulemaking approach leading
to regulations (which the Trump administration often refers to as ‘barriers’)”).
17.  Jory Heckman, NIST Sets AI Ground Rules for Agencies With-
out “Stifling Innovation,” Federal News Network (Aug. 22, 2019,
5:58  pm), https://federalnewsnetwork.com/artificial-intelligence/2019/08/
nist-sets-ai-ground-rules-for-agencies-without-getting-over-prescriptive.
18.  Id.
19.  H.R. Res. 153, 116th Cong. (2019). For an overview of the recent
developments in AI, see Kay Firth-Butterfield & Yoon Chae, Artificial Intel-
ligence Collides with Patent Law, World Economic Forum (2018), http://
www3.weforum.org/docs/WEF_48540_WP_End_of_Innovation_Protect
ing_Patent_Law.pdf.
20.  Id.
21.  Assemb. Con. Res. 215, 2018 Reg. Sess. (Cal. 2018) (enacted).
22.  Id.
23.  Asilomar AI Principles, Future of Life Institute, https://future
oflife.org/ai-principles (last visited Nov. 12, 2019).
24.  Id.
25.  Yoon Chae, Recent Trends in California’s AI Regulation Efforts,
Law360 (July 11, 2019), https://www.law360.com/articles/1176167/
recent-trends-in-california-s-ai-regulation-efforts.
26.  Asilomar AI Principles, supra note 23.
27.  Algorithmic Accountability Act of 2019, S. 1108, H.R. 2231, 116th
Cong. (2019); see also Yoon Chae, supra note 25.
28.  Sneha Revanur, In a Historic Step Toward Safer AI, Democratic
Lawmakers Propose Algorithmic Accountability Act, Medium (Apr. 20,
2019), https://medium.com/@sneharevanur/in-a-historic-step-toward-
safer-ai-democratic-lawmakers-propose-algorithmic-accountability-act-
86c44ef5326d.
29.  Algorithmic Accountability Act of 2019, supra note 27, at § 2(2) and
3(b) (emphases added). The bill would also require the covered entities to
conduct data protection impact assessments on their “high-risk information
systems.” See id. at § 2(6) and 3(b).
30.  Id. at § 3(b)(1)(C).
31.  Id. at § 3(b)(1)(D).
32.  Id. at § 2(1) (emphases added). “High risk” automated decision
systems would include those that: (1) pose a significant risk to the privacy
or security of personal information of consumers, or of resulting in or con-
tributing to inaccurate, unfair, biased, or discriminatory decisions affecting
consumers; (2) make or facilitate human decision-making based on system-
atic and extensive evaluations of consumers (including attempts to analyze/
predict sensitive aspects of their lives) and alter legal rights of consumers or
otherwise affect consumers; (3) involve the personal information of consumers
36 The Journal of Robotics, Artificial Intelligence & Law [3:17

regarding race, religion, health, gender, gender identity, criminal convictions

or arrests, and other factors; (4) monitor public places; and (5) meet any other
criteria established by the FTC. Id. at § 2(7).
33.  Id. at § 2(5).
34.  Id. at § 4 (“Nothing in this Act may be construed to preempt any
State law.”).
35.  Byungkwon Lim et al., A Glimpse into the Potential Future
of AI Regulation, Law360 (Apr. 10, 2019), https://www.law360.com/
articles/1158677/a-glimpse-into-the-potential-future-of-ai-regulation.
36.  See Algorithmic Accountability Act of 2019, supra note 27, at
§ 3(d)‑(e); Byungkwon Lim, supra note 35.
37.  See Byungkwon Lim, supra note 35; Emily J. Tail et al., Proposed
Algorithmic Accountability Act Targets Bias in Artificial Intelligence, JD Supra
(June 27, 2019), https://www.jdsupra.com/legalnews/proposed-algorithmic-
accountability-act-70186 (“The Act is the first federal legislative effort to
regulate AI systems across industries in the United States, and it reflects a
growing and legitimate concern regarding the lawful and ethical implemen-
tation of AI.”).
38.  See Byungkwon Lim, supra note 35.
39.  Jennifer Betts, Keeping an Eye on Artificial Intelligence Regulation
and Legislation, Ogletree, Deakins, Nash, Smoak & Stewart, P.C. (June
17, 2019).
40.  New Jersey Algorithmic Accountability Act, A.B. 5430, 218th Leg.,
2019 Reg. Sess. (N.J. 2019).
41.  Id. at § 3.
42.  Id. at § 2.
43.  See Charlyn Ho et al., New York and New Jersey Make an Early Effort to
Regulate Artificial Intelligence, JD Supra (July 15, 2019), https://www.jdsupra
.com/legalnews/new-york-and-new-jersey-make-an-early-73507.
44.  Ben Kelly & Yoon Chae, Insight: AI Regulations Aim at Eliminating
Bias, Bloomberg Law (May 31, 2019, 4:01 am), https://news.bloomberglaw
.com/tech-and-telecom-law/insight-ai-regulations-aim-at-eliminating-bias.
45.  A Local Law in relation to automated decision systems used by
agencies, Int. No. 1696-2017 (N.Y.C. 2018); see also Ben Kelly & Yoon Chae,
supra note 44.
46.  S.B. 5527, H.B. 1655, 66th Leg., 2019 Reg. Sess. (Wash. 2019); see also
Yoon Chae, supra note 25.
47.  S.B. 444, 2019 Reg. Sess. (Cal. 2019); see also Yoon Chae, supra note 25.
48.  Commercial Facial Recognition Privacy Act of 2019, S. 847, 116th
Cong. (2019).
49.  Makena Kelly, New Facial Recognition Bill Would Require Consent
Before Companies Could Share Data, The Verge (Mar. 14, 2019, 5:40  pm
EDT), https://www.theverge.com/2019/3/14/18266249/facial-recognition-
bill-data-share-consent-senate-commercial-facial-recognition-privacy-act.
2020] U.S. AI Regulation Guide 37

50.  Commercial Facial Recognition Privacy Act, supra note 48, at

§ 3(a)(1).
51.  Id. at § 2(3).
52.  See Jeffrey Neuburger & Daryn Grossman, Bipartisan Facial Recogni-
tion Privacy Bill Introduced in Congress, Proskauer (Mar. 26, 2019).
53.  “[F]acial recognition data” is defined as “any unique attribute or fea-
ture of the face of an end user that is used by facial recognition technology to
assign a unique, persistent identifier or for the unique personal identification
of a specific individual.” Commercial Facial Recognition Privacy Act, supra
note 48, at § 2(6). Likewise, “facial recognition technology” is defined as a
technology that analyzes facial features in still or video images and “is used
to assign a unique, persistent identifier;” or “is used for the unique personal
identification of a specific individual.” Id. at § 2(5).
54.  Id. at § 3(a)(2)-(4).
55.  Id. at § 3(b)(3).
56.  Id. at § 3(c)(1)-(2).
57.  Id. at § 3(d).
58.  Id. at § 3(e)(2) and § 2(9).
59.  Id. at § 3(e)(1)(A).
60.  Id. at § 6.
61.  Id. at § 6-7.
62.  Id. at § 4.
63.  The bill was introduced by Senator Roy Blunt (R-MO) and co-
sponsored by Senator Brian Schatz (D-HI).
64.  See Jeffrey Neuburger & Daryn Grossman, supra note 52.
65.  See Commercial Facial Recognition Privacy Act, supra note 48, at
§ 2(2)-(3).
66.  Teresa Wiltz, Facial Recognition Software Prompts Privacy, Racism
Concerns in Cities and States, Government Technology (Aug. 13, 2019),
https://www.govtech.com/products/Facial-Recognition-Software-Prompts-
Privacy-Racism-Concerns-in-Cities-and-States.html.
67.  H.R. 3875, 116th Cong. (2019).
68.  FACE Protection Act of 2019, H.R. 4021, 116th Cong. (2019).
69.  No Biometric Barriers to Housing Act of 2019, H.R. 4008, 116th
Cong. (2019).
70.  See id. at § 3.
71.  Yoon Chae, supra note 25.
72.  Id.
73.  An Act Establishing a Moratorium on Face Recognition and Other
Remote Biometric Surveillance Systems, S. 1385, 191st Leg. (Mass. 2019);
S. 5687, A.B. 7790, 2019 Reg. Sess. (N.Y. 2019).
74.  S.B. 5528, H.B. 1654, 66th Leg., 2019 Reg. Sess. (Wash. 2019).
75.  See Ben Kelly & Yoon Chae, supra note 44; Katie Lannan, Somerville
Bans Government Use of Facial Recognition Tech, WBUR (June 28, 2019),
38 The Journal of Robotics, Artificial Intelligence & Law [3:17

https://www.wbur.org/bostonomix/2019/06/28/somerville-bans-govern
ment-use-of-facial-recognition-tech. For further discussion of state and local
regulatory development on facial recognition technology, see Gibson Dunn,
Artificial Intelligence and Autonomous Systems Legal Update
(2Q19) 5-6, July 23, 2019.
76.  See Illinois Biometric Information Privacy Act, 740 ILCS 14/1; Tex.
Bus. & Com. Code § 503.001.
77.  Wash. Rev. Code § 19.375 et seq.
78.  CCPA will be codified at Cal. Civ. Code § 1798.100 to 1798.198.
79.  An Act relative to consumer data privacy, S. 120, 191st Leg. (Mass.
2019).
80.  Cal. Bus. & Prof. Code § 17940 et seq.
81.  See Noam Cohen, Will California’s New Bot Law Strengthen
Democracy?, New Yorker (July 2, 2019), https://www.newyorker.com/
tech/annals-of-technology/will-californias-new-bot-law-strengthen-
democracy; Jamie Williams & Jeremy Gillula, Victory! Dangerous Ele-
ments Removed from California’s Bot-Labeling Bill, Electronic Frontier
Foundation (Oct. 5, 2018), https://www.eff.org/deeplinks/2018/10/
victory-dangerous-elements-removed-californias-bot-labeling-bill.
82.  See Cal. Bus. & Prof. Code § 17941(a) (“It shall be unlawful for any
person to use a bot to communicate or interact with another person in Cali-
fornia online, with the intent to mislead the other person about its artificial
identity for the purpose of knowingly deceiving the person about the content
of the communication in order to incentivize a purchase or sale of goods or
services in a commercial transaction or to influence a vote in an election. A
person using a bot shall not be liable under this section if the person discloses
that it is a bot.”).
83.  Id. at § 17940(a).
84.  Id. at § 17941(a)-(b).
85.  Id. at § 17941(b)-(c); see also Harry Valetk & Brian Hengesbaugh,
California Now Has a New Bot Disclosure Law—Will Other States Follow Suit?,
b:INFORM (July 26, 2019), http://www.bakerinform.com/home/2019/7/26/
california-now-has-a-new-bot-disclosure-law.
86.  Cal. Bus. & Prof. Code § 17942(c) (“This chapter does not impose a
duty on service providers of online platforms, including, but not limited to,
Web hosting and Internet service providers.”).
87.  See Harry Valetk & Brian Hengesbaugh, supra note 85 (“Violating
the B.O.T. Act’s disclosure requirements could constitute a violation of Cali-
fornia’s Unfair Competition Act (Cal. Code BPC § 17206), which includes a
civil penalty of up to USD 2,500 for each violation. Accordingly, each ‘like,’
‘share,’ or ‘re-tweet’ performed by a bot without a proper disclosure could
result in a USD 2,500 penalty”).
88.  See Jamie Williams & Jeremy Gillula, supra note 81.
89.  See id.
2020] U.S. AI Regulation Guide 39

90.  Anti-Eavesdropping Act, A.B. 1395, 2019 Reg. Sess. (Cal. 2019),
status available at https://leginfo.legislature.ca.gov/faces/billTextClient
.xhtml?bill_id=201920200AB1395 (last amended in Senate on June 26, 2019).
91.  Id. at § 3 (proposing amendment of Cal. Bus. & Prof. Code
§ 22948.20(a)).
92.  Id. at § 3 (proposing amendment of Cal. Bus. & Prof. Code
§ 22948.20(b)(1)-(3)).
93.  Id. at § 3 (proposing amendment of Cal. Bus. & Prof. Code
§ 22948.20(d)).
94.  Artificial Intelligence Video Interview Act, H.B. 2557, 101st Gen.
Assemb., 2019 Reg. Sess., Public Act 101-0260 (Ill. 2019).
95.  Id. at § 5(1)-(3).
96.  Id. at § 10 and 15.
97.  Theodore F. Claypoole & Dominic Dhil Panakal, Illinois Law
Regulates Use of AI in Video Interviews, The National Law Review (Aug.
15, 2019), https://www.natlawreview.com/article/illinois-law-regulates-use-
ai-video-interviews; see also Justin Steffen & Heather Renée Adams, Hiring
ex Machina: Preparing for Illinois’s Artificial Intelligence Video Interview Act,
Legaltech News (Sept. 23, 2019, 7:00 AM), https://www.law.com/legaltech
news/2019/09/23/hiring-ex-machina-preparing-for-illinoiss-artificial-intel
ligence-video-interview-act (providing that the Act is silent on “whether job
candidates have a private right of action for violations of the Act”).
98.  Theodore F. Claypoole & Dominic Dhil Panakal, supra note 97.
99.  FUTURE of Artificial Intelligence Act of 2017, S. 2217, H.R. 4625,
115th Congress (2017).
100.  Ben Kelly & Yoon Chae, supra note 44.
101.  See GrAITR Act of 2019, H.R. 2202, 116th Congress (2019); AI-IA,
S. 1558, 116th Congress (2019).
102.  GrAITR Act of 2019, supra note 101, at § 101(6)(F); AI-IA, supra
note 101, at § 101(6)(F).
103.  GrAITR Act of 2019, supra note 101, at § 301(b); AI-IA, supra note
101, at § 301(b).
104.  AI in Government Act of 2019, S. 1363, H.R. 2575, 116th Congress
(2019).
105.  Id. at § 3(b)(5) and 4(a)(3).
106.  Personal Data Prot. Comm’n Singapore, A Proposed Model
Artificial Intelligence Governance Framework 3.3 (2019).
107.  Id. at 3.4.1-2.
108.  World Econ. Forum, Draft Guidelines for AI Procurement
12 (2019).
109.  See id. at 13.
110.  AI Fairness 360, IBM Developer (Nov. 14, 2018), https://developer
.ibm.com/open/projects/ai-fairness-360; see also NIST Plan, supra note 13,
at App. III (AI-Related Tools).
40 The Journal of Robotics, Artificial Intelligence & Law [3:17

111.  Matt Turek, Explainable Artificial Intelligence, Defense Advanced

Research Projects Agency, https://www.darpa.mil/program/explainable-
artificial-intelligence (last visited Sept. 25, 2019); see also NIST Plan, supra
note 13, at App. III.
112.  See id.
113.  See Personal Data Prot. Comm’n Singapore, supra note 106,
at 3.16(b).
114.  Id. at 3.16(d)-(e).
115.  Algorithmic Impact Assessment (AIA), Government of Canada,
https://www.canada.ca/en/government/system/digital-government/modern-
emerging-technologies/responsible-use-ai/algorithmic-impact-assessment.
html (last visited Sept. 25, 2019); Algorithmic Accountability Policy Toolkit,
AI Now Institute (Oct. 2018), https://ainowinstitute.org/aap-toolkit.pdf;
see also Personal Data Prot. Comm’n Singapore, supra note 106; World
Econ. Forum, Draft Guidelines for AI Procurement, supra note 108;
IEEE, Ethically Aligned Design, supra note 3.
116.  See Partnership on AI, https://www.partnershiponai.org (last
visited Sept. 25, 2019).
117.  Algorithmic Accountability Act, supra note 27, at § 2(2)(B)(i).
118.  New Jersey Algorithmic Accountability Act, supra note 40, at § 2.
119.  NIST Plan, supra note 13, at App. II (AI Standards).
120.  NIST Plan, supra note 13, at App. II; see also IEEE P7000TM Projects,
Open Community for Ethics in Autonomous and Intelligent Systems,
https://ethicsstandards.org/p7000 (last visited Sept. 24, 2019).
121.  See NIST Plan, supra note 13, at App. II; IEEE P7000TM Projects,
supra note 120.
122.  Id.
123.  Id.
124.  For example, monitoring and reviewing the AI models that have
been deployed “with a view to taking remediation measures where needed”
may be considered. See Personal Data Prot. Comm’n Singapore, supra
note 106, at 3.4.1(c)(ii).
125.  Personal Data Prot. Comm’n Singapore, supra note 106, at 3.9.
126.  See id. at 3.27–29.
 Background of Article

- title of the article (U.S. AI Regulation Guide: Legislative Overview and

Practical Considerations)

- which Journal (The Journal of Robotics, Artificial Intelligence & Law)

- name of authors (Yoon Chae)

1. Abstract

- background information of the article

This article discusses the state of U.S. regulation of artificial intelligence (“AI”),
covering legislative instruments directed to algorithmic accountability, the use of
facial recognition technology and associated data, and promoting “transparency”
when using AI, as well as pending federal bills on general governance or
research issues for AI, among other things. The author also discusses practical
considerations when using AI for businesses and in-house counsel.

- objective(s) of the article

To discuss in depth the regulations in the US that regulates the risks of AI

especially regarding the issue of algorithmic biasness, privacy data and
“transparency”.

- methodology adopted, findings

Qualitative research methodology (based on findings of law and article)

- any principles of law/legislations referred to:

European Parliament’s resolution on Civil Law Rules on Robotics (February

2017), the European Union’s Ethics Guidelines for Trustworthy AI (April 2019),
and OECD’s Council Recommendation on Artificial Intelligence (May 2019).2

2. Three interesting points

1st point (algorithmic accountability)

- what caught your attention?

- Senate and House bills for the Algorithmic Accountability Act were
introduced in Congress.
- The intention behind the bill was to require “companies to regularly
evaluate their tools for accuracy, fairness, bias, and discrimination.”
- It is the first federal legislative effort to regulate AI across industries, with
broad proposed protections that mirror the General Data Protection
Regulation (“GDPR”) in several ways.

- any issues addressed?

- Since AI is usually associated with biased outcomes, this article

addresses the issue of risks or impacts of the system’s design process
and training data on accuracy, fairness, bias, discrimination, privacy, and
security.
- Thus, in order to resolve the issue of bias decisions, it is required to
conduct ‘impact assessments’. The impact assessments should be
performed “with external third parties, including independent auditors and
independent technology experts” when “reasonably” possible.
- “Automated decision system impact assessment” requires an evaluation
of the system’s development process, including the design and training
data, for impacts on “accuracy, fairness, bias, discrimination, privacy, and
security,” and must include “a detailed description of the best practices
used to minimize the risks” and a “cost-benefit analysis,” among other
things.
- The bill also requires the covered entities to work with independent third
parties, record any bias or threat to the security of consumer’s personally
identifiable information discovered through the impact assessments, and
provide any other information that is required by the Director of the
Division of Consumer Affairs in the Department of Law and Public Safety.

- why did you find the points interesting (have they been addressed locally in
Malaysia?
 Because it is imposing a legal obligation for manufacturers to be more cautious
and conscious of their product's algorithm risk on accuracy, fairness, bias,
dis-crimination, privacy, and security instead of just blaming the messed up
algorithms as they remain difficult to track (AI Blackbox phenomenon) when the
systems facing errors.

- did it reflect a more practical approach of regulating AI that could be beneficial to

the legal realm?

- Yes. As the regulation of AI is still in infancy and mostly the law is too
general, this bill is definitely beneficial to the legal realm because it
regulates AI algorithmic accountability with more specific guidelines for
the manufacturers to apply and be accountable for.

2nd point (transparency)

(law made to improve transparency)

1. California’s Bolstering Online Transparency Act

2. California’s Anti-Eavesdropping Act

3. Illinois’s Artificial Intelligence Video Interview Act

- what caught your attention?

- it prohibits the use of “a bot to communicate or interact with another
person in California online, with the intent to mislead the other person
about its artificial identity” for commercial or political purposes.
- it would prohibit manufacturers of smart speakers from installing devices
“without prominently informing” the user. It would also prohibit any
recording or transcription that “qualifies as personal information or is not
deidentified” from being used for advertising purposes, shared or sold to
third parties, or retained without the user’s affirmative consent
- It governs an employer’s ability to use AI analysis on video interviews by
requiring the employer to (1) Notify each applicant before the interview
that AI may be used to analyze the video interview; (2) Provide the
applicant with information on how the AI works; and (3) Obtain consent
from the applicant to be evaluated by the AI program before the interview.
- any issues addressed?

- These laws were made in order to improve transparency in AI to make

sure that none of the information extracted from bot, speakers or anything
that have the kind of feature in it from extracting our information for
commercial and political purposes and also to protect it from being sold to
the 3rd party.

- why did you find the points interesting (have they been addressed locally in
Malaysia?

- These issues have not yet been addressed in malaysia, but it would be
good to apply such law in malaysia since the approach is different
because in malaysia, especially in application such as telegram, we
usually use bot to minimise our search and effort as it do anything on our
command without knowing that they could pretend to be a person to chat
with us and extract information for commercial or political purpose and we
also have this new feature in most of AI product where it able to process,
listen and record what we were saying so these laws will protect our
information from being used or given to anyone for any purposes that we
do not consent to. It is also to protect people from the new norm of
interviews which is via video sent to the related place and people. This
law surely will be useful since the person concerned might use our
information without consent and it is really worrying since it is now the
new norm but we do not have any laws to protect ourselves if any issues
regarding this thing were to happen.

- did it reflect a more practical approach of regulating AI that could be beneficial to

the legal realm?

- Definitely. All of these features namely bot, speakers that could identify
and do according to what we command and extract information from the
analysed video or picture are now being used and applied in lots of
technology in Malaysia. Thus, if Malaysia were to apply such laws, we
could prevent a lot of risk and issues regarding this matter since it
 specifically mention how to apply it and to what extent it should be applied
to and it also covers most of the used feature that is now being used
world widely especially in Malaysia thus this laws will come in handy and
very useful.

3rd point (facial recognition)

- what caught your attention?

- The Commercial Facial Recognition Privacy Act specifically in Section

847 was introduced to strengthen consumer protections and increase
transparency and to make sure “that people are given the information and
the control over how their data is shared” when it comes to facial
recognition.

- any issues addressed?

- ​The bill would generally prohibit ‘covered entities’ from using “facial
recognition technology to collect facial recognition data” of end users
without providing notice and obtaining their consent.
- Covered entities would also be prohibited from using the technology to
discriminate against the consumers, from repurposing the facial data, or
from sharing such data with third parties without obtaining further consent
from the end users.

- why did you find the points interesting (have they been addressed locally in
Malaysia?

- Because the bill provides in detail the definitions of ‘covered entity’ which
include any non-governmental entity that “collects, stores, or processes
facial recognition data such as app operators that collect facial data,
businesses that use the technology on their premises, and outside
vendors that process such data for the original data collectors.
- The bill also requires the covered entities to conduct meaningful human
review before making any final decision based on the output of the
 technology if it can result in a reasonably foreseeable harm or be “highly
offensive” to a reasonable end user.
- Besides, the bill also contains some notable exceptions. The bill exempts
“security applications” that use the technology for “loss prevention” or “to
detect or prevent criminal activity.” The bill also exempts products or
services “designed for personal file management or photo or video sorting
or storage if the facial recognition technology is not used for unique
personal identification of a specific individual,” involves “identification of
public figures for journalistic media created for public interest,” “involves
identification of public figures in copyrighted material,” or is used in an
emergency.

- Did it reflect a more practical approach of regulating AI that could be beneficial to

the legal realm?

- Yes, it does, as the bill contains an extensive explanation on the issues of

facial recognition that usually arose since there have been an increasing
number of bills on facial recognition technology.

- As far as the bill is concerned, Malaysia has not yet introduced any bill
related to facial recognition, specifically regarding users' privacy. If
Malaysia followed the development of the US by implementing the law,
our country would have a proper legal framework on how to tackle any
existing issues arising from facial recognition matters.

3. Concluding Remarks

- Anything that Malaysia can emulate?

- From this article, most of the laws that were discussed in it can surely be
applied and emulated in Malaysia since most of the technology that is
being used in Malaysia currently have the same feature as those
discussed in the article. Hence, it is possible if Malaysia were to
experience such issues and the fact is that such cases already exist but
since there is no law to govern the issues nothing could be done to the
perpetrator as well as the victim. Therefore, by applying such laws we
could prevent any similar risk or issue that has happened before in other
countries from happening to Malaysia. Moreover, with specific and certain
regulations, those who are accountable for such action may be held liable
for any risk or issue coming from the AI.
DATE DOWNLOADED: Tue Jan 4 00:34:38 2022
SOURCE: Content Downloaded from HeinOnline

Citations:

Bluebook 21st ed.

Mirko Forti, The Deployment of Artificial Intelligence Tools in the Health Sector:
Privacy Concerns and Regulatory Answers within the GDPR, 13 EUR. J. LEGAL Stud. 29
(2021).

ALWD 7th ed.

Mirko Forti, The Deployment of Artificial Intelligence Tools in the Health Sector:
Privacy Concerns and Regulatory Answers within the GDPR, 13 Eur. J. Legal Stud. 29
(2021).

APA 7th ed.

Forti, M. (2021). The Deployment of Artificial Intelligence Tools in the Health
Sector: Privacy Concerns and Regulatory Answers within the GDPR. European Journal of
Legal Studies, 13(1), 29-44.

Chicago 17th ed.

Mirko Forti, "The Deployment of Artificial Intelligence Tools in the Health Sector:
Privacy Concerns and Regulatory Answers within the GDPR," European Journal of Legal
Studies 13, no. 1 (Spring 2021): 29-44

McGill Guide 9th ed.

Mirko Forti, "The Deployment of Artificial Intelligence Tools in the Health Sector:
Privacy Concerns and Regulatory Answers within the GDPR" (2021) 13:1 Eur J Legal Stud
29.

AGLC 4th ed.

Mirko Forti, 'The Deployment of Artificial Intelligence Tools in the Health Sector:
Privacy Concerns and Regulatory Answers within the GDPR' (2021) 13 European Journal
of Legal Studies 29.

MLA 8th ed.

Forti, Mirko. "The Deployment of Artificial Intelligence Tools in the Health Sector:
Privacy Concerns and Regulatory Answers within the GDPR." European Journal of Legal
Studies, vol. 13, no. 1, Spring 2021, p. 29-44. HeinOnline.

OSCOLA 4th ed.

Mirko Forti, 'The Deployment of Artificial Intelligence Tools in the Health Sector:
Privacy Concerns and Regulatory Answers within the GDPR' (2021) 13 Eur J Legal Stud
29

-- Your use of this HeinOnline PDF indicates your acceptance of HeinOnline's Terms and
Conditions of the license agreement available at
https://heinonline.org/HOL/License
-- The search text of this PDF is generated from uncorrected OCR text.
-- To obtain permission to use this article beyond the scope of your license, please use:
Copyright Information
 THE DEPLOYMENT OF ARTIFICIAL INTELLIGENCE TOOLS IN THE
HEALTH SECTOR: PRIVACY CONCERNS AND REGULATORY ANSWERS
WITHIN THE GDPR

Mirko Forti Q

This articleexamines the privacy and dataprotection implicationsof the deployment

of machine learningalgorithms in the medical sector Researchersandphysiciansare
developing advancedalgorithmstoforecastpossible developments ofillnesses or disease
statuses, basingtheiranalysis on the processingofa wide range of data sets. Predictive
medicine aims to maximize the effectiveness of disease treatment by taking into
account individual variability in genes, environment, and lifestyle. These kinds of
predictionscouldeventually anticipateapatient'spossiblehealth conditionsyears, and
potentially decades, into the future and become a vital instrument in the future
development of diagnosticmedicine. However the current European dataprotection
legalframework may be incompatiblewith inherentfeatures ofartificialintelligence
algorithms and their constant needfor data and information. This article proposes
possible new approachesand normative solutions to this dilemma.

Keywords: artificial intelligence, algorithm, medicine, health, privacy, data

protection, GDPR

TABLE OF CONTENTS

I. IN TRO DU CTION ............................................................................................... 30

II. THE PRINCIPLES OF FAIRNESS, TRANSPARENCY AND LAWFULNESS
APPLIED TO DATA PROCESSING IN THE MEDICAL FIELD ........................... 33

1. The PrincipleofFairnessin the FunctioningofAI Algorithms ........................ 33

2. The Principleof Transparencyas a Safeguardforthe PrivacyRights of Data
S u bjects ............................................................................................................... . . 35
3. The PrincipleofLawfulness and the Secondary Use of Data............................ 36
III. FREELY-GIVEN, SPECIFIC AND INFORMED CONSENT IN BLACK-BOX
MEDICINE.........................................................................................................37

* Research Fellow - Sant'Anna School of Advanced Studies

-

mirko.forti@santannapisa.it

EJLS 13(1),June 2021, 29-44 doi:10.2 9 2 4 /EJLS.201 9 .o 4 o

{EJLS Online First, 27January 20211
30 EuropeanJournal of Legal Studies 1Vo1.113 No. i

IV. THE RIGHT TO BE FORGOTTEN: HOW CAN AN AI ALGORITHM

FORGET ITS "MEMORY"?................................................................................ 39
V. THE PROHIBITION OF AUTOMATED DECISION-MAKING AND
PROFILING ACTIVITIES UNDER THE GDPR AS A REGULATORY
CHALLENGE FOR THE DEVELOPMENT OF BLACK-BOX MEDICINE............ 39
VI. PRIVACY BY DESIGN AND PRIVACY BY DEFAULT IN THE ERA OF AI
A LG ORITH M S............................................................................................... 42

V II. CONCLUDING REMARKS........................................................................... 43

I. INTRODUCTION

The deployment of Artificial Intelligence (AI) instruments in the medical

sector has led to significant diagnostic innovations. AI tools, due to their
capacity to elaborate vast amonts of data in real-time, can individuate
common patterns undetectable for human physicians.' The global diffusion
of mobile and wearable technology, like smartphones and smartwatches,
enables the collection and uploading of vast amounts of data into AI
algorithms.2 These technological instruments take into account an extensive
array of patients' genetic features to provide tailored medical treatments. A
relatively new speciality of the medical field called predictive medicine 3
involves the processing of genetic and laboratory tests using AI tools to
predict the outbreak of a disease. Thus, the control and ownership of data are
particularly relevant legal concerns for medical care.

The General Data Protection Regulation4 (GDPR, the Regulation) is the

core of the European Union's (EU) approach regarding privacy and data

Charles A.Taylor and others, 'Predictive Medicine: Computational Techniques

in Therapeutic Decision-Making' (1999) 4 Computer Aided Surgery 231.
2 Jason P.Burnham and others, 'Using Wearable Technology to Predict Health
Outcomes: a Literature Review (2018) 25 Journal of the American Medical
Informatics Association 1221.
3 Maxwell Y. Jen and others, 'Predictive Medicine' {2020} StatPearls
<https://www.ncbi.nlm.nih.gov/books/NBK441941/> accessed 15July 2020.
4 Regulation (EU) 2016/679 of the European Parliament and of the Council of 27
April 2016 on the protection of natural persons with regard to the processing of
personal data and on the free movement of such data, and repealing Directive
95 /4 6/EC (GDPR).
2021} Deployment ofArtificialIntelligence Tools in the Health Sector 31

protection. It formulates normative standards with which algorithms must

also comply. However, the GDPR dates back to an era when AI algorithms
did not yet play a fundamental role in everyday life. As a result, central
components of AI tools may raise compliance concerns. Most significantly,
the reasoning routine of AI algorithms is obscure and undetectable due to the
inherent opaqueness of AI tools. An external human observer cannot detect
and recreate the reasoning pattern chosen by the algorithm system, even if
the output delivered by the AI tool is available.s This lack of understanding
and reproducibility, known as the 'black-box' status of AI,6 is incompatible
with the fundamental requirements of transparency, fairness and
accountability enshrined in the GDPR to ensure lawful and legitimate
processing of personal data.

However, opening the black box would mean showing the functioning
mechanism of the algorithm to market competitors, which could stifle
innovation unless more transparent forms of AI became eligible for patent
protection.7 Moreover, making the working processes of this software
detectable would be technically impossible: AI algorithms continuously
change their working routine to follow new patterns and perform tasks in an
ever more efficient way.8 Changing their computing patterns allows them to
produce more reliable diagnostic outcomes. Accordingly, a human observer
would not be able to understand the reasoning process of AI tools and its
relationship to the data processed, even if these were visible. Ultimately, this
lack of interpretability9 can result in a lack of trust in the effective

s Robin C. Feldman and others, 'Artificial Intelligence in the Health Care Space:
How We Can Trust What We Cannot Know', (2019) 30 Stanford Law and Policy
Review 399.
6 Yavar Bathaee, 'The Artificial Intelligence Black Box and the Failure of Intent
and Causation' (2018) 31 Harvard Journal of Law and Technology 889.
7 See Ana Ramalho, 'Patentability of AI-Generated Inventions: is a Reform of the
Patent System Needed?' (2018) <https://papers.ssrn.com/sol3/papers.cfm?
abstract_id=3168 7 03> accessed 19 November 2020.
8 Jenna Burrell, 'How the Machine "Thinks": Understanding Opacity in Machine
Learning Algorithms' (2016)1 Big Data and Society 10.
9 Feldman (n 5); William J. Murdoch and others, 'Definitions, Methods and
Applications in Interpretable Machine Learning' (2019) 116 PNAS 22071.
32 EuropeanJournal of Legal Studies {Vol. 113 No. i

functioning of AI algorithms because researchers and physicians must base

their clinical decisions on the correct functioning of black-box algorithms.

Can the current European legal framework adequately address the main
privacy-related issues that arise from the use of AI software for diagnostic
purposes? More specifically, can the European Regulation foster the
development of predictive medicine and, at the same time, protect the rights
of patients involved in the medical treatments? Starting from previous
scholarship, such as the work of Ann Cavoukian regarding the principle of
privacy by design,10 this article seeks to find normative solutions within the
GDPR to address the deployment of AI tools in the medical sector.
Furthermore, it attempts to find a point of balance between two opposing
interests: on the one hand, the privacy rights of every individual and, on the
other hand, the general interest to stimulate scientific and medical research
and, in more general terms, the right to public health."

The article begins by addressing the most relevant norms of the GDPR to
highlight the privacy-related issues arising from the deployment of AI tools
in the medical sector. More specifically, it focuses on the main legal
uncertainties arising from incompatibilities between the use of AI tools for
predictive medicine and norms like the principles of transparency, fairness
and lawfulness, the issue of free, informed and specific consent, the right to
be forgotten, and the prohibition of automated decisions. The article then
provides a few reflections about the main privacy threats raised by the
development of predictive medicine and how to overcome them. The
inherent opaqueness of AI algorithms may present a challenge for the
transparent and lawful functioning of predictive medicine. However, the
concepts of privacy by design and privacy by default12 could represent the

Ann Cavoukian, 'Privacy by design: the 7 foundational principles.

Implementation and mapping of fair information practices' (Data Protection
Industries) <http://dataprotection.industries/wp-content/uploads/2017/10/
privacy-by-design.pdf> accessed 24 February 2020.
"1 Shane O'Sullivan and others, 'Legal, Regulatory and Ethical Frameworks for
Development of Standards in Artificial Intelligence (AI) and Autonomous
Robotic Surgery' (2019), 15(1) The InternationalJournal of Medical Robotics and
Computer Assisted Surgery r.
12 GDPR, art 25.
2021} Deployment ofArtificialIntelligence Tools in the Health Sector 33

normative basis on which AI algorithms can be made compliant with the

provisions of the GDPR.

II. THE PRINCIPLES OF FAIRNESS, TRANSPARENCY AND LAWFULNESS

APPLIED TO DATA PROCESSING IN THE MEDICAL FIELD

The recent approval and entry into force of the GDPR established new
privacy standards for data protection at a European level.3 While in one
respect the GDPR actually reduces obligations for data controllers regarding
access to clinical data compared to previous legislation, it also limits
utilisation of health data without consent, regulates its secondary use (that is,
use of data for purposes beyond those originally planned),14 and requires data
processing activities in all fields, including predictive medicine, to comply
with certain fundamental principles. The principles of fairness, transparency
and lawfulness are the cornerstones of the current European privacy legal
framework.5 They form the main thread uniting all processing activities and
ensure the protection of the fundamental rights of people involved.

z. The PrincipleofFairnessin the FunctioningofAI Algorithms

The principle of fairness is central to the relationship between the controller

and the data subject,6 and is particularly crucial in the functioning of AI
algorithms. These predictive tools may exacerbate discriminatory trends if
they process prejudicial data. In the healthcare sector this may have fatal

'3 Melanie Bourassa Fourcier and others, 'Integrating artificial intelligence into
health care through data access: can the GDPR act as a beacon for policymakers?'
{2019}Journal of Law and the Bioscences 317.
"4 William Lowrance, 'Learning from the Experience: Privacy and the Secondary
Use of Data in Health Research', (2003), 8Journal of Health Services Research
&

Policy 2.
15 GDPR, art 5.1
16 European Union Agency for Fundamental Rights and Council of Europe,
Handbook on European DataProtection Law (Publications Office of the European
Union 2018) 118.
34 EuropeanJournal of Legal Studies (Vol. 113 No. ii

consequences for patients. Discriminatory factors, such as racel7 or gender,18

could shape the final predictive outcome, implicating the ethical obligation
of non-maleficence9 according to which every medical treatment should
promote patient safety and recovery. 20 Meanwhile, relying on a neutrality"
conception for AI algorithms, where these tools produce standardized
(neutral) outcomes ignoring the peculiar differences between patients, could
be likewise detrimental.22 The challenge is to design AI algorithms capable of
taking into account environmental and societal factors23 and inherent
biological differences to provide fair and reliable health predictive outcomes.
Not all subjects react in the same way as the average model to a specific
medical treatment. In other words, fairness does not mean equality at all
costs. A "fair" algorithm deployed for diagnostic purposes should be aware of
the limitations of model predictions caused by social determinants of health
and biological peculiarities.

Overcoming the issue of biased outputs may require human intervention.

Physicians can reformulate medical questions to reduce bias.24 They can rely
on causal knowledge to verify the algorithmic decision and identify medical
problems where the consequences of datasets biases are relatively negligible;
in other words, reformulating the input to generate a fairer output.

17 Ziad Obermeyer and others, 'Dissecting racial bias in an algorithm used to

manage the health of populations' (2019) 366 Science 447.
I8 Davide Cirillo and others, 'Sex and Gender Differences and Biases in Artificial
Intelligence for Biomedicine and Healthcare' (2020) 3(81) NPJ Digital Medicine
I.

19 Gunnar B.J. Andersson and others, 'Do Not Harm: The Balance of "Beneficence"
and "Non-Maleficence"' (2010) 3 5( 9 S) Spine S2; Vittorio Bufacchi, 'Justice as
Non-Maleficence' (2020) 67(162) Theoria 1.
20 Melissa D McRadden and others, 'Ethical Limitations of Algorithmic Fairness
Solutions in Health Care Machine Learning, (2020) 2 The Lancet Digital Health
E221.
21 Ruha Benjamin, 'Assessing Risk, Automating Racism' (2019) 366(6464) Science
421.
22 McRadden (n 20).
23 Michael Marmot, 'Social Determinants of Health Inequalities' (2005) 365(9464)
The Lancet P1o99.
24 Nanette K.Wenger, 'Cardiovascular Disease: The Female Heart Is Vulnerable. A
Call to Action grom the ioQ Report' (2012) 35 Clinical Cardiology 134.
2021} Deployment ofArtificialIntelligence Tools in the Health Sector 35

Developing guidelines to standardize reporting of predictive models

delivered by AI algorithms can also reduce discrepancies between outcomes
and help validate diagnostic products.25 Thus, physicians can choose in a
transparent way which kind of algorithms they should use according to the
peculiarities of the specific medical case.

2.The Principle of Transparency as a Safeguardfor the Privacy Rights of Data

Subjects
The principle of transparency requires the controller to inform the data
subject about every phase of the processing operation and explain these
phases in a clear and understandable way. Transparency in data processing is
strictly correlated with the principle of purpose limitation. In order for a data
subject to exercise their privacy rights, the individual must know the reasons
for which their data is being collected and processed.26 Processing without a
specific, determined goal is unlawful. However, due to the black-box nature
of AI algorithms, scientists and physicians can neither ex ante inform their
patients regarding every possible outcome of the AI working process nor
forecast possible future uses of data already elaborated.

In the medical sector, the goal is to make the diagnostic routine more user-
centric, protecting the identity rights of the patients. New technological
approaches could help.27 For instance, so-called federated learning (a
machine learning technique to process data through decentralized devices, in
which each server assembles its own dataset) facilitates collaborations across

25 Gary S.Collins and others, 'Transparent Reporting of a Multivariable Prediction

Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD
Statement' (2015) 13 BMC Medicine <https://bmcmedicine.biomedcentral.
com/articles/1o.1186/s12916-014-0241-z> accessed 23 November 2020.
26 Article 29 Working Party, 'Opinion 03/2013 on Purpose Limitation' (2 March
2013) 0o569 /13/EN WP 203 <https://ec.europa.eu/justice/article-29/
documentation/opinion-recommendation/files/2013/wp203_en.pdf> accessed 28
February 2020.
27 Casimir Wierzynski, 'Advancing Both Al and Privacy Is Not a Zero-Sum Game'
(Fortune, 27 December 2018) <https://fortune.com/2o18/12/27/ai-privacy-
innovation-machine-learning/> accessed 21July 2020.
36 European Journal of Legal Studies (Vol. 113 No. ii

multiple institutions without sharing patient data.28 Data controllers can

adjust and improve the effectiveness of their data processing model and then
share these improved algorithms with other subjects through a trusted server,
thereby obtaining a trained AI algorithm without sharing personal data with
third parties. Similarly, the advanced technique of homomorphic encryption
allows algorithms to elaborate data without decoding encrypted
information,29 and thus without identifying the underlying data subject.

3. The PrincipleofLawfulness andthe Secondary Use ofData

The principle of lawfulness requires each data processing procedure to be
grounded on one of six legal bases specified in the GDPR. These include,
inter alia, performing a contract, protecting the vital interest of a person, or
safeguarding public interests. The secondary use of sensitive data is
considered lawful when the processing activities are for scientific purposes, 3
including reasons of public health, but only if the utilised dataset does not
permit identification of any data subject previously involved.3' Using the
same dataset for several purposes is fundamental to the correct development
of scientific research,32 but the strict normative provisions of the GDPR
could discourage scientists and physicians from fully exploiting the research
possibilities intended by the European legal framework.33

28 Micah J.Seller and others, 'Federated Learning in Medicine: Facilitating Multi-

Institutional Collaborations without Sharing Patient Data' (2020) io Scientific
Reports <https://www.nature.com/articles/s41598-02o-69250-1> accessed 19
November 2020.
29 Mohamed Alloghani and others, 'A Systematic Review on the Status and Progress
of Homomorphic Encryption Technologies' (2019) 48 Journal of Information
Security and Applications 1.
30 GDPR, art 89.1 of the GDPR.
31 GDPR, recital 156.
32 Gauthier Chassang, 'The Impact of the EU General Data Protection Regulation
on Scientific Research' (2017) 11 ecancermedicalscience 709.
33 Fourcier (n 13).
2021} Deployment ofArtificialIntelligence Tools in the Health Sector 37

III. FREELY-GIVEN, SPECIFIC AND INFORMED CONSENT IN BLACK-BOX

MEDICINE

Consent is central to data-elaborating activities. The collection of so-called

sensitive data, such as information regarding an individual's health
conditions, is prohibited unless there is room to apply one of the exemptions
listed by the GDPR, including explicit consent. 34 Such consent requires the
free, informed, specific and unambiguous indication of the agreement stated
by data subjects regarding the processing of their personal data. 35 Data
subjects must have a real choice to provide legitimate consent,3 6 and thus
must be aware of specific details regarding the processing activities. These
include the identity of the data controller, the purposes of every operation
for which they gave their consent, the possibility to withdraw consent at any
time, without experiencing technical difficulties,37 and information about the
use of their data for automated decision making, if applicable.3 8 Furthermore,
the data subject should be able to understand every feature and characteristic
of the processing procedures.39 However, the GDPR does not state anything
in terms of competence and capacity of the data subject.4 0 The consequences
of the GDPR provisions about informed consent on the scientific research
context are still a highly debated issue.41

34 GDPR, art 9.
35 GDPR, art.4.11; Mary Donnelly, Maeve McDonagh, 'Health Research, Consent
and the GDPR Exemption' (2019) 26 EuropeanJournal of Health Law 97.
36 European Data Protection Board, 'Guidelines 05/2020 on Consent under
Regulation 2016/679' (4 May 2020) <https://edpb.europa.eu/sites/edpb/files/files/
file1/edpbguidelines_2o2oo5_consent_en.pdf> accessed 20 November 2020.
3 GDPR art 7.
38 European Data Protection Board (n 36).
3 Article 29 Working Party, Guidelines on Transparency under Regulation
2016/679 (11 April 2018) 17/EN WP26o rev.o1 <https://ec.europa.eu/newsroom/
article29/item-detail.cfm?item-id=623o51> accessed 2 March 2020.
40 On the issue of capacity to consent, see Michelle Biros, 'Capacity, Vulnerability,
and Informed Consent for Research' (2018) 46 The Journal of Law, Medicine
&

Ethics 72
41 Chassang (n 32); Niam Clarke and others, 'GDPR: An Impediment to Research?',
(2019) 188 Irish Journal of Medical Science 1129; Miranda Mourby and others,
'Governance of Academic Research Data under the GDPR - Lessons from the
UK' (2019) 9 International Data Privacy Law 192.
38 EuropeanJournal of Legal Studies (Vol. 113 No. ii

Respecting the consent requirements listed by the GDPR could be

problematic in the field of predictive medicine. Firstly, the functioning of AI
algorithms is unintelligible for human observers: Even if there is a clear
outcome from the working process of the system, the reasoning pattern
remains obscure. Thus, the data subject cannot understand how their
personal data are collected and processed: Consent could not be defined as
'informed' as required by the GDPR. Secondly, consent is not even 'specific'
because AI algorithms usually follow adaptive patterns to perform their
interpretative tasks, changing their working routine in light of new
circumstances. It is therefore not possible for data subjects to know all the
specific features of the processing activities when they provide consent. This
also prevents people from giving their approval freely, considering all
potential consequences, or meaningfully, taking into account all possible
variables and suitable alternatives.42

The development of machine learning systems requires a rethinking of the

legal category of consent. It is necessary to transcend the traditional
paradigm of consent, focused on a single specific purpose, to find a new legal
solution compatible with the inherent features of AI working routine. Two
approaches are worth briefly mentioning. Firstly, the broad consent 43 model,
usually applied in the context of biobanks, informs data subjects about the
overall scope and modalities of the data processing activities but not the
specific processes behind these procedures. Secondly, the dynamic consent 44
solution establishes a constant dialogue, through a digital platform, between
data subjects and controllers, allowing patients to understand how their data
is processed in successive diagnostics operations and exercise continuous
control over the processing of their personal data, including by withdrawing
their previous consent.

42 Article 29 Working Party, Guidelines on Consent under Regulation 2016/679 (10

April2018)117/EN WP2 5 9 rev.oi <https://ec.europa.eu/newsroom/article29/item-
detail.cfm?item_id=623o51> accessed 2 March 2020.
43 Mark A. Rothenstein, 'Broad Consent for Future Research: International
Perspectives' (2018) 40(6) Ethics & Human Research 7
44 Jane Kaye and others, 'Dynamic Consent: A Patient Interface For Twenty-First
Century Research Networks' (2014) 23 EuropeanJournal of Human Genetics 141.
2021} Deployment ofArtificialIntelligence Tools in the Health Sector 39

IV. THE RIGHT TO BE FORGOTTEN: HOW CAN AN AI ALGORITHM

FORGET ITS "MEMORY"?

AI algorithms' need to train datasets to improve their processing capabilities

could raise problems with one of the most relevant innovative features
introduced by the GDPR: the so-called right to be forgotten.45 Formulated
by the Court of Justice of the European Union in the famous judgement
Google Spain, it recognizes the data subject's right to obtain from the
controller the erasure of personal data concerning them without undue
delay.46 The right to be forgotten applies to different circumstances
enumerated by the GDPR itself, such as when data are no longer needed for
the original purposes, or when data subjects withdraw their consent. Where
an individual exercises their right to be forgotten, the data controller must
take reasonable measures to erase the data from the public domain, also
removing any links related to them.

From a practical perspective, the inherent technological features of AI

algorithms may complicate the application of the right to be forgotten within
the field of predictive medicine. Physicians feed medical data to the
algorithms to train the computer programmes, which acquire new
information to increase the overall knowledge of algorithms. Thus, the data
to be erased are no longer a separate unit, but part of the AI software
experience. As a result, it would be technologically impossible to extract a
single piece of data without interfering with the reasoning process of the
algorithm. Removing data from the AI system would radically change in
production of outcomes, potentially harming patients.

V. THE PROHIBITION OF AUTOMATED DECISION-MAKING AND

PROFILING ACTIVITIES UNDER THE GDPR AS A REGULATORY
CHALLENGE FOR THE DEVELOPMENT OF BLACK-BOX MEDICINE

AI algorithms can produce outcomes autonomously, or at least with minimal

involvement of human observers. This raises ethical and legal concerns about
safeguarding the fundamental rights of people subject to the action of

45 GDPR, art 17.

46 Case C-1 3 1/12, Google Spain SL, Google Inc. v Agencia Espanolade Proteccionde Datos,
Mario Costeja Gonzalez t20141 ECR 317.
40 EuropeanJournal of Legal Studies {Vol. 13 No. i

automated processing activities. Specifically, the GDPR grants the data

subject the 'right not to be subject to a decision based solely on automated
processing, including profiling, which produces legal effects concerning him
or her or similarly significantly affects him or her'.47 The main goal is to
prevent the outcomes of decision-making algorithms from infringing
people's fundamental rights,48 as machine and computer systems can
formulate decisions based on inaccurate or harmful data sets that yield a
misleading or biased interpretation of reality.49

Under the GDPR, automated decision-making activities based on sensitive

data are unlawful without the prior explicit consent of the data subject except
in cases of overriding public interest.50 On its face, this prohibition purports
to apply only to decisions taken without any human intervention whatsoever.
Since the working routine of AI tools still often requires some sort of external
action, such a restriction would apply onlyvery rarely. Thus, the precise scope
of this prohibition is open to interpretation; perhaps it applies only to
decisions made by the algorithm without a meaningfulhuman involvement.51

The general prohibition of decisions based solely on automated processes

could deter the development of black-box medicine. Humans have only a

47 GDPR, art 22.1

48 Art 22.3 of the GDPR prescribes that the data controller shall provide 'suitable
measures to safeguardthe datasubject's rights andfreedoms and legitimate interests, at
least the right to obtain human intervention on the part of the controller, to express his or
her point of view and to contest the decision'. More specifically, Recital 71 of the
GDPR explains that such safeguards should include 'specific information to the
data subject and the right to obtain human intervention, to express his or her
point of view, to obtain an explanationof the decision reached after such
assessment and to challenge the decision'. Gianclaudio Malgieri, 'Automated
Decision-Making in the EU Member States: The Right to Explanation and
"Suitable Safeguards" in the National Legislations' (2019) 35 Computer Law
&

Security Review 105327.

49 Milena A.Gianfrancesco and others, 'Potential Biases in Machine Learning
Algorithms Using Electronic Health Record Data' (2018) 178 The Journal of
American Medical Association 1544
50 GDPR, art 22.4.
51 Gianclaudio Malgieri and Gianni Comande, 'Why a Right to Legibility of
Automated Decision-Making Exists in the General Data Protection Regulation'
(2017) 7 International Data Privacy Law 243.
2021} Deployment ofArtificialIntelligence Tools in the Health Sector 41

secondary role in black-box medicine. Physicians make treatment decisions

by considering the outcomes produced by the AI algorithms, but cannot
replicate the reasoning process of the machine. Limiting the use of AI
programmes to cases of previous explicit consent or overriding public
interest is too narrow in scope. It is crucial to find an appropriate balance
between the right to privacy and data protection of every individual and the
use of innovative tools to guarantee higher health standards for the entire
community. The challenge is to set boundaries between the right to health
and the protection of personal data.

The GDPR could provide valuable indications to minimize doubt. It already

recognizes that the protection of personal data is not an absolute right, but
rather 'must be considered in relation to its function in society and be
balanced against other fundamental rights, in accordance with the principle
of proportionality'.52 It already sanctions processing activities 'carried out in
the public interest'.53 It is unquestionable that the right to public health is an
issue of public interest. Nonetheless, fundamental safeguards to privacy
rights must be preserved even in the functioning of AI algorithms for
healthcare.

Several proactive and preventive data protection measures in AI algorithms

could adequately safeguard the privacy rights of patients involved in
diagnostic treatments. Firstly, a counterfactual explanation model could
allow individuals to better understand the reasoning process behind a
predictive outcome. 54 This would explain what factors would need to change
to obtain a different result, permitting scientists and physicians to
understand the relationship between processed data and the above-
mentioned principles of data processing. Secondly, a co-governance system
of algorithms based on a multi-level design could allow humans to intervene
in the reasoning patterns to ensure the respect of fundamental rights of the

52 GDPR, recit 4.
53 GDPR, art 6.4.
54 Sandra Wachter and others, 'Counterfactual Explanations Without Opening the
Black Box' (2018) 31 Harvard Journal of Law & Technology 841.
42 EuropeanJournal of Legal Studies W01.113 No. i

patients involved.55 Thirdly, the so-called 'agonistic machine learning'5 6

approach, where AI providers should formulate alternative ways of modelling
and describing the same object, could provide new diagnostic patterns
compliant with the fundamental rights framework. AI algorithms usually rely
on machine-readable information about what is the 'truth': for instance, in
the health care sector, medical exams or diagnoses. Providing different inputs
from several sources would help algorithms overcome possible biases in the
datasets and produce more reliable outcomes. This may lead to a more
accountable and transparent decision-system, complying with the data
protection framework.

VI. PRIVACY BY DESIGN AND PRIVACY BY DEFAULT IN THE ERA OF Al

ALGORITHMS

The GDPR requires compliance with the principle of data minimisation,57

whereby controllers must process only the necessary amount of information.
The principle of privacy by design, developed by Ann Cavoukian,58 is a
proactive approach to data minimisation, integrating privacy measures into
the hardware and software utilised in data processing upon their creation to
ensure that only the necessary amount of data is processed. Privacy by design
requires data controllers to implement adequate privacy safeguards, such as
pseudonymisation, from the first phases of the processing activities. 59

This principle could conflict with the inherent nature of AI algorithms. AI

tools constantly need data to train their working routine to perform their
diagnostic tasks more efficiently. o Furthermore, algorithms adapt to
constantly changing environments; maintaining the same data protection
features may be technically impossible, though technical approaches like

5 Margot E. Kaminski, 'The Right to Explanation, Explained' (2019) 34 Berkeley

Technology LawJournal 189.
56 Mireille Hildebrandt, 'Privacy as Protection of the Incomputable Self: From
Agnostic to Agonistic Machine Learning', (2019) 19 Theoretical Inequalities of
Law 83.
5 GDPR, art 5.
58 Cavoukian (n Io).
5 GDPR, art 25.
6o European Parliamentary Research Service, The Impact of the General Data
Protection Regulation (GDPR) on ArtificialIntelligence (European Union 2020).
2021} Deployment ofArtificialIntelligence Tools in the Health Sector 43

counterfactual explanations, co-governance systems of algorithms or

agonistic machine learning could help bridge the gap.

Ultimately, the working of machine learning tools should be considered

compliant with the GDPR provisions to ensure a normative safeguard for the
rights of data subjects against the risk of obsolescence of the Regulation. The
privacy by design principle could play a fundamental role to avoid this kind of
risk and keep the pace of technological progress. This proactive and
preventive approach would make the user-the patient in the medical
setting-the focus of the entire data processing activity. Embedding privacy
issues in the construction of AI algorithms would also help to keep track of
the reasoning patterns chosen to produce a specific output. Privacy by design
would encourage dialogue between AI providers, scientists and privacy
advocates to build privacy-compliant AI algorithms that could help
physicians and scientists manage the risks related to processing health data,
taking into account the privacy rights of people involved.

VII. CONCLUDING REMARKS

In a time of new approaches to data protection, the GDPR remains the 'gold
standard' in the European framework. However, the GDPR reflects an era
when AI had not yet reached the current levels of technological development
and, more specifically, the field of predictive medicine was in its infancy. As
a result, the Regulation is not fully compatible with the inherent features of
machine learning tools. The resulting legal uncertainty could osbstruct the
development of AI technologies, increasing costs and reducing benefits for
users and patients.6 The EU must act to bridge this gap and fully regulate the
use of AI tools in everyday life, including the medical sector, and achieve a
uniform and coherent policy approach regarding AI matters. Encouragingly,
the European Commission has acknowledged the threats to privacy posed by
machine learning applications and cleared the way for adjusting relevant EU
legislative frameworks. 62

61 Ibid.
62 European Commission, 'White Paper on Artificial Intelligence - A European
Approach to Excellence and Trust' COM (2020) 65 final.
44 EuropeanJournal of Legal Studies {Vol. 113 No. i

GDPR norms are often vague and undefined,6 3 which may be a normative
choice to keep pace with the ongoing technological progress. Specific
mandatory requirements aimed at AI tools may become rapidly obsolete. It
is necessary instead to create a trustworthy environment for developing AI
applications for healthcare purposes with patient privacy rights in mind. The
GDPR already provides valuable instruments, such as the privacy by design
principle, 64 that could help reach this goal. Embedding data protection
features in diagnostics routines would help overcome the black-box barrier
of algorithms. Software providers would train their AI tools to respect
privacy rules from the very first phases of their working patterns, securing
lifecycle protection for the user during the entire duration of data processing
activities. The patient would become the focus of the diagnostic process.
Scientists and physicians would coordinate the work of AI algorithms.
Humans would control the entire process; not machines. This would ensure
the full respect of human rights of every individual involved, resolving the
tension between the privacy rights of the individual and the public health
needs of society.

63 Ibid.
64 Cavoukian (n io).
TUTORIAL WEEK 8 - ARTICLE READING

1. Background of Article
● TITLE
- THE DEPLOYMENT OF ARTIFICIAL INTELLIGENCE TOOLS IN THE HEALTH
SECTOR: PRIVACY CONCERNS AND REGULATORY ANSWERS WITHIN
THE GDPR

● JOURNAL
- European Journal of Legal Studies (Vol 13, No 1)
HEINONLINE

● NAME OF AUTHOR
- Mirko Forti

2. Abstract
● Background Information
- This article aims to examine the privacy and data security issues of
deploying machine learning algorithms in the medical sector.
● Objective
- Addressing the issue of privacy-related arising from deployment of AI tools
in medical sector
- Focusing on main legal uncertainties arising from incompatibilities
between the use of AI tools for predictive medicine and norms such as
(principles of transparency, fairness and lawfulness, issue of free,
informed and specific consent, right to be forgotten, and prohibition of
automated decision)
- Provide reflections about the main privacy threats raised by the
development of predictive medicine and how to overcome them.

● Methodology - Doctrinal Research Methodology in which the author discusses

the concerns on AI and Privacy with GDPR.

1
 ● Principles of law/legislations -
- General Data Protection Regulation (GDPR)
- Concerning privacy and data protection

3. Three Interesting Points

Point 1
● The first thing that caught our attention is concerning the issue of the principle of
transparency as a safeguard for the privacy rights of data subjects. This principle
requires the controller to inform the data subject about every phase of processing
operation and then explain those phases in a clear understandable way to them.
● The data subject could exercise their privacy rights by knowing the reasons for
which their data is being collected and processed. If any data is processed
without a specific and determined goal, it would be unlawful.
● However, the issue arises when scientists and physicians can neither ex ante
(based on forecast rather than actual result) inform their patients regarding every
possible outcome of the AI working process nor forecast possible future uses of
data already elaborated. This is caused by the black-box nature of AI algorithms.
● So how can this issue be addressed? In the medical sector, the essential goal is
to make the diagnostic routine more user-centric which is by protecting the
identity rights of patients. Thus, using new technological approaches could help
cater this issue. For example, the so-called federated learning (a machine
learning technique to process data through decentralized devices, in which each
server assembles its own dataset) facilitates collaborations across multiple
institutions without sharing patient data.
● Meaning that, the data controllers can adjust and improve the effectiveness of
their data processing model and then share these improved algorithms without
sharing personal data with third parties. Only then, the privacy rights of the data
subject could be safeguarded.
● Malaysia undoubtedly has adopted the use of AI technology in the healthcare
scene. In mid- 2020, Huawei Malaysia collaborated with the Malaysia Ministry of
Health to contribute Huawei Cloud AI-assisted diagnosis solution to a COVID-19

2
 hospital in Malaysia in order to provide faster COVID-19 diagnosis.
● The question is whether Malaysia’s use of AI tools in the medical sector has
addressed the issue of principle of transparency as discussed above? The
answer is yes. The World Health Organization (WHO) has been prompted to
issue a formal guiding principles for proper design and usage of AI in the
healthcare industry.
● There are 6 guiding principles to the basis for AI regulation and governance for
its member countries. One of the principles or known as Principle 3 concerns
about ensuring transparency, explainability, and intelligibility. It is stated that
“transparency requires that sufficient information be published or documented
before the design or deployment of an AI technology. Such information must be
easily accessible and facilitate meaningful public consultation and debate on how
the technology is designed and how it should or should not be used”
● Moreover, the issue of protection of health data in Malaysia is also primarily
governed by the Personal Data Protection Act 2010 (PDPA 2010) and its
subsidiary legislation. The Act is designed to protect personal data by imposing
certain requirements on data users and imposing certain rights on the data
subject in relation to his personal data.
● It is worth noting that the PDPA 2010 does not include a defined definition for the
terms "consent" or "explicit consent," nor does it provide any minimum standard
or formalities in terms of the consent that must be obtained. However, data users
are reminded that they must preserve a record of data subjects' consents.
Obviously, the requirement of agreement imposed by the PDPA 2010 on health
data demonstrates that our present legislation is consistent with the WHO's AI
Principle 1 which read “Protecting human autonomy: In the context of health
care, this means that humans should remain in control of health-care systems
and medical decisions; privacy and confidentiality should be protected, and
patients must give valid informed consent through appropriate legal frameworks
for data protection.

3
Point 2
● The other thing that caught our attention in this article is regarding The right to be
forgotten on ‘How can an AI algorithm forgets its memory.’
● Article 17 of the GDPR provides a right to be forgotten. The so-called right to be
forgotten is formulated by the Court of Justice of the EU in its judgment of Google
Spain case.
● The right to be forgotten applies to different circumstances such as when the
data are no longer needed for the original purposes or when data subjects
withdraw their consent. For example, a patient wants his medical data to be
erased thus the data controller must erase the data from the AI public domain
also removing any links related to them.
● Technically , AI algorithms need to be fed and trained with a huge amount of data
to improve their processing capabilities to produce a better outcome. For
example Physicians feed medical data to the algorithms to train the AI, which
acquire new information to increase the overall knowledge of algorithms.
● I would say regarding provision stipulated under Article 17 of the GDPR, which is
right to be forgotten, is something good and inline with human rights to secure
their privacy rights and data. However, in regard to AI systems it would radically
affect their algorithm in producing outcome, because AI needs this kind of
dataset to produce a better outcome decision and without this data it may
potentially harm patients.

Point 3
● Even though the GDPR was introduced, there is still a gap that needs to be
fulfilled where the regulation is not fully compatible with the advancement
inherent features of various machine learning tools. Therefore, the EU needs to
update and ensure the uniformity of the current law in order for it to be in line with
the development of technology as well as provide protection for the users.
● GDPR norms are often vague and undefined, which may be a normative choice
where it is inconsistent to keep pace with the ongoing technological progress.
● It is important to ensure in the healthcare system in which besides advancement

4
 of AI for the healthcare purposes, it can still respect human privacy.

This article will be beneficial to the legal realm as it can help as a guideline to resolve
the issue between the protection of the human rights which is on protection of the data
privacy as well as to fulfill the needs of the society. It also can help to ensure the gaps
can be fulfilled.

4. Concluding Remarks – anything that Malaysia can emulate?

● Malaysia needs to ensure the current related laws are updated especially in the
healthcare system to ensure the patients data are being protected as in the
future there will be various AI technology systems that will be introduced.
● The regulation to regulate AI must be compatible with the inherent features of
machine learning tools. As in the Article it is stated that GDPR reflects an era
when AI had not yet reached the current levels of technological development,
hence the incompatibility.
● Therefore, Malaysian Medical Device Act 2012 (MDA 2012) and Consumer
Protection Act (CPA) 1999 and Personal Data Protection Act 2010 (PDPA 2010)
need to be updated in order for it to be compatible with current AI technological
development and there should be a specific requirements for AI being introduced
in order to create a trustworthy environment.

5
DATE DOWNLOADED: Mon Jan 3 09:38:20 2022
SOURCE: Content Downloaded from HeinOnline

Citations:

Bluebook 21st ed.

Matthew Humerick, Taking AI Personally: How the E.U. Must Learn to Balance the
Interests of Personal Data Privacy & Artificial Intelligence, 34 Santa CLARA HIGH
TECH. L. J. 393 (2018).

ALWD 7th ed.

Matthew Humerick, Taking AI Personally: How the E.U. Must Learn to Balance the
Interests of Personal Data Privacy & Artificial Intelligence, 34 Santa Clara High
Tech. L. J. 393 (2018).

APA 7th ed.

Humerick, M. (2018). Taking AI Personally: How the E.U. Must Learn to Balance the
Interests of Personal Data Privacy & Artificial Intelligence. Santa Clara High
Technology Law Journal, 34(4), 393-418.

Chicago 17th ed.

Matthew Humerick, "Taking AI Personally: How the E.U. Must Learn to Balance the
Interests of Personal Data Privacy & Artificial Intelligence," Santa Clara High
Technology Law Journal 34, no. 4 (May 2018): 393-418

McGill Guide 9th ed.

Matthew Humerick, "Taking AI Personally: How the E.U. Must Learn to Balance the
Interests of Personal Data Privacy & Artificial Intelligence" (2018) 34:4 Santa Clara
High Tech L J 393.

AGLC 4th ed.

Matthew Humerick, 'Taking AI Personally: How the E.U. Must Learn to Balance the
Interests of Personal Data Privacy & Artificial Intelligence' (2018) 34 Santa Clara
High Technology Law Journal 393.

MLA 8th ed.

Humerick, Matthew. "Taking AI Personally: How the E.U. Must Learn to Balance the
Interests of Personal Data Privacy & Artificial Intelligence." Santa Clara High
Technology Law Journal, vol. 34, no. 4, May 2018, p. 393-418. HeinOnline.

OSCOLA 4th ed.

Matthew Humerick, 'Taking AI Personally: How the E.U. Must Learn to Balance the
Interests of Personal Data Privacy & Artificial Intelligence' (2018) 34 Santa Clara
High Tech L J 393

-- Your use of this HeinOnline PDF indicates your acceptance of HeinOnline's Terms and
Conditions of the license agreement available at
https://heinonline.org/HOL/License
-- The search text of this PDF is generated from uncorrected OCR text.
-- To obtain permission to use this article beyond the scope of your license, please use:
Copyright Information
 TAKING Al PERSONALLY: HOW THE E.U. MUST LEARN
TO BALANCE THE INTERESTS OF PERSONAL DATA
PRIVACY & ARTIFICIAL INTELLIGENCE

Matthew Humerickt

The race to develop artificial intelligence has begun, where

countries are heavily backing efforts to be the world leader. While this
technology promises to create a smarter, autonomous world, it is not
without its concerns. Perhaps most prevalent are concerns regarding
consumer personal data privacy and protection. While nations
worldwide have adopted varying degrees ofpersonal dataprotection,
the European Union has establisheditselfas the leader on this front.
Soon, the European Union will implement its most comprehensive
regulationyet on consumerpersonal dataprivacy and protection: the
General Data Protection Regulation ("GDPR"). However, several
aspects of this Regulation pose concerns as to the impact of its
enforcement on the algorithms and machine learningrequiredfor the
development of artificial intelligence. Until these concerns are
addressed, it remains to be seen whether artificial intelligence can be
developed in the European Union in compliance with the new GDPR's
provisions.

TMatthew Humerick holds a J.D. from Michigan State University College of Law (class of 2018)
and will sit for the Massachusetts Bar in July of 2018.

393
394 SANTA CLARA HIGH TECH. L.J. [Vol. 34

TABLE OF CONTENTS

INTRODUCTION ....................................... 395

. . . . . . . . . . . . ... . . .
I. WHAT IS ARTIFICIAL INTELLIGENCE? 396
A. Big Data, Machine Learning, and Al.................... 397
B. Al, its Expected Development, and the E. U...................... 398
II. E.U. DATA PRIVACY & AI LAw........... ............. 401
A. The Scope of the GeneralDataProtectionRegulation
("GDPR") ........................................... 402
1. Right to Consent. ....................... ..... 405
2. Right to be Forgotten (Erasure) ...................... 407
3. Right to Data Portability.......... .............. 408
4. Right to Explanation. ......................... 410
B. E. U's AI Law Efforts................. 413
III. WHERE DOES THE E.U. GO FROM HERE? ................. 414
A. AI as a Carve-Out of GDPR ................ ..... 415
B. E. U Funding ofAlternative AI Models ..... ........ 416
CONCLUSION ................................ ......... 418
2018] TAKING Al PERSONALLY 395

INTRODUCTION

Technology is continuously evolving to create a smart,

autonomous world. At the forefront of this technological revolution is
the innovation of artificial intelligence ("Al"). As stated by Russian
President Vladimir Putin, "[w]hoever becomes the leader in [artificial
intelligence] will become the ruler of the world."' President Putin's
words express the breadth of concern to which many have over the
rapid expansion of this sort of super-intelligence. 2 However, whether
or not Al is a concern, its exponential development and use may soon
stall as the European Union ("E.U.") prepares to implement its General
Data Protection Regulation ("GDPR") on May 25, 2018.3
While Al is subject to different definitions, it is generally
understood to consist of machine learning, based on algorithms that
collect, process, and adapt to data from the real world.4 Al cannot thrive
without a steady supply of data to expand its knowledge base. To
supplement its development, controllers collect vast amounts of
consumer personal data to enable algorithms to learn. Algorithms
cannot accurately learn from its environment without large amounts of
personal data. Instead, companies collect, store, process, and maintain
large sets of consumer data. As a result, data privacy and protection
have become cause for greater concerns for companies and
governments alike. Enter the E.U.'s GDPR.
The GDPR emphasizes consumer control over personally
identifiable information ("PII"), which creates stricter legal and
operational obstacles for those seeking to control and process it.5 The

1. David Meyer, Vladimir Putin Says Whoever Leads in Artificial Intelligence Will Rule
the World, FORTUNE (Sept. 4, 2017), http://bit.do/Meyer Putin (citing a recent quote by Vladimir
Putin when discussing the concept of artificial intelligence and its impact on the world).
2. See Maureen Dowd, Elon Musk's Billion-Dollar Crusade to Stop the AI Apocalypse,
VANITY FAIR (Apr. 2017), http://bit.do/DowdElon-Musk. Elon Musk, arguably one of the
greatest innovators of the 21st Century, has openly combatted the rise of AI along with only
highly-regarded minds, such as Stephen Hawking and Bill Gates. Id. This article discusses Musk's
concerns over the development of AI and how its ability to learn from humans, only to outthink
them, casts a grim outlook on humanity's future. Id.
3. GDPR Portal: Site Overview, EU GDPR, http://bit.do/EUGDPR (last visited Oct. 2,
2017) [hereinafter GDPR Portal].
4. See Jordan Novet, Everyone Keeps Talking About AI - Here's What It Really Is and
Why It's So Hot Now, CNBC (June 17, 2017), http://bit.do/Novet AI (citing the first scholarly
definition of AI, as defined in 1956 by John McCarthy, a math professor at Dartmouth College,
who defined AI as "every aspect of learning or any other feature of intelligence [that] can in
principle be so precisely described that a machine can be made to simulate it"). Nowadays, AI
has developed to include terms, such as "machine learning" and "deep learning," which describe
the complexity of technological processes used to collect and utilize data. Id.
5. See Guide to the General Data Protection Regulation (GDPR), INFO. COMM'R'S
OFFICE, http://bit.do/ICO_Overview (last visited Oct. 4, 2017).
396 SANTA CLARA HIGH TECH. L.J. [Vol. 34

GDPR codifies several E.U. consumer rights in PII that cannot coexist
with Al, including: the requirement of explicit consent, the right of
erasure, the right to explanation of automated decisions, and data
portability rights.6 While these aforementioned rights are only afforded
to E.U. citizens, the territorial scope of the GDPR applies to those
processing this protected data, wherever they may be located or
established. As a result, business leaders and legislatures across the
globe must address these compliance issues to determine whether they
can lawfully sustain Al development under the GDPR. To remain
competitive within the Al field, the E.U. must find a way to balance its
interest in data privacy against those in the advancement of Al.
Part I of this Comment provides an overview of what Al is and
how it utilizes personal data to develop. Contained within Part I is also
a brief overview of the current international Al situation and how the
E.U.'s relevance is rapidly declining. Part II discusses data privacy and
Al law within the European Union, and how new regulations may
adversely impact sustained algorithmic development under the current
Al model. Part III proposes two courses of action on how the E.U.
should adapt its views on data privacy and protection to better facilitate
the use and development of Al.

I. WHAT IS ARTIFICIAL INTELLIGENCE?

At its most basic level, Al is "a system that can learn how to
learn."" Humans write initial algorithms for a system that enables the
computer to subsequently write its own algorithms, without additional
human oversight or interaction. 9 This process allows Al to
continuously learn from, and solve new problems within, an ever-
changing environment, based on its continuing collection of data.'o
While Al may seem straightforward by this definition, there are

6. See Bernard Marr, New Report: Revealing the Secrets of AI or Killing Machine
Learning?, FORBES (Jan. 12, 2017), http://bit.do/Marr New-Report; see also Rand Hindi, Will
Artificial Intelligence Be Illegal in Europe Next Year?, ENTREPRENEUR (Aug. 9, 2017),
http://bit.do/Hindi Al-illegal.
7. See Regulation 2016/679, of the European Parliament and of the Council of 27 April
2016 on the protection of natural persons with regard to the processing of personal data and on
the free movement of such data, and repealing Directive 95/46/EC (General Data Protection
Regulation), arts. 1-3, 2016 O.J. (L 119) 32, 33 [hereinafter GDPR].
8. FRANCESCO COREA, ARTIFICIAL INTELLIGENCE AND EXPONENTIAL TECHNOLOGIES:
BUSINESS MODELS EVOLUTION AND NEW INVESTMENT OPPORTUNITIES 1-2 (2017).
9. Id. at 2.
10. Id. "While human being actions proceed from observing the physical world and
deriving underlying relationships that link cause and effect in natural phenomena, an artificial
intelligence is moved entirely by data and has no prior knowledge of the nature of the relationship
among those data." Id.
2018] TAKING Al PERSONALLY 397

numerous definitions and types of Al. Additionally, because of its

continuously learning nature, Al is expected to develop tremendously
over the next decade, making the impact of the GDPR even more
significant.
A. Big Data, Machine Learning, andAl
Big data supplies the basis for Al by supplying the environment
from which it is able to learn. While, like Al, no single definition exists,
companies and organizations commonly define "big data" as "high-
volume, high-velocity and high-variety information assets that demand
cost-effective, innovative forms of information processing for
enhanced insight and decision making."" Under this construction, its
three V's conceptualize big data, where "volume relates to massive
datasets, velocity relates to real-time data and variety relates to
different sources of data."' 2 This concentration of data includes an
abundance of information, ranging from PII to anonymous data
collections, such as Internet of Things devices, 3 machine logs, or
company reference data collections.' This supply of big data is
instrumental to Al's machine learning.
Intel, a leading company in Al development, defines machine
learning as "the set of techniques and tools that allow computers to
'think' by creating mathematical algorithms based on accumulated
data."" While this is a broad definition, machine learning functions
through complex means. For example, there are two broad categories

11. Big Data, GARTNER: IT GLOSSARY, http://bit.do/Gartner Big-Data (last visited Apr.
28, 2018).
12. See Big Data, ArtificialIntelligence, Machine Learning, and Data Protection, INFO.
COMM'R'S OFFICE, 1, 6 [hereinafter Big Data]. But see Sean Jackson, Big Data in Big Numbers
- It's Time to Forget the "Three V's" and Look at Real-World Figures, COMPUTING (Feb. 18,
2016), http://bit.do/Jackson_3Vs (redefining "big data" in away that emphasizes its presence here
and now, rather than as a number or size).
13. See Jacob Morgan, A Simple Explanation of 'The Internet of Things', FORBES (May
13, 2014), http://bit.do/Morgan Simple-Explanation ("Simply put, this is the concept of basically
connecting any device with an on and off switch to the Internet (and/or to each other). This
includes everything from cellphones, coffee makers, washing machines, headphones, lamps,
wearable devices and almost anything else you can think of. This also applies to components of
machines, for example ajet engine of an airplane or the drill of an oil rig.").
14. See Ian Murphy, Could AILead to Breaches of GDPR?, ENTERPRISE TIMES (June 21,
2017), http://bit.do/Murphy_Al.
15. See Deb Miller Landau, ArtificialIntelligenceandMachineLearning: How Computers
Learn, IQ (Aug. 17, 2016), http://bit.do/Landau_Al. Through machine learning, Al undergoes a
three-step process: "[S]tep one is perceiving the world, using data to detect patterns. Step two is
recognizing those patterns, and step three is taking an action based on that recognition." Id. See
generallyETHEM ALPAYDIN, INTRODUCTION TO MACHINE LEARNINGpassim (Thomas Dietterich
et al. eds., 2d ed. 2010).
398 SANTA CLARA HIGH TECH. L.J. [Vol. 34

of machine learning: supervised and unsupervised." With supervised

learning, the Al processes and learns from labeled data sets to develop
algorithms." This supervised approach "trains" the algorithms to
create models that can accurately map data inputs to outputs, which
allows the algorithms to predict future events.'" With supervised
learning, it is easier for programmers and analysts to oversee and
observe the Al's development. This additional control allows those
overseeing the Al development to more easily follow its logic and
introduce new data sets necessary for its continual processing.
Conversely, unsupervised learning supplies the algorithms with no
labels or prior input-output relationships, and instead leaves the
algorithms on its own to leam.' 9 Machine learning is also categorized
based on the depth of its learning.
The depth of machine learning is either shallow or deep.
Typically, shallow learning is less utilized because it only involves one
layer of data, which limits the amount of data that Al can use to expand
its knowledge. 2 0 Alternatively, deep learning involves the use of
algorithms, modeled after the human brain, to create multiple layers of
neural networks. 2 ' Such neural networks allow data to be clustered,
classified, and recognized as larger patterns, which can then be
modeled to predict future tendencies and events.22 This simplistic
overview illustrates how Al incorporates vast amounts of data to teach
its algorithms to learn in a way that builds connections between
seemingly unrelated data.

B. Al, its Expected Development, and the E. U

Because many countries share Vladimir Putin's sentiment

towards Al, 23 experts predict rapid expansion in Al research and
development over the next decade. Currently, the leaders in this field

16. See also Big Data, supra note 12, at 7.

17. Id.
18. Id. at 8.
19. Id.
20. See Jurgen Schmidhuber, Deep Learning in Neural Networks: An Overview, 61
NEURAL NETWORKS 85, 86-87 (2015).
21. See id.; COREA, supra note 8, at 12-13. See also id. at 13 ("There exist many types of
ANNs [artificial neural networks] ... but the most known ones are Recurrent Neural Networks
(RNNs); Convolutional Neural Networks (CNNs); and Biological Neural Networks (BNNs));
Introduction to Deep Neural Networks, DEEPLEARNING4J, http://bit.do/DeepLeaming4JIntro
(last visited Oct. 13, 2017).
22. See sources cited supra note 21.
23. See Dave Gershgom, AI is the New Space Race. Here's What the Biggest Countries
are Doing, QUARTZ (May 2, 2018), http://bit.do/Greshgom_ A; Meyer, supra note 1.
2018] TAKING Al PERSONALLY 399

are the United States, China, and India.24 In what is becoming a

technical arms race, these three leaders have taken different approaches
to developing Al: the United States relies heavily on the efforts of
individual companies; China's government funds Al research; and,
India relies on its work through its $143 billion outsourcing industry.25
These leaders share similar aspects with regard to balancing interests
in Al and data privacy. In each country, the respective government
facilitates Al advancement by either encouraging, or minimally
interfering with, its development. Additionally, personal data is either
not overly regulated or, the use of personal data for Al purposes is
separate and distinct from other data privacy and protection
regulations.
While the E.U. attempts to assert itself as a leading Al entity,2 6 the
United Kingdom (U.K.) dominates the region's research and
development.27 Although Europe is currently a pioneer in the
advancement of Al, it is largely because the U.K. makes up the
majority of its efforts. 28 However, in a recent move commonly known
as "Brexit,"29 the U.K. voted to leave the E.U., thus potentially
affecting the E.U.'s status in the Al field.30 While there were numerous

24. See Rishi lyengar, These Three Countries are Winning the Global Robot Race, CNN
(Aug. 21, 2017), http://bit.do/Iyengar 3-countries.
25. Id. China's government is perhaps the largest proponent of AI development, as
demonstrated by its plans to become the world leader of AI in 2030 through a $150 billion-dollar
plan. Id. See also Sherisse Pham, China Wants to Build a $150 Billion AlIndustry, CNN (July 21,
2017), http://bit.do/PhamChina.
26. See European Commission Press Release IP/18/3362, Artificial intelligence:
Commission outlines a European approach to boost investment and set ethical guidelines (Apr.
25, 2018) (detailing a recent collaborative effort amongst E.U. Member States to invest "at least
t20 billion [in AI investments and research] by the end of 2020"); see also Tania Rabesandratana,
With e1.5 Billion for Artificial Intelligence Research, Europe PinsHopes on Ethics, SCI. (Apr.
25, 2018, 12:35 PM), http://bit.do/Rabesandratana (commenting on the E.U.'s recent plans to
invest in Al and how various issues remain unaddressed).
27. See Simon Baker, Which Countries and Universities Are Leading on AI Research,
TIMES HIGHER EDUC. (May 22, 2017), http://bit.do/BakerAl-research; Fabian, The European
Artificial Intelligence Landscape | More Than 400AI Companies Built in Europe, ASGARD (July
31, 2017), http://bit.do/Fabian European-Al (showing that the majority of AI startups in the E.U.
are located within London (97), compared to the next closest city, Berlin (30)).
28. See Fabian, supra note 27 (discussing how the U.K. has "by far the strongest AI
ecosystem" in Europe, including five of Europe's top ten funded AI companies). Since 2005,
Europe has submitted the third-most Al-related patent applications, trailing behind only the U.S.
and China. China May Match or Beat America in AL, ECONOMIST (July 15, 2017),
http://bit.do/EconomistChina-match-beat. At the same time, Britain has more than double the
amount of AI companies than that of any other European country, ranking at third overall and
followed by Germany at seventh. Id.
29. Alex Hunt & Brian Wheeler, Brexit: All You Need to Know About the UK Leaving the
EU, BBC (Apr. 12, 2018), http://bit.do/HuntBrexit.
30. See Rachel Pells, UK andEurope Must Join Forces' on AI Research Despite Brexit,
TIMES HIGHER EDUC. (May 3, 2018), http://bit.do/PellsBrexit (discussing the U.K.'s expertise
400 SANTA CLARA HIGH TECH. L.J. [Vol. 34

reasons for and against the U.K.'s decision to part from the E.U.-not
discussed in this Comment-a commissioned report to the U.K.
Parliament projects that Al could lead to a f630 billion boon in the
U.K.'s economy by 2035.31 Presumably, the U.K. will use its exit from
the E.U. to develop its own laws for data privacy and Al in a timelier
and more efficient fashion than the E.U., so as to seize this competitive
advantage.32 Not only will Brexit call into question the GDPR's
applicability and territorial scope, but the U.K. will also use less
restrictive provisions to better balance the interests of consumer data
privacy rights with Al. 33 Similar to the actions of the United States,
China, and India, the U.K. plans to balance consumer interests with Al
interests rather than overregulating one or the other.3 4
While Brexit may ultimately lead to a boon in the U.K.'s Al
efforts, the E.U.'s Al industry will suffer greatly from the loss of the
U.K. The E.U should encourage Al efforts so that it remains
competitive in the technological arms race. However, the provisions of
the GDPR may be the primary inhibiting factor in the development of
Al by remaining E.U. Member States and, if unaddressed, may lead to
future departures by Member States.35 Whereas the top Al countries
have minimally regulated Al and the use of PII in big data, the E.U.'s
GDPR creates heightened standards for PII.3 6 Ultimately, the E.U.,
through its efforts to become the world-leader in consumer data

in AI research and why it is necessary for the E.U. and the U.K. to collaborate their efforts in
order to remain competitive in the AI sector).
31. See DAME WENDY HALL & JER(ME PESENTI, DEPARTMENT FOR DIGITAL, CULTURE,
MEDIA & SPORT AND DEPARTMENT FOR BUSINESS, ENERGY & INDUSTRIAL STRATEGY,
GROWING THE ARTIFICIAL INTELLIGENCE INDUSTRY IN THE UK, 2017, at 1-2; Sam Shead,A New
Report Tells the Government How It Can Supercharge'Al, BUS. INSIDER (Oct. 14, 2017, 7:01
PM), http://bit.do/Shead_Supercharge-Al.
32. See sources cited supra note 31.
33. See sources cited supra note 31.
34. See HALL & PESENTI, supra note 31, at 5, 66-74 (arguing that while regulation is
necessary, the report recommends that the industry and government work collaboratively to
establish a U.K. AI Council that oversees, rather than curbs, the development of Al).
35. While this is only speculative, support for the E.U. has declined in nearly all of its
Member States. See Rebecca Flood, REVEALED: Which Countries Could Be Next to Leave the
EU?, EXPRESS (last updated Oct. 2, 2016, 2:34 PM), http://bit.do/FloodNext-to-leave (quoting
ajoint statement issued after the Brexit vote by the foreign ministers of prominent Member States,
"We are aware that discontent with the functioning of the EU as it is today is manifest in parts of
our societies.. .. We take this very seriously and are determined to make the EU work better for
all our citizens.). Were the E.U. to incidentally hinder the use and development of AI in
comparison to the rest of the developing world, it is possible that additional Member States will
contemplate exits so as not to fall behind.
36. See Nick Wallace & Daniel Castro, Center for Data Innovation, The Impact of the
EU's New Data Protection Regulation on Al 1-4, 25-27 (2018), http://bit.do/Wallace_Impact.
2018] TAKING Al PERSONALLY 401

privacy, may be posturing itself out of relevancy in the race to develop

Al.

II. E.U. DATA PRIVACY & Al LAW

The European Union is the world leader in setting consumer data

privacy standards. In fact, the E.U. considers data privacy a
fundamental right for its citizens.37 Article 8 of the E.U.'s Charter of
Fundamental Rights provides E.U. citizens four main data privacy
rights, including the right to: protection of personal data, fair data
processing, access and rectification of collected data, and compliance
of data protection laws.38 With this mindset, the E.U. voted to replace
its existing Data Protection Directive (DPD)3 9 with the more onerous
GDPR, which is set for implementation on May 25, 2018.40 However,
the GDPR does not specifically address Al, though the primary current
model of Al is directly within the GDPR's scope.4' Accordingly, the
E.U. is beginning discussions on legislative proposals to specifically
address Al. 42 Until the E.U. implements these measures, there remains
a high likelihood that current Al models are in direct violation of the
GDPR, which may significantly impact worldwide Al development
efforts.

37. See Charter of Fundamental Rights of the European Union, 2012 O.J. (C 326)
[hereinafter Charter]. While Article 7 of the Charter of Fundamental Rights, id. art. 7, provides a
general right to respect one's private communications, Article 8, id. art. 8, explicitly addresses the
protection of personal data.
38. Charter, supra note 37, art. 8.
39. See Directive 95/46/EC, of the European Parliament and of the Council of 24 October
1995 on the protection of individuals with regard to the processing of personal data and on the
free movement of such data, 1995 O.1. (L 281) 31 [hereinafter DPD]. As a Directive, the DPD
held less authority on the E.U. Member States where each could "determine more precisely the
conditions under which the processing of personal data is lawful." Id. art. 5. However, because
the DPD is only a directive, its authority was limited to setting minimum standards rather than
prescribing a uniform standard of regulation like the way the GDPR will. See Courtney M.
Bowman, A Primer on the GDPR: What You Need to Know, PROSKAUER (Dec. 23, 2015),
http://bit.do/BowmanPrimer; DANIEL J. SOLOVE & PAUL M. SCHWARTZ, INFORMATION
PRIVACY LAW 849, 1135 (Erwin Chemerinsky et al. eds., 5th ed. 2015).
40. GDPR Portal,supra note 3.
41. See Sven Jacobs & Christoph Ritzer, DataPrivacy:Al and the GDPR, NORTON ROSE
FULBRIGHT: DATA PROTECTION BLOG (Nov. 2, 2017), http://bit.do/JacobsDataPrivacy.
42. European Parliament Press Release 201701101PR57613, Robots: Legal Affairs
Committee calls for EU-Wide Rules (Jan. 12, 2017). While there is recognition in the E.U. that it
must promulgate Al laws, discussions have mostly centered on robotics and its potential liabilities
but not how to develop Al.
402 SANTA CLARA HIGH TECH. L.J. [Vol. 34

A. The Scope of the GeneralData ProtectionRegulation

("GDPR")

The GDPR grants extensive data privacy and protection rights to

E.U. citizens, particularly through its material 43 and territorial 44 scope
provisions. The Regulation broadly defines "personal data" to mean:
[A]ny information relating to an identified or identifiable
natural person ('data subject'); an identifiable natural person
is one who can be identified, directly or indirectly, in
particular by reference to an identifier such as a name, an
identification number, location data, an online identifier or to
one or more factors specific to the physical, physiological,
genetic, mental, economic, cultural or social identity of that
natural person;45
Important within the definition are the terms "identified,"
"identifiable," "directly," and "indirectly." Rather than setting limiting
parameters on the applicable data, such as that which would only
directly identify a natural person, the definition broadens the material
scope of the Regulation. Additionally, the GDPR defines "processing"
as:
[A]ny operation or set of operations which is performed on
personal data or on sets of personal data, whether or not by
automated means, such as collection, recording, organisation,
structuring, storage, adaptation or alteration, retrieval,
consultation, use, disclosure by transmission, dissemination
or otherwise making available, alignment or combination,
restriction, erasure or destruction;46
Together, the definitions of "personal data" and "processing" serve to
expand the material scope of the GDPR, which further expands the
already liberal interpretation of data privacy laws by the E.U. courts.

43. GDPR, supra note 7, art. 2. Subject to a few narrow exceptions in subsection (2) of
Article 2, the GDPR's material scope is "to the processing of personal data wholly or partly by
automated means and to the processing other than by automated means of personal data which
form part of a filing system or are intended to form part of a filing system." Id. art. 2(1).
44. Id. art. 3. According to the GDPR, its scope "applies to the processing of personal data
in the context of the activities of an establishment of a controller or a processor in the Union,
regardless of whether the processing takes place in the Union or not." Id. art. 3(1). In addition, the
rest of the article expands the GDPR's territorial scope to those controllers and processors not
established in the union generally, and to those where Member State law applies through public
international law. Id. arts. 3 (2)-(3).
45. Id. art. 4(1). Not elsewhere defined in the GDPR, a "data subject" is an "identified or
identifiable natural person." Id.
46. Id. art. 4(2).
2018] TAKING Al PERSONALLY 403

Even prior to the GDPR, the European Court of Justice (ECJ), the
E.U.'s highest court, liberally read data privacy laws where personal
data may be processed or at risk through a convoluted process. For
example, on October 19, 2016, the ECJ heard a case that involved the
German government and its practice of capturing dynamic IP
addresses. Here, the ECJ broadly interpreted the DPD to protect
certain IP addresses because controllers could "likely reasonably"
compare their data with a third-party's separate system, which contains
identifying information, to identify individual users.4 " By broadly
holding that seemingly anonymous information can be identifying if
possibly used by a third-party, the ECJ demonstrated a liberal
interpretation of data protection policies. Since this broad holding
occurred under the less exhaustive DPD, it is fair to speculate that there
will almost be a presumption of violation in the more restrictive, but
broadly defined, GDPR.
In addition to its material scope, the GPDR attempts to create
international law through an expansive territorial scope. 49 According to
Article 3, the GDPR applies to controllers or processors who process
the personal data of E.U. citizens, and who are: (1) established in the
E.U.;o (2) not established in the E.U., but the activities concern data
subjects in the E.U.;" and, (3) to those subject to Member State law.52
While Article 3's broad territorial scope is likely to face legal
challenges by non-E.U. companies and countries prosecuted under this

47. See Case C-582/14, Breyer v. Bundesrepublik Deutschland, 2016 E.C.R. 11-779. A
dynamic IP address is an IP address assigned to a device when using a public network or internet
service that changes each time the user reconnects. This practice allows a single user to a new IP
address each time a page is reconnected to.
48. Id. Here, the German government collected reported PII when users logged into its
system while a separate system captured IP addresses. These captured IP addresses were not
logged as PII; however, because an in-depth comparison of the two systems could ultimately
identify a user, the ECJ found that consumer data could be traced back to an individual.
49. See GDPR, supranote 7, art. 3.
50. Id. art. 3(1). The first territorial component of Article 3 states that the GDPR "applies
to the processing of personal data in the context of the activities of an establishment of a controller
or a processor in the Union, regardless of whether the processing takes place in the Union or not."
Id.
51. Id. art. 3(2). The second territorial provision of Article 3 states that the GDPR
applies to the processing of personal data of data subjects who are in the Union by
a controller or processor not established in the Union, where the processing
activities are related to: (a) the offering of goods or services, irrespective of
whether a payment of the data subject is required, to such data subjects in the
Union; or (b) the monitoring of their behaviour as far as their behaviour takes place
within the Union.
Id.
52. Id. art. 3(3). The final territorial provision of Article 3 states that the GDPR "applies to
the processing of personal data by a controller not established in the Union, but in a place where
Member State law applies by virtue of public international law." Id.
404 SANTA CLARA HIGH TECH. L.J. [Vol. 34

jurisdictional component, on its face the GDPR applies to any

processing of personal data not subject to Article 2's exceptions.53
Importantly, organizations must determine (1) whether they are a
"controller"' 4 or "processor,"" under the GDPR; (2) whether their
operations involve the "processing of personal data;"5 6 and, (3) whether
they have measures in place to comply with the GDPR.17 While issues
one and two are more easily discernable, companies and organizations
continue to scramble to understand what the GDPR entails and whether
they are in compliance.' Organizations cannot guarantee compliance
with the GDPR, especially for Al, where the data encompassed within
big data fields can often trace back to individuals.
While the intent of the GDPR is to ease data processing through
uniform data privacy and protection standards, 59 its rapid
implementation, coupled with its high standards, threaten to result in
tremendous liability risks for data controllers and processors
worldwide. Specifically, based on the existing Al development models,
machine learning and big data will cause organizations worldwide to
fall under the scope of the GDPR, whether they know it or not. This
potential for liability would be even greater in unsupervised Al models,
which inherently have minimal to no human oversight. Once deemed
within its scope, organizations will need to comply with the GDPR's
provisions, which is easier said than done. Even if located outside of
the E.U., Article 3 presumably may hold anyone liable, especially those
controllers interested in conducting business with companies either
within the E.U., or who trade with E.U.-based companies. If held liable,
companies stand to face severe penalties in the form of "administrative
fines up to 20[,]000[,]000 EUR, or in the case of an undertaking, up to
4 % of the total worldwide annual turnover of the preceding financial
year, whichever is higher." 6 0 The penalties alone may dissuade the

53. Id. art. 3.

54. Id. art. 4(7). A "controller" is
the natural or legal person, public authority, agency or other body which, alone or
jointly with others, determines the purposes and means of the processing of
personal data; where the purposes and means of such processing are determined
by Union or Member State law, the controller or the specific criteria for its
nomination may be provided for by Union or Member State law.
Id.
55. Id. art. 4(8). A "processor" is "a natural or legal person, public authority, agency or
other body which processes personal data on behalf of the controller." Id.
56. Id. art. 3.
57. See id. at 39-41.
58. See, e.g., Bob Violino, Many Firms Clueless on How to Preparefor GDPR, INFO.
MGMT. (July 3, 2017, 7:09 AM), http://bit.do/ViolinoClueless.
59. See GDPR, supranote 7, at 1-3.
60. Id. art. 83(5). An "undertaking" is not defined by the GDPR or within Articles 101 or
2018] TAKING Al PERSONALLY 405

development of Al by certain organizations. There are several

provisions in particular that threaten to impede the development of Al
because its current development model does not comply with the
GDPR's provisions: (1) the right to consent; (2) the right to be forgotten
(erasure); (3) the right to data portability; and (4) the right to
explanation.'

1. Right to Consent

While the GDPR outlines several means by which the processing

of personal information is lawful, the primary method for lawfully
processing consumer PII is through explicit consent for one or more
specific purposes.62 The E.U. adopts an opt-in approach to data privacy,
meaning that controllers may only process personal data if the data
subject unambiguously consents.63 Under Article 7, the burden is on
the controller to prove that the data subject unambiguously and freely
consented, among other conditions. Additional burdens exist with
regard to children. 5 Under the GDPR, a child under the age of 16
cannot give consent, though individual Member States may lower the
age down to thirteen. A controller may mitigate its liability by making
"reasonable efforts" to verify consent, either by the parent or through
the child's age, but these efforts must take into account available
technology. However, the GDPR does not define "reasonable efforts"
and, because available technology must be taken into account, this may
heighten the scrutiny placed upon controllers and processors by the

102 of the Treaty, which are cited within the GDPR, see 2012 O.J. (C 326) 88, 89. Based on its
use, it may be inferred to mean the actions of one or a collective group. See id.
61. See GDPR, supranote 7, arts. 7, 13(f), 14(g), 15(h), 17, 20-22.
62. Id. arts. 6-7.
63. "Consent" under the GDPR is defined as "any freely given, specific, informed and
unambiguous indication of the data subject's wishes by which he or she, by a statement or by a
clear affirmative action, signifies agreement to the processing of personal data relating to him or
her." Id. art. 4(11).
64. See id. art. 7 (listing conditions on consent, such as how controllers must prove that
consent was freely given).
65. Id. art. 8 (Under the standards of Article 8, consent for children refers to "the offer of
information society services directly to a child"). The GDPR cross-references the definition of the
term "information society services" to Directive (EU) 2015/1535 of the European Parliament and
of the Council, id. art. 4(25). This Directive defines the term broadly as "any service normally
provided for remuneration, at a distance, by electronic means and at the individual request of a
recipient of services." Directive 2015/1535, of the European Parliament and of the Council of 9
September 2015 laying down a procedure for the provision of information in the field of technical
regulations and of rules on Information Society services, art. 1(b), 2015 O.J. (L 241) 1, 3
[hereinafter Information Society].
66. See GDPR, supra note 7, art. 8(1). If a child is below the minimum age, only "the
holder of parental responsibility over the child" may authorize consent. Id.
67. Id. art. 8(2).
406 SANTA CLARA HIGH TECH. L.J. [Vol. 34

courts. While the process of simply receiving consent may be

navigable, the greater obstacle is the right to withdraw consent.
Under Article 7, data subjects retain the right to withdraw consent
at any time. 8 While controllers and processors may attempt to continue
to process the data through other means of lawful processing,69 such
continued processing following withdrawn consent risks violating the
GDPR.70 Both the need for consent and the right withdraw consent
threaten the development of Al because it could limit the amount of
data available to learn from. Additionally, data subjects, in certain
situations, can exercise a right to restrict the processing of their
information. 7 ' For example, an organization may collect a large vat of
big data for the sole purpose of machine leaming. Assuming each data
subject consented, this model of Al would be lawful and uninhibited,
regardless of its methodology. Conversely, suppose a data subject, or a
group of data subjects, withdraws consent. While the prior processing
would be lawful, further processing of and leaming from these specific
data points would constitute a violation of the GDPR. Because Al
continues to learn from past data, the issue becomes how to
simultaneously stop Al's leaming from this data, without impacting its
prior development.
The current model of deep leaming through neural networks
demonstrates how Al's development hinges on utilizing multitudes of
data to continuously adapt to the surrounding environment.72 In theory,
the withdrawal of consent, coupled by the continuation of leaming
through the processing of prior learned behaviors, would constitute a
violation of the GDPR. For example, consider an Al system, which

68. Id. art. 7(3).

69. See id. art. 6(1).
70. See Gabe Maldoff, Top 10 OperationalImpacts ofthe GDPR: Part3 - Consent, IAPP
(Jan. 20, 2016), http://bit.do/Maldoff-10.
71. See GDPR, supra note 7, art. 18. Under Article 18, a data subject may restrict a
controller's processing of their PII when any one of the following situations applies:
(a) the accuracy of the personal data is contested by the data subject, for a
period enabling the controller to verify the accuracy of the personal data;
(b) the processing is unlawful and the data subject opposes the erasure of the
personal data and requests the restriction of their use instead;
(c) the controller no longer needs the personal data for the purposes of the
processing, but they are required by the data subject for the establishment,
exercise or defence of legal claims;
the data subject has objected to processing pursuant to Article 21(1)
pending the verification whether the legitimate grounds of the controller
override those of the data subject.
Id. art. 18(1).
72. Roger Parloff, Why Deep Learning is Suddenly ChangingYour Life, FORTUNE (Sept.
28, 2016, 5:00 PM), http://bit.do/ParloffLearning.
2018] TAKING Al PERSONALLY 407

uses a collection of data to learn how to respond to irate consumers

through tonal tendencies. Then, one of the data subjects withdraws
consent from the processing of its personal data, which in this case is
their voice. While all prior learning would be valid under the GDPR,
the Al could no longer use these specific data references to develop its
algorithms. Presumably, the Al could no longer continue its learning
through this data because any further processing of the learning would
be a derivative of the original set containing the withdrawn data.
Instead, the Al would need to receive new data to relearn its function,
unless the processor could somehow isolate the strand of learning,
which incorporated the now nonconsensual data. But, because the
current Al model hinges around neural networks that interweave all
sets of data, this isolation is unlikely to occur. It is likely that the
GDPR's consent provision will result in either large scale Al regression
or continual liability risks for those continuing to derive learnings from
unlawfully processed information.

2. Right to be Forgotten (Erasure)

In addition to the right to give and withdraw consent, Article 17

of the GDPR grants data subjects the right to erasure. 3 Under Article
17, controllers have an "obligation" to erase all personal data, "without
undue delay," when one of several conditions occur, including the
withdrawal of consent.7 4 Additionally, in instances where the data
requested to be erased exists in the public domain, the controller is
obligated to take reasonable steps to inform other controllers that the
information, and any links or copies of it, must be erased.7' By

73. See GDPR, supra note 7, art. 17. In addition to a right to erasure, data subjects also
have a right to the restriction of processing, id. art. 18, which, while not as potentially catastrophic,
will lead to the same or similar cumbersome results.
74. Id. art. 17. A controller must erase personal data when any of the following applies:
(a) the personal data are no longer necessary in relation to the purposes for
which they were collected or otherwise processed;
(b) the data subject withdraws consent on which the processing is based
according to point (a) of Article 6(1), or point (a) of Article 9(2), and
where there is no other legal ground for the processing;
(c) the data subject objects to the processing pursuant to Article 21(1) and
there are no overriding legitimate grounds for the processing, or the data
subject objects to the processing pursuant to Article 21(2);
(d) the personal data have been unlawfully processed;
(e) the personal data have to be erased for compliance with a legal obligation
in Union or Member State law to which the controller is subject;
(f) the personal data have been collected in relation to the offer of
information society services referred to in Article 8(1).
Id. art. 17(1).
75. Id. art. 17(2).
408 SANTA CLARA HIGH TECH. L.J. [Vol. 34

requiring all copies of the data to be erased, the potential detrimental

impact on Al operations may span numerous controllers with a single
exercise of the Article 17 right to be forgotten.
Similar to the right to consent, an exercise of the right to be
forgotten stands to severely impair the development of Al. While
controllers may easily identify individual strands of PII within a
database and delete it, the erasure of personal data that is a part of a set
of big data may impact the Al's accuracy and reliability. For example,
when Al algorithms undergo the process of machine learning, they use
the existing data to learn specific functions. By deleting parts of the
data, the future behaviors of the algorithms may not behave as they
would have when the data was present, making it unstable, less reliable,
and less accurate. For this reason, personal data utilized by Al, even
when deleted, can arguably still remain integrated as part of the neural
network.
One solution to the fear of unlawfully retaining personal data is
for companies to retrain their existing Al models using the modified
data set. However, this solution would result in the creation of Al that
is constantly at risk of destruction. The Al would then have to relearn
everything it had previously learned, which would result in additional
research, development costs, and time delays. In effect, the Al market
within the E.U. would create additional risks, liabilities, and costs not
associated with other global Al markets, which could cause companies
to cease operations with, and within, the E.U.
Alternatively, companies can develop Al models to combat this
issue by designing algorithms specifically to unlearn certain data inputs
without needing to retrain the entire Al neural network. This approach
requires increased research and development costs, additional time to
build, and may still face GDPR compliance issues. 78 Though the
logistics of forgetting information remain murky, companies need to
develop operations that allow for the isolation and deletion of an
individual's PII from a data set.

3. Right to Data Portability

Another hindrance for Al development exists through Article 20's

right to data portability. 79 Article 20 gives data subjects two main
rights: (1) the right to retrieve their personal data from a controller, and

76. Parloff, supra note 72.

77. WALLACE & CASTRO, supra note 36, at 11-13.
78. See generally WALLACE & CASTRO, supra note 36 (discussing how the GDPR will
inhibit the use and development of AI in Europe).
79. GDPR, supranote 7, art. 20.
2018] TAKING Al PERSONALLY 409

(2) the right to then transmit this data to another controller, without
hindrance.so These rights allow data subjects to facilitate the spread of
information, which may ease data collection efforts by smaller
controllers. On the other hand, the right to portability poses similar
problems to those inherent in the rights to consent and erasure."'
The right to portability requires that controllers maintain
processes to identify and isolate an individual's PII.8 2 This, and the
requirement to provide a structured report to the data subject, are fairly
simple tasks. The second right, embedded within Article 20, will
potentially cause issues for controllers: the right to transmit data to
another controller.8 3 In addition to the ability to exercise the right to be
forgotten, and all its associated issues, 4 the data subject may now
require controllers to relinquish their competitive advantages.
Development of Al under its current model hinges on the
collection of big data; large data sets provide a distinct competitive
advantage. Companies spend millions of dollars solely to improve the
data collection processes.' By allowing a data subject to retrieve or
extract this data, smaller companies can collect comparable amounts of
PII without needing to spend as much on their collection processes."
In theory, Article 20's requirements may lead to data parity, which
could in turn create more competitive Al markets that ultimately
benefit consumers. Companies would then have a less distinct
advantage, or disadvantage, based on the amount of data available to
them. While it is possible that this data parity may help with the overall
development of Al, it may also lead to greater risks for companies and
negatively impact Al.
Since consumers will be able to choose who retains their
information, companies will need to emphasize their public relations
and data security-otherwise, data subjects may not trust controllers
with their information. For example, when large scale data breaches
occur, consumers immediately feel violated. Here, the loss of trust and
desire for remedial action will likely result in consumers requiring

80. Id. art. 20(1). Upon request, a controller has an obligation to deliver the personal data
in a "structured, commonly used and machine-readable format." Id.
81. See discussion supra Sections II.A.1 - II.A.2.
82. See Article 29 Data Protection Working Party, Guidelines on the Right to Data
Portability,EUR. COMM'NDOC. WP 242 rev.01 (Apr. 5, 2017).
83. Id. at 4-5.
84. See discussion supra Section II.A.2.
85. Data & Analytics Survey: Big Data and Its Use Cases Keep Growing, IDG (2016),
http://bit.do/IDG Survey.
86. See Ruth Janal, Data Portability - A Tale of Two Concepts, 8 J. INTELL. PROP., INFO.
TECH. & E-CoM. L. 59, 60-61 (2017).
87. See id. for further discussion on this theory.
410 SANTA CLARA HIGH TECH. L.J. [Vol. 34

companies to transmit their data elsewhere, or to delete it entirely. In

turn, heavily funded Al operations may consequentially become
unlawful, or lack the necessary resources to continue its development.
While not going so far as to suggest that corporate espionage and
sabotage amongst Al companies would increase, the potential impact
of breaches and negative publicity could be catastrophic for certain
companies. Ultimately, consumer rights to data portability inherently
contain the risk that one bad instance of public relations could result in
the downfall of promising Al operations.

4. Right to Explanation

Article 22 grants data subjects the right to not be subject to

decisions based solely on automated processing." Instead, the data
subjects may, at the very least, exercise a right to human intervention
and explanation.8 9 While several exceptions to this rule exist,9 0
processors and controllers remain obligated to protecting a data
subject's rights, freedoms, and interests.91 Even if subject to automated
decision-making, individuals still have a right to know of its existence,
including profiling,92 and, if requested, must be provided with
"meaningful information about the logic involved, as well as the

88. GDPR, supra note 7, art. 22(i) ("The data subject shall have the right not to be subject
to a decision based solely on automated processing, including profiling, which produces legal
effects concerning him or her or similarly significantly affects him or her.").
89. Id. art. 22(3).
90. Id. art. 22(2). The right not to be subject to an automated decision does not apply if the
decision:
(a) is necessary for entering into, or performance of, a contract between
the data subject and a data controller;
(b) is authorised by Union or Member State law to which the controller is
subject and which also lays down suitable measures to safeguard the
data subject's rights and freedoms and legitimate interests; or
(c) is based on the data subject's explicit consent.
Id.
91. Id. art. 22(3) (When automated decisions are made either to perform a contract or with
explicit consent, as outlined in Article 22(2), a controller must still "implement suitable measures
to safeguard the data subject's rights and freedoms and legitimate interests, at least the right to
obtain human intervention on the part of the controller, to express his or her point of view and to
contest the decision.").
92. "Profiling" is defined as:
[A]ny form of automated processing of personal data consisting of the use of personal data
to evaluate certain personal aspects relating to a natural person, in particular to analyse or
predict aspects concerning that natural person's performance at work, economic situation,
health, personal preferences, interests, reliability, behaviour, location or movements[.]
Id. art. 4(4), at 33.
2018] TAKING Al PERSONALLY 411

significance and the envisaged consequences of such processing for the

data subject."93
Data subjects also have a right to object to automated processing 9 4
when data processing is for either public interest reasons or when the
data subject's fundamental rights and freedoms outweigh the interests
of the processing controller or third party. 95 Here, the burden is on the
controller to demonstrate that it has "compelling legitimate grounds for
the processing which override the interests, rights and freedoms of the
data subject or for the establishment, exercise or defence of legal
claims." 9 6 Essentially, the controller must persuade the court that its
personal objectives outweigh the highly emphasized and protected data
privacy rights of individuals. Since the ECJ liberally protects consumer
data privacy rights, a controller must determine whether the risk of
violating the GDPR is worth the continuance of processing. However,
even if the ECJ were to find for the controller, a data subject may still
be able to limit the processing of its PII through Article 18's right to
restrict processing.97
For those organizations utilizing Al, it is impractical to employ
unsupervised models of machine learning. Article 22's right to human
intervention and explanation of logic requires that Al decisions be

93. Id. art. 15(1)(h), at 43. Under Article 15, data subjects are given a right of access to
their controlled personal data, including but not limited to, the purpose of the processing, the
categories of data, and the envisaged time period. See id. art. 15(1).
94. Id. art. 21(1), at 45 ("The data subject shall have the right to object, on grounds relating
to his or her particular situation, at any time to processing of personal data concerning him or her
which is based on point (e) or (f) of Article 6(1), including profiling based on those provisions.
The controller shall no longer process the personal data unless the controller demonstrates
compelling legitimate grounds for the processing which override the interests, rights and freedoms
of the data subject or for the establishment, exercise or defence of legal claims."). Additionally,
the right to object to processing must be "explicitly brought to the attention of the data subject
and shall be presented clearly and separately from any other information." Id. art. 21(4), at 46.
Because of this, consumers know they can object to unfavorable decisions made using their
individual PII.
95. Id. art. 6(1)(e), (f), at 36. As referenced, the relevant portions of Article 6(1) referred
to by Article 2 1(1) include when:
(d) processing is necessary for the performance of a task carried out in the
public interest or in the exercise of official authority vested in the
controller; [and]
(e) processing is necessary for the purposes of the legitimate interests
pursued by the controller or by a third party, except where such interests
are overridden by the interests or fundamental rights and freedoms of the
data subject which require protection of personal data, in particular where
the data subject is a child.
Id.
96. Id. art. 21(1), at 45.
97. See id. art. 18(1), at 44.
412 SANTA CLARA HIGH TECH. L.J. [Vol. 34

explainable.98 While a supervised model of learning uses labeled sets

of data to develop algorithms, supplemented by human oversight,
unsupervised models allow Al to evolve on its own. 99 With
unsupervised models, it may not be possible to trace the Al's learning
processes or to explain its decisions, due to a lack of data labels and
relationships. Even supervised models may be too hard to explain,
which would impair one of Al's most useful purposes: automated
decisions and forecasts. loo As a result, the GDPR's extensive protection
of data privacy rights restrains the use of Al's most useful features:
autonomy and automation.
Even if explainable, protections against the profiling of
individuals also stands to eliminate the commercial usefulness of Al,
along with its ability to learn. To be effective, most commercial uses
of Al oriented towards consumers rely on analysis and forecasts based
on an individual's unique characteristics.' 0 However, under the
GDPR, individuals may object to processing used to "analyse or predict
aspects concerning that natural person's performance at work,
economic situation, health, personal preferences, interests, reliability,
behaviour, location or movements[.]"1 0 2 By restricting profiling, Al
cannot learn from human, group, or individual behaviors, and may not
be viable for extensive operational uses. Additionally, the right to
object expressly identifies direct marketing, a marketing tactic which
uses individual buyer behaviors to forecast future purposes so as to
tailor advertisements to a specific individual, as an objectionable
processing purpose.1 03 Objections to direct marketing do not have a
rebuttable aspect that allows controllers to continue their operations if

98. See id. art. 22(3), at 46.

99. For an explanation of supervised and unsupervised AI learning, see Bernard Marr,
Supervised VUnsupervisedMachine Learning-What's The Difference?, FORBES (Mar. 16, 2017,
3:13 AM), http://bit.do/Marr Supervised.
100. See, e.g., Nick Wallace, EUs Right to Explanation:A HarmfulRestriction on Artificial
Intelligence, TECHZONE360 (Jan. 25, 2017), http://bit.do/WallaceEU-Right-to-Explanation
(explaining why algorithms and AI decisions are often not easily explained, because "[a]n
algorithm can spot a correlation, but it cannot explain the link between them because it cannot
infer meaning the way a human can").
101. See generally Mike Kaput, How Brands Target ConsumersBetter and Sell More with
Artificial Intelligence, MKTG. ARTIFICIAL INTELL. INST. BLOG (Jul. 12, 2017),
http://bit.do/Kaput Brands (provides general background information on consumer profiling,
along with some specific examples).
102. See GDPR, supranote 7, arts. 4(4), 21, at 33, 45-46.
103. See id. art. 21(2)-(3), at 45. "Where personal data are processed for direct marketing
purposes, the data subject shall have the right to object at any time to processing of personal data
concerning him or her for such marketing, which includes profiling to the extent that it is related
to such direct marketing." Id. art. 21(2).
2018] TAKING Al PERSONALLY 413

they can show a legitimate purpose. 0 4 Consequently, the commercial

application of Al is limited under the GDPR, which may cause
businesses to forgo future investments in the technology.

B. E. U's ALaw Efforts

The E.U. has recognized that Al is rapidly developing and needing

regulation of sorts.' Based on a report written by the E.U.
Parliament's Committee on Legal Affairs,' 0 6 the E.U. Parliament is
calling for the creation of a European Agency to provide technical,
ethical, and regulatory expertise in the field of robotics and Al.o 7
Though the report broadly states the purpose for this agency, 0 s the
context of the report suggests greater emphasis on combatting potential
liabilities rather than assisting Al development.1 09 Also, while the
report addresses concerns regarding the protection of data privacy and
protection in the context of Al development and regulation, it does not
suggest changing existing regulations."1 0
The Information Commissioner's Office ("ICO") has also
examined the relationship between AL, data privacy, and data
protection."' In this report, the ICO examines the inherent data
protection implications that exist from the use and development of
Al.11 2 However, the ICO report continues to support the GDPR, even

104. Compare id. art. 21(1), with id. art. 21(2), (3) (while controllers may defend their
continued processing when lawfully conducted under Article 6(1)(e), (f), processing for the
purposes of direct marketing is objectionable without rebuttal).
105. See The Future ofRobotics andArtificial Intelligence in Europe, EUR. COMM'N: DIG.
SINGLE MKT. BLOG (Feb. 16, 2017), http://bit.do/DSM-blog (discussing the E.U. Commission's
adoption of the Digitising European Industry Strategy in 2016, which recognizes robotics and AI
as cornerstone technologies). See also Rich Haridy, EU Move to Bring in AI Laws, but Reject
Robot Tax Proposal, NEWS ATLAS (Feb. 16, 2017), http://bit.do/Haridy EU-move. According to
this article, the E.U. Parliament voted 396 to 123, with 85 abstentions, to pass a resolution to
regulate the development of AI and robotics. Id. This resolution has since passed to the E.U.
Commission, the E.U.'s executive branch, to determine whether or not to accept or reject the
proposal to regulate. Id. The E.U. Parliament's decision to regulate was based on a report sent to
the E.U.'s Legal Affairs Committee, which called for regulation of AI and robotics. Id.
106. See Eur. Parl. Comm'n. on Legal Affairs, DraftReport with Recommendations to the
Commission on Civil Law Rules on Robotics, 2015/2103(INL) (May 31, 2016),
http://bit.do/EuroParlInitial-report [hereinafter DraftReport].
107. See id. at 7.
108. Id. ("to ensure a timely and well-informed response to the new opportunities and
challenges arising from the technological development of robotics").
109. See id. at 10-12.
110. See id. at 8 (noting that regulation must pay particular attention to data privacy and
protection as it concerns technical integration into hardware and software, necessity and
proportionality standards, and the use of personal data as currency).
111. Big Data, supra note 12, at 19-56.
112. Id. (listing a multitude of liability and operation concerns for the use and development
of Al, including: fairness of processing, conditions for processing, and data minimization, among
414 SANTA CLARA HIGH TECH. L.J. [Vol. 34

after it points out a multitude of provisions that will negatively impact

Al and machine leaming.113 As part of its conclusion, the ICO stated:
We are aware of the view that, given some of the challenges
of applying data protection principles to big data analytics, a
different legal or regulatory approach is required. However,
we do not accept the idea that data protection, as currently
embodied in legislation, does not work in a big data context.
We maintain that big data is not a game played by different
rules. We acknowledge the increasing importance of
accountability in addressing some of these challenges, but we
do not see it as a replacement for the more traditional principle
of transparency. Transparency still has a significant role to
play and we argue it can still be achieved, even in a complex
world of Al and machine learning.
Instead of suggesting repairs to the current system, the ICO reinforces
the current E.U. stance to stay their current regulatory course. As a
result, it is likely that, even if the E.U. promulgates a new regulatory
agency and subsequent Al standards, the standards would need to
comply with the GDPR's provisions.
The GDPR will affect the way that companies use and develop
Al. Even though the E.U. plans to regulate Al specifically, these
regulations will exist harmoniously with the GDPR's expectations."
As a result, Al regulatory action is unlikely to assist in circumventing
the difficulties for Al presented by the GDPR. While the E.U. may
think that this is the best course of action, its immediate detrimental
impact will likely cause the E.U.to explore alternatives in regulating Al
with respect to data privacy.

III. WHERE DOES THE E.U. GO FROM HERE?

The E.U.'s imminent GDPR implementation and its approach

towards Al, coupled with the current, predominant model of Al,
threaten to stagnate Al research and development in the E.U.

others).
113. Id.
114. Id. at 95.
115. See, e.g., Draft Report, supra note 106 (emphasizing that privacy and data protection
guarantees must be embedded in the E.U.'s regulatory framework for robotics and AI). The E.U. 's
cautious approach to the development of AI is evident in its policy. Id. at 7 (noting that the
"potential for empowerment through the use of robotics is nuanced by a set of tensions or risks
relating to human safety, privacy, integrity, dignity, autonomy and data ownership"). In response
to the growing commercialization of the robotics and AI sector, the E.U. has stressed the
importance of consumer protection (pointing out that the "use of personal data as a 'currency'
with which services can be 'bought' . . . may not lead to a circumvention of the basic principles
governing the right to privacy and data protection). Id. at 8.
2018] TAKING Al PERSONALLY 415

Additionally, if the ECJ liberally interprets the GDPR and fully

exercises its territorial jurisdiction, international issues regarding Al
are to be expected. Alternatively, countries and companies may instead
develop Al without the use of PII from the E.U., which could impact
its ability to function there. To address these issues, the European
Council and Commission should choose to either (a) carve Al out of
the GDPR's scope and form an E.U. Al Council, or (b) assist in funding
the development and use of alternative Al models that would comply
with current GDPR standards.
A. Alas a Carve-Out of GDPR
Although the E.U. is currently exploring opportunities to regulate
Al, 6 the 2018 implementation of the GDPR is imminent. Once
effective, controllers and processors of E.U. personal data will be
subject to the GDPR's terms. While the GDPR's provisions stand to
govern all who fall within its material scope, due to its extensive
territorial jurisdiction, the question remains of how those outside of the
E.U. will respond to the GDPR's provisions. In theory, any controller
or processor that processes personal data belonging to an E.U. citizen
will fall within the GDPR's material and territorial domain.'
However, the potentially negative impact of the GDPR" on Al use
and development will result in an unwillingness to abide by its terms.
Instead, the GDPR will face significant legal challenges by those
controllers and processors prosecuted under Article 3's territorial
scope.
While the GDPR's intent is to govern all those who use personal
data belonging to E.U. citizens, legal challenges against its scope and
extraterritorial reach may lessen its effectiveness or result in a blatant
disregard for the regulation altogether.11 9 Instead, only E.U.-based
controllers and processors would bear the burden of trying to comply

116. See discussion supra Section II.B.

117. See GDPR, supranote 7, arts. 2-3, at 32-33.
118. See discussion supra Section II (detailing articles of the GDPR that may severely
impact the development and use of AI).
119. See Dennis Dayman, GDPR Impactfor Non-EU Companies, RETURN PATH (Mar. 14,
2018), http://bit.do/Dayman Non-EU (speculating that enforcement of the GDPR against non-
E.U. companies could be executed through court injunctions or potential seizures of physical
goods). Alternatively, GDPR compliance could also be regulated through international
cooperation agreements, especially between the U.S. and E.U. law enforcement agencies, such as
implementing mechanisms that allow the E.U. to issue complaints and fines against American
companies. See Aaron Winston, How the EU Can Fine US companies for Violating GDPR,
SPICEWORKS (June 21, 2017), http://bit.do/ViolatingGDPR (information based on an interview
with former Deputy General Counsel and Ethics Official of the White House Office of Drug
Policy Linda V. Priebe, a specialist in data privacy and security in both the U.S. and E.U.).
416 SANTA CLARA HIGH TECH. L.J. [Vol. 34

with the GDPR while simultaneously developing Al. While GDPR

compliance would be necessary for those conducting business in or
with E.U.-based controllers, the E.U. market may not be worth the
efforts to comply or the potential liabilities for noncompliance. Instead,
top Al companies and organizations could choose to avoid the GDPR.
In effect, avoiding the GDPR would allow the Al development in other
countries to progress unencumbered, versus those within the E.U and
subject to the GDPR. Rather than rely on the GDPR to retain global
jurisdiction, the E.U. should recognize its potential adverse impact on
Al within its own borders.
Many countries have recognized the effects of overregulating
technological advancement. These countries have chosen not to overly
protect consumer personal data or have carved out Al uses of personal
data from more onerous PII regulations.1 20 Because Al requires large
sets of data to develop, overregulation of personal data stagnates Al
use and limits research. While it is evident that the E.U. intends to
develop Al laws that mesh with the GDPR's provisions, it should
instead carve out Al from the GDPR.121 In particular, Al regulations
need to address the issues surrounding the erasure of data, in favor of
controllers and processors. For example, rather than requiring a
complete erasure of personal data, controllers and processors should be
able to retain information up to the point of erasure. In this way, the
Al's machine learning would remain at the point where it progressed,
rather than creating forced amnesia. However, all future machine
learning would not include the erased personal data. Presumably, this
would balance the interests of deleting the individual's PII without
causing the Al to regress.
Ultimately, the existing Al model emulates the neural activity of
a human. Like humans, Al observes and learns from its external
environment. While humans learn from the real world, Al learns from
big data. Rather than treating Al like ordinary data collection and
processing systems, the E.U. should recognize that Al most closely
resembles human information processing. Instead of requiring Al to
forget and unlearn, it is best for the E.U. to treat learned behaviors as
separate from the personal data it is derived from.

B. E. U FundingofAlternative Al Models

If the E.U. remains steadfast in not carving Al out of the GDPR,

its next best solution may be to fund Al development. Similar to China,

120. See discussion supra Section I.B.

121. See supra Section II.B (discussing the detrimental effects of creating a new regulatory
framework for AI that is forced to coexist with the GDPR).
2018] TAKING Al PERSONALLY 417

government Al funding would help to expedite and sustain Al

development.1 22 Additionally, government funding would entice Al
companies to stay within the E.U., even though they would remain
subject to the GDPR. Because the government would assist in funding
the Al, it may be more willing to relax its enforcement of the GDPR
against those acting on behalf of the E.U. or its Member States. With
government funding, controllers would have greater incentive and
resources to research newer models of Al that better fit within the scope
of the GDPR.
Though the current model of Al allows reverse-engineering to
trace PII to a specific individual, research suggests that there may be
ways to combat this threat to consumer data.1 23 For example, Google
and OpenAl have found that it is possible to build algorithms less
traceable to a particular individual by using "differential privacy."1 2 4
Google's theoretical model allows algorithms to function as if the
algorithm had learned from personal data without actually ever seeing
it.1 25 While not quite as accurate as the traditional model of Al, early
results indicate that this model is within an accuracy rate of 2% of the
traditional model, versus 5% by any other alternative model. 26 These
early indications support an ability for Al technology to eventually
evolve in a manner consistent with the GDPR.

122. See Pham, supranote 25.

123. See Nicolas Papernot et al., Semi-Supervised Knowledge Transferfor Deep Learning
from Private Training Data, 5 INT'L CONF. ON LEARNING REPRESENTATIONS PROC. (2017),
http://bit.do/Semi-Supervised.
124. See Dave Gershgom, Al Can Learn From Data Without Ever Having Access to It,
QUARTZ (Oct. 24, 2016), http://bit.do/Gershgom Al-can-learn (discussing differential privacy,
which allows for the protection of individual PII in a large data set and "addresses the paradox of
learning nothing about an individual while learning useful information about a population").
125. Id.
126. Id.
418 SANTA CLARA HIGH TECH. L.J. [Vol. 34

CONCLUSION

If the E.U. desires to remain competitive in the race to develop

Al, it must balance its interests in protecting personal data against its
interest in developing new Al technologies. The implementation of the
GDPR, without carveouts to ease the use of personal data in Al
systems, demonstrates the E.U.'s favor toward data privacy.
Incidentally, the GDPR's restrictions on personal data will burden the
ability of Al to learn and develop. As a result, the E.U.'s Al industry is
likely to suffer as organizations seek ways to circumvent the GDPR's
provisions. Until the E.U. recognizes and addresses the potential
impact of the GDPR on its Al industry, the E.U. will fall behind in its
Al efforts.
 TAKING Al PERSONALLY: HOW THE E.U. MUST LEARN
TO BALANCE THE INTERESTS OF PERSONAL DATA
PRIVACY & ARTIFICIAL INTELLIGENCE

By
Matthew Humerickt
(Santa Clara High
Technology Law Journal 34, no. 4 (May 2018): 393-418)

Abstract :

● Part I of this Comment provides an overview of what Al is and how it utilizes

personal data to develop. Contained within Part I is also a brief overview of the
current international Al situation and how the E.U. 's relevance is rapidly declining.

● Part II discusses data privacy and Al law within the European Union, and how new
regulations may adversely impact sustained algorithmic development under the
current Al model.

● Part III proposes two courses of action on how the E.U. should adapt its views on data
privacy and protection to better facilitate the use and development of Al.

● The legislation referred to is the General Data Protection Regulations.

First interesting point:

● Importantly, organizations must determine (1) whether they are a "controller" or

"processor," under the GDPR (2) whether their operations involve the "processing of
personal data;" and, (3) whether they have measures in place to comply with the
GDPR.

● While issues one and two are more easily discernible, companies and organizations
continue to scramble to understand what the GDPR entails and whether they are in
compliance with it or not.
 ● However, organizations cannot guarantee compliance with the GDPR, especially for
Al, where the data encompassed within big data fields can often trace back to
individuals.

● While the intent of the GDPR is to ease data processing through uniform data privacy
and protection standards, its rapid implementation, coupled with its high standards,
threaten to result in tremendous liability risks for data controllers and processors
worldwide.

● Specifically, based on the existing Al development models, machine learning and big
data will cause organizations worldwide to fall under the scope of the GDPR, whether
they know it or not.

● This potential for liability would be even greater in unsupervised Al models, which
inherently have minimal to no human oversight. Once deemed within its scope,
organizations will need to comply with the GDPR's provisions.

● Even if located outside of the E.U., Article 3 presumably may hold anyone liable,
especially those controllers interested in conducting business with companies either
within the E.U., or who trade with E.U.-based companies.

● If held liable, companies stand to face severe penalties in the form of administrative
fines up to 20,000,000 EUR, or in the case of an undertaking, up to 4 % of the total
worldwide annual turnover of the preceding financial year, whichever is higher.

Second Point:

● Article 7 of the GDPR the burden is on the controller to prove that the data subject
unambiguously and freely consented, among other conditions and that data subjects
retain the right to withdraw consent at any time.

● Both the need for consent and the right to withdraw consent threaten the development
of Al because it could limit the amount of data available to learn from.

● Data subjects, in certain situations, can exercise a right to restrict the processing of
their information.

● The current model of deep learning through neural networks demonstrates how Al's
development hinges on utilizing multitudes of data to continuously adapt to the
surrounding environment.

● The withdrawal of consent, coupled by the continuation of learning through the

processing of prior learned behaviors, would constitute a violation of the GDPR.
 ● For example, consider an Al system, which uses a collection of data to learn how to
respond to irate consumers through tonal tendencies. Then, one of the data subjects
withdraws consent from the processing of its personal data, which in this case is their
voice. While all prior learning would be valid under the GDPR, the Al could no
longer use these specific data references to develop its algorithms. Presumably, the Al
could no longer continue its learning through this data because any further processing
of the learning would be a derivative of the original set containing the withdrawn
data. Instead, the Al would need to receive new data to relearn its function, unless the
processor could somehow isolate the strand of learning, which incorporated the now
non consensual data.

● Current Al model hinges around neural networks that interweave all sets of data, this
isolation is unlikely to occur. It is likely that the GDPR's consent provision will result
in either large scale Al regression or continual liability risks for those continuing to
derive learnings from unlawfully processed information.

Third Point:

● Article 22 grants data subjects the right to not be subject to decisions based solely on
automated processing." Instead, the data subjects may, at the very least, exercise a
right to human intervention and explanation. Even if subject to automated
decision-making, individuals still have a right to know of its existence, including
profiling and, if requested, must be provided with "meaningful information about the
logic involved, as well as the significance and the envisaged consequences of such
processing for the data subject

● Data subjects also have a right to object to automated processing when data
processing is for either public interest reasons or when the data subject's fundamental
rights and freedoms outweigh the interests of the processing controller or third party.

● Protections against the profiling of individuals also stands to eliminate the

commercial usefulness of Al, along with its ability to learn. To be effective, most
commercial uses of Al oriented towards consumers rely on analysis and forecasts
based on an individual's unique characteristics.

● By restricting profiling, Al cannot learn from human, group, or individual behaviors,

and may not be viable for extensive operational uses. Additionally, the right to object
expressly identifies direct marketing, a marketing tactic which uses individual buyer
behaviors to forecast future purposes so as to tailor advertisements to a specific
individual, as an objectionable processing purpose.
Conclusion:

Having said that, the introduction of the GDPR presents an opportunity for Malaysian
businesses, particularly organisations dealing with AI, to buckle up and set a higher standard
of protection and procedure to ensure that their business processes are in sync with the
changes in both the local and global personal data protection regulatory regime. Moving
forward, it is proposed that Malaysian companies and businesses (data controllers) conduct
Data Protection Impact Assessment to understand the risks to the security and privacy of the
data processed by AI and ways to mitigate those risks. This would eventually help to balance
the interest of personal data privacy as well as the development of artificial intelligence. The
ratification of appropriate and necessary principles under the GDPR as part of Malaysia’s
personal data protection laws is vital to ensure growth of AI whilst balancing the emerging
need to improve data protection.