Creative Capabilities of Machine Learning
Evaluating music created by algorithms
Thomas Mejtoft
Umeå University
Digital Media Lab
Sweden
thomas.mejtoft@umu.se
Ulrik Söderström
Umeå University
Digital Media Lab
Sweden
ulrik.soderstrom@umu.se
ABSTRACT
The concept of creativity is an important part of human society and
the continuous evolution of artificial minds has raised questions
on creativity among machines. This aim of the this study is to
explore machine learning algorithms’ ability to be creative. The
study reported in this paper uses short samples of music generated
by IBM Watson beats that are evaluated using expert assessment of
51 music teachers together with samples generated by humans as
control samples. The results show that one of the machine learning
generated samples showed the same level of creativity as the human
generated samples. Hence, there are indications that today machine
learning algorithms can create music that is hard to distinguish
from human created music and can be considered creative.
CCS CONCEPTS
• Theory of computation → Design and analysis of algorithms;
Machine learning theory; • Human-centered computing → In-
teraction techniques.
KEYWORDS
creativity, machine learning, music
ACM Reference Format:
Thomas Mejtoft, Linus Lagerhjelm, Ulrik Söderström, and Ole Norberg.
2021. Creative Capabilities of Machine Learning: Evaluating music created
by algorithms. In European Conference on Cognitive Ergonomics 2021 (ECCE
2021), April 26–29, 2021, Siena, Italy. ACM, New York, NY, USA, 4 pages.
https://doi.org/10.1145/3452853.3452863
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
ECCE 2021, April 26–29, 2021, Siena, Italy
© 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-8757-6/21/04. . . $15.00
https://doi.org/10.1145/3452853.3452863
Linus Lagerhjelm
Umeå University
Digital Media Lab
Sweden
lila0158@student.umu.se
Ole Norberg
Umeå University
Digital Media Lab
Sweden
ole.norberg@umu.se
1 INTRODUCTION
Creativity research has been in existence for hundreds of years [15]
and throughout history, the view on the concept of creativity has
both differed and changed a lot. Most notably the modern view on
creativity, the creation of novel works, were seen as impossible by
both ancient Buddhists and Plato which both thought that noth-
ing new could ever be created. Over time, this view has gradually
evolved into today’s view where there are a consensus that cre-
ativity can exist on an individual level or, in other words, someone
can be creative. It would seem reasonable to hypothesize that if
it would be possible to construct an artificial mind with the same
complexity as the human mind it would be able to display similar
creative abilities. The externalization of this artificial creativity may
or may not look analogous to that found in humans but it would
still be there.
Artificial creativity is something that is highly controversial and
some argue that such a thing does not exist. For example Boden
[4] claims that artificial creativity cannot exist as works made by
computers lack “a subtle appreciation of relevance” and a “capacity
to remind us of musical or cultural associations that wouldn’t have
occurred to us otherwise”. Others such as Câmara [6] argue that
such argumentation is “too human-centric” and further more that
creativity is a matter of combining multiple factors of which some
are better suited for computers and vice versa. Despite this contro-
versy, the research of the topic is limited. The question of artificial
creativity will be subject to evaluation in this study.
1.1 Defining creativity
Creativity may perhaps be the most crucial definition to this paper.
Unfortunately the literature fails to agree on a formal definition.
In an article in the Annual Review of Psychology from 2010, Hen-
nessey & Amabile [10] covered more than 110 different articles
on the topic and the lack of consensus is striking. Hennessey &
Amabile [10] state, however, that most of the experts agree that
“creativity involves the development of a novel product, idea, or
problem solution that is of value to the individual and/or the larger
social group”. Aside from that, various frameworks for defining
creativity has been proposed only to be discarded by others.
ECCE 2021, April 26–29, 2021, Siena, Italy
In spite of all this there seems to exist a somewhat established
way of determine whether a piece of work can be deemed as creative
or not. This method is based on consensual assessment of experts
[10] that gets to judge a piece by their own subjective definition of
creativity. What constitutes an expert is not clearly defined in the
review but Amabile [3] discusses a similar concept which is referred
to as “appropriate observers”. These observers are defined as “those
familiar with the domain in which the product was created or the
response articulated” [3].
In this study this method when defining whether a piece of work
is creative will be used. Consequently we will assert that the creator
of a creative piece must also be considered creative. Such a creator
will be referred to as a creative agent.
1.2 Objective
The objective of this short research paper is to investigate whether
machine learning systems can be considered creative based on their
ability to compose novel works of music. This study is intended as
a springboard to further interdisciplinary research touching upon
machine learning and psychology.
2 THEORY
This section includes the theoretical framework needed to under-
stand the paper starting with an overview of machine learning
and neural networks. Later in the section some specific kinds of
networks that are central to this paper will be covered.
2.1 Machine Learning
There are plenty of problems in modern software development that
is characterized by the fact that the algorithm for solving them
are unknown [2]. We know the desired output from our program
given some input data but we lack the ability to formally specify
the required steps to derive this answer from the data. In machine
learning this problem is addressed by the construction of programs
that can extract certain characterizing hidden patterns from the
data and exploit them to produce the desired output. This process
of extracting patterns from input data to produce correct or good
enough (application dependent) output is usually referred to as
learning [2]. There are plenty of approaches to machine learning
and a wide range of different models. This paper will mainly focus
on a subset of machine learning known as neural networks.
2.2 Neural Networks
The idea of neural networks is very old. The very first research
paper where the ideas occurred was published already in 1943 [12].
The idea was to mimic how neurons work in the human brain. A
neural network in a computer is a composition of multiple simple
computational threshold units. Units that produce real valued out-
puts in response to sensory inputs [16]. These units are referred
to as neurons. Neurons are typically hierarchically organized in a
structure where the neurons are grouped together in smaller groups
to form layers. All the neurons in a layer are interconnected with
all the neurons in the previous layer via a set of connection weights.
An example of this structure are depicted in figure 1 below where
input layer denotes an input vector to the network. Hidden layer is a
group of neurons whose output is only visible to other neurons and
T. Mejtoft, L. Lagerhjelm, U. Söderström & O. Norberg
Input Hidden Output
layer layer layer
x1
x2
Output
x3
x4
Figure 1: A simple example of the structure of a neural net-
work.
the output layer is a single neuron which produces the output from
the network. Usually, neural networks consists of many hidden
layers of various sizes. The output from the neuron corresponds to
the dot product between the input vector and a weight vector. The
weight w i j between input i and neuron j determines how much a
specific input affect the output of a neuron. Learning in terms of
a neural network is the process of finding these weights in such a
way that the network produces the desired output [16] for a given
input. The output from a network typically consists of either a
binary value or real number that represents a class or percentage.
2.3 Boltzmann Machines
The Boltzmann Machine is a special kind of neural network that,
just like the traditional neural network that were described in the
previous section, consists of primitive computing units [1]. The
difference between the previous model and the Boltzmann Machine
is that the connections between units are bidirectional and can only
assume exactly one of two states: on or off represented as 1 and 0.
These states are chosen as a function of the states of the connected
units, the weights between them and a small degree of chance. Each
unit forms part of an interpretation of the input data and the on or
off state determines if this interpretation is accepted or rejected [1].
An example of an interpretation of input could be: given the pixels
of an image, do we interpret the image as representing a digit?
The different parts of the network, weights and on or off state,
can be used to describe the current state (usually referred to as
energy) of the network which is denoted E and computed as:
Õ Õ
E=− w i, j si s j + θ i si
i <j i
Where w i, j is the weight between unit i and j, si and s j are either 1
or 0 if neuron i or j are on or off. θ is a threshold for neuron i.
This energy is interpreted as how many of the interpretations
imposed by the different units are incorrect. Thus, by choosing w i, j
that minimizes E we can make correct interpretations of an input.
2.4 Restricted Boltzmann Machines
The Restricted Boltzmann Machine has exactly the same properties
as the generalized Boltzmann Machine [11] but it is restricted to
using only two layers - one visible and one hidden layer. In the
first step the visible layer consists of the actual input vector of
binary values. This input is used to update the units in the hidden
Creative Capabilities of Machine Learning
layer. In the next step, the hidden layer becomes the input to the
visible layer which is in turn updated the same way as the hidden
layer. This process is repeated back and forth until the network is
trained. Once trained, the hidden units can be used as the input
data vector to another restricted Boltzmann Machine. By stacking
restricted Boltzmann Machines like this, it is possible to achieve a
very efficient network which work very well in practice [11].
2.5 Watson Beat
Watson Beat [5] is a branch of the IBM Watson [14] project carried
out by IBM research. It is a machine learning algorithm that, when
trained on music samples, can produce unique music compositions
in response to a suggested emotion. Behind the cover, Watson Beat
consists of a stack of Restricted Boltzmann Machines that reads
and outputs MIDI files [13].
3 METHOD
This study will utilize a method that is established within creativity
research [3, 10] and briefly explained in section 1.1. The method is
built upon the insight that a piece of work is “creative to the extent
that appropriate observers independently agree it is creative”. Using
this statement as basis for the design, Amabile [3] describes a study
which uses multiple experts that assesses the creativity of the piece
of work. As expert judges, a crowd of studio artists with at least
5 years of experience were used. This study aims towards closely
mimic the study by Amabile [3]. The judges used in this study
were currently employed as music teachers and should therefore be
considered domain experts within music and the creation of music.
3.1 Participants
For this study, 51 domain experts were used. As previously men-
tioned the experts were all, or had previously been, employed as
music teachers of some form. A majority of the participants (71,7%)
had been at their current position for at least 10 years. When asked
about total experience working with music, 58,5% stated that they
had more than 40 years of experience regarding music. Everyone
of the 51 participants had at least 10 years of experience working
with music in some way.
3.2 Implementation
In order to increase the number of of participants in the study, to
be able to perform quantitative analysis, the study was performed
using an online questionnaire. The questionnaire contained four
different pieces of instrumental music, each 58 seconds in length.
Two of the samples were created by humans and two were created
by Watson Beat. The subjects were asked to rate the creativity of
each of the four clips individually on a scale of 1 - 6 (1 = not creative
at all, 6 = a very high level of creativity) . An even numbered scale
was intentionally chosen in order to prevent neutral answers [7].
In addition to grading the pieces of music, the subjects were also
provided with the ability to explain their grading in free text format.
To prevent bias in the judgments, no background on the creation of
the samples were provided. Statistical analysis on the gathered data
were performed using MATLAB. The entire study were carried out
during spring 2018.
ECCE 2021, April 26–29, 2021, Siena, Italy
4 RESULTS
The participants were asked to grade each sample individually on
a six degree scale. During this results section these samples are
referred to as ML1 & ML2 for the two samples that were gener-
ated by Watson Beats and H1 & H2 for the two samples that were
composed by humans.
4.1 Distribution
The distribution of the creativity rating has been plotted as a box
plot (figure 2). The x-axis of the box plot represents the four different
samples, labeled the same as described in the previous section. The
y-axis represents the scoring on the samples. The box represents
the interval in which 50% of the most central answer lay.
The plots look very similar for the different samples except for
the sample H1 that differs on a few points. Starting, the spread
of H1 is much less than for the other samples, the grading ranges
between 2-5 with a single outlier grading it a 6. Further more the
50% box only ranges between 3-4. ML1, ML2 and H2 all have grading
ranging from 1-6 while the 50% box of H2 differs from those of ML1
and ML2 where H1 ranges between 2.25-4 and the two ML samples
ranges between 2-4.
The median of H1 is 4 compared to the other samples that has
their median at 3. It is worth noting that the half numbers on the
y-axis were not possible choices in the questionnaire. Those are
there only for visualization purposes.
Distribution of answers
6
5.5
5
4.5
4
Grading
3.5
3
2.5
2
1.5
1
ML1 H1 ML2 H2
Sample
Figure 2: Distribution of creativity ratings among the differ-
ent samples.
4.2 Hypothesis testing
To determine whether there is a significant difference between
the ratings of the different samples, the samples were compared
in MATLAB using a paired t-test. This test is used to determine
whether there exists a significant difference between two, relatively
small, samples.
Table 1 displays the result of the significance tests. The number
is the resulting p value when performing t-test between the samples
ECCE 2021, April 26–29, 2021, Siena, Italy
Table 1: Significance levels between samples using t-test.
ML1 ML2
H1 0.0005 0.0685
H2 0.0773 0.5969
annotated by each corresponding label. For this experiment the
significance level where chosen as 95% using α = 0.05 This means
that if the t-test results in a value of p < α the null hypothesis
(the samples are equal) is rejected and we state that the results are
different.
On basis of these values of p it is stated that there is a significant
difference between the mean of the samples ML1 and H1 while for
the comparison of the other samples there are no significant differ-
ence. For completeness sake, it should be noted that the confidence
interval for the comparison between the two samples ML1 and H1
was <-0.9362, -0.2795>. Since the interval contains only negative
values, the second sample was rated significantly higher than the
first one. In this case the second sample was H1.
5 DISCUSSION
Before digging into the interpretation and implications of these
results we will spend some time discussing the methodology and
its effect on the result. Due to the relatively large number of partici-
pants, it is reasonable to believe that the statistical analysis provides
an accurate description of reality.
Factors that may have influenced the result is our choice of an
even numbered scale. As already mentioned an even numbered
scale prevents the user from providing neutral answers. The benefit
of this is of course that we will avoid the case were a majority of
the users do not want to express any opinion. Due to the nature of
this study it is likely that the subjects would have been tempted to
provide more neutral answers in order to not upset an assumed com-
poser of the sample. As the subjects were not told that some pieces
were created using machine learning algorithms, this could have
shown a noticeable effect. Our choice to leave out this information
is motivated by the fact that the subjects preconceptions about the
creative abilities of machine learning algorithms otherwise could
have influenced the result.
The generalizability of this study is somewhat limited due to the
number of available samples. It is not possible to make a reliable
statement about weather all machine learning algorithms are more,
less, or equally creative as humans. The only thing that can be stated
is that the specific algorithm used can produce musical pieces that,
to human experts, shows the same level of creativity as pieces
created by creative humans.
5.1 Conclusions and future research
As discussed previously, it is difficult to draw any generalized con-
clusion from these result. However, it is possible to say that one
sample in our experiment were rated at the same level of creativity
as its human counter part. On the other hand: this finding is quite
remarkable still as the only reasonable interpretation of this result
is that machine learning has the ability to produce creative pieces
of work. Remember that the definition of a creative agent proposed
in section 1.1 of this paper were that an agent is considered creative
T. Mejtoft, L. Lagerhjelm, U. Söderström & O. Norberg
if it has the ability to produce creative pieces of work. When using
this definition we arrive at the conclusion that machine learning
algorithms can be creative.
The extent of this creativity is, however, something that has to
be mapped out in future research. It is possible that this study found
the only piece of music generated by machine learning algorithms
that has the property of being creative. It should also be investi-
gated which machine learning algorithms that performs better on
creative tasks than others and which creative tasks are well suited
for machine learning algorithms. This latter part provides a good
starting point for inter disciplinary work between the machine
learning community and the psychology community.
This is an fast moving area and there is no doubt that machine
learning algorithms will be able to be even more “creative” in the
future, making it increasingly hard to distinguish between human-
made and machine-made. During the last couple of years Generative
Adversarial Network (GAN) [9] has proven to become a important
part of the creativity within machine learning. The goal of GAN is to
“give machines something akin to an imagination” [8]. Results from
e.g. image creation have been impressive in terms of creative effort
by machines. This kind of image generation has raised concerns
since it can be used for sinister purposes, e.g hoaxes and frauds
using so-called deepsfakes [17].
REFERENCES
[1] David H. Ackley, Geoffrey E Hinton, and Terrence J. Sejnowski. 1985. A Learning
Algorithm for Boltzmann Machines. Cognitive Science 9, 1 (1985), 147–169.
[2] Ethem Alpaydin. 2014. Introduction to Machine Learning, Third Edition. The MIT
Press, Cambridge, MA.
[3] Teresa M. Amabile. 1982. Social Psychology of Creativity: A Consensual As-
sessment Technique. Journal of Personality and Social Psychology 43, 5 (1982),
997–1013.
[4] Margaret A. Boden. 2015. Artificial Creativity. MIT Technology Review 118, 6
(2015), 12–13.
[5] Camille Charluet. 2017. IBM’s Watson Beat: who owns music made by a machine?
Retrieved August 14, 2018 from https://thenextweb.com/apps/2017/10/20/ibms-
watson-beat-who-owns-music-made-by-machine
[6] Francisco Câmara Pereira. 2014. Creativity and Artificial Intelligence: A Conceptual
Blending Approach. Mouton de Gruyter, Berlin, Germany.
[7] Ron Garland. 1991. The Mid-Point on a Rating Scale: Is it Desirable? Marketing
Bulletin 2 (1991), 66–70.
[8] Martin Giles. 2018. The GANfather: The man who’s given machines the gift of
imagination. MIT Technology Review (21 Feb. 2018). Retrieved November 28,
2020 from https://www.technologyreview.com/2018/02/21/145289/the-ganfather-
the-man-whos-given-machines-the-gift-of-imagination/
[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial
Networks. In Proceedings of the International Conference on Neural Information
Processing Systems (NIPS 2014). 2672–2680.
[10] Beth A. Hennessey and Teresa M. Amabile. 2010. Creativity. Annual Review of
Psychology 61 (2010), 569–598.
[11] Geoffrey E. Hinton. 2007. Boltzmann Machines. Scholarpedia 2, 5 (2007), 1668.
[12] Warren S. McCulloch and Walter Pitts. 1943. A Logical Calculus of the Ideas
Immanent in Nervous Activity. The bulletin of mathematical biophysics 5, 4 (1943),
115–133.
[13] IBM Research. 2015. The Music of Machines. Retrieved August 14, 2018 from
https://researcher.watson.ibm.com/researcher/view_group.php?id=6376
[14] IBM Research. n.d.. IBM Watson. Retrieved August 18, 2018 from https:
//www.ibm.com/watson/
[15] Mark A. Runco and Robert S. Albert. 2010. Creativity research: A historical
view. In The Cambridge Handbook of Creativity, James C. Kaufman and Robert J.
Sternberg (Eds.). Cambridge University Press, Cambridge, UK, 3–19.
[16] Jürgen Schmidhuber. 2015. Deep learning in neural networks: An overview.
Neural Networks 61 (2015), 85–117.
[17] Oscar Schwartz. 2018. You thought fake news was bad? Deep fakes are where
truth goes to die. The Guardian (12 Nov. 2018). Retrieved December 2, 2020
from https://www.theguardian.com/technology/2018/nov/12/deep-fakes-fake-
news-truth