Representing Musical Knowledge in a Jazz Improvisation System
Damon Horowitz
MIT Media Lab
20 Ames St.
Cambridge, MA 02139 U.S.A.
damon@media.mit.edu
Abstract
A computer system for analyzing and generating
improvisational jazz solos is currently under
development. I maintain that an enumeration and
quantification of musical common sense concepts
is necessary for the construction of musically
intelligent systems. My design of an architecture
to organize and manipulate musical knowledge is
influenced by the agent-based approach of [Minsky
86], and the spreading activation networks of [Maes
89]. A model of improvisation based upon this
approach is described, focusing on the development
of a knowledge representation and control structure
for this domain.
1 Introduction
A Generative Theory of Tonal Music [Lerdahl and Jackendoff
83] discusses the intuitions of an experienced listener in a
musical genre. These musical intuitions are described
largely by preference rules with which structures are
extracted from a musical surface. One reason why this work
is significant is for its attempt to detail aspects of musical
comprehension which are recognized as being matters of
common sense, but which have not been explicitly
enumerated previously.
Unfortunately, the motivation for GTTM is largely
neglected by computer music systems for performance and
composition. There have been systems which employ some
of the GTTM rules for surface analysis, but the more
fundamental impetus behind this work has been lost: the
need to make explicit the common intuitions of humans
experienced in a genre has been ignored. I suggest that this
is the underlying cause for the limitations of current
systems.
I share Rowe’s frustration with previous systems’
“algorithmic inability to locate salient structural chunks or
describe their function” [Rowe 93]. Generative computer
music systems are frequently noted for several categories of
shortcomings in their behavior: their inability to perceive
what is salient, and to respect this in production; their
inflexibility in dealing with novel input; and their lack of
sophistication in producing increasingly complex output.
Essentially, these systems are not equipped with human-
level musical intelligence. Interpreting instances of music
as artifacts of cognitive processes, my approach incorporates
ideas from traditional and behavior based artificial
intelligence to equip the computer program with explicit
knowledge about musical structures by developing
mechanisms which parse, organize, manipulate, and produce
these structures.
In order for music programs which emulate human
musical abilities to behave in an increasingly intelligent
fashion, they must share humans' common sense
understanding of music. The long term goal is for the
system’s knowledge representation to correlate as closely as
possible with our ideas about music, insofar as we can
formulate descriptions of our understanding of music.
Concepts such as "phrase" and "variation" must be
represented in terms of the interconnected mechanisms that
give rise to their production and recognition. These
mechanisms must demonstrate the properties that musical
perception and cognition research have revealed, while also
suggesting methods by which new musical materials may be
created. I consider a representation of musical common
sense concepts combined with the mechanisms that
manipulate them to be the basis of musical intelligence.
As an example of this approach, a model of analyzing
and generating jazz improvisations is presented. 1 Section 2
provides background relevant to this work. Section 3
describes my theory of how improvisational behavior can be
produced. The development of this model has two main
components. First, the model provides an example the
types of information about music that I suggest should be
explicitly represented in an intelligent system. Second, the
model describes an architectural framework, inspired largely
by the work in artificial intelligence by Minsky and Maes,
within which this information can be organized and used in
the service of intelligent behavior.
2 Background
The two major disciplines from which this work inherits are
the study of music and the study of intelligent processes.
The fields of music perception and music cognition seek to
identify and experimentally verify properties of human
musical processing. Particularly useful here are the results
found regarding mental concepts of tonality, rhythm, and
1 The question of whether or not a system is musically
intelligent in the sense that humans are is best pursued in an
familiar genre where its output can be judged and compared to
human output; all too frequently, the opposite approach is taken
and the curious behavior of a system which is called intelligent
is cloaked behind the mystique of an unfamiliar new genre.
melody [Bharucha 94; Desain 92; Dowling 94; Krumhansl
90]. These results suggest models for representations that
reflect musical features with proven psychological salience.
Within music theory, grammatical parallels between music
and language have been explored in [Lerdahl and Jackendoff
83], which suggests cognitively motivated and linguistically
conventional methods for the extraction of structural
information from a musical surface; several of these
segmentation and grouping rules are directly implemented in
this project. [Lidov 75; Narmour 90] provide other
contemporary concepts in general music theory which have
influenced the high-level ideas for this project and are
partially reflected in the implementation. Less general
works, specific to the jazz genre, are used for their the
enumeration of syntactic conventions which a system must
explicitly understand and respect [Amadie 90; Mehegan 59],
as well as for qualitative descriptions of jazz improvisations
[Schuler 68].
Paradigms from traditional AI which are considered here
are scripts (for solo-structuring behavior), plans (modeling
pitch trajectories over chord changes), and frames (for
organizing K-lines) [Schank 77; Agre 88; Minsky 86]. The
behavior-based approach's use of spreading activation
networks for action selection provides robustness and
flexibility unavailable from more traditional rule-based
approaches [Maes 89, 94], yet can produce some of the same
plan-following behavior. However, much of the precision
and expressive power of traditional methods is lost with
agent-based approaches, and scaling them to larger and more
cognitively challenging domains is difficult. Therefore, the
present system works with a hybrid of traditional and agent
methods [Horowitz 94]. This is appropriate because music
is a domain which presents a dynamic and unpredictable
environment combined with the need for rational high-level
structured behavior. Finally, the work in artificial
intelligence which has the largest influence over this
approach is the theory of mind described by Minsky
[Minsky 86, 89], which is elaborated upon later in this
document.
There have been several previous efforts to describe jazz
improvisation in computational terms, as well as several
computer jazz programs which attempt to consider some of
the theoretical issues posed by the problem. [Johnson-Laird
91, Pressing 85, Sudnow 78] present fundamental ideas
about the nature of cognitive processes producing
improvisations, suggesting that much of improvisation
results from the triggering of learned behaviors (similar to
the learned patterns described in [Pennycook 93], or the
PACTS of [Ramalho 94]), and instantaneous choices
regarding whether to alter or continue a triggered path.
[Levitt 81] presents an early model for a jazz improvisation
system which is more inspired by an intention to model the
cognitive and psychological factors of improvisation, and
[Widmer 92] stands out for the explicit acknowledgment that
musical systems must be knowledge-intensive to function in
a sophisticated manner reflecting the music’s context.
Agent based approaches have also been suggested, usually in
order to separate processing for different tasks; [Riecken 92;
Rowe 93] offer more direct attempts to implement portions
of a Society of Mind architecture.
3 The Model
The model presented in this paper focuses on the style of
music performed by Louis Armstrong in his famous
recordings with the Hot Fives in 1926 (although many of
the concepts here generalize to other musical domains). The
main task for the program based upon the model is to play
improvisations in 12-bar blues and 32-bar song formats
which are similar to a target set of actual Armstrong solos
from this period.2 This task includes the ability to listen to
solos as well, in order to refer to the musical material of
other players when improvising and to exchange
improvisatory phrases with another player. With this task
in mind, the goal of the model is to functionally describe the
musical style in terms of processes that create and parse it.
The model proposed here provides a knowledge
representation and control structure which can perform these
analysis and generation tasks. Thus, this model serves as a
theory of how improvisational behavior can be produced.
The resulting program will allow users to investigate
the workings of the model by adjusting state variables in the
system (variables with meaningful semantic labels, intended
to indicate the mental state of the computer performer) to
produce changes in the output behavior that remain within
the style but reflect the change in the program’s disposition
(including qualitative concepts such as emotion, as well as
more technical concepts such as submission to conventions
or use of favorite figures). This is the most simple mode of
interaction with the program, allowing clear evaluation of
the theory behind it; however, any program which
explicitly represents common sense concepts can use these
as hooks to interface elements for a wide variety of
applications.
The following sections provide a summary discussion
of the model for this program. The implementation of the
system is currently underway in C++, using mix-in classes
for the shared functionality of different agents. The program
is being written on top of a scheduler that I developed using
Apple’s MIDI Manager. An initial demonstration version of
the system will be completed by summer 1995. A more
detailed description of the model will be available at that
time as part of the documentation of the working system.
Global Description
The general task of “soloing like Armstrong” has been
restricted in order to make an implementation feasible. The
program is provided with the structure and main melody of
the songs it will play, much as a musician is given a lead
sheet, or is expected to know the chord changes of standard
songs. The program is not responsible for determining
where the beat is (beat-tracking is a difficult problem by
itself, addressed elsewhere [Desain and Honing 92]); a
daemon runs concurrently with the program and updates the
program’s conception of its position in the piece’s form (so
2 Trademark aspects of Armstrong’s playing style such as his
innovative use of the trumpet and timbre are not considered
here; I consider only MIDI information such as pitches and
rhythms here. The inventiveness of his improvised tunes
provides sufficient material for study.
 Goals and Emotions Melodic Concepts
reductions
Context
structures
meter
harmony Low-level Features
pitch
Input Solo Line
Figure 1: components of the program.
that the program always knows the metric and harmonic
context, where the “breaks” are, etc.). The program listens
only to the single-line improvisations of the other soloists,
and produces single-line improvisations in its turn; both of
these lines are represented as MIDI streams, and the
operations happen in real-time. Finally, the current system
has no provision for retaining learned material beyond a
single session; it begins each session knowing only those
musical lines, figures, and concepts which are part of its
database. In principle, the current architecture is capable of
supporting an extension which retains new musical and
conceptual material over time.
The central portion of the model is the representation of
melodic concepts, the types of musical ideas that the
program knows about and can therefore create and recognize.
This representation is used by the procedures that analyze
and generate melodic lines. These procedures make use of
the following additional modules in the program (Figure 1):
a set of mental state variables, which represents the
program’s current set of goals and emotions, as well as the
levels of activation of concepts in the concept network; a
motor-action module, which takes as input short fragments
of textural descriptions of a line by the low-level pitch and
rhythm agents and produces the actual output notes;
routines for updating the current harmonic and metric
context (e.g., spreading activation to indicate an approaching
chord change); a database of melodic lines and figures that
correspond to the wide set of tunes, sections of
improvisations, etc., that commonly occur in the solos of
this period (each musician is familiar with a set of phrases,
and also has his own “bag of tricks” that are repeatedly
used); and the set of melodic lines and figures that have been
played during the current song and are therefore active ideas
in the environment. These elements constitute the song-
environment or context that the program knows about when
analyzing and generating.
Representation of Music
I propose that multiple representations of musical material
are necessary for flexible and sophisticated analysis and
generation. My perspective in developing mechanisms
which can handle the large amount of information required
for musical intelligence is largely influenced by the theory
of mind described in Society of Mind [Minsky 86].
Minsky suggests that intelligent behavior can arise from the
interaction of many specialized smaller agents; each of our
Database of Melodic Lines
contours
Lines Active in Song
groupings
Motor-Action Module
rhythm
Output Solo Line
high-level actions and thoughts is described as requiring the
activation of many different kinds of representations and
processes with localized information. In this scheme,
concepts are represented in memory by K-lines, which are
recorded state values of different agents. For example, an
apple is represented by a set of K-lines, each of which
activates agents of color, taste, shape, weight, etc. to be in a
particular state.
In my model, musical concepts are represented in the
model by specialized agents. Each type of concept has an
agent that describes it, listing: its components and their
relations and qualities; the preconditions for its occurrence,
and the postcondition effects of its occurrence; the
parameters which determine the range of different
instantiations of the concept; and pointers to higher-level
agents for which this concept plays a role. By separating
concepts into agents, each agent can be an expert in a single
specialized and simple task; it need only be aware of that
information which relates to its concept, thus insulating it
from the complexity of dealing with unnecessary
information. Each agent is able to detect the occurrence of
the concept that it corresponds to through sensors, and also
to effect an occurrence of the concept through actuators.
For example, an agent for ornamentation of a pitch lists
the different types of ornaments which are possible
(described in terms of the parameters that distinguish them),
has sensors that parse streams of pitches to locate these
ornaments around an important pitch, and can produce
rhythm and pitch textures that realize ornaments of a given
pitch. Each agent is sensitive to context in that the
behavior of each of the mechanisms that works with each
type of information is influenced by the settings of other
agents. In this example, the ornamentation agent is told
which pitch to elaborate, and then in turn tells the rhythm
and pitch line agents how to create a specific ornament.
The agents are arranged in overlapping hierarchies
according to the nature of the concept that they represent.
The main axes of these hierarchies correspond to the
following features: the effect of the concept, what it
conceptually opens or closes, its emotional quality, etc.; and
the level of abstraction of the concept (from the musical
surface to abstract concepts, and also from small chunks of
time to larger forms), e.g., a concept that handles lists of
notes, contains contours of simple features, depicts an
abstract relation between chunks of music, etc. Thus, the
set of musical concepts that the program knows about are
arranged in a large interconnected network, linked both by
their positions in the hierarchies and by their relations as
components and pre-/post-conditions of each other.
Each concept also has an activation level, a number
which reflects its presence in the musical material when
listening, and its influence on the musical material that is
being generated. Activation is spread through the network
to guide analysis and generation. When a concept is
recognized during the analysis of input, the agent associated
with that concept becomes more active, corresponding to the
intuitive notion of semantic priming used in cognitive
science and linguistics. The more active agents are then the
first to be considered when examining new material. When
a concept is determined to be appropriate to occur in a
generated improvisation, it is also given higher activation;
the degree of activation then determines the influence that
each agent has over the output. In addition, each of the
musical lines in the known database and each of the active
musical figures in the song-environment also has a level of
activation, corresponding to the degree to which it is
recognized in incoming music or the degree to which it is
considered in generating music. Activation levels are
propagated between the figures and the concepts which
constitute the figures; this completes the intuitive model of
specific instances of figures priming the general concepts of
the figures, as well as the reverse process in which a general
concept primes the specific figures that are instances of it.
The following section describes the types of concepts that
are thus represented in the system.
Musical Concepts
The musical concepts represented by the system are those
which describe a musical phenomenon in terms of its
composition of smaller musical structures. While there are
a variety of conflicting interpretations about how humans
represent music, there are several types of structures which
seem to be generally and uncontroversially acknowledged as
contributing to our cognition of music. On a basic level,
features of notes (their relative emphases, articulations, color
[harmonically], register, etc.) must be recognized and
recorded as contours of changing parameter values over time
[Dowling 94]. These features also imply expected
continuations, as suggested by [Narmour 90]. The next
level of structure consists of groupings of events (notes and
sets of notes), based upon similarity of features. These
simple levels of comprehension are necessary for musical
tasks, and are producable by mechanisms which exist for
general purpose human cognition.
Analytic reductions of music are higher-level musical
structures; these are interpretations through which some
notes can be heard as serving subsidiary roles to others. In
particular, some events are said to be elaborations or
ornaments of major events, or are said to lead up to or
prolong major events, recursively through larger sections of
music [Lerdahl and Jackendoff 83]. This is relevant here in
so far as it relates to human comprehension and memory of
a line of music; instead of simply memorizing a list of
timed pitches as such, a musician can remember the major
events and the ways in which they are elaborated. Forming
groups and reductions requires perspectives on what in the
musical surface is perceived as standing out or being
“marked for consciousness”, in terms of the coloration of
each event and its perceived accent given the roles that it is
serving. For example, we might say that the first note of a
phrase is the accessor through which we start replaying the
phrase in our heads, but it is really the first foundational
tone that we remember as being what the phrase is about in
a reduction, while we also focus on the particular elaborating
tone that gave the phrase its distinctive quality.
Maintaining these different perspectives is useful for both
generating and analyzing new material.
These concepts are filters through which we can
examine a phrase in order to determine its qualitative effects
and its similarity to other phrases from different
perspectives. Identifying these levels in the musical surface
allows us to have a notion of a phrase that is richly
interwoven with our general knowledge about types of
phrases. It is through these devices that the concept of one
phrase referring to another (either within the solo, or from a
separate tune) is realized.
Melodic Line
Different agents capture different features of a melodic line.
Actual pieces of a line are stored as a collection of links to
the states of agents in the network. In the terminology of
SOM, a fragment of music is a set of K-lines which record
the state of different agents as they reflect the music, and can
later reactivate the agents to assume the same state. Each
agent can be seen as functioning as a feature-space onto
which a section of music is projected, thus focusing upon
some subset of its features. The state of the agent when
considering this piece of music is the set of parameter values
and pointers to other agents that it uses to understand (parse)
the music. In sum, a melodic line is represented as the set
of perspectives on the line contained by the different agents
in the system.
For example, the multiple representations of a simple
motive (a list of timed notes) are the set of K-lines reflecting
its rhythmic, melodic, and harmonic components (Figure 2).
The rhythm agent determines the level of activity (which is
roughly similar to pulse level), the amount and type of
syncopation, and the prominence of other cycles (e.g., a
crossrhythm or polyrhythm) to find salient rhythmic figures.
The melody agent looks at the pitch contour and the
abstracted simplified lines (prolongation and timespan
reductions) for key anchor points. The harmony agent
determines the hierarchical relations of tension and
resolution pitches given the harmonic framework of the
song. Each of these agents then has its own representation
of possible groupings, and points of emphasis. Finally,
there is a structural agent, which examines the patterns
present in each of the specific agents' representations (by
querying them) and indicates places of repetition or simple
variation of structure. One of the strong features of this
model is that it allows this approach to maintain the
integrity of having each agent manage its own information
in its own way; for example, each agent has different criteria
for what constitutes a similar repetition or a slight variation.
The key point is that the system has multiple
representations of a melodic line; this is appropriate
becauseof the fact that different functions require different
types of information about a line, since different tasks
 reduced melody: F g
rhythmic activity:
repeated structures:
Figure 2: a melodic line from three perspectives represented in the model.
operate from different perspectives.
The benefit of this type of representation is that a single
musical line can be manipulated in different ways depending
upon which aspects of it (i.e. which agents’ features) are
considered. A simple example of this is the separation of
rhythmic and pitch information about a line; a more
complex example is the separation of a prolongation
reduction perspective from the set of local elaboration
figures that are used. Further, since the agents themselves
exist in hierarchies of types of concepts, a given line can be
seen as a specific instantiation of general abstract types of
concepts. This abstraction allows for comparison of
musical figures according to their types along abstract
feature axes, and provides a rich set of metrics for
determining their similarity or relation (information which
is necessary for understanding music, and also used in
generating variations).
Given these types of representations of the musical line
itself, specifically of the structures of phrases, parts of
phrases and groups of phrases related to context over time,
several phenomena related to our perception, consideration,
and creation of music can also be modeled. The general idea
of chaining musical ideas, or of having musical associations
or priming of categories, is represented in this model by
spreading activation of ideas through hierarchies of types of
structures defined above. This leads directly into the model’s
description of generating music, which is summarily
described as follows: goals and intentions spread activation
to concepts which realize them (downward through a
network), while at the same time the currently active figures
and structures in the environment spread activation to their
related concepts (up through a network), and the concepts
with the highest activation are the ones which are realized in
the generated music. The following sections describe the
listening and generating functions of the program. These
functions are the inspiration for the general representation
scheme outlined above, which is designed to accommodate
the types of manipulations of information that analyzing and
playing require.
Listening
The program’s task when listening is to identify, in an
incoming stream of notes, the musical concepts which the
system knows about. In other words, the activity of
listening consists of building instances of the
representations discussed above. This is accomplished
through the use of the sensors on each type of
representation; the sensors detect a feature in the input (or
E D a C
in the agents conceptually beneath it) that is a component of
the concept it represents. In this way, the different agents
piggyback on each others’ discoveries of features in the
input. For example, the low-level rhythm agent parses the
input into notational rhythm values, which are then grouped
by a rhythm-grouping agent, and examined by a cycle-agent
for the presence of crossrhythms.
High level agents represent musical form, a concept
referring to any level of structural repetition. This concept
identifies the repetition of similar phrases and the repetition
of a figure within a phrase. These agents recognize
repeating patterns in the music by looking for patterns in
the other agents’ parses of musical lines into features.
Rather than examining each agent’s state over time, the
form-agents poll the other agents, requesting to be notified
when a repetition has occurred. Each individual agent
locates simple patterns in its type of feature by itself,
similar to the phenomenon suggested by SOM as time-
b l i n k i n g . The form-agents look for repetition in
conventional places (e.g., corresponding to metrical
divisions), but can also be activated if an agent determines
an unconventional repetition, which in turn will cause the
form-agents to suggest to the cycle-agent that a crossrhythm
may exist. Again, the abstract hierarchies present in the
representation are useful here; patterns which are not literal
repetitions in the music surface but are direct conceptual
repetitions can be easily detected with this scheme.
The other major activity of listening is the spreading of
activation between concepts. This models the experience of
priming and expectation -- predicting future material based
upon the past. The representation of concepts containing
sequential events lends itself to a direct implementation of
this idea: when a given agent’s sensors have noticed the
presence of a component which begins a sequence in a
concept, the agent primes the appropriate feature detectors
(either sensors or lower agents) to expect the next
components in the sequence. For example, an agent for
tonal resolution expects a tonic after detecting a leading
tone, and thus the pitch agent is set to look for the
appropriate note. This corresponds roughly to the
description of language understanding in SOM. The input
of a word (here, a musical “word” is an event for an agent at
any level) activates a set of frames which are interested in
this word, either semantically (e.g., an apple activates ideas
about eating) or in terms of conceptual dependencies (e.g.,
the origin role of a trans-frame). These correspond
respectively to spreading of activation to more generalized
concepts from the parsed one, and following a sequential
figure which expects later roles to be filled in. A failure of
the sensors to find the expected event requires extra
processing to recommence the parse of the input. The
occurrence of this frustration, at different levels of severity,
is a key component to Meyer’s theory of emotion in music
[Meyer 56]. The system implements this theory literally by
changing the emotional state variables of the program when
these frustrations occur. The listener thus indicates both the
comprehensibility of input (in terms of its susceptibility to
each agent's attempts to analyze it) and also its qualitative
interest (each agent can subjectively label the local
phenomena it encounters).
The final result of the listening process is that the input
musical line is memorized as part of the active environment
during the song. It remains in this buffer, accessible for
further listening and playing processes, until the conclusion
of the song. The active musical ideas already in the
environment are compared against the incoming line. A
match can be used to identify the higher-level structural
forms being played. That is, the system can recognize if
material (conceptual or literal notes) from its set of ideas is
being quoted or referred to, and can then label the current
setting appropriately (e.g., as an elaboration of the song’s
tune, or as a series of variations on the previous solo’s
closing phrase). At no point is it necessary to decide upon a
single group parsing or reduction interpretation of the input,
since all of the active agents’ states are stored as a set of K-
lines that represents the figure. However, the relationship
between consecutive phrases (or between any phrase and its
original referent) indicates which aspects of the
representation, and thus which interpretation, are focused
upon in a given instance. These sequences of relations (such
as the maintenance of a reduced melodic line or of rhythmic
figures) are then stored as the high-level structural form
describing a script that the input is assumed to have
followed. Using the original solo’s relations between its
phrases is a technique for creating a similar set of phrases in
a new solo, leading to the effect of having played the same
“sort of thing” or in the same “style”.
Generation
Generation of an improvisational line is viewed here as an
action selection problem. I assume that an improvisation
does not follow a simple set of rules, but rather is influenced
by a variety of sources concurrently. This type of decision
making is well modeled by a spreading activation network
which responds to both goals and the environment [Maes
89]. My model uses a hybrid system in which actions are
chosen by competition through spreading activation
combined with traditional AI structures of rule-based
constraints and script-following fragments. The actions in
the network are the hierarchically arranged musical concepts,
each of which is a type of thing that can happen in music.
When generating a line, the program’s goals spread
activation through the network of concepts, as do the active
ideas in the song environment. As this spreading activation
is occurring, the motor module launches sequences of a few
notes (corresponding to a single learned physical gesture, a
motor “riff”) according to the lowest level agents’
descriptions (a pitch and rhythm texture). As each motor
routine is concluded, these agents are polled again to
determine what the next motor action will be. In other
words, the decision of what to do is made by simply
checking which actions look best at the time that an action
is needed; agents with the highest values among the
competing agents when polled by the motor routine are the
ones which affect the output. This corresponds to the
intuitive notion of a player selecting a sequence of small
paths that he can realize on his instrument; this approach
assumes that all actions are motivated, if by nothing else
then by the goal to just do something, or by a kinesthetic
sense [Sudnow 78]. Note that to be silent is to have the
explicit action concept of rest be more active than the
others. Here are some other examples of action/concept
descriptions which could be actively influencing the output:
launch a new phrase, choose a motive from the melody, vary
the previous phrase, repeat a partial structure, hit a high
“C”, use filler material for two beats, emphasize a structural
element, assert a syncopated rhythmic figure, conclude this
solo, etc.
The use of goals here is based upon the standard use in
spreading activation networks for action selection [Maes 94].
The goals spread to concepts that produce the qualitative
label associated with the goal: pitch/time trajectories
modeled after [Clynes 78] relate directly to emotional
qualities, as do higher-level relations between phrases (e.g.,
call and response, amplification or elaboration, contrast,
etc.). [Rieken 92] describes a system that is similarly
interested in the relationship between the effect of musical
features (his focus is on pitch intervals) and emotions in
determining generated music. In my model, the framework
allows for the labeling of any type of concept (e.g.,
structures, groupings, reductions, roles, colorations) with an
affective quality, which can vary as a function of context.
Each concept can be seen as a short script for how to
perform a certain behavior. The actuators of an agent define
the sequence of steps and types of conditions required to
realize the concept. This occurs both on a small level, as in
the case for a script ornamenting a pitch, and on a larger
level, such as a conventional script for developing material
over the course of a solo. A script for a solo is represented
in terms of form-agent structural relations between the
phrases in the solo, with the corresponding emotional effects
for each relation; this allows for a description of rising
intensity, or trajectories through “moods”, or an entire a
dramatic form played over the course of a chorus. For
example, a script for a solo could say: begin with a
statement of material active in the environment (perhaps
from the song’s main tune); focus on a perspective of this
material that corresponds to an affective label in keeping
with the program’s mental state; maintain this aspect while
creating a variation of the original statement; evolve from
this newly played material through successive variations
while rising in intensity (modeled as a degree of
exaggeration in the parameters of a perspective) towards a
climax; then close the solo, resolving any active musical
concepts.
Spreading of activation happens in several directions.
In a top-down fashion, goals spread to those mechanisms
that realize them, such as the scripts for how to behave over
the course of a solo and the treatment of conventions. In
addition, the agent for each concept has actuators which
deliver activation to the agents which realize its
preconditions and to those which represent the its sub-
components. For example, the appoggiatura concept
activates the rhythm agent and the pitch agent in synchrony
to place the expected goal tone on a less stressed pulse
following a coloration tone in the stressed position.
Bottom-up spreading occurs from those figures which are
selected to be played to their associated successors. This is a
specific case of the general spreading of activation from lines
active in the song-environment. A figure active in the
environment spreads activation to its associated concepts’
actuators at those moments when it can begin (with respect
to the position in the solo, the chord changes, the meter,
etc.). For example, a likely figure for a turnaround or the
concept of starting a solo with two similar 4-bar phrases are
each activated at particular moments. If a figure has been
started (that is, if its initial concepts correspond to those
chosen to be played), it remains active and activates the rest
of its sequence of concepts until they are not chosen by the
system. Thus, there is spreading activation from the context
of the song (serving as the environment): it both suggests
figures and concepts that can occur at a particular location
from the set of figures active in the song-environment, and
also works on a low level by urging rhythmic actions (e.g.,
spreading activation to the agent that seeks to play rhythms
in phase with the meter) and pitch choices corresponding to
the metrical and harmonic context.
This use of spreading activation is essentially a way to
combine the interests of different musical concepts that the
system wants to realize at a given point. However, it also
does allow for simple emergent planning behavior [Maes
89]; for example, the concept of doing a leap requires that
the leap start from a chord tone and thus spreads activation
to the pitch agent to effect this. However, the most
important feature of the use of a spreading activation
network here is its capacity to model the phenomenon of
chaining musical associations and ideas in the mind of a
musician. This allows exploration of high-level ideas about
how a solo evolves over time, maintains a sense of
direction, or follows a dramatic curve of intensity. By
continually referring to what it has done, a system can be
opportunistic in its composition, cascading local processes
to create larger forms.
4 Conclusion
This paper proposes an architecture for the representation of
common sense concepts about musical improvisation, of the
structures we notice when listening and intend to create
when playing. The representation models how these
concepts may be recognized and produced. Each concept is
embedded in a network of other musical concepts, so that the
meaning and consequences of concepts are defined by their
effect upon other agents, the mental state, etc. I suggest
that it is through such explicit enumeration and
quantification of musical concepts, which can represent
different and conflicting perspectives, that progress can be
made towards understanding musical intelligence.
I share Widmer’s acknowledgment of the knowledge
intensive nature of questions about musical understanding
concerning performance expression, and believe this also
applies to beat-tracking, the nature of swing or of a
“groove”, the type of interaction between two soloists, etc.
These questions can be better pursued on top of a framework
of common sense knowledge about music. To program this
knowledge is a difficult task. Towards this end, I have
created an architecture based on ideas from Minsky and
Maes, and a specific model of a task in a restricted genre.
The proposed computational model of improvisation
reflects my approach to building intelligent music systems.
The use of multiple representations, consisting of K-lines
activating agents, combined with a spreading activation
network, allows for an interesting model of the chaining of
musical associations in the mind of a performer. As
different aspects of what is being played (the focus of
different agents) are given more attention, the active agents
can emphasize and expand this type of information in a
melodic line. The spreading activation network can model
both a temporary obsession with a particular melodic idea,
as well as its eventual fatigue. The desired result is that an
improvisation can develop a sense of direction over time
based upon the stringing together of fragments of the
representations of active musical materials. The limitations
of the current model include the absence of an ability to
learn and the restriction to a specific genre. Further
evaluation of the model and the general approach awaits
completion of the implementation.
References
Agre and Chapman (1988). “What are Plans for?”. A.I.
Memo #1050. MIT.
Amadie, J. (1990). Jazz Improv. Thornton Publications,
Pennsylvania.
Bharucha, J. (1994). “Tonality and Expectation”, in
Musical Perceptions. Oxford University Press, New York.
Clynes, M. (1978). Sentics: The Touch of Emotions.
Anchor Press, Garden City.
Desain, P. and Honing, H. (1992). Music, Mind, and
Machine: Studies in Computer Music, Music Cognition,
and Artificial Intelligence. Thesis Publishers, Amsterdam.
Dowling, W. (1994). “Melodic Contour in Hearing and
Remembering Melodies”, in Musical Perceptions. Oxford
University Press, New York.
Horowitz, D. (1994). “A Hybrid Approach to Scaling
Action Selection”. Unpublished paper, MIT Media Lab.
Johnson-Laird (1991). “Jazz Improvisation: A Theory at the
Computational Level”, in Representing Musical Structure.
Academic, London.
Krumhansl, C. (1990). Cognitive Foundations of Musical
Pitch. Oxford University Press, New York.
Lerdahl, F. and Jackendoff, R. (1983). A Generative
Theory of Tonal Music. MIT Press, Cambridge.
Levitt, D. (1981). A Melody Description System for Jazz
Improvisation. Masters Thesis, MIT, Cambridge.
Lidov, D. (1975). On Musical Phrase. Groupe de
Recherches en Semiologie Musicale, Universite de
Montreal.
Maes, P. (1989). “How to do the Right Thing”, in
Connection Science Journal Vol. 1, No. 3.
Maes, P. (1994). “Modeling Adaptive Autonomous
Agents”, in Artifical Life Journal Vol. 1, No. 1 and 2. MIT
Press, Cambridge.
Mehegan, J. (1959-1965). Jazz Improvisation, in four
volumes. Watson-Guptill Publications, New York.
Meyer, L. (1956). Emotion and Meaning in Music.
University of Chicago Press, Chicago.
Minsky, M. (1986). The Society of Mind. Simon and
Schuster, New York.
Minsky, M. (1989). “Music, Mind, and Meaning”, in The
Music Machine. MIT Press, Cambridge.
Narmour, E. (1990). The Analysis and Cognition of Basic
Melodic Structures: The Implication-Realization Model.
University of Chicago Press,
Pennycook et al. (1993). “Toward a Computer Model of a
Jazz Improvisor”, in International Computer Music
Conference 1993 Proceedings, ICMA, California.
Pressing, J. (1985). “Experimental Research into Musical
Generative Ability”, in Generative Processes in Music.
Oxford University Press, New York.
Ramalho, G. and Ganascia, J. (1994). “Simulating
Creativity in Jazz Performance”, in Proceedings of the
Twelfth National Conference on Artificial Intelligence,
Seattle, WA.
Riecken, R. (1992). “Wolfgang: A System Using Emotion
Potentials to Manage Musical Design”, in Understanding
Music with AI. MIT Press, Cambridge.
Rowe, R. (1993). Interactive Music Systems. MIT Press,
Cambridge.
Schank, R and Abelson, R. (1977). Scripts, Plans, Goals,
and Understanding. Erlbaum, Hillsdale, New Jersey.
Schuller, Gunther (1968). Early Jazz: Its Roots and
Musical Development. Oxford University Press, New York.
Sudnow, D. (1978). Ways of the Hand: The Organization
of Improvised Conduct. Harper and Row, New York.
Widmer, G. (1992). “A Knowledge Intensive Approach to
Machine Learning in Tonal Music”, in Understanding Music
with AI. MIT Press, Cambridge.