Appeared in the Proceedings of Creativity and Cognition 4, 2002

Supporting Musical Compositionby Externalizing the Composer's Mental Space

Shigeki Amitani

Department of AdvancedInterdisciplinary StudiesUniversity of Tokyo+81 3 5452 5289amitani@ai.rcast.u-tokyo.ac.jp

Koichi Hori

Department of AdvancedInterdisciplinary StudiesUniversity of Tokyohori@ai.rcast.u-tokyo.ac.jp

ABSTRACT

In the field of design and creativity support,"externalization of mental space" has been recognized as animportant challenge. In this paper, we tackle this challengeon musical composition, which is one of the importanthuman creative activities. In our research, we focus on theanalysis of the cognitive process in musical composition.That is, we analyze what the cognitive process in musicalcomposition is like and how the process is affected whenthe representation of information is changed. For thisresearch, we propose a musical composition supportingsystem named "MACSS (MAcroscopic CompositionSupporting System)" which offers a spatial representationof music to the composer to support his/her compositionprocess. Comparing with ordinary musical editors, whichgive chronological (1-dimensional, score-metaphored)representation, this system is expected to provide amacroscopic view by locating phrases on a 2-dimensionalspace. We have investigated how the cognitive processchanges when the spatial representation is introduced, i.e.,the representation of information is changed. Moreover, wehave analyzed the observed change microscopically,especially the process of "mental fixation" and "mentalleap". We have found how the spatial representationtriggers mental leap escaping from mental fixation inmusical composition.

Keywords

Creativity support, musical composition, externalization,mental space, cognitive process, spatial representation

INTRODUCTION

In the field of creativity support, it is important toinvestigate how the change of external representation of information gives effects on human cognitive processes increative activities. In this paper musical compositionprocess is selected as one of the important creativeactivities.Norman [1] claims that cognitive artifact should supportour activities by amplifying our reflective thinking. Itmeans that tools are required to provide the user withproper information in a proper way of representation whenthe user is engaged in a creative activity.In our research, we focus on the analysis of the cognitiveprocess in musical composition. Tanaka [2] points out thatfewer analysis on musical composition have beenconducted compared to those on the cognitive processes of musical listening and performance, and that most of themare macroscopic analysis through flashing an idea tocompletion of the musical piece. That is the reason wefocus on detailed analysis of composition process.

•

How composition process proceeds with an ordinarymusical editor

•

How the human cognitive process in musicalcomposition is affected when the representation of information changed.For this research, we propose a musical compositionsupporting system named "MACSS (MAcroscopicComposition Supporting System)" which offers a spatialrepresentation to the composer to support his/hercomposition process. Comparing with ordinary musicaleditors, which provide chronological (1-dimensional, score-metaphored) representation, this system is expected toprovide a macroscopic view of pieces by locating phraseson a 2-dimensional space. In the context of musicalcomposition, the term "macroscopic view" means to grabthe whole of the musical piece being composed or to grabthe relationship among the pieces (phrases) the composerhas composed. The observed change was analyzed in detailand described, especially the process of "mental fixation"which means the deadlock of the thought and "mental leap"which means to escape from the deadlock of the thought.We observed how the spatial representation triggers mentalleap escaping from mental fixation in musical compositionmore frequently than chronological representation.

RELATED WORKS

Schoen [3] explained the design process as "Seeing-Drawing-Seeing Cycle". Design process starts from"Seeing", proceeds to "Drawing", or generating(externalizing) the partial solution to the design problem,returns to "Seeing" and repeats this cycle, which is shownin Figure 1. This process can be regarded as a "dialog"between the designer and his/her material. Usually thisprocess is ambiguous and dealt in the designer's mind. It ispossible to put this design process into concrete by"externalizing the designer's mental space", which enablesthe designer to grasp his/her whole of mental space. It isexpected that this method can support the creative activities[4][5]. This process is shown in Figure 2.Zhang [6] proposed a theoretical framework for the

Appeared in the Proceedings of Creativity and Cognition 4, 2002problem-solution process based on the externalrepresentation.Figure 1

Seeing-Drawing-Seeing Cycle

Figure 2

A Cycle of Design Process with Externalizing theDesigner's Mental Space

In Zhang's research, he conducted his experiment with agame called Tic-Tac-Toe and its isomorph. The experimentdemonstrated that the actions in the problem-solutionprocess were determined by the information directlyavailable to human cognition or perception. Based on thisexperiment, Zhang suggested the representationaldeterminism, that is, the form of representation determineswhat information can be perceived, what processes can beactivated, and what structures can be discovered from thespecific representation.Suwa et al. [7] devised a scheme for coding designer'scognitive actions from video/audio design protocols.Designer's actions were coded into four cognitive levels(i.e., physical, perceptual, functional and conceptual). Therelations between each cognitive process weresystematically described and analyzed. Through thisanalysis, Suwa concluded that the interaction between thedesigner and his/her sketches provides the designer withsuch effects as re-interpretation of the design problems andunexpected discoveries. This method is useful to code andto analyze human cognitive process.

The Spatial Representation as a Way of ExternalizingMental Space

In this research, the spatial representation is adopted as away of externalizing mental space. Several studies havebeen conducted which employed the spatial representationfor externalization.Shipman et. al. [8] investigated the significance of thespatial representation. He concluded that the significance of the spatial representation is to represent mainly thesimilarity among information with the proximity of them.Their system aimed to support the process "incrementalformalization", that is, the user gradually structures theconcept.Hori [5] conducted the experiment for the effect of thespatial representation on the conceptual design. The systemnamed AA1 presents words, which represent the conceptthe user vaguely conceives, on the two-dimensional space.The user can change the location of the words on the space.This action helps the user clarify his/her concept graduallyand such phenomenon was observed as the user came togenerate new concept by looking at the blank area on thespace.Yamamoto et al. [9] presented the theoretical framework for the method of support with an interactive system at theearly stage of the creation of information. The interactivesystem aimed to support the process to create information,which is still ambiguous in the user's mind through trial-and-error. They concluded that it is not "the representationfor the design solution" but "the representation forunderstanding the design solution" that the designer needsat the early stage of the creation of information. That is, thespatial representation can support to clarify gradually thedesigner's concept that was ambiguous through theinteraction between the designer and the system in thedivergent thinking process.From these studies, the spatial representation can support tostructure and clarify the designer's ambiguous conceptgradually through the interaction. Furthermore, it can helpgenerate new idea.

Information with "Linearity" and "Gestalt"

Reybook [10] examined the limitations and possibilities of application of Gestalt concepts in relation to music. Theresearch was conducted mainly on the cognitive process of listening music, but the assertion that music analysis has tobe broadened from a structural description to a descriptionin terms of processes must be taken into consideration toanalyze the cognitive process of musical composition.Yamamoto et al. [9] also referred to "linearity" and"Gestalt" in the semiotics and they attempted to presentwith the system Gestalt to the linear information, i.e.,document in the case of their research, simultaneously.They concluded that this approach allows the designer(writer):

•

To inspect the information in detail with the linearity

Appeared in the Proceedings of Creativity and Cognition 4, 2002

•

To grasp the information from the bird's-eye viewwith GestaltThis semiotics approach can be applied to music that is alsoinformation with linearity. As well as the documentcomposition, the spatial representation that might provideGestalt allows the designer (composer) to grasp the wholestructure of the music piece, and moreover, thisrepresentation can support the musical composition process.In our research, it is attempted to introduce Gestalt conceptin musical composition process with a spatialrepresentation, which is appropriate to grasp the relationamong the objects. Introducing this representation, it isexpected that the composer use both spatial and score-metaphored representation, i.e. he/she uses a spatialrepresentation (Gestalt) to grasp the information frombird's-eye view and uses score-metaphored representation(linearity) to inspect the information in detail.

Research on Musical Composition Process

A few researches on the composition process have beenconducted in the music psychology. Sloboda [11] describedthe composition process in Figure 3.Though Umemoto [12] also presented macroscopic analysislike Sloboda's, as Tanaka [2] mentioned in his paper, theproblem in this field is that almost no research has beenconducted to investigate what the each process is like andwhat the transition among processes is like. In this paper,we investigate the cognitive process in the musicalcomposition microscopically with protocol analysismethod. It is expected that the analysis provide us with therich knowledge for designing a creativity support system inthe future. In the field of creativity support, as few practicalresearches have been conducted it is necessary to analyze alot of cognitive processes through various instances. As ourapproach to this challenge, the musical compositionprocess, which is one of the representative and importanthuman creative activities, is selected.The "mental space" in this research refers to the part named"A: idea" and "B: theme" in "CONSCIOUS" area in Figure3. It is expected that the spatial representation decrease thecognitive load when the composer grasps his/her unfinishedphrases and offers macroscopic view that the ordinarychronological editor does not provide.

MACSS (MAcroscopic Composition Supporting System)

MACSS aims at providing the composer with the ability tograsp whole of his/her piece. That is, MACSS provides the"macroscopic view". Figure 4 shows the screen shot of MACSS. MACSS arranges the phrase objects (MIDI filesonly) on the two-dimensional space by Multi-DimensionalScaling method (MDS) based on the calculated similarityamong the phrases. In the music psychology field, it is saidthat the similarity of phrases is grasped mainly based on thethree musical characteristics, such as melody contour, keyand rhythm [13]. We adopted these criteria for thecalculation of similarity among music phrases on MACSS,though there is room for further discussion to define thesimilarity [14]. MACSS was developed with JDK ver.1.3.There are composers who accumulate small phrases theyhit upon and develop or combine them to compose amusical piece. Though ordinary musical editors are usefulfor detailing a phrase, it lacks a global view of the wholepiece. MACSS provides the flexibility to reconstruct thepiece by its spatial representation. It cooperates with theordinary editor and they complement with each other.Figure 3

Sloboda's Diagram of Musical Composition Process

The composer is expected to proceed his/her compositionby exploring the "whole" and "a part" of the piece. That is,MACSS allows the composer to rearrange the phraseobjects on the space and to listen to the phrases in the orderthe composer specifies so that he/she can compose the"whole structure of the piece". On the other hand, the "EditPhrase" command allows the composer to edit the eachphrase, i.e. "a part of the piece" on the ordinarychronological editor. When the composer rearranges thephrase objects, MACSS recalculates the similarity of thephrases from the distances among the phrase objects andtunes the weight of the three musical characteristics usedfor similarity. It is expected to become easier for thecomposer to grasp the phrases in his/her long-termcomposition that can provide the composition process withthe nature of Gestalt to. Furthermore, it is also expectedsuch interaction that the composer come to generate newconcept by looking at the blank area on the space. Toenhance the nature of Gestalt, the aspect of each phraseobject is editable with "Edit Picture" command (right-click menu). It opens the picture with the external picture editorand the composer can put the image on the object. Forexample, "This color expresses the atmosphere of thisphrase", "This picture can be the jacket of this piece", etc.This function was added because one of the composers whoparticipated in this experiment wished.

The Representation of Music on a Computer

Hiraga [15] proposed Functional Representation of Music(FRM) as a representation of music on a computer. Wehave partly adopted this representation to implement thesystem. Hiraga also suggested the representation of similarity of musical phrases based on such formalelements as melody contour, contrary motion, retrograde