You are on page 1of 10

RodneyBerryTangibleRepresentationsComputationalCreativity

26June2009

TANGIBLEREPRESENTATIONSAND
MODESOFCREATIVEENGAGEMENT
RodneyBerry
CommunicationsandNewMediaProgramme
IDMIArtsandCreativityLab
NationalUniversityofSingapore
rodberry@gmail.com

ABSTRACT
Tangibleinterfacesarefindingtheirwayintovariousareasofcreativeactivity,particularlyinthecreationofmusic.
Tangiblerepresentationsofmusicalinformation,togetherwiththeirgraphicalcounterparts,playanimportantrolein
increasinglycomplexrepresentationalsystems.Thispaperexplorestherelationshipsbetweentheserepresentationsin
thecontextofvariousmodesofcompositional(andbyextension,creative)engagementadoptedbycomposersintheir
workingprocess.

INTRODUCTION
Thepurposeofthiswritingistoidentifyandexploreissuessurroundingtheuseoftangiblerepresentationsin
thecompositionofmusic.Particularly,itisconcernedwiththeproblemofhowtoscaleandmanipulatea
tangiblerepresentation.Inthecontextofcomputationalcreativity,ourprogresstowardamorecreative
partnershipwiththecomputerwilldemandrepresentationsthatengageindifferentwaysthanarepossible
withcurrentscreen,keyboardandmouseinterfaces.Mymotivationhereistorevisitissuesuncoveredinthe
developmentoftheMusicTable,anaugmentedrealitybasedtabletopmusicmakingsystem(Berry,Makino,
Hikawa,Suzukietal.2006).TheMusicTableallowsplayerstomakeshortrepeatingphrasesbymanipulating
andarrangingobjectsonatabletopwhilehearingthemusicalresultsatthesametime.AlthoughtheMusic
Tableisfuntoplay,itshareswithmost,ifnotall,tangiblemusicalinterfacesaresistancetochangesinscale
andinlevelsofabstraction.
Inaperfectworld,wemightenjoythetangible,concreteandembodiedaspectsthatmaketangible
interfacesattractive,yetstilltakeadvantageoftheplasticityandflexibilitythatmakesapurelygraphical
representationsopowerfulandflexible.Itappearshowevertobeaninescapablefactthataphysical
representationissubjecttophysicallaws.Itisalmostimpossibletocopy,paste,stretchorsquashatangible
representationinthewaythatonemightexpecttodowithapurelygraphicalrepresentation.Althoughwe
aretalkingaboutmusiccomposition,muchofwhatisdiscussedcouldwellapplytoothercreativeactivities
whereatangiblerepresentationmightbedesirable.

Dagstuhl Seminar Proceedings 09291


Computational Creativity : An Interdisciplinary Approach
http://drops.dagstuhl.de/opus/volltexte/2009/2208

RodneyBerryTangibleRepresentationsComputationalCreativity

26June2009

Itismybeliefthatacombinationofaugmentedrealitytechniques,combinedwithaninformedapproachto
manipulatingthevariouslevelsofrepresentationinsuchasystem,canallowacomposertoattempt
compositiononalongertimescalethanispossiblewithexistingtangiblemusicsystems.Makinganew
tangiblemusicsystemthatcouldrepresentmusiconalargerscaleistheultimategoalofthisexploration.To
thisend,itisnecessarytoconsiderexistingtheoryaboutcognition,representation,composition,
embodimentandtangibleinterfacesandhowtheymightinformthedesignoftangibleinterfacesthatallowa
moreflexibletypeofinteractionandrepresentation.Tobeginwith,wewilllookmorecloselyattheprocess
ofmusicalcomposition.

ONMUSICCOMPOSITION
Thereappearstobelittleempiricalresearchoncompositionasaprocess.Sloboda(Sloboda1985)notesthat
therearebasicallyfourthingswecanlookattolearnmoreaboutcomposition,particularlyasacognitive
process,theseare:existingexamplesofscoresandworkingsketchesbycomposers;composersownwritings
andremarksabouttheirprocess;liveobservationofcomposersandanalysisofthinkaloudprotocols
resultingfromobservedsessionsandfourthly,observationanddescriptionofimprovisatoryperformance.

INSPIRATIONANDEXECUTION
Usingtermscoinedbycomposer,RogerSessions,Slobodaidentifiestwodistinctphasesasfollows:
inspirationaphasewherethecomposerfirstbecomesawareofbasicthematicideas,thenexecutiona
phasewherethemostviablebasicmaterialisextendedanddeveloped.Slobodaalsostressestheimportance
ofahistoryoflisteningwherebyacomposerdevelopstheabilitytochoosewhichrawmaterialsand
subsequentmanipulationswillbeeffective.Slobodanotesthecentralroleofnotationsofvariouskindsatall
stagesasanimportantfeatureofthecompositionprocess.Differentcomposersexhibitdifferentuseof
scores,particularlyinworkingsketches.Mozartwouldoftenwritethemelodyandbasslinesalmost
completelyinonesittingthen,judgingbythechangeofink,hewouldlaterfillinthemiddleparts.Beethoven
wouldmakeendlesslittlesketchesofworksoverperiodsaslongastwentyyears,whereStravinskywould
buildupquitecomplexverticalstructuresquiteearlyintheprocess,andwoulddevelopapieceovera
relativelyshortperiodoftime.
Slobodahighlightstheimportanceofexplorationonaninstrumentasacharacteristicoftheinspirationphase
andreferstoSudnowsdescriptionsoflearningjazzimprovisationandhowhelearnedtofindtheright
notesandpatternswithhishands.Theimportanceofphysicalmovesonaninstrumentinthecreative
processsignalstomethehiddenrepresentationalroleoftheinstrument,especiallyinthehandsofa
composer.Theinstrumentprovidesaffordancesandconstraintsthatservetorestrict,butalsotomapouta
smallsubspaceofthegreaterspaceofallmusicalpossibilitiesforacomposer.Initsrepresentationalrole,
thetraditionalinstrumentusedincompositionhasmanyconnectionstoatangiblemusicinterfacesuchas
themusictable.Firstandforemostisthewaythatitallowsforexplorationofmusicalpossibilitiesand
providesfeedbackabouttheconsequencesoftheplayer/composersphysicalmovesabouttheinstrumentor
representation.Therelationshipbetweenrepresentationandcontrolissimilarinbothtraditionalmusical
instrumentsandtangibleinterfaces.Thefretsonaguitarfingerboardservetogivevisualandtactilecluesas
towherethefingersshouldbeplacedtoproduceparticularnotes.However,itistheverypresenceofafret
atthatlocationthatdeterminesthenotethatwillsound.Intangibleinterfaces,thetangibleelementsusually
2

RodneyBerryTangibleRepresentationsComputationalCreativity

26June2009

servedualrepresentationandcontrolfunctions(UllmerandIshii2000).TheoriginalMusicTablesnote
blocksrepresent,bytheirpositiononthetable,thepositionofnotesonthepitch/timeaxisbuttheyalso
sendinstructionstothecomputertotellitwhenandhowtoplaythesenotes.

BROWNSMODESOFCOMPOSITIONALENGAGEMENT
Dividingcompositionintotwophasescouldeasilybereadassequentialstagesofadevelopmental
description.However,acomposermayinfactregularlymovefromonephaseofoperationtotheother
duringthecourseofagivencomposition.Thishintsatinspirationandexecutionbeingbetterdescribedas
modesofoperationinthecompositionprocess,eachwiththeirowndifferentstyleofthinking.Browntriesto
makethismoreexplicitbybreakingdownthecompositionprocessintoanumberofmodesofcompositional
engagement(Brown2001).WhileSlobodaisspeakingprimarilyaboutcompositionofwesternEuropean
tonalmusicbytraditionalmethods,muchofwhathesayscanalsoapplytocomputerbasedandothertypes
ofcompositionthatmakeupcontemporarypractice.Theideaoftherebeingfundamentalelementsto
composition,commonbetweencontemporaryandtraditionalpractitioners,isalsoexploredbyBrown
throughinterviewswithcontemporarycomposersabouttheirworkprocessingeneralandtheroleofthe
computerinthatprocess.
Brownidentifiesfivekindsofengagementbetweenacomposerandhisorhercompositionalmedium.The
namesreflectboththevariousrolesofthecomposerandthecorrespondingroleofthemediumineachcase.
ObserverArtefact
Herethecomposerstandsapartfromthecompositionandtriestoobserveitasanaudienceorcritic
toseeitasitisandnotintermsofitspotential.
DirectorTool
Herethecomposeractstodirectandmanipulatethecompositionalmediumasatool.Inthismode
thecomposerasdirectorisverymuchincontrolofthemedium.Thiscorrespondscloselytothe
executionphasedescribedbySloboda.
PlayerInstrument
Thismodeismoreimprovisatoryandintuitivethanconsciousordeliberateinnature.Thisiscloseto
theinspirationphasedescribedearlier.Itdoesnothavetobeonaphysicalinstrument,nordoesit
requirerealtimefeedbackfromthemedium.Thismodeofengagement,whenusingalive
instrument,canappearalotlikeimprovisedperformance.However,thewayacomposerengages
withaninstrumentinthismodeisdifferenttoaperformerengagedinperformance.Hereagain,I
seetheinstrumentitselfinarepresentationalroleasamaptraversedbythecomposer.Eveninthe
caseofcomputermusicpractice,BrownquotesLanskyasdescribinghisworkonlowlevel
programmingatthecomputerassortofimprovisinginrealslowtime.Lanskysdescriptionofhis
processcouldeasilyspillintoBrownsfourthmodeofengagement,theExplorerModelmode.
ExplorerModel

RodneyBerryTangibleRepresentationsComputationalCreativity

26June2009

Herethesystems,bothtechnologicalandtheoretical(forexample,tonalormicrotonalsystemsof
harmonyandvoiceleading,serialtechniquesandrulebasedsystems),thatformpartofones
compositionalmedium,aretreatedasaplaceinwhichtoexplore.InBrownswords,
Themediumexternalisesmusicalideaswhichcanthenbeviewedfrommanyperspectives,reflected
upon,andfurtherexplored.Themediumactsasacognitiveamplifierforthecomposerinitsabilityto
leverageideasbybuildinguponthestoredknowledgeinitssymbolsystemsandexistingprocedures,
andbycapturingconceptsandfreeingupthemindfornewthoughts.
Papertusestheterm,microworlds(Papert1980)dodescribesuchmodelsforexploration,especially
inthecontextofcomputerbasedlearningaboutmathematics.Helikensittocreatingaspecial
countrywhereapersoncangotolearnthelanguage.Hedescribesthecomputerasamathland
wherechildrencangotolearnthelanguageofmathematicsfromitsnativespeaker,thecomputer.
Atangiblerepresentationcanpotentiallymakesuchamicroworldsomethingtobeexperienced
bodily.
SelectorCreator
Thefifthmodeseesthecomposerabandoningcontroltothesystemsandprocessinthe
compositionalmedium,beitasetofgamerules,analgorithm,achanceeventetc.Themedium
takestheroleofcreatorandthecomposertakesamorecuratorialroleinselectingfromtheoutput
ofwhateverprocesshasbeensetinmotion.
Itwouldbetemptingtomorecautiouslysaygeneratorinsteadofcreatorinthiscase(Brown
actuallyusesgeneratorratherthancreatorinhisPhDthesis),butthatmightbetolabourunder
theassumptionthatcreativitytakesplaceexclusivelyinsidetheheadsofhumanbeings.Writers
suchasZhangandNorman(ZhangandNorman1994)andHutchins(Hutchins1995)placeapartof
ourcognitiveprocessoutintheworldwherewethinkwiththeaidofexternal,andoftenshared,
representations.Thefeelingofpartnershiporonenesswiththecompositionalmediumreportedby
someofBrownsintervieweescouldbeseenasaninstinctiveawarenessofthedistributednatureof
cognition.
Brownsmodesofengagementarenotintendedtobeexclusiveofoneanother,infactthecomposermight
behaveinwaysthatsuggestmorethanonemodeatatimeandrapidtransitionsfromonemodetoanother
andbackagain.

BROWNSMODESANDTANGIBLEREPRESENTATIONS
AlthoughBrownspaperisnotheavilycited,Iseemuchinhismodelthatringstruefrombothmyown
experienceasacomposerandfrommyreading.Brownsemphasisontheinterplaybetweenthecomposer
andthecompositionalmediumintermsofpairedrolespointstothekindsofactivitiesthatamusic
compositiontoolmightneedtosupport.Conveniently,thecomposersrolesyieldasetofverbs:observe,
direct,play,exploreandselect.Ibelievethatthesecanhelpframemyresearchintermsofwhichofthese
activitiesarebeingsupportedbyphysicalrepresentationsatanymomentinthecompositionprocess.

RodneyBerryTangibleRepresentationsComputationalCreativity

26June2009

TangiblemusicinterfacesliketheMusicTableofferaworkingprocessthatisdeliberatebutspontaneous
wherechangesinamusicalpatternareexperiencedasphysicalmoveswithanimmediatemusicalresult.
Suchinterfacesmightthenbemoresuitedforsomemodesofengagementthanothers.Forexample,the
PlayerInstrumentmodecouldbewellsupportedbyatangiblerepresentationsinceitistraditionally
accomplishedwithaphysicalinstrumentandtherearemanyparallelsintermsofthelevelofphysical
involvementandtheintuitivelevelofactioncharacteristicofbothinstrumentsandtangibleinterfaces.
Atangibleapproachtotheexplorermodelmodewoulddependheavilyonhoweffectivelythetangible
elementsrepresentthemodel,ormodels,beingexploredbythecomposer.Itwillalsodependonhowwell
thetangibleelementsallowcontrolofthemodelandwhetherthepathwaysforcontrolareintuitively
coupledtotherepresentation.Forexample,IwaiToshiosCompositionontheTable(Iwai1999),aswellas
manyofhisotherworks,provideaninteractionenvironmentthatprovidesamodelforthegalleryvisitorto
explore.Thetangibleaspectmaybenomorethanagridofswitchesbutitprovidesamapofthemodelin
realspaceandthehumanbodymustmoveinthatspaceinordertomanipulateit.TheoriginalMusicTable
wasasimplerepresentationofpitchandtimeasaflattabletop,soitonlysupportsthismodeinalimited
way.ThereacTable*(Jord2003),ontheotherhand,providesamorecomplextangiblerepresentationofa
soundsynthesismodelthatdemandssignificantexplorationinordertounderstandit.
Thedirectortoolmodedemandsamoredepthorientedapproachandamutabilityoftoolsthatisnotwell
servedbytangibleinterfacesintheircurrentstateofdevelopment.Currenttechnologiescanstillbeusedto
supportthismodeofcompositionalengagement,butwithadegreeofcompromisedependingonwhether
onepreferstheadvantagesofapurelygraphicalinterface/representationalsystemorthoseofapurely
tangibleone.Ifthelatter,thedirectortoolmodemightbeaveryfrustratingactivity.Onecouldsimplygoto
anothertool,suchasacommercialdesktopmusiccompositionprogramwhileinthismode,however,it
wouldthenbedifficulttogobacktoothermodeswhentheneedarises.Howtosupportthisparticularmode
ofengagementusingtangiblerepresentationsandinteractionpromisestobeaninterestingandpotentially
veryrewardingchallenge.
Theobserverartefactmodeisonethatisdifficulttomaintainonatraditionalinstrumentonaccountofthe
effortandfocusrequiredtoplayanewpieceofmusicevenwithoutalsotryingtodispassionatelyreflect
uponitandhearhowitisworking.Traditionalinstrumentalcomposershadtorelyontheirskillatan
instrumenttoenterthismode,unlesstheyhadtheluxuryofothermusicianstopaythepiecebacktothem.
Comparedtoatraditionalinstrument,becausetheMusicTabledoesnotrequiretrainedandrefinedmotor
skillstooperate,itiseasiertostepbackfromonesworkandreflectonwhatisbeingmade(Makinoetal.
2005).Theabilitytoappreciatetheartefactatdifferentlevelsofscaletoassesshowwellitworksbothas
awholeandinspecificdetailsisprobablyonlymarginallyservedbyatangiblerepresentationalthoughthe
tangibleelementscouldprovideintuitivewaystocontrolonesview.Onecouldalsospeculatethat,when
lookingcriticallyataphysicalrepresentation,aplayermay,throughseeingandtouchingaphysicalpatternin
space,haveadifferentintuitionaboutthepatternandhowitworksthanbysimplyhearingit.
Ingeneral,Brownsmodesofengagementpotentiallyofferalensthroughwhichtoviewthesignificanceof
differentwaysofworking,differentactivities,differentcognitivestylesand,consequently,differenttools
associatedwithdifferentmodesofcompositionalengagement.

DISTRIBUTEDCOGNITIONANDREPRESENTATIONALSYSTEMS
5

RodneyBerryTangibleRepresentationsComputationalCreativity

26June2009

Manyauthorsmentiontheimportanceofrepresentationintangibleinterfaces,butofteninthecontextof
thetechnologicalsystemsneededtomaintainthem.Ullmer(UllmerandIshii2000)forinstancemakesa
distinctionbetweenphysicalanddigitalrepresentationstodistinguishbetweenthetangibleobjectsandthe
graphicalimagesonthescreen.Thesekindsofdistinctionsareimportantfromanengineeringperspective
wherethefunctionalityofthevarioussubsystemsisparamount.However,thegroupingsofsystem
componentsmaynotcorrespondwithhowahumaninteractingwiththesystemperceivesandmanipulates
therepresentations.Fromthehumanperspective,atangibleinteractiveenvironmentcanbeseenasone
overarchingrepresentation.
ZhangandNorman(ZhangandNorman1994)arguethatcertaintypesoftasksrequireustoprocess
informationdistributedacrossapersonsinternalmindandtheexternalenvironment.Theyclaimthat
traditionalcognitivesciencehastriedtodescribesuchtaskspurelyintermsofaninternalrepresentationand
hasdiminishedtheroleofexternalrepresentationsasmerelyaidstocognitionratherthanasapartof
cognitionitself.ThishasledtomodelsofoverlycomplexinternalrepresentationswhereZhangandNorman
wouldarguethatthecomplexityisreallyareflectionofcomplexityintheenvironment.
Theyfocusonthreemainproblems:
1.
2.
3.

Thedistributedrepresentationofinformation
Interactionbetweeninternalandexternalrepresentations
Thenatureofexternalrepresentations

Animportantpointisthatataskmayinvolvemorethanonerepresentationandtherepresentationscanbe
internalandexternal.Togethertheymakeupwhattheauthorscallarepresentationalsystem.Thistermis
helpfulformebecause,sofarinmyreadings,IhavebeenirritatedbytheseparationofwhatIseeasonebig
representationintodifferentrepresentationsaccordingtoarbitraryandusuallytechnologicalboundaries
(real/virtual,physical/digital,tangible/intangible,graphic/hapticetc.).AlthoughIseethepracticalityofsuch
divisionswhendealingwiththelogisticsofbuildingsuchenvironments,Idontthinkiteffectivelydescribes
whatisactuallyhappeningwhensomeoneengageswithatangibleinteractivesystem.Iamalsoremindedof
Dourishsassertionthatitistheuserwhomakesthemeaningoutoftheinteraction,notthedesigner
(Dourish2001p172).
Zhangsuseofthetermrepresentationalsystemhelpsresolvesomeofthisformebecause,bydescribingit
asasystem,itkeepsinmindtheideathatrepresentationisaprocessnotastaticmonolithicobject.Asa
term,italsosignalsthat,asarepresentationalsystem,therearelikelytobesubsystemsofrepresentations
orrepresentationalcomponentswithinit.Thisreconcilestheideaofoneoverarchingsystemof
representationwiththatofmultiplerepresentationsassumingdifferentrolesandlinkageswithinit.
Zhangdividesthedistributedrepresentationalspaceintointernalandexternal,representationalspaces,each
containinganynumberofrepresentationswithchanginglinkagesbetweenthem.Whentheinternaland
externalrepresentationalspacesarecombinedintoadistributedrepresentationalspace,theyforma
representationoftheabstracttaskspace.Zhangpointsoutthat,inasystemofanylevelofcomplexity,the
personengagedintheactivitydoesnotreallythinkabouttherepresentationspaceasrepresentinganything
inparticular.Insteaditisseenasthemediumthroughwhichtheactivitygetsdone.Thatiswhyatechnology
orsystemscentreddescriptionofrepresentationdoesnotreallygetathowweactuallyuserepresentations
inouractivities.
6

RodneyBerryTangibleRepresentationsComputationalCreativity

26June2009

AnotherimportantinsightfromZhangandNormansworkisthatexternalrepresentationscanbeusedto
renderimplicittheinformationthatcouldotherwiseonlybeexplicitinaninternalrepresentation.For
example,physicalconstraintscanenforcearuleinapuzzlethatwouldotherwisehavetoberemembered
andrecalledifexpressedinternally.Theusercanusetheinformationwithoutevenbeingawareofthefact
thattheyareusingit.Relatedtothis,externalrepresentationscanalsoanchorandstructurecognitive
behaviour.Byphysicallyconstrainingtherangeofpossiblebehavioursonlyactionsrelevanttotheproblem
spacearepossible.
Fromthesystemspointofview,byrenderinginformationimplicit,externalrepresentationsmakethetask
easierfortheuser.However,fromtheusersperspective,externalrepresentationsactuallychangethe
natureofataskbecausetheydonothavetobeawareofrulesandotherinformationthatotherwisethey
mighthavetoexplicitlyconsider.Finally,externalrepresentationsareanessentialanddefiningpartof
distributedcognitivetasksandonecannotreallybeconsideredwithouttheother.
Althoughintendedasanapproachtothestudyofcognition,ZhangandNormanalsoclaimthattheir
approachcanyielddesignprinciplesforeffectiverepresentations,especiallywherethetaskisdistributed
acrossinternalmindandexternalenvironment.
Iamconcernedmorewithexternalrepresentationsthaninternalones,buttherelationshipbetweenthetwo
isstillimportant,andmusicisahighlyabstractphenomenonintermsofhowwelistentoitandhowwe
createit.InmovingbetweenexternalrepresentationsintheMusicTable,theroleofinternalrepresentations
mustbealsoconsidered.Inotherwords,whatelementswillbeexplicitandrequireadecisionor
managementbytheplayer,andwhichoneswillbeimplicitinthephysicallayoutoftheinterface?For
example,eventhoughthemappingofupdowntopitchandleftrighttotimeissomethingtheplayermust
beawareofandremember,becausetraditionalwesternnotationfollowsthisconvention(asdoeswritten
textforthetimedimensionatleast),itisnotsomethingthatneedsmuchmemorizationonbehalfofthe
player.Thesameistrueofmappingrotationtoloudnessinimitationofvolumecontrolsinelectronic
equipment.ZhangandNormansstudyalsofoundthat,ofallthepossibleparametersinanexternal
representation,locationwastheonethatwasmostquicklyrememberedeasilyprocessedbythesubjects,
givingmapthetwomostsignificantmusicalparameterstotwolocationalparametersintherepresentation.
Inthecaseoftonalmusic,thesearepitchandtime.
Asthekindsofmanipulationsweusetomanagethecouplingsbetweenthevariousrepresentationalsub
systemswillrequiresomelearningonthepartoftheplayer,itisprobablybettertotryandconstrainthe
complexityoftheplayersinternalrepresentationbyconstrainingthechoicesintheenvironment.

EMBODIEDREPRESENTATION
Thewordrepresentationcanrefertoathing,butitalsoreferstotheprocessbywhichsuchathingcancome
about.Dourish(Dourish2001)echoesphenomenologicalphilosophersinsayingthatitisouractioninthe
worldthatgivestheworldmeaningforus.TherepresentationsthatconcernDourisharethosethatarein
somewayembodiedintheworld.Theembodimentitselfisnotpurelyphysicalhowever,butafunctionof
ourbeinginandparticipatingintheworld.Theculturalandsocialsettingsofouractivityarealsoapartof
whatDourishcallsembodiment.Embodiedrepresentationcanbeexternalisedanddistributed(inlinewith
ZhangandNormanandHutchins).
7

RodneyBerryTangibleRepresentationsComputationalCreativity

26June2009

Embodiedrepresentationsoftheworldarealsoparticipatingpartsofthatworld,andembodiedcomputation
notonlyrepresentstheworldbutparticipatesinandchangestheworld.
IntheMusicTable,anychangetotherepresentationalsoaffectsthemusicbeingrepresentedaswellas
played.Asanembodiedrepresentationofthemusic,therepresentationispartofthemusicitselfandwill
probablyhavesubtleeffectsontheactualmusicproducedincomparisonwiththatproducedbypiano,pen
andpaper.Inthesenseofbeingaprocess,themusicisnotjustthesoundbutthewholeprocessthatleads
toitsproductionandconsumption.Thewholepurposeoftheactofrepresentingistoenactchangetothat
whichitrepresents.Inthecaseofcreativemusicactivity,thisinitialrepresentationmaystartoutasablank
pieceofpaperoramusicalinstrumentsittingquietlyinacorneroftheroomsilentlyrepresentingaspaceof
musicalpossibilities.
Evenwhenviewedasarepresentationalsystem,technologiesandmaterialswillnecessarilybedifferentfor
differentmodesofperceptionandmodesofinteraction.Thismakesmultimodalrepresentationshardto
integratewitheachotherinpractice.Withintheonerepresentation,theremaybeseveralsub
representationsatworkatanytime.Thismakesitimportanttostudythelinkagesbetweenthem.

COUPLING
AnimportantconceptforDourishistheideaofcoupling,aconceptderivedfromHeideggerswritingsabout
tooluse.Couplingisprimarilyconcernedwithcontrolanddescribeshowweactonsomethingsthrough
others.Asimpleexampleofthiswouldbehowafamiliartool,suchasacomputerkeyboardorahammer,
ceasestobesomethingthatweactonandbecomessomethingweactthroughinstead.Couplingismanifest
asaprocessratherthanafixedstate,andvariousfunctionalelementsareregularlycoupledanddecoupledin
thecourseofaparticularactivity.Weareactingonthehammerwhenwetakeitoutofthetoolboxandbring
ittothejob,butweareactingthroughthehammerwhenwecommencebanginginthenail.Ratherthan
beingseenasabreakdownintransparency,andthereforetobeavoided,decouplingandrecouplingisan
importantpartofhowwemovebetweendifferentkindsofactivitiesandhowweengageanddisengagewith
thetoolsweemploytocarryoutthoseactivities.Themanagementofcouplingwillbeanimportantpartof
anydesignwherethecontextandscaleofarepresentationalsystemfrequentlyneedstobereconfigured.
IntheMusicTable,thereareavarietyofcouplingsatwork.Couplingsoccurbetweenelementsofcontroland
elementsofrepresentation,socouplingandmappingbecomealmostinterchangeable.Mappingcouldmore
generallybeseenascouplingatarepresentationallevel.Themostobviouscouplingisbetweenthephysical
objectsandthegraphicalelementsthatfollowthem.Theaugmentedrealityregistrationin3Dbetweenreal
andcomputergeneratedobjectsstrengthensthiscouplingbecausetheobjectsareseenonthescreenas
beingliterallystucktogetherfromthepointofviewofthecamera.Atthesametime,theplayercanseethe
realobjectsbecausetheyarethereinplainviewonthetabletopwithoutthegraphicalaugmentations.
Itisoftenenoughforustointeractwiththephysicalobjectswithoutgraphicaugmentations.Blockscan
representandcontrolpitchandtimewithoutourneedingtohearthemusicalresult.However,theblocks
alsooffercontrolovertheloudnessofnotesbyrotatingtheblockinplaceonthetabletop.Rotationisa
methodthatweareculturallyprimedtounderstandasloudnessthroughouruseofvolumecontrolson
audiodevicessoitmakessensetomapitthiswayintheMusicTable.Wecouldsaythatclockwiserotation
andloudnessofsoundisacouplingthatisembodiedinourculturalunderstandingsoftheaction.Italso
8

RodneyBerryTangibleRepresentationsComputationalCreativity

26June2009

makessensethat,whensavedtoaphraseobject,thesameclockwiserotationonaphraseobjectwould
increasetheoverallsoundlevelofthewholephrase.Here,althoughtheobjectitselfhasitsownabsolute
rotationalstaterelativetothetabletop,therotationalgestureiscoupledwithanincrementingand
decrementingoftheloudnessoftheactualsoundandthestateofthegraphicalelementsofthe
representation.Theblocksalsooffercontrolofthelengthofanoteevent(thetimebetweenturningthe
noteonandturningthenoteoff)bytiltingtheblocktotherightorleft.Herethephysicalblockhasalmostno
representationalrole,apartfromsuggestingthatonedirectionislessandtheotherismore.Thestateis
communicatedbacktotheplayerbythegraphicsandthesoundofthemusic.Onceagainitwouldserveusto
retainthiscouplingatacoarserlevelofdetailbyusingtheleftrighttiltingactiontochangetheoverallnote
lengthsorarticulationsfortheentirephrasewhenmanipulatingaphraseobject.
Whenagroupofnotesaresavedasaphrase,oneobjecttakesontheroleofseveralobjectsmanipulatedas
alargerpattern.Atsomepoint,itwillbenecessarytoretrievetheoriginalpatternofnotesinaformthat
allowsindividualnotestobeeditedwithoutaffectingtheothersinthegrouping.Unlesswecanmagically
maketheoriginalnoteobjectsreappearintheexactpositionsofthenotes,somewaymustbefoundto
smoothlyeffectachangingofrolesbetweenthevarioustangibleobjectsavailabletotheplayeratthe
tabletop.Herecouplingiscrucialasitbecomesnecessarytodecouplenoteobjectsfromnotesandtore
couplewiththemothersinordertorecover,editandreconfiguredifferentpatterns.

CONCLUSIONSANDTHOUGHTSFOREXPLORATION
Theexistenceofmultilevelledrepresentationalsystems(levelsofscaleandlevelsofabstraction)isoneof
thekeyideastoemergefromtheprocessofthiswriting,alongwiththeideathat,ifourcognitioncanbe
distributedacrossourmindandtheenvironment,ourcreativitycanbealsosomethingthatpartlyhappens
outintheworldaroundus,includingtheverymediumweaimtocreativelyshape.Ifthecomputational
processisalsointerdependentwithwhatishappeninginthephysicalworldthencomputationalcreativity
alsobecomeshardtopindownastakingplacepurelyinsidethecomputer(Berry,Makino,Hikawa,Inoueet
al.2006).Separatingrepresentationfromwhatisrepresentedbecomesanequallyslipperyconcept.
Managingthemovementbetweenlevelsofrepresentation(includinglevelsofdetailandofabstraction)is
oneofkeydesignchallengesandareasforexplorationthroughthedevelopmentofanewprototypeMusic
Tablelikesystemplannedtoaddressandexploretheseissues.Theconceptofcouplingisalsobeanareafor
expansionandinvestigation,notonlycouplinginthesenseoflinksbetweenrepresentations,butinthe
Heideggeriansenseofwhetherweactonorthroughtherepresentationalandcontrolelementsofan
interface.Howthecouplingprocessmanifestsacrossdifferentlayersandlevelsofrepresentation,aswellas
theimplicationsfordifferentfunctionalityforphysicalbehavioursininteraction,willalsobeafocusfor
ongoingresearch.

RodneyBerryTangibleRepresentationsComputationalCreativity

26June2009

REFERENCES

Berry,Rodney,MaoMakino,NaotoHikawa,NaomiInoue,YusufPisan,andErnestA.Edmonds.2006.
ProgrammingintheWorld.DigitalCreativity:SpecialIssueonProgrammingandCreativity17
(1):3648.
Berry,Rodney,MaoMakino,NaotoHikawa,MasamiSuzuki,andNaomiInoue.2006.Tunesonthetable.
MultimediaSystems11(3):280289.
Brown,AndrewR.2001.ModesofCompositionalEngagement.Mikropolyphonie6.
Dourish,Paul.2001.Wheretheactionis:thefoundationsofembodiedinteraction.Cambridge,Mass.:MIT
Press.
Hutchins,E..1995.CognitionintheWild,:MITPress.
Jord,Sergi.2003.Sonigraphicalinstruments:fromFMOLtothereacTable*.InProceedingsofthe2003
conferenceonNewinterfacesformusicalexpression.Montreal,Quebec,Canada:National
UniversityofSingapore.
Makino,M.,C.Oshima,R.Berry,N.Hikawa,K.Nishimoto,M.Suzuki,andN.Hagita.2005.ATangible
CompositionSystemforPromptingChildrentoCreateMusicbyReflectivelyManipulatingNotes.
JournalofJapanSocietyforFuzzyTheoryandIntelligentInformatics(SpecialIssueon
EntertainmentComputing)17(2):164174.
Papert,Seymour.1980.Mindstorms:children,computers,andpowerfulideas.NewYork:BasicBooks.
Sloboda,JohnA.1985.CompositionandImprovisation.InThemusicalmind:thecognitivepsychologyof
music.Oxford[Oxfordshire],NewYork:ClarendonPress;OxfordUniversityPress.
Ullmer,B.,andH.Ishii.2000.Emergingframeworksfortangibleuserinterfaces.IBMSyst.J.39(34):915
931.
Zhang,J.,andD.A.Norman.1994.Representationsindistributedcognitivetasks.CognitiveScience18
(1):87122.

10