Contemporary Music Review VOLUME 29 NUMBER 1 FEBRUARY 2010 Virtual Scores and Real-Time Playing Edited by Jason Freeman and Arthur Clay Contents Preface: Virtual Scores and Real-Time Playing Arthur Clay and Jason Freeman Real-Time Score Generation for Extensible Open Forms David Kim-Boyle Notating the Unpredictable Pedro Rebelo Extending the KLANGPILOT Score Language for Real-Time Notation Johannes Kretz Notation in the Context of Quintet.net Projects Georg Hajdu, Kai Niggemann, Adém Siska and Andrea Szigetvari LiveScore: Real-Time Notation in the Music of Harris Wulfson G. Douglas Barrett and Michael Winter You Can Play It Too: The Virtuoso Audience Arthur Clay Density Trajectory Studies: Organizing Improvised Sound Nick Didkovsky Live Random Concepts and Sight-Reading Players: The Role of the Computer in the Era of Digested Digitalism Harald Muenz The Real-Time-Score: Nucleus and Fluid Opus Gerhard E, Winkler Tools for Real-Time Music Notation Jason Freeman and Andrew Colella About the Authors 7 29 39 63 75 81 89 101 1sContemporary Music Review Routed Vol. 29, No. 1, February 2010, pp. 89-100 I a Tbe The Real-Time-Score: Nucleus and Fluid Opus Gerhard E. Winkler The author describes his concept of Real-Time-Score, which he has used in compositions since 1994, Real-Time-Scores are based on the evolution of complex dynamic systems and generate the formal structure of an entire piece, which includes the following levels: the live sound played by the musician; the ad hoc electronically transformed live sound; and the interaction between the player and the computer. These aspects define a kind of musical self-organization, where the formal structure of a piece is unique to each performance as a result of the above-mentioned complex interrelations. Keywords: Real-Time-Score; Interactive Musics Computer Simulations; Mathematic Modelling of Dynamic Systems; Live-Electronic Sound Transformation; Musical Self- Organization ‘The Development of the Real-Time-Score In the early 1990s, having previously written pieces in the traditional manner of manually recording the score on paper, I started to develop pieces which were based on computer simulations of complex dynamic systems. My goal was to create a new kind of interdependent musical structure, where the creation of different layers of a musical work are linked to one another (Winkler, 2009). This step was challenging for me as a classically trained composer, for 1 not only had to study basic mathematical concepts (e.g. Arrowsmith & Place, 1994), but also had to create software to execute music-generating algorithms (Barrigre, 1984). My first piece in this vein was ‘Emergent’ for ensemble and live-electronic sound transformation, a commission from the Salzburg Festival, performed there in 1993 by Klangforum Wien and the Experimental Studio of the Suedwestrundfunk (SWR) in Freiburg/Breisgau (Winkler, 1993). During the test runs of the simulation, I realized that minute parameter changes influenced the evolution of the entire system. This begged the question, ‘Why must musical scores be fixed on paper and defined for all ages to come?” ISSN 0749-4467 (print)/ISSN 1477-2256 (online) © 2010 Taylor & Francis DOK: 10.1080/07494467.2010.50959890 G. E. Winkler New programming tools like Max, and later Max/MSP, encouraged me to liberate my music from adherence to a fixed score and to progress to a concept of polyversional works. But, on the other hand, this decision presents the composer with new challenges: how many and which details to control, which aspects to ‘set free’, which things to ‘allow’ to happen, and where to draw the borders of the musical world. I started with a trilogy of small ensemble pieces with interactive live electronics and Real-Time-Scores. Each piece was developed at one of the large computer music and live electronic centres in Europe—‘Chambres separées’ at the ZKM, Centre for Arts and Media ‘Technology in Karlsruhe (1994/95); ‘KOMA’ at the IRCAM, Centre Pompidou (‘selection IRCAM’, with Arditti-Quartet, 1995/96); and ‘entrop’ at the experimental studio of the SWR, Freiburg (1996-1998) (Winkler, 1999). ‘At the same time, I started a cycle of solo pieces in the same manner, the “Hybrid” cycle, which currently consists of seven pieces, with a new one to premiere at the ZKM, Karlsruhe in November 2010. This series is intended as a research project about the potential and challenges of Real-Time-Scores and Interactivity. Because Real-Time- Score is a new form of notation presented on a computer screen, it is confronted by similar problems as the design of GUIs. It challenges the musician to not only ‘play music’ but also to control other aspects of the piece. Finally, the embedding of live electronic sound transformations into the artistic work presents a number of challenges. By far my largest project was the interactive opera ‘Heptameron’, commissioned by the Munich Opera-Biennale and performed in 2002 as a co-production with the ZKM, Karlsruhe. In this work, I used sensors to create a work so dynamic that even the action on stage, the stories and situations can be changed in real-time (Winkler, 2004a). Basic Definitions Regarding the preceding descriptions, it seems obvious that in my approach, Real- Time-Score not only defines a new form of notation but also includes various levels of interaction between the performer and the computer. Although a Real-Time-Score may include certain aspects of improvisation, the actual notes of a ‘playing score’ should be realized as precisely as possible by the performer. In this sense, Real-Time- Score is located between traditionally notated music and improvisational music as a third, new way of playing music. In each piece there is a defined set of interactive signals, acoustic or gesture-based, which the musician can use to communicate with the computer. ‘At the core of every Real-Time-Score piece is the computer simulation of a complex dynamic system, which generates all relevant parameters of the playing score, evolves over the time of the performance, and can be influenced by the interactive signals of the musician. The simulation also changes certain parameters of the live electronic sounds. The use of live electronic sound processing is an integral part of my Real- Time-Score concept. I define ‘Real-Time-Score’ as the use of the evolution of a complex dynamic system to control the formal evolution of a piece, which consists ofContemporary Music Review 91 the live sound played by the musician, the transformed sound of the live electronics, and the communication between the player and the computer. These aspects combine to create a new form of musical self-organization, where the formal structure of a piece is created during each separate performance as a result of the complex interrelations between the musicians, their sound and the electronics. The traditional notion of a work as an opus perfectum et absolutum is changed to a {fluid opus, without the composer giving up responsibility as the creator of the piece. ‘The act of creation is extended to the ‘setting of potentialities’ for many different versions of one piece instead of defining a single ‘valid’ version (Winkler, 2004b). Examples of Playing Scores The playing-score part of the Real-Time-Score is the actual notation executed by the musician during live performance. The playing score has to be readable with a minimum of effort during performance. Thus, the different parts of the score have to. be reduced to a number of simple elements, which can be learned in advance, and be grasped in one glance by the musician. Thave tried to solve the challenges of Real-Time-Score notation in two ways. First, the notation has to be precise enough to keep the musicians from shifting into improvisation, though experience in improvisation is quite helpful in this situation! Second, the portions of the score presented to the players during live performance should not be too long. A three-to-seven-second range has proven effective. Notation on a Computer Screen: Traditional Aspects In my interactive opera ‘Heptameron’, musicians were given a set of specific sound- types defined by different note heads, which were projected in a time-space domain (see Figure 1). 6 PP ord Figure 1 Example of the Real-Time-Score for the musicians in the interactive opera “Heptameron’ (1998-2002).94 G. E. Winkler ‘These particles appear and disappear in an unpredictable way, creating stress for the musician that is a part of the performance itself. Another way of using pre-notated material is to modify it in real-time. In ‘Chambres separées’ (1994/95) the musicians are faced with a collection of pre-written fragments of music, with fixed pitch, rhythm and metre. All other musical parameters are modified in real-time, indicated with three sliders and a metronome sign on the playing score (see Figure 4). Each pre-notated pitch/rhythm fragment can be played (and rehearsed in advance) with five different dynamics, five different articulations, four micro-differentiations, and five different tempi. As these parameters change in real-time, a unique soundworld is produced from simple, pre-written musical fragments. Regarding the notation of the player score, a mixture of symbolic elements (a traditionally notated main pitch) and graphic elements (glissando lines, sliders and bar lines) has proven to be the clearest way to represent musical instructions in real- time. Abstract elements are, in my experience, not as welcome as verbal indications (eg, ‘col legno’), which are familiar from traditional notation, Maybe over time younger musicians will get used to more unconventional notation methods; at the moment, however, a bit of tradition seems to work well in the area of real-time notation. Interactivity: Control Levels An absolutely new challenge for the interpretation of Real-Time-Score works arises from the necessity to influence the ongoing computer simulation during a performance and how the musician can communicate with the computer. The musician has to send control signals to the computer that can be clearly understood. ‘After the musician changes parameters within the computer simulation, the modified material returns to the musician and the audience in the form of the playing score \SSLIL & SIF mf ppp | PPPPPP ba Figure 4 Modification of pre-notated score files via slider indications on these parameter levels: dynamics, articulation, micro-differentiations and tempo.Contemporary Music Review 95 and live electronics. This ‘feedback loop’ creates a unique interactivity that redefines the relationship between musician, computer and composer in real-time. On the technological level, the musician has to be able to produce control signals accurately and the computer has to ‘understand’ when a signal is to be interpreted as a control signal. In general, there are two types of performer-computer interactivity, each of which has profoundly different meanings for the player. The first is direct interactivity, whereby the musicians’ signals to the computer instantly change aspects of the live electronics or real-time notation. This method is transparent and easy for the musicians to understand. The second type is indirect interactivity, whereby the signals the musicians send to the computer do not directly influence the systems output. Changes may occur over long periods of time and are not apparent to the musicians. More rehearsal time and a clear explanation by the composer can help the musician understand how his or her actions affect the electronics. One of the biggest challenges for musicians is to understand how a piece can unfold and how their signals are linked with its formal evolution. Control signals used can be time-based, sound-based, pitch-based, gesture-based, or poly-dimensional. Time-Based Control In my piece ‘Hybrid VII (FractuReflex)’ (2003) two different time-based controls are used: silence-based control, and sustain-based control. First, the musician can decide how long to wait until the indicated notes are played. The computer measures this time delay and uses the result as an input for the simulation program with consequences for the next score projections. In this case, there is no differentiation between playing score and control action. But this works only when the music consists of short notes and chords (see Figure 5), so that there are measurable pauses when the next score indication is projected. a= AI Figure 5 Short-note projection (with measured reaction time before playing) in ‘Hybrid VII (FractuReflex)’ for accordion and interactive live electronics (2003).96 G. E. Winkler ‘The other type of time-based control, sustain-based control, is also used in ‘Hybrid VII (FractuReflex)’ and consists of measuring the length of a played note. As the computer creates the player score, it ‘knows’ which type of duration has to be measured Sound-Based Control ‘The performance of ‘Hybrid II (NetWorks)’ starts with the simulation of a neural network in an amorphous state and is trained during the performance by the musician via certain sound signals. In this piece, to indicate a control signal to the computer, the musician hits the space bar on the keyboard of the notebook on which the score is displayed, and then plays the (short) signal. There are six characteristic short sound events—including knocking on the cover of the instrument, playing a sharp pizzicato note, and scratching with the bow on the lowest string—that can be selected freely by the player, but should be used consistently throughout the piece. These sounds are analysed for their spectral content and the data are used as input material to a Hopfield-type neural network. By selecting which sound signals to play, the musician influences the state of the system. The piece results in a fully trained neural network that reacts in a specific way to each of the sound signals. Pitch-Based Control Pitch-based control can be executed using fixed pitches or continuous pitches. In my newest hybrid piece, ‘Hybrid VIII (PiAnimeaux)’ for live piano and computer- controlled MIDI piano, which will be performed at the ZKM, Centre for Arts and Media Technology in Karlsruhe at the end of 2010, I use octave-independent pitch qualities to switch on and off certain sound worlds in the accompanying electronics. The use of continuous pitches is also effective. The decision to play a glissando up or down from a given starting pitch and the detection of ‘crossing the border’ of one of two given border pitches are at the centre of the control score of my piece KOMA for string quartet and interactive live electronics (see Figure 6). ‘A sequence of such control screens produces a continuous glissando, which follows the outline of one of René Thom’s ‘Catastrophe Theory’ models, the so-called butterfly (Thom, 1982). At certain critical points, there are discontinuous ‘jumps’ in the simulation, the so-called catastrophes. A virtual model of this ‘butterfly’ is also projected in the space, and the control rectangle on the right side of the screen shows the position of the projected sound in space. The slider below that rectangle gives the state of a special aspect (criticality of the catastrophes) of the dynamic system. As this unit detects both pitch and dynamics, the computer measures the difference between the played dynamics and the proposed dynamic level and the quality of the sound to influence the system.Contemporary Music Review 97 Figure 6 Control score of KOMA, Gesture-Based Control ‘The use of different sensors for gesture control during the execution of a piece is nowadays a widely used means of interactive communication, In my interactive opera “Heptameron’, I use several types of sensors, including ultrasonic, infrared and gravitational sensors, acceleration sensors, force sensors, surface-bound (Wacom) sensors, electromagnetic field sensors and optical sensors, to enable the actors on stage to communicate with the computer. Poly-Dimensional Control All main parameters of the sound played by the performer can also be used together to control the computer. For example, I use pitch, note duration, dynamics and noise level of a sound in my piece ‘Hybrid IV (Zoomed::Fringes)’ for flute and live electronics (2000/01) to control the electronic output. By modifying aspects of their sound, the musician can navigate an imaginary space that creates the live electronics, presented to the performer as part of the playing score. ‘The musician is encouraged to modify specific aspects of his or her sound in order to conform to the computer’s instructions. Each aspect of the player’s sounds, like pitch, dynamic level, etc., is displayed on the GUI. As the performer modifies his or her sound, bars on the screen move. The performer’s goal is to move all of the bars to the correct location (see Figure 7). Performances and Experiences Technical Challenges When I started working with Real-Time-Scores, I had the idea that the usability of this new type of score would be as simple as performing traditional, paper-written98 G. E. Winkler Gerhard B, Winkler ‘Hybrid IV ("Zoome ‘ringes") eb 21 Mizoprammen as Fussnoven ‘and Phe teuch dae") rr i fa 5 Foros) [ioe Te oe pin SE aor nee oa ltr 2 ee 0 2001 oy cernara.wikter Figure 7 By moving four borders on the screen according to four parameters of sound, the musician ‘navigates’ through a pre-recorded sample of ocean sounds: ‘Hybrid IV (Zoomed::Fringes)’ for flute and interactive live electronics (2000/01). scores. But several problems arose. First, the common problem of non-compatible computer systems causes installation errors on the musicians’ laptops. So for many of the performances or rehearsals, I bring along my own notebook which works as intended. Second, there is the problem of system crashes on stage during a performance. For several years, the separation of audio and visual processing on two laptops helped to avoid this disastrous situation. I prefer to assist performances by sitting at the mixing console with the audio-processing computer. This control system is connected via MIDI with a second laptop on stage, where the graphic representation for the player is projected. So it is possible to restart the control computer or shift to a redundant machine after a system crash without interruption of the ongoing performance. Though it may be sufficient for rehearsals to use one pair of speakers, in several pieces I prefer to use a more complex sound distribution, using two to four pairs of speakers. The assistance of a sound engineer is helpful for the installation of the mixing console and for controlling the live electronics during performance. All of these aspects make the use of Real-Time-Scores more complicated than just setting up a laptop and playing, but maybe one day it will be just as simple as putting a score on a music stand.Contemporary Music Review 99 Challenges of Interpretation In addition to the aspects mentioned before, I will refer to several important experiences of the practical work with Real-Time-Scores. The best way to approach the playing of a Real-Time-Score seems to be that of a relaxed, playful ‘testing’ of the system. Musicians who have already played Real-Time-Scores have compared it with ‘reading a newspaper’ or ‘watching TV’; the expectation of ‘what is to follow’ is clearly named as a part of the interpretation. In the case of a Real-Time-Score, there are additional challenges for musicians that do not exist in traditional music—most notably, the responsibility to control the computer system. This is sometimes confusing to the musicians and seems to be the biggest challenge they encounter when performing Real-Time-Scores. Some of the musicians are not used to live electronics, especially real-time sound transformations of their own playing. But this mixture of live and modified sound is the heart of my music. Finally, arguments from musicians who refuse to play Real-Time-Scores may also give an impression of the challenges connected with them. Some musicians refuse to accept the computer as part of the musical ensemble. It was sometimes difficult for the players to connect certain interactive signs they made with the reactions of the computer. A typical comment: “The computer does not do what I want it to.’ This experience may lead to frustrations in rehearsal and the eventual refusal of the musician to play the piece. Other frustrations arise when musicians are unable to understand the ‘mechanism’ of the computer system, even if it is explained to them in a very simple and metaphorical way. In this case, they feel powerless or like a victim of the ‘system’, In this respect, it is a challenge for the composer to create situations where even the most unforeseen aural results ‘make sense’ to the musicians. It is difficult to find the right balance of technical details to provide to the musicians. It is necessary for the composer to get a good personal approach to the psychology of the individual player. I see no universal solutions. Challenges for the Audience Itis important for the audience, often with the help of a short introductory comment, to understand that they are listening to a unique, non-repeatable version of a piece, an experience which lies beyond the traditional variability of interpreting a pre- written score. This helps them follow and understand the tension involved in performing a Real-Time-Score. A clear and recognizable repertoire of control signs will also help the audience to understand the interactive nature of the music. Projections of the Real-Time-Score onto video screens situated behind the musicians have been very helpful, and they show the audience the changes in the score and interactive processes. Each piece should to be performed at least twice in concert to get an idea of the differences and thus the concept behind the Real-Time- Score.100 G. E. Winkler One solution that has been found is the integration of several versions in one performance, like my Opera ‘Heptameron’, where most of the seven ‘stories’ start at least two times from a common point and develop in different directions. But even in this case, the difference between two full performances was remarkable, and people who witnessed two or even three performances of ‘Heptameron’ were impressed by the meta-experience that was attained (Hiekel, 2010). The Real-Time-Score changes not only the definition of what constitutes a ‘work’, ‘score’, ‘interpretation’ or ‘composer’, but also the performance situation itself. It can be quite rewarding for an audience's musical expectations to be challenged. Why Compose Real-Time-Scores? Luse complex dynamic systems in my concept of Real-Time-Scores not only to create a unique and challenging creative experience but also because I have a deep interest in understanding the behaviour of these systems and their commonalities with nature. We have to study interdependencies carefully, from the smallest ecological network to global interrelations, to be able to survive in a world that is more complex than we can imagine (Jantsch, 1986). References Arrowsmith, D. K., & Place, C. M. (1994). Dynamische systeme. Mathematische Grundlagen. Ubungen. Heidelberg, Berlin & Oxford: Springer. Barriére, J. B. (1984). Chréode 1: The pathway to new music with the computer. Musical Thought at IRCAM. Contemporary Music Review, 1(1), 181-202. Hiekel, J. P. (2010). Moeglichkeitsraeume. Gerhard E. Winklers interaktives Musiktheaterwerk “Heptameron’. Neue Zeitschrift fiir Musik, 3, 54-57. Jantsch, E. (1986). Die Selbstorganisation des Universums. Vom Urknall zum menschlichen Geist (3rd ed.). Miinchen: Deutscher Taschenbuch-Verlag. ‘Thom, R. (1982). Structural stability and morphogenesis (6th ed.). Reading, MA: Deest. Winkler, G. E. (1993). CD Salzburger Festspiel-Dokumente, col legno, WWE 1CD 31872, LC 7989, ORF, zeitfluss 93 (Gerhard E. Winkler: ‘Emergent’). Winkler, G. E. (1999). CD ORF, edition zeitton: Gerhard E. Winkler: Instrumente/Stimme Computer Live-EFlektronik ORF-CD 189, LC-5130, 1999. Winkler, G. E. (2004a). Heptameron, interactive opera, col legno, WWE ICD 20232, LC 07989, Miinchener Biennale, ZKM 2004. Winkler, G. B. (2004b). The Real-Time-Score: A missing link in computer-music performance. Paper presented at First Sound and Music Computing Conference, SMC 04, [RCAM, Paris, Conference Proceedings, pp. 9-14. Winkler, G. E. (2006). Hybrid II (NetWorks), or: At the Edge of Musical Self-Organization. Electronics in New Music (New Musik and Aesthetics in the 21st Century, 4), pp. 236-249. Winkler, G. E. (2008). Chaotische Attraktoren als Bezichungsmodelle, Zu den Grundlagen meiner interaktiven Oper, Heptameron (1998-2002). http://www.oefg.at/text/arge_wissenschaft kunst/Winkler-report.pdf. Winkler, G. E. (2009). ... nur erst einige Bewegungen, als Buchstaben der Natur, ... Compu- tersimulation und musikalische Komposition, Vernetzungen: Veréffentlichungen des Instituts fiir Neue Musik und Musikerziehung Darmstadt, Schott Music, 49, 44-55.