A visual language for composing interactions

The Concept

firmly

performer left foot stomp

Elements used

physical concept
actor action effect
elements clarifications

firm
movements
Concept flow diagram

up
/ pitch
performer right hand cello bow
down up / down

(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
 A visual language for composing interactions

Concept spatial layout

performer

left hand right hand

press
key move move
combination hand joystick

MIDI
key pad joystick
device + buttons

start/stop select shape

sound preset sound

(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
 A visual language for composing interactions

The physical implementation

Elements

interface
&
hardware software output
physical
objects

Directions of communication
one-way

both ways

Types of connections
wired protocol

wireless protocol

software protocol

Types of streams
audio data stream buffer

video event spectral

Example physical implementation

audio usb
microphone sound card laptop junXion

(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
 A visual language for composing interactions

Role of input / Input labels

input data stream parameter of process enable/disable trigger

Example of input labels

laptop

Process types, Math & Arithmetic operations

+ - / *
add subtract divide multiply

% 22
modulo number box

Example

foot pedal
*22 volume

sound card

(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
 A visual language for composing interactions

Conditional

Relational operations

> < >= <=

greater than less than or
greater than less than or equal to equal to

== !=
equal to not equal to

Logical operations

not and or xor

Comparison of data streams

min max
min max

Check whether value changed

Δ?
change in value

Example

accelerometer Δ >10 play sequence speaker

Ableton Live

(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
 A visual language for composing interactions

Flow control (modal/states)

external factor external factor external factor external factor

inverse
gate switch inverse switch multi switch multi switch

Input to output range

max max

min min

range

Split
375 max

250

125
min

split 0 within range

Change character
max max step size max max max max

min min min min min min

invert quantise segment nonlinear

Constrain
max max max max

min min min min

block clip wrap fold

Buffer

buffer

(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
 A visual language for composing interactions

Example of flow control

button

toggle

accelerometer sound

gate quantise

(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
 A visual language for composing interactions

Legend
operator filter

type accumulation

Filter
filter filter filter filter

windowed past input/output reset

filter filter filter filter

Δ
smoothing change change accumulation

Analysis

Amplitude/bandwidth/range analysis
analysis analysis

min max
min in stream max in stream envelope peaks

analysis analysis analysis

onsets spectral power

Example of filtering
filter 1 600
α=0,01

accelerometer sound
0 30

smoothing nonlinear

(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
 A visual language for composing interactions

Resampling

resample upsample downsample

reset
Time

3x
3s

transition timer delay repeat

Button related processes

+1
toggle gate trigger switch counting

… … 1 3
3s
4 2
multiclick timed multiclick combinations sequence timing

Mixing signals

preset
matrix sensor fusion interpolation implicit methods

(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License
 A visual language for composing interactions

Composing Interactions

an Artist's Guide to Building Expressive Interactive Systems

by Marije Baalman

https://composinginteractions.art

(c) 2022, Marije Baalman and Superposition - Creative Commons Share Alike v4.0 International License