P. 1
Real-Time Interactive Sonification

Real-Time Interactive Sonification

|Views: 22|Likes:
An essay for the 2012 Undergraduate Awards (International Programme) Competition by Danny King. Originally submitted for Computer Science Dissertation at Durham University, with lecturer Dr. N. Holliman in the category of Computer Sciences & Information Technology
An essay for the 2012 Undergraduate Awards (International Programme) Competition by Danny King. Originally submitted for Computer Science Dissertation at Durham University, with lecturer Dr. N. Holliman in the category of Computer Sciences & Information Technology

More info:

Published by: Undergraduate Awards on Aug 31, 2012
Copyright:Attribution Non-commercial


Read on Scribd mobile: iPhone, iPad and Android.
See more
See less


Real-Time Interactive Sonification

Abstract Context/Background: Sonification is the process of converting data into sound. An auditory graph is the sonified equivalent of a visual graph. Much research has been conducted into the utility and applicability of sonification as a supplement to or replacement for visualisation for various tasks and situations. Many interaction techniques have been proposed. We are interested in comparing two particular techniques: real-time and static interaction. Aims: Our first aim was to develop a sonification and visualisation engine for bivariate line graphs. Our second aim was to determine whether real-time interaction with bivariate auditory line graphs can increase performance in graph shape matching tasks when compared to static interaction. Our third aim was to determine whether it is possible to implement a trivariate auditory graph engine using the static and real-time interaction techniques. Method: We built a sonification and visualisation engine and used this to conduct a trial to determine the relative effectiveness of the two interaction techniques. We compared participants‟ performance in a matching task using the static and real-time techniques to identify graph shapes after listening to a series of auditory graphs. Lastly we extended the engine to support more complex trivariate data sets. Results: Our sonification engines performed well: they are intuitive and easy to use with minimal training. The trial results indicated that static interaction gives increased performance over real-time interaction for graph shape matching tasks on bivariate auditory line graphs. Conclusions: We built sonification engines with the ability to create bivariate and trivariate auditory graphs. Interesting results were obtained from the trial however more research is required to gain a fuller picture of when and where the two interaction techniques are most suitable. Keywords – Sonification, auditory display, interaction techniques, human-computer interaction.

INTRODUCTION Sonification & Auditory Graphs Many people will be familiar with the concept of data visualisation: the mapping of data to a visual medium such as a line graph on a computer screen. Sonification is a similar process where data points are instead mapped aurally (figure 1) and so it may be considered to be the use of sound for the exploration and analysis of data [1]. Kramer et al. [2] concisely define the process as "the use of non-speech audio to convey information." As Nees and Walker [3] point out, unlike with visualisation, time is integral to sonifications: "While visual objects exist primarily in space, auditory stimuli occur in time."

Figure 1. Visualisation & sonification.


Sonification may be more familiar to many people than they first assume. Many people change their actions based on the sounds they hear from appliances and devices. For example, many car drivers listen to engine sound to determine when to change gears. Similarly, many people intuitively listen to the sound produced when pouring liquids to know when a vessel is full [4]. There are also several well-known sonification devices such as the Geiger counter, sonar and the auditory thermometer and although sonification has been used in applications like these for many years, the field has only become a significant research area over the last two decades [5]. The appetite for sonification is increasing as the complexity and amount of data we need to process grows, as well as advances in computers making auditory displays easier and quicker to create [6]. Sonification can extend the bandwidth of the humancomputer interface, allowing us to process large data sets more easily or to look at data sets in different ways, allowing for alternative patterns to emerge [7]. Sonification can also provide an alternative interface for those unable to access visual displays such as the visually impaired or those unable to divert their attention such as drivers or pilots [7]. Auditory graphs are a common type of sonification which can take a variety of forms. For bivariate data sets, a common practice is to use changes in pitch, frequency or volume to represent changes in values on the y-axis and time to represent changes on the x-axis [3]. Hence, for example, the visual graph in figure 2 could be sonified over ten seconds using a tone that steadily decreases in pitch for four seconds, remains at constant pitch for two seconds and then decreases in pitch again over four seconds. Static and Real-Time Interaction Methods for Sonification There are numerous interaction methods for sonification and Figure 2. An example bivariate graph that could be selecting the right interface is fundamental to the utility of a sonified using our basic sonification environment [3]. Interaction techniques and models deliverable. have been proposed in the literature, which we examine in more detail in the Related Work section of this paper. For our project we were primarily interested in the real-time navigation of pre-recorded data and investigating whether this gives better performance than static playback over time of simple auditory line graphs. We define static interaction as the sonification of the data set over time, where the user has no control over the current graph position except to start and stop playback. We define real-time interaction as the sonification of the data set over time where the user controls the current graph position via the position of their mouse on the screen. Numerous trials have been conducted on static playback (e.g. [8] [9] [10]) and on real-time interaction (e.g. [11] [12] [13] [14] [15]). However, although there has been much good analysis of interaction methods for sonification, there has been comparatively little research regarding their effect on performance. We found no studies comparing static and real-time interaction with bivariate sonifications. Project Objectives We therefore asked the research question: 'Can real-time interaction with auditory graphs increase performance with a matching task for bivariate graph shape identification compared to static interaction?‟ Also, as a supplementary question: 'Is it feasible to create static and real-time interaction techniques for trivariate sonified data such as for intensity graphs?" The project was separated into three objectives: basic, intermediate and advanced. The basic objective was to build a sonification and visualisation engine to create auditory graphs and


their corresponding visual graphs for a given bivariate set of data points. This objective was to serve as a foundation for later work by providing the tools necessary to implement the intermediate and advanced objectives. Given a list of integer values representing the 𝑥 and 𝑦 values in a bivariate function, the engine was to produce a representative, accurate and continuous auditory graph by normalising and interpolating between the values and outputting a tone over time. The pitch of the tone at any one time was to correspond to the current 𝑦-value at that point in the graph. An accurate visual graph was also to be output. A basic user interface was to be created, allowing either static or real-time interaction with the auditory graphs. We achieved this objective by implementing the engine using the Processing programming language and the Minim and ToxiMath libraries [16]. The user interface (figure 3) consists of a visual representation of the graph and a grey interaction area. In the static interaction mode the user can press a „play/stop‟ button causing the graph to play over time, where the 𝑦-axis is represented by pitch and the 𝑥-axis by time, or to stop playback prematurely. In this mode the grey area acts as a status bar indicating the position of the current value being sonified in the graph. In the real-time interaction mode the user controls the 𝑥-axis: the horizontal position of their mouse on the grey area represents the position of the sonified value in the auditory a b graph. In this mode, the 𝑦-axis is again Figure 3. User interfaces for the basic objective represented by pitch but the 𝑥-axis is represented (a: static interaction, b: real-time interaction). by the position of the user‟s mouse cursor. The intermediate objective was to determine whether real-time interaction increases performance over static interaction with auditory graphs. We designed and ran a trial with fifteen participants to test whether the two techniques produced a statistically significant difference in performance using a graph shape matching task. In order to ensure a wellexecuted trial, we performed an in-depth analysis of the literature to determine relevant design considerations and best practices when implementing these auditory graphs. We produced software for the trial and incorporated the sonification and visualisation engine into it. The results were analysed using statistical methods, which we describe in detail in the Results and Evaluation sections of this paper. Neither interaction technique provided a statistically significant impact on score as a performance measure. However, when using reaction time as a performance measure, static interaction did provide a statistically significant improvement. The advanced objective was to determine whether it is feasible to build a trivariate intensity graph sonification engine that uses static and real-time interaction methods. Intensity can represent any value that varies over a two-dimensional plane, for example height (for Figure 4. Graph types the engines can support. topographic maps) or rainfall density (for weather maps). Figure 4 provides examples of the two graph types supported by the engines. We achieved this objective by extending the sonification and visualisation engine to support the extra dimension. Instead of taking in a list of integer values, the advanced engine reads intensity map images and converts the visual intensity data into interactive sonifications. Real-time interaction functions by the user moving their mouse over the (now larger) interaction area. A


higher intensity at the mouse cursor position is represented by a higher pitched output tone. Static interaction is supported although informal user testing indicates it is less useful than for bivariate graphs due to the large number of values in the trivariate data sets. A middle-ground interaction technique, between real-time and static, was therefore built as an extra feature. RELATED WORK We performed a thorough analysis of the literature to identify previously proposed interaction methods in order to determine which would be most useful to test. We also identified relevant design factors and best practices to apply. In the last two decades, sonification has become an active research topic [5], with work conducted on accessibility aids [17] [13] [18], educational use [8] [19], toolkits [11] [19], taxonomy [4], design patterns [20], design considerations [21] [22] [23] [24] and frameworks [7]. Naturally, data analysis is an area of interest in the field and interesting work has been done on auditory graphs including scatterplots [8], boxplots [9], time series [25] and line graphs [26] [27]. Levels of interactivity in sonifications can vary widely. As Walker & Nees [5] describe it, “interactivity can be considered as a dimension along which different displays can be classified, ranging from completely non-interactive to completely user-initiated.” Displays can range from the most non-interactive „concert mode‟ in which “the display is simply triggered and plays in its entirety while the user listens” to the most interactive „conversation mode‟ in which “the listener may be able to actively control the presentation of the sonification.” We built two levels of interactivity; one close to being in „concert mode‟, which we call static interaction, and the other more towards the „conversation mode‟, which we call real-time interaction. We discuss both in detail in the Solution section of this paper. In order to compare the performance of real-time and static interaction, a task type had to be selected on which to base the trial. Walker & Nees [5] identified common uses of sonification including data exploration, point estimation, point comparison, trend identification, data structure identification, and data exploration. We focused on a data structure identification task, which is when the “listener hopes to extract information about the relationships within, and structure of, the data represented.” We found this to be the most appropriate activity type for the trial due to the minimal training required. Identifying the shape of a curve is a fairly simple activity compared to, for example, determining accurate values or trends within the data. It is comparatively easy to determine participants‟ performance using a simple matching task based on this activity. Successful trials of a similar design, where participants matched auditory graphs to a choice of visual graphs, have been run for stereo versus monaural presentation in sonified line graphs and scatter plots [8] as well as for temporal mapping versus pitch in sonified box plots [9]. We also needed to decide on a definition of performance to measure. Burke et al. [28] identify three performance outcomes that are useful to evaluate sonifications: error rate, performance score and reaction time. They define error rate as the “deviation from the correct response, or the difference between an observed or calculated value and the true value.” The performance score is a quantifiable measurement, for example of task completion time or accuracy: “performance scores represent the execution of appropriate or efficient actions.” Reaction time represents the “elapsed time before a participant responds to a cue.” We were most interested in the accuracy of matches and the task completion time as these seemed the most natural performance measures for the two interaction techniques in our trial.


We then needed to determine the type of sonification to use in the trial. Common sonification techniques were categorised by de Campo [22]. We implemented „continuous data representation‟ sonifications which “treat data as quasi-analogue continuous signals [and] rely on two preconditions: equal distances along at least one dimension, typically time and/or space; and sufficient sampling rate, so that interpolation between data points is meaningful” [25]. This type of sonification is suited to shape identification: de Campo notes that using this technique, “perception of continuous shapes (curves) can be appropriate; audition is very good at structures in time.” He also states “the drawbacks include: it is often tied to linear movement along one axis only; events present in the data (e.g. global state changes in a system) can be difficult to represent well.” As we are using bivariate graphs and our own data sets, these issues do not affect us. We considered some design factors and best practices for the engines. A mapping is the representation of a variable in a sonification using a particular sound property, for example using pitch to represent temperature [7]. Common mappings are pitch, tempo, onset rate, loudness, position or timbre [3] [24]. Barrass [7] suggests some mappings which are wellsuited to certain data types. For quantitative interval data, which we used, a suggested approach is to represent the data using continuous acoustic variables such as pitch or loudness. Flowers et al. [26] also concluded that “pitch variation over time can offer a compelling description of function shape that may be nearly as effective as variation in the curve height is on a visual graph.” We used pitch variation over time as it was an intuitive way to sonify line graphs that needed minimal user training for the purposes of our trial. Humans can hear a frequency range of roughly 20Hz to 20kHz, although most people‟s upper limit is half that [1]. This provides a large resolution for mapping pitch to data. However, high frequencies can be uncomfortable to listen to. We therefore restricted the frequency range of our sonification engines to be 60Hz to 1100Hz, which is within the range of a human voice [29]. As with visualisations, providing context is important. For example, the auditory equivalents of “[axis] tick marks, axis labels, legends, etc.,” [21] should be considered for implementation. Walker and Nees found that context sounds differing in intensity by 9dB produced the best performance in interpreting auditory graphs. We focused on graph shapes rather than individual data values, so in order to keep the sonification as simple as possible and lower the learning curve, the only auditory context feature we implemented was tick marks (i.e. a click every 0.5 seconds), which was the most useful feature for shape detection. There were some similar studies from which we drew inspiration and experience. Brown and Brewster [27] ran a trial to “determine the level of accuracy with which sighted people were able to draw sketches of [auditory] graphs after listening to them.” These sketches were evaluated based on the number of key features present. They found that accuracy was over 80% on average. The study demonstrates that people can interpret bivariate auditory graphs with high accuracy, adding weight to our sonification design choices; however the sample size was extremely small at only 6 people. Heuten, Wichmann & Boll [13] developed an interactive three-dimensional sonification allowing users to navigate a city map (i.e. a twodimensional plane) using the mouse cursor to orient themselves in space. The third dimension, landmark data, was sonified. Participants were able to produce fairly accurate sketches of the sonified city maps demonstrating they were able to build accurate mental models of them; however, again, the trial size was very small at only 4 participants. Our trivariate intensity auditory graphs were built to use the same navigation style. Peres and Lane [9] ran a trial of a similar format to our trial with a large sample size of 56 participants, each of whom were asked 50 questions. “Subjects listened to a sonified box plot and then selected a visual representation from a set of four.” They “assumed that visual representation


would be easily interpreted” and their target and distracter graphs were chosen randomly for each trial. We based several trial design choices on this model. Peres and Lane‟s trial was to compare the effectiveness of pitch, panning and redundant sound dimensions for presenting the graphs. Pitch was found to be the best mapping, which we also used. SOLUTION With this project we aimed to explore the use of interactivity in auditory graphs and to determine whether the interaction method used has an impact on performance. The effectiveness of interaction methods for any technology is of fundamental importance to its applicability [30] however we could find no studies comparing static and real-time interaction with bivariate sonifications, so this became our focus. In addition, we wanted to determine whether it was practical to apply these interaction techniques to a more complex environment: trivariate auditory intensity graphs. We began our work by building on the ideas from the existing literature, discussed in the Related Work section of this document. We start this section with an overview of the architectural and design aspects common to all three project objectives and we then discuss the specifics of each in more detail. Finally we discuss optimisations, implementation issues and testing. General Architecture Across all Three Deliverables We first selected our toolset. We chose to program all three deliverables in the Processing programming language [16] due to several advantages it provides. Firstly, it is fast, stable and is designed for projects that focus on visualisation and interaction. It also has third-party libraries for data interpolation and sound synthesis (ToxiMath and Minim) so all the required functionality already existed. Secondly it is an excellent language for rapid prototyping as the syntax is both simple and powerful. Lastly, it is built upon Java, in which we have extensive experience, and so where necessary we were able to use more advanced Java functionality. Processing also creates cross-platform executables, providing an extra level of flexibility. We adopted an evolutionary iterative development model for all three deliverables as we expected that we would need flexibility with feature implementation due to the experimental nature of the work. Using this methodology, feature plans can be refined as the project progresses rather than being defined permanently at the start of the project [31]. We used the Bazaar version control system due to our familiarity with it and its ease of use. Each time we added a new feature to any deliverable we created a new branch of the codebase, separate from the main codebase. This allowed us to develop each feature in isolation without concern of breaking existing stable code. Once each feature had been developed and fully tested we merged the new branch into the main codebase. Basic Deliverable: Sonification and Visualisation Engine We started by constructing a sonification and visualisation engine to create auditory and visual graphs for arbitrary inputs of bivariate graph data. This was to be the foundation of the project as both the intermediate and advanced deliverables built upon this technology. The engine supports static and real-time interaction with the auditory graphs and produced accurate corresponding visual graphs for user training purposes. A simple graphical user interface was created for both static (figure 3a) and real-time (figure 3b) interaction modes. The static interaction mode enables the user to start or stop the sonification of the auditory graph by pressing a „start/stop‟ button. The user could stop the sonification prematurely at any time, but could not pause; playing again would start from the beginning of the data. The


auditory graph would sonify over a configurable period of time (for example, 5 seconds) and during playback, the current position in the graph was represented by a line on a status bar corresponding accurately to the visual graph which was placed immediately underneath. The real-time interaction mode was very similar except that there was no „play/stop‟ button. The button instead became a „mute/unmute‟ button and the user could interact with the graph by moving their mouse cursor over the status bar area. The horizontal position of the mouse cursor represented the current position to be sonified in the auditory graph, which was sonified continually whilst the mouse was over the status bar, and not otherwise. The sonification and visualisation engine takes an input of an array of integers representing data points and converts them into an auditory graph. Optionally, an accurate corresponding visual graph can be created too, for later use in user training or the trial. It was important that both representations therefore corresponded exactly to each other and to the input data, for the user to learn about one from the other and for accurate trial results. In order to create continuous graphs the data points were interpolated and values were Figure 5. Overview of the conversion process. mapped to pixel and hertz values for the visual and auditory outputs respectively. Figure 5 gives an overview of this procedure. Since both the visual and auditory graphs must match exactly, they must have exactly the same number of plotted points and so we were limited to either the pixel height of the visual graph or the chosen audio range, whichever was smaller. We created auditory graphs with a range of 60Hz to 1100Hz as this is within the human vocal range [29], so by default graphs are limited to 1040 data points. Figure 6 gives an overview of the program structure and components of the engine. We wrote the software in a modular fashion, keeping related functionality in the same class to minimise development time and complexity when extending functionality for later deliverables. Program control flow is dictated by the Main class, which takes in data as an input for sonification and visualisation by the AuditoryDataProducer and VisualDataProducer classes respectively. These two classes normalise, interpolate and convert the Fig. 6. Diagram representing system components. data into the relevant output formats (either hertz values or pixel values) and then export those outputs to files (either hertz value files or images). The hertz value files are then interpreted by the Sonifer class, which includes the graphical user interfaces for both real-time and static interaction modes and an output mechanism to play the given pitch values. Figure 7 outlines the main algorithms used in the engine. Both graphs must be an accurate representation of the input data. In order to create a better impression of continuous graphs the data points were interpolated so that interval data points could be plotted. The auditory and visual graphs must correspond exactly to each other if the user is to learn about one from the other so they must have the same number of plotted points. Intermediate values must be


interpolated and mapped to pixel and hertz values for the visual and auditory outputs respectively. The graph should be stretched to fill the available data range (i.e. maximum hertz and pixel ranges) as we are primarily interested in graph shape detection for the trial.
1. 2. 3. 4. 5. 6. FOR EACH pair of values, value1 and value2, IN inputArray Interpolate between value1 and value2, storing the results in tempArray END FOR conversionArray = Normalise(tempArray) pixelArray = ConvertToPixels(conversionArray) hertzArray = ConvertToHertz(conversionArray) 1. ConvertToPixels(tempArray) 2. FOR EACH tempValue IN tempArray 3. pixelValue = tempValue * GRAPH-HEIGHT 4. Store pixelValue in outputValues 5. END FOR 6. Return outputValues 7. END FUNCTION Note: GRAPH-HEIGHT is the pixel height of the graph.

Note: Interpolation was performed using the external ToxiMath library. The normalisation and conversion functions are elaborated on to the right and below. 1. ConvertToHertz(tempArray) 2. FOR EACH tempValue IN tempArray 3. hertzValue = MINHERTZ + tempValue * MAXHERTZ 4. Store hertzValue in outputValues 5. END FOR 6. Return outputValues 7. END FUNCTION Note: MINHERTZ and MAXHERTZ are the minimum and maximum allowed hertz values in the auditory graph.

1. Normalise(tempArray) 2. minValue = MIN(tempArray) 3. maxValue = MAX(tempArray) 4. FOR EACH tempValue IN tempArray 5. normalisedValue = (tempValue – minValue) / (maxValue – minValue) 6. Store normalisedValue in outputValues 7. END FOR 8. Return outputValues 9. END FUNCTION

Figure 7. Pseudocode algorithms for the conversion procedures in the sonification and visualisation engine.

Intermediate Deliverable: Trial Design and Engine We designed the trial to determine whether the interactive mode used in bivariate auditory line graphs had an impact on the performance of a simple matching task. The null hypothesis was that participants will not have increased performance when matching auditory and visual graph shapes using real-time interaction compared to static interaction. The alternative hypothesis was that participants will have increased performance when matching auditory and visual graph shapes using realtime interaction over static interaction. The software consisted of a tutorial and eight trial questions. A brief training session has been shown to increase auditory display performance [3]. In a similar trial, Brown and Brewster [27] trained participants to use the interface and interpret auditory graphs. They also included “two tasks of the same type as those in the experiment itself, after which the experimenter provided feedback in order that participants could judge their performance.” Our tutorial was similar and consisted of an explanation of sonification and auditory graphs followed by a practice screen for both real-time and static interaction techniques, showing a visual graph whilst simultaneously playing an auditory graph (figure 8a). After this, two mock trial questions were displayed, one for each mode, so that the participants could familiarise themselves with the trial layout (e.g., figure 8b). Each of the eight questions consisted of an auditory graph and four visual graphs. Of the four visual graphs, one was the target and the others were

Fig. 8a. Screenshot of one of the tutorial pages.

Fig. 8b. Screenshot of a trial question with the bottom-right choice selected as the target.


randomly chosen distracters. The participants were asked to choose the visual graph they believed matched the auditory graph by clicking on it and pressing the „next‟ button (figure 8b). Of the eight questions, four were real-time and four were static. For each real-time question there was a corresponding static question for the same auditory graph in order to compare the performance difference. Hence there were four auditory graphs tested over eight questions. The same four auditory graphs were played to all participants to ensure comparable results. In order to ensure that the results were not biased due to question design, it was important that the question order and the distracters were fully randomised. It was also important that none of the distracters appeared more than once throughout the trial as this could encourage participants to guess their answers. The position in a two-by-two grid of the four visual graphs in each question (see figure 8b) was also randomised so that the target graph would not always appear in the same location. The question order was also randomised across participants with the restriction that no two questions of the same auditory graph (i.e. the static and real-time versions of the same question) were presented consecutively as this would have been made the second instance easier. To create a fully random question sequence, all possible permutations of the question orders were pre-computed and one was chosen at random for each execution of the trial. The time and memory costs of calculating all permutations grows quickly as the number of questions grows and so this technique would not be suitable if we could not 1. Store all possible valid permutations of question order in perm2DArray guarantee that the number of 2. Select a random element from permArray and store in questionsArray questions would stay small. As we 3. Store all distracter graphs in distractersArray 4. Shuffle the elements in distractersArray to be in a random order knew that the number of questions 5. FOR EACH question IN questionsArray would be eight this was acceptable 6. Assign the first 3 distracters from distractersArray to question Remove considering the benefits of the 7. END FOR the first 3 distractors from distractersArray 8. guarantee of randomness. Using the Figure 9. High-level algorithm for question construction. inclusion-exclusion principle [32] of combinatorics we can calculate the number of non-consecutively appearing permutations of 8! 7! 6! 5! 4! four distinct questions of length 8 to be 4 ∙ 24 − 4 ∙ 23 + 4 ∙ 22 − 4 ∙ 21 + 4 ∙ 20 = 864. 0 1 2 3 4 This small number of permutations can be calculated almost instantaneously on the computer used to run the trial so there was no concern for performance issues. A high-level overview of the randomisation procedure is outlined in figure 9. We chose bivariate line graphs as stimuli because they are a simple and familiar graph type which should require minimal participant training. Bonebright et al. [8] found that certain sonified graph types are easily confused “when they have no easily discernable „envelope‟” (i.e. shape). We therefore took care to create graphs with distinct shapes as far as possible for the trial. Figure 10 shows the graphs that were used in the trial.
Qn denotes the question N° of each data line. S & R state whether the question was static or real-time. The next 4 N°s on each line denote the visual graph IDs the participant saw per question. * denotes a graph was a target. The 5th N° denotes which graph was chosen. The final N° is the reaction time per question in miliseconds.

Q1 S *3 26 19 18 18 7424 Q2 S *1 15 16 12 12 4324 Q3 S 10 *2 25 13 13 3052 Q4 R 22 14 *7 21 21 3432 Q5 R 20 *6 9 31 31 2315 Q6 R *8 30 29 28 28 8221 Q7 R *5 23 17 32 32 2425 Q8 S 11 *4 24 27 27 2521

Fig. 10. Graphs in the trial. The first 4 were targets & the rest were distracters. Fig. 11. Data format for logging. 9

For each candidate, the system took detailed logs of user activity and trial layout for later analysis. The Data points we collected included the response time for each question, question type (static or real-time), question order, visual graph allocation, distracter and target graph placement per question and the correctness of each response. This data logging is detailed in figure 11. The data we collected was more than was necessary to run our statistical tests which allowed us to conduct a detailed analysis of user behaviour. We created a simple custom interrogation database using Java, containing the data for each candidate to make this analysis faster and easier. A great variety of questions could then be asked via programmed calls to the database, for example: „for candidates who took the longest time on a question, which distracter graphs were assigned to the target graphs and were they very similar in shape?‟ The Results section of this paper elaborates on such questions and answers. We do not describe the database implementation in detail here as it was not a key project deliverable. Figure 12 gives an overview of the trial engine, which builds upon the basic engine. The Main module coordinates activity between all other modules. Each trial question has four associated visual graphs and one auditory graph, which are rendered by the basic engine and then organised and randomised by the Question module. The TrialCoordinator module draws the user interface, randomises the trial question order and logs the data to a database.

Figure 12. Diagram representing the components of the system.

Before running the trial we conducted informal „dry runs‟ with colleagues, who were not involved in the later trial, in order to get user feedback to improve the software. The main suggestions for improvement which we implemented were to make graph selection more obvious by introducing a red image border when the user clicks a graph, larger „play/stop‟ and „next‟ buttons and a full-screen option. A design flaw that emerged as a result of the „dry runs‟ was that users occasionally clicked the „next‟ button without selecting a graph. We therefore modified the software to ensure that users could not progress unless a graph had been chosen; otherwise the „next‟ button was disabled. We also made speed improvements using Java thread control mechanisms to make the sonification code more efficient. Advanced Deliverable: Trivariate Auditory Intensity Graph Engine We extended the sonification engine to support data sets of three dimensions: two spatial dimensions and one intensity dimension (which can be used to represent, for example, height, temperature, etc.). Rather than taking an input of integers, the advanced engine takes an image as an input because images are an elegant data storage solution for this application. They can be interpreted visually at a glance, are easy to edit and contain enough dimensionality to represent the necessary data points. Intensities (e.g. heights in topographic maps) are represented by colours, with lighter colours corresponding to higher intensities,


which are then mapped to higher pitches. Figure 13 shows examples of graphs supported by the engine. We modified the interaction techniques in the advanced engine to make sense in the new data landscape. Three techniques were built: static, blackbox and real-time. The new interaction technique was added due to the ill-suitability of the static playback on the larger trivariate data sets. The extra dimension means that there is too much data simply to statically play each value. For example, even a small graph of 200×200 pixels would have 40,000 data points to Figure 13. Interface for advanced objective. sonify: if each were only played for 0.01 seconds this a1: Simple topographic graph (dummy data). a2: E.g. of black-box interaction technique. would take 6. 6 minutes to sonify. The black-box b: Sonification interface for Met Office data. interaction technique we developed was between static and real-time in terms of interactivity; the user clicks a position on the image representing a vertical value on the image plane and all the values on the horizontal line at that point are sonified, allowing the user to get a static „snapshot‟ of a cross section of the intensity data at that point (figure 13-a2). In our 200×200 pixel example this would take only two seconds to sonify one row whilst retaining a somewhat static approach. We successfully implemented the engine so that it takes an input of any image with a predefined colour palette corresponding to intensity intervals. Figures 13-a1 and 13-a2 show examples of intensity graphs using dummy data and four intensity intervals. Up to 16777216 intervals are supported as this is the number of unique colours available in a JPEG image [33] although in practice only a small number of intervals are required to interact with most data sets. For example, the sunshine duration intensity graph released by the Met Office [34] has nine intensity intervals (figure 13b), for which we successfully built a sonification using the engine. Figure 14 shows the main conversion and sonification algorithms used in the engine.

Figure 14. Pseudocode algorithms for the hertz conversion and various sonification procedures in the advanced engine.


Implementation Issues, Testing and Validation We did not encounter any major implementation issues due to the flexible and agile programming techniques we used. After programming the trial software our „dry run‟ tests allowed us to check the ergonomics and usability of the software as well as how it would work under real trial conditions. We initially had some concern that the higher hertz values in the initial auditory graphs could be uncomfortable if the data set had a large variance so during the normalisation and interpolation phases we ensured that the output values always lay between the hertz values of the average human vocal range. We ran several software and non-software tests to validate our deliverables. It was very important that the data logging for the trial was accurate. We measured the accuracy of performance timings recorded by our software by using the Linux „time‟ command to check that they corresponded to the true values. We also hand-checked the accuracy of the question order logs data by running the trial whilst taking notes ourselves and comparing those to the output. We also checked that the question order, as well as the distracter graph assignment and placement, were well randomised by automatically generating a large number of trial data sets and checking the distribution of the generated questions. Output data was carefully scrutinised to ensure that it was entirely anonymous and that the only information linking it to the participant was the anonymous participant code. We also used audio tuning software to measure the accuracy of the tones produced by our engines, to check that intended outputs were playing at correct frequencies. RESULTS We started by creating the sonification and visualisation engine necessary for the trial. We were successful in building a system flexible enough to create the large variety of graph shapes required for the trial (see figure 10). We were also successful in creating useable static and real-time interaction techniques at opposite ends of the interactivity spectrum which participants were able to use very effectively with only a few minutes of training. In fact, participants did surprisingly well at interpreting the auditory graphs using both techniques. The main research question we asked was „Does the interaction technique used in bivariate auditory line graphs have an impact on the performance of a simple matching task?‟ The null hypothesis was that participants will not have increased performance when matching auditory and visual graph shapes using the real-time interaction technique compared to using the static interaction technique. The alternative hypothesis was that participants will have increased performance when matching auditory and visual graph shapes using real-time over static interaction. The trial was run on fifteen volunteer participants in the Durham Visualisation Laboratory and consisted of a pre-session questionnaire, a tutorial, eight trial questions and a post-session questionnaire. Before starting the trial, participants were tested for visual acuity using the Snellen eye test; they read a row of 9mm-high letters at a distance of 6 meters which indicates 20/30 vision if successful. They were also required to take the BSHAA online hearing test [35] in which participants listen to four sine tones at 500Hz, 1kHz, 2KHz and 4kHz. If participants could read all the Snellen letters and hear all four tones they were eligible for the trial. The trial was run on a Dell Studio XPS 1640 notebook using its integrated soundcard and Senheiser HD 215 headphones. The pre-session questionnaire consisted of demographic questions regarding participants‟ musical ability, computer usage, handedness, gender, confidence at interpreting data and their level of mathematical education. They then signed consent forms and were assisted through a short tutorial on how to use the trial software. After this they were instructed to complete the remainder of the trial


independently and were told that there was no time limit but that they should answer the questions as quickly and accurately as possible. After the trial, participants were asked to complete a non-compulsory post-session questionnaire which asked open ended questions about their interaction technique preference and for feedback on the trial itself. We have two resulting data sets from the trial as we measured two performance indicators; score and mean reaction time. For each participant, the score is the number of correct responses per interaction technique and the reaction time is the mean time to answer a question per technique. Therefore the independent variable was the interaction type used (i.e. static or real-time) and the dependent variables were score and mean reaction time. The static playback lasted for 5 seconds if played through from start to finish. Participants could choose to stop and start again at any point. The real-time playback lasted as long as the participants interacted with it. The data were partitioned into real-time and static question responses (four of each) and paired so that any differences between the two techniques for the two indicators could be measured. We first consider the scores, which turned out to be less useful due to very high scores across all participants. We then consider the reaction times, which produced much more useful results. We also consider the demographic and qualitative data collected from pre and post-session questionnaires.
Participant ID
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Mean:

Real-time Score
3 4 4 4 4 4 4 4 4 3 4 4 4 4 4 3.9

Static Score
3 4 4 4 3 4 4 4 4 3 4 4 4 4 4 3.8

Table 1. Participant scores.
Participant Static Time Real-time ID Mean Time Mean
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 17.89 11.35 23.10 9.92 14.16 20.88 12.10 10.70 11.82 18.89 20.06 17.74 11.37 14.28 11.23 15.81 11.08 25.13 24.48 12.44 24.81 16.70 11.64 12.47 32.59 17.57 35.51 14.91 14.24 16.52

We first consider the scores. For each of the eight questions asked, there was one static and one real-time version of the same question. Participants had a choice of four responses and so the chance of guessing correctly was 25%. The null hypothesis was that there was no significant difference in scores for questions of real-time verses static interaction. The alternative hypothesis was that there is a significant difference between the scores for the two interaction types. The scores for real-time and static were individually summed for each participant to compare the difference between the two techniques and are presented in table 1. All participants scored very highly (≥75% correct) and most answered all questions 15.03 19.06 Mean: correctly. There was a very slight performance increase for real-time over static (0.06 from mean). We ran a two-way Table 2. Participant mean times (sec). paired t-test on these results which returned a p-value of 0.334. We therefore fail to reject the null hypothesis for the scores analysis; there is no significant difference in the results. Our first finding was therefore that participants generally found it equally easy, based on feedback and scores, using both techniques, to identify the target from amongst the distracters. This non-result makes the reaction time data set more useful for analysis, which provides much more meaningful results. For each of the eight questions (four real-time, four static) we also timed how long participants took to answer them. The average timings for each technique were calculated by summing the four respective question reaction times for each participant and dividing by four to give the mean response time per technique per participant.


Hence, each participant has a mean static and real-time timing (table 2). The null hypothesis was that there was no significant difference in reaction times between the two techniques. The alternative hypothesis was that there was a significant difference. A two-way paired t-test was run on the data, producing a p-value of 0.028 and hence, assuming the samples came from a normal distribution, we can reject the null hypothesis and assert with 97.2% confidence that the difference was not a result of chance. A normality test was run to test this assumption by taking the absolute differences of the timings and then plotting a Q-Q plot (figure 15). If this plot follows a roughly linear distribution, the data can be assumed to be normal [36]. The plot is fairly linear but does have slight curvature and also had anomalies from three participants. As the sample size was 15 there was more flexibility in how linear the distribution could be however as this was a borderline case, for the sake of rigorous analysis we declared the normality test inconclusive Fig 15. Q-Q plot of mean timings. and ran a Wilcoxon signed-rank test on the data. This is a nonparametric version of a paired t-test and therefore does not assume the sample comes from a normal distribution [37]. The test produced a p-value of 0.017 and hence we were again able to reject the null hypothesis and with a 98.3% confidence level, declare that the difference was not due to chance, even if the data population distributions do not follow the normal distribution. Our second finding was therefore that, on average, there was a difference in reaction time between the two interaction techniques, with real-time interaction having slightly higher reaction times than static on average. Participant behaviour changed in interesting ways depending on the interaction technique used. The mean reaction time across all participants for real-time was 19.06 seconds and individual question timings ranged from 11.08 to 45.51 seconds. The mean reaction time across all participants for static was 15.03 seconds, with individual values ranging from 9.92 to 23.10 seconds. This demonstrates an interesting trend: on average, participants spent longer interpreting the graphs using the real-time technique, despite an almost identical accuracy rate for the two conditions. This may suggest that participants felt more confident in making a decision sooner with static, and yet almost always it was just as accurate. Our third finding was therefore that for a matching task with simple bivariate graphs, real-time interaction seems to be unnecessary and static interaction seems to be Figure 16. Target graphs from the trial. a quicker, more convenient way to make matches. Whether or not participants gained extra insight by Question & Mean time Mean time per mode exploring for more time using real-time interaction interaction mode per question Real-time C 20.52 remains an open question. Figure 16 shows the target graphs used in the trial, listed as A, B, C and D. Each had a corresponding real-time and static question in the trial. Overall, participants' response times for the graphs were ordered A, D, C and B from longest to shortest. Our fourth finding was that the amount of time spent answering a question varied more widely for realtime questions than for static ones. The reaction time differed by at most 1.86 seconds for static and 3.86 seconds for real-time: not only are participants spending more time answering
Real-time D 19.64 Real-time A 19.42 19.06 Real-time B 16.66 Static A 15.98 Static D 15.33 Static B 14.70 15.03 Static C 14.12 Table 3. Mean times (sec) by interaction mode per question, ordered by time (descending).


in real-time mode, they are also spending more time on certain graphs than on others. In contrast, reaction time was comparatively uniform for static interaction. This may suggest that participants spent more time thinking about the graph when using real-time interaction or alternatively it could imply that they were more confused. Table 3 lists the mean reaction times for each question by interaction technique. The time taken to evaluate each target graph also differed depending on the interaction technique used (rather than overall, across both modes). Whereas for real-time graphs the response times were ordered C, D, A, B from longest to shortest, they were ordered A, D, B, C from longest to shortest for static graphs. It is interesting that graph C, which had the longest response time for real-time (by quite a margin: 1.46 seconds from the mean) contains the most detail but the same graph had the least response time for static interaction (again by quite a margin: 0.91 seconds). This meant that the average response time for C as a static graph was 97.8% of the average response time for all static graphs, while as a real-time graph the response time for C was 107.7% of the average response time for all real-time graphs. Participant Demographics Figure 17 shows the demographic variables and post-session questionnaire statistics collected on participants. There were 15 participants, all aged 18-25. 73% were male, 87% were righthanded, 93% used a computer on a daily basis and 80% used their computers both at home and for work (whilst 13% used computers only for work, and 7% only at home). The majority of participants (73%) reported having at least average musical ability (7% expert, 20% confident, 46% somewhat confident) and most (87%) were educated in mathematics to Alevel standard or above. The majority (86%) expressed confidence at interpreting data from graphs, with 33% professing expert level, 53% confident, 7% somewhat confident and 7% novice. After completing the trial, 54% of participants expressed a preference for the static method, 13% preferred the real-time method and 33% held no preference. 47% of participants wrote in the post-session unstructured feedback question that an option to use both real-time and static interaction for different tasks (for example, gaining a general overview versus specific point interrogation) would be preferable; real-time was perceived as better for some task types and static for others.

Figure 17. Participant demographics and questionnaire statistics.


After running the trial, we applied the knowledge and experience taken from the basic and advanced deliverables and successfully created a trivariate sonification engine flexible enough to handle any given input image with a pre-determined colour palette (corresponding to intensity values). We then successfully extended that to support real-world data sets from the Met Office. The real-time interaction technique transplanted very well and in our opinion was the most intuitive interaction technique for these more complex data sets. The static technique was usable; however, it was not practical due to the sheer amount of data and the time constraints that it imposed on the listener. This is in contrast to the simpler graphs where, based on participant feedback, static was generally preferred. We were successful in creating a middle ground: the black-box interaction technique which allowed for static, but more limited feedback, one row at a time.

EVALUATION The primary aim of this study was to determine whether real-time interaction with bivariate auditory line graphs can increase performance for graph shape identification when compared to static interaction. This required building a basic sonification engine and a trial engine to test this hypothesis. The secondary aim was to determine whether it is possible to implement a trivariate auditory graph engine using the static and real-time interaction techniques. We were successful in answering these research questions. In this section we evaluate our project from three perspectives. First we will consider the strengths and weaknesses of the software produced: the two sonification engines and the trial engine. Second, we will analyse the strengths and weaknesses of the trial itself. Lastly we discuss our project organisation approach and lessons learnt. Strengths of the Sonification Software The trial engine software was robust, stable and the user interface was accessible and usable for all the participants: none had any issues in completing the trial questions. Logging was thorough, accurate and flexible enough to analyse the recorded data in depth without difficulty. The underlying basic engine for the software is flexible in its ability to take in arbitrary data sets of any range. All the graphs we required for the trial were created quickly and easily using the basic engine. The algorithms used to convert between the input data and the output visual and auditory graphs were fast, even when run on a low-end laptop: the conversion process only requires one pass through the input data requiring O(n) time. The sonification interfaces were intuitive and easy to use, as is evidenced by user feedback and by the very high scores for both the static and real-time interfaces: no users reported difficulty and the high scores corroborate this. Ergonomically, the software was successful too; all pitch hertz values were mapped to be within the typical human vocal range and no participants or other users reported any discomfort. The advanced engine is also easy to add data sets to: we were able to quickly build an auditory graph using real-world data from the Met Office. The software engineering process was successful too: we experienced no issues during the project due to careful use of versioning software, modular code and thorough testing. Limitations of the Sonification Software In all our deliverables we used a linear mapping of pitch for the auditory graphs. However, human pitch perception of musical intervals is approximately logarithmic, not linear [38]. This may cause interpretation inaccuracies at higher pitch intervals in the graphs because humans become less sensitive to pitch changes as the pitch increases: the difference in ability to detect small changes in pitch is on average 6 times less accurate at the 2kHz tone range


than at 100Hz tone range [39]. A redevelopment of the hertz conversion algorithm to take this into account would potentially produce more accurate auditory graphs. The randomisation algorithm for the trial software relies upon a pre-computed sequence of all possible permutations of the question orders to ensure the trial was fully randomised across participants. Whilst this was an excellent solution for our purpose, as we could guarantee that only eight questions would be asked and this approach ensures very random question orders, if the trial software were to be extended to support larger question sets an alternative approach would be necessary. This is because the memory and time requirements of computing all possible permutations of question orders grows very quickly as the number of questions grows. One less computationally expensive alternative would be to use a stochastic method: randomly generating permutations and checking each for suitability on the fly. Several limitations of the sonification engine are a result of its being designed primarily as a backend system for the trial software rather than a comprehensive front-end solution for nontechnical end-users. Although they are limitations of the system they do not detract from the overall success of the project because the engine worked as intended for the project. It is however still well worth considering how it could be improved for future research and extension for wider reach and utility. The user interface for the auditory graphs is basic and could be expanded to support more functionality and deeper data interrogation capabilities, particularly for the real-time interaction technique. Currently, it is difficult for non-technical users (i.e. those without programming experience) to import new data sets into the engine and a better user interface could make this easier, which would widen the reach of the software significantly. Although the sonification engine supports arbitrarily sized data inputs, the current user interfaces are hard coded to allow only a specific data set size: 15 integer data points for bivariate graphs and 210×210 pixel image maps for trivariate graphs. This is because our project required only graphs of these sizes. For the basic sonification engine, it was important that all graphs for the trial were identical in size, so to minimise the chance of errors we created the user interface with this limitation. As we were primarily interested in shape detection and matching tasks, another limitation of the bivariate engine is that it normalises and scales all data to be consistent across all graphs that it produces. This is not ideal for real-world data interrogation unless shape is the only quality of the data set that the user is interested in, as in our case. Lastly, the trivariate auditory input graphs currently need to be created by hand, requiring some basic image manipulation skills. Tools could be developed to help with or to fully automate this process and to make custom colour palettes correspond to intensity intervals without requiring programming knowledge. Strengths of the Trial The trial was very successful. The number of participants was high enough to have statistically significant results from t-tests and Wilcoxon signed-rank tests. Our results were interesting and contribute new knowledge to the field of sonification. The trial was built in such a manner as to increase the credibility of the results wherever possible: questions were fully randomised across participants and allocation of distracter graphs to target graphs for each question was also randomised to remove potential biases due to question structure. We were able to say with very high confidence levels, due to the statistical tests that we ran, that static interaction provides better performance than real-time interaction for bivariate auditory line graphs in a matching task, which was an unexpected and very interesting result. Participant feedback was positive and many of them showed an interest and appreciation for an alternative way to interact with data.


Weaknesses of the Trial Although the number of participants was good, their demographics were very similar in two areas (see figure 17). It is therefore possible that the results are not generalisable to the general population, potentially introducing external validity issues. All but two of the participants were computer science undergraduates and all were between the ages of 18 and 25. All participants had GCSE or above mathematical training which is the norm in the United Kingdom, however the majority also had training at A-level or above. It is possible that these potential external validity issues caused the borderline normality result mentioned in the Results section of this document; however these issues were greatly reduced by using the Wilcoxon signed-rank test (which does not require that participant results adhere to a normal distribution), in addition to the t-test as a precaution. The interesting result that most candidates scored 100% for both interaction modes (see table 1) may have been a result of the trial design. Although many of the graphs used in the trial have fairly complex shapes for simple bivariate graphs (see figure 10), all four target graphs had distinctive features which were not present in the majority of the distracter graphs. The graphs were also made specifically for the trial and not from real-world data. These two factors could contribute to an internal validity issue. This was minimised by our timing of participant responses as a precaution, so a useful analysis of performance was still possible by using reaction time rather than score as the primary dependent variable. Project Organisation Overall the project was well organised and managed. All objectives were achieved ahead of schedule and useful, useable results came from each deliverable. Our incremental and modular approach to software development allowed quick feature evolution and easy migration from basic to intermediate to advanced deliverables as previous modules were reused, evolved and built upon very easily. Modules were created for the basic sonification engine, the trial software and the advanced sonification engine, which in turn consisted of sub-modules. For example, the trial software consisted of a sonification sub-module, built from the basic engine module, a questioning module and a user interface module. This allowed for quick improvements to be performed on one module without affecting the others, for example new trial question formats could be added and tweaked very quickly without affecting the rest of the trial software which significantly decreased development time. Careful version control and regular backups meant that no time was lost due to computer or human failure and that new features could be introduced with confidence into the working software branch without fear of introducing hard to track bugs. Lessons Learnt If we were to run a similar project again we should not change the majority of our approach and overall we feel that the project was a success. However certain elements of the project could be improved. If we were to run the trial again we would attempt to use a larger and more diverse group of participants, especially taking into account age and educational background, in order to increase the chance that samples would come from a normal distribution. We would also take care when creating the bivariate graphs for the trial. We would increase the complexity of some of the graphs and ensure that some distracter graphs are similar to target graphs. This would increase the difficulty of the matching task and make score more meaningful as a performance measure.


CONCLUSION Overall, the project was a success: all objectives were met and we gained interesting and usable results from the trial. We performed an in-depth evaluation of the literature on sonification interaction techniques and found no research comparing the performance impact of using static over real-time interaction, so this became our focus. We created a flexible sonification and visualisation engine for simple bivariate line graphs which produces accurate corresponding visual and auditory graphs for a given data set. We then developed two interaction techniques at opposite ends of the interactivity spectrum: static and real-time interaction, which participants were able to use with minimal training. We then built a fully randomised trial to test the differences in performance on a simple graph matching task using the two interaction methods with fifteen participants. They did remarkably well on the questions, with almost all getting scores of 100% for both techniques. Reaction times were therefore used rather than scores for the bulk of the statistical analysis. We were surprised with the result that participants had increased performance using the static rather than real-time interaction technique and we suggest that this was due to the simplicity of the task and the graphs used: the extra interrogation capabilities of the real-time mode were less useful in this context. However, when asked to provide open-ended feedback, many participants did express a desire to have access to both techniques depending on the task type as each had their strengths and weaknesses. Our study provides a new and surprising result in the field of sonification: more interactivity with auditory graphs is not necessarily always beneficial for all data landscapes, even when the more interactive technique can be used to simulate the less interactive technique (for example by dragging the mouse slowly from left to right to mimic static interaction). Finally we extended the sonification engine using the lessons learned from the previous two deliverables to support trivariate data sets in the form of intensity graphs, including realworld data from the Met Office. We re-implemented the static and real-time interaction techniques to make sense in the new data landscape and introduced a third, middle ground technique to address the impracticality of the static technique for these larger data sets. Realtime interaction remained very usable in the trivariate engine. There are several interesting directions for future research. Primarily, we would be interested to determine whether the more interactive real-time technique would provide better performance and user feedback over more static interaction techniques when used in complex data landscapes such as with trivariate intensity graphs. We expect this to be the case, which would be the opposite of our findings for simple graphs and would suggest that the suitability of static and real-time interaction techniques is task dependant. Our own usage of our trivariate sonification engine and informal conversations with colleagues suggests that this may be the case and a further trial on this question would be an interesting contribution to the field. We would also be interested to determine whether the high scores in our trial were due to the simplicity of the particular graphs chosen or simply due to the simplicity of bivariate auditory graph matching tasks. We suspect that as the amount of data grows, increased interactivity becomes essential to hone in on the data sections of interest. Lastly, the engine could be extended to deal with more varied data sets and to explore different mappings rather than just pitch and time. A more efficient and robust solution could be developed using Java or C++ rather than Processing and a more flexible sound library other than Minim could be used to allow for representation of more data dimensions, for example through dynamic volume control or three-dimensional sound.


1. Data Sonification and Sound Visualization. Kaper, Hans G., Tipei, Sever and Wiebel, Elizabeth. s.l. : Computing in Science and Engineering, 1999. arXiv:cs/0007007v1. 2. Sonification Report: Status of the Field and Research Agenda. Kramer, G., et al. 2010. 3. Auditory Interfaces and Sonification. Nees, M. A. and Walker, B. N. The Universal Access Handbook pp.507--521 [book auth.] Constantine Stephanidis. New York : CRC Press Taylor & Francis, 2009. 4. Taxonomy and Definitions for Sonification and Auditory Display. Hermann, Thomas. s.l. : Proceedings of the 14th International Conference on Auditory Display, 2008. 5. Theory of Sonification. Walker, Bruce N. and Nees, M. A. Principles of Sonification: An Introduction to Auditory Display [book auth.] T. Hermann, A. Hunt and J. Neuhoff. In press. 6. Desktop Data Sonification: Comments On Flowers et al. Flowers, J. H. and Turnage, K. D. s.l. : ICAD, 1996. ACM Trans. Appl. Percept., 2(4):473–476. 7. A Perceptual Framework for the Auditory Display of Scientific Data. Barrass, Stephen. 4, s.l. : ACM Transactions on Applied Perception, 2005, Vol. 2. ACM 1544-3558/05/1000-0389. 8. Testing the Effectiveness of Sonified Graphs for Education: a Programatic Research Project. Bonebright, Terri L., et al. s.l. : Proceedings of the 2001 International Conference on Auditory Display, 2001. ICAD01-62. 9. Sonification of Statistical Graphs. Peres, Camille S. and Lane, David M. s.l. : Proceedings of the 2003 International Conference on Auditory Display, 2003. 10. Brief Training for Performance of a Point Estimation Sonification Task. Walker, B. N. and Nees, M. A. s.l. : Proceedings of the International Conference on Auditory Display, 2005. ICAD05-1. 11. A Toolkit for Interactive Sonification. Pauletto, S. and Hunt, A. s.l. : Proceedings of the International Conference on Auditory Display, 2004. 12. "I Hear the Pattern": Interactive Sonification of Geographical Data Patterns. Zhao, Haixia, Plaisant, Catherine and Shneiderman, Ben. s.l. : CHI Extended Abstracts on Human Factors in Computing Systems, 2005. 1-59593-002-7. 13. Interactive 3D Sonification for the Exploration of City Maps. Heuten, Wilko, Wichmann, Daniel and Boll, Susanne. s.l. : Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles, 2006. 1-59593-325-5. 14. The Importance of Interaction in Sonification. Hunt, Andy and Hermann, Thomas. s.l. : Proceedings of the International Conference on Auditory Display, 2004. ICAD04-1. 15. Real-time Control of Sonification Models with a Haptic Interface. Hermann, Thomas, Krause, Jan and Ritter, Helge. s.l. : Proceedings of the 2002 International Conference on Auditory Display, 2002. ICAD02-1. 16. Processing. [Online] http://processing.org. 17. Digitizer Auditory Graph: Making Graphs Accessible to the Visually Impaired. Choi, Stephen H. and Walker, Bruce N. s.l. : Proceedings of the 28th International Conference Extended Abstracts on Human Factors in Computing Systems, 2010. 978-1-60558-930-5. 18. The vOICe. Seeing With Sound. Meijer, Peter. [Online] [Cited: 11 January 2012.] http://www.seeingwithsound.com. 19. Sonification Sandbox: A Graphical Toolkit for Auditory Graphs. Walker, Bruce N. and Cothran, Joshua T. s.l. : Proceedings of the 2003 International Conference on Auditory Display, 2003. ICAD03-1. 20. Cultivating Design Patterns for Auditory Displays. Adcock, Matt and Barrass, Stephen. s.l. : Proceedings of the 10th Meeting of the International Conference on Auditory Display, 2004. 21. Relative Intensity of Auditory Context for Auditory Graph Design. Nees, Michael A. and Walker, Bruce N. s.l. : Proceedings of the 12th International Conference on Auditory Display, 2006. 22. A Data Sonification Design Space Map. de Campo, Albert. s.l. : Proceedings of the 2nd International Workshop on Interactive Sonification, 2007. Ison07-1. 23. Universal Design of Auditory Graphs: A Comparison of Sonification Mappings for Visually Impaired and Sighted Listeners. Walker, Bruce N. and Mauney, Lisa M. 3, s.l. : ACM Transactions on Accessible Computing, 2010, Vol. 2. 24. Sonification Design and Metaphors: Comments on Walker and Kramer, ICAD 1996. Walker, Bruce N. and Kramer, Gregory. 4, s.l. : ACM Transactions on Applied Perception pp.413-417, 2005, Vol. 2. 25. Using Sonification for Mining Time Series Data. Last, Mike and Gorelik, Anna. s.l. : Proceedings of the 9th International Workshop on Multimedia Data Mining, 2008. 978-1-60558-261-0. 26. Data Sonification from the Desktop: Should Sound Be Part of Standard Data Analysis Software? Flowers, John H., Buhman, Dion C. and Turnage, Kimberly D. 4, s.l. : ACM Transactions on Applied Perception, 2005, Vol. 2. 27. Drawing By Ear: Interpreting Sonified Line Graphs. Brown, Lorna M. and Brewster, Stephen, A. s.l. : Proceedings of the 2003 International Conference on Auditory Display, 2003. ICAD03-1. 28. Comparing the Effects of Visual-Auditory and Visual-Tactile Feedback on User Performance: a Meta-Analysis. Burke, Jennifer L., et al. s.l. : Proceedings of the 8th international conference on Multimodal interfaces, 2006. doi>10.1145/1180995.1181017. 29. Fundamentals of Microelectronics. Razavi, Behzad. s.l. : Wiley, 2008. 30. Human-Computer Interaction. Dix, Alan, et al. s.l. : Pearson Education, 2004. 31. Iterative and Incremental Development: A Brief History. Larman, Craig and Basili, Victor. s.l. : Computer (IEEE Computer Society), 2003. 32.Inclusion-Exclusion Principle. Weisstein, Eric W. MathWorld. [Online] http://mathworld.wolfram.com/InclusionExclusionPrinciple.html. 33. Photoshop for the Web. Aaland, Mikkel. s.l. : O'Reilly Media, 1999. 34. Met Office. [Online] [Cited: 12 April 2012.] http://www.metoffice.gov.uk/climate/uk/interesting/2011_spring/. 35. British Society of Hearing Aid Audiologists. [Online] [Cited: 14 2 2012.] http://www.bshaa.com. 36. Q-Q plots. Murdoch University School of Chemical and Mathematical Sciences. [Online] 2009. [Cited: 1 April 2012.] http://www.cms.murdoch.edu.au/areas/maths/statsnotes/samplestats/qqplot.html. 37. The Wilcoxon Signed-Rank Test. Lowry, Richard. Concepts & Applications of Inferential Statistics. [Online] [Cited: 2 April 2012.] http://vassarstats.net/textbook/ch12a.html. 38. Intonational Equivalence: an Experimental Evaluation of Pitch Scales. Nolan, F. s.l. : Proceedings of the 15th International Congress of Phonetic Sciences, 2003. 39. An Acoustics Primer. Hass, J. Center for Electronic and Computer Music. [Online] Indiana University, 2003. [Cited: 1 April 2012.] http://www.indiana.edu/~emusic/etext/acoustics/chapter1_pitch.shtml.


You're Reading a Free Preview

/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->