See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/320282066

Co-creating a Gamified Solution for Music Learning

Conference Paper · October 2017

CITATIONS READS

2 943

3 authors:

Margoudi Maria George Waddell

HighSkillz Ltd Royal College of Music
21 PUBLICATIONS 122 CITATIONS 49 PUBLICATIONS 189 CITATIONS

SEE PROFILE SEE PROFILE

Manuel Fradinho Duarte de Oliveira

KIT-AR
145 PUBLICATIONS 1,133 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Target View project

ECOSPACE View project

All content following this page was uploaded by Margoudi Maria on 09 October 2017.

The user has requested enhancement of the downloaded file.

Co-creating a Gamified Solution for Music Learning
1 2 1
Maria Margoudi , George Waddell , Manuel Oliveira
1
HighSkillz, London, UK
2
Centre for Performance Science, Royal College of Music, London, UK
maria.margoudi@highskillz.com
george.waddell@rcm.ac.uk
manuel.oliveira@highskillz.com

Abstract: The practices and techniques of learning and teaching a musical instrument are highly tradition-
driven. Therefore, the use of technology – hardware or software – is often strictly limited to certain instances
of students’ or teachers’ practices. Despite technological progress and its application across numerous
educational fields, instrumental learning has not yet harvested the full potential of these advancements. To fill
this gap, the TELMI (Technology Enhanced Learning of Musical Instrument Performance) project aims to
develop a complete technology-enhanced solution for violin learning and teaching. A distinctive feature of the
TELMI approach is the application of Co-creation techniques to the design of the learning solution. Co-creation
is “an act of collective creativity, shared by two or more people”. Building on Living Lab and Co-design
principles, the customized TELMI Co-creation approach involves end-users and domain experts – violin
students and teachers – in the design process, allowing them to create their own solution. In this paper, we’ll
elaborate on the collaborative Co-creation methods that have been applied thus far in the project, including
storyboards, personas, timelines, scenarios, and problem definition. In particular, guidelines and lessons from
a series of interviews and workshops carried out with violin students and teachers will be introduced. The
main aim of these sessions was the in-depth exploration of the use of technology in instrumental learning, as
well as the acceptance of specific gamification techniques.

Keywords: Co-creation, violin, music education, gamification, deliberate practice

1. Introduction
TELMI (Technology Enhanced Learning of Musical Instrument Performance) is examining how people learn
musical instruments from a pedagogical and scientific perspective. Using the violin as a case study, the project
will design and develop an interactive, assistive system to complement traditional teaching and learning
methods. In addition to state-of-the-art systems of audio, video, and motion-capture equipment, a key feature
of the project is the incorporation of these technologies in a gamified platform that enhances self-directed
learning and encourages communication with peers and teachers.

In the first phase of this project, the partners constructed a pedagogical framework defining the contexts in
which musical instruments are learned, the challenges musicians face, and their current use of technology and
games in music learning (see Margoudi et al, 2016 for a full discussion). Key to this framework was the
adaption of principles of self-regulated learning, where students set concrete, achievable goals and use
deliberate strategies to achieve them (Clark et al, 2016), with the recognition that, while numerous
technology- and gamification-enhanced solutions have been developed, they have seen limited uptake. This
resulted in six recommendations for the incorporation of games-based principles into musical learning:

§ Provide feedback systems within games that can reduce complex physical and acoustic information into
simplified outputs; provide cross-modal interfaces that can e.g. visualize complex aural information,
employing metaphor and intuitive concepts.
§ Use games to bring the collaborative nature of music learning to one’s own practicing; apply social design
to encourage engagement.
§ Limit use of competition as a game design pattern to avoid aggravating the negative impact of competitive
driven behaviour within musical practice.
§ Incorporate a wide variety of skills, beyond technical and artistic execution (e.g. planning, stage skills,
memorization, etc.) when defining success in music learning.
§ Design solutions that either accommodate the current practices and tools of musicians or make more
efficient use of their time and resources through better self-regulated learning.
§ Improve communication between student and teacher through systems that can monitor, analyze, and
provide feedback on students’ practice, guided by the expertise of the teacher.
These principles formed the basis of the first stage of Co-creating the TELMI solution to the challenges of
learning a musical instrument. To achieve this, TELMI adopted a structured, literature-driven, and customized
Co-creation process, tailored to its particular needs and characteristics (Best, 2006). A review of processes
based on Social Design (Kimbell et al, 2013), User-Centered Design (Doorley et al, 2012), Service Design (NC
State University, 2016), Innovation Design (Brown, 2008), and Transformation Design (Sauder d.studio, n.d.)
was carried out. The final Co-creation approach consisted in the combination of two pre-existing processes:
the “Double Diamond” design process (The Design Council, 2007) and the more education-oriented “Design
Thinking for Educators” process (IDEO LLC, 2012).

In the following sections, we will elaborate on the Co-creation activities that have already been realised in the
TELMI project, towards the definition of the final solution. For the practical implementation of the TELMI Co-
creation Process, a set of 27 different co-design methods and techniques was compiled. Following the
structure of the four main phases of the TELMI Process, described above, we’ll elaborate on several of the
exploited techniques including interviews with violin students and teachers, storyboards, and mockups.

2. The TELMI Co-creation Process

Fostered under the broader field of Social Design and Design Thinking, Co-creation is applied in a variety of
fields. Sanders (2008) described it very generally as “an act of collective creativity, shared by two or more
people”. The origin of these approaches can be traced back to the marketing and business fields as a highly
facilitated team-based process, heavily dependent on the expertise and the domain knowledge of its
practitioners. The participants in Co-creation approaches are experts coming from different fields, all working
together in order to design, realize, and evaluate an innovation that will address a concrete need. Compared to
other, more traditional user-centred design methods, this results in greater impact on the traditional roles of
process participants. These roles are now shifted and the end-user becomes the expert, simultaneously
participating in and profiting from the process. This new, active role of the participants/end-users applies to all
the aspects of the process from knowledge development and idea generation to concept development
(Sanders, 2008).

The TELMI Co-creation Process consists of four central phases, contextualised by transitional stages at the
beginning and end:
§ Preparation: This is an introductory phase, where the ideas are formed and remain largely unstructured.
The hypothesis or problem is identified and phrased.
§ Discover: This is the preparatory stage, after the identification of the problem. The main aim is to form an
understanding of the challenge, identify the people involved, and document the needs that emerge.
§ Define: This phase comprises the interpretation of the ideas collected in the previous phase, as well as
ideation - the formation and refinement of the final solution.
§ Develop: The fourth phase focuses on bringing the final solution to life. This is an iterative process that
consists of three steps: (a) prototyping, (b) testing, and (c) refining.
§ Deliver: In this phase, the product is finalized through testing and evaluation and is finally launched into
the market.
§ Evolution: The final phase of the TELMI Co-creation Process is a transitional phase, aimed at the
development of the project over time and identifying potential for related, follow-up, or spin-off work.

Figure 1 maps the current progress of the TELMI project onto this TELMI Co-creation process. This paper will
focus on the first two phases (“Discover” and “Define”), where the collaboration among the partners and
subject matter experts (SMEs) was augmented by interviews and workshops with violin students and teachers
to first discover the challenges faced and then define potential solutions. Recommendations on how to
facilitate and enhance this process are then discussed.
 Figure 1: The TELMI Co-creation Process

3. The Discover Phase: Mapping the Area

In the introductory phase of the project, we identified the broad challenges faced by those striving to learn a
musical instrument (see above and Margoudi et al, 2016 for a full discussion). In the Discover Phase, we
examined how these challenges related to their past and current use of technology in their musical practice
and learning, and their reactions to hypothetical systems that could address the general challenges they face.

To facilitate this procedure, we developed a set of storyboards in collaboration with the project partners and
subject matter experts at the Royal College of Music (RCM) (see Figure 2). The use of storyboards is adapted
from the cinematographic tradition. It utilizes a series of drawings or pictures put together in a narrative
sequence, highlighting key touchpoints between the user and the system. The creation of these Storyboards
incorporated several other Co-creation methods:
§ Problem Definition: This is the first step towards solving any problem. End-users are involved to ensure
that a genuine problem is targeted. In the case of TELMI project, in collaboration with RCM experts, the
main problems in music learning to be addressed were drafted.
§ Personas: These are archetypes built after observation of the potential users. A persona is based on a
fictional character, whose profile gathers up the features of an existing social group. In TELMI, three
distinct personas were identified – teacher, conservatoire student, and mainstream student.
§ Scenarios: The aim of this method is to elicit different scenarios of use from the actual stakeholders. It is a
tool for thinking about, sharing, discussing, and analysing design challenges. It describes a concrete and
real challenge, and proposes a possible way of addressing it. In TELMI, three scenarios were described,
and relevant storyboards were generated for each of them: a student practicing on their own, a student
with a teacher in a lesson, and a teacher-led online group studio.
§ Timeline: This method provides a linear representation of a time period where important events are
marked. The TELMI storyboards used a week-in-the-lift approach, depicting the different ways a student
and teacher might interact with the system across a single week.
Figure 2: Examples from the TELMI storyboards, depicting (top left) voice-activated interaction with an
intelligent practice guide, (top right) a student reviewing practice session progress, (bottom left) a teacher
and student reviewing practice distribution over a week, and (bottom right) a teacher video-recording a
student for review and discussion.

To validate and refine these storyboards and further define relationship between violinists and technology,
interviews were conducted with nine violinists in the United Kingdom and Canada. The cohort comprised five
female and three male violinists with a mean age of 28 years (SD = 4.5). All violinists had or were completing
undergraduate music training; seven had additional postgraduate training, and two performed with
professional orchestras. Six were active as violin teachers with students from the beginner to intermediate
level. Interviews were conducted individually (with one exception where two were interviewed tougher) in
person and via Skype, were audio recorded, and lasted 1-2 hours. The research was conducted following the
ethical guidelines of the British Psychological Society, and informed consent was obtained from all participants.
The interviews followed a semi-structured design, utilizing a pre-determined set of open, discussion-prompting
questions allowing the interviewer to further explore particular themes or responses (Robson et al, 2016). It
comprised three general lines of questioning:
§ Where have you/do you currently use technology in learning and teaching the violin?
§ What new technologies or technological innovations would you like to see developed?
§ What are your reactions to these hypothetical technologies (i.e. storyboards, video examples of violin
motion-tracking feedback)?

After giving the violinists the opportunity to answer freely, the interviewer prompted where necessary to
determine whether each engaged with standard uses of technology (e.g. metronomes, tuners, audio- and
video-recording, digital planners) and any reasons why they did or did not use particular technologies. While
an in-depth discussion of the reactions to the first two sections will be the subject of a separate manuscript, in
general the musicians demonstrated a relatively stable use of the standard technologies, with limited use of
video recording overall or returning to review past audio or video recordings. Their concerns with new
technology fell into four themes:
§ Time: Over all else, musicians emphasized the need to maximize use of limited time. Technology was
reflected upon both positively and negatively in facilitating this; students saw the potential of technology
to help organize and maximize use of time in the practice room. However, they showed concern at the
amount of time it could take to learn to use, operate, and troubleshoot the technology.
§ Equipment: The musicians expressed concern at whether their existing devices (e.g. smartphones, tablets,
laptops, etc.) would be capable of operating the systems as presented. While most were open to investing
in equipment purchase, cost was identified as a concern.
§ Customization: While musicians commended the number of functionalities proposed in the storyboards,
they emphasized the need to be able to customize which aspects would be used and in what way. In
particular, the ability to determine the degree to which the system operated as a passive collector and
organizer of practice data versus an active tool, offering guidance, judgement, and recommendations.
They also highlighted the need to have direct control over who had access to their data in cases where it
may be shared with teachers, peers, the public, etc.
§ Functionality/reliability: The musicians identified technology functionality and reliability as a concern,
citing experiences of technology failure and frustration (e.g. the unreliability of voice recognition software)
or issues where low audio or video quality prevented useful feedback.

4. The Define Phase: Interpretation & Identification

Following the Discover Phase, the Define Phase aimed to narrow down the scope of the storyboards into a
concrete definition of the challenges to be addressed by the TELMI system and how they might be addressed,
taking into account user-feedback from the previous stage.

The Co-creation method of prototyping was used for this phase. Prototypes may take many different forms
according to the aim of the activity and the available resources. They can be tested in a workshop context or in
the users’ own living, working, or social context. For our purposes, we chose to create an interactive
prototype. This has been an iterative process, involving the testing of different technologies and the
employment of a multidisciplinary team of developers, artists, subject matter experts, and user experience
designers. The main aim of the prototypes was to represent selected basic functions of the storyboards based
on their technical feasibility and user feedback, and drive the development process towards the final TELMI
product. Consequently, the mockups were focused on the following four elements (see Figure 3):
§ Planning: The planning aspect was greatly emphasised by all the interviewees and forms a central role in
the pedagogical underpinning, driven by theories of self-regulated learning. In the interactive mockup we
incorporated the notion of short, medium, and long-term planning (see Figure 3, part A).
§ Wizards: The Wizards function acts as an extension of the planning aspect. Suggested by subject matter
experts, Wizards allow the learner to choose among pre-defined sets of activities that employ specific
strategies to work towards a particular goal (e.g. mastering a specific vibrato skill). This way the planning
process becomes simpler and quicker for the user, and they learn to augment their goal-setting with the
strategies they can use to achieve them (see Figure 3, part B).
§ PlayTools: The PlayTools aspect of the prototype guides the user through an individual planned practice
session, providing a visualization of time planned versus time used. At the core of this feature is the
presentation of performance feedback driven by aural, visual, and motion-capture analysis provided by
related initiatives in the TELMI project. These feedback tools take the form of widgets, which drive user
review of and reflection on practice and allow for more efficient self-diagnosis of performance issues.
They also expanding the range of tools a teacher may use to demonstrate and model success both in and
beyond the lesson. As the widgets remain under development, they were not visualized in the mockups
but were the subject of a separate discussion with students in the workshop (see Figure 3, part C and
section 5: Experimentation with Students).
§ Studio: Finally, the Studio mode represents the social component of the TELMI solution. Following on the
pedagogical model of group masterclasses, this mode enables students to upload performance recordings
to a communal virtual space for peer- and teacher-review and commentary (see Figure 3, part D).
Figure 3: Examples from the TELMI Interactive Mockups: (A) Planning, (B) Wizards, (C) PlayTools, (D) Studio.

5. Experimentation with Students

To evaluate the success of the mockups, a workshop was organized with Royal College of Music students (N=6;
five violinists and one cellist). The cohort consisted of three undergraduate and three postgraduate students
(two male, four female) with a mean age of 23.7 (SD = 1.4) years and a mean 18.2 (SD = 1.4) years of musical
experience. Five of the six taught beginner to intermediate students. The workshop comprised two sections:
(1) responses to the mockups and (2) a brainstorming co-design session on visual feedback procedures (i.e. the
widgets within the PlayTools) for intonation (tuning), dynamics (volume), articulation, rhythm/tempo, tone
quality, movements of the bow, and movements of the body. Students were given paper (both blank and with
sample musical material) and coloured pencils to aid in demonstrating visualisations while a researcher
facilitated with video and still image examples of existing feedback systems and a whiteboard.

In the first phase, responses to the mockups were positive, with further emphasis placed on the need for
customisation elements to fit the learning styles of individual students, as was reflected in the interviews. This
concerned scheduling, particularly the need to balance flexibility in short-term planning to adapt to quickly
changing schedules while maintaining progress towards long-term goals. A disconnect between relatively non-
specific goals (e.g. learn a piece of repertoire; improve a technique) with specific short-term strategies to
achieve them was identified, highlighting the need for the system to not only capture progress towards goal
achievement but also to encourage, suggest, and monitor series of specific rehearsal strategies to achieve
them. Verbal, written, and drawn feedback (see Figure 4) centred on the following three general principles,
which expand upon the first gamification recommendation discussed above (i.e. provide feedback systems
within games that can reduce complex physical and acoustic information into simplified outputs):

1. Visualizations should be clean, colourful and use few (or no) words or numbers. It should be easy to
understand their context at first sight.
2. Elements should be presented separately by default. Aural feedback should demonstrate a single
concept, and visual elements should engage a single visual perspective (e.g. frontal versus lateral) at a
time to avoid confusion. Users can then choose to activate multiple widgets.
3. Wherever possible, bow movement should be shown as a visual analogue, relative to a depiction of
the violin that is fixed in space (i.e. isolated from the movement of a performer).
Figure 4: Participant co-created drawings of violin performance feedback, indicating (A) dynamics, (B)
incorrect rhythms (with horizontal position of marks indicating notes performed before the correct time),
and (C) articulation (i.e. breaks in a continuous sound).

6. Discussion
The adopted Co-creation process, as intended, has led to a solution that is shaped by the needs and
expectations of the target end-users. The driving principles for designing Digital Games-Based Learning
solutions that engage and support the learning of music instrumentation, as outlined in previous work
(Margoudi et al, 2016), are addressed in the final design of the TELMI application being developed. However,
upon reflection on the Co-creation process, imbued with design thinking principles, several new guidelines can
be considered:

§ The very nature of a multidisciplinary team entails the involvement of stakeholders from different
disciplines with different expertise and experience, working towards creating new enriched solutions
through the intersection of the different perspectives. To engage a multidisciplinary team over a period of
time requires frequent and continuous communication amongst the key participants to refine the
understanding of the user needs and common shaping of the foundational concepts. The involvement of
pedagogical experts, subject matter experts, and end-users is fundamental to understand the behaviours
that should be encouraged and potentially accepted by the end-users.
§ Recommendation 1: Game designers should mediate between the different stakeholders,
communicating on a continuous basis with frequent interactions.

§ In the Co-creation process, namely for the Discovery Phase, various methods and tools are used with
strong visual emphasis (e.g. scenarios, timeline, and storyboards) that are beneficial for facilitating the
discussion amongst the different stakeholders. Although the visual artefacts are relevant for exploration,
one needs to gauge carefully how to transition to the definition phase and towards ultimately proceeding
with development. Otherwise, there is a risk in increasing the specification gap between initial design and
implementation, blocking progress. Regarding gamification, the broad concepts captured in the visual
artefacts may become less relevant when addressing the specific details and feasibility of monitoring
behaviours and implementing the corresponding game mechanics.
§ Recommendation 2: Attribute a short expiration date to the visual artefacts and adopt the driving
principle of discarding the stories and storyboards soon after validation with the end-users.

§ The Co-creation process results in visual artefacts supported by some documentation, but a significant
amount of tacit knowledge is generated that is not externalised as the effort required makes it impractical
considering the fast pace of change. Concepts are shaped by the interplay amongst all the stakeholders;
consequently, it is not possible for the co-creation process to be facilitated just with the artefacts created,
irrespective of the experience of the facilitator.
§ Recommendation 3: Ensure that the co-creation process is facilitated by a consistent team of
individuals to ensure continuity of the tacit knowledge generated throughout all co-creation activities.

§ When assessing the potential of gamification despite the overpowering hype, Burke (2012) communicated
that 80% of the gamified solutions would fail to achieve their business objectives due to poor design.
Although gamification has braved the through of disillusionment, it remains a crafted experience with high
 probability of failure due to poor design. In highly competitive environments, such as within the Royal
College of Music, poorly gamified designs are easily derided – “it is just a game”.
§ Recommendation 4: The first step towards gamification is to understand the target end-users, the
desired behaviours to influence and the key barriers towards change. Do not apply game mechanics
without purpose, and simplicity is often more effective.

7. Acknowledgements
This research was conducted as part of the TELMI (Technology Enhanced Learning of Musical Instrument
Performance) project, which has received funding from the European Union’s Horizon 2020 research and
innovation programme under grant agreement No 688269.

References

Best, K. (2006). Design management: managing design strategy, process and implementation. AVA publishing.
Brown, T. (2008). Design thinking. Harvard business review, 86(6), 84.
Burke, B. (2012). Gamification 2020: What is the future of gamification. Gartner, Inc., Nov, 5.
Clark T., Lisboa T. and Williamon, A. (2014), Learning to be an instrumental musician, in I Papageorgi & G Welch
(eds), Advanced Musical Performance: Investigations in Higher Education Learning, Ashgate, pp 287-300.
Doorley, S. and Witthoft, S. (2012). Make space: How to set the stage for creative collaboration. Hasso Plattner
Institute of Design at Stanford University.
Kimbell, L., and Julier, J. (2012). The social design methods menu. perpetual beta.
Margoudi, M., Oliveira M. and Waddell, G. (2016). Game-Based Learning of Musical Instruments: A Review and
Recommendations. In 10th European Conference on Games Based Learning: ECGBL 2016, p. 426.
NC State University (2016). “Service Design Process”, [online], Learning Space Toolkit: A Resource for Designing
and Sustaining Technology-Rich Informal Learning Spaces, http://learningspacetoolkit.org/services-and-
support/service-design-process/
Robson, C. and McCartan, K. (2016). Real world research. John Wiley & Sons.
Sauder d.studio. (n.d.). “Design Processes”, [online], The University of British Columbia,
http://dstudio.ubc.ca/toolkit/processes/
The Design Council (2007). “Eleven lessons: managing design in eleven global companies. Desk research
report.” [online], 2nd edition, © 2012 IDEO LLC. All rights reserved. http:// designthinkingforeducators.com/

View publication stats