Music SOFA
An architecture for semantically informed recomposition of Digital Music Objects
David De Roure Graham Klyne
University of Oxford University of Oxford
Oxford, UK Oxford, UK
david.deroure@oerc.ox.ac.uk graham.klyne@oerc.ox.ac.uk
David M. Weigl
University of Oxford
Oxford, UK
david.weigl@oerc.ox.ac.uk
ABSTRACT
We describe the design and implementation of a semantic music
system which illustrates the assembly of a music composition using
semantically annotated music fragments. The system, which we
call SOFA (SOFA Ontological Fragment Assembler), demonstrates
architectural design principles which may have more general appli-
cability in the semantic music domain, notably the adoption of the
Linked Data Platform to realise service components of the end-to-
end pipeline so that data specialisation takes the place of service
specialisation. The prototype builds upon two existing tools devel-
oped by the authors: Music Encoding and Linked Data (MELD),
which augments and extends MEI structures with semantic Web
Annotations capable of addressing musically meaningful score sec-
tions, and Numbers Into Notes, an algorithmic composition tool
that acts as a ‘semantic signal generator’ to drive the tool chain.
The system demonstrates the concept of Digital Musical Objects
(DMOs), and in particular DMO processing and recomposition.
CCS CONCEPTS
• Information systems → Semantic web description languages;
• Applied computing → Sound and music computing;
KEYWORDS
Semantic Music, Digital Music Object, Linked Data Platform
ACM Reference Format:
David De Roure, Graham Klyne, John Pybus, David M. Weigl, and Kevin Page.
2018. Music SOFA: An architecture for semantically informed recomposition
of Digital Music Objects. In Proceedings of ISWC Workshop on Semantic
Applications for Audio and Music (SAAM) (SAAM’2018). ACM, New York,
NY, USA, Article 4, 9 pages. https://doi.org/10.475/123_4
1 BACKGROUND
Our research explores the idea that music can be described as an
assembly of musical fragments, and uses semantic annotation of
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
SAAM’2018, October 2018, Monterey, California USA
© 2018 Copyright held by the owner/author(s).
ACM ISBN 123-4567-24-567/08/06.
DOI: 10.1145/3243907.3243912
John Pybus
University of Oxford
Oxford, UK
john.pybus@oerc.ox.ac.uk
Kevin Page
University of Oxford
Oxford, UK
kevin.page@oerc.ox.ac.uk
fragments to help guide the process of assembly into a composition.
This notion of Semantic Music differs from semantically described
metadata pertaining to a musical work, or semantic representations
at note or chord level. We are particularly interested in developing
this approach as a basis for Digital Music Objects (DMOs), i.e. rich
representations of music works which are semantically described
and may carry information about process and provenance. In this
section we describe the background to our work.
1.1 Historical context and Numbers Into Notes
The research came about as a result of a historical thought experi-
ment. In 2015 we were involved in organising an event to celebrate
the 200th Anniversary of the birth of Ada Lovelace, who wrote of
Charles Babbage’s proposed Analytical Engine: [15]
Supposing, for instance, that the fundamental rela-
tions of pitched sounds in the science of harmony and
of musical composition were susceptible of such ex-
pression and adaptations, the engine might compose
elaborate and scientific pieces of music of any degree
of complexity or extent.
Using a simulator for Charles Babbage’s Analytical Engine, and
with input from mathematicians and musicologists, we generated
fragments of music algorithmically. We then made a Web version
of our software, called Numbers into Notes1 , which exports the frag-
ments in multiple representations, including MIDI, W3C PROV,
and a musical score. [22]. As well as a demonstration at the event,
generated fragments have subsequently been used to create com-
positions, for example a performance at Audio Mostly in London
in 2017.
The success of this experiment in combining semantically de-
scribed fragments to create a composition has led to the work we
describe here, which we believe to be of general utility. In particular,
the architecture we describe demonstrates some design principles
which should be applicable to a broader body of work in the seman-
tic music domain.
1.2 Contextual Semantics for Music
Our approach draws on music metadata and Linked Data patterns,
which we incorporate and extend to inform the design of our se-
mantic music system.
1 http://numbersintonotes.net
SAAM’2018, October 2018, Monterey, California USA
Broadly we draw on three approaches to music metadata, as
follows:
Content-derived feature data. Feature extractors operating over
music content generate mathematical approximations of human
music perception: basic percepts (pitch, note onsets); models of
structure and form (e.g. key detection; meter estimation; structural
segmentation); higher-level percepts such as instrument classifica-
tion. By augmenting content with derived feature data, music infor-
mation can be structured to provide for automated analysis, discov-
ery, and retrieval at scale. To overcome the semantic gap between
human experience and mathematical approximation, provenance
information regarding the precise mechanism of derivation—plugin
names, versions, and parameterisations–must be captured and asso-
ciated with the feature data. An exemplar Linked Data publication
describing a large community-contributed collection of live music
recordings [1] adopts a ‘layered’ approach to digital library meta-
data [13] that augments the audio recordings and semi-structured,
user-provided descriptions of the source archive with additional
layers of structured metadata describing content, computational
analysis, and associated provenance information [18]. This provides
for automated exploration, analysis, re-interpretation and re-use
without modification of the source archive.
Bibliographic context. From library catalogues, to media broad-
cast records, to musicological and music industry databases, web-
accessable music datasets provide access to varied descriptions of
musical works, composers, performers, performances, and myr-
iad other kinds of related contextual information. In the absence
of shared identifiers, these datasets remain isolated, restricting
combined access. Ambiguities, such as names shared between indi-
viduals, hinder automated reconciliation; while manual alignment
requires exponential effort with growing datasets. Computational
assistance, generating match proposals while deferring to human
insight for confirmation, can address these challenges, providing
unified access to complementary datasets, while supporting schol-
arly communication through match provenance (e.g. in cases of
contested attribution). Matches can be proposed according to string
similarity [2], but more ambiguous matching requires rich semantic
context2 [24].
Music encoding. Notated music precisely describes compositional
intent independently of any particular performance. Annotations
upon written music have long supported edition, analysis, rehearsal,
and performance [26]. The Music Encoding Initiative’s XML schema
[5] (MEI) offers such affordances in digital form through a com-
prehensive machine-readable representation of musical structure.
Elements at any level of abstraction, from a complete work to an ac-
centuation mark on a particular note, are identifiable with standard
XML element IDs, which become web-addressible when employed
as fragment identifiers at the end of MEI file URIs. In turn, the re-
sulting element URIs can be incorporated as Linked Data providing
for annotation and other semantic associations while bypassing the
need for formal ontological (RDF) descriptions of musical structure.
2 Where a composer is associated with a work that is included within a book of
music which is in turn attributed to an author, this provides contextual cues that the
composer and the author may be the same individual, particularly when they share
other contextual similarities, e.g. in birth or death date, or place of birth.
D. De Roure et al.
1.3 Music Encoding and Linked Data (MELD)
Information systems typically describe music and other timed mul-
timedia information resources using timeline-based anchors ex-
pressed as millisecond offsets from the start of an audio recording,
or as MIDI clock ticks (e.g. [10, 20]). While such temporal offsets are
widely used to link to positions within video or audio recordings,
their usefulness is limited when targetting music in conceptual
terms, as when annotating musical score.
By drawing on machine readable, addressable music encodings
such as those offered by the MEI schema, Music Encoding and
Linked Data (MELD)3 offers an alternative approach, anchoring
semantic Web Annotations [4] within a meaningful representation
of musical structure (Fig. 1). The MELD semantic framework [25]
combines and augments pertinent subsets of established ontologies,
including: the FRBR-aligned Music Ontology [21] to discriminate
compositional entities at different levels of abstraction (e.g. a mu-
sical work at conceptual level, its expression as musical score, and
its manifestation as a particular music encoding); and the Segment
Ontology [9] to describe musical segments comprising a segment
line, bridged to ontological structures expressing elements of mu-
sical form appropriate to specific subdomains (e.g. verse, chorus;
sonata, fugue). Collections of MEI fragments, manifestations of
musical segments embodied within a published score (MEI file),
anchor annotations associating and interlinking musical content
and extra-musical information resources with elements of musical
structure.
Each MELD Web Annotation is associated with a motivation
formalising the annotation’s intended purpose. In a given MELD
application, web clients interpret annotations according to these
motivations, effecting corresponding actions through domain- and
use-case-specific rendering and interaction handlers. A library of
ReactJS web client components4 is available, simplifying the cre-
ation of MELD applications for varied contexts and use-cases. Such
apps have been developed to drive musical performances [11], sup-
port collaboration between musicians in jam session scenarios [25],
and support contextual interpretation of digital music notation for
musicological scholarship [17]. Here, we apply MELD in the speci-
fication of DMOs representing the musical fragments generated by
Numbers Into Notes, progressively augmented with additional meta-
data through an iterative annotation process. The resulting set of
music flows and data structures supports the creative (re-)assembly
of such fragments into new compositions according to semantic
notions of musical compatibility.
1.4 Separation of Concerns using Linked Data
A goal of our design is to separate the semantic analysis and an-
notation of music fragment compatibility from the use of this in-
formation in fragment assembly, in order to facilitate the later
adoption of new forms of semantic annotation. Linked data and
web standards underpin the information flows between analytical
and consumptive (assembly-related) elements: LDP [23] provides
a uniform interface (per REST architectural style [8]) for exchang-
ing information via web-accessible resources, Web Annotations
3 http://github.com/oerc-music/meld
4 http://github.com/oerc-music/meld-clients-core
Music SOFA
Figure 1: Overview of the Music Encoding and Linked Data
(MELD) framework. Traditional multimedia alignments em-
ploying timeline anchors (red arrows) are augmented with
semantic relationships anchored in musical structure (green
arrows).
[4] are used to associate musical resources with semantic informa-
tion, MEI [5] is used to represent fragments of music, and MELD
[25] provides a framework for referencing music fragments, their
arrangements, and associated semantic annotations.
Using Web Annotations, MELD incorporates specialised seman-
tic information about fragments in a common access framework.
By materialising semantic information in LDP containers, we are
able to use off-the-shelf LDP software for the service elements (we
discuss later (section 3.3) how specialised LDP service software
might enable on-the-fly materialization of matches). We use sep-
arate LDP containers to capture different types of inter-fragment
compatibility (e.g. key, pitch range, length of fragment). The mem-
bers of each container are Web Annotations, arranged to enable
the discovery and enumeration of compatible fragments using a
follow-your-nose pattern [7]5 .
Thus we represent compatibility semantics through specialised
representations, while using existing LDP and related Web stan-
dards for access and “query”. This approach of adding specialised
representations to existing standards has been put to good use in
other projects: Memento [6] uses a small number of specialised
HTTP header fields; ORE [16]defines vocabulary for representing
exchange of composite objects using HTTP; Web Annotations [4]
defines an extensible vocabulary for use with LDP containers.
2 SEMANTIC MUSIC FLOWS
Having related the contextual landscape available to semantic music
applications, we now consider how music fragments can be cre-
ated and annotated by leveraging these approaches to contextual
metadata, and how ‘music flows’ of iterative semantic enrichment
can create contextual webs of information about and around the
fragments, providing for their re-interpretation and consumption.
In section 3, we will detail the implementation of an ontological
fragment assembler that practically applies these ideas within a
semantic architecture accessible via a web interface. Finally, we
provide reflections and outline avenues for future work in section 4.
5 http://patterns.dataincubator.org/book/follow-your-nose.html
SAAM’2018, October 2018, Monterey, California USA
2.1 Semantic Signal Generator (SSG)
For the purposes of our research we need a supply of music frag-
ments, and for this we turn to our other semantic music tooling.
The Numbers into Notes software can be used to generate fragments
in multiple formats, and provides a rich supply of source content
which is parameterised by the settings of the application.
This is a novel deployment of the software as part of our tool
chain, and we describe it as a Semantic Signal Generator (SSG),
by analogy with the signal or function generator that an audio
engineer may use in testing audio signal processing. From a creative
viewpoint it also provides a particular approach to algorithmic
composition, but we can also work with other sources of music,
whether audio or symbolically encoded.
The original implementation of Numbers into Notes supported
MIDI and Lilypond (music engraving) export formats, plus W3C
PROV to provide a record of how a fragment was created (and
could be recreated). We extended this to include MEI in order to
drive MELD. It is straightforward to add further formats, and we
anticipate future versions supporting MusicXML and interoperating
with other semantic music tooling and ontologies.
2.2 The SOFA Ontological Fragment Assembler
(SOFA)
The SOFA Ontological Fragment Assembler (SOFA) is an appli-
cation which takes a set of musical fragments produced by the
Semantic Signal Generator, along with annotations informed by
the semantic context, and provides a remixer interface offering a
guided environment for the user to experiment with assembling
the fragments to create a new musical composition.
The user starts by pointing the application to a URI representing
a set of fragments exported from the Semantic Signal Generator; this
will form the group of fragments available for use in the assembly
interface. The remixer allows for multiple parts to be specified, and
the user configures the number of parts, associating with each an
instrument chosen from the instrument taxonomy described in 3.4.
The main interface takes the form of a grid where the user can
place fragments to assemble the piece. Each fragment is represented
by an engraved score, and the ‘cursor’ where the current selection
is being made is highlighted. The selection being a modal ‘pop-up’
which shows a list of available fragments rendered as engraved
score. Each fragment having a play button to allow the user to hear
a midi representation.
To assist the user, fragments available for selection are filtered
according to annotations which identify various forms of compat-
ibility. The horizontal rows, associated with a particular musical
instrument, are constrained to be within the playing range of that
instrument, so the interface filters out any fragments which contain
notes outside the range as represented in the instrument taxon-
omy. In order that the fragments line up in the output piece, the
system constrains all of those in the same column to be of the same
length. After one fragment has been selected in a column, the selec-
tions for the other spaces are filtered to only show those fragments
annotated as being of the same length.
In addition to the annotations informing the filtering by compat-
ible instrument and matching length, there may be other types of
match annotation, identifying compatible fragments according to
SAAM’2018, October 2018, Monterey, California USA
other schemes. One such we have implemented is compatible key
where fragments composed in the same key, or in a neighbouring
key on the circle of fifths, are matched. The interface has no prior
knowledge of these types of annotation, but adapts to the annota-
tion data it receives and offers a checkbox for the user to optionally
filter by this criteria. The options offered to the user depend on
which semantic annotations have been added to the fragments.
Figure 2: The SOFA remixer interface showing the fragment
assembly grid and the fragment selection list
2.3 Music Flows using Digital Music Objects
(DMOs)
The previous two subsections have introduced first the Semantic
Signal Generator as a source of music fragments, and secondly the
SOFA remixer as a user application taking advantage of Linked
Data to creatively assemble fragments. These two components de-
fine the start and end of a music flow; at mid-points in the flow
each fragment is associated with additional semantic context, re-
alised through annotations, which encode the properties by which
remixing in the SOFA can proceed.
For the SOFA scenario introduced above, annotations must be
added within a music flow to indicate a fragment’s key, length, and
the instruments for which the fragment is within performable range.
We can imagine further annotations marking the scale mapping,
start and end notes, time signatures, and provenance information
such the user who created the fragment, and the algorithmic map-
ping used in the Semantic Signal Generator.
Annotation agents analyse the fragments within these specific
semantic contexts (e.g. key, instrumentation) and indicate compati-
bility through addition of annotations. At a later stage in the flow,
matching agents use the semantic annotations to indicate groups
of compatible fragments which are candidates for remixing (e.g. of
D. De Roure et al.
the same or related key, playable on the same instrument). This
generic music flow is represented in the top half of Figure 3.
SOFA can only provide for remixing according to semantic con-
texts in which annotation agents and matching agents have oper-
ated, as its remixing rules rely on their output; a greater number
of annotated fragments offers the composer using SOFA a corre-
sponding higher degree of creative flexibility. Conversely fragments
not thus augmented with metadata cannot be utilised in SOFA ac-
cording to our chosen scenario (section 3). Just as the Semantic
Signal Generator can usefully add to the pool of available music
fragments at any time, so to can annotation agents make additional
indications. We see these interactions within the music flow not as
a series of tightly coupled service calls, but as dynamic, iterative,
additions to the Web of Linked Data of which the fragments form
just one part.
On the other hand, there is clearly a pipeline of increasingly
enriched fragments moving from the SSG to the SOFA remixer—a
music flow of fragments, gathering semantic context according to
the annotation and matching agents. At the point an agent interacts
with one, we conceptualise a fragment and its associated scoped
semantic context (drawn from the Web of Linked Data) as a DMO,
with a series of related and derivative DMOs the artefacts of the
music flow (illustrated in the lower half of Figure 3).
We can thus consider our architecture at a high level through
two lenses: as a music flow with fragments moving from agent to
agent towards a particular goal; and as a Web of DMOs which are
individually crystallised into existence when an agent requires a
semantically scoped fragment upon which to operate.
MUSIC FLOWS
Semantic
Annotation Matching SOFA
Signal
agents agents remixer
Generator
WEB OF DMOs
Music Fragment Materialized Output
fragment
DMOs
compatibility
DMOs
match
DMOs
composition
DMO
Figure 3: Music Flows and Digital Music Objects (DMOs)
3 SOFA IMPLEMENTATION
In this section we describe the achitecture of the SOFA Ontological
Fragment Assembler and its implementation to provide a proof of
concept system.
3.1 Architectural Overview
Figure 4 shows the overall system of which the SOFA remixer is a
part, consisting of:
Semantic Signal Generator (SSG) - in our example, this is Num-
bers Into Notes, but could be any application that generates anno-
tated music fragments.
Annotated music fragments - musical fragments and accom-
panying metadata that can be recognised and analyzed by the
compatibility analysis agents. The Annotated music fragments are
grouped into Working sets for subsequent processing.
 Music SOFA
Compatibility analysis agents - these are non-interactive pro-
cesses that analyze the annotated music fragments in a working set
and generate fragment compatibility annotations, and save these for
subsequent access using the Match retrieval service. Many agents
may be applied to a working set, with each typically handling
a single analysis (e.g. key compatibility, pitch range constraints,
fragment length matching, etc.).
Fragment compatibility annotations - web annotations that
record pairwise compatibility between fragments, possibly with
additional metadata than can be used by the remixer software to
refine the matches presented for selection.
Match retrieval service - a web service providing a uniform
interface for accessing fragment compatibility annotations. The
basic design allows for finding all fragments in a working set that
are compatible according to a selected criterion with a specified
fragment. Variations on this allow for fragments that are compatible
with some other criterion; e.g. using a pitch range that is playable
by a given instrument.
Materialised matches - sets of web annotations that represent
a particular compatibility query (e.g. all fragments that are key-
compatible with a given fragment, or that are playable by a specified
instrument.
SOFA remixer - an interactive application that assembles a com-
position using these musical fragments from a working set, offering
guidance to the user regarding compatible fragments. The guidance
is based on semantic annotations provided via the match retrieval
service, and does not require the remixer itself to have inbuilt
knowledge of the compatibility criteria.
Output composition - an assembly of music fragments that can
be presented and performed as a completed work.
MUSIC FLOW
Semantic
Signal
Compatibility
analysis
retrieval
Generator agent
DMO LIFECYCLE
Annotated Fragment
music compatibility
fragment annotation
Figure 4: SOFA design overview
3.2 SSG and Fragment DMOs
The SSG generates and publishes music fragment DMOs compris-
ing web-accessible MEI encodings of the fragment composition (in
which each note and measure is made addressable by association
with XML identifiers); as well as semantic descriptions of the frag-
ment as mo:Score (Music Ontology) mo:published_as the MEI
file—corresponding to an expression–manifestation relationship in
FRBR terms. Light-weight provenance information regarding the
fragment’s creator and its title, as well as MELD references to its
first and last note, are also included within the initially generated
fragment DMO. Upon generation, a fragment reference pointing
at the newly created DMO may be added (HTTP POST) to an LDP
SAAM’2018, October 2018, Monterey, California USA
container representing a working set, thus becoming available to
subsequent stages of the music flow. This fragment reference ab-
straction is used to allow fragments to be incorporated within
multiple different working sets.
Upon inclusion within a working set, annotation agents employ-
ing feature extraction heuristics annotate the music fragment DMO
with semantic descriptors characterising aspects of the composi-
tion. In the current implementation, these derived features include:
length as a simple count of notes and rests, which are isochronous
in the musical universe of our SSG; key by minimising semi-tone
deviations between canonical key profiles and the musical content
of a composition; and pitch range according to the lowest and high-
est pitches within the composition, exposed both in scientific pitch
notation and as corresponding MIDI values. We implemented these
feature extraction heuristics as they straightforwardly inform the
fragment assembly task at the later stages of our music flow: length,
to synchronise the durations of fragments associated with different
voices; key and mode, to provide creative control of harmony and
discord within the assembly; and pitch range, to inform appropriate
instrumentation choices for each fragment6 .
The derived features are associated with Web Annotations tar-
getting (oa:hasTarget) the DMO of the fragment, motivated by
(oa:motivatedBy) feature type (e.g. “key”), with bodies (oa:hasBody)
encoding the feature extraction outcome (e.g. “D minor”).
3.3 Compatibility Service
The annotated fragment DMOs within the working set are analyzed
by compatibility analysis agents which generate further Web Anno-
tations establishing the compatibility of fragment pairs according
to a given match type criterion. Compatibility analysis is driven by
the semantic annotations associated with the fragment DMOs in
prior parts of the music flow; no further domain-specific (musically
informed) heuristics are applied at this stage, and matching is done
Match
SOFA according to a generic process through a unified HTTP interface
service
remixer
using LDP.
Figure 5 shows the compatibility service presenting as an LDP
container, which contains further LDP containers:
Materialized
matches
Output
composition
A Working set index, also referenced by the compatibility ser-
vice container using link sofa:working_sets, which contains Web
(by match type)
annotations that target working set containers and whose bodies
link to Working set service containers.
Working set service containers, one for each working set, con-
tain annotations that connect a Match type description with a Match
service.
Match service containers, one for each combination of working
set and match type identifier, containing annotations that target
selected a fragment, or some other entity, and whose bodies link to
a musical fragment that is compatible with the target according to
the criterion represented by the match service.
The targets of the contained annotations are one of the following:
A Working set is an LDP containter of references to musical
fragments over which compatibility information is provided, and
from which the resulting SOFA output will be composed.
6 Due to the open world assumption implicit in Linked Data, additional annotation
agents can be easily incorporated applying additional feature heuristics resulting in
the addition of further RDF triples associated with each annotated music fragment.
SAAM’2018, October 2018, Monterey, California USA
ldp:BasicContainer
Compatibility
service
ldp:BasicContainer
ldp:contains Working set
sofa:working_sets index
ldp:contains
ldp:contains
ldp:BasicContainer
Match service oa:hasBody
index
ldp:contains
ldp:contains
ldp:BasicContainer
oa:hasBody
Match service
ldp:contains
sofa:FragmentRef
Candidate
match
fragment
Figure 5: Compatibility service data model
A Match type description is essentially an identifier that de-
notes a particular kind of musical fragment compatibilitty (e.g.
compatible key, or same length, or containing a range of notes
playable by a specified instrument).
Selected fragment or other Compatibility target: when look-
ing for compatible music fragments, this references a given musi-
cal fragment, or some other target (e.g. an instrument for which
playable fragments are required, based on the range of notes used).
In general there will be multiple annotations for any given target,
each corresponding to a different compatible fragment.
Figure 6 shows how these nested LDP containers may be used to
find music fragments that meet some compatibility criteria, starting
with the URI (CS_URI) of the compatibility service:
GET CS_URI - retrieve a representation of the compatibility ser-
vice container as an RDF graph. Initial construction of the compat-
ibility service container includes a link to the working set index
that is extracted in the next step.
Find Working set index URI - look for a triple in the CS container
representation:
CS_URI sofa:working_sets WSI_URI
and extract the value found for WSI_URI.
The next steps all use a common function: Find [service] anno-
tations targeting [some resource]. A simple implementation in the
client would be to iterate through the service container contents,
returning all annotations whose target is the given resource. Other
possible implementations are discussed later.
Find WSI annotation targeting the required working set - the
resulting annotation refers to a Match Service Index (MSI) for the
designated working set.
Find MSI annotation targeting the required match type - the
resulting annotation refers to a Match Service (MS) for the desig-
nated match type and working set.
D. De Roure et al.
Initial deployment
creates these two
containers
oa:Annotation sofa:WorkingSet
Working set oa:hasTarget Working set
entry
oa:Annotation sofa:MatchType
Match type oa:hasTarget Match type
entry description
oa:Annotation sofa:FragmentRef
Fragment
oa:hasTarget Selected
match
fragment
annotation
oa:hasBody
Find MS annotations targeting the selected fragment - the
resulting annotations refer to fragments in the designated work-
ing set that are compatible with the selected fragment under the
designated match type.
What we show here is a common interaction pattern that ap-
plies to a range of possible compatibility criteria, whose details are
represented in the annotation data rather than being reflected in
the interaction interface. This uniform interface allows us to build
the compatibility service using an off-the-shelf LDP container, and
also allows additional compatibility criteria to be added without
changing the SOFA remixer application. This approach isolates de-
tailed knowledge of the compatibility criteria to the compatibility
analysis agents (Figure 4) that create the match service containers.
3.3.1 Nested match service containers. Consider the example of
looking for fragments that are compatible with the musical key of
another fragment. As described previously, this will require search-
ing through all possible pairs of compatible fragment annotations
to find those that could apply. We have extended the match service
design to allow nested service containers, avoiding the need to
examine every possible matching fragment pair.
This extended design allows compatible fragments to be grouped,
for example, by musical key, with each annotation in the main match
service container linking to a subsidiary container containing only
fragments that are compatible with the selected target fragment.
Implementation of this extended design for matching fragment
retrieval is contained entirely within the Find [service] annotations
targeting [some resource] function. It uses a special motivation value
attached to the matching annotation, which tells the client to look
at contents in a referenced container. Note that this is not a func-
tional change to the match retrieval service, but an optimization.
As before, details of optimised container construction are isolated
to the compatibility analysis agents that create them.
Music SOFA
Compatibility
service (CS)
Retrieve Fragment Match
annotations
GET
200 + content
Extract from content:
CS-URI sofa:working_sets WSI_URI
(Find annotation that targets required working set)
(annotation)
Extract MSI-URI from annotation
(Find annotation that targets required match type)
Extract MS-URI from annotation
(Find annotations that target identified fragment)
Figure 6: Compatibility service interactions (example)
3.3.2 Annotation retrieval optimizations. As described above,
finding compatible fragments has a client function for the Find
[service] annotations targeting [some resource] function retrieve all
annotations in a match service container.
An alternative implementation would have a specialised LDP
server, or some web service middleware, implement this iterate-
and-filter functionality, avoiding the need for the client to retrieve
un-needed annotations. It is not clear at this stage how much of this
could be handled using the existing LDP API and a specialised server
(e.g. by on-the-fly creation of response containers for a submitted
match query).
3.3.3 Distributed compatibility service. Our particular imple-
mentation has all of the compatibility service containers hosted
by a single LDP server. But the design does not require this, and
the compatibility service agents are free to create match service
containers on any LDP server, so the choice to use an alternative
match service container implementation (see previous sections) can
be made by a compatibility service agent with knowledge of the de-
tails of a particular compatibility criterion, and of likely associated
performance bottlenecks.
3.4 Instrument characteristics
The compatibility analysis for playability of a fragment by a desig-
nated instrument requires access to information about instruments
and their playable pitch-ranges. As no Linked Data publication of
this information was available, we used data for about 500 instru-
ments defined for Musescore7,8 and Annalist [12] to create Linked
Data 9 that can be accessed programmatically as JSON-LD or Turtle.
The intent is for the various instruments to be represented by the
Library of Congress Medium of Performance Thesaurus for Music
instrument type identifiers 10 following reconciliation. Currently,
7 https://musescore.org/
8 https://raw.githubusercontent.com/musescore/MuseScore/master/share/
instruments/instruments.xml
9 http://fast-project.annalist.net/annalist/c/FAST_musical_instruments_catalogue/
10 http://id.loc.gov/authorities/performanceMediums.html
SAAM’2018, October 2018, Monterey, California USA
Fragment
Working Set Match Service Match Service
Match
Index (WSI) Index (MSI) (MS)
Annotations
(annotation)
(get match annotations)
(Fragment match annotations)
we use generated URIs embedding musicXMLid values contained
in the Musescore data. Playable note ranges are represented using
MIDI note numbers obtained from Musescore data, and also as
notes symbols and frequency values obtained by mapping based
on the Scientific Pitch Notation table in Wikipedia11 .
3.5 User Interface
The user interface to the SOFA remixer application is implemented
as a Javascript web browser application, fitting within the MELD
client architecture described in section 1.3 and making use of addi-
tional, newly created ReactJS components.
The web application retrieves the set of fragments under con-
sideration in the current session by dereferencing the Working
Set LDP container which contains references to fragments output
by the Semantic Signal Generator. These Fragment References are
retrieved as Turtle formatted RDF and parsed and queried using the
rdflib.js library12 . Fragments of score are rendered to SVG engrav-
ings for display using the Verovio toolkit[19], this is also used to
create MIDI data which, using a Javascript MIDI player, allows the
user to listen to fragments as they are interacting with the system.
Candidate fragments for selection are filtered using the com-
patibility service (section 3.3) to display just those fragments that
are likely be be useful in the context of that selection. Annotations
identifying the Match services available for the current Working
Set are retrieved and the options for compatibility type shown to
the user are generated from the match type identifiers used. In this
particular application, an instrument playability match service and
a length match service are required for the built in functionality of
the remixer application, and any other match service types, such as
the key compatibility match service, are used to generate an option
in the UI so the user may further constrain the fragment selection.
Figure 6 illustrates the pattern of LDP container interactions
used to identify and locate compatible fragments. The client inter-
face locates the Working Set Index from a reference at the base
11 https://en.wikipedia.org/wiki/Scientific_pitch_notation
12 https://github.com/linkeddata/rdflib.js/
SAAM’2018, October 2018, Monterey, California USA
Compatibility Service URI and then retrieves annotations from this
location. It checks these annotations for one targeting the Working
Set under consideration and in the body of this annotation finds
the Match Service Index corresponding to this Working Set. The
same pattern of annotations within an LDP container is used to
find the compatibility services supported. If, for example, there is
an annotation with oa:hasTarget sofa:keyCompatibility then
the interface knows that key compatibility filtering data is available.
It looks for an rdfs:label at sofa:keyCompatibility in order
to generate a selection option in the UI and it uses the URI in the
body of the annotation to find the container of annotations actually
identifying compatibility between the fragments. It is these which
are used to filter the fragments offered within the interface.
4 DISCUSSION AND FUTURE WORK
We have presented the design and implementation of SOFA, a se-
mantic music system which illustrates a number of ideas which we
hope might contribute to this emerging field:
Music composition as assembly of semantically described
fragments. This contrasts with other work that describes the over-
all piece of music. As well as providing a more pervasive semantic
description of the music content, our approach also describes the
assembly (composition) of the fragments, and furthermore it assists
that assembly process. This in turn provides a demonstration to
the Semantic Web community of music as a semantically described
content type.
A new approach to Digital Music Objects. Building on our
previous MELD work, SOFA demonstrates use of Digital Musical
Objects (DMOs) as associations of semantically described fragments
that are drawn together as required from a sea of linked data. These
DMOs describe both musical content and options for its processing.
MELD as an architecture for music processing and recom-
position. This new application of MELD further validates it as a
common semantic framework for diverse music applications.
The notion of a semantic signal generator as part of the se-
mantic music workbench (which might be considered to have a
role analogous to “publishers’ latin” in document composition).
This methodology helps with implementing and testing the SOFA
architecture. Numbers into Notes could be used in this way by other
projects.
A "disappearing infrastructure" architecture that pushes se-
mantics from services into data, something that is particularly well-
supported by linked data standards. Rather than creating a new
service for each aspect of our distributed application, we have been
able to use common off-the-shelf LDP server software and move
the corresponding semantics into the data. There are parallels here
with the use of combinators in functional programming, where
common access and traversal patterns can be implemented just
once and used with a semantically diverse range of data types.
There are many directions of future work. We will work with
other forms of semantically described music content, and bring
other semantic music tools into the toolchain. In terms of technical
approaches, we see the potential of Linked Data Notifications [3],
and the architecture should support SoLiD [14] integration. There
is further work to be done on scalability and performance, and we
D. De Roure et al.
note the Semantic Signal Generator has proven effective in testing
these aspects.
EPILOGUE
Our semantic music system plays out the original thought experi-
ment. In Numbers into Notes, the role of the human is to make the
selection of numbers and the mapping to notes and an instrument
(of compatible range), always remaining faithful to the algorithmic
output. With the remixer, the computer is assisting in the assem-
bly process, using notions of compatibility that could have been
adopted in 1843.
We wonder what Lovelace might have made of our design—of
the ability to define a music machine, and of the ability of that
music machine to reconfigure itself: a data-defined, self-assembling
music machine.
ACKNOWLEDGMENTS
We are grateful to Emily Howard, Pip Willcox, and Alan Cham-
berlain for their essential contributions to the Numbers into Notes
concept and software, and contributing to its development and
creative applications. This research was supported by Fusing Au-
dio and Semantic Technologies for Intelligent Music Production and
Consumption funded by the UK Engineering and Physical Sciences
Research Council (EPSRC) under grant number EP/L019981/1, a col-
laboration between Queen Mary University of London, University
of Nottingham and University of Oxford.
REFERENCES
[1] Sean Bechhofer, Kevin Page, David M. Weigl, György Fazekas, and Thomas
Wilmering. 2017. Linked Data Publication of Live Music Archives and Analyses.
In The Semantic Web – ISWC 2017, Claudia d’Amato, Miriam Fernandez, Valentina
Tamma, Freddy Lecue, Philippe Cudré-Mauroux, Juan Sequeda, Christoph Lange,
and Jeff Heflin (Eds.). Springer International Publishing, Cham, 29–37.
[2] David Bretherton, Daniel Alexander Smith, Joe Lambert, and M. C. Schraefel.
2011. MusicNet: Aligning Musicology’s Metadata. In Music Linked Data Workshop.
http://eprints.soton.ac.uk/272408/
[3] Sarven Capadisli and Amy Guy. 2017. Linked Data Notifications. W3C Recom-
mendation. W3C. https://www.w3.org/TR/ldn/.
[4] Paolo Ciccarese, Benjamin Young, and Robert Sanderson. 2017. Web Annotation
Data Model. W3C Recommendation. W3C. https://www.w3.org/TR/2017/REC-
annotation-model-20170223/.
[5] Tim Crawford and Richard Lewis. 2016. Review: Music Encod-
ing Initiative. Journal of the American Musicological Society 69, 1
(Spring 2016), 273–285. https://doi.org/10.1525/jams.2016.69.1.273
arXiv:http://jams.ucpress.edu/content/69/1/273.full.pdf
[6] Herbert Van de Sompel, Michael Nelson, and Robert Sanderson. 2013. HTTP
Framework for Time-Based Access to Resource States – Memento. Technical Report
7089. https://doi.org/10.17487/RFC7089
[7] Leigh Dodds and Ian Davis. 2012. Linked Data Patterns. Online book. http:
//patterns.dataincubator.org/book/
[8] Roy Thomas Fielding. 2000. Architectural Styles and the Design of Network-based
Software Architectures. Ph.D. Dissertation.
[9] Ben Fields, Kevin Page, David De Roure, and Tim Crawford. 2011. The segment
ontology: Bridging music-generic and domain-specific. In Multimedia and Expo
(ICME), 2011 IEEE International Conference on. IEEE, 1–6.
[10] Martin Gasser, Andreas Arzt, Thassilo Gadermaier, Maarten Grachten, and Ger-
hard Widmer. 2015. Classical Music on the Web–User Interfaces and Data Rep-
resentations. In Proc. 16th International Society for Music Information Retrieval
Conference.
[11] Maria Kallionpää, Chris Greenhalgh, Adrian Hazzard, David M Weigl, Kevin R
Page, and Steve Benford. 2017. Composing and realising a game-like performance
for disklavier and electronics. In New Interfaces for Musical Expression (NIME
2017) Proceedings.
[12] Graham Klyne, Cerys Willoughby, and Kevin Page. 2016. Annalist: A practical
tool for creating, managing and sharing evolving linked data. Proceedings of the
Workshop on Linked Data on the Web (LDOW). http://ceur-ws.org/Vol-1593/
#article-10
Music SOFA SAAM’2018, October 2018, Monterey, California USA

[13] Clifford Lynch. 2002. Digital Collections, Digital Libraries and the Digitization
of Cultural Heritage Information. First Monday 7, 5 (2002). https://doi.org/10.
5210/fm.v7i5.949
[14] Essam Mansour, Andrei Vlad Sambra, Sandro Hawke, Maged Zereba, Sarven
Capadisli, Abdurrahman Ghanem, Ashraf Aboulnaga, and Tim Berners-Lee. 2016.
A Demonstration of the Solid Platform for Social Web Applications. In Proceedings
of the 25th International Conference Companion on World Wide Web (WWW ’16
Companion). International World Wide Web Conferences Steering Committee,
Republic and Canton of Geneva, Switzerland, 223–226. https://doi.org/10.1145/
2872518.2890529
[15] Luigi Federico Menabrea and Ada Augusta Countess of Lovelace. 1843. Sketch of
the Analytical Engine invented by Charles Babbage, Esq. In Scientific Memoirs,
Selected from the Transactions of Foreign Academies of Science and Learned Societies,
and from Foreign Journals (vol. 3), Richard Taylor (Ed.). Richard and John E. Taylor,
London, UK, 666–731.
[16] Open Archives Initiative. 2008. ORE Specification — Abstract Data Model. Techni-
cal Report. http://www.openarchives.org/ore/1.0/datamodel
[17] Kevin Page, David Lewis, and David Weigl. 2017. Contextual interpretation of
digital music notation. Digital Humanities (DH2017), Montréal, Canada (2017).
[18] Kevin R Page, Sean Bechhofer, Gyorgy Fazekas, David M Weigl, and Thomas
Wilmering. 2017. Realising a layered digital library: exploration and analysis
of the live music archive through linked data. In Digital Libraries (JCDL), 2017
ACM/IEEE Joint Conference on. IEEE, 1–10.
[19] Laurent Pugin, Rodolfo Zitellini, and Perry Roland. 2014. Verovio: A library for
Engraving MEI Music Notation into SVG. In Proc. 15th International Society for
Music Information Retrieval Conference. 107–112.
[20] Fazilatur Rahman and Jawed Siddiqi. 2012. Semantic annotation of digital music.
J. Comput. System Sci. 78, 4 (2012), 1219 – 1231.
[21] Yves Raimond, Samer A Abdallah, Mark B Sandler, and Frederick Giasson. 2007.
The Music Ontology. In Proc. 8th International Conference on Music Information
Retrieval. 417–422.
[22] D. De Roure and P. Willcox. 2017. Experimental Humanities: An Adventure
with Lovelace and Babbage. In 13th International Conference on e-Science. IEEE,
194–201. https://doi.org/10.1109/eScience.2017.32
[23] Steve Speicher, John Arwe, and Ashok Malhotra. 2015. Linked Data Platform 1.0.
W3C Recommendation. W3C. http://www.w3.org/TR/2015/REC-ldp-20150226/.
[24] David M Weigl, David Lewis, Tim Crawford, Ian Knopke, and Kevin R Page. 2017.
On providing semantic alignment and unified access to music library metadata.
International Journal on Digital Libraries (2017), 1–23.
[25] David M Weigl and Kevin R Page. 2017. A Framework for Distributed Semantic
Annotation of Musical Score: “Take It to the Bridge!”. In Proceedings of the 17th
International Society for Music Information Retrieval Conference.
[26] Megan A Winget. 2008. Annotations on musical scores by performing musicians:
Collaborative models, interactive methods, and music digital library tool devel-
opment. Journal of the Association for Information Science and Technology 59, 12
(2008), 1878–1897.