Proceedings of the 2015 Winter Simulation Conference

L. Yilmaz, W. K V. Chan, I. Moon, T. M. K. Roeder, C. Macal, and M. D. Rossetti, eds.

A Case Study of Aspect Oriented Design in Distributed Simulations

Arthur Valadares

Cristina V. Lopes

Donald Bren School of

Information and Computer Sciences
University of California, Irvine
6210 Donald Bren Hall
Irvine, CA 92697-3425, USA

Donald Bren School of

Information and Computer Sciences
University of California, Irvine
6210 Donald Bren Hall
Irvine, CA 92697-3425, USA

ABSTRACT
Within the Modeling and Simulation field, a major challenge that lies ahead is integrating massively complex
distributed simulations. Powerful standards such as the High Level Architecture (HLA) successfully establish
interoperability, yet the architecture and design of distributed simulators are neglected. The Distributed Scene
Graph (DSG) proposed an architecture similar to HLA, where simulators are separated by functionalities,
such as physics and scripting. These simulation concerns can be thought of as aspects, capturing cross-cutting
concerns of the same objective: the simulation.
This paper presents a case study in designing and implementing simulators as aspects in the DSG
architecture. We investigate both DSG and its evolution, DSG-M, for evidences of an increase in information
scattering and break in modularity. Our findings indicate that an aspect-oriented approach, such as in DSG
and DIS/HLA, leads to code tangling caused by interdependencies from cross-cutting concerns. We provide
suggestions to improve modularity and reduce tangling effect.
1

INTRODUCTION

One effective way of scaling up complex simulations is distribution. By leveraging hardware resources
from multiple sources, a simulation can make use of thousands of CPUs and large memory space to
achieve fine-grained and complex simulations that would be otherwise unfeasible. A recent study (Barnes
et al. 2013) shows how distributed simulation performance evolved in 6 orders of magnitude from 1990 to
2013, in most part due to improved parallelization from multi-core CPUs. However designing distributed
systems is challenging, requiring skilled software engineers to carefully design the architecture, protocols,
fault-tolerant distributed algorithms, and other non-functional requirements that may penalize functionality
and performance. Interoperable standards such as Distributed Interactive Simulation (DIS) (DIS Steering
Committee 1998) and High Level Architecture (IEEE Standards Association 2000) provide the networking
protocols and a powerful set of tools for distributed simulations, while implying a functionality oriented
simulation design. However, these standards do not provide simulation practitioners with the knowledge
for designing distributed simulation software architectures.
A Distributed Virtual Environment (DVE) called DSG was designed and implemented based on functional
division, where each simulator was responsible for parts of the simulation of a virtual environment. This
approach was shown to be highly scalable (Liu et al. 2010), while also providing separation of concerns:
simulators with different functionality could be evolved independently. A similar approach is also followed
by HLA simulations, as for instance, the Space Exploration Experience (SEE) (Taylor et al. 2014). Looking
closely at how such distributed systems are built, we notice a parallel between functional simulation workload

Valadares and Lopes

partitioning and a well known software engineering programming paradigm: Aspect-Oriented Programming
(AOP).
Our previous work in evolving the DSG architecture (Valadares et al. 2014) relies on separation of
concerns into modules, in the same way aspects capture cross-cutting concerns in AOP. However, while
our results were positive in terms of performance and workload scalability, we noticed an increase in
source code complexity which could threaten software evolution in the longer term. In this article we
pursue an investigation in the form of a case study of the modifications done to implement microcell
space partitioning (De Vleeschauwer et al. 2005) for our DVE. This paper explores the extent to which
aspects are a good conceptual model for engineering distributed simulations, and the software engineering
consequences resulting from inappropriate support for that conceptual model.
2

CURRENT PRACTICES

2.1 Software Engineering

The software engineering research community has formalized several programming and architectural
paradigms that aid developers to frame complex software systems in ways that are easier to comprehend,
evolve, and maintain. One such example is modularisation (Parnas 1972), on which the software designer
encapsulates data and procedures in a module, enabling developers of one module to be mostly unaware
of the design of other modules of the system. Parnas states that a system is better modularized when
information is hidden from most programmers, which is referred to as the information hiding property.
We present strengths and shortcomings of 2 paradigms that we find most relevant to the distributed and
real-time computing requirements of distributed simulations: object-oriented programming (OOP) and
aspect-oriented programming (AOP).
Object-oriented programming (OOP) decomposes modules in terms of abstract components. Modules
encapsulate data structures and modifying procedures as objects, used to encapsulate design decisions
regarding the implementation of the component. This way, objects can be replaced by different implementations, as long as they fulfill the same interface. OOP is a widely used paradigm, seen in many popular
languages, such as Java, Python, and C++. However, encapsulation by abstract data types leads to a form
of source-code tangling, as the main procedure has to transfer control between objects frequently. This
concern is made in Parnas On the Criteria to be Used in Decomposing Systems into Modules (Parnas
1972), in his analysis of efficiency and implementation of modularisation. Distributing an object-oriented
source-code may aggravate the issue of constant transfers of control, when network communication is
required to perform remote procedure calls.
Aspect-Oriented Programming (AOP) was introduced by Kiczales et al. (Kiczales et al. 1997), and
presented a critical problem found in source code development regarding modularisation: cross-cutting
concerns. Cross-cutting concerns are properties that fulfill a requirement that 1) is usually non-functional,
and 2) cannot be encapsulated in a procedural manner. Kickzales et al. propose that cross-cutting concerns
should be encapsulated separately to avoid code tangling, in this case referring to lines of code that fulfill
a different purpose that the procedure it is encapsulated by. These encapsulations of cross-cutting concerns
into modules are called aspects. When a method is cross-cut by an aspect, it triggers a call to another
method, in order to resolve the cross-cutting concern. The method called is referred to as advice. The
locations where advice are triggered from are called pointcuts.
A large part of research in AOP has been in terms of aspects as programming language concern, as
opposed to an architectural or design approach. AOP as a programming language has many different
implementations (Spinczyk et al. 2002, Pawlak et al. 2001, De Meuter 1997), the most popular being
Aspect J (Kiczales et al. 2001). Steimann (Steimann 2006) did a survey of AOP languages in attempt to
answer two questions: what is AOP? is AOP a positive contribution in terms of modularity? As AOP has
multiple implementations with different hypothesis, Steimann uses the definition by (Filman and Friedman
2000), where AOP is defined as quantification + obliviousness. Quantification expresses that aspects can

Valadares and Lopes

arbitrarily affect the code in many different points. Obliviousness express that advice invocation should
be automated, and oblivious to the developer of the methods where they are executed from.
With this definition in mind, Steimann argues that AOP may actually cause harm in terms of modularity.
Since aspects are oblivious to developer and may affect the state of any point of the code, there are no longer
guarantees that traditional modularity brings. When an advice is invoked, it may modify or even delete
variables, objects, and state without the knowledge from the basic functionality context. Thus, instead
of information hiding by co-locating code that executes in order, information becomes scattered across
aspects, requiring engineers, designers, and developers to be knowledgeable in a domain that may be large.
Despite critique of AOP as a language concern, cross-cutting concerns resulting in code tangling are a
fact. It is our view that the issues presented in Steimann were with the definition of AOP as quantification
+ obliviousness, a definition that is geared towards AOP as a programming language concern. Instead, for
the purpose of our case study, we formulate the following definitions for aspects as software design and
software architectural concern:

Aspect-Oriented Programming (AOP) as a software design and architectural concern, not programming language.
Advice need not be implicit calls for a software design to be AOP.
An event published in a pub-sub system is the equivalent of a pointcut in AOP.
An event listener method in a pub-sub system is the equivalent of an advice in AOP.
Synchronization between nodes of the distributed simulation is the equivalent of aspect weaving,
which we treat as an aspect in itself.
Distributed simulations with distributed functionality are inherently an AOP design.

Our previous work on distributed simulations, explained further in Section 3, showed that, under the
modified definitions, it is possible to encapsulate simulator functionality as aspects while gaining the benefit
of improved scalability and separation of concerns.
2.2 Distributed Virtual Environments
Distributed Virtual Environments (DVEs) are online multi-user interactive systems that simulate shared 3D
spaces. The first successful standard for distributed simulation was SIMNET (Miller and Thorpe 1995),
funded by the DoD and developed by DARPA. With SIMNET, it became possible to link hundreds of
simulators to produce a virtual world, used for real-time, man-in-the-loop, coordination and tactic simulations.
SIMNET eventually evolved into the Distributed Interactive Simulation (DIS) standard, becoming an IEEE
standard (IEEE 1278-1993) (DIS Steering Committee 1998).
Two foundational papers in DVE are DIVE (Carlsson and Hagsand 1993) and Diamond Park (Waters
et al. 1997), both proposing VEs that scale by partitioning the workload in terms of space. Each process
handles an area of the virtual world, achieving optimal scalability in terms of virtual land area, and improved
scalability of users and objects. Most DVEs used today utilize the same scalable approach of partitioning
the world by space.
2.3 Distributed Simulations
Concurrently to the effort of developing DVEs, another effort unfolded to generalize the concept of distributed
simulations beyond interactive environments. Simulations dont always have an interactive component, and
even if they do, it is important to generalize the model by which the different simulators work together to
produce the collaborative, complex simulation.
For this purpose, another standard was developed with the participation of industry and academia
partners: the High Level Architecture (HLA) (IEEE Standards Association 2000). The HLA is, as the
name implies, higher level when compared to DIS. In DIS, most of the simulation design decisions, such
as networking protocols, are fixed. The HLA is flexible, and enforces an API that simulation designers

Valadares and Lopes

can use to make decisions for their own simulation. The HLA defines two separate entities: the federates
and the runtime infrastructure (RTI). The RTI is the communication bus for simulators to share events and
updates. It is also used to specify and share the object model template (OMT), a shared specification of
what objects are available to be instatiated. New objects can be defined and shared between simulators.
The federates are the simulators, and together with the RTI, they form the federation. The API from the
HLA defines how federates should share object models, updates, events, and interactions.
The RTI has many commercial implementations, including VT Mak RTI and Pitch pRTI. Two well
known open-source implementations are OpenHLA and Portico. Simulators can be adapted to the HLA
protocol through what is called an HLA ambassador: a software module that translates events from the
RTI to the simulator, and interprets internal events and routes back to RTI.
3

PREVIOUS WORK

Our case study describes the modifications made to evolve DSG in to DSG with Microcells, based on the
work we did in collaboration with Intel Labs. In this Section, we describe in detail the architecture of
DSG, and the modifications made for DSG-M.
3.1 Distributed Scene Graph
The Distributed Scene Graph is an architecture for distributed simulations of virtual environments. Traditionally, virtual environments scale up by partitioning the virtual land: different processes handle different
areas of space, and objects are migrated between processes if they move outside the area of authority of
the process they belong. While this approach enables infinite scalability of land, it fails when simulated
objects and users are agglomerated, which is the typical use case in virtual environments(Chen and Lei
2006, Pittman and GauthierDickey 2007).
DSG provided a novel way of partitioning virtual environment simulation workload, using functionality
instead of space. A virtual environment simulation computes multiple characteristics of the real world
that we are accustomed, such as physical properties (gravity, momentum), scripted objects (traffic lights,
bots), inventory, and client connections and commands. DSG breaks these characteristics of the simulation
apart by functionality, and runs them separately in processes that are distributed, synchronizing the virtual
in-world objects between them. We see this partitioning of virtual environments by functionality as aspects:
characteristics of the simulation that cannot be cleanly encapsulated in independent modules. (Liu et al.
2010) demonstrates how using space partitioning techniques such as binary space partitioning (BSP) alone
leads to region fragmentation and high overhead for migration and communication between servers.
DSG is implemented as plugin for OpenSimulator (OpenSimulator 2015). OpenSimulator is a highly
customizable virtual environment platform, enabling its functionalities to be pluggable, and even removed.
DSG picks which functionality of OpenSimulator to use depending on the actors purpose. For example,
non-physics actors do not have a have a physics engine: they bypass physics calculations. The same
happens with non-script actors: they have script objects, but do not execute the scripts.
3.2 Distributed Scene Graph with Microcells
Despite the successful results presented by the functionality workload partitioning in (Liu et al. 2010),
space partitioning is still the best solution for some workloads. Physics, for instance, does not scale well as
more objects are added to the virtual world, but has locality properties that make space partitioning viable.
Consequently, the next step in our research was scaling virtual environments by enabling both forms of
workload partitioning simultaneously. Thus we introduced microcell space partitioning to DSG. We use the
microcell partitioning approach from (De Vleeschauwer et al. 2005), where space is divided into equally
sized areas of space that are small enough to not be overwhelmed.
We named the improved architecture as DSG with Microcells (DSG-M), which was successfully built,
evaluated, and shown to improve of workload partitioning even under worst case conditions (Valadares,

Valadares and Lopes

Project
OS
DSG
DSG-M
DSG
Ratio

Cyclomatic
Complexity
51,879
2,261
2,640
379
16.76%

Class
Coupling

508
525
17
3.23%

Lines of
Code
297,981
9,302
10,922
1,620
17.42%

Number
of Types
3,421
98
111
13
11.49%

Number of
Methods
31,007
1,534
1,682
148
9.65%

Table 1: Traditional software engineering metrics for OpenSimulator (OS), Distributed Scene Graph (DSG),
and DSG with Microcells (DSG-M). DSG is the difference between DSG-M and DSG, and Ratio is DSG
divided by the respective metric of DSG.
Lopes, and Liu 2014). DSG-M improved load balancing, affording the execution of a simulation that was
otherwise impossible with DSG alone. However, we noticed that the source code became far more complex
than it used to be, which prompted an investigation. DSG-M provided a unique opportunity to show how
the original design was challenging to scale, by weaving a new aspect to DSG: space partitioning. The
DSG-M source code is available online, open-sourced, at http://github.com/arthur00/DSG.
Our case study focuses on the aspect-oriented design of DSG and DSG-M. We identify a basic
functionality and 3 different types of aspects in DSG-M:

Actors: Each actor is an aspect responsible for fulfilling a part of the simulation (i.e. a cross-cutting
concern), such as performing physics simulations, executing scripts, or handling clients.
Synchronization: The synchronization aspect maintains the state consistent across actors.
Space Partitioning (DSG-M only): The space partitioning aspect partitions the workload by virtual
space, so that if the workload of an actor type is too large, it can be divided.
Basic Functionality: Basic functionality are all the common simulation workload that is not divided
by actors, such as handling objects, object groups, attachments, and inventory.

CASE STUDY: QUANTITATIVE ANALYSIS

To illustrate the concepts we have discussed so far, we perform an in-depth analysis of a distributed real-time
application we have worked on in the past, with collaboration from Intel Labs, the Distributed Scene Graph
with Microcells, or DSG-M.
In this Section and in Section 5, we present evidence of high aspect complexity in DSG. This section
presents quantitative metrics from DSG, DSG-M, and OpenSimulator. In Section 5, we use our metrics
to categorize aspect complexity of DSG in terms of architectural erosion, cross-cutting concerns and code
tangling. Then we discuss the impact of introducing space partitioning with microcells, and show evidence
of code tangling and architectural erosion permeating through the design.
4.1 Software Engineering Metrics
We start our study by providing some general numbers and OOP software engineering metrics for the
code base of DSG, DSG-M, and the base simulator platform, OpenSimulator. Table 1 shows cyclomatic
complexity, class coupling, total lines of code, and number of methods and types for OpenSimulator, DSG,
and DSG-M. In the last lines, we subtract DSG metrics from DSG-M and show the ratio increase in
complexity when space partitioning was introduced in DSG-M.
OpenSimulator alone has 297,981 lines of source code, 3,421 types in 98 modules. Both DSG and
DSG-M are organized into 2 modules, where one is a small physics related plugin, with only 219 lines of
code. We exclude this module from our analysis, as it was not touched from DSG to DSG-M, nor does it
present significant cross-cutting concerns.

Valadares and Lopes

Aspect Metrics Modified
Methods
Types
Files

272
24
13

Non-Empty
Ratio
Total
683
39.82%
88
27.27%
28
46.43%

Table 2: Metrics for aspects: number and ratio of modified methods, types, and files, based on source code
modifications of the evolution from DSG to DSG-M.
From the last line of Table 1 we can see that the increase in size (i.e. lines of code) was similar to
the increase in cyclomatic complexity. Class coupling increase was small, and number of methods and
types were also proportional to both size and cyclomatic complexity. These simple metrics, however, do
not capture the true increase in complexity.
4.2 Aspect Metrics
Measuring the presence and impact of aspects with metrics is still a challenge to the AOP research community.
For our case study, we have based our metrics from (Lopez-Herrejon and Apel 2007), particularly the
Aspects Code Fraction (ACF), meaning the fraction of lines of code that come from aspects relative to
the number of lines of code in the method. We use number of methods, types, and files modified as our
metrics for aspects.
Table 2 shows both the aspect metrics, and their ratios in our source code. One difference when using
ratio was picking DSG methods that were non-empty (683) as opposed to all DSG methods (1534). Empty
methods were mostly hidden methods such as getters, setters, event adders, and other language-specific
methods.
Comparing to Table 1, which showed only a modest increase in source code complexity and size,
Table 2 indicates the increase was in great part due to modifications spread across the original source.
Considering that all new code was for the implementation of the space partitioning aspect, the ratios
represent a significant increase in cross-cutting concerns. Next, we will have a closer look at the source
code and understand why space partitioning had a high ratio of method modifications.
5

CASE STUDY: QUALITATIVE ANALYSIS

The DSG architecture encapsulates 2 aspects: actors and synchronization. The actors aspect partitions
the simulation workload by functionality. The synchronization aspect is a non-functional, but an essential
requirement of DSG. All updates to objects must be synchronized with all other actors, as they all share the
data representation of the same virtual environment entities. To demonstrate code tangling and cross-cutting
concern, we use a simplified snippet example from DSG, seen in Source Code 1.
5.1 Code Tangling
Source Code 1 is highlighted in different colors, to show different concerns tangled in the same procedure.
We highlight the lines in two colors: dark gray is code concerned with DSG actors, such as physics and
scripting. Light gray is code concerned with basic functionality, the simulator. Lines without highlighting
are concerned with the functional requirement of the procedure: synchronization of updated properties.
The method is an event handler for updates to in-world entities. Ideally the procedure would not have any
highlighted lines, encapsulating only the synchronization aspect.
There are 19 lines of code in the procedure, and only 4 of those lines are concerned entirely with
synchronization of updated properties. For each aspect that cross-cuts the procedure forces the developer
to know and understand the cross-cutting aspect. This is a symptom of a break in modularity: the code
meant for synchronization has semantic dependencies on basic and DSG aspect functionalities. The result

Valadares and Lopes

1
2
3
4

private void OnSceneObjectPartUpdated(SceneObjectPart part) {

if (IsLocallyGeneratedEvent(UpdatedProperties || NewObject))
return;

if (part.PhysicsActor == null)

6
7
8
9

properties.ExceptWith(PhysicsActorProperties);
HashSet<PropertyTypes> typesUpdated = UpdateSyncInfoByLocal(uuid, properties);

10

if (uuid != part.ParentGroup.UUID) {

11

propertiesWithSyncInfoUpdated.ExceptWith(GroupProperties);

12
13

14

if (part.ParentGroup.IsAttachment) {

15
16
17
18
19

typesUpdated.ExceptWith(AttachmentNonSyncProperties);
}
EnqueueUpdatedProperty(uuid, typesUpdated);
}

Source Code 1: Method in DSG-M for synchronizing object updates across actors.

is a failure of realizing information hiding, requiring developers to have a vast knowledge of all aspects
and basic functionality to perform even small modifications. We discuss next some of the causes of the
observed code tangling.
5.2 Cross-Cutting of Aspects
An overlooked concern of AOP design is the unpredictability of how aspects cross-cut each other (Steimann
2006, Elrad, Filman, and Bader 2001). In DSG, each actor is an aspect of the simulation, and actors are
interwoven to deliver the simulation as if there was no distribution. Ideally, actors and synchronization
would be independent aspects. Yet we see cases where the computation of one aspect depends intrinsically
on another.
One such example is on lines 5 and 6 of Source Code 1. This method is handling an event caused by
an update performed on an in-world entity, referred to in the code as a SceneObjectPart. Line 5 checks
for physical properties in the entity. The meaning of a PhysicsActor attribute in a SceneObjectPart is
that it has physical properties, only present in DSG actors with physics. The separation of physics and
non-physics properties exist so that simulators can separate visual representation from physical values.
While many actors have position and rotation values for entities, typically only one actor is responsible
for the simulation of the physical properties.
Lines 5 and 6 violate aspect independence: synchronization aspect has now a dependency on an DSG
actor. This dependency leads to a break in modularity: changes made to the physics actor in DSG may have
unintended consequences with synchronization. There is a large semantic gap between these aspects, such
that a developer modifying properties in the physics would not intuitively keep in mind synchronization
dependencies. Ideally, such distinctions would not be necessary for synchronization, but they are required
for improving synchronization, both in terms of performance and consistency.
The break in modularity goes beyond aspect to aspect cross-cutting. Aspect cross-cutting basic
functionality also creates a dependency, breaking information hiding. Two examples can be seen in lines
10-12 and 14-16 on Source Code 1. Line 10 checks if the updated entity is part of a group or not. Grouped
entities are used for building complex forms, such as table. If you move or resize a part, all the parts must
move and resize in the same amount. This concern is basic functionality, and a developer handling this
synchronization method would have to be aware of.
Line 14 has a similar basic functionality concern, now regarding attachments. When the updated
property is an user attachment, there is no need to synchronize all the regular properties, they can, and
should, be calculated from the avatar. The takeaway from both cases is that this procedure imposes the

Valadares and Lopes

need of understanding OpenSimulator attachments and object groups when modifying a synchronization
aspect procedure, hindering software maintenance and evolution.
5.2.1 Quantification and Events
Aspects, as defined by quantification + obliviousness, introduces an issue that is similar to the condemned
go to statements: lack of linearity. With quantification, an advice can be executed from any point of the
code, breaking modularity. Uncoincidentally, pub-sub systems face the same design issue of traceability
and debugging. To illustrate these concerns, we discuss two examples.
To illustate, we discuss a real bug discovered during the implementation of DSG-M. When a user left
the simulator, we noticed several warning messages on the console referring to objects that could not be
deleted, as they no longer existed. Moreover, the number of warnings was different for each actor. Figure
1 shows the sequence of events that led to the bug. On the top, is the execution pertaining DSG data
synchronization, and on the bottom, the basic functionality of the simulator.

Figure 1: Flowchart describing the sequence of events that led to a bug, caused by the mix of aspects and
basic functionality.
Following the arrows of Figure 1 from left to right, these are the events that triggered the bug: a)
user disconnects, prompting a RemoveClient event on the Client Manager actor; b) the simulator triggers
an OnRemovePresence event, indicating the client is leaving; c) DSG synchronization aspect picks up
on this event and sends a message informing other actors that the client left; d) control is returned to
the simulator, which moves on to destroy the users attachments; e) each attachment is an entity, firing
several RemoveObject events; f) DSG synchronization aspect picks up on RemoveObject events and sends
them to other actors. The undesired cross-cutting between synchronization and basic functionality led to
unexpected system behavior.
5.3 Space Partitioning
The design of DSG-M and the use of microcells for space partitioning is described in Section 3. In
Section 4, we showed metrics that summarizes the changes from DSG to DSG-M, with the introduction of
space partitioning. All modifications made from DSG to DSG-M were related to space partitioning with
microcells.
We observe that cyclomatic complexity, class coupling, and inheritance depth, increased to a proportion
similar to the increase in lines of code. Yet the aspect metrics in Table 2 showed that there was a significant
increase in aspect complexity: with 39.82% of non-empty methods being modified. These numbers suggest
that the new aspect, space partitioning, is cross-cutting a large number of our active (i.e. non-empty)
methods.
When introducing space partitioning in DSG-M, we defined a new class called the MicrocellManager, responsible for encapsulating methods and data relating to space partitioning. This class contains all concerns regarding space partitioning with microcells, including translation of x,y coordinates into microcell coordinates,
maintaining publish-subscription lists for determining what actors subscribe to which microcells, and implement both passive and active subscription. Despite our effort in encapsulating all space partitioning concerns,
it was simply not possible to cleanly modularize space partitioning under DSGs functional architecture for two
reasons:

Valadares and Lopes

1) Synchronization is deeply dependent on space partitioning.

The motivation behind space partitioning is of filtering. By reducing the number of interested nodes for a
publication, we reduce network communication, and can scale better for the same limited hardware resources.
Thus, space partitioning influences the way we synchronize data across actors. When a synchronization message is received, we must check with the MicrocellManager to determine if it should be forwarded, to whom it
should be forwarded, if it should be applied to the current simulator, and if we should handle it locally without
synchronization.
2) Space partitioning has dependencies on basic functionality.
In addition to modifying the code path of all synchronization procedures, we must also track position
updates of objects to detect region crossings. By doing so, we create dependency between space partitioning
and position updates within the simulator. These updates are caught by the same event handler seen in Source
Code 1, completing a chain of dependencies from basic functionality (position update), to synchronization
(OnUpdatedProperties), to space partitioning, checking for positions outside of the boundaries the actor
can publish to.
When objects migrate from one actor to another, we have the same dependency the other way around. The
MicrocellManager (space partitioning aspect) uses the position update to create a migration message. The
object migration is received (synchronization aspect), leading to creation of the object (basic functionality).
6

DISCUSSION

Distributed simulations that follow an aspect-oriented design claim one major advantage in terms of software
design: modularity. If simulators can be built independently and attached together by an interoperable
standard, the complexity of making a simulation distributed is gone, and simulation practitioners would be
free to focus on their simulation expertise. Unfortunately, our case study shows evidence of the contrary:
aspect-oriented distributed simulations may lead to code tangling, and the inter-dependency between aspects.
In Section 5.3, we show that the introduced space partitioning aspect had strong ties with synchronization,
and indirect ties with basic functionality. As we have shown before that synchronization and actors are
also dependent, it follows that space partitioning is also dependent on synchronization, actors, and basic
functionality. We also show that when objects migrate dependencies exist both ways. In summary, despite
DSG-Ms effort in encapsulating cross-cutting concerns in separate modules (i.e. aspects), the essential
dependency between these modules break modularity, and little to no information hiding is accomplished.
We have shown that within the DSG-M application, it is not possible to separate the dependencies
between the cross-cutting concerns, leading to the break in modularity. From this finding, we have two
general research questions that require investigation:
RQ 1) Are dependencies between cross-cutting concerns of distributed simulations accidental (i.e. the
result of inappropriate design) or essential (i.e. inherent to the problem of distributed simulations)?
RQ 1.1) If they are accidental, how can distributed simulations be designed so that dependencies are
removed?
RQ 1.2) If they are essential, how can dependencies be detected early in the design and architectural
phases of a distributed simulation?
The answers to these questions are outside the scope of our case study, but we can argue that some
aspects are essential, and will always cross-cut aspect-oriented distributed simulations. For instance,
synchronization of state is an aspect that creates a dependency with individual actors (or federates, in
HLA), making it difficult to abstract away the state synchronization from the simulator. This is possibly
the reason why HLA is considered to be highly complex by simulation practitioners (Boer, De Bruin, and
Verbraeck 2006). Next, we provide some suggestions to address our research questions by mitigating code
tangling and cross-cutting concerns.

Valadares and Lopes

6.1 Distributed Object Model
One way to reduce dependency based on functionality (e.g. DSG actors) is to design object-oriented models
for functionally distributed simulators. A similar concept is used in the object models of HLA. In HLA,
a component called RTI (runtime infrastructure) manages the pub-sub of objects attributes. Only one
distributed node is allowed to write (i.e. publish), but multiple nodes may subscribe to different attributes
of the same object. The RTI creates a proxy object, based on a metaclass description of the object, and
automatically updates the attributes as updates are published.
Our suggestion is to use the same pub-sub approach of HLA, which has proven to scale well in
distributed simulations, but add a new form of inheritance, which we call view objects.

Figure 2: Class diagram demonstrating how an example distributed object-oriented model of car across 3
aspects: physics, visual/interactive, and statistics.
Figure 2 shows an example of view objects. The idea is to create objects that provide views of the
original, in the same manner a database view relates to the database tables. In Figure 2, a car abstract data
type is first defined, without any methods or data types, except a unifying id. The physics simulator defines
its own view of the object (PhysicalDistributedCar), containing physical properties for position, speed,
acceleration, and others. A virtual environment simulator defines another view object (VisualDistributedCar,
subscribed (read-only) to the physical properties of the object in the physics simulator. Finally, a statistics
module implements one more view object (StatsDistributedCar for maintaining statistical data about
the car. This view has attributes from PhysicalDistributedCar (position, velocity and acceleration) and
VisualDistributedCar (color).
What makes view objects different from traditional objects are that all view objects in Figure 2 share
the same id, and represent one simulator object from different points of view. The separation of concerns
are embedded in the abstract data types design. In addition to the known advantage of OOP in saving
memory by omitting unnecessary attributes of an object, we also reduce code tangling, as no code paths
need to be taken to check if updates of different views apply to the actor. Compare to Source Code 1,
where Line 5 checked if the actor had physical properties when handling an updated property event.
6.2 Improve Aspect Complexity Metrics
During the design and implementation of a distributed simulation, automatic detection of cross-cutting
aspects could feasibly be done if the developer could tags methods as part of an aspect. This approach
is not an automated detection of aspect, but rather tool-supported flagging. An IDE could allow tagging
of methods per aspect name or basic functionality. When method calls are done that cross aspects, the
IDE reports it as bad practice, and measure the amount of aspect/basic functionality cross-cutting that is
happening. This information will give the architect an idea of how much information scattering is being
caused by inter-aspect procedural calls or data access. If the issue is accidental rather than essential, the
developer can untangle the source code to improve information hiding.

Valadares and Lopes

7

CONCLUSION

We have thoroughly analyzed a real-world complex distributed simulation called DSG-M. DSG-M is an
evolved version of DSG, an architecture for distributing virtual environments in terms of functionality (a.k.a.
actors), such as physics or scripting. During our collaboration with Intel, we have developed DSG-M as
an evolved version, that also distributes actors in terms of space.
DSG has a basic functionality, which is the simulator, and two aspects: synchronization and actors.
In DSG-M, we added a third aspect: space partitioning. We analyzed closely the changes from DSG to
DSG-M in order to understand a perceived increase in software complexity. The number of lines of code
grew proportional to cyclomatic complexity, inheritance depth, and class coupling. However, when looking
closely at metrics for AOP, we see that the change touched 39.82% of the methods in DSG, implying a
high amount of cross-cutting caused by the space partitioning concern.
We cannot make generalizations to all aspect-oriented models based on one case study, yet we see
evidence that some cross-cutting concerns identified in our study are a common issue to all distributed
simulations, and lead to source code tangling and break in modularity caused by inter-dependencies between
aspects. We presented two suggestions for approaching this problem. First, the design of view object
models: partial representation of distributed objects with cherry-picked attributes. Second, we suggest
tool-supported measurement of cross-cutting concerns for distributed simulation design.
REFERENCES
Barnes, P. D., C. D. Carothers, D. R. Jefferson, and J. M. LaPre. 2013. Warp speed: executing time warp
on 1,966,080 cores. In Proceedings of the 2013 ACM SIGSIM conference on Principles of advanced
discrete simulation - SIGSIM-PADS 13, 327. New York, New York, USA: ACM Press.
Boer, C., A. De Bruin, and A. Verbraeck. 2006, December. Distributed Simulation in Industry - A Survey
Part 1 - The Cots Vendors. In Proceedings of the 2006 Winter Simulation Conference, 10531060:
IEEE.
Carlsson, C., and O. Hagsand. 1993. DIVE A multi-user virtual reality system. Proceedings of IEEE
Virtual Reality Annual International Symposium - VRAIS 93:394400.
Chen, K.-t., and C.-L. Lei. 2006. Network game design: hints and implications of player interaction. In
Proceedings of 5th ACM SIGCOMM workshop on Network and system support for games - NetGames
06, 17.
De Meuter, W. 1997. Monads as a theoretical foundation for AOP. In International Workshop on AspectOriented Programming at ECOOP 97, Volume 97.
De Vleeschauwer, B., B. Van Den Bossche, T. Verdickt, F. De Turck, B. Dhoedt, and P. Demeester. 2005.
Dynamic microcell assignment for massively multiplayer online gaming. In Proceedings of 4th ACM
SIGCOMM workshop on Network and system support for games - NetGames 05, 17. New York, New
York, USA: ACM Press.
DIS Steering Committee 1998. IEEE Standard for Distributed Interactive Simulation-Application Protocols. IEEE Standard 1995.
Elrad, T., R. E. Filman, and A. Bader. 2001. Aspect-oriented programming: Introduction. Communications
of the ACM 44 (10): 2932.
Filman, R. E., and D. P. Friedman. 2000. Aspect-oriented programming is quantification and obliviousness.
Technical report.
IEEE Standards Association 2000. 1516-2000 IEEE Standard for Modeling and Simulation (M&S) High
Level Architecture (HLA)-Framework and Rules.
Kiczales, G., E. Hilsdale, J. Hugunin, M. Kersten, J. Palm, and W. G. Griswold. 2001. An Overview of
AspectJ. ECOOP 01 2072 (4): 327353.

Valadares and Lopes

Kiczales, G., J. Lamping, A. Mendhekar, C. Maeda, C. V. Lopes, J.-m. Loingtier, and J. Irwin. 1997.
Aspect-Oriented Programming. In ECOOP97 Object-Oriented Programming, edited by M. Aksit
and S. Matsuoka, Volume 220-242, 220242. Springer Berlin / Heidelberg.
Liu, H., M. Bowman, R. Adams, J. Hurliman, and D. Lake. 2010, December. Scaling virtual worlds:
Simulation requirements and challenges. In Proceedings of the 2010 Winter Simulation Conference
(WSC), edited by B. Johansson, S. Jain, J. Montoya-Torres, J. Hugan, and E. Yucesan, WSC 10,
778790. Baltimore, MD: IEEE Computer Society.
Lopez-Herrejon, R., and S. Apel. 2007. Measuring and Characterizing Crosscutting in Aspect-Based
Programs: Basic Metrics and Case Studies. Fundamental Approaches to Software Engineering:423
437.
Miller, D., and J. Thorpe. 1995. SIMNET: the advent of simulator networking. Proceedings of the IEEE 83
(8): 11141123.
OpenSimulator 2015. OpenSimulator. http://opensimulator.org.
Parnas, D. L. 1972. On the criteria to be used in decomposing systems into modules. Communications
of the ACM 15 (12): 10531058.
Pawlak, R., L. Duchien, G. Florin, and L. Seinturier. 2001. JAC: A flexible solution for aspect-oriented
programming in Java. In Metalevel architectures and separation of crosscutting concerns, 124.
Springer.
Pittman, D., and C. GauthierDickey. 2007. A measurement study of virtual populations in massively
multiplayer online games. In Proceedings of the 6th ACM SIGCOMM workshop on Network and
system support for games - NetGames 07, 2530.
Spinczyk, O., A. Gal, and W. Schroder-Preikschat. 2002. {AspectC++}: an aspect-oriented extension to
the {C++} programming language. In Proceedings of the Fortieth International Conference on Tools
Pacific, CRPIT 02, 5360. Darlinghurst, Australia, Australia: Australian Computer Society, Inc.
Steimann, F. 2006. The paradoxical success of aspect-oriented programming. ACM SIGPLAN Notices 41:481.
Taylor, S. J., N. Revagar, J. Chambers, M. Yero, A. Anagnostou, A. Nouman, N. R. Chaudhry, and P. R.
Elfrey. 2014, October. Simulation Exploration Experience: A Distributed Hybrid Simulation of a
Lunar Mining Operation. 2014 IEEE/ACM 18th International Symposium on Distributed Simulation
and Real Time Applications:107112.
Valadares, A., C. Lopes, and H. Liu. 2014. Enabling fine-grained load balancing for virtual worlds with
distributed simulation engines. In Proceedings of the 2014 Winter Simulation Conference, edited by
J. A. M. A. Tolk, S. Y. Diallo, I. O. Ryzhov, L. Yilmaz, S. Buckley, 34593470.
Waters, R. C., D. B. Anderson, J. W. Barrus, D. C. Brogan, M. A. Casey, S. G. McKeown, T. Nitta, I. B.
Sterns, and W. S. Yerazunis. 1997. Diamond park and spline: A social virtual reality system with
3D animation, spoken interaction, and runtime modifiability. Presence: Teleoperators and Virtual
Environments 6 (4): 461480.
AUTHOR BIOGRAPHIES
ARTHUR VALADARES is a PhD Informatics candidate in the Bren School of Information and Computer
Sciences at the University of California, Irvine. His research is centered in scaling virtual environments to
accommodate thousands of users and virtual objects. He received his M.S. at University of California, Irvine,
and his B.S. at Universidade Estadual de Campinas in Brazil. His e-mail address is avaladar@ics.uci.edu.
CRISTINA V. LOPES is a Professor of Informatics in the Bren School of Information and Computer
Sciences at the University of California, Irvine. Her research focuses on software engineering for large-scale
data and systems. She has a PhD from Northeastern University, and MS and BS degrees from Instituto
Superior Tecnico in Portugal. Her email address is lopes@ics.uci.edu.