From Simulation to Online Gaming:

the need for adaptive solutions

Gabriele D’Angelo
<gda@cs.unibo.it>
http://www.cs.unibo.it/gdangelo/

Department of Computer Science

University of Bologna
Presentation outline

Simulation

Parallel and Distributed Simulation (PADS)

Adaptive migration-based approach

Simulation of scale-free networks

Gossip protocols

Scale-free networks and gossip protocols for online gaming

Conclusions and future work

2
Simulation and much more

“A simulation is a system that represents or emulates the

behavior of another system over time. In a computer

simulation the system doing the emulating is a computer

program” (Richard M. Fujimoto)

The simulation is a widely used technique in the performance

evaluation, risk assessment and security analysis of physical

systems (or during the design phase)

It is fundamental for the creation of Digital Virtual

Environments (DVEs) and Online Games

3
Limitations of the monolithic approach

Many of the systems of interest are composed by a very large

number of entities, with a really dynamic nature and interaction

pattern

For example, wireless networks, under the simulation viewpoint, have

very strict requirements in terms of level of detail

Monolithic simulators are unable to fulfill these scalability

requirements due to memory constrains and excessive amount of

time required to obtain the results

A Parallel And Distributed Simulation (PADS) can be used to

aggregate memory and computational resources

4
Parallel And Distributed Simulation (PADS)
In this case, the state of the simulation and its evolution is shared
among a set Logical Processes (LPs)

Each LP is responsible to manage only a subset of Simulated

Entities and is allocated on a different Physical Execution Unit
(PEU)

The PEUs are interconnected together using tightly coupled (parallel

simulation) loosely coupled (distributed simulation) communication
networks

To obtain correct results it is necessary that the causal order of

events is respected, two main approaches have been proposed:

Conservative (Chandy-Misra-Bryant, CMB)

Optimistc (Time-Warp)

5
The main drawbacks of PADS

The PADS approach can aggregate many memory and computational

resources: it has potential for very good scalability

This approach is not free from drawbacks, the distributed execution

architecture needs appropriate mechanisms for:

Synchronization (and more in general, communication)

Load-balancing

Many approaches and techniques have been proposed to cope with

these problems: in particular to reduce the communication

overhead and to build-up efficient synchronization schemes

6
Partitioning problem

A simulation can be seen as a set of Simulation Entities (SE) that

interact together

In our vision, the reduction of the communication overhead and

the computational load-balancing problems have to be

considered at the same time

In practice, it is an instance of a dynamic partitioning problem in

which there are many SEs, a (variable size) set of heterogeneous

PEUs (with the corresponding LPs) and a communication network

(with performance varying in time)

7
Migration-based approach

All the static approaches are suboptimal given the variability in the

load generated by the simulated mode, the execution platform and

the communication network

It is necessary to employ a dynamic and adaptive approach that

reacts to imbalances in the architecture and proactively enhances the

model partitioning

We propose a migration-based approach: each entity in the

simulation can be dynamically relocated (migrated), moving from a

source PEU to a new destination PEU

8
Migration-based approach

Clustering in the same LP the highly interacting entities it is

possible to reduce the costly inter-LP communication and conversely

increasing the rate of low cost intra-LP communication

This can be done considering the communication pattern of each

model entity that is in the simulation

Given the unpredictable nature of parts the simulated model and of

many parameters in the execution architecture it is not feasible to

compute the optimal solution at runtime:

we have to rely on heuristics (simple, fast and scalable)

9
An example: communication overhead reduction

A set of Simulated Entities The Simulated Entity x is

X
X allocated on PEUs A and B Migrated from Host A to B
with the aim to reduce the
The entity x (PEU A) communication overhead
produces a
“transmission-event”
that involves some other
Simulated Entities

Physical Execution Units Physical Execution Units

A B A B

network delay network delay

X

The most of the interactions Clustering the entities it is possible to

experience a (quite large) increase the amount of
network delay memory-to-memory interactions

10
Migration-based approach: load balancing
This approach can be also used to improve the computation load-
balancing in the execution architecture

The synchronization points in the distributed architecture are

exploited to tag each LP as “fast” or “slow”

A Logical Process is “slow” if:

its PEU (i.e. CPU) is overloaded

its communication network has a higher delay with respect to

other parts of the execution architecture

In both cases, to speed up the simulation, the solution is to migrate

some entities from the slow LPs to faster LPs, therefore acting to
reduce imbalances

11
Migration-based approach: load balancing

This description of the mechanism is very high level, the “real world”

implementation has to take in account many subtle details such as:

each migration has a cost that depends on many factors (e.g.

the state size of entities, the network bandwidth and latency)

the algorithm has to be “fully distributed” (without any

centralization point) and therefore has to take decisions

without any “global vision” of the distributed system

the mechanism has to react very quickly to internal (i.e. the

creation of new simulated entities) and external events (i.e. a

burst of CPU or network load) but without introducing

oscillatory behaviors
12
Positive side-effects

This approach can control the communication overhead and can be

used to enhance the partitioning of simulated entities

the overhead introduced by the mechanism can be controlled

very easily: tuning the rate of migrations

it adaptively reacts to the behavior of the simulated model,

such as hotspots in the simulated area

the execution platform can be very heterogeneous and shared

with other tasks. The system is able to adaptively

reconfigure at runtime looking for the better allocation

pattern (this is very well suited for cloud computing and “on

demand resources”)
13
ARTÌS + GAIA
As part of our research effort we have implemented a PADS middleware
called ARTÌS in which the GAIA module implements the migration
mechanism previously described

14
Wireless network simulation
The migration-based approach works very well for the simulation of
wireless networks

This is due to the nature of the wireless propagation and the

locality of most wireless transmissions

In the last years we have tested the ARTÌS middleware in many case
studies and simulating wireless network with different degrees of
detail in the simulated model

In all cases the results have confirmed that it is possible to obtain a

very good speed-up also using simple and general heuristics

In this way, massively populated networks (e.g. 1 million of

nodes) using detailed MAC protocols (e.g. IEEE 802.11) can be
simulated using Commodity-Off-The-Shelf (COTS) hardware

15
A more challenging scenario: scale-Free networks

A graph can be used to represent a

network and its connectivity

Degree of a node = number of neighbor

nodes attached to them

A scale-free network is a network whose

degree distribution follows a power law

If pk is the probability that a node has a degree equal to k then:

pk ~ k-α, for some constant value α

(usually: 2 < α < 3)

16
Scale-Free networks: meaning and examples
This means:

a few highly connected nodes, called hubs

a very large number of poorly connected nodes

a very small network diameter

Quite good to model several types of real networks:

computer networks (e.g. Web, Internet (?))

evolving networks in biology

transmission of diseases

citation graphs

social networks, etc.

17
Simulation of scale-free networks

Real networks are usually composed of a huge number of nodes

Under the simulation point of view:

large amount of memory used to represent the massively

populated models

huge amount of communication when simulating real-world

protocols on top of such models

nodes in real-world networks are very heterogeneous (i.e. hubs

leaf nodes, different hardware and software characteristics)

Lack of powerful tools to simulate such complex networks

18
Gossiping protocols: fixed probability

Gossiping protocols have been implemented on top of the simulated

scale-free networks, to build realistic case-studies

Algorithm
Gossip #1: Fixed Probability
function INITIALIZATION()

v ← CHOOSE_PROBABILITY()

For each message, the node

function GOSSIP(msg)
randomly selects those edges
for all nj in Πj do
through which the message
if RANDOM() < v then
must be propagated
SEND(msg, nj)
(v = threshold value)
end if

end for

19
Gossiping protocols: fixed fanout

Algorithm
Gossip #2: Fixed Fanout function INITIALIZATION()

fanout ← RETRIEVE_FANOUT()

Each message is sent to a fixed

function GOSSIP(msg)
number of nodes (fanout), the
if fanout ≥ |Πj| then
receivers are selected at
toSend ← Πj
random among the neighbors
else

SELECT_NODES()

end if

for all nj in toSend do

SEND(msg, nj)

end for
20
Gossiping protocols: probabilistic broadcast

Algorithm

Gossip #3: Probabilistic Broadcast function INITIALIZATION()

pb ← PROBABILITY_BROADCAST()

If the message is locally generated

function GOSSIP(msg)
then it is always broadcasted to all
if (RANDOM() < pb or
neighbors, otherwise it is randomly
FIRST_TRANSMISSION()) then
decided if it has to be broadcasted or
for all nj in Πj do
ignored
SEND(msg, nj)
(pb = probability of dissemination)
end for

end if

21
PaScaS: the scale-free network simulator

Parallel and distributed Scale-free network Simulator (PaScaS)

Implements the building of the scale-free networks (based on the

preferential attachment algorithm), the gossiping protocols

and some data analysis utilities

Based on the Advanced RTI System (ARTÌS), a middleware

used to implement sequential/parallel/distributed simulations that

follows an event-based approach

It can exploit the adaptive simulation features provided by the

Generic Adaptive Interaction Architecture (GAIA)

PaScaS will be soon freely available: http://pads.cs.unibo.it

22
Experimental evaluation: model parameters

Model parameters and simulation scenario

Parameter Value

number of nodes 3000, 6000, 9000, 12000

message generation
cache size (local to each node)
message Time To Live (TTL)
probability of dissemination (v)
fanout value
probability of broadcast (pb)
simulated time
exponential distribution
mean = 50 time-steps
10 slots
6 (fixed prob. and fanout)
4 (conditional broadcast)
0.5 (i.e. 50%)
5
0.5 (i.e. 50%)
1000 time-steps (after building)

23
Experimental evaluation: execution architecture

Hardware:

Dual processor unit, each processor equipped with a

single-core Intel Xeon “Gallatin” CPU 2.80 GHz Hyper-

Threading, 2 GB RAM

Software:

Debian GNU/Linux, Kernel 2.6.16.19

PaScaS 0.9, ARTÌS 1.9.3

Note: in case of monolithic (sequential) execution, PaScaS is forced

to use only a single processor (also in presence of multiprocessors or

multi-cores)

24
Experimental evaluation: sequential execution

A single process is responsible to manage the whole simulation. The fixed

probability has computational requirements higher than other gossip protocols

25
Experimental evaluation: parallel execution

The simulation is obtained through the coordinated execution of a set

of components (Logical Processes, LPs)

Each LP manages the evolution of a part of the model and it is

usually run by a different CPU

Each node in the scale-free net is modeled as a Simulated Entity

(SE), therefore each LP manages a set of SEs

SEs are randomly allocated in the LPs

It is worth noting that the performance of a Parallel And Distributed

Simulation (PADS) is a trade-off between: a) load parallelization; b)

communication overhead in the parallel/distributed architecture

(also including synchronization)

26
Experimental evaluation: parallel execution (fixed probability)

27
Experimental evaluation: parallel execution (fixed fanout)

28
Experimental evaluation: parallel execution (cond. broadcast)

29
Experimental evaluation: monolithic vs. parallel execution

Performance gap (%) between LP=1 (monolithic) and LP=4

Nodes Gossip #1 Gossip #2 Gossip #3
3000 3.46 -12.22 -9.1
6000 0.19 -4.49 -6.23
9000 -5.35 -0.63 -3.36
12000 -9.07 -0.25 -2.42

green = parallel faster than monolithic

The parallel execution gives very unsatisfactory results

The communication overhead introduced by the parallel execution

does not balance the gain given by the load parallelization

30
Experimental evaluation: adaptive parallel execution

31
Experimental evaluation: adaptive parallel execution

32
Experimental evaluation: adaptive parallel execution

33
Parallel execution vs. adaptive parallel execution

Performance gap (%) between LP=1 and LP=4 GAIA ON

Nodes Gossip #1 Gossip #2 Gossip #3

3000 34.18 -1.33 19.75
6000 38.63 6.87 23.37
9000 30.97 11.07 24.59
12000 26.47 9.65 22.56

green = parallel faster than monolithic

The results obtained by adaptive parallel execution are quite good

The best performances are obtained for gossip #1 and gossip #3,

because both protocols are communication-bounded

34
Putting it all together

Parallel approach gives unsatisfactory results in the simulation of

such complex networks

An approach based on dynamic and adaptive clustering of the

simulated entities can give valuable results

From the simulator to the simulation model: are gossip

protocols a good choice to build dissemination mechanisms on

scale-free networks?

35
MOGs: scalability and responsiveness

Scalability and responsiveness are open problems in

Multiplayer Online Games (MOGs)

Several architectures have been proposed to support MOGs:

client/server

mirrored servers

peer-to-peer

Given the distributed nature of this kind of applications, the

dissemination of game events can be very costly

Under the scalability viewpoint the peer-to-peer approach

is very promising

36
MOGs: a peer-to-peer approach
 Each peer locally manages its copy of the game state

 The peers are organized in some form of overlay network

 The dissemination of game events is obtained by passing

messages through the overlay

 What is the best form of overlay?

 tree

 fully connected graph

 random graph

 scale-free network (?)

37
Scale-free networks for MOGs

The idea is to implement MOGs using a peer-to-peer

architecture that is partially “unstructured” and “spontaneous”

Some properties of scale-free networks (e.g. the very small

diameter: d ~ ln ln n, with a network of n nodes) are very
valuable in supporting scalability and responsiveness

In our view, the dissemination of game events will be

obtained through probabilistic approaches, for example using
gossip protocols

The game events generated at peers are disseminated to the

whole network, using very simple gossip protocols and
without any form of centralization or predefined routing

38
Performance evaluation: simulation-based

Parameter Value
number of nodes 0-500
message generation exponential distribution
mean = 50 time-steps
cache size (local to each node) 256 slots
message Time To Live (ttl) 6
probability of dissemination (v) 0.5, 0.8, 1 (i.e. 50-80-100%)
fanout value 5 (# of nodes)
probability of broadcast (pb) 0.5, 0.8, 1 (i.e. 50-80-100%)
simulated time (gaming time) 1000 time-steps (after building)

39
Performance evaluation: metrics

Coverage

percentage of nodes that have received all the messages

that have been produced during the whole game execution
“are the game events received by all gamers?”

Delay

average number of hops (that is time-steps) necessary to

receive a message after its creation
“is the data dissemination really responsive?”

Messages

total number of messages routed in the game execution

“what is the overhead due to the events dissemination?”
40
Evaluation: coverage rate (%)

41
Evaluation: number of hops

42
Evaluation: total number of messages

43
Evaluation: coverage rate (%)

44
Evaluation: number of hops

45
Evaluation: total number of messages

46
Conclusions and Future work

The low diameter of scale-free networks is very good for

fast data dissemination

Common gossip protocols are unable to disseminate the whole

event trace and their overhead is very high

Simple mechanisms such as caching of packets, ttl and

protocols tweaking are quite ineffective or with limited
impact on performances (e.g. # of routed messages)

Smarter / adaptive protocols are necessary:

push / pull approaches for data dissemination

adaptive protocols and behaviors

more information shared among network nodes

47
Further Conclusions and Future work

Adaptive mechanisms based on entities migration are very

effective in improving the scalability of simulators

The benefits are not limited to wireless models: also more

general environments (such as scale-free networks) can obtain
very satisfactory results

The next steps are:

more complex aggregation mechanisms in the migration

of model entities

the implementation of simulators based on this paradigm

and taking advantage of cloud-computing (on-demand)
resources

48
References
G. D'Angelo, M. Bracuto. Distributed Simulation of Large-Scale and Detailed Models. International Journal of Simulation and

Process Modelling 2009 - Vol. 5, No. 2. Special issue on "Parallel and Distributed Simulation", Inderscience Publishers

L. Bononi, M. Di Felice, G. D'Angelo, M. Bracuto, L. Donatiello. MoVES: A framework for parallel and distributed simulation of

wireless vehicular ad hoc networks. ComNet Journal, Special issue on "Emerging Wireless Networks: Performance Modeling and

Analysis", Volume 52, Issue 1, Elsevier

S. Ferretti, G. D'Angelo. Multiplayer Online Games over Scale-Free Networks: a Viable Solution? Workshop on DIstributed

SImulation and Online gaming (DISIO 2010)

G. D'Angelo, S. Ferretti. Simulation of Scale-Free Networks. Proceedings of 2nd ACM/ICST International Conference on

Simulation Tools and Techniques (SIMUTools 2009)

M. Bracuto, G. D'Angelo. Detailed Simulation of Large-Scale Wireless Networks. Proceedings of the 11-th ACM/IEEE

International Symposium on Distributed Simulation and Real Time Applications (DS-RT 2007)

L. Bononi, M. Bracuto, G. D'Angelo, L. Donatiello. Exploring the Effects of Hyper-threading on Parallel Simulation. Proceedings

of the 10-th ACM/IEEE International Symposium on Distributed Simulation and Real Time Applications (DS-RT 2006)

L. Bononi, M. Bracuto, G. D'Angelo, L. Donatiello. An Adaptive Load Balancing Middleware for Distributed Simulation.

Proceedings of Frontiers of High Performance Computing and Networking - ISPA 2006 Workshops

L. Bononi, G. D'Angelo, L. Donatiello. HLA-based adaptive distributed simulation of wireless mobile systems. Proceedings of

the 17th ACM/IEEE/SCS Workshop on Parallel and Distributed Simulation (PADS '03)

49
Online References, Preprints and Software

Parallel and Distributed Simulation (PADS) research group:

http://pads.cs.unibo.it

Personal homepage:

http://www.cs.unibo.it/gdangelo/pubblicazioni-eng.html

Gabriele D'Angelo

Department of Computer Science, University of Bologna

E-mail: <g.dangelo@unibo.it>

http://www.cs.unibo.it/gdangelo/
50
From Simulation to Online Gaming:

the need for adaptive solutions

Gabriele D’Angelo
<gda@cs.unibo.it>
http://www.cs.unibo.it/gdangelo/

Department of Computer Science

University of Bologna