CT6054 Network Planning and

Simulation

Lecture 10

Simulation Outline

CT6054 Network Planning and Simulation 1

 Introduction
The best advice to those about to embark on a very large simulation is
often the same as Punch’s famous advice to those about to marry: ‘Don’t!’
– Bratley, Fox and Schrage (1986)

• System to be characterized may not be available

– During design or procurement stage
• Still want to predict performance
• Or, may have system but want to evaluate
wide-range of workloads
 Simulation
• However, simulations may fail
– Need good programming, statistical analysis and
performance and evaluation knowledge
CT6054 Network Planning and Simulation 2
 WHY Simulation?

• We need to conduct experiments "on some

reality" and the reality - although pre-existing -
is not available for our experiments.
• Examples:
• a) a busy network of computers that cannot be taken
over just for the experiment;
• b) a busy superhighway system on which we want to
"change the rules of traffic";
• c) a chemical plant whose production cannot be
stopped so that "we can tinker with it"; etc..
CT6054 Network Planning and Simulation 3
 WHY Simulation?
• What characteristic do these example share?
• They simply have to do with our lack of access to an existing
artefact: the simulation allows us to construct a useful model of
the artefact, that we can then use as though it were the
inaccessible artefact.
• The goal is to determine whether a planned change to the USE of the
artefact can be implemented while producing the desired results and
no undesired ones.
• A more specific example would be the introduction of the use of a
"group productivity package", like Lotus Notes or a Configuration
and Version Manager for a software producing organization. In
both cases the traffic patterns - and bottlenecks - in a LAN might
not be predictable without extensive testing, and any meaningful
REAL testing will result in many lost productivity hours for the
whole group or organization that is adopting the package.

CT6054 Network Planning and Simulation 4

 WHY Simulation?
• A second set of examples (with the
absence of an appropriate artefact:
a) An automobile frame that must meet certain
stiffness and crushability criteria, while also
meeting geometry, materials, production
method and weight constraints;
b) An algorithm to manage certain types of (not
yet available?) traffic in networks with as yet
non-existent (but likely, or already possible)
properties (e.g., 20 TH bandwidth);
c) The design of drugs with special properties.

CT6054 Network Planning and Simulation 5

 WHY Simulation?
• What characteristic do these example share?
• There is NO artefact on which to perform
experiments, and the construction of any such
artefact is not feasible (e.g. too expensive because
current technology is too immature or too
dangerous) without knowledge that the finished
artefact will behave (with high probability) as
desired.
• There MAY exist earlier versions of similar artefacts,
with different characteristics, that MIGHT be used
as guides for the design of a simulation, but with
no guarantee that the results of the simulation can
be compared to "real" data in the regions of
interest.
CT6054 Network Planning and Simulation 6
 Time
•
• Time
physical system: the actual or imagined system being modeled
simulation: a system that emulates the behavior of a physical system
main()
{ ...
double clock;
...

physical system simulation

• physical time: time in the physical system
– 11:00, November 17, 2008 to 13:00, November 17, 2008
• simulation time: representation of physical time within the simulation
– floating point values in interval [0.0, 24.0]
• wallclock time: time during the execution of the simulation, usually output from a
hardware clock
– 9:00 to 9:15 AM on November 17, 2008

CT6054 Network Planning and Simulation 7

 Summary: Advantages of Simulation
a) it permits controlled experimentation
• you KNOW what parameters are being changed.
b) it permits time compression
• e.g., weather forecasting...
c) it permits sensitivity analysis (change input variables)
d) it does not disturb the real system
• which may not even exist, anyway.
e) it is an effective training tool
• you are not likely to crash a flight simulator,
• or a big chunk of the Internet

CT6054 Network Planning and Simulation 8

 The Stages of a Simulation
Project (1 of 3)
• Planning
a) Problem Formulation: what is it and what do I want
to do with it?
b) Resource Estimation: time, people and money.
c) System and Data Analysis
• Modeling
a) Model Building: find relationships.
b) Data Acquisition: find and collect appropriate data.
c) Model Translation: program and debug.

CT6054 Network Planning and Simulation 9

 The Stages of a Simulation
Project (2 of 3)
• Verification/Validation
a) Verification: does the PROGRAM execute as
intended?
b) Validation: does the PROGRAM represent reality as
intended?
• Application
a) Experimentation: run it!
b) Analysis: how do I analyze and interpret the results?
c) Implementation/Documentation: how do I
implement the decisions resulting from the simulation,
and how do I document the model and its use?

CT6054 Network Planning and Simulation 10

 The Stages of a Simulation
Project (3 of 3)
• Performance Measures.
What is it that we are measuring?
What (statistical) properties of the "measured" are
we interested in?
• For example:
• maximum, minimum, totals, mean, variance, higher
moments, specific frequency distribution, interarrival times,
service times, lengths of queues, loss rates, error rates,
etc.

CT6054 Network Planning and Simulation 11

 Common Mistakes in
Simulation (1 of 4)
• Inappropriate level of detail
– Level of detail often potentially unlimited
– But more detail requires more time to develop
• And often to run!
– Can introduce more bugs, making more inaccurate not less!
– Often, more detailed viewed as “better” but may not be the
case
• More detail requires more knowledge of input parameters
• Getting input parameters wrong may lead to more inaccuracy
(Ex: disk service times exponential vs. simulating sector and
arm movement)
– Start with less detail, study sensitivities and introduce detail
in high impact areas
CT6054 Network Planning and Simulation 12
 Common Mistakes in
Simulation (1 of 4)
• Improper language
– Choice of language can have significant impact on time
to develop
– Special-purpose languages can make implementation,
verification and analysis easier
– C++Sim (http://cxxsim.ncl.ac.uk/), JavaSim (
http://javasim.ncl.ac.uk/), SimPy(thon) (
http://simpy.sourceforge.net/) …
• Unverified models
– Simulations generally large computer programs
– Unless special steps taken, bugs or errors

CT6054 Network Planning and Simulation 13

 Common Mistakes in
Simulation (3 of 4)
• Invalid models
– No errors, but does not represent real system
– Need to validate models by analytic, measurement or
intuition
• Improperly handled initial conditions
– Often, initial trajectory not representative of steady
state
• Including can lead to inaccurate results
– Typically want to discard, but need method to do so
effectively

CT6054 Network Planning and Simulation 14

 Common Mistakes in
Simulation (4 of 4)
• Too short simulation runs
– Attempt to save time
– Makes even more dependent upon initial conditions
– Correct length depends upon the accuracy desired
(confidence intervals)
• Poor random number generators and seeds
– “Home grown” are often not random enough
• Makes artifacts
– Best to use well-known one
– Choose seeds that are different

CT6054 Network Planning and Simulation 15

 More Causes of Failure (1 of 2)
Any given program, when running, is obsolete. If a program is useful, it will
have to be changed. Program complexity grows until it exceeds the capacity
of the programmer who must maintain it. - Datamation 1968

Adding manpower to a late software project makes it later.

- Fred Brooks

• Large software
– Quotations above apply to software development
projects, including simulations
– If large simulation efforts not managed properly, can
fail
• Inadequate time estimate
– Need time for validation and verification
– Time needed can often grow as more details added
CT6054 Network Planning and Simulation 16
 More Causes of Failure (2 of 2)
• No achievable goals
– Common example is “model X”
• But there are many levels of detail for X
– Goals: Specific, Measurable, Achievable, Repeatable,
Through (SMART)
– Project without goals continues indefinitely
• Incomplete mix of essential skills
– Team needs one or more individuals with certain
skills
– Need: leadership, modeling and statistics,
programming, knowledge of modeled system

CT6054 Network Planning and Simulation 17

Simulation Checklist (1 of 2)
• Checks before developing simulation
– Is the goal properly specified?
– Is detail in model appropriate for goal?
– Does team include right mix (leader, modeling,
programming, background)?
– Has sufficient time been planned?
• Checks during simulation development
– Is random number random?
– Is model reviewed regularly?
– Is model documented?

CT6054 Network Planning and Simulation 18

 Simulation Checklist (2 of 2)
• Checks after simulation is running
– Is simulation length appropriate?
– Are initial transients removed?
– Has model been verified?
– Has model been validated?
– Are there any surprising results? If yes,
have they been validated?

CT6054 Network Planning and Simulation 19

 Selecting a Simulation Tool (1
of 2)
• Four choices:
• general-purpose language
• extension of general purpose language
• simulation language
• simulation package
• General-purpose language – known to developer, available on
more systems, flexible
• Simulation language – built in facilities for time steps, event
scheduling, data collection, reporting
• The major difference is the cost tradeoff – simulation language
requires startup time to learn, while general purpose may
require more time to add simulation flexibility
– Recommendation may be for all analysts to learn one simulation
language so understand those “costs” and can compare

CT6054 Network Planning and Simulation 20

Selecting a Simulation Language (2 of
2)
• Extension of general-purpose – collection of routines and tasks
commonly used. Often, base language with extra libraries that
can be called
• Simulation packages – allow definition of model in interactive
fashion. Get results in one day
– Tradeoff is in flexibility, where packages can only do what developer
envisioned, but if that is what is needed then is quicker to do so

CT6054 Network Planning and Simulation 21