PERFORMANCE EVALUATION

 Often in Computer Science you need to:

 demonstrate that a new concept, technique, or
algorithm is feasible
 demonstrate that a new method is better than an
existing method
 understand the impact of various factors and
parameters on the performance, scalability, or
robustness of a system

1
Performance Evaluation
 Performance evaluation is the application of the scientific
method to the study of computer systems.
 Viewed as distinct from computer system design, the goal
of performance evaluation is to determine the
effectiveness and fairness of a computer system that is
assumed to work correctly.
 Performance evaluation techniques have been developed to
accurately measure the effectiveness with which computer
system resources are managed while striving to provide
service that is fair to all customer classes.
PERFORMANCE EVALUATION

 There is a whole field of computer science called

computer systems performance evaluation that is devoted
to exactly this
 One classic book is Raj Jain’s “The Art of Computer
Systems Performance Analysis”, Wiley & Sons, 1991
 Much of what is outlined in this presentation is described
in more detail in [Jain 1991] The art of computer systems
performance analysis.

3
 Network Performance Metrics

 Metric :: a descriptor used to represent some aspect of a

computer network’s performance.
 The goal is objective performance indices.
 For computer networks, metrics can capture performance at
multiple layers of the protocol stack, e.g.,
 UDP throughput
 IP packet round trip time
 MAC layer channel utilization
 Performance metrics can be positive and negative.
 e.g., goodput, packet loss rate, MAC layer retries

4
Wide Area Network (WAN)
Host
Host M
Host N Host
A L

2 3 Host
4 J
1
Host 5
B 16
routers 14
11
Host 12 17
C 15 6

10 13
7
9 Host
8 H
Host Host
D G
Host
E
Host
F

5
Wireless Local Area Network
(WLAN)

Server

Clients

AP

6
 Sample Performance Measures

Category Metric Units

productivity throughput Mbps

effective capacity
responsiveness delay milliseconds
round trip time
queue size packets
utilization channel utilization percentage of
time busy
losses packet loss rate loss percentage
frame retries
buffer problems AP queue overflow packet drops
playout buffer underflow rebuffer events
7
Wide Area Network (WAN)
Host
Host M
Host N Host
A L

2 3 Host
4 J
1
Host 5
B 16
nodes 14
11
Host 12 17
C 15 6

10 13
7
9 Host
8 H
Host Host
D G
Host
E
Host
F

8
 Local Area Network (LAN)

A C X Z

B Y

9
Wireless Local Area Network
(WLAN)

Server

Client

AP

10
PERF EVAL: THE BASICS
 There are three main methods/techniques used in the
design of performance evaluation studies:
 Analytic approaches
 the use of mathematics, Markov chains, queueing theory,
Petri Nets, abstract models…
 Simulation approaches
 design and use of computer simulations and simplified
models to assess performance
 Experimental approaches (workload characterization)
 measurement and use of a real system

11
 Performance Evaluation Techniques
 Workload characterization for computer networks
involves the design and choice of traffic types that provide the
inputs for computer network performance evaluation.
 Performance measures of computer networks are all
dependent to some extent on the input workload, the
network topology and the choices in controlled parameters
or network default settings.
 An evaluation study of a computer network seeks to determine
the values for network performance indices under a given
traffic workload and network configuration.

12
 Typical Network Traffic Types
 Web Traffic between a Browser and an Internet
Server.
 Long-Lived File Transfers
 FTP downloads.
 Multimedia Streaming
 Video clip downloads (UDP and/or TCP)
 Audio VOIP (Voice Over IP)
 Peer-to-Peer Exchanges
 Concurrent downloads and uploads
 Telnet file edits

13
Wireless Local Area Network
(WLAN)

Server

Client

AP

14
Performance Evaluation Techniques
Network evaluation utilizes the actual network, an emulated network or a model of
the network.

 Models
 Simulation Modeling
 Analytic Modeling
 Both modeling techniques tend to rely on queuing theory.
 Measurement Studies
 Empirical measurement of real networks
 Measurements where some aspect of the network architecture or topology is
emulated via software or hardware.
The primary focus of this presentation is on the design and techniques
used in experiments to measure real computer networks.

15
Conceptual Models
 Researchers utilize knowledge about the interactions
of network components to understand and explain the
workings of a computer network via a conceptual
model.
 Models are partitioned into simulation models or
analytic models. Both model types rely on simplifying
assumptions that enable the model to capture
important characteristics of networks (usually in terms
of networks of queues).

16
Analytical Example: Queueing Theory

 Queueing theory is a mathematical technique that

specializes in the analysis of queues (e.g., customer
arrivals at a bank, jobs arriving at CPU, I/O requests
arriving at a disk subsystem, lineup at Tim Hortons)
 General diagram:

Customer
Arrivals Departures
Buffer Server
17
Queueing Theory (cont’d)
 The queueing system is characterized by:
 Arrival process (M, G)
 Service time process (M, D, G)
 Number of servers (1 to infinity)
 Number of buffers (infinite or finite)
 Example notation: M/M/1, M/D/1
 Example notation: M/M/ , M/G/1/k
8

18
Queueing Theory (cont’d)
 There are well-known mathematical results for the mean
waiting time and the number of customers in the system
for several simple queueing models
 E.g., M/M/1, M/D/1, M/G/1
 Example: M/M/1
 q = rho/ (1 - rho) where rho = lambda/mu < 1

19
Queueing Theory (cont’d)
 These simple models can be cascaded in series and in
parallel to create arbitrarily large complicated
queueing network models
 Two main types:
 closed queueing network model (finite pop.)
 open queueing network model (infinite pop.)
 Software packages exist for solving these types of
models to determine steady-state performance (e.g.,
delay, throughput, util.)

20
Simulation Models
 Simulation attempts to reproduce the behavior of the
network in the time domain.
 Event-driven simulation defines a network in terms of
states and transitions where events trigger transitions.
 Simulation is essentially a numeric solution that
utilizes systems of equations and data structures to
capture the behavior of the simulated network in
terms of logical conditions.

21
Simulation Models
 The three types of simulators are:
 Trace-driven
 Program-driven
 Distribution-driven
 The choice of the duration of a simulation run is subject to
the same issues of estimating variance and variance
reduction as found in the design of empirical
measurements.

22
Simulation Example: TCP Throughput
 Can use an existing simulation tool, or design and build
your own custom simulator
 Example: ns-2 network simulator
 A discrete-event simulator with detailed TCP protocol
models
 Configure network topology and workload
 Run simulation using pseudo-random numbers and
produce statistical output

23
OTHER ISSUES
 Simulation run length
 choosing a long enough run time to get statistically
meaningful results (equilibrium)
 Simulation start-up effects and end effects
 deciding how much to “chop off” at the start and end of
simulations to get proper results
 Replications
 ensure repeatability of results, and gain greater statistical
confidence in the results given

24
Experimental Example: Benchmarking
 The design of a performance study requires great care in
experimental design and methodology
 Need to identify
 experimental factors to be tested
 levels (settings) for these factors
 performance metrics to be used
 experimental design to be used

25
FACTORS

 Factors are the main “components” that are varied in an

experiment, in order to understand their impact on
performance
 Examples: request rate, request size, read/write ratio,
num concurrent clients
 Need to choose factors properly, since the number of
factors affects size of study

26
LEVELS
 Levels are the precise settings of the factors that are to be
used in an experiment
 Examples: req size S = 1 KB, 10 KB, 1 MB
 Example: num clients C = 10, 20, 30, 40, 50, 100,..
 Need to choose levels realistically
 Need to cover useful portion of the design space

27
PERFORMANCE METRICS

 Performance metrics specify what you want to measure in

your performance study
 Examples: response time, throughput, pkt loss
 Must choose your metrics properly and instrument your
experiment accordingly

28
EXPERIMENTAL DESIGN
 Experimental design refers to the organizational structure
of your experiment
 Need to methodically go through factors and levels to get
the full range of experimental results desired
 There are several “classical” approaches to experimental
design

29
EXAMPLES

 One factor at a time

 vary only one factor through its levels to see what the
impact is on performance
 Two factors at a time
 vary two factors to see not only their individual effects, but
also their interaction effects, if any
 Full factorial
 try every possible combination of factors and levels to see
full range of performance results

30
 Empirical Measurement Studies
The planning phase objectives of an empirical measurement are:

1. To decide what to measure.

2. To choose the measurement tools
3. To design the experiments.

Network measurements can be either active or passive.

 Active measurement involves purposely adding traffic to the network

workload specifically to facilitate the measurement (e.g., sending
packet pair probes into the network to estimate the available bandwidth
along a flow path).
 An example of a passive measurement tool is a network sniffer
running in promiscuous mode to collect information about all packets
traversing a network channel.

31
 What to Measure?
 The overall objective of the computer network measurement
study guides the choice of performance indices to be
measured.
 Metrics are either direct or indirect indices. Indirect indices
require some type of data reduction process to determine
metric values.
 Due to the large data volume associated with network traffic,
measurement of computer networks often involves filtering
of data or events (e.g., It is common for network
measurement tools to only retain packet headers for off-
line analysis).
 When the measurement strategy involves probabilistic
sampling, the duration of the experiments is determined
using confidence interval techniques.

32
 Network Measurement Tools
 While hardware probes provide the best quality
measurements, they are expensive and not always
available.
 The availability of software tools for computer
networks depends on the ability to get inside the
components of the network protocol stack and the
ability to access nodes of the network topology.
 Network software measurement tools provide ‘hooks’
within the network layering software to capture and
store network measurement data.

33
Choice of Measurement Tools
Key issues in the usability of network measurement
tools are:
1. Tool location
2. Interference or bias introduced by the tool.
3. Accuracy of the tool.
4. Tool resolution
- This has become a problem with respect to the granularity of
system clocks relative to the speed of modern high speed
network links.

34
Wireless Local Area Network
(WLAN)

Server

Clients

AP

23rd March
35
 The Design of Measurement
Experiments
Measurement Experiments are divided into two major categories:

1. Live measurements

 With live empirical studies, the objective is to measure the performance of

the computer network while it is handling real traffic.
 The advantage of this type of study is that the measurement involves a real
workload.
 One disadvantage of measuring live traffic is being convinced that this
measurement involves ‘typical’ traffic for this network.
 Another disadvantage of live traffic measurement is that reproducibility
of the exact same traffic workload is usually not possible. This is
problematic when the goal is to evaluate the impact of changing network
components on overall performance.

36
 The Design of Measurement
Experiments
2. Controlled-traffic measurements

 Traffic generator tools or traffic script files provide repeatable, controlled

traffic workloads on the network being measured.
 Controlled-traffic workloads are chosen when the goal of the performance
study is to evaluate the impact of different versions of a network
component, strategy or algorithm on network performance.
 Controlled, repeatable traffic makes it easier to conduct cause-and-
effect performance analysis.
 One difficulty with controlled-traffic is being confident in the accuracy of
the traffic generator tool and the ability to conduct measurement
experiments where the traffic workload choices are adequately varied to
provide representative, robust network performance evaluation.

37
 Measurement Design Decisions
 Understanding which network components (or
independent variables) significantly impact network
performance.
 Deciding which network parameters are to be
controlled and/or held fixed during experimental runs.
 How long to run a single experiment?
 How many times to repeat an experiment?

38
39
Throughput (Mbps)

Time (sec)
 RSSI (dB)

Time (sec)

40
 Measurement Design Decisions
 When to run experiments?
 Namely, to determine whether time of day or other temporal
periods influence performance measurements.
 How to control, minimize and/or understand physical
phenomenon or other interference sources that can
produce discrepancies and variability in the measurement
results?

41
42
Throughput (Mbps)

Time (sec)
 RSSI (dB)

Time (sec)

43
 Measurement Design Decisions
 What data filters to use?
 How and where to store experimental results?
 Determining the best choices of graphical and tabular
forms of data representation to facilitate network
performance analysis while providing a clear view of
the results of the computer network performance
evaluation.

44
45
MAC Layer Retries

Time (sec)
Cumulative Distribution Function
(CDF)

46
Coming Attractions

Discussions on:

 The Scientific Method applied to Computer Networks

 Statistical Techniques used in Experimental
Measurement Design

47
SUMMARY
 Computer systems performance evaluation defines
standard methods for designing and conducting
performance studies
 Great care must be taken in experimental design and
methodology if the experiment is to achieve its goal,
and if results are to be fully understood
 We will see examples of these methodologies and their
applications over the rest of the sem

48