Performance metrics

What is a performance metric?

Characteristics of good metrics

Standard processor and system metrics

Speedup and relative change

Copyright 2004 David 1

What is a performance metric?
Count
Of how many times an event occurs
Duration
Of a time interval
Size
Of some parameter
Avalue derived from these fundamental
measurements

Copyright 2004 David 2

Time-normalized metrics
Rate metrics
Normalize metric to common time basis
Transactions per second
Bytes per second
(Number of events) (time interval over which
events occurred)
Throughput

Useful for comparing measurements over

different time intervals
Copyright 2004 David 3
What makes a good metric?
Allows accurate and detailed comparisons
Leads to correct conclusions

Is well understood by everyone

Has a quantitative basis

A good metric helps avoid erroneous

conclusions

Copyright 2004 David 4

Good metrics are
Linear
Fits with our intuition
If metric increases 2x, performance should
increase 2x
Not an absolute requirement, but very appealing
dB scale to measure sound is nonlinear

Copyright 2004 David 5

Good metrics are
Reliable
If metric A > metric B
Then, Performance A > Performance B
Seems obvious!
However,
MIPS(A) > MIPS(B), but
Execution time (A) > Execution time (B)

Copyright 2004 David 6

Good metrics are
Repeatable
Same value is measured each time an
experiment is performed
Must be deterministic
Seems obvious, but not always true
E.g. Time-sharing changes measured execution time

Copyright 2004 David 7

Good metrics are
Easy to use
No one will use it if it is hard to measure
Hard to measure/derive
less likely to be measured correctly
A wrong value is worse than a bad metric!

Copyright 2004 David 8

Good metrics are
Consistent
Units and definition are constant across systems
Seems obvious, but often not true
E.g. MIPS, MFLOPS
Inconsistent impossible to make comparisons

Copyright 2004 David 9

Good metrics are
Independent
A lot of $$$ riding on performance results
Pressure on manufacturers to optimize for a
particular metric
Pressure to influence definition of a metric
But a good metric is independent of this pressure

Copyright 2004 David 10

Good metrics are
Linear -- nice, but not necessary
Reliable -- required

Repeatable -- required

Easy to use -- nice, but not necessary

Consistent -- required

Independent -- required

Copyright 2004 David 11

Clock rate
Faster clock == higher performance
1 GHz processor always better than 2 GHz
Butis it a proportional increase?
What about architectural differences?
Actual operations performed per cycle
Clocks per instruction (CPI)
Penalty on branches due to pipeline depth
What if the processor is not the bottleneck?
Memory and I/O delays

Copyright 2004 David 12

Clock rate
(Faster clock) Linear
(better performance)
Reliable
A good first-order metric
Repeatable
Easy to
measure
Consistent
Independent

Copyright 2004 David 13

MIPS
Measure of computation speed
Millions of instructions executed per second

MIPS = n / (Te * 1000000)

n = number of instructions
Te = execution time
Physical analog
Distance traveled per unit time

Copyright 2004 David 14

MIPS
But how much actual Linear
computation per instruction?
E.g. CISC vs. RISC Reliable
Clocks per instruction (CPI)
Repeatable
MIPS = Meaningless Indicator
Easy to
of Performance measure
Consistent

Independent

Copyright 2004 David 15

MFLOPS
Better definition of distance traveled
1 unit of computation (~distance) 1 floating-
point operation
Millions of floating-point ops per second

MFLOPS = f / (Te * 1000000)

f = number of floating-point instructions
Te = execution time
GFLOPS, TFLOPS,

Copyright 2004 David 16

MFLOPS
Integer program = 0 MFLOPS Linear
But still doing useful work
Reliable
Sorting, searching, etc.
How to count a FLOP? Repeatable
E.g. transcendental ops, roots
Easy to
Too much flexibility in measure
definition of a FLOP
Consistent
Not consistent across
machines Independent

Copyright 2004 David 17

SPEC
System Performance Evaluation Coop
Computer manufacturers select
representative programs for benchmark
suite
Standardized methodology
1. Measure execution times
2. Normalize to standard basis machine
3. SPECmark = geometric mean of normalized
values
Copyright 2004 David 18
SPEC
Geometric mean is Linear
inappropriate (more later)
Reliable
SPEC rating does not
correspond to execution times Repeatable
of non-SPEC programs
Easy to
Subject to tinkering measure
Committee determines which
programs should be part of the Consistent
suite
Independent
Targeted compiler optimizations

Copyright 2004 David 19

QUIPS
Developed as part of HINT benchmark
Instead of effort expended, use quality of
solution
Quality has rigorous mathematical definition

QUIPS = Quality Improvements Per Second

Copyright 2004 David 20

QUIPS
HINT benchmark program
Find upper and lower rational bounds for
1 (1 x)
0 (1 x)dx
Use interval subdivision technique
Divide x,y into intervals that are power of 2
Count number of squares completely above/below
curve to obtain u and l bounds
Quality = 1/(u l)
Copyright 2004 David 21
 QUIPS

From: http://hint.byu.edu/documentation/Gus/HINT/ComputerPerformance.html

Copyright 2004 David 22

 QUIPS

From: http://hint.byu.edu/documentation/Gus/
HINT/ComputerPerformance.html
Copyright 2004 David 23
QUIPS
Primary focus on Linear
Floating-point ops
Memory performance Reliable
Good for predicting
performance on numerical
Repeatable
programs Easy to
But it ignores measure


I/O performance
Multiprograming
Consistent
Instruction cache Independent
Quality definition is fixed
Copyright 2004 David 24
Execution time
Ultimately interested in time required to
execute your program
Smallest total execution time == highest
performance
Compare times directly
Derive appropriate rates
Time == fundamental metric of performance
If you cant measure time, you dont know
anything

Copyright 2004 David 25

Execution time
Stopwatch measured execution time
Start_count = read_timer();
Portion of program to be measured
Stop_count = read_timer();
Elapsed_time = (stop_count start_count)
* clock_period;
Measures wall clock time
Includes I/O waits, time-sharing, OS overhead,
CPU time -- include only processor time

Copyright 2004 David 26

Execution time
Best to report both wall clock Linear
and CPU times
Includes system noise effects
Reliable
Background OS tasks


Repeatable
Virtual to physical page mapping
Easy to
Random cache mapping and measure
replacement
Variable system load Consistent
Report both mean and
variance (more later)
Independent

Copyright 2004 David 27

Performance metrics summary
Clock MIPS MFLOPS SPEC QUIPS TIME

Linear
Reliable
Repeatable
Easy to
measure
Consistent
Independent
Copyright 2004 David 28
Other metrics
Response time
Elapsed time from request to response
Throughput
Jobs, operations completed per unit time
E.g. video frames per second
Bandwidth
Bits per second
Ad hoc metrics
Defined for a specific need

Copyright 2004 David 29

Means vs. ends metrics
Means-based metrics
Measure what was done
Whether or not it was useful!
Nop instructions, multiply by zero,
Produces unreliable metrics
Ends-based metrics
Measures progress towards a goal
Only counts what is actually accomplished

Copyright 2004 David 30

Means vs. ends metrics

Means-based Ends-based

Execution time
Clock rate

MFLOPS

QUIPS
SPEC
MIPS

Copyright 2004 David 31

Speedup
Speedup of System 1 w.r.t System 2
S2,1 such that: R2 = S2,1 R1
R1 = speed of System 1
R2 = speed of System 2
System 2 is S2,1 times faster than System 1

Copyright 2004 David 32

Speedup
Speed is a rate metric
Ri = Di / Ti
Di ~ distance traveled by System i
Run the same benchmark program on both
systems
D 1 = D2 = D
Totalwork done by each system is defined to
be execution of the same program
Independent of architectural differences

Copyright 2004 David 33

Speedup

R2 D2 / T2 D / T2
S 2,1
R1 D1 / T1 D / T1
T1

T2

Copyright 2004 David 34

Speedup

S 2,1 1 System 2 is faster than System 1

S 2,1 1 System 2 is slower than System 1

Copyright 2004 David 35

Relative change
Performanceof System 2 relative to
System 1 as percent change

R2 R1
2,1
R1

Copyright 2004 David 36

Relative change

R2 R1 D / T2 D / T1
2,1
R1 D / T1
T1 T2

T2
S 2,1 1
Copyright 2004 David 37
Relative change

2,1 0 System 2 is faster than System 1

2,1 0 System 2 is slower than System 1

Copyright 2004 David 38

Important Points
Metrics can be
Counts
Durations
Sizes
Some combination of the above

Copyright 2004 David 39

Important Points
Good metrics are
Linear
More natural
Reliable
Useful for comparison and prediction
Repeatable
Easy to use
Consistent
Definition is the same across different systems
Independent of outside influences
Copyright 2004 David 40
Important Points

Not-so-good metric Better metrics

Execution time
Clock rate

MFLOPS

QUIPS
SPEC
MIPS

Copyright 2004 David 41

Important Points
Speedup = T2 / T1
Relative change = Speedup - 1

Copyright 2004 David 42