Computer Performance:

Metrics, Measurement, & Evaluation

1
OUTLINE

• Defining computer performance

• Performance metrics
• Measuring Performance
• Amdahl’ Law

2
Defining Computer
performance

3
What is Performance?

➢ As a computer user, YOU define the performance!

➢ Different users have different definitions of performance

➢ Some known performance metrics

Clock Frequency:
My computer runs at 500MHz, yours only at 300MHz, so my computer is
better than yours!

➢ Millions of Instructions Per Second (MIPS):

In a second, my computer can execute 8 million instructions, while yours
only 5 million instructions, so my computer is better than yours

4
Example: Which Airplane is “best”?
Airplane Passengers Range (mi) Speed (mph)
Boeing 737-100 132 630 598
Boeing 747 470 4150 610
BAC/Sud Concorde 101 4000 1350
Douglas DC-8-50 146 8720 544 Which Airplane
performs better?
• How much faster is the Concorde than the 747?
– 2.213 times faster
• How much larger is the 747’s capacity than the Concorde?
– 4.65 larger
• It is roughly 4000 miles from Raleigh to Paris. What is the throughput of
the 747 in passengers/hr? The Concorde?
610 1350
470  = 71.675 passengers/hr 101 = 34.0875 passengers/hr
4000 4000
• What is the latency of the 747? The Concorde?
– 4000/610 = 6.56 hours ; 4000/1350 = 2.96 hours
 5
Who is interested in Computer Performance?

• Everyone who specifies or buys a computer should be interested in

performance because they need to be able to compare competing
machines in order to make an informed choice. Computer users also need
to understand computer performance in order to get the best results from
their machine.
• Each type of computer user may view computer performance in a different
way
➢ The Computer Designer – they create computer, hence the computer need to
be better than the existing computer
➢ The Corporate User – bank, government agencies (and etc). Very large sums
of money are involved in such transactions and it is important to make an
informed decision.
➢ The End User - wide range of computer abilities. Each type of user is interested
in performance in their own way

6
What is PERFORMANCE?

• Performance is an indication of how rapidly a computer performs a

task. Example:
▪ computer A performs a task in tA seconds
▪ computer B performs the (same) task in tB seconds.
▪ The performance of computer A relative to computer B is defined
as tB/tA.
▪ Suppose computer A performs a task in 48 seconds and
computer B performs the same task in 64 seconds, then the
performance of computer A relative to computer B is 64/48 = 1.33.

7
What is performance? (cont…)

▪ You can also describe relative performance as a percentage using the

formula
Performance = 100 x (tB – tA)/tA
▪ In terms of the previous example we have:
▪ Performance = 100 x (64 – 48)/48 = 33.3%; that is, the performance of
computer A is 33.3% greater than B.
▪ Performance can be measured in terms of the time taken to perform a
task or the rate at which tasks are performed (i.e., the number of jobs or
programs per unit time);
▪ for example, we can say that computer P performs a task in 0.025s or
we can say that computer P performs 40 tasks/s.

8
What Does Performance Mean?

➢ Computer performance is the amount of work accomplished by a

computer system. Depending on the context, high computer
performance may involve one or more of the following:
▪ Short response time for a given piece of work
▪ High throughput (rate of processing work)
▪ Low utilization of computing resource(s)
▪ High availability of the computing system or application
▪ Fast (or highly compact) data compression and decompression
▪ High bandwidth
▪ Short data transmission time

Note:
• Response time - Elapsed time from request to response
• Throughput - Jobs, operations completed per unit time
9
Why is Performance Difficult to Measure?

• Recall comparison of airplane (parameter: speed, mileage, capacity). Drag

race car...
• Let’s look at some of the factors that affect computer performance:
– The clock rate → manufacturer A: 3GHz and manufacturer B: 3.5GHz
– The chip architecture → Different families of chips (e.g., ARM or Intel
iCore or AMD)
– The chip architecture → different organization (e.g. bus, register etc)
– Number of cores → different program might utilize or use different number
of cores
– Cache memory
– Main memory
– Secondary storage → hard drives vs SSD

10
Performance
Metrics

11
Performance Metric

• Let look at some of the ways in which the performance of computers is (or
has been) measured:

1. Clock rate
• The speed of a processor’s clock determines how fast individual
operations are carried out within the chip.
• Clock rate simply tells us how rapidly a device is clocked but
NOT how many operations are carried out at each clock pulse or
how much computation they perform.
• Clock rates have increased dramatically.
– The clock rate of the very first processors in the 1970s was
approximately 100 kHz (105 Hz).
– The clock rate of some modern processors is 4 GHz (4 x109 Hz).
Experimental devices have been clocked as rapidly as 500 GHz (5
x1011 Hz).
http://www.intel.com/pressroom/kits/quickreffam.htm 12
Performance Metric (cont…)

2. MIPS - millions of instructions per second

• A common means of comparing processors is their MIPs rating.
• MIPS provides some idea of the work actually performed (unlike
clock rate).
• However, even MIPS is not a good indication of performance
because not all instructions perform the same amount of
computation.

3. MFLOPS - Millions of floating point operations per second

• Generally used to characterize the performance of computers
dedicated to scientific and numerical applications
• MFLOPS suffers from the same limitations as MIPS (i.e., it
measures the number of operations and not the amount of work
performed); however, MFLOPS does at least indicate the amount
of useful calculations performed by the computer
13
 Measuring
Performance

14
Computer Performance
➢ Response Time (latency)
✓ How long does it take for my job to run?
✓ How long does it take to execute a job?
✓ How long must I wait for the database query?

➢ Throughput
✓ How many jobs can the machine run at once?
✓ What is the average execution rate?
✓ How much work is getting done?

➢ If we upgrade a machine with a new processor what do we increase?

• Latency

➢ If we add a new machine to the lab what do we increase?

• Throughput

15
Why measure performance?
➢ Make intelligent choices
➢ See through the marketing hype
➢ Key to understanding underlying organizational motivation

➢ We are interested in answering questions like:

Why is some hardware better than others for different programs?

What factors of system performance are hardware related? (e.g., Do we

need a new machine, or a new OS?)

How does the machine's instruction set affect performance?

16
Response Time and Throughput

• Response time
– How long it takes to do a task

• Throughput
– Total work done per unit time
• e.g., tasks/transactions/… per hour

• How are response time and throughput affected by

– Replacing the processor with a faster version?
– Adding more processors?

• We’ll focus on response time for now…

17
Relative Performance

• Define Performance = 1/Execution Time

• “X is n time faster than Y”

Performance X Performance Y
= Execution time Y Execution time X = n

◼ Example: time taken to run a program

◼ 10s on A, 15s on B
◼ Execution TimeB / Execution TimeA
= 15s / 10s = 1.5
◼ So A is 1.5 times faster than B

18
Relative Performance

• Define Performance = 1/Execution Time

• “X is n time faster than Y”

Performance X Performance Y
= Execution time Y Execution time X = n

◼ Example: time taken to run a program

◼ 60s on A, 30s on B
◼ Execution TimeB / Execution TimeA= 30s / 60s = 0.5
So A is 0.5 times faster than B
◼ or B is 2 times faster than A

19
Measuring Execution Time

• Elapsed time
– Total response time, including all aspects
• Processing, I/O, OS overhead, idle time
– Determines system performance
• CPU time
– Time spent processing a given job
– Comprises user CPU time and system CPU time
– Different programs are affected differently by CPU and system
performance

20
CPU Clocking

• Operation of digital hardware governed by a

constant-rate clock

Clock period

Clock (cycles)

Data transfer
and computation
Update state

◼ Clock period: duration of a clock cycle

◼ e.g., 250ps = 0.25ns = 250×10–12s
◼ Clock frequency (rate): cycles per second
◼ e.g., 4.0GHz = 4000MHz = 4.0×109Hz

21
CPU Time

CPU Time = CPU Clock Cycles Clock Cycle Time

CPU Clock Cycles
=
Clock Rate
• Performance improved by
– Reducing number of clock cycles
– Increasing clock rate

22
CPU Time Example

• Computer A: 2GHz clock, 10s CPU time

• Designing Computer B
– Aim for 6s CPU time
– Can do faster clock, but causes 1.2 × clock cycles
• How fast must Computer B clock be?

Clock CyclesB 1.2  Clock CyclesA

Clock RateB = =
CPU Time B 6s
Clock CyclesA = CPU Time A  Clock Rate A
= 10s  2GHz = 20  10 9
1.2  20  10 9 24  10 9
Clock RateB = = = 4GHz
6s 6s
23
Instruction Count and CPI

Clock Cycles = Instruction Count  Cycles per Instruction

CPU Time = Instruction Count  CPI  Clock Cycle Time
Instruction Count  CPI
=
Clock Rate

• Instruction Count for a program

– Determined by program, ISA and compiler
• Average cycles per instruction
– Determined by CPU hardware
– If different instructions have different CPI
• Average CPI affected by instruction mix

24
CPI Example

• Computer A: Cycle Time = 250ps, CPI = 2.0

• Computer B: Cycle Time = 500ps, CPI = 1.2
• Which is faster, and by how much?

CPU Time = Instruction Count  CPI  Cycle Time

A A A
= I  2.0  250ps = I  500ps A is faster…
CPU Time = Instruction Count  CPI  Cycle Time
B B B
= I  1.2  500ps = I  600ps

B = I  600ps = 1.2
CPU Time
CPU Time I  500ps …by this much
A

25
CPI in More Detail

• If different instruction types take different numbers

of cycles

n
Clock Cycles =  (CPIi  Instruction Counti )
i=1

◼ Weighted average CPI

Clock Cycles n
 Instruction Counti 
CPI = =   CPIi  
Instruction Count i=1  Instruction Count 

Relative frequency

26
CPI Example

▪ Alternative compiled code sequences using

instructions in type INT, FP, MEM
Type INT FP MEM
CPI for type 1 2 3
IC in Program 1 2 1 2
IC in Program 2 4 1 1

▪ Program 1: IC = 5
◼ Clock Cycles
= 2×1 + 1×2 + 2×3
= 10
◼ Avg. CPI = 10/5 = 2.0
◼ Program 2: IC = 6
◼ Clock Cycles
= 4×1 + 1×2 + 1×3
=9
◼ Avg. CPI = 9/6 = 1.5

27
Performance Summary

The BIG Picture

Instructions Clock cycles Seconds

CPU Time =  
Program Instruction Clock cycle

• Performance depends on
– Algorithm: affects IC, possibly CPI
– Programming language: affects IC, CPI
– Compiler: affects IC, CPI
– Instruction set architecture: affects IC, CPI, Tc

28
Amdahl’s law

29
• Amdahl’s Law implies that the overall performance
improvement is limited by the range of impact when an
optimization is applied. Consider the following equations:
• Improvement rate N = (original execution time) / (new execution
time)
• New execution time = time_unaffected + time_affected /
(Improvement Factor)
• Time = (instruction count) × CPI × CCT

30
• CPI = (execution_time x clock_rate)/instructions

• (cpu_ex_time_B/cpu_ex_time_A) – formula to know which is faster

31
Amdahl’s law

• If a program currently takes 100 seconds to execute and loads and stores
account for 20% of the execution times, how long will the program take if
loads and stores are made 30% faster?
• For this, you can use Amdahl's law or you can reason it out step by step.
Doing it step by step gives (1) Before the improvement loads take 20
seconds (2) If loads and stores are made 30 percent faster they will take
20/1.3 = 15.385 seconds, which corresponds to 4.615 seconds less. (3)
Thus, the final program will take 100 - 4.615 = 95.38

32
Amdahl’s law

• Amdahl's law, named after computer architect Gene

Amdahl, is used to find the maximum expected
improvement to an overall system when only part of the
system is improved.
• Amdhal's law can be interpreted more technically, but in
simplest terms it means that it is the algorithm that decides
the speedup not the number of processors.

Wikipedia
33
Amdahl’s law

• Amdahl's law is a demonstration of the law of diminishing returns:

while one could speed up part of a computer a hundred-fold or more,
if the improvement only affects 12% of the overall task, the best the
speedup could possibly be is
1 = 1.136 times faster.
1 – 0.12

Wikipedia
34
Amdahl’s law

• More technically, the law is concerned with the speedup achievable

from an improvement to a computation that affects a proportion P of
that computation where the improvement has a speedup of S.

Wikipedia
35
Amdahl’s law

• For example, if an improvement can speedup 30% of the

computation, P will be 0.3; if the improvement makes the portion
affected twice as fast, S will be 2.
• Amdahl's law states that the overall speedup of applying the
improvement will be

Wikipedia
36
Amdahl’s law

• To see how this formula was derived, assume that the

running time of the old computation was 1, for some unit of
time. The running time of the new computation will be the
length of time the unimproved fraction takes (which is
1 − P) plus the length of time the improved fraction takes.
The length of time for the improved part of the computation
is the length of the improved part's former running time
divided by the speedup, making the length of time of the
improved part P/S.

Wikipedia
37
Amdahl’s law

• The final speedup is computed by

dividing the old running time by the
new running time, which is what the
above formula does.

Wikipedia
38
Amdahl’s law

Here's another example. We are given

a task which is split up into four
parts: P1 = .11 or 11%, P2 = .18 or
18%, P3 = .23 or 23%, P4 = .48 or
48%, which add up to 100%. Then
we say P1 is not sped up, so S1 = 1
or 100%, P2 is sped up 5x, so S2 = 5
or 500%, P3 is sped up 20x, so S3 =
20 or 2000%, and P4 is sped up
1.6x, so S4 = 1.6 or 160%.

Wikipedia
39
Amdahl’s law

By using the formula

we find the running time is

or a little less than 1/2 the original running time

which we know is 1.
Therefore the overall speed boost is

or a little more than double the original speed

using the formula
Wikipedia
40