Analysis of Algorithms
Data Structures and Algorithms
Prof. Dr. Mohamed Amine Chatti
Social Computing Group, University of Duisburg-Essen
www.uni-due.de/soco
Agenda
Introduction
Observations
Mathematical Models
Order-of-Growth Classifications
http://algs4.cs.princeton.edu Theory of Algorithms
Chapter 1.4
Data Structures and Algorithms - Analysis of Algorithms 2
Agenda
Introduction
Observations
Mathematical Models
Order-of-Growth Classifications
http://algs4.cs.princeton.edu Theory of Algorithms
Chapter 1.4
Data Structures and Algorithms - Analysis of Algorithms 3
Cast of Characters
Programmer needs to develop
a working solution
Student might play
any or all of these
roles someday
Client wants to solve
problem efficiently
Theoretician wants
to understand
Data Structures and Algorithms - Analysis of Algorithms 4
Running Time
“As soon as an Analytic Engine exists, it will necessarily guide the future
course of the science. Whenever any result is sought by its aid, the question
will arise—By what course of calculation can these results be arrived at by
the machine in the shortest time?” — Charles Babbage (1864)
How many times do you have to
turn the crank?
Analytic Engine
Data Structures and Algorithms - Analysis of Algorithms 5
Reasons to Analyze Algorithms
Predict performance
Compare algorithms this course
Provide guarantees
Understand theorical basis “Berechenbarkeit und Komplexität”
Primary practical reason: avoid performance issues
Client gets poor performance because programmer
did not understand performance characteristics
Data Structures and Algorithms - Analysis of Algorithms 6
In-class Exercise
• Assume that the running time of an algorithm is related to frequency of
execution of its operations.
• Suppose that 𝑛 equals 1 million. Approximately how much faster is an
algorithm (A) that performs 𝑛 ∙ lg 𝑛 operations versus another algorithm (B)
that performs 𝑛2 operations? Recall that lg is the base-2 logarithm function.
• (A) 20x
• (B) 1000x Compute
𝑟𝑢𝑛𝑛𝑖𝑛𝑔 𝑡𝑖𝑚𝑒 𝑜𝑓 𝐵
𝑟𝑢𝑛𝑛𝑖𝑛𝑔 𝑡𝑖𝑚𝑒 𝑜𝑓 𝐴
• (C) 50000x
• (D) 1000000x
2 minutes
Data Structures and Algorithms - Analysis of Algorithms 7
The Challenge
Why is my Why does it run
• Q: Will my program be able to solve program so slow? out of memory?
a large practical input?
• Insight
• Use scientific method to understand
performance [Knuth 1970s]
Data Structures and Algorithms - Analysis of Algorithms 8
Scientific Method Applied to Analysis of Algorithms
• A framework for predicting performance and comparing algorithms
• Scientific Method
• Observe some feature of the natural world
• Hypothesize a model that is consistent with the observations
• Predict events using the hypothesis
• Verify the predictions by making further observations
• Validate by repeating until the hypothesis and observations agree
• Principles
• Experiments must be reproducible
• Hypotheses must be falsifiable
Data Structures and Algorithms - Analysis of Algorithms 9
Agenda
Introduction
Observations
Mathematical Models
Order-of-Growth Classifications
http://algs4.cs.princeton.edu Theory of Algorithms
Chapter 1.4
Data Structures and Algorithms - Analysis of Algorithms 10
Example: 3-Sum
• Given N distinct integers, how many triples sum to exactly zero?
𝒂𝒊 𝒂𝒋 𝒂𝒌 𝒔𝒖𝒎
$ cat data/8ints.txt
30 -40 10 0
30 -40 -20 -10 40 0 10 5
30 -20 -10 0
$ python3 3sum.py data/8ints.txt
4 -40 40 0 0
-10 0 10 0
Data Structures and Algorithms - Analysis of Algorithms 11
3-Sum: Brute-Force Algorithm
import sys
How to measure
import DSA the running time of
this program?
def count(a): In Java
N = len(a)
count = 0
for i in range(0,N):
for j in range(i+1,N):
for k in range(j+1,N): Check each triple
if a[i] + a[j] + a[k] == 0:
count+=1
return count
f = DSA.In(sys.argv[1]) $ cat data/8ints.txt
30 -40 -20 -10 40 0 10 5
a = f.readAllInts()
print(count(a)) $ python3 3sum.py data/8ints.txt
4
Data Structures and Algorithms - Analysis of Algorithms 12
Measuring The Running Time
• Q. How to time a program?
• A. Manual
Data Structures and Algorithms - Analysis of Algorithms 13
Measuring The Running Time
class Stopwatch (part of DSA.py)
Stopwatch() Create a new stopwatch
float: elapsedTime() Time since creation (in seconds)
• Q. How to time a program?
• A. Automatic
f = DSA.In(sys.argv[1])
a = f.readAllInts()
s = DSA.Stopwatch()
c = count(a)
time = s.elapsedTime()
print("elapsed time:",time,"seconds")
print(c)
Data Structures and Algorithms - Analysis of Algorithms 14
Empirical Analysis
$ python3 3sum.py data/8ints.txt
elapsed time: 0.00 seconds
4
$ python3 3sum.py data/1Kints.txt
elapsed time: 18.62 seconds
• Run the program for various input 70
sizes and measure running time $ python3 3sum.py data/1Kints.txt
Elapsed time: 18.79 seconds
70
$ python3 3sum.py data/2Kints.txt
|
Data Structures and Algorithms - Analysis of Algorithms 15
Empirical Analysis
Input 𝑵 time (seconds)†
size 𝑵 𝑻 𝑵
250 0
500 0
• Run the program for various input 1000 0,1
sizes and measure running time 2000 0,8
4000 6,4
8000 51,1
16000 ?
Data Structures and Algorithms - Analysis of Algorithms 16
Data Analysis – Standard plot
Input 𝑵 time (seconds)†
size 𝑵 𝑻 𝑵
250 0
500 0
1000 0,1
2000 0,8
4000 6,4
8000 51,1
16000 ?
• Plot running time 𝑇 𝑁 vs. input size 𝑁
Data Structures and Algorithms - Analysis of Algorithms 17
Data Analysis – Log-Log Plot
log-log plot
• Plot running time 𝑇 𝑁 vs. input size 𝑁 7
Straight
using a log-log scale 6
5 line of
Input 𝑵 time (seconds)† 4 slope 3
𝒍𝒈(𝑵) 𝒍𝒈(𝑻(𝑵)) 3
size 𝑵 𝑻 𝑵
lg(T(N))
2
1000 9,966 0,1 -3,322 1
0
2000 10,966 0,8 -0,322
-1 0 5 10 15
4000 11,966 6,4 2,678 -2
-3
8000 12,966 51,1 5,675
Power law -4
lg(N)
• Regression
• Fit a straight line through data points: 𝑎 ∙ 𝑁 𝑏 slope lg 𝑇 𝑁 = 𝑏 ∙ lg 𝑁 + 𝑐
𝑏 = 2,999
• Hypothesis 𝑐 = −33,210
• The running time is about 1,006 ∙ 10−10 ∙ 𝑁 2,999
seconds 𝑇 𝑁 = 𝑎 ∙ 𝑁 𝑏 , 𝑤ℎ𝑒𝑟𝑒 𝑎 = 2𝑐
Data Structures and Algorithms - Analysis of Algorithms 18
In-class Exercise
• How to find the value of 𝑏? 7
log-log plot
• How to find the value of 𝑐? 6
5
Straight
line of
4 slope 3
3
lg(T(N))
2
1
0
Input 𝑵 time (seconds)† -1 0 5 10 15
𝒍𝒈(𝑵) 𝒍𝒈(𝑻(𝑵))
size 𝑵 𝑻 𝑵 -2
1000 9,966 0,1 -3,322 -3
-4
2000 10,966 0,8 -0,322 lg(N)
4000 11,966 6,4 2,678
8000 12,966 51,1 5,675 lg 𝑇 𝑁 = 𝑏 ∙ lg 𝑁 + 𝑐
𝑏 =?
𝑐 =?
2 minutes
Data Structures and Algorithms - Analysis of Algorithms 19
Logarithms and Exponential Rules
Logarithm Rules Exponential Rules
• lg 𝑛 = log 2 𝑛 (binary logarithm) • 𝑎 = 𝑏 log𝑏 𝑎 • 𝑎𝑚 ∙ 𝑎𝑛 = 𝑎𝑚+𝑛 ,
𝑎𝑚
• ln 𝑛 = log 𝑒 𝑛 (natural logarithm) • log 𝑐 𝑎𝑏 = log 𝑐 𝑎 + log 𝑐 𝑏 • = 𝑎𝑚−𝑛 , 𝑎≠0
𝑎𝑛
• lg 𝑘 𝑛 = lg 𝑛 𝑘
(exponential) • log 𝑏 𝑎𝑛 = 𝑛 ∙ log 𝑏 𝑎 , • 𝑎𝑏 𝑚 = 𝑎𝑚 𝑏𝑚
log 𝑎 𝑎 𝑚 𝑎𝑚
• lg lg 𝑛 = lg lg 𝑛 (composition) • log 𝑏 𝑎 = log𝑐 𝑏 • = , 𝑏≠0
𝑐 𝑏 𝑏𝑚
1
• log 𝑏 = − log 𝑏 𝑎 • 𝑎𝑚 𝑛 = 𝑎𝑚𝑛
𝑎
1
• log 𝑏 𝑎 = log
𝑎 𝑏
• 𝑎log𝑏 𝑐 = 𝑐 log𝑏 𝑎
Data Structures and Algorithms - Analysis of Algorithms 20
In-class Exercise
• Prove that 𝑇 𝑁 = 𝑎 ∙ 𝑁 𝑏 where, 7
log-log plot
𝑎 = 2𝑐 from the log-log plot 6 Straight
5 line of
4 slope 3
3
• Exponential on both sides of the
lg(T(N))
2
equation 1
0
-1 0 5 10 15
• Use rules -2
-3
• 𝑎𝑚 ∙ 𝑎𝑛 = 𝑎𝑚+𝑛 , -4
• 𝑎𝑚 𝑛 = 𝑎𝑚𝑛 lg(N)
• 𝑎 = 𝑏 log𝑏 𝑎
lg 𝑇 𝑁 = 𝑏 ∙ lg 𝑁 + 𝑐
𝑏 = 2,999
𝑐 = −33,210
𝑇 𝑁 = 𝑎 ∙ 𝑁 𝑏 , 𝑤ℎ𝑒𝑟𝑒 𝑎 = 2𝑐
4 minutes
Data Structures and Algorithms - Analysis of Algorithms 21
Prediction and Validation
• Hypothesis
• The running time is about 1,006 ∙ 10−10 ∙ 𝑁 2,999 seconds
• Predictions “order of growth” of running time
• 51,0 seconds for 𝑁 = 8000 is about 𝑁 3 [discussed later]
• 408,1 seconds for 𝑁 = 16000
• Observations
Input size 𝑵 𝑻(𝑵)
8000 51,1
8000 51 validates hypothesis!
8000 51,1
16000 410,8
Data Structures and Algorithms - Analysis of Algorithms 22
Doubling Hypothesis
T N = a ∙ 𝑁𝑏
𝑇 2𝑁 𝑎 ⋅ 2𝑁 𝑏 𝑏
= = 2
• Quick way to estimate b in a power- 𝑇 𝑁 𝑎 ⋅ 𝑁𝑏
law relationship
𝑵 time
Input 𝒍𝒈
• Run program, doubling the size of the size 𝑵
(seconds)† ratio
ratio
𝑻 𝑵
input
250 0 -
500 0 4,8 2,3
• Hypothesis 1000 0,1 6,9 2,8
2000 0,8 7,7 2,9
• Running time is about T N = a ∙ 𝑁 𝑏 b = lg ratio
4000 6,4 8 3
with b = lg ratio, i.e., 0,998 ∙ 10−10 ∙ 𝑁 3 lg(6,4/0,8) = 3,0
8000 51,1 8 3
seems to converge to a constant b ≈ 3
Data Structures and Algorithms - Analysis of Algorithms 23
Doubling Hypothesis
T N = a ∙ 𝑁𝑏
• Q. How to estimate a (assuming we
know 𝑏)? Input size
𝑵 time
(seconds)†
𝑵
• A. Run the program (for a sufficiently 𝑻 𝑵 51,1 = 𝑎 × 80003
large value of 𝑁) and solve for 𝑎 8000 51,1 ⇒ 𝑎 = 0,998 ∙ 10−10
8000 51,0
8000 51,1
• Hypothesis
• Running time is about 0,998 ∙ 10−10 ∙ 𝑁 3
seconds
almost identical hypothesis
to one obtained via linear regression
Data Structures and Algorithms - Analysis of Algorithms 24
Experimental Algorithmics
• System independent effects T N = a ∙ 𝑁𝑏
• Algorithm
determines exponent in power law (b)
• Input data
determines constant in power law (a)
• System dependent effects
• Hardware: CPU, memory, cache, …
• Software: compiler, VM, …
• System: operating system, network, other apps, …
• Bad news
• Difficult to get precise measurements
• Good news
• Much easier and cheaper than other science e.g., can run huge number of experiments
Data Structures and Algorithms - Analysis of Algorithms 25
What’s Next?
• Next session on April 15
• Lecture on Analysis of Algorithms (Cont.)
• In Campus Essen R14 R00 A04 Audimax
• Live stream for students in Campus Duisburg
http://algs4.cs.princeton.edu
Chapter 1.4
Data Structures and Algorithms - Analysis of Algorithms 26
