Professional Documents
Culture Documents
Theoretician wants
to understand.
2
Running time
Analytic Engine
3
Reasons to analyze algorithms
Predict performance.
this course
Compare algorithms.
Provide guarantees.
theory of algorithms
Understand theoretical basis.
4
Some algorithmic successes
time
quadratic
64T
32T
16T
linearithmic
8T
linear
size 1K 2K 4K 8K
5
Some algorithmic successes
N-body simulation.
・Simulate gravitational interactions among N bodies.
・Brute force: N steps.
2
time
quadratic
64T
32T
16T
linearithmic
8T
linear
size 1K 2K 4K 8K
6
The challenge
Scientific method.
・Observe some feature of the natural world.
・Hypothesize a model that is consistent with the observations.
・Predict events using the hypothesis.
・Verify the predictions by making further observations.
・Validate by repeating until the hypothesis and observations agree.
Principles.
・Experiments must be reproducible.
・Hypotheses must be falsifiable.
3-SUM. Given N distinct integers, how many triples sum to exactly zero?
4 -10 0 10 0
return count;
}
11
Measuring the running time
70
% java ThreeSum 2Kints.txt
528
% java ThreeSum 4Kints.txt
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
tick tick tick tick tick tick tick tick
4039
13
Empirical analysis
Run the program for various input sizes and measure running time.
14
Empirical analysis
Run the program for various input sizes and measure running time.
N time (seconds) †
250 0.0
500 0.0
1,000 0.1
2,000 0.8
4,000 6.4
8,000 51.1
16,000 ?
15
Data analysis
6.4
lg (T(N ))
30 3.2
1.6
20 .8
.4
10 .2
.1
1K 2K 4K 8K 1
problem size N
Analysis of experimental data (the running time of ThreeSum)
16
Data analysis
Log-log plot. Plot running time T (N) vs. input size N using log-log scale.
12.8
lg(T (N)) = b lg N + c
6.4 b = 2.999
lg (T(N ))
3.2 c = -33.2103
1.6
.8 T (N) = a N b, where a = 2 c
.4
.2
.1
8K 1K 2K 4K 8K
lgN
ental data (the running time of ThreeSum) power law
8,000 51.1
8,000 51.0
8,000 51.1
16,000 410.8
validates hypothesis!
18
Doubling hypothesis
250 0.0 –
N time (seconds) †
8,000 51.1
51.1 = a × 80003
8,000 51.0
⇒ a = 0.998 × 10 –10
8,000 51.1
・Input data.
in power law
determines constant a
System dependent effects. in power law
21
1.4 A NALYSIS OF A LGORITHMS
‣ introduction
‣ observations
‣ mathematical models
Algorithms
‣ order-of-growth classifications
Donald Knuth
1974 Turing Award
25
Cost of basic operations
assignment statement a = b c2
int count = 0;
for (int i = 0; i < N; i++)
if (a[i] == 0)
count++;
operation frequency
variable declaration 2
assignment statement 2
equal to compare N
array access N
increment N to 2 N
27
Example: 2-SUM
int count = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
if (a[i] + a[j] == 0)
count++;
1
0 + 1 + 2 + . . . + (N 1) = N (N 1)
2 ⇥
N
=
operation frequency 2
equal to compare ½ N (N − 1)
tedious to count exactly
array access N (N − 1)
increment ½ N (N − 1) to N (N − 1)
28
Simplifying the calculations
Cost model. Use some basic operation as a proxy for running time.
int count = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
if (a[i] + a[j] == 0)
count++;
1
0 + 1 + 2 + . . . + (N 1) = N (N 1)
2 ⇥
N
=
operation frequency 2
equal to compare ½ N (N − 1)
30
Simplification 2: tilde notation
Ex 1. ⅙ N 3 + 20 N + 16
~ ⅙N3
166,666,667 N 3/6 ! N 2/2 + N /3
Ex 2. ⅙N 3 + 100 N 4/3 + 56
~ ⅙N 3
Ex 3. ⅙N3 - ½N 2 + ⅓ N
~ ⅙N3 166,167,000
N 1,000
discard lower-order terms Leading-term approximation
(e.g., N = 1000: 500 thousand vs. 166 million)
f (N)
Technical definition. f(N) ~ g(N) means lim = 1
N→ ∞ g(N)
31
Simplification 2: tilde notation
equal to compare ½ N (N − 1) ~ ½ N2
array access N (N − 1) ~ N2
increment ½ N (N − 1) to N (N − 1) ~ ½ N2 to ~ N2
32
Example: 2-SUM
int count = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
if (a[i] + a[j] == 0) "inner loop"
count++;
1
0 + 1 + 2 + . . . + (N 1) = N (N 1)
2 ⇥
A. ~ N 2 array accesses. =
N
2
Bottom line. Use cost model and tilde notation to simplify counts.
33
Example: 3-SUM
int count = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
for (int k = j+1; k < N; k++)
if (a[i] + a[j] + a[k] == 0) "inner loop"
count++;
⇥
N N (N 1)(N 2)
=
3 3!
A. ~ ½ N 3 array accesses. 1 3
⇥ N
6
Bottom line. Use cost model and tilde notation to simplify counts.
34
Estimating a discrete sum
N ⇥ N
1 2
Ex 1. 1 + 2 + … + N. i x dx N
i=1 x=1 2
N N
k 1
Ex 2. 1k + 2k + … + N k. i xk dx N k+1
i=1 x=1 k+1
N ⇥
1 N
1
Ex 3. 1 + 1/2 + 1/3 + … + 1/N. dx = ln N
i=1
i x=1 x
N N N ⇥ ⇥ ⇥
N N N
1 3
Ex 4. 3-sum triple loop. 1 dz dy dx N
i=1 j=i k=j x=1 y=x z=y 6
35
Mathematical models for running time
In practice,
・Formulas can be complicated.
・Advanced mathematics might be required.
・Exact models best left for experts.
costs (depend on machine, compiler)
TN = c1 A + c2 B + c3 C + c4 D + c5 E
A= array access
B= integer add
frequencies
C= integer compare
(depend on algorithm, input)
D= increment
E= variable assignment
log-log plot
512T
exponential
tic
ic
cubi
hm
dra
r
r it
ea
qua
ea
lin
lin
64T
time
8T
4T
2T
logarithmic
T
constant
1K 2K 4K 8K size 512K
39
Common order-of-growth classifications
order of
name typical code framework description example T(2N) / T(N)
growth
add two
1 constant a = b + c; statement 1
numbers
while (N > 1)
log N logarithmic divide in half binary search ~1
{ N = N / 2; ... }
divide
N log N linearithmic [see mergesort lecture] mergesort ~2
and conquer
40
Practical implications of order-of-growth
tens of hundreds of
N millions billions
millions millions
hundreds of hundreds of
N log N millions millions
thousands millions
tens of
N2 hundreds thousand thousands
thousands
2N 20 20s 20s 30
Bottom line. Need linear or linearithmic alg to keep pace with Moore's law.
41
Binary search demo
Goal. Given a sorted array and a key, find index of the key in the array?
6 13 14 25 33 43 51 53 64 72 84 93 95 96 97
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
lo hi
42
Binary search: Java implementation
Trivial to implement?
・First binary search published in 1946; first bug-free one in 1962.
・Bug in Java's Arrays.binarySearch() discovered in 2006.
Invariant. If key appears in the array a[], then a[lo] ≤ key ≤ a[hi].
43
Binary search: mathematical analysis
given
T (N) ≤ T (N / 2) + 1
apply recurrence to first term
≤ T (N / 4) + 1 + 1
...
= 1 + lg N
44
An N2 log N algorithm for 3-SUM
input
Sorting-based algorithm.
30 -40 -20 -10 40 0 10 5
・Step 1: Sort the N (distinct) numbers.
・Step 2: For each pair of numbers a[i] sort
-40 -20 -10 0 5 10 30 40
and a[j], binary search for -(a[i] + a[j]).
binary search
(-40, -20) 60
(-40, -10) 50
(-40, 0) 40
(-40, 5) 35
(-40, 10) 30
⋮ ⋮
(-40, 40) 0
⋮ ⋮
(-20, -10) 30
only count if
⋮ ⋮
a[i] < a[j] < a[k]
(-10, 0) 10 to avoid
Analysis. Order of growth is N2 log N. ⋮ ⋮ double counting
45
Comparing programs
32,000 14.88
64,000 59.16
ThreeSumDeluxe.java
50
Theory of algorithms
Goals.
・Establish “difficulty” of a problem.
・Develop “optimal” algorithms.
Approach.
・Suppress details in analysis: analyze “to within a constant factor”.
・Eliminate variability in input model by focusing on the worst case.
Optimal algorithm.
・Performance guarantee (to within a constant factor) for any input.
・No algorithm can provide a better performance guarantee.
51
Commonly-used notations in the theory of algorithms
½ N2
asymptotic 10 N2 classify
Big Theta Θ(N2)
order of growth 5 N2 + 22 N log N + 3N algorithms
⋮
10 N2
100 N develop
Big Oh Θ(N2) and smaller O(N2)
22 N log N + 3 N upper bounds
⋮
½ N2
N5 develop
Big Omega Θ(N2) and larger Ω(N2)
N3 + 22 N log N + 3 N lower bounds
⋮
52
Theory of algorithms: example 1
Goals.
・Establish “difficulty” of a problem and develop “optimal” algorithms.
・Ex. 1-SUM = “Is there a 0 in the array? ”
Upper bound. A specific algorithm.
・Ex. Brute-force algorithm for 1-SUM: Look at every array entry.
・Running time of the optimal algorithm for 1-SUM is O(N ).
Lower bound. Proof that no algorithm can do better.
・Ex. Have to examine all N entries (any unexamined one might be 0).
・Running time of the optimal algorithm for 1-SUM is Ω(N ).
Optimal algorithm.
・Lower bound equals upper bound (to within a constant factor).
・Ex. Brute-force algorithm for 1-SUM is optimal: its running time is Θ(N ).
53
Theory of algorithms: example 2
Goals.
・Establish “difficulty” of a problem and develop “optimal” algorithms.
・Ex. 3-SUM.
Upper bound. A specific algorithm.
・Ex. Brute-force algorithm for 3-SUM.
・Running time of the optimal algorithm for 3-SUM is O(N 3 ).
54
Theory of algorithms: example 2
Goals.
・Establish “difficulty” of a problem and develop “optimal” algorithms.
・Ex. 3-SUM.
Upper bound. A specific algorithm.
・Ex. Improved algorithm for 3-SUM.
・Running time of the optimal algorithm for 3-SUM is O(N 2 logN ).
Start.
・Develop an algorithm.
・Prove a lower bound.
Gap?
・Lower the upper bound (discover a new algorithm).
・Raise the lower bound (more difficult).
Golden Age of Algorithm Design.
・1970s-.
・Steadily decreasing upper bounds for many important problems.
・Many known optimal algorithms.
Caveats.
・Overly pessimistic to focus on worst case?
・Need better than “to within a constant factor” to predict performance.
56
Commonly-used notations
10 N2 provide
Tilde leading term ~ 10 N2 10 N2 + 22 N log N approximate
10 N2 + 2 N + 37 model
½ N2
asymptotic classify
Big Theta Θ(N2) 10 N2
growth rate algorithms
5 N2 + 22 N log N + 3N
10 N2
develop
Big Oh Θ(N2) and smaller O(N2) 100 N
upper bounds
22 N log N + 3 N
½ N2
develop
Big Omega Θ(N2) and larger Ω(N2) N5
lower bounds
N3 + 22 N log N + 3 N
Byte. 8 bits.
Megabyte (MB). 1 million or 220 bytes.
Gigabyte (GB). 1 billion or 230 bytes.
60
Typical memory usage for primitive types and arrays
boolean 1 char[] 2N + 24
byte 1 int[] 4N + 24
char 2 double[] 8N + 24
int 4
for one-dimensional arrays
float 4
long 8
type bytes
double 8
char[][] ~2MN
for primitive types
int[][] ~4MN
double[][] ~8MN
61
Typical memory usage for objects in Java
32 bytes
2N + 64 bytes
substring example
63
String genome = "CGCCTGGCGTCTGTAC";
Typical memory usage summary
64
Example
16 bytes
public class WeightedQuickUnionUF (object overhead)
{
private int[] id; 8 + (4N + 24) each
private int[] sz; reference + int[] array
A. 8 N + 88 ~ 8 N bytes.
65
1.4 A NALYSIS OF A LGORITHMS
‣ introduction
‣ observations
‣ mathematical models
Algorithms
‣ order-of-growth classifications
Empirical analysis.
・Execute program to perform experiments.
・Assume power law and formulate a hypothesis for running time.
・Model enables us to make predictions.
Mathematical analysis.
・Analyze algorithm to count frequency of operations.
・Use tilde notation to simplify analysis.
・Model enables us to explain behavior.
Scientific method.
・Mathematical model is independent of a particular system;
applies to machines not yet built.
・Empirical analysis is necessary to validate mathematical models
and to make predictions.
67
Algorithms R OBERT S EDGEWICK | K EVIN W AYNE