Professional Documents
Culture Documents
6.046J/18.401J/SMA5503
Lecture 1
Prof. Charles E. Leiserson
Welcome to Introduction to
Algorithms, Fall 2001
Handouts
1. Course Information
2. Calendar
3. Registration (MIT students only)
4. References
5. Objectives and Outcomes
6. Diagnostic Survey
1. Staff 8. Website
2. Distance learning 9. Extra help
3. Prerequisites 10.Registration (MIT only)
4. Lectures 11.Problem sets
5. Recitations 12.Describing algorithms
6. Handouts 13.Grading policy
7. Textbook (CLRS) 14.Collaboration policy
Example:
Input: 8 2 4 9 3 6
Output: 2 3 4 6 8 9
Day 1 Introduction to Algorithms L1.6
Insertion sort
INSERTION-SORT (A, n) A[1 . . n]
for j 2 to n
do key A[ j]
ij1
pseudocode while i > 0 and A[i] > key
do A[i+1] A[i]
ii1
A[i+1] = key
1 i j n
A:
key
sorted
Day 1 Introduction to Algorithms L1.7
Example of insertion sort
8 2 4 9 3 6
Math:
(g(n)) = { f (n) : there exist positive constants c1, c2, and
n0 such that 0 c1 g(n) f (n) c2 g(n)
for all n n0 }
Engineering:
Drop low-order terms; ignore leading constants.
Example: 3n3 + 90n2 5n + 6046 = (n3)
MERGE-SORT A[1 . . n]
1. If n = 1, done.
2. Recursively sort A[ 1 . . n/2 ]
and A[ n/2+1 . . n ] .
3. Merge the 2 sorted lists.
1 2
1 2
1 2 7
1 2 7
1 2 7 9
1 2 7 9
1 2 7 9 11
1 2 7 9 11
1 2 7 9 11 12
1 2 7 9 11 12
(1)
(1)
(1)
(1)
(1)
(1) #leaves = n (n)
(1) #leaves = n (n)
Total = (n lg n)
Day 1 Introduction to Algorithms L1.51
Conclusions
Lecture 2
Prof. Erik Demaine
Solving recurrences
The analysis of merge sort from
Lecture 1 required us to solve a
recurrence.
Recurrences are like solving integrals,
differential equations, etc.
o Learn a few tricks.
Lecture 3: Applications of recurrences.
(1)
(1)
(1)
(1)
(1) Total = n 2
( 5 5
1 + 16 + 16 ( ) +( )
2 5 3
16
+L )
= (n2) geometric series
Day 3 Introduction to Algorithms L2.17
The master method
i i ) = 1.
(
k
p
a /b
i =1
Then, the answers are the same as for the
master method, but with np instead of nlogba.
(Akra and Bazzi also prove an even more
general result.)
Day 3 Introduction to Algorithms L2.23
Idea of master theorem
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) f (n/b2) a2 f (n/b2)
#leaves = ah
C ASE 1:
CASE 1: The
The weight
weight increases
increases
geometrically
geometrically from
from the
the root
root to
to the
the nlogba (1)
(1) leaves.
leaves. The
The leaves
leaves hold
hold aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight. (nlogba)
Day 3 Introduction to Algorithms L2.25
Idea of master theorem
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) f (n/b2) a2 f (n/b2)
CCASEASE 2:
2: (k
(k == 0)
0) The
The weight
weight
isis approximately nlogba (1)
(1) approximately the the same
same on
on
each
each ofof the
the log
logbbnn levels.
levels.
(nlogbalg n)
Day 3 Introduction to Algorithms L2.26
Idea of master theorem
Recursion tree:
f (n) f (n)
a
f (n/b) f (n/b) f (n/b) a f (n/b)
h = logbn a
f (n/b2) f (n/b2) f (n/b2) a2 f (n/b2)
C ASE 3:
CASE 3: The
The weight
weight decreases
decreases
geometrically
geometrically from
from the
the root
root to
to the
the nlogba (1)
(1) leaves.
leaves. The
The root
root holds
holds aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight. ( f (n))
Day 3 Introduction to Algorithms L2.27
Conclusion
n +1
1 x
1 + x + x2 + L + xn = for x 1
1 x
2 1
1+ x + x +L = for |x| < 1
1 x
Return to last
slide viewed.
Lecture 3
Prof. Erik Demaine
The divide-and-conquer
design paradigm
1. Divide the problem (instance)
into subproblems.
2. Conquer the subproblems by
solving them recursively.
3. Combine subproblem solutions.
n
cij = aik bkj
k =1
Day 4 Introduction to Algorithms L3.17
Standard algorithm
for i 1 to n
do for j 1 to n
do cij 0
for k 1 to n
do cij cij + aik bkj
H(n)
Area = (n)
Lecture 4
Prof. Charles E. Leiserson
Quicksort
Proposed by C.A.R. Hoare in 1962.
Divide-and-conquer algorithm.
Sorts in place (like insertion sort, but not
like merge sort).
Very practical (with tuning).
66 10
10 13
13 55 88 33 22 11
11
i j
66 10
10 13
13 55 88 33 22 11
11
i j
66 10
10 13
13 55 88 33 22 11
11
i j
66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
i j
66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
i j
66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
i j
66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
66 55 33 10
10 88 13
13 22 11
11
i j
66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
66 55 33 10
10 88 13
13 22 11
11
i j
66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
66 55 33 10
10 88 13
13 22 11
11
66 55 33 22 88 13
13 10
10 11
11
i j
66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
66 55 33 10
10 88 13
13 22 11
11
66 55 33 22 88 13
13 10
10 11
11
i j
66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
66 55 33 10
10 88 13
13 22 11
11
66 55 33 22 88 13
13 10
10 11
11
i j
66 10
10 13
13 55 88 33 22 11
11
66 55 13
13 10
10 88 33 22 11
11
66 55 33 10
10 88 13
13 22 11
11
66 55 33 22 88 13
13 10
10 11
11
22 55 33 66 88 13
13 10
10 11
11
i
Day 6 Introduction to Algorithms L4.16
Pseudocode for quicksort
QUICKSORT(A, p, r)
if p < r
then q PARTITION(A, p, r)
QUICKSORT(A, p, q1)
QUICKSORT(A, q+1, r)
(1)
(1)
(1)
1 9
What if the split is always 10 : 10 ?
T (n) = T (101 n ) + T (109 n ) + (n)
What is the solution to this recurrence?
T (100
1
n ) T (100
9
n ) T (100
9
n )T (100
81
n)
(1) O(n)
O(n) leaves
leaves
(1)
(1) O(n)
O(n) leaves
leaves
(n lg n) (1)
Lucky! cn log10n T(n) cn log10/9n + (n)
Day 6 Introduction to Algorithms L4.32
More intuition
Suppose we alternate lucky, unlucky,
lucky, unlucky, lucky, .
L(n) = 2U(n/2) + (n) lucky
U(n) = L(n 1) + (n) unlucky
Solving:
L(n) = 2(L(n/2 1) + (n/2)) + (n)
= 2L(n/2 1) + (n)
= (n lg n) Lucky!
How can we make sure we are usually lucky?
Day 6 Introduction to Algorithms L4.33
Randomized quicksort
IDEA: Partition around a random element.
Running time is independent of the input
order.
No assumptions need to be made about
the input distribution.
No specific input elicits the worst-case
behavior.
The worst case is determined only by the
output of a random-number generator.
Day 6 Introduction to Algorithms L4.34
Randomized quicksort
analysis
Let T(n) = the random variable for the running
time of randomized quicksort on an input of size
n, assuming random numbers are independent.
For k = 0, 1, , n1, define the indicator
random variable
1 if PARTITION generates a k : nk1 split,
Xk =
0 otherwise.
E[Xk] = Pr{Xk = 1} = 1/n, since all splits are
equally likely, assuming elements are distinct.
Day 6 Introduction to Algorithms L4.35
Analysis (continued)
T(0) + T(n1) + (n) if 0 : n1 split,
T(1) + T(n2) + (n) if 1 : n2 split,
T(n) =
M
T(n1) + T(0) + (n) if n1 : 0 split,
n 1
= X k (T (k ) + T (n k 1) + (n)) .
k =0
Linearity of expectation.
2a 1 n 2 lg n 1 n 2 + (n)
n 2 8
Use fact.
2a 1 n 2 lg n 1 n 2 + (n)
n 2 8
= an lg n an (n)
4
Express as desired residual.
= 2a 1 n 2 lg n 1 n 2 + (n)
n 2 8
= an lg n an (n)
4
an lg n ,
if a is chosen large enough so that
an/4 dominates the (n).
Day 6 Introduction to Algorithms L4.46
Quicksort in practice
Lecture 5
Prof. Erik Demaine
How fast can we sort?
All the sorting algorithms we have seen so far
are comparison sorts: only use comparisons to
determine the relative order of elements.
E.g., insertion sort, merge sort, quicksort,
heapsort.
The best worst-case running time that weve
seen for comparison sorting is O(n lg n) .
Is O(n lg n) the best we can do?
Decision trees can help us answer this question.
2001 by Charles E. Leiserson Introduction to Algorithms Day 8 L5.2
Decision-tree example
Sort a1, a2, , an 1:2
1:2
2:3
2:3 1:3
1:3
123
123 1:3 213
213 2:3
1:3 2:3
132
132 312
312 231
231 321
321
A: 44 11 33 44 33 C:
B:
A: 44 11 33 44 33 C: 00 00 00 00
B:
for i 1 to k
do C[i] 0
A: 44 11 33 44 33 C: 00 00 00 11
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
A: 44 11 33 44 33 C: 11 00 00 11
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
A: 44 11 33 44 33 C: 11 00 11 11
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
A: 44 11 33 44 33 C: 11 00 11 22
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
A: 44 11 33 44 33 C: 11 00 22 22
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
A: 44 11 33 44 33 C: 11 00 22 22
B: C': 11 11 22 22
for i 2 to k
do C[i] C[i] + C[i1] C[i] = |{key i}|
A: 44 11 33 44 33 C: 11 00 22 22
B: C': 11 11 33 22
for i 2 to k
do C[i] C[i] + C[i1] C[i] = |{key i}|
A: 44 11 33 44 33 C: 11 00 22 22
B: C': 11 11 33 55
for i 2 to k
do C[i] C[i] + C[i1] C[i] = |{key i}|
A: 44 11 33 44 33 C: 11 11 33 55
B: 33 C': 11 11 22 55
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
2001 by Charles E. Leiserson Introduction to Algorithms Day 8 L5.23
Loop 4
1 2 3 4 5 1 2 3 4
A: 44 11 33 44 33 C: 11 11 22 55
B: 33 44 C': 11 11 22 44
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
2001 by Charles E. Leiserson Introduction to Algorithms Day 8 L5.24
Loop 4
1 2 3 4 5 1 2 3 4
A: 44 11 33 44 33 C: 11 11 22 44
B: 33 33 44 C': 11 11 11 44
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
2001 by Charles E. Leiserson Introduction to Algorithms Day 8 L5.25
Loop 4
1 2 3 4 5 1 2 3 4
A: 44 11 33 44 33 C: 11 11 11 44
B: 11 33 33 44 C': 00 11 11 44
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
2001 by Charles E. Leiserson Introduction to Algorithms Day 8 L5.26
Loop 4
1 2 3 4 5 1 2 3 4
A: 44 11 33 44 33 C: 00 11 11 44
B: 11 33 33 44 44 C': 00 11 11 33
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
2001 by Charles E. Leiserson Introduction to Algorithms Day 8 L5.27
Analysis
for i 1 to k
(k) do C[i] 0
for j 1 to n
(n)
do C[A[ j]] C[A[ j]] + 1
for i 2 to k
(k) do C[i] C[i] + C[i1]
for j n downto 1
(n) do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
(n + k)
2001 by Charles E. Leiserson Introduction to Algorithms Day 8 L5.28
Running time
If k = O(n), then counting sort takes (n) time.
But, sorting takes (n lg n) time!
Wheres the fallacy?
Answer:
Comparison sorting takes (n lg n) time.
Counting sort is not a comparison sort.
In fact, not a single comparison between
elements occurs!
2001 by Charles E. Leiserson Introduction to Algorithms Day 8 L5.29
Stable sorting
Counting sort is a stable sort: it preserves
the input order among equal elements.
A: 44 11 33 44 33
B: 11 33 33 44 44
Replica of punch
card from the
1900 U.S. census:
[Howells 2000]
system
Pantograph card
punch
Hand-press reader
Dial counters
Sorting box
Lecture 6
Prof. Erik Demaine
Order statistics
Select the ith smallest of n elements (the
element with rank i).
i = 1: minimum;
i = n: maximum;
i = (n+1)/2 or (n+1)/2: median.
Naive algorithm: Sort and index ith element.
Worst-case running time = (n lg n) + (1)
= (n lg n),
using merge sort or heapsort (not quicksort).
2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.2
Randomized divide-and-
conquer algorithm
RAND-SELECT(A, p, q, i) ith smallest of A[ p . . q]
if p = q then return A[ p]
r RAND-PARTITION(A, p, q)
krp+1 k = rank(A[r])
if i = k then return A[ r]
if i < k
then return RAND-SELECT(A, p, r 1, i )
else return RAND-SELECT(A, r + 1, q, i k )
k
A[r]
A[r] A[r]
A[r]
p r q
2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.3
Example
Select the i = 7th smallest:
66 10
10 13
13 55 88 33 22 11
11 i=7
pivot
Partition:
22 55 33 66 88 13
13 10
10 11
11 k=4
Linearity of expectation.
greater
2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.22
Choosing the pivot
greater
2001 by Charles E. Leiserson Introduction to Algorithms Day 9 L6.24
Analysis (Assume all elements are distinct.)
Lecture 7
Prof. Charles E. Leiserson
Symbol-table problem
Symbol table T holding n records:
record
x Operations on T:
key[x]
key[x]
INSERT(T, x)
DELETE(T, x)
Other fields
containing SEARCH(T, k)
satellite data
k1 h(k1)
k5 h(k4)
K k4 h(k2) = h(k5)
k2 k3
h(k3)
U
m1
When a record to be inserted maps to an already
As each key
occupied slotisininserted, h maps
T, a collision it to a slot of T.
occurs.
2001 by Charles E. Leiserson Introduction to Algorithms Day 11 L7.4
Resolving collisions by
chaining
Records in the same slot are linked into a list.
T
i 49
49 86
86 52
52
h(49) = h(86) = h(52) = i
204 collision
481
m1
204
481
m1
m1
Lecture 8
Prof. Charles E. Leiserson
A weakness of hashing
Problem: For any hash function h, a set
of keys exists that can cause the average
access time of a hash table to skyrocket.
An adversary can pick all keys from
{k U : h(k) = i} for some slot i.
IDEA: Choose the hash function at random,
independently of the keys.
Even if an adversary can see your code,
he or she cannot find a bad set of keys,
since he or she doesnt know exactly
which hash function will be chosen.
2001 by Charles E. Leiserson Introduction to Algorithms Day 12 L8.2
Universal hashing
Definition. Let U be a universe of keys, and
let H be a finite collection of hash functions,
each mapping U to {0, 1, , m1}. We say
H is universal if for all x, y U, where x y,
we have |{h H : h(x) = h(y)}| = |H|/m.
= n 1 . Algebra.
m
2001 by Charles E. Leiserson Introduction to Algorithms Day 12 L8.9
Constructing a set of
universal hash functions
Let m be prime. Decompose key k into r + 1
digits, each with value in the set {0, 1, , m1}.
That is, let k = k0, k1, , kr, where 0 ki < m.
Randomized strategy:
Pick a = a0, a1, , ar where each ai is chosen
randomly from {0, 1, , m1}.
r
Define ha (k ) = ai ki mod m . Dot product,
i =0
modulo m
How big is H = {ha}? |H| = mr + 1. REMEMBER
THIS!
2001 by Charles E. Leiserson Introduction to Algorithms Day 12 L8.10
Universality of dot-product
hash functions
Theorem. The set H = {ha} is universal.
Example: m = 7.
z 1 2 3 4 5 6
z1 1 4 5 2 3 6
Lecture 9
Prof. Charles E. Leiserson
Binary-search-tree sort
T Create an empty BST
for i = 1 to n
do TREE-INSERT(T, A[i])
Perform an inorder tree walk of T.
Example: 33
A = [3 1 8 2 6 7 5] 11 88
Tree-walk time = O(n), 22 66
but how long does it 55 77
take to build the BST?
2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.2
Analysis of BST sort
BST sort performs the same comparisons as
quicksort, but in a different order!
3 1 8 2 6 7 5
1 2 8 6 7 5
2 675
5 7
The expected time to build the tree is asymptot-
ically the same as the running time of quicksort.
2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.3
Node depth
The depth of a node = the number of comparisons
made during TREE-INSERT. Assuming all input
permutations are equally likely, we have
Average node depth
n
= E (# comparisons to insert node i )
1
n i =1
= 1 O(n lg n) (quicksort analysis)
n
= O(lg n) .
2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.4
Expected tree height
But, average node depth of a randomly built
BST = O(lg n) does not necessarily mean that its
expected height is also O(lg n) (although it is).
Example.
lg n
h= n
Ave. depth 1 n lg n + n n
n 2
= O(lg n)
2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.5
Height of a randomly built
binary search tree
Outline of the analysis:
Prove Jensens inequality, which says that
f(E[X]) E[f(X)] for any convex function f and
random variable X.
Analyze the exponential height of a randomly
built BST on n nodes, which is the random
variable Yn = 2Xn, where Xn is the random
variable denoting the height of the BST.
Prove that 2E[Xn] E[2Xn ] = E[Yn] = O(n3),
and hence that E[Xn] = O(lg n).
2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.6
Convex functions
A function f : R R is convex if for all
, 0 such that + = 1, we have
f(x + y) f(x) + f(y)
for all x,y R.
f
f(y)
f(x) + f(y)
f(x)
f(x + y)
x x + y y
2001 by Charles E. Leiserson Introduction to Algorithms Day 17 L9.7
Convexity lemma
Lemma. Let f : R R be a convex function,
and let {1, 2 , , n} be a set of nonnegative
constants such that k k = 1. Then, for any set
{x1, x2, , xn} of real numbers, we have
n n
f k xk k f ( xk ) .
k =1 k =1
Proof. By induction on n. For n = 1, we have
1 = 1, and hence f(1x1) 1f(x1) trivially.
Convexity.
Induction.
Linearity of expectation.
Lecture 10
Prof. Erik Demaine
Balanced search trees
Balanced search tree: A search-tree data
structure for which a height of O(lg n) is
guaranteed when implementing a dynamic
set of n items.
AVL trees
2-3 trees
Examples: 2-3-4 trees
B-trees
Red-black trees
33 18
18
NIL NIL
10
10 22
22 h=4
NIL
88 11
11 26
26
NIL NIL NIL NIL NIL NIL
33 18
18
NIL NIL
10
10 22
22
NIL
88 11
11 26
26
NIL NIL NIL NIL NIL NIL
33 18
18
NIL NIL
10
10 22
22
NIL
88 11
11 26
26
NIL NIL NIL NIL NIL NIL
33 18
18
NIL NIL
10
10 22
22
NIL
88 11
11 26
26
NIL NIL NIL NIL NIL NIL
33 18
18 bh = 2
NIL NIL bh = 1 10
10 22
22
NIL
bh = 1 88 11
11 26
26
bh = 0 NIL NIL NIL NIL NIL NIL
4. All simple paths from any node x to a
descendant leaf have the same number of
black nodes = black-height(x).
2001 by Charles E. Leiserson Introduction to Algorithms Day 18 L10.8
Height of a red-black tree
Theorem. A red-black tree with n keys has height
h 2 lg(n + 1).
Proof. (The book uses induction. Read carefully.)
INTUITION:
Merge red nodes
into their black
parents.
CC LEFT-ROTATE(A) CC
y y
AA BB
x BB x AA
Transform to Case 3.
CC
RIGHT-ROTATE(C)
BB
y
BB
AA CC
x AA
Done! No more
violations of RB
property 3 are
possible.
2001 by Charles E. Leiserson Introduction to Algorithms Day 18 L10.28
Analysis
Lecture 11
Prof. Erik Demaine
Dynamic order statistics
OS-SELECT(i, S): returns the i th smallest element
in the dynamic set S.
OS-RANK(x, S): returns the rank of x S in the
sorted order of Ss elements.
key
key
Notation for nodes:
size
size
2001 by Charles E. Leiserson Introduction to Algorithms Day 20 L11.2
Example of an OS-tree
M
M
99
CC PP
55 33
AA FF NN QQ
11 33 11 11
DD HH
11 11
7 3 3 4
RB-INSERT and RB-DELETE still run in O(lg n) time.
2001 by Charles E. Leiserson Introduction to Algorithms Day 20 L11.8
Data-structure augmentation
Methodology: (e.g., order-statistics trees)
1. Choose an underlying data structure (red-
black trees).
2. Determine additional information to be
stored in the data structure (subtree sizes).
3. Verify that this information can be
maintained for modifying operations (RB-
INSERT, RB-DELETE dont forget rotations).
4. Develop new dynamic-set operations that use
the information (OS-SELECT and OS-RANK).
These steps are guidelines, not rigid rules.
2001 by Charles E. Leiserson Introduction to Algorithms Day 20 L11.9
Interval trees
Goal: To maintain a dynamic set of intervals,
such as time intervals.
i = [7, 10]
low[i] = 7 10 = high[i]
5 11 17 19
4 8 15 18 22 23
int
int
mm
2001 by Charles E. Leiserson Introduction to Algorithms Day 20 L11.11
Example interval tree
17,19
17,19
23
23
5,11
5,11 22,23
22,23
18
18 23
23
4,8
4,8 15,18
15,18
88 18
18
high[int[x]]
7,10
7,10 m[x] = max m[left[x]]
10
10
m[right[x]]
30
30 14
14 14
14 19
19
Lecture 12
Prof. Erik Demaine
Computational geometry
Algorithms for solving geometric problems
in 2D and higher.
Fundamental objects:
point line segment line
Basic structures:
11 17
17 43
43
66 88 12
12 14
14 26
26 35
35 41
41 42
42 59
59 61
61
11 14
14 35
35 43
43
11 66 12
12 17
17 26
26 41
41 43
43 59
59
66 88 12
12 14
14 26
26 35
35 41
41 42
42 59
59 61
61
11 14
14 35
35 43
43
11 66 12
12 17
17 26
26 41
41 43
43 59
59
66 88 12
12 14
14 26
26 35
35 41
41 42
42 59
59 61
61
RANGE-QUERY([7, 41])
2001 by Erik D. Demaine Introduction to Algorithms Day 21 L12.11
General 1D range query
root
split node
Crossproduct v1 v2 = x1 y2 y1 x2
= |v1| |v2| sin .
Thus, sign(v1 v2) = sign(sin ) > 0 if convex,
< 0 if reflex,
= 0 if borderline.
2001 by Erik D. Demaine Introduction to Algorithms Day 21 L12.20
Primitive operations:
Orientation test
Given three points p1, p2, p3 are they p3
in clockwise (cw) order,
in counterclockwise (ccw) order, or p2
collinear?
p1
(p2 p1) (p3 p1)
collinear
> 0 if ccw
< 0 if cw p2 p3
= 0 if collinear p1 cw p1 ccw
p3 p2
c
f
b
e
d
a
b f
Lecture 13
Prof. Erik Demaine
Fixed-universe
successor problem
Goal: Maintain a dynamic subset S of size n
of the universe U = {0, 1, , u 1} of size u
subject to these operations:
INSERT(x U \ S): Add x to S.
DELETE(x S): Remove x from S.
SUCCESSOR(x U): Find the next element in S
larger than any element x of the universe U.
PREDECESSOR(x U): Find the previous
element in S smaller than x.
Example: u = 16, u = 4 .
W0 W1 W2 W3
0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 1
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
W0 represents 0, 1, , u 1 U;
W1 represents u , u + 1, , 2 u 1 U;
:
Wi represents i u , i u + 1, , (i + 1) u 1 U;
:
W u 1 represents u u , u u + 1 , , u 1 U.
Lecture 14
Prof. Charles E. Leiserson
How large should a hash
table be?
Goal: Make the table as small as possible, but
large enough so that it wont overflow (or
otherwise become inefficient).
Problem: What if we dont know the proper size
in advance?
Solution: Dynamic tables.
IDEA: Whenever the table overflows, grow it
by allocating (via malloc or new) a new, larger
table. Move all items from the old table into the
new one, and free the storage for the old table.
2001 by Charles E. Leiserson Introduction to Algorithms Day 24 L14.2
Example of a dynamic table
1. INSERT 1
2. INSERT overflow
1. INSERT 11
2. INSERT overflow
1. INSERT 11
2. INSERT 2
1. INSERT 11
2. INSERT 22
3. INSERT overflow
1. INSERT 1
2. INSERT 2
3. INSERT overflow
1. INSERT 1
2. INSERT 2
3. INSERT
1. INSERT 1
2. INSERT 2
3. INSERT 3
4. INSERT 4
1. INSERT 1
2. INSERT 2
3. INSERT 3
4. INSERT 4
5. INSERT overflow
1. INSERT 1
2. INSERT 2
3. INSERT 3
4. INSERT 4
5. INSERT overflow
1. INSERT 1
2. INSERT 2
3. INSERT 3
4. INSERT 4
5. INSERT
1. INSERT 1
2. INSERT 2
3. INSERT 3
4. INSERT 4
5. INSERT 5
6. INSERT 6
7. INSERT 7
i 1 2 3 4 5 6 7 8 9 10
sizei 1 2 4 4 8 8 8 8 16 16
ci 1 2 3 1 5 1 1 1 9 1
i 1 2 3 4 5 6 7 8 9 10
sizei 1 2 4 4 8 8 8 8 16 16
1 1 1 1 1 1 1 1 1 1
ci
1 2 4 8
$0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0
2001 by Charles E. Leiserson Introduction to Algorithms Day 24 L14.22
Accounting analysis of
dynamic tables
Charge an amortized cost of i = $3 for the i th
insertion.
$1 pays for the immediate insertion.
$2 is stored for later table doubling.
When the table doubles, $1 pays to move a
recent item, and $1 pays to move an old item.
Example:
$0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $2 $2 $2
2001 by Charles E. Leiserson Introduction to Algorithms Day 24 L14.23
Accounting analysis
(continued)
Key invariant: Bank balance never drops below 0.
Thus, the sum of the amortized costs provides an
upper bound on the sum of the true costs.
i 1 2 3 4 5 6 7 8 9 10
sizei 1 2 4 4 8 8 8 8 16 16
ci 1 2 3 1 5 1 1 1 9 1
i 2* 3 3 3 3 3 3 3 3 3
banki 1 2 2 4 2 4 6 8 2 4
*Okay, so I lied. The first operation costs only $2, not $3.
2001 by Charles E. Leiserson Introduction to Algorithms Day 24 L14.24
Potential method
IDEA: View the bank account as the potential
energy ( la physics) of the dynamic set.
Framework:
Start with an initial data structure D0.
Operation i transforms Di1 to Di.
The cost of operation i is ci.
Define a potential function : {Di} R,
such that (D0 ) = 0 and (Di ) 0 for all i.
The amortized cost i with respect to is
defined to be i = ci + (Di) (Di1).
2001 by Charles E. Leiserson Introduction to Algorithms Day 24 L14.25
Understanding potentials
i = ci + (Di) (Di1)
potential difference i
i = ci + (Di) (Di1)
Lecture 15
Prof. Charles E. Leiserson
Dynamic programming
Design technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)
Given two sequences x[1 . . m] and y[1 . . n], find
a longest subsequence common to them both.
a not the
x: A B C B D A B
BCBA =
LCS(x, y)
y: B D C A B A
functional notation,
but not a function
2001 by Charles E. Leiserson Introduction to Algorithms Day 26 L15.2
Brute-force LCS algorithm
Check every subsequence of x[1 . . m] to see
if it is also a subsequence of y[1 . . n].
Analysis
Checking = O(n) time per subsequence.
2m subsequences of x (each bit-vector of
length m determines a distinct subsequence
of x).
Worst-case running time = O(n2m)
= exponential time.
2001 by Charles E. Leiserson Introduction to Algorithms Day 26 L15.3
Towards a better algorithm
Simplification:
1. Look at the length of a longest-common
subsequence.
2. Extend the algorithm to find the LCS itself.
Notation: Denote the length of a sequence s
by | s |.
Strategy: Consider prefixes of x and y.
Define c[i, j] = | LCS(x[1 . . i], y[1 . . j]) |.
Then, c[m, n] = | LCS(x, y) |.
2001 by Charles E. Leiserson Introduction to Algorithms Day 26 L15.4
Recursive formulation
Theorem.
c[i1, j1] + 1 if x[i] = y[j],
c[i, j] = max{c[i1, j], c[i, j1]} otherwise.
Proof. Case x[i] = y[ j]:
1 2 i m
x: L
1 2 = j n
y: L
Optimal substructure
An optimal solution to a problem
(instance) contains optimal
solutions to subproblems.
2,4
2,4 same 3,3
3,3
subproblem
1,4
1,4 2,3
2,3 2,3
2,3 3,2
3,2 m+n
1,3
1,3 2,2
2,2 1,3
1,3 2,2
2,2
Lecture 16
Prof. Charles E. Leiserson
Graphs (review)
Definition. A directed graph (digraph)
G = (V, E) is an ordered pair consisting of
a set V of vertices (singular: vertex),
a set E V V of edges.
In an undirected graph G = (V, E), the edge
set E consists of unordered pairs of vertices.
In either case, we have | E | = O(V 2). Moreover,
if G is connected, then | E | | V | 1, which
implies that lg | E | = (lg V).
(Review CLRS, Appendix B.)
2001 by Charles E. Leiserson Introduction to Algorithms Day 27 L16.2
Adjacency-matrix
representation
The adjacency matrix of a graph G = (V, E), where
V = {1, 2, , n}, is the matrix A[1 . . n, 1 . . n]
given by
1 if (i, j) E,
A[i, j] =
0 if (i, j) E.
A 1 2 3 4
22 11 1 0 1 1 0 (V 2) storage
2 0 0 1 0 dense
33 44 3 0 0 0 0 representation.
4 0 0 1 0
2001 by Charles E. Leiserson Introduction to Algorithms Day 27 L16.3
Adjacency-list representation
An adjacency list of a vertex v V is the list Adj[v]
of vertices adjacent to v.
22 11 Adj[1] = {2, 3}
Adj[2] = {3}
Adj[3] = {}
33 44 Adj[4] = {3}
For undirected graphs, | Adj[v] | = degree(v).
For digraphs, | Adj[v] | = out-degree(v).
Handshaking Lemma: vV = 2 |E| for undirected
graphs adjacency lists use (V + E) storage
a sparse representation (for either type of graph).
2001 by Charles E. Leiserson Introduction to Algorithms Day 27 L16.4
Minimum spanning trees
Input: A connected, undirected graph G = (V, E)
with weight function w : E R.
For simplicity, assume that all edge weights are
distinct. (CLRS covers the general case.)
6 12
5 9
14 7
8 15
3 10
Greedy-choice property
A locally optimal choice
is globally optimal.
T: v
u
A
(u, v) = least-weight edge
VA connecting A to V A
T: v
u
A
(u, v) = least-weight edge
VA connecting A to V A
Consider the unique simple path from u to v in T.
T: v
u
A
(u, v) = least-weight edge
VA connecting A to V A
Consider the unique simple path from u to v in T.
Swap (u, v) with the first edge on this path that
connects a vertex in A to a vertex in V A.
T : v
u
A
(u, v) = least-weight edge
VA connecting A to V A
Consider the unique simple path from u to v in T.
Swap (u, v) with the first edge on this path that
connects a vertex in A to a vertex in V A.
A lighter-weight spanning tree than T results.
2001 by Charles E. Leiserson Introduction to Algorithms Day 27 L16.13
Prims algorithm
IDEA: Maintain V A as a priority queue Q. Key
each vertex in Q with the weight of the least-
weight edge connecting it to a vertex in A.
QV
key[v] for all v V
key[s] 0 for some arbitrary s V
while Q
do u EXTRACT-MIN(Q)
for each v Adj[u]
do if v Q and w(u, v) < key[v]
then key[v] w(u, v) DECREASE-KEY
[v] u
At the end, {(v, [v])} forms the MST.
2001 by Charles E. Leiserson Introduction to Algorithms Day 27 L16.14
Example of Prims algorithm
A
6 12
VA
5 9
14 7
8 15
00
3 10
A
6 12
VA
5 9
14 7
8 15
00
3 10
A
6 12
VA
5 9
77
14 7
8 15
00 15
15
3 10
10
10
A
6 12
VA
5 9
77
14 7
8 15
00 15
15
3 10
10
10
A 12
12
6 12
VA
5 9
55 77 99
14 7
8 15
00 15
15
3 10
10
10
A 12
12
6 12
VA
5 9
55 77 99
14 7
8 15
00 15
15
3 10
10
10
A 66
6 12
VA
5 9
55 77 99
14 7
8 15
14
14 00 15
15
3 10
88
A 66
6 12
VA
5 9
55 77 99
14 7
8 15
14
14 00 15
15
3 10
88
A 66
6 12
VA
5 9
55 77 99
14 7
8 15
14
14 00 15
15
3 10
88
A 66
6 12
VA
5 9
55 77 99
14 7
8 15
33 00 15
15
3 10
88
A 66
6 12
VA
5 9
55 77 99
14 7
8 15
33 00 15
15
3 10
88
A 66
6 12
VA
5 9
55 77 99
14 7
8 15
33 00 15
15
3 10
88
A 66
6 12
VA
5 9
55 77 99
14 7
8 15
33 00 15
15
3 10
88
Lecture 17
Prof. Erik Demaine
Paths in graphs
Consider a digraph G = (V, E) with edge-weight
function w : E R. The weight of path p = v1
v2 L vk is defined to be
k 1
w( p ) = w(vi , vi +1 ) .
i =1
Example:
vv11 4 2 vv33 5 1 vv55
vv22 vv44
w(p) = 2
Proof.
(u, v)
uu vv
(u, x) (x, v)
xx
Example:
<0
uu vv
3
CC 2 EE
3
Q: A B C D E CC 2 EE
0
S: {}
3
Q: A B C D E CC 2 EE
0
S: { A }
3
Q: A B C D E CC 2 EE
0 3
10 3
S: { A }
3
Q: A B C D E CC 2 EE
0 3
10 3
S: { A, C }
3
Q: A B C D E CC 2 EE
0 3 5
10 3
7 11 5
S: { A, C }
3
Q: A B C D E CC 2 EE
0 3 5
10 3
7 11 5
S: { A, C, E }
3
Q: A B C D E CC 2 EE
0 3 5
10 3
7 11 5
7 11 S: { A, C, E }
3
Q: A B C D E CC 2 EE
0 3 5
10 3
7 11 5
7 11 S: { A, C, E, B }
3
Q: A B C D E CC 2 EE
0 3 5
10 3
7 11 5
7 11 S: { A, C, E, B }
9
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.18
Example of Dijkstras
algorithm
7 9
D EXTRACT-MIN(Q): 2
BB D
D
10
8
0 AA 1 4 7 9
3
Q: A B C D E CC 2 EE
0 3 5
10 3
7 11 5
7 11 S: { A, C, E, B, D }
9
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.19
Correctness Part I
Lemma. Initializing d[s] 0 and d[v] for all
v V {s} establishes d[v] (s, v) for all v V,
and this invariant is maintained over any sequence
of relaxation steps.
Proof. Suppose not. Let v be the first vertex for
which d[v] < (s, v), and let u be the vertex that
caused d[v] to change: d[v] = d[u] + w(u, v). Then,
d[v] < (s, v) supposition
(s, u) + (u, v) triangle inequality
(s,u) + w(u, v) sh. path specific path
d[u] + w(u, v) v is first violation
Contradiction.
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.20
Correctness Part II
Theorem. Dijkstras algorithm terminates with
d[v] = (s, v) for all v V.
Proof. It suffices to show that d[v] = (s, v) for every
v V when v is added to S. Suppose u is the first
vertex added to S for which d[u] (s, u). Let y be the
first vertex in V S along a shortest path from s to u,
and let x be its predecessor:
uu
ss
S, just before xx yy
adding u.
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.21
Correctness Part II
(continued)
S uu
ss
xx yy
aa ff hh
dd
bb gg
ee ii
cc
Q:
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.26
Example of breadth-first
search
0 aa ff hh
dd
bb gg
ee ii
cc
0
Q: a
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.27
Example of breadth-first
search
0 aa 1 ff hh
dd
1 bb gg
ee ii
cc
1 1
Q: a b d
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.28
Example of breadth-first
search
0 aa 1 ff hh
dd
1 bb gg
ee ii
2 cc 2
1 2 2
Q: a b d c e
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.29
Example of breadth-first
search
0 aa 1 ff hh
dd
1 bb gg
ee ii
2 cc 2
2 2
Q: a b d c e
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.30
Example of breadth-first
search
0 aa 1 ff hh
dd
1 bb gg
ee ii
2 cc 2
2
Q: a b d c e
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.31
Example of breadth-first
search
0 aa 1 ff hh
dd
3
1 bb gg
ee ii
2 cc 2 3
3 3
Q: a b d c e g i
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.32
Example of breadth-first
search
4
0 aa 1 ff hh
dd
3
1 bb gg
ee ii
2 cc 2 3
3 4
Q: a b d c e g i f
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.33
Example of breadth-first
search
4 4
0 aa 1 ff hh
dd
3
1 bb gg
ee ii
2 cc 2 3
4 4
Q: a b d c e g i f h
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.34
Example of breadth-first
search
4 4
0 aa 1 ff hh
dd
3
1 bb gg
ee ii
2 cc 2 3
4
Q: a b d c e g i f h
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.35
Example of breadth-first
search
4 4
0 aa 1 ff hh
dd
3
1 bb gg
ee ii
2 cc 2 3
Q: a b d c e g i f h
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.36
Example of breadth-first
search
4 4
0 aa 1 ff hh
dd
3
1 bb gg
ee ii
2 cc 2 3
Q: a b d c e g i f h
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.37
Correctness of BFS
while Q
do u DEQUEUE(Q)
for each v Adj[u]
do if d[v] =
then d[v] d[u] + 1
ENQUEUE(Q, v)
Key idea:
The FIFO Q in breadth-first search mimics
the priority queue Q in Dijkstra.
Invariant: v comes after u in Q implies that
d[v] = d[u] or d[v] = d[u] + 1.
2001 by Charles E. Leiserson Introduction to Algorithms Day 29 L17.38
Introduction to Algorithms
6.046J/18.401J/SMA5503
Lecture 18
Prof. Erik Demaine
Negative-weight cycles
Recall: If a graph G = (V, E) contains a negative-
weight cycle, then some shortest paths may not exist.
Example:
<0
uu vv
m . maximizing .
A x b cT x
2001 by Charles E. Leiserson Introduction to Algorithms Day 31 L18.17
Linear-programming
algorithms
Algorithms for the general problem
Simplex methods practical, but worst-case
exponential time.
Ellipsoid algorithm polynomial time, but
slow in practice.
Interior-point methods polynomial time and
competes with simplex.
Feasibility problem: No optimization criterion.
Just find x such that Ax b.
In general, just as hard as ordinary LP.
2001 by Charles E. Leiserson Introduction to Algorithms Day 31 L18.18
Solving a system of difference
constraints
Linear programming where each row of A contains
exactly one 1, one 1, and the rest 0s.
Example: Solution:
x1 x2 3 x1 = 3
x2 x3 2 xj xi wij x2 = 0
x1 x3 2 x3 = 2
0 vv11
vv99 Note:
No negative-weight
s vv44 cycles introduced
vv33 shortest paths exist.
vv77
2001 by Charles E. Leiserson Introduction to Algorithms Day 31 L18.21
Proof (continued)
Claim: The assignment xi = (s, vi) solves the constraints.
Consider any constraint xj xi wij, and consider the
shortest paths from s to vj and vi:
(s, vi)
ss vvii
minimum separation
Problem: Compact (in one dimension) the
space between the features of a VLSI layout
without bringing any features too close together.
11
2
x1 x2
Constraint: x2 x1 d 1 +
Bellman-Ford minimizes maxi{xi} mini{xi},
which compacts the layout in the x-dimension.
2001 by Charles E. Leiserson Introduction to Algorithms Day 31 L18.25
Introduction to Algorithms
6.046J/18.401J/SMA5503
Lecture 19
Prof. Erik Demaine
Shortest paths
Single-source shortest paths
Nonnegative edge weights
Dijkstras algorithm: O(E + V lg V)
General
Bellman-Ford: O(VE)
DAG
One pass of Bellman-Ford: O(V + E)
All-pairs shortest paths
Nonnegative edge weights
Dijkstras algorithm |V| times: O(VE + V 2 lg V)
General
Three algorithms today.
2001 by Charles E. Leiserson Introduction to Algorithms Day 32 L19.2
All-pairs shortest paths
Input: Digraph G = (V, E), where |V | = n, with
edge-weight function w : E R.
Output: n n matrix of shortest-path lengths
(i, j) for all i, j V.
IDEA #1:
Run Bellman-Ford once from each vertex.
Time = O(V 2E).
Dense graph O(V 4) time.
Good first try!
ii kk kk kk kk jj
(k1)
k
cik ckj(k1)
ii jj
cij(k1)
intermediate vertices in {1, 2, , k}
Notes:
Okay to omit superscripts, since extra relaxations
cant hurt.
Runs in (n3) time.
Simple to code.
Efficient in practice.
2001 by Charles E. Leiserson Introduction to Algorithms Day 32 L19.11
Transitive closure of a
directed graph
1 if there exists a path from i to j,
Compute tij =
0 otherwise.
IDEA: Use Floyd-Warshall, but with (, ) instead
of (min, +):
tij(k) = tij(k1) (tik(k1) tkj(k1)).
Time = (n3).
Lecture 20
Prof. Erik Demaine
Disjoint-set data structure
(Union-Find)
Problem: Maintain a dynamic collection of
pairwise-disjoint sets S = {S1, S2, , Sr}.
Each set Si has one element distinguished as the
representative element, rep[Si].
Must support 3 operations:
MAKE-SET(x): adds new set {x} to S
with rep[{x}] = x (for any x Si for all i).
UNION(x, y): replaces sets Sx, Sy with Sx Sy
in S for any x, y in distinct sets Sx, Sy .
FIND-SET(x): returns representative rep[Sx]
of set Sx containing element x.
2001 by Erik D. Demaine Introduction to Algorithms Day 33 L20.2
Simple linked-list solution
Store each set Si = {x1, x2, , xk} as an (unordered)
doubly linked list. Define representative element
rep[Si] to be the front of the list, x1.
Si : x1 x2 xk
rep[Si]
MAKE-SET(x) initializes x as a lone node. (1)
FIND-SET(x) walks left in the list containing x
until it reaches the front of the list. (n)
UNION(x, y) concatenates the lists containing
x and y, leaving rep. as FIND-SET[x]. (n)
2001 by Erik D. Demaine Introduction to Algorithms Day 33 L20.3
Simple balanced-tree solution
Store each set Si = {x1, x2, , xk} as a balanced tree
(ignoring keys). Define representative element
rep[Si] to be the root of the tree.
Si = {x1, x2, x3, x4, x5}
MAKE-SET(x) initializes x
as a lone node. (1) rep[Si] x1
FIND-SET(x) walks up the
tree containing x until it x4 x3
reaches the root. (lg n)
UNION(x, y) concatenates
x2 x5
the trees containing x and y,
changing rep. (lg n)
2001 by Erik D. Demaine Introduction to Algorithms Day 33 L20.4
Plan of attack
We will build a simple disjoint-union data structure
that, in an amortized sense, performs significantly
better than (lg n) per op., even better than
(lg lg n), (lg lg lg n), etc., but not quite (1).
To reach this goal, we will introduce two key tricks.
Each trick converts a trivial (n) solution into a
simple (lg n) amortized solution. Together, the
two tricks yield a much better solution.
First trick arises in an augmented linked list.
Second trick arises in a tree structure.
2001 by Erik D. Demaine Introduction to Algorithms Day 33 L20.5
Augmented linked-list solution
Store set Si = {x1, x2, , xk} as unordered doubly
linked list. Define rep[Si] to be front of list, x1.
Each element xj also stores pointer rep[xj] to rep[Si].
rep
Si : x1 x2 xk
rep[Si]
FIND-SET(x) returns rep[x]. (1)
UNION(x, y) concatenates the lists containing
x and y, and updates the rep pointers for
all elements in the list containing y. (n)
2001 by Erik D. Demaine Introduction to Algorithms Day 33 L20.6
Example of
augmented linked-list solution
Each element xj stores pointer rep[xj] to rep[Si].
UNION(x, y)
concatenates the lists containing x and y, and
updates the rep pointers for all elements in the
list containing y.
rep
Sx : x1 x2 rep
rep[Sx]
Sy : y1 y2 y3
rep[Sy]
2001 by Erik D. Demaine Introduction to Algorithms Day 33 L20.7
Example of
augmented linked-list solution
Each element xj stores pointer rep[xj] to rep[Si].
UNION(x, y)
concatenates the lists containing x and y, and
updates the rep pointers for all elements in the
list containing y.
Sx Sy : rep
x1 x2 rep
rep[Sx]
y1 y2 y3
rep[Sy]
2001 by Erik D. Demaine Introduction to Algorithms Day 33 L20.8
Example of
augmented linked-list solution
Each element xj stores pointer rep[xj] to rep[Si].
UNION(x, y)
concatenates the lists containing x and y, and
updates the rep pointers for all elements in the
list containing y.
rep
Sx Sy :
x1 x2
rep[Sx Sy]
y1 y2 y3
rep
Sx : x1 x2
rep
rep[Sx]
Sy : y1 y2 y3
rep[Sy]
2001 by Erik D. Demaine Introduction to Algorithms Day 33 L20.10
Alternative concatenation
UNION(x, y) could instead
concatenate the lists containing y and x, and
update the rep pointers for all elements in the
list containing x.
rep
x1 x2
Sx Sy : rep
rep[Sx]
y1 y2 y3
rep[Sy]
2001 by Erik D. Demaine Introduction to Algorithms Day 33 L20.11
Alternative concatenation
UNION(x, y) could instead
concatenate the lists containing y and x, and
update the rep pointers for all elements in the
list containing x.
rep
x1 x2
Sx Sy : rep
y1 y2 y3
rep[Sx Sy]
2001 by Erik D. Demaine Introduction to Algorithms Day 33 L20.12
Trick 1: Smaller into larger
To save work, concatenate smaller list onto the end
of the larger list. Cost = (length of smaller list).
Augment list to store its weight (# elements).
Let n denote the overall number of elements
(equivalently, the number of MAKE-SET operations).
Let m denote the total number of operations.
Let f denote the number of FIND-SET operations.
Theorem: Cost of all UNIONs is O(n lg n).
Corollary: Total cost is O(m + n lg n).
Lecture 21
Prof. Charles E. Leiserson
Take-home quiz
Lecture 22
Prof. Charles E. Leiserson
Flow networks
Definition. A flow network is a directed graph
G = (V, E) with two distinguished vertices: a
source s and a sink t. Each edge (u, v) E has
a nonnegative capacity c(u, v). If (u, v) E,
then c(u, v) = 0.
Example: 2
3 3
1 3 1
ss 3 2 tt
2 2
3
2001 by Charles E. Leiserson Introduction to Algorithms Day 38 L22.2
Flow networks
Definition. A positive flow on G is a function
p : V V R satisfying the following:
Capacity constraint: For all u, v V,
0 p(u, v) c(u, v).
Flow conservation: For all u V {s, t},
p(u, v) p(v, u ) = 0 .
vV vV
The value of a flow is the net flow out of the
source:
p ( s , v ) p (v, s ) .
vV vV
Skew symmetry:
f (u, v) = p(u, v) p(v, u)
= (p(v, u) p(u, v))
= f (v, u).
2001 by Charles E. Leiserson Introduction to Algorithms Day 38 L22.8
Proof (continued)
() Let
f (u, v) if f(u, v) > 0,
p(u, v) = 0 if f(u, v) 0.
2:2 2:2
3:3
| f | = f (s, V) = 4 f (V, t) = 4
c(S, T) = (3 + 2) + (1 + 2 + 3)
= 11
G: uu vv Gf : uu vv
3:5 2
Lecture 23
Prof. Charles E. Leiserson
Recall from Lecture 22
Flow value: | f | = f (s, V).
Cut: Any partition (S, T) of V such that s S
and t T.
Lemma. | f | = f (S, T) for any cut (S, T).
Corollary. | f | c(S, T) for any cut (S, T).
Residual graph: The graph Gf = (V, Ef ) with
strictly positive residual capacities cf (u, v) =
c(u, v) f (u, v) > 0.
Augmenting path: Any path from s to t in Gf .
Residual capacity of an augmenting path:
c f ( p ) = min {c f (u , v)} .
(u ,v ) p
2001 by Charles E. Leiserson Introduction to Algorithms Day 40 L23.2
Max-flow, min-cut theorem
Theorem. The following are equivalent:
1. | f | = c(S, T) for some cut (S, T).
2. f is a maximum flow.
3. f admits no augmenting paths.
Proof.
(1) (2): Since | f | c(S, T) for any cut (S, T) (by
the corollary from Lecture 22), the assumption that
| f | = c(S, T) implies that f is a maximum flow.
(2) (3): If there were an augmenting path, the
flow value could be increased, contradicting the
maximality of f.
2001 by Charles E. Leiserson Introduction to Algorithms Day 40 L23.3
Proof (continued)
(3) (1): Suppose that f admits no augmenting paths.
Define S = {v V : there exists a path in Gf from s to v},
and let T = V S. Observe that s S and t T, and thus
(S, T) is a cut. Consider any vertices u S and v T.
ss uu vv
path in Gf S T
We must have cf (u, v) = 0, since if cf (u, v) > 0, then v S,
not v T as assumed. Thus, f (u, v) = c(u, v), since cf (u, v)
= c(u, v) f (u, v). Summing over all u S and v T
yields f (S, T) = c(S, T), and since | f | = f (S, T), the theorem
follows.
2001 by Charles E. Leiserson Introduction to Algorithms Day 40 L23.4
Ford-Fulkerson max-flow
algorithm
Algorithm:
f [u, v] 0 for all u, v V
while an augmenting path p in G wrt f exists
do augment f by cf (p)
Can be slow:
109 109
G: ss 1 tt
109 109
G: ss 0:1 tt
0:109 0:109
G: ss 0:1 tt
0:109 0:109
G: ss 1:1 tt
0:109 1:109
G: ss 1:1 tt
0:109 1:109
G: ss 0:1 tt
1:109 1:109
G: ss 0:1 tt
1:109 1:109
G: ss 1:1 tt
1:109 2:109