(Warshalls) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
12.4 Sparse graphs: Johnsons algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
12.4.1 Proof weight works: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
13 Flow Networks 50
13.1 Flow network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
13.1.1 Maxow methods: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
13.2 FORDFULKERSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
13.3 EDMONDSKARP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
13.4 Maximal bipartite matching. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
14 NPCompleteness 53
14.1 Other Graph problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
14.1.1 Euler Tour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
14.1.2 Hamiltonian cycles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
14.1.3 Traveling Salesman . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
14.1.4 Constructive vs. Decision vs. Verication Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
14.2 Problem complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
14.2.1 Formal Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
14.2.2 Decision problem in Formal language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
14.2.3 Decision in polynomial time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
14.2.4 P, NP, coNP in terms of formal languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
14.3 Reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
14.4 NP, NPcomplete, NPhard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
14.4.1 How to show L NPComplete? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
14.4.2 Examples Circuit Satisability: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
14.4.3 NP completeness proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
14.5 NPC Proof for (Formula) Satisability problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
14.5.1 3CNF satisability (3SAT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3
14.6 NPC proof contd: kclique problem (CLIQUE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
15 Approx. Algorithms 58
15.1 kVERTEXCOVER problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
15.1.1 Observation: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
15.1.2 Approximation algorithm for Vertex Cover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
15.2 Hamiltonian (HAMCYCLE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
15.3 Traveling Salesman Problem (TSP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
15.3.1 Approximated TSP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
15.3.2 General TSP: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
15.4 Subsetsum problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
15.4.1 Exact solution: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
15.4.2 NP Complete proof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
15.4.3 polynomialtime approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
15.5 Set Covering problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
15.5.1 Greedy approximation for SETCOVER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4
Lecture 1
Introduction to Analysis and Design of Algorithms
Administrative
Course syllabus: http://e3.uci.edu/03f/15845/syllabus.html
1.1 Algorithm
1.1.1 Whats an algorithm?
input, output problem statement
sequence of steps
data structures
must always halt and output must satisfy property
1.1.2 Classes of algorithms
sorting (classied by problem)
tree traversal, graph algorithms (classied by data struc
tures / representation)
dynamic programming (classied by technique)
incremental, greedy, divide&conquer algorithm (clas
sied by decision criteria)
Why this class?
fundamental cross cutting applications
analysis aspect: problem complexity, what problems are
easy/hard, how to quantify
many solutions to the same problem
many applications of a given solution
1.1.3 Example: sorting problem
input: array A = [a
1
, a
2
, . . . a
n
];
output: array A
/
= [a
/
1
, a
/
2
, . . . , a
/
n
] such that a
/
1
a
/
2
. . . a
/
n
1.1.4 Many sorting algorithms available:
insertion sort
See Chapter 2 Sec. 2.1 for explanation; We will revisit sorting
later.
idea: partition A into two: [1. . . ( j 1)] already nonde
creasing, [ j . . . n] not yet sorted.
grow this sorted subarray by one more element
insert it into the tail of the sorted one, but need to make
it sorted
shift everybody up by one until the nondecreasing
property is satised again
done when j 1 = n
5 2 4 6 1 3 input A
5 2 4 6 1 3
sorted
A[1..j1]
not sorted
A[j .. n]
A[j]
5 4 6 1 3
new A[j]
j = 2
j = 3
2 5 6 1 3
2
4 j = 4
new A[j]
j =5
2 5 6 4 1 3
new A[j]
j =6
2 5 6 4 3
new A[j]
1
Output 2 5 6 4 1 3
5
1.2 Analysis
1.2.1 Execution model assumption:
RAM (random access machine) as opposed to stack,
queue, or FSM!
sequential instruction execution
alternative assumptions: parallelism, communication,
synchronization
associative memory (contentaddressable)
data structures pointers/linked list instead of copying
1.2.2 Algorithm runtime
measure it, but want a formula T(n) where n is problem
size
want to factor out processor speed as a scaling factor
worst case, best case, average case
algorithm memory consumption (visit later)
Chapter 3 deals with Order of growth (n
2
)
Insertion sort: O(n
2
)
Quick sort: O(n
2
) worst case, O(nlgn) avg.
Merge sort, Heap Sort: (nlgn)
1.3 Algorithm design
1.3.1 Approaches:
incremental
divideandconquer (e.g., MergeSort, QuickSort)
greedy (e.g., MinSpanning Tree)
backtracking
randomization
genetic
1.3.2 Example: merge sort
divide array into two halves
sort left and sort right
interleave the two halves
uses Recursion
(nlgn) complexity, where lg = log base 2
but uses at least twice as much memory in order to keep
merge linear
5 2 4 6 1 3 input A[1:6]
5 2 4
MergeSort
A[1:3]
6 1 3
MergeSort
A[4:6]
(1)
(2)
MergeSort
A[1:2]
(3)
5 2
(4)
4
5
MergeSort
A[1:1]
(5)
MergeSort
A[2:2]
2
(6)
Merge
(A[1:1],
A[2:2])
2 5
MergeSort
A[3:3]
(7)
4
(8)
Merge
(A[1:2],
A[3:3])
2 4 5 1 3 6
(9)
1 2 3 4 5 6
Merge
(A[4:5],
A[6:6])
(15)
(16)
Merge(A[1:3], A[4:6])
output A[1:6]
1.4 Problem complexity vs. Algorithm Com
plexity
Measures how hard the problem is
Bounds the complexity of all algorithms for this problem
Example: Cant do better than O(nlgn) for comparision
based sorting
But can do better (linear time) using noncomparison
based algo.
The problem itself may be hard! No efcient algorithm
exists for those problems.
NPComplete problems:
Guess an answer, verify (yes/no) in polynomial time;
otherwise exponential or worse.
Approximation algorithms: not minimum/optimal cost,
but comes within a factor.
1.5 Data Structures
xed: arrays, matrices
growable: stacks, queues, linked list, tree, heap
key addressable: hash table
structures: graphs, ow networks
1.6 Python
Scripting language, free!
Already installed on unix; download for any platform
Easy, powerful, interactive
Best way is to try it out; helps you debug algorithm
Tutorial at http://www.python.org/doc/current/tut/tut.html
6
Important features
No Semicolons, no Cstyle { }
Indentation is signicant
Loosely typed
# starts a comment, unless inside string
Interactive mode
% python
>>> 2 + 3
5
>>> x = (2 + 3) * 4
>>> x
20
>>> print "hello world"
hello world
Lists and Arrays
>>> x = [1, 2, 3]
>>> x
[1, 2, 3]
>>> x[0]
1
>>> x.append(20)
[1, 2, 3, 20]
>>> len(x) # get the number of elements
4
>>> x.pop() # pop from tail like a stack
20
>>> x
[1, 2, 3]
>>> x.pop(0) # pop from head like a queue
1
>>> x
[2, 3]
You can have other data types in lists and mix them:
>>> dayofweek = [Sun,Mon,Tue,Wed]
>>> dayofweek[3]
Wed
Hash Table (Dictionary)
Q: What if we want to look up the number from the name?
Hash tables let you use the name string as an index. How
ever, one big difference is that lists are ordered, whereas hash
tables are not.
>>> d = {} # make an empty hash table
>>> d[Sun] = 0
>>> d[Mon] = 1
>>> d[Tue] = 2
>>> d[Wed] = 3
>>> d
{Wed: 3, Tue: 2, Mon: 1, Sun: 0}
>>> d[Tue]
2
>>> d.has_key(Mon)
1 # this means true
>>> d.has_key(Blah blah)
0 # this means false
>>> d.keys()
[Wed, Tue, Mon, Sun]
7
Lecture 2
Growth of Functions, Data structures
This lecture:
big O, big , and big , how they grow
fundamental data structures
2.1 Asymptotic Notations
purpose: express run time as T(n), n = input size
assumption: n N =0, 1, 2, . . . natural numbers
2.1.1 big O = asymptotic upper bound: (page 44)
f (n) = O(g(n)) is a set of functions that are upper
bounded by 0 f (n) c g(n) for some n n
0
f (n) grows no faster than g(n)
example: 2n
2
= O(n
3
), for c = 1, n
0
2
funny equality, more like a membership: 2n
2
O(n
3
)
n
2
and n
3
are actually functions, not values.
sloppy but convenient
2.1.2 big = asymptotic lower bound (page 45)
swap the place between f (n) and c g(n):
0 c g(n) f (n) for some n n
0
example:
n = (lgn), for c = 1, n
0
= 16
2.1.3 big = asymptitic tight bounds
need two constants c
1
and c
2
to sandwich f (n).
example:
1
2
n
2
2n = (n
2
) for c
1
=
1
4
, c
2
=
1
2
, n
0
= 8
Theorem: (O and )
g(n)
f(n)
O(g(n))
0
g(n)
(g(n))
f(n)
0 0
g(n)
g(n)
f(n)
(g(n))
Example: insertion sort
O(n
2
) worst case, (n) best case
T(n) = c
1
n // for loop
+ c
2
(n1) // key A
+
n
j=2
Insert( j) // loop to insert j
+ c
8
(n1) // A[i +1] key
Best case of Insert( j) is (1), Worst is ( j)
Issue: (n
2
) +(n) is still (n
2
)
Example: MergeSort(A, p, r)
where p, r are the lower and upper bound indexes to array A.
if p < r then
q :=(p+r)/2
MergeSort(A, p, q), MergeSort(A, q+1, r)
Merge(A, p, q, r)
T(n) =
_
(1) if n = 1
2 T(
n
2
) +(n) if n > 1
T(n) is expressed in terms of T itself! This is called recur
rence. We want a closedform solution (no T on the righthand
side of the equation).
To solve recurrence:
rewrite as T(n) =
_
c if n = 1
2 T(
n
2
) +c n if n > 1
T(n) = 2T(
n
2
) +cn Expand
= 2(2T(
n/2
2
) +c
n
2
) +cn
= 4T(
n
4
) +2c
n
2
+cn = 4T(
n
4
) +2cn
= 8T(
n
8
) +3cn = . . .
= (2
i
)T(
n
2
i
) +i cn
We can divide n by 2 at most lgn times before n = 1. So,
T(n) = (lgn) cn+cn = (nlgn).
Chapter 4 will showhowto solve recurrence systematically.
2.1.4 littleo, little not asymptotically tight
o(g(n)) = f (n) : c > 0n
0
> 0 : 0 f (n) < cg(n) n n
0
8
Example
2n
2
= O(n
2
) is a tight bound 2n
2
,= o(n
2
).
2n = O(n
2
) is not tight 2n = o(n
2
).
does not make sense to have little
2.2 Polynomials, log, factorial, bonacci
2.2.1 Polynomial in n of degree d:
p(n) =
d
i=0
a
i
n
i
= (n
d
)
Polynomially bounded: f (n) = O(n
k
) for constant k.
2.2.2 logarithm
useful identity: a
log
b
c
= c
log
b
a
.
polylogarithmically bounded if f (n) = O(lg
k
n) for con
stant k
grows slower than any polynimial: lg
b
n = o(n
a
) for any
constant a > 0.
logstar: lg
n = mini 0 : lg
(i)
n 1
(log of log of log ... of log of n) grows very very slowly.
Example: lg
(2
65536
) = 5.
2.2.3 Factorial:
Loose bound: n! is faster than 2
n
but slower than n
n
precisely, n! = (n
n+1/2
e
lgn
) (Stirlings approxima
tion)
lg(n!) = (nlgn)
2.2.4 Fibonacci
grows exponentially
2.2.5 misc
2
2
n
grows faster than n!
(n+1)! grows faster than n!
n! grows faster than n 2
n
faster than 2
n
2
lgn
= n
2.3 Python: Functions
function denition: def functionName (paramList) :
variables have local scope by default
control ow: for, while, if
range(a, b) function: returns the list [a, ... b1]
>>> range(1,5)
[1, 2, 3, 4]
Algorithm from book (p.24) Python code (le isort.py)
INSERTIONSORT(A) def InsertionSort(A):
1 for j 2 to length[A] for j in range(1, len(A)):
2 do key A[ j] key = A[j]
3 i j 1 i = j 1
4 while i > 0 and A[i] > key while (i>= 0) and (A[i]>key):
5 do A[i +1] A[i] A[i+1] = A[i]
6 i i 1 i = i 1
7 A[i +1] key A[i+1] = key
Dont forget the colon : for def, for, and while constructs
Book A[1. . . n] Python A[0. . . n1]
Books 2. . . n Python 1. . . n1.
line 1: range(1,len(A))[1...len(A)1]
% python
>>> from isort import * # read le isort.py
>>> x = [2,7,3,8,1] # create test case
>>> InsertionSort(x) # call routine
>>> x # look at result
[1, 2, 3, 7, 8]
Works for other data types, too!
2.4 Data Structures Review
Concepts:
Stacks: Lastin Firstout
Queue: Firstin Firstout
Trees: Root, children
2.4.1 Array implementation
Stack S: need a stack pointer (top)
Push(x): S[++top] x
Pop(): return S[top]
Note: Actually stacks can grow downward also.
Note: To be safe, check stack underow/overow.
Queue Q: need 2 pointers (head, tail)
Enqueue(x): Q[tail] x
tail (tail mod len(Q)) + 1
Dequeue(): temp Q[head]
head (head mod len(Q)) + 1
return temp
Note: To be safe, check queue underow/overow.
2.4.2 Stack and Calls/Recursion
Calling routine Push local, param, return address
Return Pop and continue
9
Example: MergeSort (cf. MergeSort Fig. from last lecture)
L (Left), R (Right), M (Merge)
(1) (2) (3) (4) (5) (6) (7) (8)
[1, 6] [1, 6]L
[1, 3]
[1, 6]L
[1, 3]L
[1, 2]
[1, 6]L
[1, 3]L
[1, 2]L
[1, 1]
[1, 6]L
[1, 3]L
[1, 2]R
[1, 6]L
[1, 3]L
[1, 2]R
[2, 2]
[1, 6]L
[1, 3]L
[1, 2]M
[1, 6]L
[1, 3]R
[1, 6]L
[1, 3]R
[3, 3]
[1, 6]L
[1, 3]M
[1, 6]R
2.4.3 Linked List x
Singly linked (next) or Doubly linked (next, prev)
need a key
need a NIL value
How to implement linked lists?
Preallocate arrays prev[1 : n], next[1 : n], key[1 : n], and
x is an index 0. . . n, where 0 is NIL.
Records (structures in C, classes in Java): x.prev,
x.next, x.key, and NULL for NIL.
Rearrange by adjusting links, not moving data (key)
Recall: INSERTIONSORT with array need to copy object
to insert.
Linked list representation: once we know where to in
sert, adjust link in O(1) time.
Array version: best case O(1), worst case O(n), average
case O(
n
2
) = O(n).
2 5 6 4 1
2 5 6 4 1 6
2 5 4 1 6 5
2 5 4 1 6 4
2 5 3 1 6 4
2 5 6 4 1
insert(3)
3
2 5 6 4 1
(1) alllocate a
node for the
data
head
insert(3)
(2) adjust links
3
Search for the item
e.g., where to insert?
list is unsorted, worst/average case O(n) time to nd it!
(compare keys one at a time).
list is sorted:
Array: binary search in O(lgn) times
Singly linked list: O(n) avg/worst case, need to do
linear search
Doubly linked list: does not help! still O(n) cant
jump to the middle of the list, because link is to the
head/tail.
2.4.4 Binary search tree
pointer to root; each node has pointer to leftchild, right
child
optional pointer to parent
left smaller, right bigger key value
O(lgn) search, if BALANCED
Can still be O(n) if not balanced!!
Redblack tree (Ch.13) is a way to keep BST balanced
a Heap (Ch.6) implements balanced BST using array; no
pointers
18
2 6 13 4 1
head tail
18 20
6
root
Doubly
Linked
List
Binary
(Search)
Tree
2
1 4 13 20
nary tree
w/ leftchild,
rightsibling
18
6
root
2
1 4 13 20
2.4.5 rooted trees w/ unbounded branches
Could have arbitrary number of children!
leftchild: head of linked list to children (down one level)
rightsibling: link to next on same level
optional link back to parent
10
Lecture 3
Recurrences; QuickSort
This lecture:
Chapter 4: Recurrences, Master method
Substitution method
Recursion tree
Master method: T(n) = aT(n/b) + f (n)
Quicksort (Ch. 7.17.2) (if we have time today)
Summation review (Appendix A)
3.1 Recurrence: by Substitution method
guessandverify
rename variables if necessary.
mathematical induction: assume true for n
0
, show true
for n n
0
.
issue: boundary conditions, oor/ceiling use inequal
ity to bound those cases.
avoid pitfall with constant terms.
Example w/ oor
T(n) =
_
(1) if n = 1,
2T(n/2) +n if n > 1.
Guess T(n) = O(nlgn).
sufcient to show T(n) cnlgn for some c > 0.
Assume T(n/2) cn/2lg(n/2).
T(n) 2(cn/2lg(n/2)) +n substitution
cnlg(n/2) +n drop oor
= cnlgncnlg2+n (lg(a/b) = lgalgb)
= cnlgncn+n (lg2 = 1)
cnlgn c 1
.
Avoid Pitfalls
What if we guessed T(n) = O(n)??
Try to show T(n) cn
T(n) 2(cn/2) +n subst. into T(n) 2T(n/2) +n
cn+n drop oor
= (c +1)n factor out const
,cn Fails!
.
must be careful not to drop constants arbitrarily.
subtractive term in guessed solution
Example: T(n) = T(n/2) +T(n/2) +1
Guess: O(n). First attempt: verify T(n) cn:
T(n) cn/2 +cn/2 +1
= cn+1 ,cn
Oops! But this does not mean T(n) ,= O(n).
works if we guess T(n) cnb for some b 0:
T(n) (cn/2 b) +(cn/2 b) +1
= cn2b+1
cnb works for b 1.
11
Variable Renaming
introduce a new variable to replace a difcult expression, make
it easier to solve
Example T(n) = 2T(
n) +lgn
Rename: let m = lgn: T(n = 2
lgn
= 2
m
) = 2T(2
m
2 ) +m
Rename: let S(m) = T(2
m
) S(m) = 2S(
m
2
) +m
This looks like S(m) = O(mlgm)
Substitute back, lgn = m: T(2
m
= 2
lgn
= n) = S(m)
= O(mlgm) = O((lgn)lg(lgn)).
3.2 Recursion tree, convert to summation
Example: T(n) = 3T(n/4) +(n
2
).
Visualize as recursion tree: T(n) = 3T(n/4) +cn
2
for
some c > 0
cn
2
T(n/4) T(n/4)
T(n/4) = c(n/4)
2
T(n/16) T(n/16) T(n/16)
cn
2
(3/16) cn
2
T(1) T(1) T(1)
Convert to summation:
T(n) = cn
2
+
3
16
cn
2
+(
3
16
)
2
cn
2
+ +(
3
16
)
Levels1
+width
# Levels = 1 +log
4
n, similar argument to MergeSort.
root is 1 level, divide log
4
n times.
width at level i is 3
i
, so at the bottom level, width is
3
log
4
n
.
Apply the identity a
log
b
c
= c
log
b
a
, rewrite 3
log
4
n
=
n
log
4
3
.
T(n) =
(log
4
n)1
i=0
(
3
16
)
i
cn
2
+(n
log
4
3
)
<
i=0
(
3
16
)
i
cn
2
+(n
log
4
3
)
apply geometric series
k=0
x
k
=
1
1x
=
1
1(3/16)
cn
2
+(n
log
4
3
)
=
16
13
cn
2
+(n
<1
)
= O(n
2
).
Can also show T(n) = (n
2
) and therefore T(n) = (n
2
).
Unbalanced Recursion Tree
Example: T(n) = T(n/3) +T(2n/3) +O(n)
Root level: cn
Next level: c
n
3
+c
2n
3
= cn
Tree height: How many times can we divide n into
1
3
and
2
3
?
log
3
n times for
1
3
(most shallow branch), and
log
3/2
n times for
2
3
division (deepest branch).
# nodes at level i = 2
i
(binary tree)
# nodes at bottom level < 2
log
3/2
n
= n
log
3/2
2
(not a full binary tree)
Note log grows slower than any polynomial, so
n
log
3/2
2
= n
log
1.5
2
= n
>1
= (nlgn).
Total cost = cn #levels +#leaves
= O(cnlog
3/2
n) +(nlgn) = O(nlgn).
Can use substitution method to verify upper bound:
T(n) dnlgn: (see book for proof)
3.3 The Master Method
General form: T(n) = aT(n/b) + f (n), a 1, b > 1
intuition: divideandconquer
away branch (recursive calls) per level,
each subproblem size = n/b,
Combine cost is f (n).
#levels = log
b
n
#nodes(at level i) = a
i
#nodes(at leaf level) = a
log
b
n
= n
log
b
a
Example: MERGESORT
a = 2, b = 2, f (n) = (n) oor/ceiling dont matter
12
Three cases, compare f (n) to n
log
b
a
if f (n) = then T(n) =
1. O(n
(log
b
a)
) (n
log
b
a
)
2. (n
log
b
a
) (n
log
b
a
lgn).
3. (n
(log
b
a+
) ( f (n)).
and af (n/b) c f (n)
for const c < 1, n n
0
,
Restated:
1. if f (n) is polynomially slower than #leaves = n
(log
b
a)
,
then
# leaves dominate cost; its (a
log
b
n
) = (n
log
b
a
)
2. f (n) (conquer) and n
log
b
a
(divide) are equally fast,
(with lgn levels) then its (n
log
b
a
lgn).
3. The conquer cost f (n) is dominant.
Important note:
Master method does not cover all recurrences!
f (n) and n
log
b
a
comparison must be differ polynomially
(i.e., n
n
k=1
k =
1
2
n(n+1) = (n
2
).
Happens when list already sorted!
Prove more strictly:
13
T(n) = max
0qn1
(T(q) +T(nq1)) +(n)
max
0qn1
(cq
2
+c(nq1)
2
) +(n)
= c max
0qn1
(q
2
+(nq1)
2
) +(n)
c(n1)
2
+(n) = c(n
2
2n+1) +(n)
= cn
2
c(2n1) +(n)
cn
2
3.4.2 Best case: balanced partitioning
T(n) 2T(n/2) +(n)
Can apply 2nd case of the Master theorem:
if f (n) = (n
log
b
a
), then T(n) = (n
log
b
a
lgn).
T(n) = (n
log
2
2=1
lgn) = (nlgn)
3.4.3 Average case
(next time)
14
Lecture 4
Heap sort, Priority Queues
This lecture:
Quick sort (continued from last time)
Heap property, Heap sort
Python implementation (for homework)
4.1 Heap data structure: Binary heap
nearly balanced binary tree
use array instead of pointer data structure
Denition:
parent(i) =i/2
leftChild(i) = 2i
rightChild(i) = 2i +1
heap property
by default: Max Heap: A[parent(i)] A[i]
Min Heap: A[parent(i)] A[i]
height of heap as a binary tree is (lgn).
x
x
(max) heap
zs Left and R
both z
x
y z
x
y z
ys L and R
both y
1
2, 3
4, 5, 6, 7
y, z both x
index
Examples: Are these (max)heaps?
[7, 5, 6, 2, 3, 4]?
(a) 7 5, 6; (b) 5 2, 3; (c) 6 4 Yes.
[7, 6, 5, 4, 3, 2]?
(a) 7 6, 5; (b) 6 4, 3; (c) 5 2 Yes.
[2, 5, 7, 4, 3, 6]?
No. However, it is almost a heap if we dont consider
the root: (b) 5 4, 3; (c) 7 6.
This can be xed with a call to HEAPIFY.
Heap API
HEAPIFY(A, i) (incremental x) assume L and R subtrees
of i are heaps, make A[i . . . n] a heap.
BUILDHEAP(A) input unsorted array, rearrange to make it a heap
HEAPSORT(A) sorting alg., uses HEAPIFY and BUILDHEAP
4.2 Heapify:
Assumption: called when L(i), R(i) are heaps, but
A[i] itself isnt its children
after call: propagate the largest of children up to root
position, so that the whole thing is a heap again.
MAXHEAPIFY(A, i, n) (see page 130, slightly different)
1 l LEFT(i) index for left child
2 r RIGHT(i) index for right child
3 if (l n) and (A[l] > A[i])
4 then largest l left child exists and is bigger
5 else largest i either no L or root is bigger
6 if (r n) and (A[r] > A[largest])
7 then largest r right child exists and is bigger
no else clause: either no R or root is bigger.
already taken care of by line 5
8 if largest ,= i
9 then exchange A[i] A[largest]
10 MAXHEAPIFY(A, largest, n)
2
5 7
4 3 6
index
1
2, 3
4 7
largest
7
5 2
4 3 6
largest
7
5
2 4 3
6
done
Correctness: by induction.
15
Time complexity of Heapify
O(lgn) binary tree argument.
Or, O(h) where h = height of the heap as a binary tree.
4.3 BuildHeap
input unsorted array, output a heap after the call
leaf = trivial heap, start with one level above the leaf
BUILDMAXHEAP(A) (page 133)
1 n length[A]
2 for i n/2 downto 1
3 do MAXHEAPIFY(A, i, n)
7
2
3 4
5 6 7
index
1
2, 3
4 7
7
6
2 5 3
4
done
HEAPIFY
2
3
5 6 4
2
6 7
5 3 4
HEAPIFY
2
3 4
5 6 7
trivial heaps
1
2, 3
4 7
HEAPIFY
Build (Max) Heap:
7
6
5 3
2
4
largest
largest
Correctness: assumption for Heapify always hold.
4.3.1 Complexity:
loose upper bound is O(
n
2
lgn) = O(nlgn)
MAXHEAPIFY is O(lgn) iterate n/2 times.
However! tighter bound is O(n).
HEAPIFY near leaf level is much cheaper than near root
level!
Height of nelement heap =lgn. (binary tree)
# nodes at level h is n/2
h+1

Height 0 (leaf) n/2
0+1
 =n/2 nodes.
One level higher half as many nodes
HEAPIFY called on a node at height h is O(h) (see last
page if you forgot)
T(n) =
lgn
h=0
(#of HEAPIFY calls at level h)
Time of HEAPIFY at level h
=
lgn
h=0
(
n
2
h+1
)O(h), factor out n, mult. by 2,
= O(n
lgn
h=0
h
2
h
).
Apply the integrating and differentiating series (Ap
pendix A)
k=0
kx
k
=
x
(1x)
2
.
plug in x =
1
2
, we get
h=0
(
h
2
h
) =
(1/2)
(11/2)
2
= 2.
T(n) = O(n 2) = O(n)
4.3.2 HeapSort
Use HEAPIFY to nd the largest, stick it in the end A[n]
Use HEAPIFY on [1. . . n 1] to nd the second largest,
stick it in position A[n1]
HEAPSORT(A) (page 136, slightly different)
1 BUILDMAXHEAP(A, n) O(n) time
2 for i n downto 2 n1 times
3 do exchange A[1] A[i] O(1) time
4 MAXHEAPIFY(A, 1, i 1) O(lg(i 1)) time
Total time:
O(n)BUILDHEAP
+ O(n)loop O(lgn)HEAPIFY
= O(nlgn)
7 6 2 5 3 4
input: [2, 3, 4, 5, 6, 7]
Line 1: BUILDMAXHEAP
Line 2: loop, i = 6
heap
Line 3: swap, A[1], A[i]
7 6 2 5 3 4
sorted almost a heap
(root is wrong, but
children are OK)
Line 4: Heapify(A, 1, 5)
7 5 6 2 3 4
7 5 3 2 6 4
heap again sorted
Line 3: swap, A[1], A[i]
Line 2: loop, i = 5
sorted almost a heap
Line 4: Heapify(A, 1, 4)
7 3 5 2 6 4
Line 2: loop, i = 4
heap again
sorted
Continue until the whole array is sorted.
4.4 Python Programming
See http://e3.uci.edu/01f/15545/heaphint.html; or for PDF, try
http://e3.uci.edu/01f/15545/heaphint.pdf
4.5 Priority Queues
want to get the highest priority element,
MAXHEAPINSERT(A, x)
EXTRACTMAX(A)
16
Implementation choices:
unsorted: O(n) to extract, O(1) to insert (anywhere)
sorted linked list: O(1) to extract, O(n) to insert
Heap: O(lgn) to insert , O(lgn) to extract
4.5.1 Heap Insert:
assume A is already a heap
increase size of A by one, put newelement at the end, and
propagate up until its parent is larger (or its the largest)
MAXHEAPINSERT(A, key) (diff. from book page 140)
1 heapsize[A] heapsize[A] +1
2 i heapsize[A]
3 while i > 1 and A[Parent(i)] < key
4 A[i] A[Parent(i)]
5 i Parent(i)
6 A[i] key
7
7
6
5 3 2
key = 8
7
6
5 3 2
key = 8 7
6 4
5 3 2
make room
MaxHeapInsert(8)
4 4
4 7
8
6
5 3 2
key = 8
4
O(lgn) time (climb up from tail to at most root)
assume short circuit evaluation (i.e., check index, if i ,>1
then A[parent(i)] is not even evaluated!
Even though the algorithm says
heapsize[A] heapsize[A] +1,
you probably cant do that in any programming lan
guage. You need to actually make room for A[n +1], or
else your program can crash. In Python, call A.append
to increase physical size of A.
in Python, heapsize[A] can be derived from len(A),
rather than a global variable.
4.5.2 Extract max:
pull the max (already at the root)
move the last one to the front and heapify (since both
children are heaps)
in Python, use A.pop to accomplish heapsize[A]
heapsize[A] 1.
HEAPEXTRACTMAX(A)
1 if (heapsize[A] < 1)
2 then error heap underow;
3 max A[1]
4 A[1] A[heapsize[A]]
5 heapsize[A] heapsize[A] 1
6 MAXHEAPIFY(A, 1)
7 return max
4.5.3 Heap Increase key
HEAPINCREASEKEY(A, i, key)
A is a heap, i is an index to an element.
Purpose: overwrite element A[i] with key (which is ex
pected to be larger than A[i] hence increase key),
while making minor changes to restore its heap property.
Strategy: propagate A[i] up towards the root, similar to
MAXHEAPINSERT.
HEAPINCREASEKEY(A, i, key) (page 140)
1 if key < A[i]
2 then error new key is smaller than current key
3 A[i] key
4 while i > 1 and A[Parent(i)] < A[i]
5 do exchange A[i] A[Parent(i)]
6 i Parent(i)
not a common routine in priority queues; used later in
minspanning tree algorithms
very unusual to implement MAXHEAPINSERT as
shown on page 140.
implies a max heap. (HEAPDECREASEKEY would im
ply a min heap.)
timing complexity similar to HEAPINSERT except no re
allocation of storage size
17
Lecture 5
Linear Time sorting, Medians/Order Stat
This lecture
problem complexity of comparisonbased sorting (8.1)
noncomparison sorting: counting sort (8.2), radix sort
(8.3), bucket sort (8.4)
5.1 Comparisonbased sorting: prob. complex
ity
Decision Tree argument
Use a tree to represent all possible execution traces of an
algorithm (Fig. 8.1)
1:2 a[1] a[2]
<1, 2>
2 : 3
a[1] a[2] a[3]
<1, 2, 3>
a[2] a[3]
1 : 3
a[1] a[3] < a[2]
<1, 3, 2>
a[1] a[2]
a[3] < a[2]
a[1] a[3]
a[3] < a[1] a[2]
<3, 1, 2>
a[3] < a[1]
1 : 3
2 : 3 <2, 1, 3>
<2, 3, 1> <3, 2, 1>
>
>
a[1] > a[2]
<2, 1>
twoway branch on >, <, , or pairwise comparison
operator
Q: How many ways can the elements be ordered in a list of size
n?
n! ways (nfactorial, permutation)
rst position: n choices; second position: n1 choices..
leaf of tree algorithm nishes
path length from root to leaf number of steps (leaf
itself is just the result)
# leaves in decision tree:
# leaves # of possible outcomes.
# of leaves n!
less efcient algorithm would actually have more leaves
(redundant comparisons)
Runtime bounded by tree height: T(n) = (h)
roottoleaf path length = height = lower bound on run
time
a decision tree must have at least n! leaves,
binary tree of height h has 2
h
leaves (upper bound).
n! 2
h
. Take log on both sides,
lg(n!) lg(2
h
) = h.
from Ch. 3, page 55, lg(n!) = (nlgn) using Stirlings
approximation
h = (nlgn).
[Corollary 8.1] Any comparison sort algorithm requires
(nlgn) comparisons in the worst case.
[Corollary 8.2] HEAPSORT, MERGESORT, median partitioned
QUICKSORT are asymptitic optimal comparison sorts.
5.2 Noncomparison sorts
domain knowledge: e.g., exam scores, SAT scores, etc.
Clearly a lot more students than #of possible scores.
observation: sorting is based on a key of an object,
which might not be a unique identier
example of key: exam score; weight, height; income;
5.2.1 Counting sort
Idea:
1. count how many times each key occurs
2. convert the counts to offsets (to the end)
3. from the back, copy each key from input to ouput buffer
based on the offset; decrement
A[1..n] original source array
B[1..n] output array
18
C[1..k] key count; overloaded to be index
Initialize C[1..k] 0
foreach A[i]
do C[A[i]] ++
foreach i 2 to k
do C[i]+ =C[i 1]
computes Offset[i] := Offset[i 1] +C[i]
converts C from count to index to the end of its region
for i n downto 1 do
B[Offset[i] ] A[i]
2 5 3 0 2 3 0 3 A
input
count #times each key appears in input
2 2 3 1
0 1 2 3 4 5
C
output
layout
2 5 3 0
2 2 3 1
index to end
of region
C
2 4 7 8
Copy from
A[i] to B[C[i]]
in reverse
order
B 3
3 0
3 0 3
2 5 3 0 2 3 0 3 A
3 0 3 2
3 0 3 2 0
3 0 3 2 0 3
3 0 3 2 0 3 5
3 0 3 2 0 3 5 2
6
1
5
3
0
4
7
2
stable sort (relative positions preserved)
uses 2 buffer
complexity = O(n+k), effectively linear if k = O(n).
5.2.2 RadixSort
Idea: For keys that have digits
use stable sorting to sort by one digit at a time
start from least signicant digit!
3 2 9
4 5 7
6 5 7
8 3 9
4 3 6
7 2 0
3 5 5
input
Stablesort by
least significant
digit first
7 2 0
3 5 5
4 3 6
4 5 7
6 5 7
3 2 9
8 3 9
Stablesort by
2nd LSD
7 2 0
3 2 9
4 3 6
8 3 9
3 5 5
4 5 7
6 5 7
Stablesort
by MSD
3 2 9
3 5 5
4 3 6
4 5 7
6 5 7
7 2 0
8 3 9
Important!! does not work if you start from MSD!!!
3 2 9
4 5 7
6 5 7
8 3 9
4 3 6
7 2 0
3 5 5
input
Stablesort by
MSD first
Stablesort by
2nd MSD
7 2 0
3 2 9
4 3 6
8 3 9
3 5 5
4 5 7
6 5 7
Stablesort
by MSD
3 2 9
3 5 5
4 3 6
4 5 7
6 5 7
7 2 0
8 3 9
3 2 9
3 5 5
4 5 7
4 3 6
6 5 7
7 2 0
8 3 9
Oops! Not sorted
Need to do a lot of bookkeeping if really want to sort from
MSD.
3 2 9
4 5 7
6 5 7
8 3 9
4 3 6
7 2 0
3 5 5
input
Stablesort by
MSD first
Stablesort by
2nd MSD
7 2 0
3 2 9
4 3 6
8 3 9
3 5 5
4 5 7
6 5 7
Stablesort
by MSD
3 2 9
3 5 5
4 3 6
4 5 7
6 5 7
7 2 0
8 3 9
3 2 9
3 5 5
4 5 7
4 3 6
6 5 7
7 2 0
8 3 9
also useful for sorting by multiple keys by name, by
SAT score, etc. start from the secondary before the
primary key
runtime complexity: (d(n+k)), where
d = # of digits in key,
k = radix (# possible values per digit)
reason: make d passes. Each pass is (n + k) (e.g.,
counting sort)
5.2.3 Bucket sort
assume values are distributed within some range.
project the value into n buckets, (insertion) sort each
bucket
input
78 17 39 26 72 94 21 12 23 68
bucket
#
7 1 3 2 7 9 2 1 2 6
1
2
3
9
12 17
21 26 23
.
.
.
39
94
0
Timing Complexity: expected to be linear (to be revisited)
19
5.3 Median and Order statistics
5.3.1 Find min? max? both min and max?
n1 comparisons for min (keep the smallest so far)
also n1 comparisons for max
both min/max: 2n2
but could do better: 3n/2 comparisons
1. compare pairs (n/2 comparisons)
2. search for min in the smaller set
3. search for max in the bigger set
3 7 5 8 6 1 9 4
input
3
7
5
8 6
1
9
4
smaller
bigger
n/2 for pairwise compare
n/2 1 for min
n/2 1 for max
1
9
5.3.2 How to nd the ith smallest w/out sorting?
keep track of min & max so far
nd the median
Use QuickSorts partitioning:
each pass: pivot is in the correct rank (absolute sorted
position)
if pivot is the position we want, return it
otherwise, continue partitioning in either one of the two
intervals.
RANDOMIZEDSELECT(A, p, r, i)
1 if p = r
2 then return A[p]
3 q RANDOMIZEDPARTITION(A, p, r)
4 k qp+1
5 if i = k the pivot value is the answer
6 then return A[q]
7 else if i < k
8 then return RANDOMIZEDSELECT(A, p, q1, i)
9 else return RANDOMIZEDSELECT(A, q+1, r, i k)
Complexity of (randomized) Select:
Best case: O(n) rst pass, very lucky, pivot is what we
want
Even partitioning: T(n) + T(n/2) + T(n/4) + ... +
T(2) +T(1) = T(2n) = O(n)
Worst case: T(n) +T(n1) +T(n2).. = O(n
2
)
Average time: use Randomized
1/n probability of each position i as pivot. expected cost
= sum up probability cost of i as pivot
T(n) =
n1
i=1
1
n
T(max(i, ni)) +O(n)
=
n1
i=n/2
2
n
T(i) +O(n)
Guess T(n) an,
Verify
T(n)
2
n
n1
i=n/2
(a i) +cn
2
n
n
2
(a(n1) +an/2
2
+cn =
3
4
an+cn.
Choose a = 4c O(4n) = O(n)
5.3.3 Can show that worstcase select is O(n)
(Theoretical interest only)
Divide n elements into sublists of LENGTH 5 each, for
n/5 lists
Find the medians from these lists. Brute force (constant
time) on each list.
Recurse to nd the median x of these n/5 medians.
Partition around x. Let k = rank(x)
if (i = k) return x
if (i < k) then Select recursively to nd ith smallest in
rst part
else Select recursively to nd (i k)
th
smallest in second
part.
5.3.4 Justication:
(x is dont know, s = smaller, L = larger)
x x x L L L L
x x x L L L L
s s s * L L L
s s s s x x x
s s s s x x x
<
<
input
n elements
medians
of 5
median of medians
larger
smaller
larger smaller
< < < < < <
<
<
<
<
<
<
<
<
<
<
<
<
<
<
?
?
20
Claim: the median of medians of a list of length n has a
rank between 3n/106 and 7n/10+6.
Proof: There are (1/2 n/5) 1 columns to the left
and right of the MofM.
The number of elements < or > than the MofM is
3#of columns to the left and right, respectively, plus
two in the MofMs own column.
Its rank is 3(
1
2
n/5 2) 3n/106
y x
y
LEFTROTATE(x)
RIGHTROTATE(y)
Right rotate:
x is left child of y, want to make y the right child of
x.
x gets to keep its left child as before.
xs old right tree (which all have values x <
t < y) now becomes ys left child (i.e. promoted up
to xs old position)
(1) time! just x up pointers.
Acutally, can also think in terms of Rotate Up/Down:
Left Rotate x: x goes down to left child, y got ro
tated Up to be new root
Right Rotate x: x was the left child, now gets pro
moted one level up to root position; at the same
time y gets rotated down from root position.
6.2 Python Programming with Trees
objectoriented programming: Tree and Node classes
tuple encoding
6.2.1 Node data structure
class Node:
# constructor
def init (self, key):
self.key = key
self.left = None
self.right = None
self.p = None
To instantiate, call the constructor (name of the class) with
the params:
>>> x = Node(3) # makes a new node with key = 3
>>> x.key
3
Note that calling Node(3) actually calls Node classs
init (constructor) routine.
Use x.key, x.left instead of key[x], left[x], etc.
We can insert additional elds even after the constructor
is called!
example: x.color = red # add a new color at
tribute
Example routine:
def TreeMaximum(x):
while x.right != None:
x = x.right
return x
6.2.2 Tupleencoded Trees
Tuples vs Lists:
list: L = [1, 2, a, b, c ]
tuple: T = (1, 2, a, b, c)
Similarities:
Both tuples and lists let you access elements by their in
dices: e.g. L[3] and T[3] both yield the value of b.
len works for both. Both len(L) and len(T) yield
5.
Differences:
syntax: tuples use ( ); lists use [ ]
tuples are immutable (read only)
not allowed to say T[3] = z
OK to do lists L[3] = z
tuples can be compared lexigraphically:
(1, 2) < (1, 2, 3) < (1, 3)
Can have tuples within tuples:
(1, (2, 3, (4, 5)), 6)
24
Tuples as Tree Nodes
Preorder encoding: (key, left, right)
Example: (5, (3, (2, None, None), (5, None, None)), (7,
None, (8, None, None)))
5
3
2 5
7
8
convenient way of specifying a tree test case
Also for printing out the trees
To convert between tree data structure and tuples:
def TupleToTree(t):
if t == None:
return None
# make a new node
n = Node(t[0])
n.left = TupleToTree(t[1])
n.right = TupleToTree(t[2])
return n
def TreeToTuple(x):
if x == None:
return None
return (x.key,
TreeToTuple(x.left),
TreeToTuple(x.right))
6.3 RedBlack Trees
6.3.1 Five Properties: (important!!)
1. Every node is either red or black.
2. The root is black.
3. (conceptual) Leaf is black actually, there are real
nodes vs. conceptual nodes. All real nodes have two
children. A NIL is not a real node and is the only kind of
node that can have no children.
4. Red node has two black children (could be NIL)
5. All simple paths from a node to a descendant leaf contain
the same number of black nodes. (black height)
Denition: black height bh(x)
# of black nodes on a path to a leaf, not including x itself.
a heighth node x has blackheight bh(x) h/2.
This is by property 4: since you cant see two red nodes
in a row, the most you can see is every other red, or h/2.
number of vertices n is bounded by the height h:
upperbound: full tree w/ red/blacks: n 2
h
1
lower bound: black only: n 2
bh(x)
1.
but since bh(x) h/2, we have
n 2
h/2
1.
rewrite lower bound:
n+1 2
h/2
, take log on both sides
lg(n+1) h/2, move 2, swap sides
h 2lg(n+1)
6.3.2 How to maintain the RedBlack Tree proper
ties?
Example: color scheme?
7
5 9
12
7
5 9
12
(a) This is OK.
7 is root (forced
black); 12 red pre
serves bh
7
5 9
12
(b) not OK
7
5 9
12
(c) Is this one
ok?
Insertion using BST routines
insert(8) (as new leaf) to tree (a) above:
left child of 9.
need to color it red like 12.
7
5 9
12 8
insert(11) as left child of 12. What color should 11 be?
7
5 9
12 8
11
what color?
7
5 9
12 8
11
7
5 9
12 8
11
bh = 4
everywhere else bh = 3
red nodes must have
two black children!
if red 12 is red... but 12 must have two black
children. (property 4)
if black 12 black height thru 11 is longer than
other paths (violates property 5)
Solution: recolor the tree!
7
5 9
12 8
11
insert as red
change color
scheme
But! what if we Insert(10)? (left child of 11)
25
7
5 9
12 8
11
10
no feasible coloring
scheme!
need structural x
want to x it in O(lgn) time.
6.3.3 Insertion
call BST Insert z as a red leaf.
black height didnt change
if the new node z has a black parent, done.
only possible violation: z and its parent both red
To x it: (1) recoloring, (2) rotating
3 cases for Insertion xup
problem: Two reds on adjacent levels
approach: separate the two reds with a black by swaping
colors from above if possible. push the red bubble up
towards the root)
given parent[z] = red Grandma[z] must be black.
7
5 9
12 8
11
10
insert(z)
as red
grandma[z]
must be black!
?
red: case 1,
=> swap color levels
black: case 2 or 3
=> rotate followed by color swap
{
Case 1: grandma has two red children
solve by colorlevel swapping
We didnt violate black height!
we can push the red level up the tree, by swapping level
colors.
7
12
11
10
insert(z)
as red
grandma[z]
12
11
10
insert(z)
as red
push
red up
these
are
OK
}
}
if great
grandma is
red, then
need more
fixing
great
grandma[z]
However, weve moved the problem somewhere else.
Grandma is now red could have twoconsecutive red.
climb up the tree to x it.
Cases 2 and 3: mom is red, aunt is black
solve by Rotations followed by color swapping
Case 2: (Grandma, mom, myself) are related as (Lchild,
Lchild) chain or (Rchild, Rchild) chain
1. uprotate mom to grandmas position
2. problem: mom and I are still both red! solution: swap
mom and (original) grandmas colors
12
11
10
insert(z)
as red
grandma[z]
Step 1: rotate zs mom up to
grandmas position
aunt[z]
is black
12
11
10
Step 2: swap colors betw.
mom & grandma (original)
12
11
10
Tree is OK
Case 3: (Grandma, mom, myself) are either LR or RL
chain
1. uprotate myself to moms position.
2. This transforms it into an LL or RR chain, which is
case 2. Continue as case 2.
12
10
11
insert(11)
as red
grandma[z]
Step 1: uprotate z up to its
moms position
aunt[z]
is black
12
11
10
insert(z)
as red
grandma[z]
Step 2: we just transformed
case 3 into LL chain or R
R chain,(which is case 2)
so we resume as case 2..
aunt[z]
is black
6.3.4 Deletion
Modied based on the BST delete routine
4 cases instead of 3 when xup
13.4 as optional reading.. dont worry about details.
6.3.5 Tree data structure
maintains the root
to act as a frontend to the nodes, and serves as a good
place to group together related routines as methods.
class Tree:
# constructor
def init (self, key):
self.root = None
def insert(self, key):
# your code for insert
def delete(self, key):
# your code for delete
Objectoriented vs. Procedureoriented programming:
Procedure (textbook): TREEINSERT(T, x)
Object oriented method call: T.insert(x)
The method must be declared as
def insert( self , key)
self must be the rst parameter! (corresponds to
T)
to access instance variables (e.g., root), must say
self.root.
Can mix both styles
26
Lecture 7
Dynamic Programming
This lecture: Dynamic programming
Optimization problems and Dynamic programming
Assembly line scheduling
matrix multiply
longest common subsequencce
optimal binary search tree
7.1 Optimization problems
many possible solutions with different costs
want to maximize or minize some cost function
unlike sorting its sorted or not sorted.. partially sorted
doesnt quite count.
examples: matrixchain multiply (same results, just
faster or slower)
knapsack problem (thief lling up a sack), compression
7.1.1 Dynamic programming
programming here means tabular method
Instead of recomputing the same subproblem, save re
sults in a table and look up (in constanttime!)
signicance: convert an otherwise exponential/factorial
time problem to a polynomialtime one!
Problem characteristic: Recursively decomposable
Search space: a lot of repeated subcongurations
optimal substructure in solution
7.2 Case study: Assembly line scheduling
two assembly lines 1 and 2
each line i has n stations S
i, j
for n stages of assembly
process
each station takes time a
i, j
chassis at stage j must travel stage ( j +1) next
option to stay in same assembly line or switch to
the other assembly line
time overhead of t
i, j
if decided to switch to line to
go to S
i, j
.
assembly
line 1
assembly
line 2
stage 1 stage 2 stage 3 stage 4
2
2
1
3
2
1
stage 5 stage 6
2
3
1
4
4
2
3
2
7 9
5
3
6
4
4
8
5
4
7 5
Want to minimize time for assembly
Line1 only: 2+7+9+3+4+8+4+3 = 40
Line2 only: 4+8+5+6+4+5+7+2 = 41
Optimal: 2+7+(2) +5+(1) +3+(1) +4+5+
(1) +4+3 = 38
How many possible paths? 2
n
(two choices each stage)
Optimal substructure
Global optimal contains optimal solutions to subprob
lems
Fastest way through any station S
i, j
must consist of
shortest path from beginning to S
i, j
shortest path from S
i, j
to the end
That is, cannot take a longer path to S
i, j
and make
up for it in stage ( j +1). . . n.
27
Notation: f
i
[ j] = fastest possible time from start through
station S
i, j
(but not continue)
e
i
, x
i
are entry/exit costs on line i
Goal is to nd f
global optimal
initially, at stage 1 (for line l = 1 or 2),
f
l
[1] = e
l
(entry time) +a
l,1
(assembly time)
at any stage j >1, line l, (and m donotes the other line)
f
l
[ j] = min
_
f
l
[ j 1] same line l
f
m
[ j 1] +t
m,( j1)
other line m + transfer
_
+ a
l, j
(assembly time at station S
l, j
)
Can write this as a recursive program:
F(i, j)
if j = 1
then return e
i
+a
i,1
else return min(F(i, j 1),
F(i%2+1, j 1) +t
i%2+1, j1
)
+a
i, j
But! There are several problems:
many repeat evaluation of F(i, j), could be O(2
n
) time
use a 2D array f [i, j] to remember the running mini
mum
does not track the path
use array l
i
[ j] to remember which path gave us this
min
Iterative version shown in book on p. 329.
Run time is (n).
7.3 MatrixChain multiply
Basic Matrix Multiply
A is pq, B is qr
q
p
q
r
p
r
A B AB
product is pr matrix:
c
i, j
=
y=1...q
a
i,y
b
y, j
total number of scalar multiplications = pqr
Multiply multiple matrices
matrix multiplication is associative:
(AB)C = A(BC)
Both yield ps matrix
Total # multiplications can be different! (added)
(AB)C is
pqr to multiply AB rst,
+ prs to multiply (AB)(pr) w/ C(r s)
= pqr + prs total # multiplications
On the other hand, A(BC) is pqs +qrs
Example: if p = 10, q = 100, r = 5, s = 50, then
pqr + prs = 5000+2500 = 7500
pqs + qrs = 50000 + 25000 = 75000 ten times as
many!
Generalize to matrix chain: A
1
, A
2
, A
3
. . . A
n
But there are many ways!
P(1) = 1 (A) nothing to multiply
P(2) = 1 (AB)
P(3) = 2 A(BC), (AB)C
P(4) = 5 A(B(CD)), A((BC)D), (AB)(CD), (A(BC))D, ((AB)C)D
P(5) = 14 . . . Exponential growth
P(n) =
_
_
_
1 if n = 1,
n1
k=1
P(k) P(nk) for n 2
(4
n
/n
3/2
) at least exponential!
7.3.1 optimial parenthesization to inimize # scalar
mults
((A
1
A
2
)
A
1...2
A
3
) (A
4
A
5
)
A
4...5
1..k k+1..n
A
1...5
A
1...3
Notation:
let A
i... j
denote matrix product A
i
A
i+1
A
j
matrix A
i
has dimension p
i1
p
i
Optimal substructure
if optimal parenthesization for A
1... j
at the top level is
(L)(R) = (A
1...k
)(A
k+1... j
), then
L must be optimal for A
1...k
, and
R must be optimal for A
k+1... j
Proof by contradiction
Let M(i, j) = Minimum cost from the i
th
to the j
th
matrix
M(i, j) =
_
_
0 if i = j,
min
ik<j
M(i, k) +M(k +1, j) + p
i1
p
k
p
j
if i < j
As a recursive algorithm (very inefcient!):
M(i, j)
if i = j
then return 0
else return M(i, k)+ M(k +1, j) + p
i1
p
k
p
j
28
Observation
dont enumerate the space!
bottom up no need to take the min so many times!
instead of recomputing M(i, k), remember it in array
m[i, k]
book keeping to track optimal partitioning point. See
Fig.7.1
O(n
3
) time, (n
2
) space (for m and for s arrays)
7.4 Longest common subsequence
Example sequence X =A, B, C, B, D, A, B),
Y =B, D, C, A, B, A)
a subsequence of X is Z =B, C, D, B)
Longest common subsequence (LCS) of length 4:
B, C, B, A), B, D, A, B)
This is a maximization, also over addition, but add cost
by 1 (length increment)
Brute force:
enumerate all subsequences of x (length m), check if its
a subsequence of y (length n)
#of subsequences of x = 2
m
(binary decision at each
point whether to include each letter)
worst case time is (n 2
m
) because for each one check
against ys length = n
Better way:
Notation: X
k
= lengthk prex of string X
x
i
is i
th
character in string X
Z = z
1
. . . z
k
) is an LCS of X = x
1
. . . x
m
) and Y =
y
1
. . . y
n
)
if x
m
= y
n
then z
k
= x
m
= y
n
, and
Z
k1
is an LCS of X
m1
,Y
n1
.
if x
m
,= y
n
then
if z
k
,= x
m
then Z is LCS of X
m1
,Y
if z
k
,= y
n
then Z is LCS of X,Y
n1
.
c[i, j] = LCS length of X
i
,Y
j
c[i, j] =
_
_
_
c[i 1, j 1] +1 if x[i] = x[ j](match)
max
_
c[i, j 1]
c[i 1, j]
_
(no match, advance either)
Algorithm
c[1 : m, 0] 0; c[0, 1 : n] 0
for i 1 to m
do for j 1 to n
do if (x
i
= y
j
)
then c[i, j] c[i 1, j 1] +1
b[i, j] match ()
else if (c[i 1, j] c[i, j 1])
then c[i, j] c[i 1, j] copy the longer length
b[i, j] dec i ()
else c[i, j 1] > c[i 1, j]
c[i, j] c[i 1, j]
b[i, j] dec j ()
0
0 0 0 0 0 0 0
j 0 1 2 3 4 5 n =6
i
0
1
2
3
4
5
6
m=7
A
B
C
B
D
A
B
B D C A B A
0 0 0 0 0 0 0
0
0
0
0
0
0
0
i=1 A
0
B D C A B A
0
0 0 0 1 1 1
i=2 B
B D C A B A
1 1 1 1 2 2
i=3 C
0
B D C A B A
1 1 2 2 2 2
i=4 B
0
B D C A B A
1 1 2 2 3 3
i=5 D
0
B D C A B A
1 2 2 2 3 3
i=6 A
0
B D C A B A
1 2 2 3 4 3
i=7 B
0
B D C A B A
1 2 2 3 4 4
Time (mn), Space (mn).
7.5 Optimal Binary Search Trees
input: n keys K =k
1
, k
2
, . . . , k
n
)
n+1 dummy keys D =d
0
, d
1
, . . . , d
n
)
d
0
< k
1
< d
1
< k
2
< d
2
< . . . < k
n
< d
n
key k
i
has probability p
i
, and
dummy key d
i
has probability q
i
, and
n
i=1
p
i
+
n
i=0
q
i
= 1
want: Binary tree that yields fastest search: (fewer steps)
for frequently used words
k
i
keys should be internal nodes, and d
i
dummy keys
should be leaves.
29
MATRIXCHAINORDER(p)
1 n length[p] 1
2 for i 1 to n
3 do m[i, i] 0
4 for l 2 to n: l = length of interval considered
5 do for i 1 to (nl) +1:
starting index, from 1 up to n length for each length
6 do j i +l 1
ending index, always length away from the starting index
7 m[i, j]
8 for k i to j 1:
different partitions between i and j
9 do q m[i, k] +m[k +1, j] + p
i1
p
k
p
j
.
10 if (q < m[i, j]):
11 then m[i, j] q
12 s[i, j] k remember best k between i, j
13 return m, s
A
i...j
length
l
i = starting
index
of chain
n
n l + 1
j = i + l 1
= ending
index
of chain
1
k from i to j 1
chain length l from 2 to n
starting index i from 1 to n l + 1
(implies ending index j)
Figure 7.1: MatrixChainOrder Algorithm and graphical illustration.
optimize for common case. Balanced tree might not be
good!
Example tree (not optimal):
k
2
0. 10
k
1
0. 15
k
4
0. 10
d
0
0. 05
d
1
0. 10
k
3
0. 05
k
5
0. 20
d
2
0. 05
d
3
0. 05
d
4
0. 05
d
5
0. 10
Expected search cost of tree T
Optimal substructure: if root=k
r
, L = (i . . . r 1), R =
(r +1. . . j) L, R must be optimal subtrees.
Expected cost e[i, j]
e[i, j] =
_
_
q
i1
(a dummy leaf ) if j = i 1
min
irj
e[i, r 1] (left subtree)
+e[r +1, j] (right subtree)
+w(i, j) (add one depth) if i j
k
2
0. 10
k
1
0. 15
k
4
0. 10
d
0
0. 05
d
1
0. 10
k
3
0. 05
k
5
0. 20
d
2
0. 05
d
3
0. 05
d
4
0. 05
d
5
0. 10
d
0
0. 05
e[1,0]
= q
0
= 0.05
d
1
0. 10
e[2,1]
= q
1
= 0.10
e[1,1] =
p
1
(0.15 for k
1
)
+ e[1,0] (left tree)
+ e[1,1] (right tree)
+ one more depth
for all nodes in
Left & Right subtrees
k
1
0. 15
d
0
0. 05
d
1
0. 10
d
0
0. 05
d
1
0. 10
k
1
0. 15
d
0
0. 05
d
1
0. 10
e[1,5] =
w[1, 5]
+ e[1,0] (left tree)
+ e[1,1] (right tree)
w[1,1] = p
1
+ one more depth
for all nodes
in subtrees
e[1,1] = w[1,1] + e[1,0] + e[1, 1]
k
1
0. 15
d
0
0. 05
d
1
0. 10
k
4
0. 10
k
3
0. 05
d
2
0. 05
d
3
0. 05
k
5
0. 20
d
4
0. 05
d
5
0. 10
use arrays to remember e[i, j], w[i, j] instead of recom
puting
use array root[i, j] to remember root positions
OPTIMALBST(p, q, n) (page 361)
1 for i 1 to n+1
2 do e[i, i 1] q
i1
3 w[i, i 1] q
i1
4 for l 1 to n
5 do for i 1 to nl +1
6 do j i +l 1
7 e[i, j]
8 w[i, j] w[i, j 1] + p
j
+q
j
9 for r i to j
10 do t e[i, r 1] +e[r +1, j] +w[i, j]
11 if t < e[i, j]
12 then e[i, j] t
13 root[i, j] r
14 return e, root
30
Lecture 8
Greedy Algorithm
This lecture
greedy algorithms: make choice that looks best at the
moment
activity selection problem
Huffman (with Python programming hints)
8.1 Activity Selection problem
Set S of n activities
s
i
= start time of activity i,
f
i
= nish time of i.
assume 0 s
i
< f
i
< . (nite, positive/nonzero execu
tion delay)
i, j are compatible
if f
i
< f
j
implies f
i
< s
j
. (i.e., no overlap)
Goal: nd a maximal subset A of compatible activities
maximize [A[ = # of activities
NOT maximize utilization!!
there many be several optimal solutions, but its
sufcient to nd just one.
Greedy Solution: pick next compatible task with earliest
nish time.
Example
0 2 4 6 8 10 12 14
l
i
s
t
o
f
t
a
s
k
s
s
o
r
t
e
d
b
y
f
i
n
i
s
h
t
i
m
e
f
i
o
p
t
i
m
a
l
s
c
h
e
d
u
l
e
s
Notation
S =a
1
. . . a
n
set of activities.
a
0
= start, a
n+1
= end (fake) activities: f
0
=
0, s
n+1
= .
S
i j
=a
k
S : f
i
s
k
< f
k
s
j
_
0 if S
i, j
= / 0
max
i<k<j
c[i, k] +c[k, j] +1 if S
i, j
,= / 0
(note: S
i,k
and S
k, j
are disjoint and do not include a
k
)
looks like dynamic programming formulation, but
we can do better with greedy.
if a
m
S
i, j
has earliest nish time, then
1. S
i,m
= / 0
because if not empty, there must exist a
k
that starts
after a
i
and nish before a
m
,
but if a
k
nishes before a
m
, then a
k
should have
been picked. Contradiction.
2. a
m
is a member of some maximal A
i, j
if not, we can always construct A
/
i, j
= A
i, j
a
k
a
m
, and [A[ =[A
/
[.
a
i
S
i,j
S
i,m
S
i,m
a
j
a
k
a
m
A
ij
A
ij
a
m
a
k
S
i,m
should be empty
incompatible
activities
If a maximal A
ij
starts with
a
k
a
m
, it should always be
possible to create another A
ij
by replacing a
k
with a
m
.
What does this mean?
earliestnishtimerst is always as good as any optimal
optimal substructures for later half dont depend on ear
lier optimal substructure
Greedy Algorithm (iterative version, p. 378)
GREEDYACTIVITYSELECTOR(s, f )
assume f already sorted!
1 n length[s]
2 A a
1
3 i 1
4 for m 2 to n
5 do if s
m
f
i
next compatible with earliest nish
6 then A Aa
m
;
7 j m
8 return A
Another way to view this:
intuition  leave the largest remaining time
What about least duration?
8.2 Greedy choice property
global optimal is made up of the greedy choice plus an
optimal subproblem
prove greedy choice is a member of the global optimal
Optimal substructure:
showing that removing the greedy choice yields a solu
tion to the subproblem.
Greedy vs. Dynamic programming: Knapsack prob
lem
Thief with a knapsack
weight limited rather than volume limited
Goal: maximize the value of stolen goods under weight
limit
Variations:
01 knapsack problem: whole items
(watches, nuggets)
fractional knapsack problem can take parts
(sugar, salt)
Important distinction:
Greedy works for the fractional knapsack problem
always take things with maximum value per unit weight
Greedy does not work for 01
by counter example:
Limit: 50 lb, items $60/10, $100/20, $120/30
32
Ratios $6, $5, $4
Greedy solution: take item 1, item 2, but
item 3 wont t get $160/30
Optimal is take item 2, item 3, leave item 1
$220/50
$60
10kg
0
10
20
30
40
50
kg
$100
20kg
$120
30kg
knap
sack
50kg
$:wt
ratio
6 5 4
Greedy is not optimal for 01 knapsack:
highest $:wt first, keep adding until exceeding
knapsack capacity
$60
10kg
$100
20kg
add first
add second
$120
30kg
oops! wont fit!
must leave behind
result: $160 / 30kg
Optimal for 01:
in this case, best fit
$100
20kg
$120
30kg
result: $220 / 50kg
Greedy would work for fractional knapsack:
take a fraction of the pink part
result: $240 / 50kg
How to solve 01 knapsack?
dynamic programming, in O(nW) time.
idea: iterate over total weights up to the limit
need to assume unit weight
8.3 Huffman codes
xedlength code:
same # bits for all characters
variable length code:
more frequent use fewer bits (e.g. vowels like a e
i o u vs. q z)
Goal: minimize
codewordcC
frequency(c) bitlength(c)
nonlossy! preserves the same information
Comparison example: 6 characters
a b c d e f
frequency 45 13 12 16 9 5
xed length codeword 000 001 010 011 100 101
varlength codeword 0 101 100 111 1101 1100
Why is variable length better?
xed (3bits per char) 100,000 chars 300,000 bits
variable length: (45 1+(13+12+16) 3+(9+5) 4) =
224, 000 bits (25% saving)
problem: how do we know where to begin each code
word?
prex codes: no codeword is a prex of another code
word. e.g., only a starts with 0. So, 00 means (a a).
1 need to look at more bits:
10 is prex for either b (101) or c (100)
11 is prex for d, e, f
Example: 001011101 uniquely parses as
0(a) 0(a) 101(b) 1101(e), no ambiguity
Algorithm
HUFFMAN(C) (page 388)
C is the array of nodes that carry (letter, frequency)
1 n [C[
2 Q C put all elements into priority queue
3 for i 1 to n1
4 do z MAKE NEW NODE
5 left[z] x EXTRACTMIN(Q)
6 right[z] y EXTRACTMIN(Q)
7 f [z] f [x] + f [y]
8 INSERT(Q, z)
9 return EXTRACTMIN(Q)
Python version
http://e3.uci.edu/02f/15845/huffman.html
33
14
25 30
55
100
0 1
1
1
1
0
0
0
0 1
initially: minpriority Q gets character/frequency nodes
'f': 5 'e': 9 'c': 12 'b': 13 'd': 16
Q
i=1:
14
1. extract two nodes
2. make a binary tree
z with sum freq.
3. insert z into Q
'c': 12 'b': 13 'a': 45
Q 14
'f' 'e'
'f': 5 'e': 9
z:
i=2:
25
'c' 'b'
Q
'a': 45
14
'f' 'e'
25
'c' 'b'
'a': 45
'd': 16
'd': 16
i=3:
Q
'a': 45
25
'c' 'b'
14
'f' 'e'
'd': 16
30
30
14
'f' 'e'
'd'
continued... when all done,
'a' : 45
'b' : 13 'c' : 12 'd' : 16
'e' : 9 'f' : 5
Time complexity:
[n[ 1 calls, heap is O(lgn), so this is O(nlgn) for n
characters
Correctness
Greedy property:
C = alphabet. x, y C are two chars with lowest frequen
cies. (most rarely used)
then there exist an optimal prex code such that
the codewords for x and y have the same length and
differ only by the last bit value.
If a, b are leaves at deepest depth and have higher
frequency, then we can swap a, b with x, y and ob
tain a new tree with a lower cost of sum of freq
depth.
Optimal substructure:
T
/
is an optimal tree for alphabet C
/
T is an optimal tree for alphabet C if
C and C
/
are the same except x, y C while z C
/
,
f is the same except f [z] = f [x] + f [y]
we construct T from T
/
by replacing leaf node for
z with internal node that is parent to x, y.
34
Lecture 9
Basic Graph Algorithm
This lecture
Graph data structure
Graph traversal ordering denition
9.1 Graphs G = (V, E)
V: vertices. Notation: u, v V
E: edges, each connects a pair of vertices
E V V; (u, v) E
Variations
undirected or directed graph (digraph)
weighted: by default, weighted edges
w : E R
Notation: weight for edge (u, v) is written as w(u, v)
Also possible to have weighted vertices
Undirected Graphs
degree of a vertex: # of edges connected to the vertex
complete graph, aka clique:
undirected graph where all pairs of vertices are con
nected
bipartite graph:
undirected graph whose V =V
1
V
2
, and E V
1
V
2
multigraph: can have multiple edges between the same
pair of vertices (including self edges)
hypergraph: an edge can connect more than two vertices
Digraphs
indegree: # incoming edges,
outdegree: # outgoing edges
path: v
0
, v
1
, . . . , v
k
), where (v
i
, v
i+1
) E
simple path: a path where all v
i
are distinct
cycle: a nontrivial simple path plus (v
k
, v
0
) to close the
path
DAG: directed acyclic graph (no cycle)
strongly connected: digraph whose vertices are all reach
able from each other.
Representation
Adjacency list
Associate each vertex with a linked list of its neighbors
on its outgoing edges
good for sparse graphs
for weighted graph, need to store weight in linked list
Adjacency matrix
A bit matrix to indicate presence/absence of edge
weighted: can store weight in matrix
u v
y x
w
V
u
v
w
x
y
Adjacency list
v y
x w
x
y
v
u
v
w
x
y
u v w x y
1 1
1 1
1
1
1
Adjacency matrix
tradeoffs:
Adj. List: easier to grow #of vertices, good for sparse
Adj. Matrix: faster access, but O(V
2
) storage
35
9.2 Breadthrst search
distance here means # of edges, not edge weight!
e.g., start from u:
= 0: u
= 1: v, y
= 2: w, x
u v
y x
w
= 1
= 2
start
visit vertices in increasing shortest distance
visit all of distance d before visiting distance d +1 ver
tices
BFS order may not be unique!!
Example: BFS(G, u) can be
[u, v, y, w, x], [u, v, y, x, w], [u, y, v, w, x], [u, y, v, x, y]
Algorithm
BFS(G, s)
1 for each u V[G] s
2 do color[u] WHITE
3 d[u]
4 [u] NIL
5 color[s] COLOR
6 d[s] 0
7 [s] NIL
8 Q / 0
9 ENQUEUE(Q, s)
10 while Q ,= / 0
11 do u DEQUEUE(Q)
12 for each v Adj[u]
13 do if color[v] = WHITE
14 then color[v] GRAY
15 d[v] d[u] +1
16 [v] u
17 ENQUEUE(Q, v)
18 color[u] BLACK
main idea: for each vertex of d, enqueue vertices of d +1
avoid queuing more than once: use color
WHITE: never been enqueued
GRAY: currently in queue
BLACK: already visited
actually COLOR is somewhat redundant... enough
to test (predecessor)
Lemma 22.1: for any edge (u, v) E, (s, v) (s, u)+1
if we can reach u in edges, we can reach v in +1
edges (by taking edge (u, v))
v might actually be closer than u, but thats ok
d[v] computed by BFS satises d[v] (s, v)
initialize to d[u] = (s, u)
base case: since we start with d[s] = 0, all those u
with (s, u) will have d[u] = (s, u) = 1
induction: assume d[u] = (s, u), consider edge
(u, v).
if color[v] ,= WHITE then it has been visited before
and d[v] < (s, u) +1.
if color[v] = WHITE then it is being visited for the
rst time: d[v] = (s, u) +1 = d[u] +1 = (s, v).
9.3 Depthrst search
depth is not necessarily the same as distance in BFS!
discover vertices (use a stack) before we visit!
pushing vertices on stack (if we havent pushed it before)
DFS order may not be unique!
Example: DFS with tree
u
v
x y
w
z
Output when closing parenthesis: [x, z, y, v, w, u]
start
u
v
x
z
w
(u (v (x x) (y (z z) y) v) (w w) u)
1 2 3 4 5
y
6 7 8 9 10 11 12
DFS(G)
1 for each v V[G]
2 do color[u] WHITE
3 [u] NIL
4 time 0
5 for each u V[G]
6 do if color[u] = WHITE
7 then DFSVISIT(u)
DFSVISIT(u)
1 color[u] GRAY discovered u
2 timetime +1
3 d[u] time
4 for each v Adj[u]
5 do if color[v] = WHITE
6 then [v] u
7 DFSVISIT(v)
8 color[u] BLACK
9 f [u] time time +1
36
Example: DFS with DAG
u v
x y
w
z
One possible DFS order:
u
v
y
x
Output: [x, y, v, u, z, w]
Another possible DFS order:
u v
x y
w
z
Output: [z, v, x, y, w, u]
(u (v (y (x x) y) v) u) (w (z z) w) (w (z z) (y (x (v v) x) y) w) (u u)
1 2 3 4 5 6 7 8 9 10 11 12
w
z
w
v
y
x
1 2 3 4 5 6 7 8 9 10 11 12
u
z
generalization of of postorder traversal
doesnt need a root: all vertices are considered
automatically discovers nodes when disconnected
Time complexity: (V +E):
DFS main routine is (V),
DFSVISIT is [Adj[v][ = (E) times
Color Encoding
WHITE: never been visited
GRAY: been pushed on the stack, not yet popped
BLACK: been visited (popped from stack).
Classication of edges
G
z
15
z
15
Dijkstras terminated with z = 15 (via s y z),
even though (s, z) = 14 (via s t y z)
11.3 BellmanFord
Allows negative edge weight
Can detect negative cycles!
11.3.1 Algorithm:
BELLMANFORD(G, w, s)
1 INITIALIZESINGLESOURCE(G, s)
that is, d[u V] ; d[s] 0
2 loop [V[ 1 times
3 for edge (u, v) E order does not matter!
4 RELAX(edge (u, v))
now check if we converged
5 for edge (u, v) E
6 do if RELAX(u, v) still havent converged after V 1 times!
7 raise Negative Cycle;
if converged (test dirty bit) then return to make it faster!
11.3.2 Correctness:
Need to prove d[v] = (s, v) after [V[ 1 passes if no negative
cycle.
Lemma: d[v] (s, v)
Proof by contradiction:
(1) Suppose d[v] < (s, v), and edge (u, v) exists, and
(2) Suppose d[v] = d[u] +w(u, v)
d[v] < (s, v) assumption (1) to contradict
(s, u) +w(u, v) by triangular inequality
d[u] +w(u, v) assumption that v is rst violation.
But this contradicts assumption (2) d[v] = d[u] +w(u, v).
Lemma (stated informally)
s v
1
v
2
v
3
. . . v
initially d[s] = 0 is correct.
after 1 pass, d[v
1
] is correct, so d[v
1
] = (s, v
1
)
after 2 passes, d[v
2
] is correct....
11.3.3 Complexity: O(VE)
observation: no Priority Queue
Compare: DAG shortest paths
Relax in (any) topologically sorted order. just 1
pass.
O(V +E)
idea: use strongly connected component graphs to
speed up Bellman Ford.
11.4 BellmanFord: special case of linear pro
gramming
conjunctive pairwise maxdifference constraints. For example:
x
1
x
2
3
x
2
x
3
2
x
1
x
3
2
Want
feasibility check negative cycle not feasible
solution to the variables x
i
.
11.4.1 Bellman Ford as a Constraint Solver!
one vertex per variable, plus anchor s
one edge per constraint (including redundant constraint
elimination)
x
1
x
2
5 would be redundant! ( 3 constraint is
stronger)
Run BellmanFord, then solution X = ((s, x
1
), (s, x
2
), . . .)
is a feasible solution.
45
s
t x
z y
6
7
8
5
9
3
7
2
4
2
Relaxation order: over Adj of
From: t: x: y: z: s: 0
To: x: y: z: t: x: z: x: s: 0 t: 6 y: 7
1st
pass:
s
t x
z y
6
7
8
5
9
3
7
2
4
2
2nd
pass:
From: t: 6 x: 11 y: 7 z: 2 s: 0
To: x: 11 y: 7 z: 2 t: 6 x: 4 z: 2 x: 4 s: 0 t: 6 y: 7
keep the shorter path keep the shorter path
s
t x
z y
6
7
8
5
9
3
7
2
4
2
3rd
pass:
From: t: 6 x: 4 y: 7 z: 2 s: 0
To: x: 4 y: 7 z: 2 t: 2 x: 4 z: 2 x: 4 s: 0 t: 6 y: 7
d[s] didnt change,
so nothing happens
to these
d[t], d[y], d[z], d[s] didnt change
since the previous iteration, so nothing
happens to these.
s
t x
z y
6
7
8
5
9
3
7
2
4
2
4th
pass:
From: t: 2 x: 4 y: 7 z: 2 s: 0
To: x: 4 y: 7 z: 2 t: 2 x: 4 z: 2 x: 4 s: 0 t: 6 y: 7
keep the shorter path
Converged after 4th pass
46
Lecture 12
All Pairs Shortest Paths
This lecture
Single source contd: special case of linear program
ming, constraint solver
Allpairs shortest paths
FloydWarshall: O(V
3
) and Transitive closure (War
shalls)
Johnsons algorithm
12.1 Special case Linear Programming (contd)
ab 2 as edge (b, a) with weight = 2
ac 1 as edge (c, a) with weight = 1
bc 3 as edge (c, b) with weight =3
Finally, add a source vertex s and edges
(s, a), (s, b), (s, c) with weight 0.
s
a
b
c
0
0
0
2
1
3
Solve this with BELLMANFORD, we get
a =1 (path s c b a)
b =3 (path s c b)
c = 0 (path s c)
check our answers:
(1) (3) 2 ? yes
(1) (0) 1 ? yes
(3) (0) 3 ? yes
12.1.1 Correctness sketch:
Key point: triangle inequality: (s, v) (s, u) +w(u, v)
By construction, d[v] = min over all such (u) +w(u, v)
Min over all means these properties hold conjunctively.
If there is a negative cycle, then
(w(i +1%n, i) < 0
But algebraically, x
(i+1)%n
x
i
= 0.
since 0 cant be < 0, there exists no feasible solution.
BELLMAN FORD nds the best (i.e., minimizes)
(maxx
i
minx
i
)
12.2 All Pairs shortest path
First attempt: DIJKSTRAs or BELLMANFORD once on
each vertex as s
BELLMAN FORD good for sparse graph; but
for dense graph, O(V
2
E), about O(V
4
) for dense
graphs
FLOYDWARSHALLs assumes adjacency matrix
JOHNSONs for sparse, adjacency list
12.2.1 Formulation
Singlesource Allpairs
curr best d[u] d[i, j]
optimal (s, u) (i, j)
on termination d[u] = (s, u) d[i, j] = (i, j)
relax: RELAX(u, v) RELAXPAIR(i, k, j)
try s u v try i k j
data struct adj. list? adj. matrix
Adjacency matrix representation
matrix W =
_
w
i j
_
m
th
iteration: W
(m)
=
_
w
(m)
i j
_
47
Recursive solution
matrix L =
_
l
i j
_
l
(m)
i j
= minimum path weight i j containing at most m
edges
l
(0)
i j
=
_
0 if i = j,
if i ,= j
for m
th
iteration, want to RELAXPAIR(i, k, j)
if i k (m1 steps) + k j (in 1 step) is shorter than
the current best i k ((m1) steps), then take path
i k j.
l
(m)
i j
= min
_
l
(m1)
i j
curr. best in m1 steps
min
k1..n
l
(m1)
ik
+w
k j
m1 steps to k, then (k, j)
= min
1kn
_
l
(m1)
ik
+w
k j
_
We know we can reach any node from any other node in
V 1 edges. This means
Loop V 1 times (V)
for k 1 to [V[ (V)
for all i, j V
2
times
if l
i j
> l
ik
+w
k j
then shorter to go thru k
l
i j
l
ik
+w
k j
.
Total is O(V
4
) times essentially same as BELLMAN
FORD.
12.2.2 Repackage this as a matrix structure:
L
(m)
= L
(m1)
W looks a lot like a matrix multiply C = A B:
EXTEND PATH MATRIX MULTIPLY
L
(m)
= L
(m1)
W C = A B
l
ik
+w
k j
a
ik
b
k j
l
i j
min(l
i j
, l
ik
+w
k j
) c
i j
c
i j
+a
ik
b
k j
EXTENDPATH(L,W)
1 for i 1 to n
2 for j 1 to n
3 l
i j
4 for k 1 to n
5 l
i j
min(l
i j
, l
ik
+w
k j
)
6 return L
MATRIXMULTIPLY(A, B)
1 for i 1 to n
2 for j 1 to n
3 c
i j
0
4 for k 1 to n
5 c
i j
c
i j
+a
ik
b
k j
6 return C
Why do it this way? can be faster
matrixmult is associative, so we can do repeated squar
ing:
W
2k
=W
k
W
k
.
so we just need to do this lgn1 times to compute
W
(n1)
.
Ok to overshoot because it should converge, unless there
is a negative cycle.
Complexity: (V
3
lgV): each matrix multiply is (V
3
),
(lgV) squarings.
12.2.3 FloydWarshall: O(V
3
)
quintessential dynamic programming, bottom up
Observation: wasteful to just consider expanding the dis
tance by one edge segment at a time.
instead of of w
ik
, why dont we use d
ik
from the previous
iteration?
d
(m1)
ik
(path) grow by
REPEAT SQUARING i k k j (edge)
FLOYDWARSHALL i k k j (path)
d
(k)
i j
=
_
_
w
i j
if k = 0
min
k1..n
_
d
(k1)
i j
,
d
(k1)
ik
+d
(k1)
k j
_
if k 1
12.2.4 Algorithm  very simple!
FLOYDWARSHALL(W)
D
(0)
W
for k 1 to n
for i 1 to n
for j = 1 to n
d
(k)
i j
= min(d
(k1)
i j
, d
(k1)
ik
+d
(k1)
k j
)
return D
(n)
as a side effect, record predecessor when taking min
in an implementation, can just reuse the same space!!!
(it wont hurt to use d
(k)
i j
when we need d
(k1)
i j
.
O(n
3
) time, O(n
2
) space
12.3 Transitive closure: E
(Warshalls)
Boolean yes/no question:
is there a path from i to j?
very useful for transitive relations:
(e.g., a > b, b > c, then a > c)
(e.g. a b, b c, then a c)
Question becomes is there a directed path?
nite path yes; innite no.
can just replace min with AND, + with OR.
yes TRUE (or 1), no FALSE (or 0)
FLOYDWARSHALL(W)
D
(0)
W
for k 1 to n
for i 1 to n
for j 1 to n
d
i j
min(d
i j
, d
ik
+d
k j
)
return D
(n)
TRANSITIVECLOSURE(G)
T
(0)
Adj[G]
for k 1 to n
for i 1 to n
for j 1 to n
t
i j
t
i j
(t
ik
t
k j
)
return T
(n)
48
t
i, j
= (i, j) E ? true : false;
t
i,i
= true
t
(k)
i j
t
(k1)
i j
(t
(k1)
ik
t
(k1)
k j
) means
if (i, j) not already E, and we have both (i, k) and
(k, j) E, then add (i, j) to E.
Summary: FloydWarshall use adj matrix, better for dense;
but for sparse, want the E factor closer to V.
12.4 Sparse graphs: Johnsons algorithm
Idea: run DIJKSTRAs if no negative weight (preprocess
ing pass to check)
But if there are, do reweighing add an offset to shift
the weights up to nonnegative.
Cant just add a constant weight to each edge.
why not?
because path length is a separate concept from path
count.
could screw up path length by (difference in path count)
(edgeweight offset to make everybody positive).
u v
w
5
2 3
x
1
u v
w
7
0 5
x
1
shift all
weights up
by 2 (to
eliminate
negative
weights)
shortest:
uvwx: 2
shortest (incorrect):
ux : 5
idea:
add a new source vertex s to all other vertices in G
weight = 0 on all these new edges.
Run BELLMANFORD to compute d[v] for all in G.
use d[v] from BELLMANFORD as the new offset func
tion h(v).
Reweighing each edge (u, v) as w(u, v) = w(u, v) +
h(u) h(v)
for each u, run DIJKSTRAs on this reweighed graph
shift weight back:
(u, v) +h(v) h(u).
12.4.1 Proof weight works:
need to show w(p) = (v
0
, v
k
) w(p) =
(v
0
, v
k
).
basically along any path, all intermediate ones cancel
(telescope series)
w(v
1
, v
2
) + w(v
2
, v
3
) + . . .
= w(v
1
, v
2
) +h(v
1
) h(v
2
) +w(v
2
, v
3
) +h(v
2
) h(v
3
) + . . .
= w(v
1
. . . v
n
) +h(v
1
) h(v
n
).
49
Lecture 13
Flow Networks
This lecture
Flow Networks
FORDFULKERSON method
EDMONDSKARP algorithm
13.1 Flow network
directed graph G = (V, E)
source s, sink t
capacity on edge, c(u, v) 0.
s t
u
w
v
x
16
13
10 4
12
14
9
7
4
20
two views: net ow vs. positive ow
Book uses net ow f : V V R
positive ow G is p : V V R
p(u, v) =
_
f (u, v) if f (u, v) > 0
0 if f (u, v) 0
Basically f (u, v) = p(u, v) p(v, u); f (u, v) =f (v, u)
Example:
p(w, x) = 11; p(x, w) = 0 because net ow is 11.
f (w, x) = 11; f (x, w) =11 = p(x, w) p(x, w) =
11.
s t
u
w
v
x
11/16
8/13
0/10 1/4
12/12
11/14
4/9 7/7
4/4
15/20
Three properties:
capacity constraint: 0 p(u, v) c(u, v)u, v V
skew symmetry: f (u, v) =f (v, u)
ow conservation: u V s, t :
vV
: p(u, v) =
vV
p(v, u)
vV
: f (u, v) = 0
e.g., u: p(u, v) + p(u, w) = p(s, u) + p(w, u)?
12+0 = 11+1 yes.
e.g., w: p(w, u) + p(w, x) = p(s, w) + p(u, w) + p(v, w)?
1+11 = 8+0+4 yes.
This means Inow(u) = Outow(u) for all except source
and sink. Kirchhoffs current law.
Defn: value of ow value( f ) = Outow(s) Inow(s) =
Inow(t) Outow(t)
Goal: maximize value( f ). Could be less.
Example: s
5
x
4
t
max out at ow of 4 (min of capacities along any seg
ment)
(x, y) is the bottleneck
Example:
50
s t
u
y
v
x
5/5
5/6
3/5
0/2
2/4
5/5
1/1
7/7
4/6
2/5
w
z
2/4
3/3
2/4
this flow has
value f = 13
Can do better:
s t
u v
x
2/
4/4
7/7
2/
4/5
w
3/
1/3
3 /
5/5
2/
4/4
an
augmenting
path
Note: were actually reducing f(u, x) in
order to increase overall flow
Residual network G
f
include only edges w/ nonzero capacities.
c
f
(u, v) = c(u, v) f (u, v) for forward edge,
c
f
(v, u) = f (u, v) for added back edge.
example: G is u
3/5
v, v
0/1
u then
v
3/5
u v
c(u, v) = 2 = 5 3
u
G
f
G
c(v, u) = 4 = 1 + 3
0/1
G
f
is u v (2 capacity, after subtracting ow 3
from 5 capacity, because we have 2 capacity left)
and v u is (4 capacity, because its capacity 1
minus ow of 3 in opposite direction = 4.)
Augmenting path: p
a path p from s to t in ow can be increased by c
f
(p) =
minc
f
of any edge on path p.
[E
f
[ 2[E[.
12
s t
u
w
v
x
11/16
8/13
0/10 1/4
12/12
11/14
4/9 7/7
4/4
15/20
G
s t
u
w
v
x
11/16
8/13
5
0/10
11
1/4
3
12/12 0
11/14 3
4/9
5
7/7
4 / 4
15/20
5
G
f
5 11
8
11
7
15
4
4
s t
w
v
5
5
Augmenting path in residual network
4
s t
u
w
v
x
11/16
8/13
12/13
0/10 1/4
12/12
11/14
4/9
0/9
7/7
4/4
15/20
19/20
Flow improvement:
original graph G has ow f ,
residual graph G
f
has ow f
/
Improved ow =[ f + f
/
[ =[ f [ +[ f
/
[ (i.e., just add resid
ual graphs ow)
Maxow, min cut: (S, T)
Defn: a Cut of a ow network is a partition of V into
(S, T) such that s S, t T,
(or, look at as a set of edges that, when removed, discon
nects the graph).
s t
u
w
v
x
11/16
8/13
0/10 1/4
12/12
11/14
4/9 7/7
4/4
15/20
S T
Example: Green edges form a cut.
capacity of a cut (S, T) =
c(S, T) =
uS,vT
c(u, v)
Example: c(S, T) = 12 + 14 = 26 (not considering
(v, w)!! dont do 9)
ow across the cut (S, T) is f (S, T) that is same as the
whole networks ow f !!!
Example: f (S, T) = 12+(4) +11 = 19
same as Outow(s), Inow(t)
Theorem
1. f is a maximum ow
2. f contains no augmenting paths
3. [ f [ = c(S, T) for some cut (S, T).
Proof:
(1) f maxow (2) f has no aug. paths
trivial: because if there exists an aug. path, we can in
crease ow by f
/
(which means it wasnt max)
(2) no aug. paths (3)[ f [ = c(S, T)
no augmenting path means residual graph has no path
from s to t.
S = sv : path s
p
v, v T
f (u, v) = c(u, v)foru S, v T
otherwise c
f
(u, v) = c(u, v) f (u, v) > 0,
v should have been in S in the rst place.
[ f [ = f (S, T) = c(S, T)
51
[) where f
w
2
v
1
x
3
u
3
5
2
4
1
1
u v
w x
Easy to nd a cycle in a clique, but hard to nd
minimumweight cycle
But if we dont need the minimum, we can nd good
cycles in polynomial time (approximation algorithm)
14.1.4 Constructive vs. Decision vs. Verication
Problems
Constructive: produce the solution (hardest)
e.g., show me a hamiltonian cycle in G
Decision: Yes/No, Can/Cannot (easier)
e.g., does G have a hamiltonian cycle?
Verication: Yes/No is a given solution valid? (easiest)
e.g., is path p a valid hamiltonian cycle for G?
certicate = the alledged solution to be veried
e.g., the path p to be veried as a hamiltonian cycle
53
14.2 Problem complexity
P: set of problems solvable in polynomial time. O(n
k
)
example: sorting, singlesource shortest path
example: verify if p is a hamiltonian cycle in G.
NP: set of decision problems that can verify a certicate
(guessed solution) in polynomial time.
NP does not mean not polynomial!!
N means nondeterministic (as in nondeterminis
tic nite automaton)
must be a DECISION PROBLEM!!!!!!!!!!!!!!
a decision problem is NP
the corresponding constructive problem is hard
polynomial time solution is unlikely to exist
14.2.1 Formal Languages
alphabet = nite set of symbols
language L over = set of strings whose characters are
example: =0, 1, L =10, 11, 101, 111, 1011, . . .
empty string
L
example: L =10, 11, 101, 111, 1011
L =, 0, 1, 00, 01, 000, 001, 100, . . .
concatenation: L
1
L
2
=x
1
x
2
: x
i
L
i
example: L
1
=10, 11, L
2
=00, 01
L
1
L
2
=1000, 1001, 1100, 1101
selfconcatenation: L
k
= LLL. . . for a total of k times
L
= closure of L =LL
1
L
2
L
3
14.2.2 Decision problem in Formal language
x
: Q(x) = 1
i.e., L is the set of valid certicates to a decision problem
Q.
Example
= ASCII character set
p
L
2
: L
1
is polynomialtime reducible to L
2
if a polynomialtime function f such that
x L
1
iff f (x) L
2
.
for reduction of Decision problems, trivial post
processing (0/1 output)
14.4 NP, NPcomplete, NPhard
Memorize this now: NP NPComplete NPHard
NP: (review)
set of decision problems: guess and verify in poly
nomial time
example: does G have a hamiltonian cycle?
NPHard:
L NPH: L
/
NP L
/
p
L
i.e., an NPhard algorithm can be called to solve
any NP problem,
after the input is preprocessed in polynomial time.
L itself might actually run in exponential time!!!!
(e.g., the constructive hamiltonian problem)
L could also be in NP, but need not be in NP.
NPComplete = NP NPH
1. L NP
2. L is NPHard
it turns out the hamiltoniandecision is NPhard
(no need to use the constructive hamiltonian)
since hamiltonian decision is NP and NPH NPC
14.4.1 How to show L NPComplete?
1) rst show L NP
easy: showability to verify certicate in polynomial time
2a) show for all L
/
NP reduce to L in polynomial time.
(recall: all of NP NPC NPH)
hard! How to cover all of NP? (could be many)
2b) Alternative: if there is a known L
/
NPC, show L
/
p
L
since NP
p
L
/
( NPC)
p
L
p
NPH
So... we need to
Figure out the rst L NPC by showing (1), (2a) hard.
(CIRCUITSAT problem)
For all other L we want to show NPC, do (1), (2b) eas
ier.
(Circuit Sat
p
L)
if any NPcomplete problem is P, then P = NP.
55
14.4.2 Examples Circuit Satisability:
input: combinational logic (p. 989)
Q: is there an assignment of 0, 1 to inputs so it outputs
1?
OR
NOT
AND
x1
x2
x3
1
1
1
1
1
1
1
0
To show CIRCUITSAT is NPcomplete:
1) Show its NP:
easy: someone gives us circuit and answer, we verify.
(actually could be linear)
2a) Show its NPhard:
assume a decision algorithm A for any problem
in NP.
Construct a new combinational circuit C by un
rolling!!!!
combinational
circuit
M
output
next
state
input
current
state
latch
M M
init
state
next
state
curr.
state
next
state
... M
curr.
state
next
state
...
0/1
new circuit C obtained by inlining a polynomial
number of copies of M
machine that contains verification algorithm A that
is decomposed into circuit M and latch
call CIRCUITSAT(C) to effectively execute A:
returns 1 iff A(x) = 1; returns 0 iff A(x) = 0.
14.4.3 NP completeness proofs
No need to show EVERY L
/
p
L for all L
/
NP.
only need to show polynomialtime reduction from one
L
/
that is known to be NPcomplete.
14.5 NPC Proof for (Formula) Satisability
problem.
allow (AND), (OR), (NOT),
(implies): a b means ab,
(if and only if):a b means (ab) (ab))
easy to show equivalent to the combinational logic prob
lem.
Example: = ((x
1
x
2
) ((x
1
x
3
) x
4
)) x
2
Is a satisable formula? the SAT problem
SAT is NPcomplete.
Show (1) SAT NP and (2b) an existing NPC prob
p
SAT.
(1) easy: guess assignment, verify:
example: certicate for above is x
1
, x
2
, x
3
, x
4
) =
0, 0, 1, 1) verify: ((0 ) ((0 1) 1)) 0 =
(1. . .) 1 = 1
(2b) CIRCUITSAT is known to be NP complete; lets use it
to show CIRCUITSAT
p
SAT.
To solve CIRCUITSAT using SAT:
Note: is equality in binary.
ANDgate: (output (input1 input2))
ORgate: (output (input1 input2),
NOTgate: (output input)
Example:
OR
NOT
AND
x
1
x
2
x
3
x
5
x
6
x
7
x
8
x
9
x
10
= x
10
(x
4
x
3
)
(x
5
(x
1
x
2
))
(x
6
x
4
)
(x
7
(x
1
x
2
x
4
))
(x
8
(x
5
x
6
))
(x
9
(x
6
x
7
))
(x
10
(x
7
x
8
x
9
)))
14.5.1 3CNF satisability (3SAT)
Motivation: a restricted form of formula:
equally expressive as general SAT
polynomial time translation to/from general SAT for
mula
polynomial time transformation from circuits
want to show CIRCUITSAT
p
3SAT
p
SAT
by transitivity, CIRCUITSAT
p
SAT
3CNF:
CNF (restricted form) = Conjuctive Normal Form: =
AND of Clauses
3CNF: Each clause is an OR of 3 different literals
each literal is either a variable or its complement (NOT)
form.
example: (abc) (ad e) (c e f )
bag of tricks: boolean identities:
56
AND: o (ab)
a b o = o ab
0 0 0 1
0 1 0 1
1 0 0 1
1 1 0 0
0 0 1 0
0 1 1 0
1 0 1 0
1 1 1 1
So,
= (a b o) (a b o) (a b o) (a
bo)
Use DeMorgans laws:
= (abo) (abo) (abo) (a
bo)
can construct OR gates similarly
inserting identities if needed to t 3CNF:
e.g, (ab) = (ab p) (abp)
14.6 NPC proof contd: kclique problem
(CLIQUE)
input: undirected graph G
denition: clique = subset of V that form a complete sub
graph
kclique: complete graph with k vertices
problem: does a clique of size k exist in G?
It is NP Complete
1) NP: obvious: guess and verify.
2b) NPhard: show reduction 3CNFSAT
p
CLIQUE
for each Clause C
i
= (a, b, c),
make 3 vertices a
i
, b
i
, c
i
for group C
i
link each literal to all except its complement in
other clauses.
add edges (a
i
, x
j
) if i ,= j, x ,=a
example: =C
1
C
2
C
3
= (x
1
x
2
x
3
) (x
1
x
2
x
3
) (x
1
x
2
x
3
)
x
1 x
2
x
3
C
1
= x
1
x
2
x
3
x
1
x
2
x
3
C
2
= x
1
x
2
x
3
x
1
x
2
x
3
C
3
= x
1
x
2
x
3
x
1 x
2
x
3
x
1
x
2
x
3
x
1
x
2
x
3
x
1 x
2
x
3
x
1
x
2
x
3
x
1
x
2
x
3
x
1 x
2
x
3
x
1
x
2
x
3
x
1
x
2
x
3
edges for x
1
edges for x
2
edges for x
3
if we can nd a kclique for k clauses, then the clique
represents a certicate.
x
1 x
2
x
3
x
1
x
2
x
3
x
1
x
2
x
3
in the example, there are several solutions!
57
Lecture 15
Approx. Algorithms
This lecture
Approximation algorithms
ratio bound: bounding cost of suboptimal solution within
some factor from ideal
15.1 kVERTEXCOVER problem
input: undirected graph G(V, E)
denition: vertex cover = subset of V thats part of all
edges.
kvertex cover: a vertex cover with k vertices
NP Complete
NP: easy: test whether each vertex in vertex cover is u
or v for every edge (u, v).
reduce kCLIQUE
p
kVERTEXCOVER
15.1.1 Observation:
Graph G has a kCLIQUE iff Gcomplement has a ([V[
k) vertex cover.
G = (V, E): complement of G = (V, E).
i.e., G has same vertices as G, but only those edges that
are not in G.
Claim: if V
/
is a kclique, then V V
/
is a vertex cover
in G.
u
z
v
w
y x
u
z
v
w
y x
complement graph with
vertex cover = { w, z }
= V clique C
Graph G with
six vertices
clique of size 4
C = { u, v, x, y }
Proof: V G are guaranteed not to have edges between
each other; so they are entirely unnecessary in the vertex
cover of G. (or else theyd be redundant)
since G contains all other vertices, all other edges will be
covered.
15.1.2 Approximation algorithm for Vertex Cover
decision problem is NP Complete
nding an optimal vertex cover is NPhard.
However: if we just want to nd a reasonably good
vertex cover, can solve it in polynomial time.
APPROXVERTEXCOVER(G)
1 C / 0
2 E
/
E[G]
3 while E
/
,= / 0
4 do (u, v) EXTRACTRANDOM(E
/
)
5 C Cu, v
6 E
/
E
/
all edges in the form (u, x) or (x, v)
7 return C
This algorithm has a ratio bound of 2.
b
a
c d
e f g
C = { }
b
a
c d
e f g
pick edge (b, c)
C = { b, c }
remove all edges incident to b or c
b
a
c d
e f g
pick edge (e, f)
C = { b, c, e, f }
remove all edges incident to e or f
b
a
c d
e f g
pick edge (d, g)
C = { b, c, e, f, d, g }
remove all edges incident to d or g
approximated vertex cover = { b, c, e, f, d, g }
twice as many as optimal vertex cover = { b, d, e }
b
a
c d
e f g
58
First: C is a vertex cover by construction.
(an edge is removed after its covered by either u or v)
C always adds two vertices per iteration.
The best that an optimal solution can do is to use one
instead of two vertices to cover those same vertices.
15.2 Hamiltonian (HAMCYCLE)
recall: Hamiltonian cycle = simple cycle that visits all
vertices exactly once.
HAMCYCLE (decision problem) is NP Complete.
proof:
1) NP: easy; walk the path
2b VERTEXCOVER
p
HAMCYCLE
use a 12vertex widget for each edge
detail not important
Issue: No approximation of HAMCYCLE
15.3 Traveling Salesman Problem (TSP)
TSP =G, c, k) : G = (V, E)a complete graph,
c : V V Z
k Z,
tour A with cost c(A) k
c(A) =
(u,v)A
c(u, v)
Hamiltonian cycle on a complete graph with cost k
easy to show NP Complete:
1) NP : test if cost k
2b) reduction: Hamiltonian
p
TSP
c(i, j) =
_
0 if there is an edge (i, j)
1 if (i, j) , E; assume no self loop (v, v)
then check for TSP with cost k = 0 (real edges).
any nonedges will have cost > 0.
15.3.1 Approximated TSP
Nearoptimal solution assuming triangle inequality:
simplifying assumption: going direct is always cheaper
than a hop:
c(u, w) c(u, v) +c(v, w)
Approximation: with Minspanning tree.
Ratiobound of 2.
APPROXTSPTOUR(G, c)
pick some vertex r V as root
grow minspanning tree using MSTPRIM(G, c, r)
return preorder tree walk of T
(i.e., visit root before visiting children recursively
as a hamiltonian cycle)
b
a
c
d
e
f g
h
b
a
c
d
e
f g
h
preorder: a, b, c, h, d, e, f, g
1
2
3
4
5
6
7
8
Approximated TSP, assuming
triangular inequality
Proof:
First: Minspanning tree tour would be 2x optimal.
This is because at each point youre going the shortest
distance, but you do it twice.
Second: the preorder traversal order would jump direct,
rather than backtrack to go indirectly. so it can only do
better.
There are better approximations. 2 is still not very
good.
15.3.2 General TSP:
does not have a ratio bound () unless P = NP
Proof by contradiction: if such exists then can reduce
Hamiltonian cycle to TSP by cost assignment:
1. form complete graph E
/
from E such that
c(u, v) =
_
1 if (u, v) E,
[V[ +1 otherwise
b
a
c
d
e
b
a
c
d
e
HAMCYCLE?
cost = 1 (original)
cost = V + 1
Solve as an
instance of TSP
w/ ratio bound =
2. optimal TSP is to take only edges in E. (unit weight).
taking any purple edge means
cost ([V[ +1)
. .
+([V[ 1)
. .
purple cost remaining originals
= (+1)[V[
> [V[
approximation algorithm for TSP with ratio bound ex
ists only if P = NP.
59
15.4 Subsetsum problem
input: S, t) : S =x
1
, x
2
, ...x
n
set of integers;
t = integer target sum, certicate S
/
= subset of S.
Example: S =1, 2, 7, 12, 18, t = 27
S
/
=2, 7, 18 S because 2+7+18 = 27.
15.4.1 Exact solution:
enumerate set of all possible subsets for the sums.
there are 2
n
such subsets (two choices with each mem
ber: include or exclude.)
iterative algorithm to enumerate sums, prune out sums
>t.
15.4.2 NP Complete proof
1) NP obvious (just sum up)
2b) NPH reduction from 3CNFSAT
express clauses as numbers
C
1
= (x
1
x
2
),C
2
= (x
1
x
2
)
x
1
x
2
C
1
C
2
v
1
1 0 1 0
v
/
1
1 0 0 1
v
2
0 1 0 1
v
/
2
0 1 1 0
s
1
0 0 1 0
s
/
1
0 0 2 0
s
2
0 0 0 1
s
/
2
0 0 0 2
t 1 1 4 4
v
i
has 1 in x
i
column, 0 in x
j
columns (i ,= j), and
1 if x
i
appears in clause C
k
, 0 if not.
v
/
i
has 1 in x
i
column, 0 in x
j
columns (i ,= j), and
1 if x
i
does not appear in clause C
k
, 0 if does
s
i
has all 0 in x
j
columns, 1 in column C
i
, 0 otherwise
s
/
i
has all 0 in x
j
columns, 2 in column C
i
, 0 otherwise.
interpret these as decimal numbers (no carry can happen)
15.4.3 polynomialtime approximation
idea: compute a sum that is close to the target sum within
a factor of 1+, 0 < < 1, rather than exactly
trim out an element y if it is can be approximated by z:
y
1+
z y
APPROXSUBSETSUM(S, t, )
1 n [S[
2 L
0
0)
3 for i 1 to n
4 do L
i
MERGE(L
i1
, L
i1
+x
i
),
5 L
i
TRIM(L
i
, /2n)
6 L
i
L
i
elements >t
7 return z
Much more than documents.
Discover everything Scribd has to offer, including books and audiobooks from major publishers.
Cancel anytime.