Faster Algorithm S For Well-Behaved Optimizat Ion Problems

Algorith
ms
Faster
Algorithm
s for
Well-
Behaved
Optimizat
ion
Problems
Dynamic Programming is Blind
■ The algorithms we have studied are relatively inefficient:

■ Matrix chain multiplication: O(n ) 3
■ Longest common subsequence: O(mn)

■ Optimal binary search trees: O(n ) 3
■ Why?
■ We have many choices in computing an optimal solution.
■ We exhaustively (blindly) check all of them.
■ We would much rather like to have a way to decide which choice is the best or at
least restrict the choices we have to try.
■ Greedy algorithms work for problems where we can decide what’s the best
choice.
CSci 3110 — Introduction to Algorithms Greedy Algorithms

Making a Greedy Choice
Greedy choice:
We make the choice that looks best at the moment.
(Every time we make a choice, we greedily try to maximize our profit.)
An example where this does not work:
Finding a longest monotonically increasing subsequence.
Given sequence ‹3 4 5 17 7 8 9›,
a longest monotonically increasing subsequence is ‹3 4 5 7 8 9›.
The greedy choice after choosing ‹3 4 5› is to choose 17, which precludes the
addition of another element and thus results in the suboptimal sequence ‹3 4 5
17›.

A Scheduling Problem
A classroom can be used for one class at a time.

There are n classes that want to use the classroom.
Every class has a corresponding time interval Ij = [sj, fj) during which the the room
would be needed for this class.
Our goal is to choose a maximal number of classes that can be scheduled to use the
classroom without two classes ever using the classroom at the same time.
Assume that the classes are sorted according to increasing finish times; that is,
f 1 < f 2 < … < f n.
5
2 6
1 7
3 8
4

The Structure of an Optimal Schedule
Let Si,j be the set of classes that begin after time fi and end before time sj; that is,
these classes can be scheduled between classes Ci and Cj.
We can add two fictitious classes C0 and Cn + 1 with f0 = –∞ and sn + 1 = +∞. Then
S0,n + 1 is the set of all classes.
Assume that class Ck is part of an optimal schedule of the classes in Si,j.

Then i < k < j, and the optimal schedule consists of a maximal subset of Si,k, {Ck},
and a maximal subset of Sk,j.
Ci Ck Cj
Si,k Sk,j

The Structure of an Optimal Schedule
Hence, if Q(i, j) is the size of an optimal schedule for set Si,j, we have

Making a Greedy Choice
Lemma: There exists an optimal schedule for the set Si,j that contains the class Ck
in Si,j that finishes first, that is, the class Ck in Si,j with minimal index k.
Lemma: If we choose class Ck as in the previous lemma, then set Si,k is empty.

A Recursive Greedy Algorithm
Recursive-Activity-Selector(s, f, i, j)
1 m ← i+1
2 while m < j and sm < fi
3 do m ← m+1
4 if m < j
then return {am} U Recursive-Activity-Selector(s, f, m, j)
5 else return ∅
Line 2 finds the finds the first activity in Sij
Assume the activities have already been sorted by finish times, the running time of
the call Recursive-Activity-Selector(s, f, i, j) is Θ(n)

An Iterative Linear-Time Greedy Algorithm
Iterative-Schedule(S)
1 n ← |S|
2 m ← –∞
3 O←∅
3 for i = 1..n
4 do if si ≥ m
5 then O ← O ∪ {Ci}
6 m ← fi
7 return O

1 n ← |S|
2 m ← –∞
3 O←∅
3 for i = 1..n
4 do if si ≥ m
6 m ← fi
7 return O

1 n ← |S|
2 m ← –∞
3 O←∅
3 for i = 1..n
4 do if si ≥ m
6 m ← fi
7 return O

1 n ← |S|
2 m ← –∞
3 O←∅
3 for i = 1..n
4 do if si ≥ m
6 m ← fi
7 return O

1 n ← |S|
2 m ← –∞
3 O←∅
3 for i = 1..n
4 do if si ≥ m
6 m ← fi
7 return O

1 n ← |S|
2 m ← –∞
3 O←∅
3 for i = 1..n
4 do if si ≥ m
6 m ← fi
7 return O

1 n ← |S|
2 m ← –∞
3 O←∅
3 for i = 1..n
4 do if si ≥ m
6 m ← fi
7 return O

1 n ← |S|
2 m ← –∞
3 O←∅
3 for i = 1..n
4 do if si ≥ m
6 m ← fi
7 return O

1 n ← |S|
2 m ← –∞
3 O←∅
3 for i = 1..n
4 do if si ≥ m
6 m ← fi
7 return O

1 n ← |S|
2 m ← –∞
3 O←∅
3 for i = 1..n
4 do if si ≥ m
6 m ← fi
7 return O

1 n ← |S|
2 m ← –∞
3 O←∅
3 for i = 1..n
4 do if si ≥ m
6 m ← fi
7 running
The returntime
O of this algorithm is obviously linear.
It’s correctness follows from the following lemma:
Lemma: Let O be the current set of selected classes, and let Ck be the last class
added to O. Then any class Cl, l > k, that conflicts with a class in O conflicts with Ck.

A Variation of the Problem
Instead of maximizing the number of classes we want to schedule, we want to

maximize the total time the classroom is in use.
Our dynamic programming approach still remains unchanged.
But none of the obvious greedy choices would work:
■Choose the class that starts earliest/latest

■Choose the class that finishes earliest/latest
■Choose the longest class

When Do Greedy Algorithms Produce an Optimal Solution?
The problem must have two properties:

Greedy choice property:
An optimal solution can be obtained by making choices that seem best at the
time, without considering their implications for solutions to subproblems.
Optimal substructure:
An optimal solution can be obtained by augmenting the partial solution
constructed so far with an optimal solution of the remaining subproblem.

Text Encoding
Our next goal is to develop a code that represents a given text as compactly as
possible.
A standard encoding is ASCII, which represents every character using 7 bits:
“An English sentence”
1000001 (A) 1101110 (n) 0100000 ( ) 1000101 (E) 1101110 (n) 1100111 (g)
1101100 (l) 1101001 (i) 1110011 (s) 1101000 (h) 0100000 ( ) 1110011 (s)
1100101 (e) 1101110 (n) 1110100 (t) 1100101 (e) 1101110 (n) 1100011 (c)
1100101 (e)
= 133 bits ≈ 17 bytes

Of course, this is wasteful because we can encode 12 characters in 4 bits:
‹space› = 0000 A = 0001 E = 0010 c = 0011 e = 0100 g = 0101 h = 0110
i = 0111 l = 1000 n = 1001 s = 1010 t = 1011
Then we encode the phrase as
0001 (A) 1001 (n) 0000 ( ) 0010 (E) 1001 (n) 0101 (g) 1000 (l) 0111 (i)
1010 (s) 0110 (h) 0000 ( ) 1010 (s) 0100 (e) 1001 (n) 1011 (t) 0100 (e)
1001 (n) 0011 (c) 0100 (e)
This requires 76 bits ≈ 10 bytes

An even better code is given by the following encoding:
‹space› = 000 A = 0010 E = 0011 s = 010 c = 0110 g = 0111 h = 1000

i = 1001 l = 1010 t = 1011 e = 110 n = 111
Then we encode the phrase as
0010 (A) 111 (n) 000 ( ) 0011 (E) 111 (n) 0111 (g) 1010 (l) 1001 (i)
010 (s) 1000 (h) 000 ( ) 010 (s) 110 (e) 111 (n) 1011 (t) 110 (e) 111 (n)
0110 (c) 110 (e)
This requires 65 bits ≈ 9 bytes

Codes That Can Be Decoded
Fixed-length codes:
Every character is encoded using the same number of bits.
To determine the boundaries between characters, we form groups of w bits, where
w is the length of a character.
Examples:
■ASCII
■Our first improved code
Prefix codes:
No character is the prefix of another character.
Examples:
■Fixed-length codes
■Huffman codes

Why Prefix Codes?
Consider a code that is not a prefix code:
a = 01 m = 10 n = 111 o = 0 r = 11 s = 1 t = 0011
Now you send a fan-letter to you favorite movie star. One of the sentences is “You
are a star.”
You encode “star” as “1 0011 01 11”.

Your idol receives the letter and decodes the text using your coding table:
100110111 = 10 0 11 0 111 = “moron”
Oops, you have just insulted your idol.

Non-prefix codes are ambiguous.

Why Are Prefix Codes Unambiguous?
It suffices to show that the first character can be decoded unambiguously. We then
remove this character and are left with the problem of decoding the first character of
the remaining text, and so on until the whole text has been decoded.
c
Assume that there are two characters c and c' that could potentially be the first
characters in the text. Assume that the encodings are x0x1…xk and y0y2…yl. Assume
further that k ≤ l.
c
c'
Since both c and c' can occur at the beginning of the text, we have xi = yi,
for 0 ≤ i ≤ k; that is, x0x1…xk is a prefix of y0y2…yl, a contradiction.

Representing a Prefix-Code Dictionary
Our example:
‹space› = 000 A = 0010 E = 0011 s = 010 c = 0110 g = 0111 h = 1000
i = 1001 l = 1010 t = 1011 e = 110 n = 111
0 1
0 1 0 1
0 1 0 1 0 1 0 1
‹spc› s e n
0 1 0 1 0 1 0 1
A E c g h i l t

Representing a Prefix-Code Dictionary
Our example:
‹space› = 000 A = 0010 E = 0011 s = 010 c = 0110 g = 0111 h = 1000
i = 1001 l = 1010 t = 1011 e = 110 n = 111
0 1
0 1 0 1
0 1 0 1 0 1 0 1
‹spc› s e n
0 1 0 1 0 1 0 1
A E c g h i l t

The Cost of Prefix Codes
Let C be the set of characters in the text to be encoded, and let f(c) be the frequency
of character c.
Let dT(c) be the depth of node (character) c in the tree representing the code. Then
is the number of bits required to encode the text using the code represented by
tree T. We callInB(T)
Observation: theTcost
a tree of tree T. an optimal prefix code, every internal node
representing
has two children.

Huffman’s Algorithm
Huffman(C)
1 n ← |C| f:5 e:9 c:12 b:13 d:16 a:45
2 Q←C
3 for i = 1..n – 1
4 do allocate a new node z
5 left[z] ← x ← Delete-Min(Q)
6 right[z] ← y ← Delete-Min(Q)
7 f[z] ← f[x] + f[y]
8 Insert(Q, z)
9 return Delete-Min(Q)

14
Huffman(C)
1 n ← |C| f:5 e:9 c:12 b:13 d:16 a:45
2 Q←C
3 for i = 1..n – 1
7 f[z] ← f[x] + f[y]
8 Insert(Q, z)

Huffman(C)
1 n ← |C| c:12 b:13 14 d:16 a:45
2 Q←C
3 for i = 1..n – 1
f:5 e:9
7 f[z] ← f[x] + f[y]
8 Insert(Q, z)

25
Huffman(C)
1 n ← |C| c:12 b:13 14 d:16 a:45
2 Q←C
3 for i = 1..n – 1
4 do allocate a new node z f:5 e:9
7 f[z] ← f[x] + f[y]
8 Insert(Q, z)

Huffman(C)
1 n ← |C| 14 d:16 25 a:45
2 Q←C
3 for i = 1..n – 1
f:5 e:9 c:12 b:13
7 f[z] ← f[x] + f[y]
8 Insert(Q, z)

30
Huffman(C)
1 n ← |C| 14 d:16 25 a:45
2 Q←C
3 for i = 1..n – 1
4 do allocate a new node z f:5 e:9 c:12 b:13
7 f[z] ← f[x] + f[y]
8 Insert(Q, z)

Huffman(C)
1 n ← |C| 25 30 a:45
2 Q←C
3 for i = 1..n – 1
c:12 b:13 14 d:16
6 right[z] ← y ← Delete-Min(Q) f:5 e:9
7 f[z] ← f[x] + f[y]
8 Insert(Q, z)

55
Huffman(C)
1 n ← |C| 25 30 a:45
2 Q←C
3 for i = 1..n – 1
4 do allocate a new node z c:12 b:13 14 d:16
6 right[z] ← y ← Delete-Min(Q) f:5 e:9
7 f[z] ← f[x] + f[y]
8 Insert(Q, z)

Huffman(C)
1 n ← |C| a:45 55
2 Q←C
3 for i = 1..n – 1
25 30
6 right[z] ← y ← Delete-Min(Q) c:12 b:13 14 d:16
7 f[z] ← f[x] + f[y]
8 Insert(Q, z)
f:5 e:9

100
0 1
Huffman(C)
1 n ← |C| a:45 55
2 Q←C 0
0 1
3 for i = 1..n – 1
4 do allocate a new node z 25 30
5 left[z] ← x ← Delete-Min(Q) 0 1 0 1
6 right[z] ← y ← Delete-Min(Q) c:12 b:13 14 d:16
7 f[z] ← f[x] + f[y] 100 101 111
0 1
8 Insert(Q, z)
9 return Delete-Min(Q) f:5 e:9
1100 1101

Greedy Choice
Why is merging the two nodes with smallest frequency into a subtree a greedy
choice?
We can alternatively define B(T) as
where
B(v) = 0 if v is a leaf and

B(v) = f(left[v]) + f(right[v]) if v is an internal node.
By merging the two nodes with lowest frequency, we greedily try to minimize the
cost the new node contributes to B(T).

Greedy Choice
Lemma: There exists an optimal prefix code such that the two characters with
smallest frequency are siblings and have maximal depth in T.
Let x and y be two such characters, and let

T be a tree representing an optimal prefix T
code.
Let a and b be two sibling leaves of x y

maximal depth in T.
a b
Assume that f(x) ≤ f(y) and
f(a) ≤ f(b). T'
This implies that f(x) ≤ f(a) and

f(y) ≤ f(b).
a b
Let T' be the tree obtained by exchanging
a and x and b and y. x y

The cost difference between trees T and T' is
T T'
x y a b
a b x y

Optimal Substructure
After joining two nodes x and y by making them children of a new node z, the
algorithm treats z as a leaf with frequency f(z) = f(x) + f(y).
Let C' be the character set in which x and y are replaced by the single character z
with frequency f(z) = f(x) + f(y), and let T' be an optimal tree for C'.
Let T be the tree obtained from T' by making x and y children of z.
We observe the following relationship between B(T) and B(T'):

B(T) = B(T') + f(x) + f(y)

Lemma: If T' is optimal for C', then T is optimal for C.
Assume the contrary. Then there exists a better tree T'' for C.
Also, there exists a tree T''' at least as good as T'' for C where x and y are sibling
leaves of maximal depth.
The removal of x and y from T''' turns their parent into a leaf; we can associate this
leaf with z.
The cost of the resulting tree is B(T''') – f(x) – f(y) < B(T) – f(x) – f(y) = B(T').
This contradicts the optimality of B(T').
Hence, T must be optimal for C.

Summary
Greedy algorithms are efficient algorithms for optimization problems that exhibit
two properties:
Greedy choice property: An optimal solution can be obtained by making locally
optimal choices.
Optimal substructure: An optimal solution contains within it optimal solutions
to smaller subproblems.
If only optimal substructure is present, dynamic programming may be a viable
approach; that is, the greedy choice property is what allows us to obtain faster
algorithms than what can be obtained using dynamic programming.

Faster Algorithm S For Well-Behaved Optimizat Ion Problems

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Faster Algorithm S For Well-Behaved Optimizat Ion Problems

Uploaded by

Copyright:

Available Formats

Algorith

■ The algorithms we have studied are relatively inefficient:

■ Longest common subsequence: O(mn)

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

A classroom can be used for one class at a time.

CSci 3110 — Introduction to Algorithms Greedy Algorithms

Assume that class Ck is part of an optimal schedule of the classes in Si,j.

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

Line 2 finds the finds the first activity in Sij

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

It’s correctness follows from the following lemma:

CSci 3110 — Introduction to Algorithms Greedy Algorithms

Instead of maximizing the number of classes we want to schedule, we want to

■Choose the class that starts earliest/latest

CSci 3110 — Introduction to Algorithms Greedy Algorithms

The problem must have two properties:

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

‹space› = 000 A = 0010 E = 0011 s = 010 c = 0110 g = 0111 h = 1000

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

Consider a code that is not a prefix code:

You encode “star” as “1 0011 01 11”.

100110111 = 10 0 11 0 111 = “moron”

Oops, you have just insulted your idol.

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

B(v) = 0 if v is a leaf and

CSci 3110 — Introduction to Algorithms Greedy Algorithms

Let x and y be two such characters, and let

Let a and b be two sibling leaves of x y

This implies that f(x) ≤ f(a) and

CSci 3110 — Introduction to Algorithms Greedy Algorithms

CSci 3110 — Introduction to Algorithms Greedy Algorithms

Let T be the tree obtained from T' by making x and y children of z.