You are on page 1of 57

Greedy Algorithms

Greed is good. (Some of the time)

Outline

Elements of greedy algorithm
 

Greedy choice property Optimal substructures Kruskal’s algorithm Prim’s algorithm

Minimum spanning tree
 

Huffman code  Activity selection

Jaruloj Chongstitvatana

Chapter 3: Greedy Algorithms

2

Introduction

Concepts
 

Choosing the best possible choice at each step. This decision leads to the best over all solution.

Greedy algorithms do not always yield optimal solutions.

Jaruloj Chongstitvatana

Chapter 3: Greedy Algorithms

3

Elements of Greedy Algorithms

Greedy-choice property

A globally optimal solution is derived from a locally optimal (greedy) choice. When choices are considered, the choice that looks best in the current problem is chosen, without considering results from subproblems.

Jaruloj Chongstitvatana

Chapter 3: Greedy Algorithms

5

This property is important for both greedy algorithms and dynamic programming.Optimal substructures  A problem has optimal substructure if an optimal solution to the problem is composed of optimal solutions to subproblems.  Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 6 .

 Dynamic-programming problems are often solved in a bottom-up manner. but it cannot depend on any future choices or on the solutions to subproblems. Dynamic Programming Greedy algorithm  The best choice is made at each step and after that the subproblem is solved.  A greedy strategy usually progresses in a top-down fashion.  The choice made by a greedy algorithm may depend on choices so far. Jaruloj Chongstitvatana Dynamic programming  A choice is made at each step.s. reducing each given problem instance to a smaller one. making one greedy choice after another. Chapter 3: Greedy Algorithms 7 .  The choice made at each step usually depends on the solutions to subproblems.Greedy Algorithm v.

Convert the recursive algorithm to an iterative algorithm. Thus. Show that all but one of the subproblems induced by having made the greedy choice are empty. Chapter 3: Greedy Algorithms 8    Jaruloj Chongstitvatana .Steps in Design Greedy Algorithms    Determine the optimal substructure of the problem. Develop a recursive solution. one of the optimal choices is the greedy choice. Develop a recursive algorithm that implements the greedy strategy. it is always safe to make the greedy choice. Prove that at any stage of the recursion.

Demonstrate that. what remains is a subproblem with the property that if we combine an optimal solution to the subproblem with the greedy choice we have made. we arrive at an optimal solution to the original problem. so that the greedy choice is always safe. after a choice is made and there is only one subproblem left to be solved.  Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 9 . having made the greedy choice. Prove that there is always an optimal solution to the original problem that makes the greedy choice.Shortcuts Design   Form the optimization problem so that.

Minimum Spanning Tree .

Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 11 . E) be an undirected graph.v)∈T  Let A be a subset of some minimum spanning tree. v)} is also a subset of a minimum spanning tree. T is a minimum spanning tree of G if T ⊆ E is an acyclic subset that connects all of the vertices and whose total weight w(T ) = ∑ w(u. An edge (u. v) is a safe edge for A if A ∪ {(u. (u. v) is minimized.Definitions  Let G = (V.

E) be an undirected graph. A cut (S. A cut respects a set A of edges if no edge in A crosses the cut. An edge (u. v) ∈ E crosses the cut (S. V − S) if one of its endpoints is in S and the other is in V − S.Definitions  Let G = (V. An edge is a light edge crossing a cut if its weight is the minimum of any edge crossing the cut. V − S) of G is a partition of V. Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 12 .

v) is safe for A. A be a subset of E that is included in some minimum spanning tree for G. (S. Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms S A x u light edge v y V-S 13 . E) be a connected. v) is a light edge crossing (S. then edge (u.Theorem Let G = (V. V − S). If (u. V − S) be any cut of G that respects A. undirected graph with a real-valued weight function w defined on E.

Proof Let T be a minimum spanning tree that includes A. then (u. Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms S A x u light edge T y v V-S 14 .v) and (x.v) is not in T.y) crossing S and V-S.v) is in T.v).y) cross S and V-S. Since both (u. there is a cycle for edges in T ∪ (u. Another spanning tree T’ can be created from T. If (u. then T contains an edge (x. If (u.v) is safe for A.

Proof S A x S A x u light edge u light edge T v y T’ v y V-S V-S Remove edge (x.v) -> from cycle Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 15 . T’ (u.y)create athe cycle Add edge is created -> cutT.

Proof Thus.v).y)} ∪ {(u. T’ is a spanning tree that includes A.v)}.y). As a result.y) + w(u. Next. w(T’ ) ≤ w(T).w(x. w(T’ ) = w(T). Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 16 . T’ is also a minimum spanning tree . w(u. T’ = T-{(x. w(T’ ) = w(T) . Thus.v) is a light edge cross S and V-S. From the construction of T’ .v) ≤ w(x. we need to show that T’ is a minimum spanning tree. v) is safe for A. Since T is a minimum spanning tree. Thus. (u. Since (u. Then.

EC) be a connected component (tree) in the forest GA = (V. Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 17 . V−VC) respects A. then (u. Proof (VC. E) be a connected. v) is safe for A. A). undirected graph with a realvalued weight function w defined on E. A be a subset of E that is included in some minimum spanning tree for G. v) is a light edge for this cut.Corollary Let G = (V. Therefore. (u. and C = (VC. v) is safe for A. v) is a light edge connecting C to another tree in GA. If (u. and (u.

because at each step it adds to the forest an edge of least possible weight. choose the edge of the least weight. Repeatedly connect the trees to create a subset of a minimum spanning tree. In connecting two trees. v) is a safe edge for C1. v). until all nodes are covered.Kruskal’s Algorithm  Concept    Build a forest of minimum spanning trees.  Let C1 and C2 denote the two trees that are connected by (u. Chapter 3: Greedy Algorithms 18  Jaruloj Chongstitvatana . Since (u. v) must be a light edge connecting C1 to some other tree. Kruskal’s algorithm is a greedy algorithm. we need to prove that (u.

Example of Kruskal’s Algorithm 12 2 5 8 7 9 3 3 5 1 8 9 11 1 10 Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 19 .

v) return A Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 20 . taken in nondecreasing order by weight do if FIND-SET(u) ≠ FIND-SET(v) then A = A ∪ {(u. v) in E.w) A=∅ for each vertex v in V [G] do MAKE-SET(v) sort the edges of E in nondecreasing order by weight w for each edge (u.Kruskal’s Algorithm MST-KRUSKAL(G. v)} UNION(u.

Kruskal’s Algorithm: Complexity MST-KRUSKAL(G.w) A=∅ for each vertex v in V [G] O(v) do MAKE-SET(v) O(e lg e) sort the edges of E in nondecreasing order by weight w for each edge (u. taken in nondecreasing order by weight do if FIND-SET(u) ≠ FIND-SET(v) then A = A ∪ {(u. v)} UNION(u. O(e lg v) Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 21 . v) in E. v) O((v+e) lg v) return A 2 e ≤ v .

 Grow the tree until it spans all the vertices in V. A). a light edge is added to the tree A that connects A to an isolated vertex of GA = (V.   At each step.  The tree starts from an arbitrary root vertex r.Prim’s Algorithm Prim’s algorithm has the property that the edges in the set A always form a single tree. Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 22 .

Example of Prim’s Algorithm 12 2 5 8 7 9 3 3 5 1 8 9 11 1 10 Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 23 .

r) for each u in V[G] do key[u] = ∞ π[u] = NIL key[r] = 0 Q = V[G] while Q ≠ ∅ do u = EXTRACT-MIN(Q) for each v in Adj[u] do if v in Q and w(u.Prim’s Algorithm PRIM(G. v) < key[v] then π[v] = u key[v] = w(u.w. v) Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 24 .

Execution of Prim’s Algorithm 0 2 5 8 12 9 12 3 9 1 3 7 3 8 9 8 2 1 3 5 11 5 1 1 7 10 10 5 Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 25 .

r) for each u in V[G] Also build min-heap do key[u] = ∞ O(v) π[u] = NIL key[r] = 0 O(e Q = V[G] while Q ≠ ∅ do u = EXTRACT-MIN(Q) O(lg v) O(v for each v in Adj[u] do if v in Q and w(u.w. v) < key[v] then π[v] = u key[v] = w(u.Prim’s Algorithm:Complexity PRIM(G. v) O(lg v) Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms lg v) lg v) 26 .

Huffman Code .

110. 101. where each character has a frequency of occurrences. 110. 111. 101. {0. 111. 1010. 1000. 1001} is a prefix code. {0.Problem Find an optimal prefix code representing a set of characters in a file. Chapter 3: Greedy Algorithms 28  Optimality  Jaruloj Chongstitvatana . 0111} is not prefix code The code yields a file with the minimum number of bits.  Prefix code     Codes in which no codeword is also a prefix of some other codeword.

Creating Huffman Code: Example 100 1 0 64 0 34 1 30 1 0 1 0 0 15 1 ก: 36 0 ข: 17 100 ค: 17 101 ง: 15 110 จ: 10 ฉ: 5 1110 1111 29 Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms .

Chapter 3: Greedy Algorithms 30  Jaruloj Chongstitvatana . one for each letter of the alphabet. then the tree for an optimal prefix code has exactly |C| leaves. If C is the alphabet from which the characters are drawn and all character frequencies are positive. in which every nonleaf node has two children.Optimal Code  An optimal code for a file is always represented by a full binary tree. and exactly |C|−1 internal nodes.

Full Binary Trees for Prefix Code Tree for 1 letter Tree for 2 letters 1 1 Tree for 3 letters A 2 C 4 2 B 2 3 Tree for 4 letters B A 3 C 4 3 1 B A 2 31 B 3 1 A 1 A 1 2 Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms .

Full Binary Trees for Prefix Code Tree for 4 letters Tree for 5 letters B C 4 3 C 4 3 1 Jaruloj Chongstitvatana B A 1 2 5 B A 2 D 4 D 5 C 4 3 1 Chapter 3: Greedy Algorithms C 3 1 A 2 B A 2 32 .

Creating Huffman Code: Example กขคงจฉ: 100 1 0 0 ขคงจฉ: 64 1 ขค: 34 0 1 0 งจฉ: 30 1 จฉ: 15 0 1 1111 33 ก: 36 0 Jaruloj Chongstitvatana ข: 17 100 ค: 17 101 ง: 15 110 จ: 10 ฉ: 5 1110 Chapter 3: Greedy Algorithms .

the full binary tree for an optimal prefix code for C letters has exactly C leaves. and 0 internal node. If there is one letter. and exactly C−1 internal nodes. the binary tree requires only 1 leaf node. Induction hypotheses: For C< n. and exactly C−1 internal nodes. Proof by induction.Theorem A full binary tree for an optimal prefix code for C letters has exactly C leaves. Basis: C=1.  Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 34 .

Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 35 . To create a full binary tree or optimal prefix code. we can take a full binary tree for an optimal prefix code for C letters. and add another leaf node L by either   adding a new root node R and put L and the old root of T as its children. In either case.Theorem  Induction Step: Let T be a full binary tree for an optimal prefix code for C+1 letters. or replacing a leaf node N of T by an internal node and put L and N as its children. the number of leaf nodes of T is C+1 and the number of internal nodes is C.

f INSERT(Q.left = x = EXTRACT-MIN(Q) z.Creating Huffman Code: Algorithms HUFFMAN(C) Use min-heap for Q n = |C| O(n) to build min-heap Q=C for i = 1 to n − 1 do allocate a new node z z.right = y = EXTRACT-MIN(Q) z. z) ►Return the root of the tree.f + y.f =x. return EXTRACT-MIN(Q) Jaruloj Chongstitvatana O(n lg n) O(lg n) Chapter 3: Greedy Algorithms 36 .

and x and y be two characters in C having the lowest frequencies.  Then.Greedy-choice property of Huffman Code Let C be an alphabet in which each character c ∈ C has frequency f [c]. there exists an optimal prefix code for C in which the codewords for x and y have the same length and differ only in the last bit. Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 37 .

 If we can do this. then their codewords will have the same length and differ only in the last bit.Proof  The idea of the proof is to :   take the tree T representing an arbitrary optimal prefix code modify it to make a tree representing another optimal prefix code such that the characters x and y appear as sibling leaves of maximum depth in the new tree. Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 38 .

Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 39 . in order. Since f [x] and f [y] are the two lowest leaf frequencies. Assume that f [a] ≤ f [b] and f [x] ≤ f [y]. we have f [x] ≤ f [a] and f [y] ≤ f [b]. and f [a] and f [b] are two arbitrary frequencies. in order.Proof Let a and b be two characters that are sibling leaves of maximum depth in T .

Proof Exchange the positions of a and x in T to produce T’ . B(T) − B(T’ ) = ∑ f(c) dT (c) − ∑ f(c)dT’ (c) c∈C c∈C = f [x] dT (x) + f [a] dT (a) − f [x] dT’ (x) − f [a] dT’ (a) = f [x] dT (x) + f [a] dT (a) − f [x] dT (a) − f [a] dT (x) = ( f [a] − f [x])( dT (a) − dT (x)) ≥ 0 T x y a Jaruloj Chongstitvatana T’ a y b x Chapter 3: Greedy Algorithms b 40 .

Similarly. B(T’’ ) = B(T). it does not increase the cost. T’’ Thus. B(T) ≤ B(T’’ ). T’’ is an optimal tree in which x and y appear as sibling leaves of b maximum depth. B(T’’ ) ≤ B(T). exchange the positions of b T’ and y in T’ to produce T’’. Then. and so B(T’ ) − B(T’’ ) ≥ 0. Since T is optimal. Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms a x b a x y 41 . y Therefore.Proof Then.

C’ be the alphabet C with characters x. x and y be two characters in C with minimum frequency. represents an optimal prefix code for the alphabet C. so that C’ = C − {x.Optimal-substructure Property Let C be a given alphabet with frequency f [c] defined for each character c ∈ C. y} ∪ {z}. obtained from T ’ by replacing the leaf node for z with an internal node having x and y as children. Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 42 . y removed and (new) character z added. and T be any tree representing an optimal prefix code for the alphabet C ’. define f for C’ as for C. except that f [z] = f [x] + f [y].  Then the tree T .

we have f [x]dT (x) + f [y]dT (y) = ( f [x] + f [y])( dT’ (z) + 1) = f [z] dT ’ (z) + ( f [x] + f [y]) Thus. f [c] dT (c) = f [c] dT ’ (c). y}. B(T) = B(T ’) + f [x] + f [y].Proof Show that the cost B(T) can be expressed in terms of the cost B(T’) by considering the component costs. Since dT (x) = dT (y) = dT ’(z) + 1. we have   dT (c) = dT ’ (c). Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 43 . B(T ’) = B(T) − f [x] − f [y] . That is. For each c ∈ C − {x.

Then there exists a tree T ’’ such that B(T ’’) < B(T).Proof We now prove by contradiction. Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 44 . Suppose T does not represent an optimal prefix code for C. T ’’ has x and y as siblings. Without loss of generality.

Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 45 . B(T ’’’) = B(T ’’) − f [x] − f [y] < B(T) − f [x] − f [y] = B(T ’). Thus.Proof Let T ’’’ be the tree T ’’ with the common parent of x and y replaced by a leaf z with frequency f [z] = f [x] + f [y]. Then. T must represent an optimal prefix code for the alphabet C. We reach a contradiction to the assumption that T ’ represents an optimal prefix code for C’.

Interval Scheduling .

. . an} of n proposed activities that wish to use the same resource. f j ) do not overlap . a2. activity ai takes place during the half-open time interval [si .Problem definition Let S be {a1. where 0 ≤ si < fi < ∞. Activities ai and aj are compatible if the intervals [si . .  Only one activity can use the resource at a time. fi ) and [s j . Chapter 3: Greedy Algorithms 47   Jaruloj Chongstitvatana . . The activity-selection problem is to select a largest subset of mutually compatible activities. fi ).   If selected.  Each activity ai has a start time si and a finish time fi.

Example S A1 A2 A3 Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 48 .

and 0 ≤ i. Chapter 3: Greedy Algorithms 49 Jaruloj Chongstitvatana .n +1 .   Add activities a0 and an+1 such that f0 = 0 and sn+1 =∞. Then S = S0.Subproblems  Sij = {ak ∈ S : fi ≤ sk < fk ≤ s j }  the subset of activities in S that can start after activity ai finishes and finish before activity aj starts. j ≤ n + 1.

j ] = c[i. Let c[i. j ] =  0  i< j k< max {c[i. Then.Optimal Solution   Let Aij be an optimal solution to Sij . k] + c[k.  c[i. j ] be the number of activities in a maximum-size subset of mutually compatible activities in Sij . j ] + 1 if Sij = ∅ if ak be an activity in Aij . j ] = 0 for i ≥ j (i. j ] + 1} if Sij ≠ ∅ ak∈Sij Chapter 3: Greedy Algorithms 50 Jaruloj Chongstitvatana .  c[i. Sij = ∅) ak  c[i. k] + c[k.e.

so that choosing am leaves the subproblem Smj as the only one that may be nonempty. Then. and let am be the activity in Si j with the earliest finish time: fm = min { fk : ak ∈ Si j } . Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 51 .  Activity am is used in some maximum-size subset of mutually compatible activities of Si j .  The subproblem Sim is empty.Theorem  Consider any nonempty subproblem Si j .

  If ak = am. and Ai j is sorted in monotonically increasing order of finish time.Proof: First part Let Ai j be a maximum-size subset of mutually compatible activities of Si j .  Let ak be the first activity in Ai j. we construct the subset A’i j = Ai j − {ak} ∪ {am}. am is used in some maximum-size subset of mutually compatible activities of Si j. S Aij ak A’ij  am Chapter 3: Greedy Algorithms 52 Jaruloj Chongstitvatana . If ak ≠ am.

we see that A’i j is a maximum-size subset of mutually compatible activities of Si j that includes am.Proof: First part  The activities in A’i j are disjoint. Noting that A’i j has the same number of activities as Ai j . since the activities in Aij are. S Aij ak A’ij am  Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 53 . ak is the first activity in Ai j to finish. and fm ≤ fk .

We conclude that Sim is empty.Proof: Second part Let Sim be nonempty.   Then. ai am S ak Chapter 3: Greedy Algorithms  aj 54 Jaruloj Chongstitvatana . there is an activity ak such that fi ≤ sk <fk ≤ sm< fm. which contradicts our choice of am. Then. ak is also in Si j and it has an earlier finish time than am.

Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 55 .Greedy Solution Ai j = {ak} ∪ Akj where ak is the activity which finishes earliest among activities in Si j.

m. f. f. n) else return ∅ Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 56 . i.n+1. n) m=i+1 while m ≤ n and s[m] < f [i] ► Find the first activity in Si. do m= m+1 if m ≤ n then return {am} ∪ ACTIVITY-SELECTOR(s.Recursive Algorithms ACTIVITY-SELECTOR(s.

f ) n = length[s] A = {a1} i=1 for m = 2 to n do if s[m ] ≥ f[i] then A = A ∪ {am} i=m return A Jaruloj Chongstitvatana Chapter 3: Greedy Algorithms 57 .Iterative Algorithm GREEDY-ACTIVITY-SELECTOR(s.