# Greedy Algorithm

Lecturer notes is purely design and developed by Er. Mohd. Arif Siddique
Lecturer Dept. of Computer Science & IT RGEC, Meerut

unit5

1

General form of Greedy Algorithms
Algorithm greedy (C) //C is an input set having n element { S←∅ // S is a solution set while not solution (S) and C ≠ ∅ do x ← extract best element from C. if (x is feasible) Then C ← C - {x} // extract X from input set C S ← S ∪ {x} // add X into solution if solution (S) = ∅ then return “no solutions” }

Time complexity excluding sorting: O(n)
Unit 3 Greedy Algorithms 2

A Greedy Algorithm
A greedy algorithm always makes the choice that looks best at the moment.  In dynamic programming, the optimal solution is described in a recursive manner, and then is computed ``bottom-up''. Dynamic programming is a powerful technique, but it often leads to algorithms with higher than desired running times.  An alternative design technique, called greedy algorithms. This method typically leads to simpler and faster algorithms, but it is not as powerful or as widely applicable as dynamic programming.  The greedy concept make the choice that looks best at the moment in this hope that local optimal choices lead to global optimal solution  Even when greedy algorithms do not produce the optimal solution, they often provide fast heuristics (non-optimal solution strategies), and are often used in finding good approximations.

Unit 3

Greedy Algorithms

3

Problem 1: Activity-Selection Problem
Problem:Given a set S = {1, 2, …, n} of n proposed activities, with a start time si and a finish time fi for each activity i, select a maximumsize set of mutually compatible activities. Compatible Activities: Activities i and j are compatible if the internal [si, fi) and [sj, fj) do not overlap, i.e, i and j are compatible if si≥ fj and sj ≥ fi

Goal: Select a maximum-size set of mutually compatible activities
Unit 3 Greedy Algorithms 4

Greedy Activity-Selection Algorithm
Sort the input activities by increasing finishing time. f 1 ≤ f2 ≤ . . . ≤ fn Algo GREEDY-ACTIVITY-SELECTOR (s[ ], f[ ]) { n = length [s] A={i} j=1 For(i = 2 to n ){ if (si ≥ fj ){ A= AU{i} j=i } } return set A }

Time complexity excluding sorting: O(n)
Unit 3 Greedy Algorithms 5

Example
Given 11 activities
A s f p 1 4 q 3 5 r 0 6 s 5 7 t 3 8 u 5 9 v 6 10 w 8 11 x 8 12 y 2 13 z 12 14

Here all input activities are already sorted by increasing finishing time.
z y x w v u t s r q p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Unit 3

Greedy Algorithms

6

Example cont…
z y x w v u t s r q
0 1

p
2 3 4 5 6 7 8 9 10 11 12 13 14

Initialization A={p}

Activity ={p}

Unit 3

Greedy Algorithms

7

Example cont…
z y x w v u t s r q p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ‘p’ and ‘q’ are not compatible (overlapped) So don't add it in to solution

Activity ={p}
Unit 3 Greedy Algorithms 8

Example cont…
z y x w v u t s r q p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ‘p’ and ‘r’ are not compatible (overlapped) So don't add it in to solution

Activity ={p}
Unit 3 Greedy Algorithms 9

Example cont…
z y x w v u t s r q p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ‘p’ and ‘s’ are compatible, So add ‘s’ in to solution

Activity ={p, s, }
Unit 3 Greedy Algorithms 10

Example cont…
z y x w v u t s r q p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ‘s’ and ‘t’ are not compatible (overlapped) So don't add it in to solution

Activity ={p, s, }
Unit 3 Greedy Algorithms 11

Example cont…
z y x w v u t s r q p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ‘s’ and ‘u’ are not compatible (overlapped) So don't add it in to solution

Activity ={p, s, }
Unit 3 Greedy Algorithms 12

Example cont…
z y x w v u t s r q p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ‘s’ and ‘v’ are not compatible (overlapped) So don't add it in to solution

Activity ={p, s, }
Unit 3 Greedy Algorithms 13

Example cont…
z y x w v u t s r q p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ‘s’ and ‘w’ are compatible, So add ‘w’ in to solution

Activity ={p, s, w, }
Unit 3 Greedy Algorithms 14

Example cont…
z y x w v u t s r q p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ‘s’ and ‘x’ are not compatible (overlapped) So don't add it in to solution

Activity ={p, s, w, }
Unit 3 Greedy Algorithms 15

Example cont…
z y x w v u t s r q p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ‘w’ and ‘y’ are not compatible (overlapped), So do not add ‘y’ in to solution

Activity ={p, s, w, }
Unit 3 Greedy Algorithms 16

Example cont…
z y x w v u t s r q p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ‘w’ and ‘z’ are compatible, So add ‘z’ in to solution

Activity ={p, s, w, z }
Unit 3 Greedy Algorithms 17

Example cont
z y x w v u t s r q p
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Activity ‘w’ and ‘z’ are compatible, So add ‘z’ in to solution

Activity ={p, s, w, z }
Unit 3 Greedy Algorithms 18

Activity Selection (Interval Scheduling)
B (1) C (2) A (3) E (4) D (5) F (6) G (7) 0 1 2 3 4 5 6 7 8 H (8) 9 10 11 Time

0
Unit 3

1

2

3

4

5

6

7

8

9

10

11
19

Greedy Algorithms

Activity Selection (Interval Scheduling)
B (1) C (2) A (3) E (4) D (5) F (6) G (7) 0 1 2 3 4 5 6 7 8 H (8) 9 10 11 Time

B 0
Unit 3

1

2

3

4

5

6

7

8

9

10

11
20

Greedy Algorithms

Activity Selection (Interval Scheduling)
B (1) C (2) A (3) E (4) D (5) F (6) G (7) 0 1 2 3 4 5 6 7 8 H (8) 9 10 11 Time

B 0
Unit 3

1

2

3

C 4

5

6

7

8

9

10

11
21

Greedy Algorithms

Activity Selection (Interval Scheduling)
B (1) C (2) A (3) E (4) D (5) F (6) G (7) 0 1 2 3 4 5 6 7 8 H (8) 9 10 11 Time

0
Unit 3

1

B A 2 3

4

5

6

7

8

9

10

11
22

Greedy Algorithms

Activity Selection (Interval Scheduling)
B (1) C (2) A (3) E (4) D (5) F (6) G (7) 0 1 2 3 4 5 6 7 8 H (8) 9 10 11 Time

B 0
Unit 3

E 3 4 5 6 7 8 9 10 11
23 Greedy Algorithms

1

2

Activity Selection (Interval Scheduling)
B (1) C (2) A (3) E (4) D (5) F (6) G (7) 0 1 2 3 4 5 6 7 8 H (8) 9 10 11 Time

B 0
Unit 3

D E 3 4 5 6 7 8 9 10 11
24 Greedy Algorithms

1

2

Activity Selection (Interval Scheduling)
B (1) C (2) A (3) E (4) D (5) F (6) G (7) 0 1 2 3 4 5 6 7 8 H (8) 9 10 11 Time

B 0
Unit 3

E 3 4 5 6
Greedy Algorithms

1

2

F 7

8

9

10

11
25

Activity Selection (Interval Scheduling)
B (1) C (2) A (3) E (4) D (5) F (6) G (7) 0 1 2 3 4 5 6 7 8 H (8) 9 10 11 Time

B 0
Unit 3

E 3 4 5 6 7
Greedy Algorithms

1

2

G 8

9

10

11
26

Activity Selection (Interval Scheduling)
B (1) C (2) A (3) E (4) D (5) F (6) G (7) 0 1 2 3 4 5 6 7 8 H (8) 9 10 11 Time

B 0
Unit 3

E 3 4 5 6 7 8 9
Greedy Algorithms

H 10 11
27

1

2

Elements of the Greedy Strategy

․ When to apply greedy algorithms?

Greedy-choice property: A global optimal solution can be arrived at by making a locally optimal (greedy) choice.
Dynamic programming needs to check the solutions to subproblems.

Optimal substructure: An optimal solution to the problem contains within its optimal solutions to subproblems.

․ Greedy algorithms (heuristics) do not always produce ․
optimal solutions. Greedy algorithms v.s. dynamic programming (DP)
  

E.g., if A is an optimal solution to S, then A' = A - {1} is an optimal solution to S' = {i ∈ S: si ≥ f1}.

Common: optimal substructure Difference: greedy-choice property DP can be used if greedy solutions are not optimal.

Unit 3

Greedy Algorithms

28

Problem 2: Fractional Knapsack Problem
There are n items in a store. For i =1,2, . . . , n, item i has weight wi > 0 and benefit bi > 0. Thief can carry a maximum weight of W pounds in a knapsack. In this version of a problem the items can be broken into smaller piece, so the thief may decide to carry only a fraction of object i.

Classification of Knapsack Problem:

Knapsack Problem

The fractional knapsack problem (Allow to take fraction of items.)
Unit 3 Greedy Algorithms

The 0-1 knapsack problem (Each item is either taken or not taken)
29

Fractional Knapsack Problem

cont..

Algo Greedy-fractional-knapsack (Item[n], w[ ], b[ ], W) { Knap=0 Weight = 0 Benefit=0 for each item i These line takes θ(n) v[i] = b[i] / w[i] // value per pound while (weight <=W) { i= Extract item of maximum value from list if(weight + w[i] ≤ W) { Knap=Knap U Item[i] weight = weight + w[i] benefit= benefit +v[i] These line takes θ(n) } else{ Knap=Knap U Item[i] weight = W benefit = (W - weight) * v[i] / w[i] // Factoring item, W – weight is needed weight } } return x }
Unit 3 Greedy Algorithms 30

Fractional Knapsack Problem

cont..

Algo Greedy-fractional-knapsack (Item[n], w[ ], b[ ], W) { ) Knap=0 (n Weight = 0 :O Benefit=0 ing t for each item i or s v[i] = b[i] / w[i] // value per pound g while (weight <=W) { din clu i= Extract item of maximum value from list ex if(weight + w[i] ≤ W) y xit { ple Knap=Knap U Item[i] om weight = weight + w[i] c e benefit= benefit +v[i] tim } So else{ Knap=Knap U Item[i] weight = W benefit = (W - weight) * v[i] / w[i] // Factoring item, W – weight is needed weight } } return Knap }
Unit 3 Greedy Algorithms 31

Example
Let S ={a, b, c, d, e, f, g} denote a set of objects with weights and benefits as given in the table below. What is an optimal solution to the fractional knapsack problem for S assuming that we have a sack that can Item objects B hold A C D E F G
Benefits (Rs.) 12 with total weight10 18? Weight (Kg.) 4 6 8 5 11 7 14 3 7 1 9 6

Carrying capacity W = 18 Kg.

Unit 3

Greedy Algorithms

32

Example cont… First we must calculate the “value" for the each items, which is defined as value = benefits/weight. I do this and Item A B C D give them in the table below: E F G
benefits Weight Value 12 4 3 10 6 1.67 8 5 1.6 11 7 1.57 14 3 4.67 7 1 7 9 6 1.5

value = benefit / weight

Unit 3

Greedy Algorithms

33

Example cont…
Item benefits Weight Value A 12 4 3 B 10 6 1.67 C 8 5 1.6 D 11 7 1.57 E 14 3 4.67 F 7 1 7 G 9 6 1.5

Sort this table according to decreasing value
Item benefits Weight Value F 7 1 7 E 14 3 4.67 A 12 4 3 B 10 6 1.67 C 8 5 1.6 D 11 7 1.57 G 9 6 1.5

Unit 3

Greedy Algorithms

34

Example cont…
Item benefits Weight Value F 7 1 7 E 14 3 4.67 A 12 4 3 B 10 6 1.67 C 8 5 1.6 D 11 7 1.57 G 9 6 1.5

W=18

Initially Knapsack weight = 0 benefits =0 W=18

Unit 3

Greedy Algorithms

35

Example cont…
Item benefits Weight Value F 7 1 7 E 14 3 4.67 A 12 4 3 B 10 6 1.67 C 8 5 1.6 D 11 7 1.57 G 9 6 1.5

W=18

Select maximum valued item ‘F’, Here (weight+w[F])<W Put whole item ‘F’ into knapsack. Add weight[F] with weight and benefit[F] with benefit.
Knapsack weight = 1 benefits =7
F

W=18

Unit 3

Greedy Algorithms

36

Example cont…
Item benefits Weight Value F 7 1 7 E 14 3 4.67 A 12 4 3 B 10 6 1.67 C 8 5 1.6 D 11 7 1.57 G 9 6 1.5

W=18

Now select next maximum valued item ‘E’, Here (weight+w[E])<W Put whole item ‘E’ into knapsack. Add weight[E] with weight and benefit[E] with benefit.

Knapsack weight = 1+3 =4 benefits =7+14 =21

F, E

W=18

Unit 3

Greedy Algorithms

37

Example cont…
Item benefits Weight Value F 7 1 7 E 14 3 4.67 A 12 4 3 B 10 6 1.67 C 8 5 1.6 D 11 7 1.57 G 9 6 1.5

W=18

Now select next maximum valued item ‘A’, Here (weight+w[A])<=W Put whole item ‘A’ into knapsack. Add weight[A] with weight and benefit[A] with benefit.

Knapsack weight = 1+3 =4+4=8

F, E, A

W=18

benefits =7+14 =21+12=33

Unit 3

Greedy Algorithms

38

Example cont…
Item benefits Weight Value F 7 1 7 E 14 3 4.67 A 12 4 3 B 10 6 1.67 C 8 5 1.6 D 11 7 1.57 G 9 6 1.5

W=18

Now select next maximum valued item ‘B’, Here (weight+w[B])<=W Put whole item ‘B’ into knapsack. Add weight[B] with weight and benefit[B] with benefit.

Knapsack

F, E, A, B

W=18

weight = 1+3 =4+4=8+6=14 benefits =7+14 =21+12=33+10=43

Unit 3

Greedy Algorithms

39

Example cont…
Item benefits Weight Value F 7 1 7 E 14 3 4.67 A 12 4 3 B 10 6 1.67 C 8 5 1.6 D 11 7 1.57 G 9 6 1.5

W=18

Now select next maximum valued item ‘C’, Here (weight+w[C])

⊀W Put whole item ‘B’ into knapsack. And calculate
weight and benefit as follows: needed weight = W – weight = 18 – 14 = 4 Put whole item ‘B’ into knapsack. Knapsack
F, E, A, B, C

W=18

weight = W = 18 benefits =7+14 =21+12=33+10=43+(needed weight)* value[C] =43+(4*1.6) = 43+6.4 = 49.4
Unit 3 Greedy Algorithms 40

Example cont…
Item benefits Weight Value F 7 1 7 E 14 3 4.67 A 12 4 3 B 10 6 1.67 C 8 5 1.6 D 11 7 1.57 G 9 6 1.5

W=18

Remaining items D, G could not put into Knapsack (bag) because bag is full i.e weight = W

Knapsack =

F, E, A, B, C

Weight in bag = W = 18 Benefits = Rs. 49.4

Unit 3

Greedy Algorithms

41

Complexity Analysis
If the items are already sorted into decreasing order of vi / wi, then  the while-loop takes a time in O(n); Therefore, the total time including the sort is in O(n log n). If we keep the items in heap with largest vi/wi at the root. Then creating the heap takes O(n) time while-loop now takes O(log n) time (since heap property must be restored after the removal of root)

Unit 3

Greedy Algorithms

42

Introduction
Huffman coding is an algorithm used for lossless data compression developed by David A. Huffman as a PhD student at MIT in 1952, and published in A Method for the Construction of MinimumRedundancy Codes. "Huffman Codes" are widely used applications that involve the compression and transmission of digital data, such as: fax machines, modems, computer networks, and high-definition television (HDTV), etc.

Professor David A. Huffman
(August 9, 1925 - October 7, 1999)
Unit 3 Greedy Algorithms 43

Motivation

 Reducing the space required to store files on disk or tape  Reducing the time to transmit large files.

 Huffman savings are between 20% - 90%

Unit 3

Greedy Algorithms

44

Basic Idea
Huffman Codes: Suppose we wish to save a text (ASCII) file on the disk or to transmit it though a network using an encoding scheme that minimizes the number of bits required. In fixed-size-encoding -scheme, without compression, characters are typically encoded by their ASCII codes with 8 bits per character. We can do better if we variable-size-encoding. In the variable-size-encoding-scheme we assign different code of different length according to their frequencies of occurrences.

Unit 3

Greedy Algorithms

45

Example:
Suppose you have a file with 100K characters. For simplicity assume that there are only 6 distinct characters in the file from a through f, with frequencies as indicated below. We represent the file using a unique binary string for each character.
a Frequency (in 1000s) Fixed-length codeword 45 000 b 13 001 c 12 010 d 16 011 e 9 100 f 5 101

Space =

(45*3 + 13*3 + 12*3 + 16*3 + 9*3 + 5*3) * 1000

= 300K bits
Unit 3 Greedy Algorithms 46

Can we do better ?? YES !!
By using variable-length codes instead of fixed-length codes. Idea : Giving frequent characters short codewords, and infrequent characters long codewords. i.e. The length of the encoded character is inversely proportional to that character's frequency.
a Frequency (in 1000s) Fixed-length codeword Variable-length codeword 45 b 13 c 12 d 16 e 9 f 5 Bit Required

000

001

010

011

100

101

300K bits 224K bits

0

101

100

111

1101

1100

Space = (45*1 + 13*3 + 12*3 + 16*3 + 9*4 + 5*4) * 1000=224K bits

Savings = 25%
Unit 3 Greedy Algorithms 47

PREFIX CODES

Codes in which no codeword is also a prefix of some other codeword.
("prefix-free codes" would have been a more appropriate name)
Variable-length codeword 0 101 100 111 1101 1100

It is very easy to encode and decode using prefix codes. No Ambiguity !! It is possible to show (although we won't do so here) that the optimal data compression achievable by a character code can always be achieved with a prefix code, so there is no loss of generality in restricting attention to prefix codes.
Unit 3 Greedy Algorithms 48

Constructing a Huffman code Huffman invented a greedy algorithm that constructs an optimal prefix code called a Huffman code. The algorithm builds the tree T corresponding to the optimal code in a bottom-up manner. It begins with a set of |C| leaves and performs a sequence of | C| - 1 "merging" operations to create the final tree.

Greedy Choice?

The two smallest nodes are chosen at each step, and this local decision results in a globally optimal encoding tree. In general, greedy algorithms use small-grained, or local minimal/maximal choices to result in a global minimum/maximum

Unit 3

Greedy Algorithms

49

Greedy Algorithm for Huffman Tree
Algo Greedy-Huffman(C,f) { n ← length[C] Q ← C:f for (i ← 1 to n-1) { z ← Allocate-Node x ← left[z] ← Extract-Min(Q) y ← right[z] ← Extract-Min(Q) f[z] ← f[x]+f[y] Insert(Q, z) } return Extract-Min(Q) }

Unit 3

Greedy Algorithms

50

Constructing a Huffman Coding Tree
14

From following Lines z ← Allocate-Node x ← left[z] ← Extract-Min(Q)

f:5

e:9

c:12

b:13

d:16

a:45

y ← right[z] ← Extract-Min(Q) f[z] ← f[x]+f[y]

c:12

b:13

14

d:16

a:45

From following Lines Insert(Q, z)

f:5

e:9

Unit 3

Greedy Algorithms

51

Constructing a Huffman Coding Tree
25

From following Lines z ← Allocate-Node x ← left[z] ← Extract-Min(Q) y ← right[z] ← Extract-Min(Q) f[z] ← f[x]+f[y]
f:5 e:9

c:12

b:13

14

d:16

a:45

From following Lines Insert(Q, z)
14

d:16

25

a:45

f:5

e:9

c:12

b:13

Unit 3

Greedy Algorithms

52

Constructing a Huffman Coding Tree
30

From following Lines z ← Allocate-Node
d:16 25 a:45

x ← left[z] ← Extract-Min(Q) y ← right[z] ← Extract-Min(Q) f[z] ← f[x]+f[y]

14

f:5

e:9

c:12

b:13

25

30

a:45

From following Lines Insert(Q, z)

c:12

b:13

14

d:16

f:5
Unit 3

e:9
Greedy Algorithms 53

Constructing a Huffman Coding Tree From following Lines
a:45 25 55

z ← Allocate-Node x ← left[z] ← Extract-Min(Q)
30

y ← right[z] ← Extract-Min(Q)
d:16

f[z] ← f[x]+f[y] Insert(Q, z)

c:12

b:13

14

f:5

e:9

Unit 3

Greedy Algorithms

54

Constructing a Huffman Coding Tree From following Lines
100 a:45 55

z ← Allocate-Node x ← left[z] ← Extract-Min(Q) y ← right[z] ← Extract-Min(Q) f[z] ← f[x]+f[y]
30

25

Insert(Q, z)
d:16

c:12

b:13

14

f:5

e:9

Unit 3

Greedy Algorithms

55

Adding binary code in this tree
0

100

1

a:45
0

55
1

25
0 1 0

30
1

Add ‘0’ with left child, and ‘1’with right child of each node

c:12

b:13
0

14
1

d:16

f:5

e:9

Unit 3

Greedy Algorithms

56

Finding binary code from this tree
0

100

1

Char Freq Codeword
55

a:45
0

a
1

45 13 12 16 9 5

0 101 100 111 1101 1100

b c
1

25
0 1 0

30

d e f

c:12

b:13
0

14
1

d:16

f:5

e:9

Unit 3

Greedy Algorithms

57

Running Time analys
The analysis of the running time of Huffman's algorithm assumes that Q is implemented as a binary min-heap. • For a set C of n characters, the initialization of Q in line 2 can be performed in O(n) time using the BUILD-MIN-HEAP procedure. • The for loop in lines 3-8 is executed exactly |n| - 1 times. Each heap operation requires time O(log n). The loop contributes = (|n| - 1) * O(log n) = O(nlog n) Thus, the total running time of HUFFMAN on a set of n characters = O(n) + O(nlog n) = O(n log n)
Unit 3 Greedy Algorithms 58

Thank You

Unit 3

Greedy Algorithms

59

Sign up to vote on this title