Greedy Algorithm Immediate Benefit Optimal Solution

✬ ✩
BS-MS & MS-PhD Maths & Computing: SMCS (IACS) 1
Greedy Algorithms
• A greedy algorithm approaches a solution by

taking decision that gives immediate benefit.
• For some problem it gives optimal solution.
• A popular example is the problem of finding
the minimal spanning tree of an weighted
undirected grapha .
a Even a bubble sort may be viewed as a greedy approach.
✫ ✪
Greedy Algorithms (7): COM 4201: Design and Analysis of Algorithms Goutam Biswas
✬ ✩
Minimal Spanning Tree
• For a connected, undirected graph

G = (V, E), a spanning tree is a connected,
acyclic subgraph of all the vertices,
T = (V, E ′ ).
• If the graph is weighted, G = (V, E, w),
different spanning trees may have different
sum of edge weights.
✫ ✪
✬ ✩
Minimal Spanning Tree
• A minimal spanning tree has the minimal

sum of edge weights.
• A minimal spanning tree need not be unique.
• A greedy algorithm at every step chooses an
edge of lowest weight that does not form a
cycle.
✫ ✪
✬ ✩
An Example
5 4
0 1 2
3
4 1 3 4
2
1 3
3 4 5
✫ ✪
✬ ✩
A Minimal Spanning Tree
w(1,3) = 1
4 w(3,4) = 1
0 1 2
w(1,4) = 2
1
3 w(1,5) = 3
w(0,4) = 3
3 w(4,5) = 3
4 5 w(0,3) = 4
3
1 w(1,2) = 4
w(2,5) = 4
Total weight: 12
w(0,1) = 5
✫ ✪
✬ ✩
Cut and Cut Property
Let G = (V, E, w) be an weighted undirected

graph.
• A cut is a partition of V in two groups of
vertices S and V \ S.
• Let X be a subset of edges of a minimal
spanning tree T of G.
✫ ✪
✬ ✩
• Let S ⊆ V be such that there is no edge

e = {u, v} ∈ X so that u ∈ S and v ∈ V \ S.
• Let e′ = {u′ , v ′ } be an edge with minimal
weight such that u′ ∈ S and v ′ ∈ V \ S.
• Then X ∪ {e′ } is part of some minimal
spanning tree.
✫ ✪
✬ ✩
• If e′ is an edge of T , then there is nothing to

prove. We assume that e′ is not a part of T .
• We add e′ to T . But then T is a spanning
tree, so it is connected and a cycle will be
formed by e′ across the cut.
✫ ✪
✬ ✩
T
e’
S V\S
✫ ✪
✬ ✩
• If we remove e we get the tree

T ′ = (T ∪ {e′ }) \ {e}.
• We claim that T ′ is also a minimal spanning
tree, where w(T ′ ) = w(T ) − w(e) + w(e′ ).
• As w(e′ ) ≤ w(e)a , we have w(T ′ ) ≤ w(T ).
• But then w(T ) is the minimal, and so is
w(T ′ ).
a e′
✫ ✪
is smallest across the cut
✬ ✩
A Greedy Scheme Based on Cut Property
mst (G = (V, E, w))

1 X←∅
2 do while|X| < |V | − 1:
Find S ⊆ V so that X has no
edge across S and V \ S. Pick
the minimum weight edge e between S and V \ S.
X ← X ∪ {e}
✫ ✪
✬ ✩
Kruskal’s Algorithm
• Start with vertices without any edge (forest

of vertices) and keep the edges sorted.
• Repeatedly add an edge of lowest weight
without forming a cycle.
• It is a greedy algorithm as the decision is
taken on immediate advantage just avoiding
a cycle.
✫ ✪
✬ ✩
The Example
5 4
0 1 2
3
4 1 3 4
2
1 3
3 4 5
✫ ✪
✬ ✩
Forest of Vertices and Sorted Edges
w(1,3) = 1
0 w(3,4) = 1
1 2
w(1,4) = 2
w(1,5) = 3
w(0,4) = 3
w(4,5) = 3
4 5 w(0,3) = 4
3
w(1,2) = 4
w(2,5) = 4
w(0,1) = 5
✫ ✪
✬ ✩
Final Minimal Spanning Tree
w(1,3) = 1
4 w(3,4) = 1
0 1 2
w(1,4) = 2
1
3 w(1,5) = 3
w(0,4) = 3
3 w(4,5) = 3
4 5 w(0,3) = 4
3
1 w(1,2) = 4
w(2,5) = 4
Total weight: 12
w(0,1) = 5
✫ ✪
✬ ✩
Kruskal’s Algorithm
kmst (G = (V, E, w))

1 for each u ∈ V :
2 makeset(u)
3 Et ← ∅
4 sort E by edge weight
5 ∀(u, v) ∈ E in non-decreasing order:
6 if u, v does not belong to the same set:
7 Et ← Et ∪ (u, v)
8 union(u, v):
✫ ✪
✬ ✩
Data Structure and Complexity
• The state of Kruskal’s algorithm is a set of

connected components where each one is a
tree.
• The next edge added connects two such
component trees.
• It is necessary to check whether the end
vertices of the selected edge (u, v) belongs to
two different components.
✫ ✪
✬ ✩
Joining Components
comp−I comp−II comp−III w(1,3) = 1

0 w(3,4) = 1
1 2
w(1,4) = 2
1
3 w(1,5) = 3
w(0,4) = 3
w(4,5) = 3
1
4 5 w(0,3) = 4
3
w(1,2) = 4
comp−IV w(2,5) = 4
w(0,1) = 5
✫ ✪
✬ ✩
• Once an edge is selected, two components

are merged to a single component.
• Disjoint components are modeled by disjoint
set data structure.
• Three essential operations on this data
structure are makeset(x), find(x) and
union(x, y).
✫ ✪
✬ ✩
• makeset(x) creates a singleton set of its

parameter.
• find(x) finds the set in which x belongs.
There is some representative of every set.
• union(x, y) makes the union of the sets of x
and y.
✫ ✪
✬ ✩
Complexity of Kruskal’s Algorithm
• There are |V | number of makeset(x)

operations.
• There are 2 × |E|, find(x) operationsa .
• And |V | − 1 union operationsb .
• The time complexity of the algorithm
depends on the cost of these operations.
a For each edge (u, v) we perform find(u) and find(v).
b It starts with n components and each union() reduces the number of com-
✫ ✪
ponent by 1.
✬ ✩
Disjoint Set Data Structure
• A disjoint sets are stored as a directed trees.

A tree node has an element of the set.
• There is no ordering, but each node has a
parent pointer. The path through the
pointer leads to the root of the tree.
• The parent pointer of the root points to
itself. The root is the representative of the
set.
✫ ✪
✬ ✩
Directed Trees
Two Disjoint Sets
3 2
5 4 7 0 6
{3,4,5}
{0,1,2,6,7}
1
✫ ✪
✬ ✩
Disjoint Set Data Structure
• Each node has a rank.

• The rank of a node is the hight of the
subtree hanging from it.
• For every node x other than the root,
rank(x) < rank(π(x)), where π(x) is the
parent of x.
✫ ✪
✬ ✩
Three Operations on Directed-Trees
makeset (x)
1 π(x) ← x // parent pointer points to itself
2 rank(x) ← 0
find(x)
1 while x 6= π(x):
2 x ← π(x) // root is the representative
3 return x
✫ ✪
✬ ✩
Three Operations on Directed-Trees
union (x, y)
1 rx ← find(x)
2 ry ← find(y)
3 if rx = ry : return // in the same set
4 if rank(rx ) > rank(ry ): // x in a deeper tree
5 π(ry ) = rx
6 else: // x in a lesser or equal height tree
7 π(rx ) = ry
8 if rank(rx ) = rank(ry ):
9 rank(ry ) ← rank(ry ) + 1.
✫ ✪
✬ ✩
Building Trees
Eight Disjoint Sets (all rank 0)
0 1 2 3 4 5 6 7
0 0 0 0 0 0 0 0
union(4,3), union(1,0), union(7,2)
3 0 2 5 6
1 1 0 0
1
4 1 7
0 0 0
✫ ✪
✬ ✩
Building Trees
union(3,4), union(0,1), union(2,7)
3 0 2 5 6
1 1 0 0
1
4 1 7
0 0 0
union(1,7), union(4,5)
2 3 6
2 1 0
0
1
7 5 4
0 0 0
✫ ✪
1
0
✬ ✩
Complexity of Operations
• makeset(x) is an O(1) operation.

• find(x) is an O(h) operation where h is the
height of the tree.
• union(x, y) is also O(h) operation. This
operation builds the tree and attempt should
be made to keep the height low.
✫ ✪
✬ ✩
Properties
• If the ranks of the root nodes of both x and y

are equal, then the rank of the root node for
y increase during the union(x, y) operation.
• Property I: If the rank of any node is k ≥ 0,
there are at least 2k nodes in the tree. This
can be proved by mathematical induction.
• Rank of an internal node does not change
from its last value as a root.
✫ ✪
✬ ✩
Properties
• If there are all total n elements, the number

of nodes of rank k cannot be more than n/2k .
• Therefore the height h of a tree is ≤ log n.
• Both find() and union() are of O(log n).
✫ ✪
✬ ✩
Time Complexity of Kruskal’s Algorithm
• Sorting of edges:
O(|E| log |E|) = O(|E| log |V |) as
log |E| = O(log |V |).
• Another O(|E| log |V |) for union() and find()
operations.
✫ ✪
✬ ✩
Modified find()
find(x)
1 if x 6= π(x):
2 π(x) ← find(π(x)), Make x and its
ancestors point to root
3 return π(x)
The benefit is not immediate, but in long
sequence of find(), the amortized cost is much
better than O(log n).
✫ ✪
✬ ✩
An Example
3 3
1 1
find(10) 10
0
0 2 1 3 2 5 0 2 1 3 2 5
0 4 1
6 1 7 8 0 0 4 1
6 1 7 8 0
0 9 0 10 11 0 0 11 0
9
✫ ✪
✬ ✩
An Example
0
3 11
3
1 1
10 find(11) 10
0 0
2
3 0 2 1 3 5
0 2 1 2 5
0 4 1
0 4 1
6 1 7 8 0 6 1 7 8 0
0 11 0 0 9
9
Rank of a node cannot be interpreted as the

height of the tree.
✫ ✪
✬ ✩
Prim’s Algorithm
• X, the set of edges, forms a subtree of a

minimal spanning tree at each intermediate
stage of computation.
• S is the set of vertices of this subtree.
• In each iteration a new edge, one with lowest
weight between S and V \ S, is added to X
and the subtree grows.
✫ ✪
✬ ✩
Prim’s Algorithm
• The computation structure of Prim’s

algorithm is very similar to the Dijkstra’s
shortest path algorithm.
• The difference is in the key of the priority
queue. It is the weight of the minimal weight
edge that connects a node v ∈ V \ S to a
node u ∈ S.
✫ ✪
✬ ✩
Prim’s Algorithm
primMST (G = (V, E, w), s)

1 for each u ∈ V :
2 cost[u] ← ∞;
3 previous[u] ← nil
4 cost(s) ← 0 // s is any initial node
5 initMinQ(Q, V ) # cost value as the key
✫ ✪
✬ ✩
An Example
5 4
0 1 2
3
4 1 3 4
2
1 3
3 4 5
Initial State (cost, previous)

Set S 0 1 2 3 4 5
∅ (0, •) (∞, •) (∞, •) (∞, •) (∞, •) (∞, •)
✫ ✪
✬ ✩
Prim’s Algorithm (cont.)
6 while not empty(Q)

7 u ←deleteMin(Q)
8 for all edges(u, v) ∈ E
9 if cost(v) > w(u, v)
10 cost(v) ← w(u, v);
11 previous(v) = u
✫ ✪
✬ ✩
(cost, previous)
Set S 0 1 2 3 4 5
{0} (5, 0) (∞, •) (4, 0) (3, 0) (∞, •)
{0, 4} (2, 4) (∞, •) (1, 4) (3, 4)
{0, 4, 3} (1, 3) (∞, •) (3, 4)
{0, 4, 3, 1} (4, 1) (3, 4)
{0, 4, 3, 1, 5} (4, 1)
✫ ✪
✬ ✩
Prim’s MST
4
0 1 2
3
1
1 3
3 4 5
✫ ✪
✬ ✩
Prim’s Algorithm
• The time complexity of the algorithm

depends on the implementation of minQ. We
assume that it is implemented on a binary
min-heap.
• Line 5: initMinQ takes O(|V |) time.
• The while-loop (6) is executed |V | times.
• Line 6-7: the total time deleteMin takes is
O(|V | log |V |) time.
✫ ✪
✬ ✩
Prim’s Algorithm
• 8-11: the total time the for-loop is executed

is |E| times.
• 10: the cost in the binary min-heap can be
updated in O(log |V |) time.
• The total time is O((|V | + |E|) log |V |).
✫ ✪
✬ ✩
Data Encoding
• There is a string of length 107 (approx)

letters over a four letter alphabet
Σ = {a, b, c, d}.
• This string is to be encoded in binary i.e.
each letter of Σ is encoded as a short binary
string.
• Two bits are sufficient to encode four letters:
a : 00, b : 01, c : 10, d : 11.
✫ ✪
✬ ✩
Data Encoding
• The length of encoded data is 2 × 107 bits.

• The question is can we shorten the length of
encoded data if there is some more
information about it?
• In the string the number of A are 68%, B are
2%, C are 16% and D are 14%.
✫ ✪
✬ ✩
Variable Length Encoding
• We may choose shorter code length for

symbols that are present in higher frequency.
• But the code should not create decoding
ambiguity.
• If we encode a : 0, b : 001, c : 11, d : 01, then
001 is ambiguous (b or ad).
✫ ✪
✬ ✩
Prefix-Free Huffman Encoding
• The code should be prefix-free.

• Code of one letter should not be a prefix of
the code of another letter.
• A prefix-free binary code can be represented
by a full binary tree.
✫ ✪
✬ ✩
A Prefix-Free Code
0 1
Codes
(32) a: 0
a (68) 0 1 b: 100
c: 11
(16) d: 101
0 1 c (16)
b (2) d (14)
✫ ✪
✬ ✩
Prefix-Free Huffman Encoding
• Elements of Σ are at the leaf nodes.

• Left-edge of a node is interpreted as zero (0)
and the right-edge is interpreted as one (1).
• A path from the root to the leaf represents
the code for the symbol at the leaf.
✫ ✪
✬ ✩
Decoding
• A string of bits is decoded by reading from it

left to right and traversing the tree from the
root down towards a leaf.
• Once a leaf is reached, the symbol of the leaf
is the output.
• The process starts again from the root.
✫ ✪
✬ ✩
Decoding
1000011101 ⇒ baacd.
1
Codes
0 a: 0
a (68) b: 100
c: 11
0 d: 101
c (16)
b (2)
d (14)
✫ ✪
✬ ✩
Size of Encoded Data
• a : 68 × 105 bits, b : 3 × 2 × 105 bits,

c : 2 × 16 × 105 bits and d : 3 × 14 × 105 bits.
• Total number of bits:
(68 + 6 + 32 + 42) × 105 = 148 × 105 bits.
• Improvement is (200 − 148)/2 ≡ 26%.
✫ ✪
✬ ✩
Cost of Coding Tree
• Let Σ = {a1 , · · · , an } be the input alphabet.

• Their frequencies are f1 , · · · , fn respectively.
• Let di be the depth of ai in the
encoding-decoding tree (root to leaf).
• di is also the length of code (path length) of
ai .
✫ ✪
✬ ✩
Cost of a Coding Tree
• The cost of a decoding-tree is

Xn
(fi × di )
i=1
• Every symbol of Σ is associated with a leaf

node along with its frequency.
• We can associate a frequency with the
internal nodes and can give a new definition
✫ ✪
of the tree cost.
✬ ✩
• The frequency of an internal node is the sum

of the frequencies of the leaf-nodes of its
subtree.
• The encoding emits a bit at each non-root
node on the path of the symbol from root to
leaf.
✫ ✪
✬ ✩
• The cost of a tree can also be written as sum of the

frequencies of all leave and internal nodes except the
root.
0 1
(32)
a (68) 0 1
(16)
0
c (16)
1
b (2)
d (14)
Cost-I = 68 × 1 + 2 × 3 + 14 × 3 + 16 × 2 = 148
Cost-II = 68 + 2 + 14 + 16 + 16 + 32 = 148
✫ ✪
✬ ✩
Greedy Algorithm
• Symbols with lowest frequencies will be at

the lowest level of the optimal tree.
• The tree can be constructed greedily.
• If fi and fj are two leaves of lowest
frequencies, construct a tree with them as
two children.
✫ ✪
✬ ✩
Greedy Algorithm
• The frequency of their root is fi + fj by the

second cost function.
• Treat the frequency of their root along with
other siblings of fi and fj and continue.
• Remove fi and fj from the list of
frequencies, insert fi + fj in the list.
• Continue with this smaller problem.
✫ ✪
✬ ✩
An Example
Frequencies: 5 8 14 18 25 30
(13)
0
1
(5) (8)
✫ ✪
✬ ✩
An Example
Frequencies: 13 14 18 25 30
(27)
0 1
(13) (14)
0
1
(5) (8)
✫ ✪
✬ ✩
An Example
Frequencies: 18 25 27 30
(43)
(27)
0 1 0
(18) (25) (13) (14)
0
1
(5) (8)
✫ ✪
✬ ✩
Boolean variables, Literals and Clauses
• Boolean variables: variables x, y, x1 , y1 , z, · · ·

taking values either true or false. They stand
for statements.
• A literals l is a variable x (positive literal) or
its negation ¬x or x̄ (“not x”, negative
literal).
• A clause is a disjunction (or,‘∨’) of literals
e.g. C = ¬x1 ∨ x2 ∨ ¬x3 ∨ x4 .
✫ ✪
✬ ✩
Horn Formula and CNF
• A Boolean formula is in conjunctive normal

form (CNF) if it is conjunction of clauses
e.g. C1 ∧ C2 ∧ C3 .
• A clause is called a Horn clause is it has at
most one positive literal.
• A Horn formula is a conjunction of Horn
clauses.
✫ ✪
✬ ✩
Types of Horn Clauses
• Negative clause (no positive literal):

(¬x ∨ ¬y ∨ ¬z).
• Implication: (¬x ∨ ¬y ∨ z) or (x ∧ y) ⇒ x.
• Unit clause: there are two special cases x
(⇒ x) and ¬x.
✫ ✪
✬ ✩
Boolean Satisfiability
• A Boolean formula is satisfiable if it is

evaluated to true by assigning truth values
to its variables. It is unsatisfiable if no such
valuation is possible.
• (x ∨ ¬y ∨ ¬z) ∧ (¬x ∨ y ∨ z) ∧ (¬x ∨ ¬y ∨ z)
is satisfiable with x = T, y = F, z = T .
• The formals (x ∨ y) ∧ (x ∨ ¬y) ∧ (¬x ∨ y) ∧
(¬x ∨ ¬y ∨ ¬z) ∧ (¬x ∨ z) is unsatisfiable
✫ ✪
✬ ✩
Boolean Satisfiability
• Boolean satisfiability is a decision problem

to check whether a formula is satisfiable.
• The collection of all satisfiable Boolean
formulas is called SAT.
✫ ✪
✬ ✩
P, NP, 2-SAT and 3-SAT
• 2-SAT is the collection of CNF formulas

where each clause has at most two literals.
• 3-SAT is the collection of CNF formulas
where each clause has at most three literals.
• 2-SAT is in P and 3-SAT is in NP. In fact it
is NP-complete.
✫ ✪
✬ ✩
Horn-SAT
• Horn-SAT is the collection of satisfiable

Horn formulas i.e. a collection of CNF
formulas where each clause is a Horn clause.
• Horn-Sat is in P and there is a greedy
algorithm for it.
✫ ✪
✬ ✩
Greedy Algorithm
HornSat()
Input: a Horn Formula
Output: satisfying truth assignment (if any)
1 Assign false to all variables
2 while there is an implication whose left-hand
side is true, but the right hand variable is false,
change the assignment of right side variable to true.
3 if all negative clauses are satisfied:
return the assignment
4 else: return ”unsatisfiable”.
✫ ✪
✬ ✩
Correctness Proof
• If the algorithm returns a satisfying

assignment, it satisfies the implications as
well as the negative clauses.
• The satisfying assignment of the implications
changes a variable assignment from false to
true only when it is essential, and for any
satisfying assignment, they must be so.
✫ ✪
✬ ✩
Greedy Algorithm
There is a linear time algorithm. But we stop here!
✫ ✪
✬ ✩
References
[DPV] Algorithms, by Sanjoy Dasgupta, Christos

Papadimitriou & Umesh Vazirani, ISBN
978-0-07-063661-3, Pub. Tata McGraw-Hill, 2008.
[KT] Algorithm Design by Jon Kleinberg & Éva Tardos,
ISBN 81-317-0310-X, Pub. Pearson Education, 2006.
[CLRS] Introduction to Algorithms by Thomas H.
Cormen, Charles E. Leiserson, Ronald L. Rivest &
Clifford Stein, ISBN 81-203-2141-3, Pub. PHI, 2001.
✫ ✪

Greedy Algorithm Immediate Benefit Optimal Solution

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Greedy Algorithm Immediate Benefit Optimal Solution

Uploaded by

Copyright:

Available Formats

✬ ✩

BS-MS & MS-PhD Maths & Computing: SMCS (IACS) 1

• A greedy algorithm approaches a solution by

Minimal Spanning Tree

• For a connected, undirected graph

Minimal Spanning Tree

• A minimal spanning tree has the minimal

A Minimal Spanning Tree

Cut and Cut Property

Let G = (V, E, w) be an weighted undirected

Cut and Cut Property

• Let S ⊆ V be such that there is no edge

Cut and Cut Property

• If e′ is an edge of T , then there is nothing to

Cut and Cut Property

Cut and Cut Property

• If we remove e we get the tree

A Greedy Scheme Based on Cut Property

mst (G = (V, E, w))

• Start with vertices without any edge (forest

Forest of Vertices and Sorted Edges

Final Minimal Spanning Tree

kmst (G = (V, E, w))

Data Structure and Complexity

• The state of Kruskal’s algorithm is a set of

comp−I comp−II comp−III w(1,3) = 1

Data Structure and Complexity

• Once an edge is selected, two components

Data Structure and Complexity

• makeset(x) creates a singleton set of its

Complexity of Kruskal’s Algorithm

• There are |V | number of makeset(x)

Disjoint Set Data Structure

• A disjoint sets are stored as a directed trees.

Two Disjoint Sets

Disjoint Set Data Structure

• Each node has a rank.

Three Operations on Directed-Trees

Three Operations on Directed-Trees

Eight Disjoint Sets (all rank 0)

union(4,3), union(1,0), union(7,2)

union(3,4), union(0,1), union(2,7)

• makeset(x) is an O(1) operation.

• If the ranks of the root nodes of both x and y

• If there are all total n elements, the number

Time Complexity of Kruskal’s Algorithm

Rank of a node cannot be interpreted as the

• X, the set of edges, forms a subtree of a

• The computation structure of Prim’s

primMST (G = (V, E, w), s)

Initial State (cost, previous)

Prim’s Algorithm (cont.)

6 while not empty(Q)

• The time complexity of the algorithm

• 8-11: the total time the for-loop is executed

• There is a string of length 107 (approx)

• The length of encoded data is 2 × 107 bits.

Variable Length Encoding

• We may choose shorter code length for

Prefix-Free Huffman Encoding

• The code should be prefix-free.

Prefix-Free Huffman Encoding