Algorithms Analysis & Design

Greedy Algorithms
Paradigm & Examples
 Announcements
Assignment#2: DESIGN#1
• Assess: D&C & Greedy design paradigms
• Location: @LMS under “Tests/Quizzes”
• Due: SUN 9 APR 23:55 isA [2 weeks]
• Submission: Upload “[PROBLEM_NAME].cs” file
• Evaluation: 12 (+) Marks
 Announcements
ta rt
S E LY
Assignment#2: DESIGN#1 D IAT
M E
IM
• TWO C# Template-based Problems
– GIVEN: reading inputs + checking outputs + sample test cases
– REQUIRED: fill the missing function that solve the problem
Sample Cases UNSEEN Large Cases
Problem Assess (Correctness) (Efficiency) Total
D&C
Problem#1 Taken Problem 2 Marks 6 Marks 8 MARKS
D&C or GREEDY
Problem#2 New Problem 1 Mark 3 Marks 4 MARKS
 Announcements
Assignment#2: DESIGN#1
• Problem#1: COMPETITION & BONUSES
Criteria BONUS

Just Faster +1 Mark

(on Large
Vs. Naïve
Cases)

1x Faster +3 Marks

[N]x Faster +[N]x2 Marks

TOP5 Correct & Speed 2~4 Marks

 GENERAL NOTE
Hidden Slides are Extra Knowledge
Not Included in Exams
 Roadmap

Algorithm Analysis & Design

Think…Design…Analyze…

Analysis Design

Statements, Recursion Brute- Divide & Dynamic Greedy Increm.

Conditions force Conquer Program. Method Improv.
& Loops (3 methods)
 Agenda
• Greedy Paradigm

• Ex.1: Fractional Knapsack

• Ex.2: Activity Selection

• Ex.3: Max Sequence Sum

• [EXTRA]
– Huffman Coding

• Summary & Questions

 Learning Outcomes

1. For Greedy strategy, identify a practical example to which it would apply.

2. Use a Greedy approach to solve an appropriate problem
3. Determine if the Greedy rule chosen leads to an optimal solution.
4. Determine an appropriate algorithmic approach to a problem.
 References
• Chapter 16
– Section 16.1 (Activity selection)
– Section 16.2 (Elements of Greedy + Fractional KS)
• MIT 6.046
– Lecture1 (Variations of activity selection)
Algorithm Design
Techniques: 4
 follow definition /
Brute Force & try all possibilities
Exhaustive
Search
break problem
Divide & into distinct
Iterative Conquer sub-problems
Improvement

repeatedly improve
current solution Algorithm
convert
Design Transformation problem to
another one
Techniques
Randomization

use Dynamic break problem

random Programming into overlapping
numbers sub-problems
repeatedly do
Greedy what is best now
11
 follow definition /
Brute Force & try all possibilities
Exhaustive
Search
break problem
Divide & into distinct
Iterative Conquer sub-problems
Improvement

repeatedly improve
current solution Algorithm
convert
Design Transformation problem to
another one
Techniques
Randomization

use Dynamic break problem

random Programming into overlapping
numbers sub-problems
repeatedly do
Greedy what is best now
12
 Greedy
• What?
 Repeatedly makes locally best choice, ignoring effect on future
 Greedy = ONE greedy choice + ONE subproblem
 ≈ D&C with ONE subproblem
 Divide = ONE greedy choice + ONE subproblem
 = one of the DP/Brute-force possible solutions (the best one)
1) Divide Main Problem ONE Greedy choice
D&C

2) Conquer Subproblem ONE Greedy choice

Subproblem ONE Greedy choice

3) Combine
Subproblem ONE Greedy choice
.
.
.
 Greedy
• Why?
– leads to efficient solutions (usually reduce the polynomial of DP)

• When to use?
– Often in optimization problems, in which
• There can be many possible solutions,

• Choosing one of them (Greedy) each time  final optimal solution.

 Greedy
• Conditions?
1. Optimal substructure:
• Optimal sol. to problem contains within it optimal sol. to ONE sub-problem
• i.e. Divide and Conquer
2. Greedy-choice property:
• Global optimal can be arrived at by making a local optimal (greedy) choice.
• Make the choice that looks best in the current problem, without considering results from sub-
problems.
Looks
Optimal Solution ONE Greedy choice
BEST

Optimal Sol. To DON’T Depend on

Subprob.

…
 Greedy
• Solution Steps:
1. Cast the problem:
• make a choice  one sub-problem to solve.

2. Prove greedy choice is always SAFE:

• there is always an optimal solution to the original problem that makes the greedy choice

3. Ensure optimal substructure:

• Final optimal = optimal solution to sub-problem + greedy choice

Optimal Solution ONE Greedy choice

=
+
Optimal Sol. To
Subprob.
 Greedy
• Compared with D&C:
1. Easy to propose multiple greedy algorithms for many problems
2. Easy run-time analysis
3. Hard to establish correctness

CAUTION: Most greedy algorithms are NOT correct

• Prove of correctness:
– Art more than science
1. Method1: Induction
2. Method2: Exchange Arguments
3. Method3: Whatever Works!
 Agenda
• Greedy Paradigm

• Ex.1: Fractional Knapsack

Idea
• Ex.2: Activity Selection Naïve Solution
Greedy Solution
• Ex.3: Max Sequence Sum Analysis
Trace Example
• [EXTRA]
– Huffman Coding

• Summary & Questions

 Before Start!
• Understand the problem VERY WELL

Slight change in problem statement

CAN YIELD TO
Different solution with different complexity

CAUTION: Most greedy algorithms are NOT correct

 Ex1: Fractional Knapsack
• Given
– There are n different items in a store
– Item i weighs wi pounds and is worth $vi
– A thief breaks in
– He can carry up to W pounds in his knapsack
– Fractions of items can be taken
• e.g. A thief stealing gold dust

• Required
– What should he take to maximize his benefit?

20
 Ex1: Fractional Knapsack

• Naïve Solution:
1. Find all possible sets of items
2. Chose the max one
– Complexity = O(2N)

21
 Ex1: Fractional Knapsack
• Greedy Solution:
1. Cast the problem: (make a choice  one sub-problem to solve)
1. Choice: start with ONE from the N items
2. Sub-problem: Find max benefit to pack the remaining weight from N – 1 items

WHICH ONE to Chose: Max-benefit to pack W

from N
Select ONE Item

 Max cost
Max-benefit to pack
 Min weight remain. W from N-1

 Max cost per unit

22
 Ex1: Fractional Knapsack
• Greedy Solution:
1. Cast the problem: (make a choice  one sub-problem to solve)
1. Choice: start with ONE from the N items
2. Sub-problem: Find max benefit to pack the remaining weight from N – 1 items

Select Item with

WHICH ONE to Chose: Max-benefit to pack W
from N max cost/unit
 Max cost
Max-benefit to pack
 Min weight remain. W from N-1

 Max cost per unit

23
 Ex1: Fractional Knapsack
• Greedy Solution:
1. Cast the problem: (make a choice  one sub-problem to solve)
1. Choice: start with MAX cost/unit from the N items
2. Sub-problem: Find max benefit to pack the remaining weight from N – 1 items

2. Prove greedy choice is always SAFE:

• Take as much as possible of the item with the greatest value per unit
• This fill each unit of the knapsack with max cost
• Which leads to maximize the sum of all units (i.e. max total cost)

3. Ensure optimal substructure:

• Final optimal = max benefit to pack remain. weight + item with max c/u
24
 Ex1: Fractional Knapsack
• High-level pseudo-code:
1. Calculate cost per unit for each item: ci / wi

2. Sort items by cost/unit (descending)

3. Repeatedly select the items until filling the knapsack
1. If wi ≤ W  take the entire item

2. Else  take fraction of it (= W x ci / wi)

Complexity = Θ(Nlog(N))
BONUS: Can we make it O(N)?
HINT: Think of selecting Kth element
25
 Ex1: Fractional Knapsack
• CAN greedy choice work with 0-1 knapsack?
• NO
– Unable to fill the knapsack to capacity  empty space lowers the effective value per
pound of the load
– We must compare two options:
• the solution to the sub-problem in which the item is included

• the solution to the sub-problem in which the item is excluded

before we can make the choice

 Dynamic Programming 26
 Ex1: Fractional Knapsack
• Ex:

0/1 Knapsack Fractional

Knapsack

27
 Agenda
• Greedy Paradigm

• Ex.1: Fractional Knapsack

Idea
• Ex.2: Activity Selection Naïve Solution
Greedy Solution
• Ex.3: Max Sequence Sum Analysis
Trace Example
• [EXTRA]
– Huffman Coding

• Summary & Questions

 Ex2: Activity-Selection

• In general: Scheduling a resource among several competing activities

– (e.g. activities on a lecture hall)

3
Max Set 4 6
2
1 5
• Given:
– A single resource
– A set S of N activities, each with:
si = start time of activity i
fi = finish time of activity i
• Required: 29
 Ex2: Activity-Selection

• Compatible Activity
– Activities i and j are compatible if [si, fi) and [sj, fj) don’t overlap

si  fj or sj  fi

• Naïve Solution:
1. Find all possible sets of compatible activities
2. Chose the max-size one
– Complexity = O(2N)

30
 Example
• Here are a set of start and finish times

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
1 6
2 7 11
10
3 9
5
4 8
• What is the maximum number of activities that can be completed?
• {a3, a9, a11} can be completed
• But so can {a1, a4, a8’ a11} which is a larger set
• But it is not unique, consider {a2, a4, a9’ a11} 31
 Ex2: Activity-Selection
• Greedy Solution:
1. Cast the problem: (make a choice  one sub-problem to solve)
1. Choice: start with ONE from the N activity
2. Sub-problem: Find max-size set of compatible activities from N – 1 activity

WHICH ONE to Chose: Max-size set from N Select ONE Activ.

 Min duration
 Earliest start time Max-size set from
N-1
 Last start time
 Earliest finish time
 Last finish time
 Has min number of overlaps
32
 Ex2: Activity-Selection
• Greedy Solution:
1. Cast the problem: (make a choice  one sub-problem to solve)
1. Choice: start with ONE from the N activity
2. Sub-problem: Find max-size set of compatible activities from N – 1 activity

Select Earliest
WHICH ONE to Chose: Max-size set from N Finish Activ.
 Min duration
 Earliest start time Max-size set from
N-1
 Last start time
 Earliest finish time
 Last finish time
 Has min number of overlaps
33
 Ex2: Activity-Selection
• Greedy Solution:
1. Cast the problem: (make a choice  one sub-problem to solve)
1. Choice: start with EARLIEST FINISH or LAST START from the N activity
2. Sub-problem: Find max-size set of compatible activities from N – 1 activity

2. Prove greedy choice is always SAFE:

• It leaves as much opportunity as possible for the remaining activities to be scheduled
• Hence, it maximizes the amount of unscheduled time remaining

3. Ensure optimal substructure:

• Final optimal = max-size set from (N – 1) + chosen activity with first finish

34
 Ex2: Activity-Selection
• High-level pseudo-code:
1. Sort activities by finish time (ascending)
2. Repeatedly select the compatible activities: Sk+1 ≥ Fk

Complexity = ??? O(Nlog(N))

35
Sorted by Finish
 Ex2: Activity-Selection
• VARIATION#1: MIT 6.046 Lecture1
– Given:
• a set of activities to schedule
• large number of lecture halls

– Required:
• Schedule all the activities using as few lecture halls as possible?

GREEDY or DP?!
Complexity? 37
 Ex2: Activity-Selection
• VARIATION#2: MIT 6.046 Lecture1
– Given:
• A set of activities to schedule, each with start si and finish fi

• Single resource

• A value (weight) for each activity vk

– Required:
• Choose a set of compatible activities such that is maximized?
GREEDY or DP?!
Complexity? 38
 Agenda
• Greedy Paradigm

• Ex.1: Fractional Knapsack

Idea
• Ex.2: Activity Selection Naïve Solution
Greedy Solution
• Ex.3: Max Sequence Sum Analysis
Trace Example
• [EXTRA]
– Huffman Coding

• Summary & Questions

 Ex3: Max Sequence Sum

• Given: array of N numbers

• Required: find a sequence of contiguous numbers with max sum

31 -41 59 26 -53 58 97 -93 -23 84

• BRUTE-FORCE • D&C
1. Find all possible sequences Divide: array into 2 halves
2. Chose one with max sum Conquer: each half to find its max seq.
• Complexity? Θ(N3) or Θ(N2) Combine: max of 2 halves & cross sum
• Complexity? Θ(N Log(N))
Can we do BETTER?
 Ex3: Max Sequence Sum
• Greedy Solution:
1. Cast the problem: (make a choice  one sub-problem to solve)
1. Choice: start with ONE from the N numbers
2. Sub-problem: Find max sequence sum from remaining numbers

WHICH ONE to Chose: Max Seq. Sum from N Select ONE number
 Min overall
 Max overall Max Seq. Sum from
remaining
 Min +ve
 Max +ve
 First +ve
 First +ve followed by +ve
41
 Ex3: Max Sequence Sum
• Greedy Solution:
1. Cast the problem: (make a choice  one sub-problem to solve)
1. Choice: start with ONE from the N numbers
2. Sub-problem: Find max sequence sum from remaining numbers

WHICH ONE to Chose: Max Seq. Sum from N Select ONE number
 Min overall
 Max overall Max Seq. Sum from
remaining
 Min +ve
 Max +ve
 First +ve
 First +ve followed by +ve
42
 Ex3: Max Sequence Sum
• Greedy Solution:
1. Cast the problem: (make a choice  one sub-problem to solve)
1. Choice: start with 1st +ve number and add it to max(0, current sum)
2. Sub-problem: Find max sequence sum from remaining numbers

2. Prove greedy choice is always SAFE:

• If current sum is +ve, then continue (as there's possibility to increase it by adding further num’s)
• if current sum is -ve, then reset the sum to 0 and restart from next +ve num
– as starting from 0 is better (i.e. give us greater sum) than starting from –ve num)

3. Ensure optimal substructure:

• Final optimal = max-sequence sum starting from non-negative over remaining numbers
43
 Ex3: Max Sequence Sum
• High-level pseudo-code:
1. Initialize maxSoFar to 0 maxSoFar = 0
curSum = 0
2. Initialize curSum to 0
for i = 1 to n
3. Repeatedly add next number to curSum {

1. If curSum still +ve, check it with maxSoFar, curSum += x[i]

then continue curSum = max(0, curSum)

maxSoFar = max(maxSoFar, curSum)
2. If becomes –ve, reset and start with 0
}

Complexity = ??? Θ(N)

44
 Agenda
• Greedy Paradigm

• Ex.1: Fractional Knapsack

Idea
• Ex.2: Activity Selection Naïve Solution
Greedy Solution
• Ex.3: Max Sequence Sum Analysis
Trace Example
• [EXTRA]
– Huffman Coding

• Summary & Questions

 Agenda
• Greedy Paradigm

• Ex.1: Fractional Knapsack

• Ex.2: Activity Selection

• Ex.3: Max Sequence Sum

• [EXTRA]
– Huffman Coding

• Summary & Questions

 SUMMARY

• Greedy = D&C with 1 subproblem = 1-choice + 1 subproblem

• Commonly used for optimization

• Safety checking is a MUST… Most choices are NOT SAFE!

• It significantly reduce complexity of some problems

1. Fractional KS: 2N or W×N N

2. Activity Selection: 2N  N Log(N) (can be N)

2
 Questions
• Greedy SHEET with Solutions
1. 7 Design Problems [with solutions]
2. 4 Design Problems [without solutions]
3. Other Questions (MCQ, T/F, Trace…)