Design and Analysis

of Algorithms

D r D i v ya S i n d h u L e k h a
A s s i sta n t P ro fe s s o r

References: Online Course Materials, Analysis of algorithms, https://aofa.cs.princeton.edu/online/,

Introduction to Algorithms (Cormen et al.), Algorithm Design (Kleinberg and Tardos)

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2021 1

Fundamental Concepts for Algorithm
Analysis
❑ Asymptotic notations – for comparing growth of functions
O, Ω, θ, o, ω
❑ Average, Best and Worst Case Analysis
❑ Common functions – Time Complexity

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2021 2

Growth of common functions

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2021 3

Recurrences
An equation that recursively defines a sequence.
✓To model cost in program.
✓Telescoping a (linear first-order) recurrence

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2021 4

Types of recurrences

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2021 5

Types of recurrences

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2021 6

Types of recurrences

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2021 7

Types of recurrences

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2021 8

Types of recurrences

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2021 9

Divide-and-conquer recurrences
An effective technique in algorithm design.

https://www.enjoyalgorithms.com/blog/merge-sort-algorithm

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2021 10

Divide-and-conquer - Classic examples
Binary search

https://mikebuss.com/2016/04/21/binary-search/

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2021 11

Divide-and-conquer - Classic examples
Merge sort

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2021 12

Divide-and-conquer - Classic examples
Strassen matrix multiplication

https://www.raywenderlich.com/5740-swift-algorithm-club-strassen-s-algorithm

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2021 13

Divide and Conquer!

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 14

Divide-and-conquer
✓ Divide
the problem into a number of sub-problems that are smaller instances of
the same problem.
✓ Conquer
the sub-problems by solving them recursively.
✓ Combine
the solutions to the sub-problems into the solution for the original
problem.

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 15

Recurrence
➢ A recurrence is an equation or inequality
➢ Describes a function in terms of its value on smaller inputs.

➢ Solution – complexity

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 16

Methods - Solving Recurrence
1. Substitution Method
2. Recursion-Tree Method
3. Master Method

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 17

Sorting problem - Applications
Given a list L of n elements, rearrange them in ascending order.
1. Organize an MP3 library.
2. Display Google PageRank results.
3. List RSS news items in reverse chronological order.
Some problems become easier once elements are sorted.
1. Binary search in a database.
2. Remove duplicates in a mailing list.

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 18

Example: Merge Sort
➢ Divide:
Divide the n-element sequence to be sorted into two subsequences of n=2
elements each.
➢ Conquer:
Sort the two subsequences recursively using merge sort.
➢ Combine:
Merge the two sorted subsequences to produce the sorted answer.

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 19

Merge sort: Divide

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 20

Merge sort - Combine

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 21

Question
What is the auxiliary space complexity of merge sort?
Hint: Auxiliary space – Temporary space requirement.

A. O(1)
B. O(log n)
C. O(n)
D. O(n log n)

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 22

Explanation
An additional space of O(n) is required in order to merge two sorted arrays.
Merge sort is not an in-place sorting algorithm.

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 23

Merge sort - Algorithm

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 24

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 25
ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 26
Question
Merge sort is preferred for arrays over linked lists.
True or False?

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 27

Explanation
Merge sort is preferred for linked list over arrays.
In a linked list the insert operation takes only O(1) time.
Implement merge operation in constant time.

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 28

Analysis (To be continued…)

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 29

Analysis - Divide and Conquer Algorithms
➢ Algorithm contains a recursive call to itself.
Recurrence relation.
➢ Running time (problem of size n) in terms of the running time on smaller
inputs.
➢ Use mathematical tools to solve the recurrence.
➢ Obtain bounds on the performance of the algorithm
Solution of recurrence –> complexity of algorithm!

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 30

Analysis - Divide and Conquer Algorithms
Recurrence relation

Solution – complexity

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 31

Methods - Solving Recurrence
1. Substitution Method
2. Recursion-Tree Method
3. Master Method

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 32

Analysis: Merge sort
◦ Assumption: n is even. (for simplicity)

Divide: Computes middle of array. Constant time.

𝐷(𝑛) = 𝜃(1)
Conquer: Recursively solve 2 sub-problems of size 𝑛Τ2 .
𝐶 𝑛 = 2𝑇(𝑛Τ2)
Combine: Merge procedure on n-element subarray.
M(𝑛) = 𝜃(𝑛)?

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 33

1-3: Constant
8-11: Constant
4-7: for loops
𝜃 𝑛1 + 𝑛2 = 𝜃 𝑛
12-17: for loop
𝜃 𝑛

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 34

Analysis: Merge sort
◦ Assumption: n is even. (for simplicity)

Divide: Computes middle of array. Constant time.

𝐷(𝑛) = 𝜃(1)
Conquer: Recursively solve 2 sub-problems of size 𝑛Τ2 .
𝐶 𝑛 = 2𝑇(𝑛Τ2)
Combine: Merge procedure on n-element subarray.
M(𝑛) = 𝜃(𝑛)

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 35

Analysis: Merge sort

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 36

Solving recurrence
1. Substitution Method
2. Recursion-Tree Method
3. Master Method

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 37

Recursion tree
𝑇 𝑛 = 2𝑇 𝑛Τ2 + 𝜃 𝑛 = 2𝑇 𝑛Τ2 + 𝑐𝑛

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 38

Recursion tree 𝑇 𝑛 = 2𝑇 𝑛Τ2 + 𝜃 𝑛 = 2𝑇 𝑛Τ2 + 𝑐𝑛

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 39

Merge sort: Complexity

𝑇 𝑛 = 𝑐𝑛 log 𝑛 + 𝑐𝑛 = 𝜃(𝑛 log 𝑛)

(Ignoring low-order term and constant c.)

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 40

Design technique: Divide and Conquer

✓ Divide
✓ Conquer (recursively)
✓ Combine

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 41

Merge sort - Algorithm

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 42

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 43
ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 44
Analysis - Divide and Conquer Algorithms
Recurrence relation:
Running time (problem of size n) in terms of the running time on smaller
inputs.

Solution:

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 45

Methods - Solving Recurrence
1. Substitution Method
2. Recursion-Tree Method
3. Master Method

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 46

Substitution Method
Two steps
1. Guess the form of the solution.
2. Use mathematical induction to find the constants and show that the solution
works

We can use the substitution method to establish either upper or lower bounds
on a recurrence!

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 47

Substitution Method
Proof by induction

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 48

Substitution Method
𝑇 𝑛 = 2𝑇 (𝑛Τ2) + 𝑛

Step 1: Guess:
𝑇 𝑛 = 𝑂 (𝑛 log 𝑛 )

Remember the recursion tree of log n height!

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 49

Substitution Method
𝑇 𝑛 = 2𝑇 (𝑛Τ2) + 𝑛
𝐺𝑢𝑒𝑠𝑠: 𝑇 𝑛 = 𝑂 (𝑛 log 𝑛 )

Step 2: Prove that 𝑻 𝒏 ≤ 𝒄𝒏 𝐥𝐨𝐠 𝒏

𝑛
Assume that the bound holds for 𝑚 =
2
𝑛 𝑛 𝑛
𝑇 ≤𝑐 log( )
2 2 2
Substitute into recursion.

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 50

Substitution Method

Note: 𝑐 ≥ 1

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 51

Substitution Method
Base cases:
Assume 𝑇 1 = 1. 𝑢𝑛𝑖𝑡 𝑡𝑖𝑚𝑒
Then
𝑇 2 = 2𝑇 1 + 2 = 4
𝑇 3 = 2𝑇 1 + 3 = 5
𝑇 2 ≤ 𝑐2 log 2
𝑇 3 ≤ 𝑐3 log 3
For any choice of 𝑐 ≥ 2. (log base 2)

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 52

Substitution Method
➢No general method to make guesses.
➢Can use Recursion Tree to make guesses.
➢Similar solution for similar recurrences.
E.g. 𝑇 𝑛 = 2𝑇 (𝑛Τ2 + 17) + 𝑛
Another method:
•Prove loose lower bound (e.g. Ω(𝑛)) and loose upper bound (e.g. 𝑂(𝑛2 ))
•Gradually lower upper bound and increase lower bound until you reach a tight
bound.

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 53

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 54
ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 55
Methods - Solving Recurrence
1. Substitution Method
2. Recursion-Tree Method
3. Master Method

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 56

Substitution Method
Two steps
1. Guess the form of the solution.
2. Use mathematical induction to prove
Courtesy: https://www.askiitians.com

We can use the substitution method to establish either upper or lower bounds
on a recurrence!

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 57

Recursion Tree Method

Courtesy: https://www.codesdope.com

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 58

Recursion Tree
➢ Best used to generate a good guess.
➢ Verify by the substitution method.

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 59

Recursion Tree
Node
❑ For each recursive function invocation
❑ Represents the cost of a single sub-problem

Cost
❑ Per-level cost : Sum of costs within each levels
❑ Total cost: sum of all the per-level costs.

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 60

Recursion tree for good guess
❑ Focus on finding an upper bound

❑ Floors and ceilings not taken into account.

❑ Assume n as power of 4.

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 61

Recursion tree for good guess

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 62

Recursion tree for good guess

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 63

Recursion tree for good guess

Use an infinite decreasing geometric series as an upper bound.

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 64

Recursion tree for good guess

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 65

Recursion tree for good guess
Derived a guess.

For original recurrence:

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 66

Using substitution method to verify
If is an upper bound for
Show that

Holds true as long as 𝑑 ≥ 16Τ 𝑐

13

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 67

ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 68
ICS 211 DESIGN AND ANALYSIS OF ALGORITHMS, ODD SEMESTER, 2020 69