Design and Analysis of Algorithms

Unit - I

Problem Solving Strategies & Algorithm

Analysis
Contents
• Algorithms
• Properties of Algorithms
• How to write an Algorithm
• Analyzing Algorithms (Space, Time, Input Size, Running Time)
• Problem Solving Principle
• Classification of Problem
• Problem Solving Strategies
• Classification of Time Complexities
• Order of Growth
• Asymptotic Notations
• Best Case, Worst case and Average case analysis
• Performance Analysis of basic Programming Constructs
• Recurrences – Solving Recurrence Equations
• Divide and Conquer
• Applications of Divide and Conquer
An algorithm is a sequence of unambiguous instructions for solving a
problem, i.e., for obtaining a required output for any legitimate input in
a finite amount of time.
Properties of Algorithm
• Input: should be specified, if not A will move to infinite state
• Output: At least one quantity is produced
• Non-ambiguity (Definiteness): Clear and Precise instruction, no
conflicts.
• Finiteness: should terminate after performing required operations
• Effectiveness: Every instruction must be very basic.
Performance Evaluation
• Performance evaluation is divided into two phases
• 1. Performance Analysis (A priori estimates)
• 2. Performance Measurement (Posteriori testing)
Performance Analysis
• If There are problems then algorithms is required to solve them.
• Problems and problem instances.
• Example: Sorting data in ascending order.
• Problem: Sorting
• Problem Instance: e.g. sorting data (2 3 9 5 6 8)
• Algorithms: Bubble sort, Merge sort, Quick sort, Selection sort, etc.
• Which is the best algorithm for the problem? How do we judge?

9
Performance Analysis (Priori estimates)
• Two criteria are used to judge algorithms: (i) time complexity (ii)
space complexity.
• Space Complexity of an algorithm is the amount of space needed to
solve an instance of a computational problem.
• Time Complexity of an algorithm is the amount of time it needs to
run an Algorithm.

10
Space Complexity
• Memory space S(P) needed by a program P, consists of two
components:
• A fixed part: needed for instruction space (byte code), simple variable space,
constants space etc.  c
• A variable part: dependent on a particular instance of input and output data.
 Sp(instance)
• S(P) = c + Sp(instance)

11
Space Complexity: Example 1
1. Algorithm abc (a, b, c)
2. {
3. return a+b+b*c+(a+b-c)/(a+b)+4.0;
4. }
For every instance 3 computer words required to store variables: a,
b, and c. Therefore Sp()= 3. S(P) = 3.

12
Space Complexity: Example 2
1. Algorithm Sum(a[], n)
2. {
3. s:= 0.0;
4. for i = 1 to n do
5. s := s + a[i];
6. return s;
7. }

13
Space Complexity: Example 2.
• Every instance needs to store array a[] & n.
• Space needed to store n = 1 word.
• Space needed to store a[ ] = n floating point words (or at
least n words)
• Space needed to store i and s = 2 words
• Sp(n) = (n + 3). Hence S(P) = (n + 3).

14
Time Complexity
• Time required T(P) to run a program P also consists of two
components:
• A fixed part: compile time which is independent of the problem instance  c.
• A variable part: run time which depends on the problem instance 
tp(instance)
• T(P) = c + tp(instance)
• Fixed part is usually ignored; only the variable part tp() is measured.

15
 i=1 i=1
For(i<=n)
For(i<=n)
{ i=i+2

{ i=i+1 }
}

AlgoSum(A[],m, n)

AlgoSum(A[], n) for i=1 to n do

{
{ s=0 for j=1 to n do

for i=1 to n do { s=s+a[i][j]

}
{ s=s+a[i] return s;
} }
return s;
}
Time Complexity: Example 1
Statements S/E Freq. Total

1 Algorithm Sum(a[],n) 0 - 0
2 { 0 - 0
3 S = 0.0; 1 1 1
4 for i=1 to n do 1 n+1 n+1
5 s = s+a[i]; 1 n n
6 return s; 1 1 1
7 } 0 - 0
2n+3 18
Time Complexity: Example 2
Statements S/E Freq. Total

1 Algorithm Sum(a[],n,m) 0 - 0

2 { 0 - 0

3 for i=1 to n do; 1 n+1 n+1

4 for j=1 to m do 1 n(m+1) n(m+1)

5 s = s+a[i][j]; 1 nm nm

6 return s; 1 1 1

7 } 0 - 0

2nm+2n+2 19
Orders of Growth
Classification of Time Complexities
Name of Efficiency Classes Order of Growth Examples

Constant 1 Scanning Array Examples

Logarithmic logn Performing Binary Search

Operation

Linear n Sequential Search Operation

nlogn nlogn Merge sort or quick sort

Quadratic n2 Scanning matrix elements

Cubic n3 Matrix multiplication

Exponential 2n Generating all subset of n elements

Factorial n! Generating all permutations

Performance Measurement (Posteriori
testing)
• Which is better?
• T(P1) = (n+1) or T(P2) = (n2 + 5).
• T(P1) = log (n2 + 1)/n! or T(P2) = nn(nlogn)/n2.
• Complex step count functions are difficult to compare.
• For comparing, ‘rate of growth’(Asymptotic notation) of time and
space complexity functions is easy and sufficient.

22
Asymptotic Notations

• Big Oh Notation : denotes upper bound.(Worst Case)

• Omega Notation : denotes lower bound (Best case)

• Theta Notation : average case

Big Oh (O)
• Let f(n) and g(n) be two non negative functions. Let and constant
c are two integers such that denotes some value of input and n>
and c>0.

Then we can write f(n) ≤c*g(n)

Example: f(n) = 3n+2 is O(n) because 3n+2 <= 4n for all n >= 2. c = 4, n0
= 2. Here g(n) = n.
Big O Notation

= n0

25
Big O Notation
• Example: f(n) = 10n2+4n+2 is O(n2) because 10n2+4n+2 <= 11n2 for all
n >=5.
• Example: f(n) = 6*2n+n2 is O(2n) because 6*2n+n2 <=7*2n for all n>=4.
• Algorithms can be: O(1)  constant; O(log n)  logrithmic; O(nlogn);
O(n) linear; O(n2)  quadratic; O(n3)  cubic; O(2n)  exponential.

26
Big O Notation
• Now it is easy to compare time or space complexities of algorithms.
Which algorithm complexity is better?
• T(P1) = O(n) or T(P2) = O(n2)
• T(P1) = O(1) or T(P2) = O(log n)
• T(P1) = O(2n) or T(P2) = O(n10)

27
Omega Notation

A function f(n) is said to be in ῼ(g(n)) if f(n) is bounded below by some

positive constant multiple of g(n) such that

f(n) ≥ c*g(n) for all n≥

ɵ Notation
• Let f(n) and g(n) be two non negative functions. There are two
positive constants c1 and c2 such that

c2g(n) ≤ f(n) ≤ c1g(n)

Then we can write

f(n) ɛ ɵ (g(n))
Problem Solving Principle
• Understand the Problem: Problem Stmt, Known, Unknown
• Make Decisions: Suitable DS, Algorithmic Technique
• Design or Specify the Problem: Flowcharts, Natural Languages
• Verify the Algorithm: Checking the correctness of Algorithm
• Analyze the Algorithm: Space and Time
• Implement: Suitable Programming Languages
• *Testing, Documentation, Maintenance and follow up
Classification of Problem
• Sorting: Ascending, Descending: numbers, Chars, Strings: Key & Unique
• Searching: Search for desired element. Search Key, Many algorithms
• Numerical Problems: Math Eqns, System of equations, definite integrals
• Geometric Problems: Points, Lines, Polygon
• Combinatorial Problems: Solvable, Unsolvable. No Algorithms-Finite
Time
• Graph Problems: Vertices and Edges: CN-Graph Traversals, Shortest Path
Problem Solving Strategies
• Brute Force: Straight Forward, Naïve Approach
• Divide and Conquer: Break a problem in to subProblem
• Dynamic Programming: The results of smaller, re-occuring instances
are obtained to solve the problem.
• Greedy Technique: locally optimal decisions are made
• Backtracking: Trial and error
• Branch & Bound: setting bound on live branches.
Divide-and-Conquer
The most-well known algorithm design strategy:
1. Divide instance of problem into two or more smaller instances

2. Solve smaller instances recursively

3. Obtain solution to original (larger) instance by combining these

solutions
Application of Divide and Conquer
• Binary Search
• Merge Sort
• Quick Sort
• Multiplication of Large Integers and Strassen’s Matrix Multiplication
• The Closest-Pair and Convex-Hull Problems by Divide-and-Conquer
• Max-Min Problem
Control Abstraction for D&C
• Algo D&C (P)
•{
• If P is small then return S(P);
• Else
•{
• Divide P in p1, p2,p3……pn>=1
• Apply D&C on (p1 p2,p3,….. pn)
• Return Combine (D&C(p1), D&C(p2), D&C(P3)…..)
•}
•}
Recurrence Equation for D& C

• T(n)= T(1)
aT(n/b)+f(n)
Examples of Recurrence relation
• Void Test (n)
•{
• if (n>0)
• {
• print(“%d”, n);
• Test(n-1)
• }
•}
• Void Test (n)
•{
• if(n>0)
• {
• for(i=0; i<n;i++)
• {
• printf(“%d”, i)
• }
• Test(n-1)
• }
•}
• Void Test (n)
•{
• if(n>0)
• {
• for(i=1; i<n;i*2)
• {
• printf(“%d”, i)
• }
• Test(n-1)
• }
•}
Mergesort
• Split array A[0..n-1] into about equal halves and make
copies of each half in arrays B and C
• Sort arrays B and C recursively
• Merge sorted arrays B and C into array A as follows:
• Repeat the following until no elements remain in one of the
arrays:
• compare the first elements in the remaining unprocessed portions
of the arrays
• copy the smaller of the two into A, while incrementing the index
indicating the unprocessed portion of that array
• Once all elements in one of the arrays are processed, copy
the remaining unprocessed elements from the other array
into A.
Pseudocode of Mergesort
Pseudocode of Merge

Time complexity: Θ(p+q) = Θ(n) comparisons

Mergesort Example
8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 71 5 4

8 3 2 9 7 1 5 4
The non-recursive
version of Mergesort
3 8 2 9 1 7 4 5 starts from merging
single elements into
2 3 8 9 1 4 5 7 sorted pairs.

1 2 3 4 5 7 8 9
Quicksort
• Select a pivot (partitioning element) – here, the first element
• Rearrange the list so that all the elements in the first s
positions are smaller than or equal to the pivot and all the
elements in the remaining n-s positions are larger than or
equal to the pivot

A[i]p A[i]p
• Exchange the pivot with the last element in the first (i.e., )
subarray — the pivot is now in its final position
• Sort the two subarrays recursively
Partitioning Algorithm

or i > r
< or j = l

Time complexity: Θ(r-l) comparisons

Quicksort Example
5 3 1 9 8 2 4 7
2 3 1 4 5 8 9 7
1 2 3 4 5 7 8 9
1 2 3 4 5 7 8 9
1 2 3 4 5 7 8 9
1 2 3 4 5 7 8 9
Analysis of Quicksort
• Best case: split in the middle — Θ(n log n)
• Worst case: sorted array! — Θ(n2) T(n) = T(n-1) + Θ(n)
• Average case: random arrays — Θ(n log n)

• Improvements:
• better pivot selection: median of three partitioning
• switch to insertion sort on small subfiles
• elimination of recursion
These combine to 20-25% improvement

• Considered the method of choice for internal sorting of

large files (n ≥ 10000)
Analysis of D&C
• Analysis can be done using recurrence equations
• Recurrence for D&C:
• T(n) = aT(n/b) + f (n)
• Method to solve recurrence equations are:
• 1. Substitution Method
• 2. Masters Theorem
Substitution Method
• Recurrence for Merge sort:
• T(n)= aT(n/b)+f(n)
A=2 b=2 f(n)=n
T(n)= 2T(n/2)+n
= 2[2T(n/4)+n/2]+n
= 4T(n/4)+2n
= 4[2T(n/8)+n/4]+2n
= 8T(n/8)+3n
= …
=…
= 2iT(n/2i) +in //2i=n
= nT(n/n) + log2n log2i = logn
= nT(1)+ nlog2n i= log2n//
= 0+ nlog2n
= nlog2n
Max-Min Problem
• Find the Maximum And Minimum Value from the given array
2. Matrix multiplication

The problem:
 
Multiply two matrices A and B, each of size n  n

     
 A   B   C 
     
  nn   nn   nn

56
 The traditional way:
 
n
C ij   Aik  Bkj
k 1

use three for-loop

3
 T ( n)  O ( n )

57
The Divide-and-Conquer way:
C11 C12   A11 A12   B11 B12 
 C  A  B 
     
C 21 C 22   A21 A22   B21 B22 

C11  A11  B11  A12  B21

C12  A11  B12  A12  B22
C 21  A21  B11  A22  B21
C 22  A21  B12  A22  B22
transform the problem of multiplying A and B, each of size [n×n]
into 8 subproblems, each of size  n  n 
2 2

59
 n
 T (n)  8  T ( )  an 2
2
 O(n 3 )
which n2 is for addition

so, it is no improvement compared with the traditional

way

62
Example: 1 1 1 1 1 1
1 1 1  1 1 1
   
1 1 1 1 1 1
use Divide-and-Conquer way to solve it as following:
1 1 1 0  1 1 1 0  3 3 3 0
1 1 1 0 1 1 1 0 3 3 3 0
  
1 1 1 0  1 1 1 0  3 3 3 0
     
0 0 0 0  0 0 0 0 0 0 0 0

1 1 1 1 1 0 1 1 3 3
C11          
1 1 1 1 1 0 0 0 3 3
1 1 1 0 1 0 1 0 3 0
C12      
1 1 1 0 1 0 0 0 3 0
1 1 1 1 1 0 1 1 3 3
C 21     
0 0 1 1 0 0 0 0 0 0
1 1 1 0 1 0 1 0 3 0
C 22           
0 0 1 0 0 0 0 0 0 0

63
Strassen’s matrix multiplication:
·Discover a way to compute the Cij’s using 7 multiplications
and 18 additions or subtractions
P  ( A11  A22 )( B11  B22 )
Q  ( A21  A22 ) B11
R  A11 ( B12  B22 )
S  A22 ( B21  B11 )
T  ( A11  A12 ) B22
U  ( A21  A11 )( B11  B12 )
V  ( A12  A22 )( B21  B22 )
C11  P  S  T  V
C12  R  T
C 21  Q  S
C 22  P  R  Q  U
64
•Algorithm :

procedure Strassen (n, A, B, C) // n is size, A,B the input

matrices, C output matrix
begin
if n = 2,

C11  a11  b11  a12  b21;

C12  a11  b12  a12  b22 ;
C21  a21  b11  a22  b21;
C22  a21  b12  a22  b22 ;
else
(cont.)

65
else
Partition A into 4 submatrices: A11 , A12 , A; 21 , A22
Partition B into 4 submatrices: B11 , B12 , B;21 , B22
n
call Strassen ( 2
;
, A11  A22 , B11  B22 , P)

n
call Strassen ( , A21  A22 , B11 , Q);
2
n
call Strassen ( , A11 , B12 ; B22 , R )
2
n
call Strassen ( , A22 , B21 ; B11 , S )
2
n
call Strassen ( , A11  A12;, B22 , T )
2

66
 n
call Strassen ( , A21  A11 , B11 ; B12 ,U )
2
n
call Strassen ( , A12  A22 , B21 ; B22 ,V )
2
C11  P  S  T  V ;
C12  R  T ;
C 21  Q  S ;
C 22  P  R  Q  U ;
end;

Young
CS 331 D&A of Algo. Topic: Divide and Conquer 67
 Analysis:
 n
7  T ( )  an 2 n2
T ( n)   2
b n2

n
T (n)  7  T ( )  an 2
2
n 7
 7  T ( 2 )  ( )an 2  an 2
2
2 4
n 7 2 7
 7  T ( 3 )  ( )  an  ( )  an 2  an 2
3 2
2 4 4

Young
CS 331 D&A of Algo. Topic: Divide and Conquer 69
Assume n = 2k for some integer k
k 1 n 2  7 k 2 
 7  T ( k 1 )  an  ( )    1
2  4 
 7 k 1 
 ( )  1
 7 k 1  b  an 2  4 
 7
1 
 4 
7
 b  7k  c  n2  ( )k
4
7
7 Lg
 b  7 Lgn  cn 2  ( ) Lgn  b  7 Lgn  cn 2 (n) 4
4
 b  n Lg 7  cn Lg 7  (b  c)  n Lg 7
 O(n Lg 7 )  O (n 2.81 )

70
Masters Theorem
• T(n)=aT(n/b)+nklogpn
• Condition: a>=1; b>1; k>=0; p is a real no
• Case1: if a>bk then T(n)=(nlog a) b

• Case2: if a= bk
i) if p>-1 then T(n)= (nlogba logp+1)
ii) if p=-1 then T(n)= (nlogba loglogn)
iii) if p<-1 then T(n)= (nlogba )
• Case3: if a<bk
i) if p>=0 then T(n)= (nk logpn)
ii) if p<0 then T(n)= (nk)