Design & Analysis of Algorithms

M. Tech.(Software Eng.),
BITS-WIPRO Collaborative Program

Yogeshri Gaidhani
 Instructor: Yogeshri Gaidhani

yogeshrigaidhani@gmail.com
(M) 9422774507
Textbooks

Introduction to Algorithms Introduction to the

(2nd edition) Theory of Computation
by Cormen, Leiserson, Rivest (2nd edition)
and Stein (CLRS) by Michael Sipser
Some other good books:
1. Design of Algorithms
By Goodrich, Tamassia
2. Fundamentals of Algorithms
By Horowitz, Sahani
3. Art of Computing
By Knuth
Course goals
 Useful algorithms
 Some sorting algorithms
 Implementing some important data structures

 General techniques
 Dynamic programming
 Greedy Method
 Linear programming
 Randomization

 Analysis methodology
 Correctness
 Complexity
 Limitations of computation
What we want to do…
 Understanding term Algorithm

 “good” Algorithm- Measure of “good” or

efficient algorithm
 Learning some fundamental strategies to
write algorithm / Designing Algorithm
 Role of Data Structures in Implementing
Algorithms
What we want to do
 Introduce new models of
Computation such as combinatorial
circuits or parallel computers
 Discuss some known limitations to
design of efficient algorithms
 Introduce methods for coping with
those limitations
Algorithms

 Computational procedure which

accepts input(s) and produces some
value(s), output
 Correctness of Algorithm – for all
problem instances
 Correct algorithm : An algorithm is
said to be correct if it halts with
correct output, for every input
instance.
 We say that a correct algorithm
solves the given computational
problem.
Algorithms

 Efficiency of Algorithm – Speed / Time Complexity

 Efficiency of Algorithm – Memory / Space
Complexity
Performance Analysis – Time
Complexity
 Experimental Study – Calculate actual time
required for execution on various test inputs
 Limitations of Experimental Studies:

1. Experiments can be done only on limited set of

i/p. Care must be taken to make sure that these are
representatives.
2. Comparison possible only when software and
hardware platforms are same.
3. Implementation and execution of algorithm is
must.
Need of Analytical framework that -
 Takes into account all possible inputs.
 Allows us to compare efficiency of
two algorithms, independent of
software and hardware environments.
 Can be performed by studying high
level description of algorithm without
implementing it
 Components in Analytical
Methodology
 A language for describing algorithm
 Computational model that algorithms
execute within
 A metric for measuring algorithm
running time
 An Approach for characterizing
running time, including those for
recursive algorithms.
Metric for measuring time –
Counting primitive Steps
 Primitive operation – a syntactically or
semantically meaningful statement of a
program that has execution time that is
independent of instance characteristics.
 Primitive operation corresponds to a low
level operation with execution time that
depends on hardware / software
environment, but is constant.
 Count number of primitive operations
instead of actual run time.
 Implicit assumption – running time of all
primitive operations is fairly same.
Counting primitive operations:
 Comments count zero.
 Assignment count for 1 step.
 Evaluation of expression 1 step.
 Loops – each execution in control part
counts 1.
 return statement counts for 1 step.
Insertion Sort
InsertionSort(A) // No of Steps
{ for(j:=2 to length(A) // n
{key:=A[j]; //1 * (n-1)
i:=j-1; //1 * (n-1)
while(i>0 and A[i]>key) // n-1
{ A[i+1]:=A[i]; //<(n-1)*(n-1)
i:=i-1; //<(n-1)*(n-1)
} //
A[i+1]:=key; //n-1
} //Total <=K*n2
}
Recall : Orders of Growth
 Big-Oh Notation : We say that f(n)
is big Oh of g(n) and write
f(n)=O(g(n)) iff there exists a real
number c>0 and an integer n0>=1
such that, f(n)<=c*g(n) n>=n0.
Ex: f(n) = 4n +6 <= 5n n>=6
f(n) = O(n)
If P(n) is a polynomial of degree k
then P(n) = O(nk)
Recall : Orders of Growth
 Big-Omega Notation : We say that
f(n) is big Omega of g(n) and write
f(n)=Ω(g(n)) iff there exists a real
number c>0 and an integer n0>=1
such that, f(n)>=c*g(n) n>=n0.
Ex: f(n) = 4n +6 <= 5n n>=6
f(n) = Ω(n)
If P(n) is a polynomial of degree k
then P(n) = Ω(nk)
Recall: Orders of Growth
 Big-Theta Notation : We say that f(n) is
big Theta of g(n) and write f(n)=Θ(g(n)) iff
there exists a real number c1, c2>0 and an
integer n0>=1 such that,
c1*g(n)>=f(n)>=c2*g(n) n>=n0.
Ex: f(n) = 4n +6 <= 5n and f(n)>= 4n
n>=6
f(n) = Θ(n)
If P(n) is a polynomial of degree k then
P(n) = Θ(nk)
Orders of Growth:
O(lg n)< O(n) <O(n2)< … <O(exp)
Average Case, Best Case, and
Worst Case Analysis:
90
Worst Case
80
70
60
50
40 Running Time
30
20
Best Case
10
0
A B C D
Review: Asymptotic Performance

 Asymptotic performance: How does algorithm behave

as the problem size gets very large?
 Running time
 Memory/storage requirements
 Remember that we use the RAM model:
 All memory equally expensive to access
 No concurrent operations
 All reasonable instructions take unit time
 Except, of course, function calls
 Constant word size
 Unless we are explicitly manipulating bits
 Bubble Sort Example
9, 6, 2, 12, 11, 9, 3, 7
6, 9, 2, 12, 11, 9, 3, 7
Bubblesort compares the numbers in pairs from left to right
exchanging when necessary. Here the first number is compared
to the second and as it is larger they are exchanged.

6, 2, 9, 12, 11, 9, 3, 7
Now the next pair of numbers are compared. Again the 9 is the
larger and so this pair is also exchanged.

6, 2, 9, 12, 11, 9, 3, 7
In the third comparison, the 9 is not larger than the 12 so no
exchange is made. We move on to compare the next pair without
any change to the list.

6, 2, 9, 11, 12, 9, 3, 7
The 12 is larger than the 11 so they are exchanged.

6, 2, 9, 11, 9, 12, 3, 7
The twelve is greater than the 9 so they are exchanged
The end of the list has been reached so this is the end of the first pass. The

6, 2, 9, 11, 9, 3, 12, 7
twelve at the end of the list must be largest number in the list and so is now in
The 12
the correct is greater
position. Wethan
now the 3 so
start theypass
a new are exchanged.
from left to right.

6, 2, 9, 11, 9, 3, 7, 12
The 12 is greater than the 7 so they are exchanged.
 Bubble Sort Example
First Pass
6, 2, 9, 11, 9, 3, 7, 12
Second Pass

6, 6,
2, 2, 9, 9,
11,3,
11,
9,7,
11,
3,11,
7, 12
Notice that this time we do not have to compare the last two
numbers as we know the 12 is in position. This pass therefore only
requires 6 comparisons.
 Bubble Sort Example
First Pass
6, 2, 9, 11, 9, 3, 7, 12
Second Pass

Third Pass
2, 6, 9, 9, 3, 7, 11, 12
2, 6, 9, 3,
9, 9,
7, 9,
3, 7, 11, 12
This time the 11 and 12 are in position. This pass therefore only
requires 5 comparisons.
 Bubble Sort Example
First Pass
6, 2, 9, 11, 9, 3, 7, 12
Second Pass

Third Pass
2, 6, 9, 9, 3, 7, 11, 12
Fourth Pass
2, 6, 9, 3, 7, 9, 11, 12
2, 6, 3,
9, 9,
7, 9,
3, 7, 9, 11, 12
Each pass requires fewer comparisons. This time only 4 are needed.
 Bubble Sort Example
First Pass
6, 2, 9, 11, 9, 3, 7, 12
Second Pass

Third Pass
2, 6, 9, 9, 3, 7, 11, 12
Fourth Pass
2, 6, 9, 3, 7, 9, 11, 12
Fifth Pass
2, 6, 3, 7, 9, 9, 11, 12
2, 6,
3, 3,
6, 7, 9, 9, 11, 12
The list is now sorted but the algorithm does not know this until it
completes a pass with no exchanges.
 Bubble Sort Example
First Pass
6, 2, 9, 11, 9, 3, 7, 12
Second Pass

Third Pass
2, 6, 9, 9, 3, 7, 11, 12
Fourth Pass
2, 6, 9, 3, 7, 9, 11, 12
2, 6, 3, 7, 9, 9, 11, 12
Fifth PassThis pass no exchanges are made so the algorithm knows the list is
sorted. It can therefore save time by not doing the final pass. With

Sixth Pass
2, 3, 6, 7, 9, 9,
other lists this check could save much more work.
11, 12
2, 3, 6, 7, 9, 9, 11, 12
Bubble Sort Questions Time
1. Which number is definitely in its correct position at the end of
the first pass?
Answer: The last number must be the largest.
2. How does the number of comparisons required change as the
pass number increases?
Answer: Each pass requires one fewer comparison than the last.
3. How does the algorithm know when the list is sorted?

Answer: When a pass with no exchanges occurs.

4. What is the maximum number of comparisons required for a
list of 10 numbers?

Answer: 9 comparisons, then 8, 7, 6, 5, 4, 3, 2, 1 so total 45

Exercise: Bubble Sort

 Write a function
bsort1(int A[], int n)
and apply the bubble sort algorithm to sort
elements in array A in ascending order.
 You may test your function with the
following main program.
int main()
{
    int a[] = { 1, 3, 5, 7, 2, 4, 6 };
    int size = sizeof(a) / sizeof(a[0]);
    bsort1(a, size);    print_array(a, size);   
    return 0;
}
 bsort2
 Write a function
bsort2(int A[], int n)
and apply the bubble sort algorithm to
sort elements in array A in descending
order.
 You may test your function with the following
main program.
int main()
{
    int a[] = { 1, 3, 5, 7, 2, 4, 6 };
    int size = sizeof(a) / sizeof(a[0]);
    bsort1(a, size);    print_array(a, size);
bsort2(a, size);    print_array(a, size);    
    return 0;
}
 bsort3
 Write a function
bsort3(int A[], int n)
which will sort the array A so that even
numbers will be in front of odd numbers.
 For example,
 { 1, 3, 5, 7, 2, 4, 6 } → {2, 4, 6, 1, 3, 5,
7}
 { 7, 3, 5, 1, 6, 4, 2} → {6, 4, 2, 7, 3, 5, 1 }
 bsort4
 Write a function
bsort4(int A[], int n)
which will sort the array A so that odd
numbers will be in front of even
numbers.
 For example,
 {2, 4, 6, 1, 3, 5, 7 } → { 1, 3, 5, 7, 2, 4,
6}
 {6, 4, 2, 1, 3, 5, 7 } →{ 1, 3, 5, 7, 6, 4, 2}
 bsort5
 Write a function
bsort5(int A[], int n)
which will sort the array A so that even
numbers will be in front of odd numbers.
 Moreover, the sequences of odd numbers
and even numbers are also independently
sorted in ascending order.
 For example,
 { 1, 3, 5, 7, 6, 4, 2} → {2, 4, 6, 1, 3, 5, 7 }
Analysis
 Simplifications
 Ignore actual and abstract statement
costs
 Order of growth is the interesting
measure:
 Highest-order term is what counts
 Remember, we are doing asymptotic analysis
 As the input size grows larger it is the high
order term that dominates
Upper Bound Notation
 We say BubbleSort’s run time is O(n2)
 Properly we should say run time is in O(n2)
 Read O as “Big-O” (you’ll also hear it as
“order”)
 In general a function
 f(n) is O(g(n)) if there exist positive constants
c and n0 such that f(n)  c  g(n) for all n  n0
 Formally
 O(g(n)) = { f(n):  positive constants c and n0
such that f(n)  c  g(n)  n  n0
Bubble Sort Is O(n2)
 Proof
 Suppose runtime is an2 + bn + c
 If any of a, b, and c are less than 0 replace the
constant with its absolute value
 an2 + bn + c  (a + b + c)n2 + (a + b + c)n +
(a + b + c)
  3(a + b + c)n2 for n  1
 Let c’ = 3(a + b + c) and let n0 = 1
 Question
 Is BubbleSort O(n3)?
 Is BubbleSort O(n)?
Big O Fact
 A polynomial of degree k is O(nk)
 Proof:
 Suppose f(n) = bknk + bk-1nk-1 + … + b1n
+ b0
 Let ai = | bi |
i k-1
 f(n)  akknk + ank-1n + …k + a1n + a0
 n a i
n k
 n a
i  cn k
Lower Bound Notation
 We say Bubble Sort’s run time is (n)
 In general a function
 f(n) is (g(n)) if  positive constants c and
n0 such that 0  cg(n)  f(n)  n  n0
 Proof:
 Suppose run time is an + b
 Assume a and b are positive (what if b is
negative?)
 an  an + b
Asymptotic Tight Bound
 A function f(n) is (g(n)) if  positive
constants c1, c2, and n0 such that

c1 g(n)  f(n)  c2 g(n)  n  n0

 Theorem
 f(n) is (g(n)) iff f(n) is both O(g(n)) and
(g(n))
 Proof: someday
Practical Complexity
250

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Practical Complexity
500

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Practical Complexity
1000

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 3 5 7 9 11 13 15 17 19
Practical Complexity
5000

4000
f(n) = n
f(n) = log(n)
3000
f(n) = n log(n)
f(n) = n^2
2000 f(n) = n^3
f(n) = 2^n

1000

0
1 3 5 7 9 11 13 15 17 19
Practical Complexity
10000000

1000000

100000

10000

1000

100

10

1
1 4 16 64 256 1024 4096 16384 65536
Why is it necessary for you to
study algorithms?

Algorithms are at the core of

technologies used in computers.
Real Goal of This Course

This course will teach you techniques

of algorithm design and analysis so
that you can develop algorithms on
your own, show that they give correct
answer, and understand their
efficiency.