Analysis of Algorithm

Chapter One

Introduction And Elementary Data Structures

1
 Outlines
• Introduction to Algorithm analysis
• Asymptotic Notations
• Analysis of Algorithm
• Review of elementary Data Structures
 Heaps
 Hashing
 Set Representation UNION, FIND Operation

2
 What is Algorithm
 An algorithm is a sequence of unambiguous instructions for solving a problem. I.e., for

obtaining a required output for any legitimate input in a finite amount of time.

• To read a specification of input

• To solve a given problem

• To produce a specification output according to a given input

 Each instruction can be carried out in a finite amount of time.

Input Algorithm Output

 An Algorithm tool for solving a well-specified computational problem.

 An Algorithm can be implemented using different programming languages 3on

 CONT…
 A computational problem specifies an input-output relationship
• What does the input look like?
• What should the output be for each input?
 Algorithms must be:
 Correct: For each input produce an appropriate output
 Algorithm is correct if, for every input instance, it ends with the correct
output. We say that a correct algorithm solves the given computational
problem.
 An incorrect algorithm might not end at all on some input instances, or
it might end with an answer other than the desired one.
4
 We shall be concerned only with correct algorithms.
 CONT…
 Efficient: run as quickly as possible, and use as little
memory as possible
Same problem can have many ways (algorithms) to solve it

 Which one is more efficient?

Simply Efficiency considers:

•How long will it take?

•How much storage will it need?

5
 Methods for Developing an Algorithm
1. Define the problem
 State the problem you are trying to solve in clear and concise terms.

2. List the inputs and the outputs

 Information needed to solve the problem.

 What the algorithm will produce as a result.

3. Describe the steps

 Needed to convert or manipulate the inputs to produce the outputs.

 Start at a high level first, and keep refining the steps until they are
effectively computable operations
4. Test the Algorithm
6
 Choose data sets and verify that your algorithm works!
 CONT…
 There are two commonly used tools to help to design program logic
(algorithm).
These are flowcharts and Pseudo-code
 Flowcharts work well for small problems but Pseudo-code is
used for larger problems.
• Pseudo-code (or Program Design Language)
 Consists of natural language-like statements that precisely
describe the steps of an algorithm or program.
 Focuses on the logic of the algorithm or program.
 Avoids language-specific elements. 7
 CONT…
• Flowcharts:
 A graphical tool that diagrammatically depicts the steps and
structure of an algorithm or program.

8
 Order Notation or Order of Growth
 An algorithm’s proportional time requirement is known as growth rate.
• E.g. number of elements in a list for a sorting algorithm,
•Algorithm A requires 5*n2 time units to solve a problem of size n.
•Algorithm B requires 7*n time units to solve a problem of size n.
• The most important thing to learn is how quickly the algorithm’s time
requirement grows as a function of the problem size.
 For the large enough inputs, the multiplicative constants and lower-order
terms are dominated by the effects of the input size itself. The growth rate
is not affected by
 constant factors or lower-order terms
• Algorithm A requires time proportional to n2.
• Algorithm B requires time proportional to n.
We measure an algorithm’s time requirement as a function of the
problem size n.

9
 Asymptotic Notation
• Used to describe the asymptotic running time of an algorithm are defined
in terms of functions whose domains are the set of natural numbers
N={1,2,3,…..n}
• Such notations are convenient for describing the worst-case running-time
function T(n), which usually is defined only on integer input sizes.
• Let f be a nonnegative function. Then we define the three most common
asymptotic bounds as follows
1. Big-Oh O : Upper Bounds
 O(g(n)): class of functions f(n) that grow no faster than g(n)

2. Big-Omega Ω : Lower Bounds

 Ω(g(n)): class of functions f(n) that grow at least as fast as g(n)
10
 CONT…
3. Theta Θ(g(n)) : Tight Bounds:
 Θ(g(n)): class of functions f(n) that grow at same rate as g(n)

•Big-Oh O : Upper Bounds

 O(g(n)): class of functions f(n) that grow no faster than g(n)

 Big-Oh notation mostly used to describe Worst-Case.

• Definition of Big-Oh O

 We say that f (n) is Big-O of g(n), written as f (n) = O(g(n)), iff

there are positive constants c and n0 such that

0 < f (n) ≤ c g(n) for all n ≥ n0

 If f (n) = O(g(n)), we say that g(n) is an upper bound on f (n).11
 CONT…
Example:
F(n) = 4n2 + 16n +2 and g(n) = n4
F(n) is O(g(n)) = ?
n 4n2 + 16n +2 n4 Result
0 2 0 F
1 22 1 F
2 50 16 F
3 86 81 F
4 130 256 T
Yes f(n) is O(g(n))if C=1 and No=4

t i c
m a
ra m
iag
D y
ll 12
 CONT…
Big-Omega Ω : Lower Bounds
 Ω(g(n)): class of functions f(n) that grow at least as fast as g(n)
 Big-Omega notation mostly used to describe Best-Case

Definition of Big-Omega Ω
 We say that f (n) is Big-Omega of g(n), written as f (n) = Ω(g(n)),
iff there are positive constants c and n0 such that
0 < c g(n) ≤ f (n) for all n ≥ n0
 If f (n) = Ω(g(n)), we say that g(n) is a lower bound on f (n).

13
 CONT…
• Example:
• F(n) = 4n2 + 16n +2 and g(n)=n2
• F(n) is Ω(g(n))
n =?4n + 16n +2 n Condition
2 2

0 2 0 T

Yes f(n) is Ω (g(n)) When C =1 and N0 =0

t i c
m a
r a m
iag
D y
a ll 14
 CONT….
Theta Θ(g(n)) : Tight Bounds:
 Θ(g(n)): class of functions f(n) that grow at same rate as g(n)

 Theta notation mostly used to describe Average-Case

Definition of Theta Θ(g(n

 Both f(n) ∈ O(g(n)) and f(n) ∈ Ω(g(n)).

 We say that f (n) is Big-Theta of g(n), written as f (n) = Θ(g(n)), iff

there are positive constants c1 , c2 and n0 such that

0 < c1 g(n) ≤ f (n) ≤ c2 g(n) for all n ≥ n0

Equivalently, f (n) = Θ(g(n)) if and only if f (n) = O(g(n)) and f (n) =

Ω(g(n)). If f (n) = Θ(g(n)), we say that g(n) is a tight bound on f (n).
15
 CONT…
• Example:
Yes f(n) is Θ(g(n))
• F(n) = 4n + 16n +2 and g(n) = 4n
2 2
When C =5 and N0 =2
• F(n) is Θ (g(n)) = ?
n 4n2 + 16n +2 C.4n2 Condition
0 2 0 F
1 22 20 F
2 50 80 T

t i c
m a
ra m
a g
Di y
a ll 16
 Property of growth rate function
1. f(n)  O(f(n))
2. f(n)  O(g(n)) iff g(n) (f(n))

3. If f(n)  O(g (n)) and g(n)  O(h(n)) , then f(n)  O(h(n)) Note
similarity with a ≤ b

4. If f1(n)  O(g1(n)) and f2(n)  O(g2(n)) , then f1(n) + f2(n) 

O(max{g1(n), g2(n)})
5. We can ignore low-order terms in an algorithm’s growth-rate function.
• If an algorithm is O(n3+4n2+3n), it is also O(n3).
• We only use the higher-order term as algorithm’s growth-rate
function.
17
 CONT…
6. We can ignore a multiplicative constant in the higher order term of
an algorithm’s growth-rate function.
• If an algorithm is O(5n3), it is also O(n3).
7. O(f(n)) + O(g(n)) = O(f(n)+g(n))
• We can combine growth rate functions.
• If an algorithm is O(n3) + O(4n), it is also O(n3 +4n)  So, it
is O(n3).
• Similar rules hold for multiplication.

18
Comparison of growth rate functions

19
 Analysis of Algorithm
 Analysis deals with performance evaluation (complexity analysis).
 The analysis of an algorithm provides background information that gives us a
general idea of how long an algorithm will take for a given problem set.

 For analyzing the efficiency of algorithms the two kinds are time
efficiency and space efficiency.
 Time efficiency indicates how fast an algorithm in question runs;

 Space efficiency deals with the extra space the algorithm requires.

 The space requirement is not of much concern, because now we

have the fast main memory, cache memory etc. so we concentrate
more on time efficiency.
20
 Analysis of Algorithm
 Studying the analysis of algorithms gives us the tools to choose
between algorithms.
• Arithmetic Operators:

 We will count arithmetic operators in two groups

 Additive and Multiplicative

 Additive operators (Additions) include addition, subtraction, increment, and

decrement.
 Multiplicative operators (Multiplications) include multiplication, division,
and modulus.
 These two groups are counted separately because multiplications are
considered to take longer than additions
 Logarithms and Geometric functions are another group even more21 time
 Memory space efficiency
 Memory space efficiency: is a function describing the amount of
memory (space) an algorithm takes in terms of the amount of input to
the algorithm.
 We often speak of "extra" memory needed, not counting the memory
needed to store the input itself. Again,
 we use natural (but fixed-length) units to measure this. We can use
bytes, but it's easier to use, say, number of integers used, number of
fixed-sized structures, etc.
 Space complexity is sometimes ignored because the space used is
minimal and/or obvious, but sometimes it becomes as important an
22
 Run Time complexity
 Running time efficiency: is a function describing the amount of time an
algorithm takes in terms of the amount of input to the algorithm.
 "Time" can mean
• The number of memory accesses performed,

• The number of comparisons between integers,

• The number of times some inner loop is executed,

• Or some other natural unit related to the amount of real time the
algorithm will take.
• Since many factors unrelated to the algorithm itself can affect the real
time (like the language used, type of computing hardware, proficiency of
23
the programmer, optimization in the compiler, etc.).
 EXAMPLES
Algorithm A Algorithm B
//Sum of n No’s //Sum of n No’s
//Input n //Input n
//Output Sum of (1+2+… //Output Sum of (1+2+…+n)
+n) 1. Declare total=0, n, i
1. Declare total, n 2. Read n
2. Read n 3. For (i <- 1 to n)
3. total=n(n+1)/2 4. total = total +I
4. Display total 5. Display n

Algorithm A
1. Memory space:- takes 2 memory space for total and n
2. Running time :- takes 4 sec or micro second O(1)

Algorithm B
1. Memory space:- takes 3 memory space for total, n and i
2. Running time :- takes 2n + 3 sec or micro second O(n) 24
Example of time and space complexity
Algorithm A() Algorithm B()
1. {
1. {
2. int i, j, total, n;
n 3. Read n; n
2. int i, total,n;
3. read n;
n 4. For (i = 1 to n) n
4. For (i = 1 to n)
5. { n2
5. { 6. For (j = 1 to n)
6. total
T(n) = total +i
= O(n)
7. } 7. {
8. } 8. total = total 2+I
9.
T(n)
}
= O(n )
10. }
Algorithm A Algorithm
11. } B
1. Memory space:- takes 3 =O(1) 1. Memory space:- takes 4= O(1)
memory space for i, total, n memory space for i,j, total, n
2. Running time :- takes O(n) 2. Running time :- takes O(n2)
25
Example of time and space complexity
Algorithm C()

1. { i 1 3 5 7 9 … n/2

2. int i, total, n;
3. read n; n/2 = n
O(n/2)
4. For (i = 1;i<=n; i=i+2)
T(n) = O(n)
5. {
6. total = total +i
Algorithm
7. }
C
1.
8. Memory
} space:- takes 3 memory space for i, total, n
2. Running time :- takes O(n)

26
Example of time and space complexity
Algorithm D()
i 1 2 4 8 16 .. N
1. {
2 20 21 22 23 24 … 2k
2. int i, total, n;
3. read n; 2k = n
4. For (i = 1;i<=n; i=i*2) k log2 n
O(Log2 n)
5. {
6. total =Dtotal + i
Algorithm
1.7. }
Memory
8. }
space:- takes 3 memory space for i, total, n
2. Running time :- takes O(log2n)

27
Example of time and space complexity
Algorithm G()
1. {
2. Int i, j,k,n;
3. Read n
4. For (i=n/2;i<=n; i++) n n*log2n*log2n
5. {
n(log2n)2
6. For (j=1;j<=n;j*2)
7. { log2n O(n(log2n)2)
8. For (k=1;k<=n;k*2) log2n
9. { T(n) = O(n(log2n)2
10. Print(“Hello class”)
11. }
12. }
Algorithm G()
1. Memory space:- takes 4 memory space for i, j, n
28
2. Running time :- takes O(n(log n) 2)
 Types of Analysis of Algorithm
 There are three types of Analysis
• An algorithm can require different times to solve different problems of the same
size input n.
 Worst-Case Analysis
 The maximum amount of time that an algorithm requires to solve a problem of
size n.
• Normally, we try to find worst-case behavior of an algorithm.
 Best-Case Analysis
 The minimum amount of time that an algorithm require to solve a problem of
size n.
• The best case behavior of an algorithm is NOT so useful.
 Average-Case Analysis
 The average amount of time that an algorithm require to solve a problem of size
n.
• We use average case when worst case and best case are the same.
• Worst-case analysis is more common than average-case analysis.
29
 CONT…
 There are three types of Analysis
 Worst-Case Analysis –
2 7 1 0 5 4 9 3 8

 Find number 8 in the array -> worst case

 Best-Case Analysis

8 2 7 1 0 5 4 9 3
 Find number 8 in the array -> Best case
 Average-Case Analysis
3 2 7 1 8 0 5 4 9

 Find number 8 in the array -> Average case

30
 Heap sort
Heap sort is one of the sorting mechanisms which sort a given data (elements), in
ascending or descending order.
 The (binary) heap data structure is an array object that we can view as a
complete binary tree or almost (nearly) complete binary tree.

 Input: A sequence of n numbers (a1, a2, a3, …………., an)

 Output: A permutation (reordering) (a1’, a2’, a3’, …., an’) of the input sequence

such that a1’≤ a2’ ≤ a3’ ≤ …. ≤ an’

 Heap sort mechanism has three stages to sort a given data.

1. max- heapify/ min-heapify

2. Build-max-heap / build-min-heap
3. Heap-sort
31
 CONT…
 Heap is represented by Complete Binary tree, or Almost
Complete Binary tree .
 What is a Complete binary tree :- a tree w/h has Equal
number of left and right children

a) Level 1 CBT b)
Level 2 CBT

32
 How to Implement heap sort
 An array A that represents a heap is an object with two
attributes:
1. A.length, which (as usual) gives the number of
elements in the array, and
2. A.heap-size, which represents how many elements in
the heap are stored within array A, w/h indicate
indexes.
Or how many element follow the heap property.
 There are two kinds of property:
1. Max-heap property and
2. Min-heap property.

33
 Max-heap
• MAX-HEAP means that the root of a tree should be
Maximum from the left and right sub tree.
30
• Example
10 20

All leaves are already a max-heap.

The largest element in a max-heap is stored at the

root, and the sub tree rooted at a node contains
values no larger than the main root of the tree. 34
 Min heap
• MIN-HEAP means that the root of a tree should be
Minimum from the left and right sub tree or children.
10
• Example:
30 20

All leaves are already a Min-Heap

the smallest element in a min-heap is stored at the

root, and the sub tree rooted at a node contains
values no smaller than the main root of the tree. 35
Implement Max-heap property
 Root = First element , Left child(i) = [2 * i],
Right child(i) = 2 * i + 1], Parent(i) = [i/2]
4
1 1
2 3 2
4 16
7 9
8 10
9 10
14 14
8 16
7
1 2 3 4 5 6 7 8 9 10
1

4
2 3

1 3
4 5 6 7

2 16 9 10
8 9 10

14 8 77
1. MAX-HEAPIFY/ MIN-HEAPIFY
Implement MAX- Heap Property
It could be more than one max - heap
If the array is on Descending order already
satisfy Max - Heap property

If the array is on Ascending order already

satisfy Min- Heap property

37
 Maintaining the heap property
MAX-HEAPIFY(A , i)
1 l = LEFT (2*i)
2 r = RIGHT(2*i+1)
3 if l ≤ A.heap-size and A[l] > A[i]
4 largest = l
else
5 largest = i
6 if r ≤ A.heap-size and A[r] > A[largest]
7 largest = r
8 if largest ≠ i
9 exchange A[i] with A[largest]
10 MAX-HEAPIFY (A.largest)
 Algorithm of the heap property
heap-sort(A)
1 build-max-heap(A)
2 for i = A.length downto 2
3 exchange A[1] with A[i] exchange
4 A.heap-size = A.heap-size - 1
5. MAX-HEAPIFY(A,1)
The HEAP-SORT procedure takes time O(n logn),
 since the call to Build-Max-Heap takes time O(n)
and each of the n - 1 calls to MAX-HEAPIFY
takes time O(log n).
T(n) =O(Log n)
39
 Run time of Max- HEAPIFY
 What is the relationship b/n height and Max No of Node.
h 1 2 3 4 5 6 .. .. n
Max No 3 7 15 31 63 127 .. .. ?

n= 2h+1 - 1
2h+1 – 1 Log2 n = n
Log2 3 = 1
Log2 7 = 2
Log2 15 = 3
Log2 31 = 4
T(n) = O(Log n)
 Hash Table
• It is one of the most important data structures that uses a special function

known as a hash function that maps a given value with a key to access the

elements faster.

• A Hash table is a data structure that stores some information, and the

information has basically two main components, i.e., key and value.

• The hash table can be implemented with the help of an associative array.

• A Hash function assigns each value with a unique key.

• Sometimes hash table uses an imperfect hash function that causes a collision

because the hash function generates the same key of two different values.

41
 Hashing
• Hashing is one of the searching techniques that uses a constant

time. The time complexity in hashing is O(1).

• In hashing technique, the hash table and hash function are used.

Using the hash function, we can calculate the address at which the

value can be stored.

• The main idea behind the hashing is to create the (key/value) pairs.

• If the key is given, then the algorithm computes the index at which

the value would be stored. It can be written as:

Index = hash(key) 42
 Cont.…
• There are three ways of calculating the hash function:
 Division method
 Folding method
 Mid square method

• In the division method, the hash function can be defined as: map a
key k into one of the m slots by taking the remainder of k divided
by m. That is:
h(k) = k mod m
where m is the size of the hash table.
• Example: m = 31 and k = 78  h(k) = 16. 43
 Collision
• Multiple keys can hash to the same slot – collisions are possible.
• The following are the collision techniques:
• Open Hashing: It is also known as closed addressing.
 Chaining (separate chaining)

• Closed Hashing: It is also known as open addressing.

 Linear probing
 Quadratic probing
 Double hashing

44
 Cont.…
• Open Hashing
 In Open Hashing, one of the methods used to resolve the collision is known as a
chaining method.

• Example: Suppose we have a list of key values

• A = 3, 2, 9, 6, 11, 13, 7, 12 where m = 10, and h(k) = 2k+3

• In this case, we cannot directly use h(k) = ki/m as h(k) = 2k+3

• The index of key value 3 is:

45
 Cont.…
The value 3 would be stored at the index 9.
The index of key value 2 is:
index = h(2) = (2(2)+3)%10 = 7
The value 2 would be stored at the index 7.
The index of key value 9 is:
index = h(9) = (2(9)+3)%10 = 1
The value 9 would be stored at the index 1.
The index of key value 6 is:
index = h(6) = (2(6)+3)%10 = 5
The value 6 would be stored at the index 5.
The index of key value 11 is:
index = h(11) = (2(11)+3)%10 = 5
The value 11 would be stored at the index 5.
Now, we have two values (6, 11) stored at the
same index, i.e., 5
This leads to the collision problem, so we will use the
chaining method to avoid the collision. We will create
one more list and add the value 11 to this list. After the
creation of the new list, the newly created list will be
linked to the list having value 6.
The index of key value 13 is:
index = h(13) = (2(13)+3)%10 = 9
The value 13 would be stored at index 9. Now, we have
two values (3, 13) stored at the same index, i.e., 9. This
leads to the collision problem, so we will use the
chaining method to avoid the collision. We will create
one more list and add the value 13 to this list. After the
creation of the new list, the newly created list will be
linked to the list having value 3.
The index of key value 7 is:
46
index = h(7) = (2(7)+3)%10 = 7
 Cont.…
after doing all steps finally you got the order as shown below

index
0
1 9
2
3
4
5 6 11
6
7 2 7 12
8
9 3 13

47
 Cont.…
• Closed Hashing:

 Linear Probing: search sequentially for an unoccupied position.

• Example: Consider the above example for the linear probing:

• A = 3, 2, 9, 6, 11, 13, 7, 12 where m = 10, and h(k) = 2k+3

• The key values 3, 2, 9, 6 are stored at the indexes 9, 7, 1, 5 respectively. The calculated index
value of 11 is 5 which is already occupied by another key value, i.e., 6. When linear probing is
applied, the nearest empty cell to the index 5 is 6; therefore, the value 11 will be added at the
index 6.

• The next key value is 13. The index value associated with this key value is 9 when hash function
is applied. The cell is already filled at index 9. When linear probing is applied, the nearest empty
cell to the index 9 is 0; therefore, the value 13 will be added at the index 0.

• The next key value is 7. The index value associated with the key value is 7 when hash function is
applied. The
0 cell1is already
2 filled
3 at4index57. When
6 linear
7 probing
8 is
9 applied, the nearest empty
cell to the 13
index 9
7 is 8; 12
therefore, the value 67 will 11 2 at the
be added 7 index
3 8. 48
 Cont.…
• In case of linear probing, searching is performed linearly.
• In contrast, quadratic probing is an open addressing technique that
uses quadratic polynomial for searching until a empty slot is found.
• It can also be defined as that it allows the insertion ki at first free
location from (u+i2)%m where i=0 to m-1.
• Example: Consider the same example which we discussed in the
linear probing.
• A = 3, 2, 9, 6, 11, 13, 7, 12 where m = 10, and h(k) = 2k+3
• The key values 3, 2, 9, 6 are stored at the indexes 9, 7, 1, 5,
respectively. We do not need to apply the quadratic probing technique
on these key values as there is no occurrence of the collision.
49
 Cont.…
The index value of 11 is 5, but this location is already occupied by the 6. So, we apply the
quadratic probing technique.
When i = 0
Index= (5+02)%10 = 5
When i=1
Index = (5+12)%10 = 6
Since location 6 is empty, so the value 11 will be added at the index 6.
The next element is 13. When the hash function is applied on 13, then the index value
comes out to be 9, which we already discussed in the chaining method. At index 9, the cell
is occupied by another value, i.e., 3. So, we will apply the quadratic probing technique to
calculate the free location.
When i=0
Index = (9+02)%10 = 9
When i=1
Index = (9+12)%10 = 0
Since location 0 is empty, so the value 13 will be added at the index 0.
The next element is 7. When the hash function is applied on 7, then the index value comes
out to be 7, which we already discussed in the chaining method. At index 7, the cell is
occupied by another value, i.e., 7. So, we will apply the quadratic probing technique50 to
 Cont.…
Reading assignment
 Double hashing
 Set Representation UNION, FIND Operation

51
 D ! !
E N

52