What is an Algorithm?

-Definition & characteristics

1.Input
2.Output
3.Definiteness
4.Finiteness
5.Effectiveness
The study of algorithms includes
a) How to devise algorithms
b) How to validate algorithms
c) How to analyze algorithms
d) How to test a program
Performance of an algorithm
• Time and space complexity
-Space complexity
1.Fixed part
2.Variable part
S(P)=c+Sp(instance characteristics)
Algorithm sum(a,n) Algorithm Rsum(a,n)
{ s=0.0 {
for i=1 to n do if(n<=0) then return
s=s+a[i] else
return s; return Rsum(a,n-1) +a[n]
}
Ssum(n)>=(n+3)
Performance of an algorithm
• Time and space complexity
-Space complexity
1.Fixed part
2.Variable part
S(P)=c+Sp(instance characteristics)
Algorithm sum(a,n) Algorithm Rsum(a,n)
{ s=0.0 {
for i=1 to n do if(n<=0) then return
s=s+a[i] else
return s; return Rsum(a,n-1) +a[n]
}
Ssum(n)>=(n+3) SRsum>=3(n+1)
formal parameter,local parameter and return address
(3)
Computing Time Complexity of Algorithm

- A priori Analysis
-A posteriori Analysis
Frequency Count
-Best case,average case& worst case
Asymptotic Notations :-
-Big-O
-Omega
-Theta
Introduction

• Data structures
• Methods of organizing data
• What is Algorithm?
• a clearly specified set of simple instructions on the data to be
followed to solve a problem
• Takes a set of values, as input and
• produces a value, or set of values, as output
• May be specified
• In English
• As a computer program
• As a pseudo-code
• Program = data structures + algorithms
Analysis of Algorithms

• An algorithmis a finite set of precise instructions for performing a

computation or for solving a problem.
• •What is the goal of analysis of algorithms?
• –To compare algorithms mainly in terms of running time but also in
terms of other factors (e.g., memory requirements,programmer's
effort etc.)
• •What do we mean by running time analysis?
• –Determine how running time increases as the size of the problem
increases.
• Five conditions for an algorithm
• Input:Zero or more quantities (data) are supplied externally
• Output:At least one quantity (data) is produced
• Definiteness: Each instruction must be clear and
unambiguous
• Finiteness: The algorithm is required to terminate after a
finite number of steps
• Effectiveness: Every instruction must be sufficiently basic so
that anyone can follow
• Problem
• Solution
Introduction

• Why need algorithm analysis ?

• writing a working program is not good enough
• The program may be inefficient!
• If the program is run on a large data set, then the running time becomes an
issue
Example: Selection Problem

• Given a list of N numbers, determine the kth largest, where k  N.

• Algorithm 1:
(1)   Read N numbers into an array
(2)   Sort the array in decreasing order by some simple algorithm
(3)   Return the element in position k
• Algorithm 2:
(1)   Read the first k elements into an array and sort them in decreasing order
(2)   Each remaining element is read one by one
• If smaller than the kth element, then it is ignored
• Otherwise, it is placed in its correct spot in the array, bumping one element out of the
array.
(3)   The element in the kth position is returned as the answer.
• Which algorithm is better when
• N =100 and k = 100?
• N =100 and k = 1?
• What happens when
• N = 1,000,000 and k = 500,000?
• We come back after sorting analysis, and there exist better
algorithms
Algorithm Analysis

• We only analyze correct algorithms

• An algorithm is correct
• If, for every input instance, it halts with the correct output
• Incorrect algorithms
• Might not halt at all on some input instances
• Might halt with other than the desired answer
• Analyzing an algorithm
• Predicting the resources that the algorithm requires
• Resources include
• Memory
• Communication bandwidth
• Computational time (usually most important)
• Factors affecting the running time
• computer
• compiler
• algorithm used
• input to the algorithm
• The content of the input affects the running time
• typically, the input size (number of items in the input) is the main
consideration
• E.g. sorting problem  the number of items to be sorted
• E.g. multiply two matrices together  the total number of elements in
the two matrices
• Machine model assumed
• Instructions are executed one after another, with no concurrent
operations  Not parallel computers
Different approaches

• Empirical: run an implemented system on real-world data. Notion of

benchmarks.
• Simulational: run an implemented system on simulated data.
• Analytical: use theoretic-model data with a theoretical model system.
We do this in 171!
 Example
• Calculate N

 i
i 1
3

1
1
2 2N+2
3 4N
4 1

• Lines 1 and 4 count for one unit each

• Line 3: executed N times, each time four units
• Line 2: (1 for initialization, N+1 for all the tests, N for all the increments)
total 2N + 2
• total cost: 6N + 4  O(N)
Worst- / average- / best-case

• Worst-case running time of an algorithm

• The longest running time for any input of size n
• An upper bound on the running time for any input
 guarantee that the algorithm will never take longer
• Example: Sort a set of numbers in increasing order; and the data is in
decreasing order
• The worst case can occur fairly often
• E.g. in searching a database for a particular piece of information
• Best-case running time
• sort a set of numbers in increasing order; and the data is already in
increasing order
• Average-case running time
• May be difficult to define what “average” means
Running-time of algorithms

• Bounds are for the algorithms, rather than programs

• programs are just implementations of an algorithm, and almost
always the details of the program do not affect the bounds

• Algorithms are often written in pseudo-codes

• We use ‘almost’ something like C++.

• Bounds are for algorithms, rather than problems

• A problem can be solved with several algorithms, some are more
efficient than others
Growth Rate

• The idea is to establish a relative order among functions for large n

•  c , n0 > 0 such that f(N)  c g(N) when N  n0
• f(N) grows no faster than g(N) for “large” N
Typical Growth Rates
Growth rates …

• Doubling the input size

• f(N) = c  f(2N) = f(N) = c
• f(N) = log N  f(2N) = f(N) + log 2
• f(N) = N  f(2N) = 2 f(N)
• f(N) = N2  f(2N) = 4 f(N)
• f(N) = N3  f(2N) = 8 f(N)
• f(N) = 2N  f(2N) = f2(N)
• Advantages of algorithm analysis
• To eliminate bad algorithms early
• pinpoints the bottlenecks, which are worth coding carefully
Asymptotic notations

• Upper bound O(g(N)

• Lower bound (g(N))
• Tight bound (g(N))
Asymptotic upper bound: Big-Oh

• f(N) = O(g(N))
• There are positive constants c and n0 such that
f(N)  c g(N) when N  n0

• The growth rate of f(N) is less than or equal to the growth

rate of g(N)
• g(N) is an upper bound on f(N)
• In calculus: the errors are of order Delta x, we write E =
O(Delta x). This means that E <= C Delta x.

• O(*) is a set, f is an element, so f=O(*) is f in O(*)

• 2N^2+O(N) is equivelent to 2N^2+f(N) and f(N) in O(N).
Big-Oh: example

• Let f(N) = 2N2. Then

• f(N) = O(N4)
• f(N) = O(N3)
• f(N) = O(N2) (best answer, asymptotically tight)

• O(N2): reads “order N-squared” or “Big-Oh N-squared”

Some rules for big-oh

• Ignore the lower order terms

• Ignore the coefficients of the highest-order term
• No need to specify the base of logarithm
• Changing the base from one constant to another changes the value
of the logarithm by only a constant factor

If T1(N) = O(f(N) and T2(N) = O(g(N)),

 T1(N) + T2(N) = max( O(f(N)), O(g(N)) ),

 T1(N) * T2(N) = O( f(N) * g(N) )
 Big Oh: more examples
• N2 / 2 – 3N = O(N2)
• 1 + 4N = O(N)
• 7N2 + 10N + 3 = O(N2) = O(N3)
• log10 N = log2 N / log2 10 = O(log2 N) = O(log N)
• sin N = O(1); 10 = O(1), 1010 = O(1)
•


N 2
i 1
i  N  N  O(N )

 i 1
N 2 2 3
i  N  N  O ( N )
• log N + N = O(N)
• logk N = O(N) for any constant k
• N = O(2N), but 2N is not O(N)
• 210N is not O(2N)
Math Review
a
x  b iff log x b  a
log ab  log a  log b
log m b
log a b 
log m a
log a b  b log a
a log n  n log a
logb a  (log a )b  log a b
d loge x 1

dx x
lower bound

•  c , n0 > 0 such that f(N)  c g(N) when N  n0

• f(N) grows no slower than g(N) for “large” N
Asymptotic lower bound: Big-
Omega

• f(N) = (g(N))
• There are positive constants c and n0 such that
f(N)  c g(N) when N  n0

• The growth rate of f(N) is greater than or equal to the

growth rate of g(N).
• g(N) is a lower bound on f(N).
Big-Omega: examples

• Let f(N) = 2N2. Then

• f(N) = (N)
• f(N) = (N2) (best answer)
tight bound

• the growth rate of f(N) is the same as the growth rate of g(N)
Asymptotically tight bound: Big-Theta

• f(N) = (g(N)) iff f(N) = O(g(N)) and f(N) = (g(N))

• The growth rate of f(N) equals the growth rate of g(N)

• Big-Theta means the bound is the tightest possible.
• Example: Let f(N)=N2 , g(N)=2N2
• Since f(N) = O(g(N)) and f(N) = (g(N)),
thus f(N) = (g(N)).
Some rules

• If T(N) is a polynomial of degree k, then

T(N) = (Nk).

• For logarithmic functions,

T(logm N) = (log N).
General Rules
• Loops
• at most the running time of the statements inside the for-loop
(including tests) times the number of iterations.
• O(N)
• Nested loops

• the running time of the statement multiplied by the product of the

sizes of all the for-loops.
• O(N2)
• Consecutive statements
• These just add
• O(N) + O(N2) = O(N2)

• Conditional: If S1 else S2
• never more than the running time of the test plus the larger of the
running times of S1 and S2.
• O(1)
This is rarely used in 171, as we know the relative growth
rates of most of functions used in 171!
Using L' Hopital's rule
• ‘rate’ is the first derivative
• L' Hopital's rule lim g ( N )  
lim f ( N )   n
• If n  and
f (N ) f ( N )
lim lim
n g(N ) n   g ( N )
then =

• Determine the relative growth rates (using L' Hopital's rule if necessary)
f (N )
• compute lim
n g(N )
• if 0: f(N) = o(g(N)) and f(N) is not (g(N))
• if constant  0: f(N) = (g(N))
• if : f(N) = (f(N)) and f(N) is not (g(N))
• limit oscillates: no relation
Our first example: search of an
ordered array
• Linear search and binary search
• Upper bound, lower bound and tight bound
Time Complexity
Example-Nonrecursive Algorithm

Statement s/ Frequency Total

e steps
1 Algo sum(a,n) 0 - 0
2{ 0 - 0
3 s:=0.0 1 1 1
4 for i=1 to n do 1 n+1 n+1
5 s=s+a[i] 1 n n
6 Return s; 1 1 1
7} 0 - 0
Total 2n+3
Time Complexity of recursive algo
Statement s/e Frequency Total
N=0 n=1 steps
1 Algo Rsum(a,n) 0 - 0
2{ - 0
3 if(n<=0) then 1 1 1 1 1
4 Return 0.0 1 1 0 1 0
5 else 1 n n
6return
Rsum(a,n-1)+a[n]; 1+x 0 1 0 1+x
7}
Total 2 2+x

tRsum(n)= 2 if n=0
2+tRsum(n-1) if n>0
Statement s/ Frequen Total
e cy steps

1 add(a,b,c,n,n) 0 - -
2{ 0 - -
3 for i=1 to n do 1 n+1 n+1
4 for j=1 to n do 1 n(n+1) N*n+n
5 c[I,j)=a(i,j) 1 n*n n*n
+b(i,j) 0 - -
6}
Total 2n2+2
n+1
• Logarithmic Loops
Multiply Loops Divide Loops
1. i=1
2. loop(i<100) stmts
i=i x 2
3. end loop

To calculate no of iterations

Multiply =2^iterations<100

f(n)=[log 2 n]
• Logarithmic Loops
Multiply Loops Divide Loops
1. i=1 1. i=100
2. loop(i<100) 2. loop(i>0)
stmts stmts
i=i x 2 i=i / 2
3. end loop 3.end loop

Divide= 100/2^iterations >=1

f(n)=[log 2 n]
• Dependant Quadratic
i=1
loop(i<=10)
j=1
loop(j<=i)
application code
j=j+1
end loop
i=i+1
end loop
Inner loop=1+2+3….+10=55 avg=55/10=5.5=(n+1)/2
f(n)=n((n+1)/2)
• Linear Logarithmics Loops
i=1
loop(i<=10)
j=1
loop(j<=10)
application code
j=j+2
end loop
i=i+1
end loop
Inner loop=log210
outer loop=10xlog210
f(n)=n log2n
Expo(x,n)//return x^n for int >=0
{
m=n; power=1;
z=x;
while(m>0) do
{
while((m mod 2)=0) do
{
m=m/2; z=z^2
}
m=m-1; power=power * z;
}
}
//logn iterations
Review: Asymptotic Performance

• Asymptotic performance: How does algorithm behave as the problem

size gets very large?
• Running time
• Memory/storage requirements
• Remember that we use the RAM model:
• All memory equally expensive to access
• No concurrent operations
• All reasonable instructions take unit time
• Except, of course, function calls
• Constant word size
• Unless we are explicitly manipulating bits

David Luebke
47
03/12/2023
Review: Running Time

• Number of primitive steps that are executed

• Except for time of executing a function call most statements roughly require
the same amount of time
• We can be more exact if need be
• Worst case vs. average case

David Luebke
48
03/12/2023
An Example: Insertion Sort

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
49
03/12/2023
An Example: Insertion Sort
30 10 40 20 i =  j =  key = 
A[j] =  A[j+1] = 
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
50
03/12/2023
An Example: Insertion Sort
30 10 40 20 i=2 j=1 key = 10
A[j] = 30 A[j+1] = 10
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
51
03/12/2023
An Example: Insertion Sort
30 30 40 20 i=2 j=1 key = 10
A[j] = 30 A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
52
03/12/2023
An Example: Insertion Sort
30 30 40 20 i=2 j=1 key = 10
A[j] = 30 A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
53
03/12/2023
An Example: Insertion Sort
30 30 40 20 i=2 j=0 key = 10
A[j] =  A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
54
03/12/2023
An Example: Insertion Sort
30 30 40 20 i=2 j=0 key = 10
A[j] =  A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
55
03/12/2023
An Example: Insertion Sort
10 30 40 20 i=2 j=0 key = 10
A[j] =  A[j+1] = 10
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
56
03/12/2023
An Example: Insertion Sort
10 30 40 20 i=3 j=0 key = 10
A[j] =  A[j+1] = 10
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
57
03/12/2023
An Example: Insertion Sort
10 30 40 20 i=3 j=0 key = 40
A[j] =  A[j+1] = 10
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
58
03/12/2023
An Example: Insertion Sort
10 30 40 20 i=3 j=0 key = 40
A[j] =  A[j+1] = 10
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
59
03/12/2023
An Example: Insertion Sort
10 30 40 20 i=3 j=2 key = 40
A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
60
03/12/2023
An Example: Insertion Sort
10 30 40 20 i=3 j=2 key = 40
A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
61
03/12/2023
An Example: Insertion Sort
10 30 40 20 i=3 j=2 key = 40
A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
62
03/12/2023
An Example: Insertion Sort
10 30 40 20 i=4 j=2 key = 40
A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
63
03/12/2023
An Example: Insertion Sort
10 30 40 20 i=4 j=2 key = 20
A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
64
03/12/2023
An Example: Insertion Sort
10 30 40 20 i=4 j=2 key = 20
A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
65
03/12/2023
An Example: Insertion Sort
10 30 40 20 i=4 j=3 key = 20
A[j] = 40 A[j+1] = 20
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
66
03/12/2023
An Example: Insertion Sort
10 30 40 20 i=4 j=3 key = 20
A[j] = 40 A[j+1] = 20
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
67
03/12/2023
An Example: Insertion Sort
10 30 40 40 i=4 j=3 key = 20
A[j] = 40 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
68
03/12/2023
An Example: Insertion Sort
10 30 40 40 i=4 j=3 key = 20
A[j] = 40 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
69
03/12/2023
An Example: Insertion Sort
10 30 40 40 i=4 j=3 key = 20
A[j] = 40 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
70
03/12/2023
An Example: Insertion Sort
10 30 40 40 i=4 j=2 key = 20
A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
71
03/12/2023
An Example: Insertion Sort
10 30 40 40 i=4 j=2 key = 20
A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
72
03/12/2023
An Example: Insertion Sort
10 30 30 40 i=4 j=2 key = 20
A[j] = 30 A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
73
03/12/2023
An Example: Insertion Sort
10 30 30 40 i=4 j=2 key = 20
A[j] = 30 A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
74
03/12/2023
An Example: Insertion Sort
10 30 30 40 i=4 j=1 key = 20
A[j] = 10 A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
75
03/12/2023
An Example: Insertion Sort
10 30 30 40 i=4 j=1 key = 20
A[j] = 10 A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
76
03/12/2023
An Example: Insertion Sort
10 20 30 40 i=4 j=1 key = 20
A[j] = 10 A[j+1] = 20
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
77
03/12/2023
An Example: Insertion Sort
10 20 30 40 i=4 j=1 key = 20
A[j] = 10 A[j+1] = 20
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Done!
David Luebke
78
03/12/2023
Animating Insertion Sort

• Check out the Animator, a java applet at:

http://www.cs.hope.edu/~alganim/animator/Animator.ht
ml

• Try it out with random, ascending, and descending inputs

David Luebke
79
03/12/2023
Insertion Sort
What is the precondition
InsertionSort(A, n) { for this loop?
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
80
03/12/2023
Insertion Sort

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
} How many times will
} this loop execute?

David Luebke
81
03/12/2023
Insertion Sort

Statement Effort
InsertionSort(A, n) {
for i = 2 to n { c1 n
key = A[i] c2(n-1)
j = i - 1; c3(n-1)
while (j > 0) and (A[j] > key) { c4T
A[j+1] = A[j] c5(T-(n-1))
j = j - 1c6(T-(n-1))
} 0
A[j+1] = keyc7(n-1)
} 0
}
T = t2 + t3 + … + tn where ti is number of while expression evaluations for the ith for loop iteration
David Luebke
82
03/12/2023
 Analyzing Insertion Sort
• T(n) = c1n + c2(n-1) + c3(n-1) + c4T + c5(T - (n-1)) + c6(T - (n-1)) + c7(n-1)
= c8T + c9n + c10
• What can T be?
• Best case -- inner loop body never executed
• ti = 1  T(n) is a linear function
• Worst case -- inner loop body executed for all previous elements
• ti = i  T(n) is a quadratic function
• Average case
• ???

David Luebke
83
03/12/2023
Analysis

• Simplifications
• Ignore actual and abstract statement costs
• Order of growth is the interesting measure:
• Highest-order term is what counts
• Remember, we are doing asymptotic analysis
• As the input size grows larger it is the high order term that dominates

David Luebke
84
03/12/2023
Upper Bound Notation

• We say InsertionSort’s run time is O(n2)

• Properly we should say run time is in O(n2)
• Read O as “Big-O” (you’ll also hear it as “order”)
• In general a function
• f(n) is O(g(n)) if there exist positive constants c and n0 such that f(n)  c  g(n)
for all n  n0
• Formally
• O(g(n)) = { f(n):  positive constants c and n0 such that f(n)  c  g(n)  n  n0

David Luebke
85
03/12/2023
Insertion Sort Is O(n2)

• Proof
• Suppose runtime is an2 + bn + c
• If any of a, b, and c are less than 0 replace the constant with its absolute value
• an2 + bn + c  (a + b + c)n2 + (a + b + c)n + (a + b +
c)
•  3(a + b + c)n2 for n  1
• Let c’ = 3(a + b + c) and let n0 = 1
• Question
• Is InsertionSort O(n3)?
• Is InsertionSort O(n)?

David Luebke
86
03/12/2023
Big O Fact

• A polynomial of degree k is O(nk)

• Proof:
• Suppose f(n) = bknk + bk-1nk-1 + … + b1n + b0
• Let ai = | bi |
• f(n)  aknk + ak-1nk-1 + … + a1n + a0

i
n
 n  ai k k
 n k
a
i  cn k

David Luebke
87
03/12/2023
Lower Bound Notation

• We say InsertionSort’s run time is (n)

• In general a function
• f(n) is (g(n)) if  positive constants c and n0 such that 0  cg(n)  f(n)  n 
n0
• Proof:
• Suppose run time is an + b
• Assume a and b are positive (what if b is negative?)
• an  an + b

David Luebke
88
03/12/2023
Asymptotic Tight Bound

• A function f(n) is (g(n)) if  positive constants c1, c2, and n0 such that

c1 g(n)  f(n)  c2 g(n)  n  n0

• Theorem
• f(n) is (g(n)) iff f(n) is both O(g(n)) and (g(n))
• Proof: someday

David Luebke
89
03/12/2023
Practical Complexity
250

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

David Luebke
90
03/12/2023
Practical Complexity
500

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

David Luebke
91
03/12/2023
Practical Complexity
1000

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 3 5 7 9 11 13 15 17 19

David Luebke
92
03/12/2023
Practical Complexity
5000

4000
f(n) = n
f(n) = log(n)
3000
f(n) = n log(n)
f(n) = n^2
2000 f(n) = n^3
f(n) = 2^n

1000

0
1 3 5 7 9 11 13 15 17 19

David Luebke
93
03/12/2023
Practical Complexity
10000000

1000000

100000

10000

1000

100

10

1
1 4 16 64 256 1024 4096 16384 65536

David Luebke
94
03/12/2023
Other Asymptotic Notations

• A function f(n) is o(g(n)) if  positive constants c and n0 such that

f(n) < c g(n)  n  n0
• A function f(n) is (g(n)) if  positive constants c and n0 such that
c g(n) < f(n)  n  n0
• Intuitively,

 o() is like <  () is like >  () is like =

 O() is like   () is like 
David Luebke
95
03/12/2023
 Linear search:

// Given an array of ‘size’ in increasing order, find ‘x’

int linearsearch(int* a[], int size,int x) {
int low=0, high=size-1;

for (int i=0; i<size;i++)

O(N)
if (a[i]==x) return i;
return -1;
}
 Iterative binary search:

int bsearch(int* a[],int size,int x) {

int low=0, high=size-1;

while (low<=higt) {
int mid=(low+high)/2;
if (a[mid]<x)
low=mid+1;
else if (x<a[mid])
high=mid-1;
else return mid;
}
return -1
}
 Iterative binary search:

int bsearch(int* a[],int size,int x) {

int low=0, high=size-1;

while (low<=higt) {
int mid=(low+high)/2;
if (a[mid]<x)
low=mid+1;
else if (x<a[mid])
high=mid-1;
else return mid;
}
return -1
} n=high-low
n_i+1 <= n_i / 2
i.e. n_i <= (N-1)/2^{i-1}
N stops at 1 or below
there are at most 1+k iterations, where k is the smallest such
that (N-1)/2^{k-1} <= 1
so k is at most 2+log(N-1)
O(log N)
 Some standard Time Complexity
Functions
• Linear O(n)
• Logarithmic O(logn)
• Linear logarithmic O(nlogn)
• Quadratic O(n^2)
• Cubic O(n^3)
• Exponential O(a^n / 2^n)
permutation problems
Graphical Representation of Time complexity Functions

N log n

Log n

1 2 4 8 16 32 64 128
logn n n logn n^2 n^3 2^n

0 1 0 1 1 2

1 2 2 4 8 4

2 4 8 16 64 16

3 8 24 64 542 256

4 16 64 256 4096 6553

6
5 32 160 1024 3276 Big
8 no
More Complexity Examples
Example 1
I=1
P=1
while (I < N) do
{P=P*I
I=I+1
}
T(n) = 2 + (N - 1) * (1 + 1)
= 2 + (N - 1) * (2)
= O(2 + (2N - 2))
= O(N)
Example2

for(i=0=sum=0;i<n;i++)
sum+=a[i]
Solution:-
i=0-----1 time
sum=0-----1 time

sum+=a[i] ----n
i++ ----n
Total frequency count =2n+2
Asynptotic complexity=O(n)
Example3---Solve
for(i=0;i<n;i++)
for(j=1,sum=a[0];j<=i;j++)
sum+=a[j];
Example 4
for I = 1 to N - 1 do
for J = N downto I + 1 do
if A[J-1] > A[J] then
swap(A[J-1], a[J])
inner loop O((n - i) * 1) = O(n-i)

outer loop
                              
 Solve example 5
z = 1.0
t=a
k=n
while K > 0 do ---1
{
if odd(k) then ----1
z=z*t
k = k/2 ----1
if K <> 0 then ----1
t=t*t
}
 Answer of example 3
• k = n, n/2, n/4, n/8, .... stop m iterations
 Bubble sort
bubblesort(int array[],int array_size)
{
for(i=array_size-1;i>=0;i--)
{
for(j=1;j<=i;j++)
{
if(array[j-1]>array[j]
{
swap(array[j-1],array[ j])
}
}
(n-1)+(n-2)+(n-3)+……..+1=n(n-1)/2
 Selection sort
• Void selectionsort(int array[],int array_size)
{
for(i=0;i<array_size;i++)
{ min=i
for(j=i+1;j<array_size; j++)
{
if(array[j]>array[min]
{
min=j
} swap(array[i],array[ min])
}
(n-1)+(n-2)+(n-3)+……..+1=n(n-1)/2
Insertion sort’s Time Complexity
• Void insertion sort(int array[],int array_size)
{
for(i=1;i<array_size;i++)
{
index=array[i]
j=i
while(j>0 && array[j-1]>index)
{
array[j]=array[j-1]
}
array[j]=index
}
}
best- O(n)
worst(1+2+3+……+(n-3)+(n-2)+(n-1)
=n*(n-1)/2
Recurrence Relations
• Can easily describe the runtime of recursive algorithms
• Can then be expressed in a closed form (not defined in terms of itself)
• Consider the linear search:
Int recursive_lin(int arr[10],int n,int key)
{ if(n>=0)
{
if(arr[n]==key)
return n;
recursive_lin(arr,n-1,key)
}
else
return -1;
}
 Eg. 1 - Linear Search
• Recursively
• Look at an element (constant work, c), then search the remaining
elements…

• T(n) = T( n-1 ) + c
• “The cost of searching n elements is
the cost of looking at 1 element, plus
the cost of searching n-1 elements”
Linear Seach (cont)

Caveat:
• You need to convince yourself (and others) that the single step,
examining an element, *is* done in constant time.
• Can I get to the ith element in constant time, either directly, or from
the (i-1)th element?
• Look at the code
 Methods of Solving Recurrence
Relations
• Substitution (we’ll work on this one in this lecture)
• Accounting method
• Draw the recursion tree, think about it
• The Master Theorem*
• Guess at an upper bound, prove it

* See Cormen, Leiserson, & Rivest, Introduction to Algorithms

Linear Search (cont.)

• We’ll “unwind” a few of these

T(n) = T(n-1) + c (1)
But, T(n-1) = T(n-2) + c, from above
Substituting back in:
T(n) = T(n-2) + c + c
Gathering like terms
T(n) = T(n-2) + 2c (2)
Linear Search (cont.)

• Keep going:
T(n) = T(n-2) + 2c
T(n-2) = T(n-3) + c
T(n) = T(n-3) + c + 2c
T(n) = T(n-3) + 3c (3)
• One more:
T(n) = T(n-4) + 4c (4)
Eg. 1 – list of intermediates

Result at ith unwinding i

T(n) = T(n-1) + 1c 1

T(n) = T(n-2) + 2c 2

T(n) = T(n-3) + 3c 3

T(n) = T(n-4) + 4c 4
Linear Search (cont.)

• An expression for the kth unwinding:

T(n) = T(n-k) + kc
• We have 2 variables, k and n, but we have a relation
• T(d) is constant (can be determined) for some constant d (we know
the algorithm)
• Choose any convenient # to stop.
Linear Search (cont.)

• Let’s decide to stop at T(0). When the list to search is empty, you’re
done…
• 0 is convenient, in this example…
Let n-k = 0 => n=k
• Now, substitute n in everywhere for k: T(n) = T(n-n) + nc
T(n) = T(0) + nc = nc + c0 = O(n)
( T(0) is some constant, c0 )
Write a recurrence relation for factorial and
analyse same?
• Int fact( int n)

{
if(n==0 or n==1)
return 1;
else
return n*fact(n-1)
}
N=5 5*fact(4),=5*24=120
N=4 4*fact(3)=4*6=24
N=3 3*fact(2)=3*2=6
N=2 =2*fact(1) =2*1=2
N=1 return 1
Divide and Conquer algorithmic Strategy
-General Method
Algo DAandC(P)
{
if(small(P) then return S(P);
else
{
divide p into P1,P2,….Pk,k>=1
Apply DAandC to each subproblems return
Combine(DAandC(P1),….
DAandC(Pk))
}
}
Divide-and-Conquer Technique (cont.)

a problem of size n
(instance)

subproblem 1 subproblem 2
of size n/2 of size n/2

a solution to a solution to
subproblem 1 subproblem 2

a solution to It general leads to a

the original problem
recursive algorithm!
Time Complexity
T(n)= g(n) n small
T(n1)+T(n2)+…..+T(nk) +f(n)

In the Recurrence form

T(n)= T(1) n=1
aT(n/b)+f(n) n>1
Examples
1)Merge Sort
2)Quick sort
3)Binary Search
 Recursive binary search:

int bsearch(int* a[],int low, int high, int x) {

if (low>high) return -1;

O(1)
else int mid=(low+high)/2;
T (1)  1
if (x=a[mid]) return mid; N
T (N ) T ( )  1
O(1)
else if(a[mid]<x) 2

bsearch(a,mid+1,high,x); T(N/2)
else bsearch(a,low,mid-1);

}
Binary Search

• Algorithm – “check middle, then search lower ½ or upper ½”

• T(n) = T(n/2) + c
where c is some constant, the cost of checking the middle…
Binary Search (cont)

Let’s do some quick substitutions:

T(n) = T(n/2) + c (1)
but T(n/2) = T(n/4) + c, so
T(n) = T(n/4) + c + c
T(n) = T(n/4) + 2c (2)
T(n/4) = T(n/8) + c
T(n) = T(n/8) + c + 2c
T(n) = T(n/8) + 3c (3)
Binary Search (cont.)

Result at ith unwinding i

T(n) = T(n/2) + c 1

T(n) = T(n/4) + 2c 2

T(n) = T(n/8) + 3c 3

T(n) = T(n/16) + 4c 4
Binary Search (cont)

Result at ith unwinding i

T(n) = T(n/2) + c =T(n/21) + 1c 1

T(n) = T(n/4) + 2c =T(n/22) + 2c 2

T(n) = T(n/8) + 3c =T(n/23) + 3c 3

T(n) = T(n/16) + 4c =T(n/24) + 4c 4

Binary Search (cont)

• After k unwindings:
T(n) = T(n/2k) + kc
• Need a convenient place to stop unwinding – need to relate k & n
• Let’s pick T(0) = c0 So,
n/2k = 0 =>
n=0
Hmm. Easy, but not real useful…
Binary Search (cont)

• Okay, let’s consider T(1) = c0

• So, let:
n/2k = 1 =>
n = 2k =>
k = log2n = lg n
Binary Search (cont.)

• Substituting back in (getting rid of k):

T(n) = T(1) + c lg(n)
= c lg(n) + c0
= O( lg(n) )
Quick sort(Partition Exchange Sort)
Void quick(x,lb,ub)
{
int j;
if (lb>=ub)
return;
else
{
partition(x,lb,ub,&j);
quick(x,lb,j-1);
quick(x,j+1,ub);
}
}
Partition of Quick sort
1.Take pivot element (x[lb])
2.Take down=lb, up=ub
a. Repeatedly increment down index until
x[down]>pivot
b. Repeatedly decrement up index until x[up]<pivot
c.if(down<up)
exchange x[down] & x[up]and repeat step a,b,c til
down<up
3.Up is position of pivot element
swap(x[up],x[lb])

4. Return pivot element position j=up;

 Analysis of Quick sort
T(n) = c1 n=1 ,c is constant
c*n+ 2T(n/2) n>1
When n is power of 2 ,n=2k
T(n)= c*n + 2( c*n/2 + 2T(n/4))
= c*n+c*n +4 T(n/4)=2c*n+4 T(n/4)
=2c*n+4(c*n/4+2T(n/8))
.
.
=2kT(1)+k *n*c
= 2 logn*c1+log2n *n*c=O(nlog2n)
Worst case analysis of quick sort
T(n) =c*n +T(n-1)
= c*n+c*(n-1)+T(n-2)
=2cn-c+T(n-2)
= 2cn-c+c*(n-2)+T(n-3)
= 2cn-c+cn-2c+T(n-3)
= 3cn-3c+T(n-3)
.
.
= n*cn-n*c+T(1) = O(n2)
complexity of quick sort
Time Complexity
1.Best case –O(log2n)
2.Average case-O(n log2n)
3.Worst case –O(n2)

Space Complexity
Best - O(log2n)
worst –O(n)
Merge Sort Using Divide and
Conquer
Mergesort
• Split array A[0..n-1] into about equal halves and make copies of
each half in arrays
• Sort sub arrays recursively
• Merge sorted sub arrays of A into new array B as follows:
• Repeat the following until no elements remain in one of the
subarrays of A:
• compare the first elements in the remaining unprocessed portions of the arrays
• copy the smaller of the two sub array into B, while incrementing the index indicating the
unprocessed portion of that array
• Once all elements in one of the arrays are processed, copy the
remaining unprocessed elements from the other subarray into B.
Merge Sort
Algo mergesort(a,low,high)
if(low <high)//more than 1 elem
{ //divide p into subproblem
mid=(low+high)/2;
mergesort(a,low,mid);
mergesort(a,mid+1,high);
//combine solutions
merge(a,low,mid,high);
}
Merge Sort
 Merging of two subarrays using additional storage
Merge(a,low,mid,high)
{ h=low; i=low; j=mid+1;
While(h<=mid and j<=high)
{ if(a[h]<=a[j]) then
b[i]=a[h]; h=h+1
else
b[i]=a[j]; j=j+1;
i=i+1;
}
Continue….
If(h>mid) then
for k=j to high
{ b[i]=a[k];
i=i+1;
}
else
{ for k=h to mid
{
b[i]=a[k]; i=i+1;
}
}
Merge Sort
Mergesort Example 8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 71 5 4

8 3 2 9 7 1 5 4

The non-
3 8 2 9 1 7 4 5
recursive
version of
Mergesort
2 3 8 9 1 4 5 7
starts from
merging single
1 2 3 4 5 7 8 9 elements into
sorted pairs.
Merge Sort: Complexity

MERGE-SORT(A, p, r)
if p < r
then q = (p + r)/2
MERGE-SORT(A, p, q) (1)
MERGE-SORT(A, q + 1, r) T(n/2)
MERGE(A, p, q, r)
T(n/2)
If n1, T(n) = 2 T(n/2) + (n) (n)
If n=1, T(n) = (1)
T(n) = (n lg n)
Solving Recurrence Equation
T(n) = 2 T(n/2) + (n) when n > 1
T(n) = (1) when n = 1

cn cn cn cn cn … cn cn cn

2c 2c 2c … 2c 2c
lg n
4c … 4c
cn/2 cn/2
cn
That is, T(n) = (n lg n)
 Analysis Of merge sort
T(n)= a n=1 ,a a constant
2T(n/2) +cn n>1
c a const
Solve the recursion n=2^k
 Complexities of sorting methods
Sorting Run Avg worst bes Spac Comments
time t e
bubble n2 n2 n2 n2 no Good for
small n
selectio n2 n2 n2 n2 no Partially
n sorted data &
small n
insertio n2 N-1 n2 n2 no Almost sorted
n data
Quick N(log2n N(log2 N(log log2 log2n Excellent
) n) 2n) n

Shell N5/3 N5/3 N3/2 N3/2 no Good

Radix n n n n n Excellent

merge nlog2n nlog2n nlog2 nlog n

External file
n n
Another Example of recurrence

• Maximum Subsequence Sum Problem

• Given (possibly negative) integers A1, A2, ...., An, find the maximum
value of For convenience, the maximum subsequence j sum is 0 if all
the integers are negative 
k i
Ak

• E.g. for input –2, 11, -4, 13, -5, -2

• Answer: 20 (A2 through A4)
Algorithm 1: Simple
• Exhaustively tries all possibilities (brute force)

N
N-i, at most N

j-i+1, at most N

• O(N3)
 Algorithm 2: improved
// Given an array from left to right
int maxSubSum(const int* a[], const int size) {
int maxSum = 0;

for (int i=0; i< size; i++) {

int thisSum =0;
N
for (int j = i; j < size; j++) {
N-i, at most N
thisSum += a[j];
if(thisSum > maxSum)
maxSum = thisSum;
}
}

return maxSum;
}

O(N^2)
 Algorithm 3: Divide-and-conquer
• Divide-and-conquer
• split the problem into two roughly equal subproblems, which are
then solved recursively
• patch together the two solutions of the subproblems to arrive at a
solution for the whole problem

  The maximum subsequence sum can be

 Entirely in the left half of the input
 Entirely in the right half of the input
 It crosses the middle and is in both halves
• The first two cases can be solved recursively
• For the last case:
• find the largest sum in the first half that includes the last element in
the first half
• the largest sum in the second half that includes the first element in the
second half
• add these two sums together
// Given an array from left to right
int maxSubSum(a,left,right) {
if (left==right) return a[left]; O(1)
else
mid=(left+right)/2
maxLeft=maxSubSum(a,left,mid);
T(N/2)
maxRight=maxSubSum(a,mid+1,right);
T(N/2)
maxLeftBorder=0; leftBorder=0;
for(i = mid; i>= left, i--) {
leftBorder += a[i];
if (leftBorder>maxLeftBorder) O(N)
maxLeftBorder=leftBorder;
}

// same for the right

maxRightBorder=0; rightBorder=0;
for … {
O(N)
}

return max3(maxLeft,maxRight, maxLeftBorder+maxRightBorder);

} O(1)
• Recurrence equation

T (1)  1
N
• 2 T(N/2): T (N
two )  2 T (each)ofsizeNN/2
subproblems,
• N: for “patching” two solutions2to find solution to whole problem
Change of variable
T(n)=T(n/2)+c ->sub,tree,change,masters
T(i)-T(i-1)=c put n=2^i n->i
compare ->(b^n).P(n) b=1,p(n)=1 ->d=0
(x-1)(x-b)^d+1=>(x-1)(x-1)x=1->2 times
T(i)=c1(r1)^i+i*c2(r1)^i i=>logn
T(i)=c1+i*c2
T(n)=c1+logn*c2
T(n)=logn
Proof Techniques
 Proof Techniques-Contradiction
Statement- There is no Largest number
• Proof: Proof by contradiction.
• Step 1. Contrary assumption: Assume that there is a largest integer.
Call it B (for “biggest”).
• Step 2. Show this assumption leads to a contradiction: Consider
C = B + 1. C is an integer because it is the sum of two integers.
Also, C > B, which means that B is not the largest integer after all. Thus,
we have reached a contradiction. The only flaw in our reasoning is the
initial assumption that the theorem is false. Thus, we conclude that the
theorem is correct.

160
Mathematical Induction

161
162
Example 1

163
Example 2

164
Example 2

165
Example3

166