Complexity Analysis

Text is mainly from Chapter 2 by Drozdek

 Computational complexity

• Developed by Juris Hartmanis & Richard E. Stearns

• Compares efficiency of algorithms
• Assesses the degree of difficulty of an algorithm
• How much effort is needed to APPLY an algorithm
• Estimate the resources required by an algorithm
• How costly it is
 What is the COST of an algorithm???

• Time
• Space
 How can you practically compare 2
algorithms???
• Run them on the same machine
• Implement them using the same language
• Only time units such as nano-seconds or micro
seconds should not be used…WHY???
• Logical units expressing the relation between
the size of file ‘n’ and the amount of time ‘t’
required to process it should be used
• Same input data should be used
 Problems with this approach
• Effort
• One program may be better written than the
other
• Choice of tests may favor one of the
algorithms

We want a method in which we compare two algorithms WITHOUT executing them

The method should be independent of hardware/compiler/operating system
 What do we need?????
We require algorithm analysis!!!

• Estimate the performance of an algorithm through

– The number of operations required to process an input
– Process an input of certain size
• Require a function expressing relation between n & t called
time complexity function T(n)
• For calculating T(n) we need to compute the total number of
program steps … (can be the number of executable statements
or meaningful program segment)
 Calculating T(n)
• A program step is the syntactically / semantically
meaningful segments of a program
• A step DOESNOT correspond to a definite time unit
• A step count is telling us how run time for a program
changes with change in data size
• Calculate the total number of steps/executable
statements in a program
• Find the frequency of each statement and sum them
up
• Don’t count comments and declarations
 Analysis of ‘for’ Loop
1: for (i=0;i<n;++i)
2: sum++;
3: cout << sum

You can see….

That statement 1 executes n+1 times
Statement 2 executes n times
Statement 3 executes 1 time
T(n) = ???

The condition always executes one more time than the loop
itself
 Nested ‘for’ loops

for (i=0;i<n;++i)
{ for (j=0;j<m;++j)
sum++
}
The statement sum++ executes n*m times
So in a nested for loop if the loop variables are independent then:

The total number of times a statement executes =

outer loop times * inner loop times
 Problems with T(n)
• T(n) is difficult to calculate
• T(n) is also not very meaningful as step size is not exactly
defined
• T(n) is usually very complicated so we need an approximation
of T(n)….close to T(n).
• This measure of efficiency or approximation of T(n) is called
ASYMPTOTIC COMPLEXITY or ASYMPTOTIC
ALGORITHM ANALYSIS

Asymptotic complexity studies the efficiency of an algorithm as

the input size becomes large
 Example
• If T(n) = 7n+100
• What is T(n) for different values of n???
n T(n) Comment
1 107 Contributing factor is 100
5 135 Contributing factor is 7n and 100
10 170 Contributing factor is 7n and 100
100 800 Contribution of 100 is small
1000 7100 Contributing factor is 7n
10000 70100 Contributing factor is 7n
106 7000100 What is the contributing factor????

When approximating T(n) we can IGNORE the 100 term for very large value of n
and say that T(n) can be approximated by 7(n)
 Example 2
• T(n) = n2 + 100n + log10n +1000
n T(n) n2 100n log10n 1000

Val % Val % Val % Val %

1 1101 1 0.1% 100 9.1% 0 0% 1000 90.8%
10 2101 100 5.8% 1000 47.6% 1 0.05% 1000 47.6%
100 21002 10000 47.6% 10000 47.6% 2 0.99% 1000 4.76%
105 10,010,001,005 1010 99.9% 107 .099% 5 0.0% 1000 0.00%

When approximating T(n) we can IGNORE the last 3 terms and say say that T(n) can
be approximated by n2
 Big-Oh or Big-O
• Definition:
f(n) is O(g(n)) if there exist positive numbers c & N such that f(n)<=cg(n)
for all n>=N

g(n) is called the upper bound on f(n) OR

f(n) grows at the most as large as g(n)
Example:
T(n) = n2 + 3n + 4
n2 + 3n + 4 <= 2 n2 for all n>10
(What is f(n) and what is g(n)????)
(What is c ???? & What is N???)

so we can say that T(n) is O(n2) OR

T(n) is in the order of n2.

T(n) is bounded above by a + real multiple of n2

What does it mean???? An example…

• For three constants c1, c2, c3 …(doesn’t

matter what the values are) there will be an n
beyond which the program with complexity
c3n will be faster than the one with
complexity c1n2+c2n
• This value of n is called the break even
point
• If n=0 then c3n will always be faster
 COMPLEXITY

• Space complexity is the amount of memory a

program needs to run to completion
• Time complexity is the amount of computer
time a program needs to run to completion
• If
f(n) = amnm+am-1nm-1+….+a1n+a0
then f(n) = O(nm)
 FACTS ABOUT O(n)

• Gives us means for comparing algorithms

• It tells us about the growth rate of an algorithm
as input size becomes large
• Its also called the asymptotic growth rate or
asymptotic order or simply order of the
function. ASYMPTOTIC NOTATION
 Properties of Big-Oh

• If f(n) is O(g(n)) and g(n) is O(h(n)) then f(n) is O(h(n))

• If f(n) is O(h(n)) and g(n) is O(h(n)) then f(n) + g(n) is O(h(n))
• ank is O(nk) where a is a constant
• nk is O(nk+j ) for any positive j
• If f(n)=cg(n) then f(n) is O(g(n))
• The function logan is O(logbn) for all positive a & b ≠ 1
so we can just write O(log n) (without the base)
 Growth rates

This is O(1)

sum++;
This is O(n)
for (i=0;i<n;++i)
sum++;
This is O(n2)
for (i=0;i<n;++i)
for (j=0;j<n;++j)
sum++;
 Growth rates
Which growth rate is better???
• Add n!, nn , 2n to the
graph and see where
they fit
Here we have:
• Constant
• Linear
• Logarithmic
• Quadratic
• cubic

This is certainly what I

would like my program
to be!!! (but this is just
wishful thinking)

Graph from Adam Drozdek’s book

 Comparison of Growth Rates
N log2N N log2N N2 N3 2N

1 0 0 1 1 2

2 1 2 4 8 4

8 3 24 64 512 256

64 6 384 4096 262,144 About 5 years

128 7 896 16,384 2,097,152 Approx 6 billion years,

600,000 times more than
age of univ.
(If one operation takes 10-11 seconds)
REF: Nell Dale
 EXAMPLE OF O(log n)

sum = 0; i sum
for (i=1;i<n;i=i*2) 1 1
sum = sum+1; 2 1+1=2
4 2+1=3
8 3+1=4
…
n/2 log2n
n “exit loop”…
sum not incremented
loop will not execute for i=n further

The value of sum is log2n at the end of the loop

EXAMPLE OF O(log n)…suppose you are coloring the table

take half

discarded take half

discarded take half

take half

take half
You didn’t color all cells…..everytime you discarded half the cells
EXAMPLE OF O(n)…suppose you are coloring the table

You will color all the cells one by one

take half
EXAMPLE OF O(n2)…suppose you are coloring the table

You will color each cell n times

take half
 EXAMPLE OF O(2n)…suppose you are coloring the table

The number of times you color a cell is written there

1 2 4 8 16 32 64 128 256 512 1024 2048

 Math: Arithmetic series
• Arithmetic series notation (useful for analyzing
runtime of loops): k

Expr
i j
– the sum of all values of Expr with each value of i between j--k
• Example: 4

2i 1
i0

= (2(0) + 1) + (2(1) + 1) + (2(2) + 1) + (2(3) + 1) + (2(4)

+ 1)
=1+3+5+7+9
= 25
 Arithmetic series identities
• sum from 1 through N inclusive:
N  N  1
 
N


i 1
i 
2
 O N 2

• Intuition:
– sum = 1 + 2 + 3 + ... + (N-2) + (N-1) + N
– sum = (1 + N) + (2 + N-1) + (3 + N-2) + ... // rearranged
// N/2 pairs total
• sum of squares:

N  N  1 2 N  1
 
N


i 1
2
i 
6
 O N 3
 Series runtime examples
• What is the exact runtime and complexity class
(Big-Oh)?
int sum = 0;
for (int i = 1; i <= N; i++) {
for (int j = 1; j <= N * 2; j++) {
sum++;
}
}
– Runtime = N · 2N = O(N2).

int sum = 0;
for (int i = 1; i <= N; i++) {
for (int j = 1; j <= i; j++) {
sum++;
}
}
 SOME MATH
Normally these formulas are very handy :
If x y  z then y  log x z

n
n
 ai 
i 1 2
(a1  a n )
n
x  (n  1) x ( n 1)  nx ( n  2 )

k 0
kx 
k

( x  1 ) 2
n
n
i 
i 1 2
(n  1)

n n(n  1)( 2n  1)
i2  ( for n  1)
i 1 6
m 1
m
1  r

k 0
r k

1 r
 SOME RULES
Rule 1
for (i=0;i<n;i=i+k) Anything inside the loop will run approximately n/k times

Rule 2
for (i=n;i>0;i=i-k) Anything inside the loop will run approximately n/k times

Rule 3
for (i=1;i<n;i=i*k) Anything inside the loop will run approximately logkn times

Rule 4
If the loop variables are independent then the total times a statement inside a
nested loop is executed is equal to the product of the times the individual loops
run
e.g. for (i=0;i<n;++i)
for (j=0;j<m;++j)
A statement inside the above nested loop will run n*m times
 Some rules…(contd.)
Rule 5
for(i=1;i<=n;++i)
for (j=1;j<=i;++j)
The above nested loop approximately runs (1/2)n(n+1) times.
The variable j depends upon the value of i

Rule 6
for(i=1;i<=n;i=i*2)
for (j=1;j<=i;++j)
The statements in the above nested loop approximately run 2n-1 times.

Rule 7
for(i=0;i<n;i=i+1)
for (j=i;j<n;++j)
The statements in the above nested loop approximately run n(n+1)/2
times.