C Programming

Efficiency of Programs
Abstract Machine Model

Machine Model

Instructions are executed sequentially.

Impractical to define instructions of each machine, and their
corresponding costs.
Therefore, a set of commonly found instructions in a computer are
assumed:
Arithmetic: add, multiply, divide,
Data movement: load, store, copy
Control: jumps, calls to functions, return.
Assume each instruction takes one unit of time.

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 1 / 11

C Programming
Efficiency of Programs
Abstract Machine Model

Machine Model

No distinction of operations on integers and floating point types.

Precision of values is not considered.
Each integers require c log n bits, where c ≥ 1.
Since, c is a constant, word size is limited.

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 2 / 11

C Programming
Efficiency of Programs
Running Time

Notion of Running Time

Sorting time 1000 elements > time for sorting 3 elements. input size.
A given algorithm takes different amount of times for same
Data shifting is not required in insertion sort for a sorted sequence.
But required for a reverse sorted sequence.

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 3 / 11

C Programming
Efficiency of Programs
Running Time

Example

f o r ( i = 0 ; i < n ; i ++)
sum += a [ i ] ;

Example
Description Times executed
Initialization step 1
Comparison step n+1
Addition and assignment steps together 2n
Increment step n
Total 4n + 2.

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 4 / 11

C Programming
Efficiency of Programs
Running Time

Example

f o r ( i = 0 ; i < n ; i ++)
f o r ( j = 0 ; j < n ; j ++)
sum += b [ i ] [ j ] ;

Example
Description Times executed
Initialization 1 + n (1 for i, n for j)
Comparison step (n + 1) + n(n + 1)
Addition and assignment steps together 2n × n
Increment step for first loop n
Increment step for second loop n2
Total 4n2 + 4n + 2

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 5 / 11

C Programming
Efficiency of Programs
Running Time

Example

f o r ( i = 0 ; i < n ; i ++)
f o r ( j = i + 1 ; j < n ; j ++)
if (a [ i ] < a [ j ])
swap ( a [ i ] , a [ j ] ) ;

Example
The input has n element in array, so, we know that above code will
result in:
n − 1 comparisons for a[0]
n − 2 comparisons for a[1], and so on.
In general, n − i − 1 comparisons for a[i]
Pn−1
Thus
Pn−1total number of comparisons = i=0 (n − i − 1) or
i=1 i = n(n − 1)/2

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 6 / 11

C Programming
Efficiency of Programs
Running Time

Efficiency of Algorithms

Comparing Algorithms
Suppose we have two algorithms A and B
A takes time 5000n and B takes 1.1n time
For n = 10, A requires 50000 steps and B takes 5500 steps
But for n = 1000, A requires 5,000,000 steps and B takes 2.5×1041
steps.
B can not be used for large inputs whereas A is ok.

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 7 / 11

C Programming
Efficiency of Programs
Running Time

Efficiency of Algorithms

Comparing Algorithms

Input Program A Program B

n 5000n 1.1n
10 50000 5500
100 500,000 13,781
1000 5,000,000 2.5 ×1041
1,000,000 5.109 4.8.1041392

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 7 / 11

C Programming
Efficiency of Programs
Running Time

Efficiency of Algorithms

Speed of Machine

Function Size of Large Problem Instance in 1 hour

With M1 With M2 With M3
n N1 100N1 1000N1
n2 N2 10 N2 31.6 N2
n3 N3 4.64 N2 10 N2
2n N4 N4 + 6.64 N4 + 9.97
3n N5 N5 + 4.19 N5 + 6.29

For 2n case, in 1hr = N 4 with present computer

Since 100×2N 4 = 2N x , N x = N 4 + (log 100/ log 2) = N 4 + 6.64

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 7 / 11

C Programming
Efficiency of Programs
Big Oh Notation

Complexity
Growth Rate and Big O
Important measure of efficiency of a program is how the number of
steps (time) grows as the input grows.
Growth function is defined by big-O notation.

Definition (Big-O)
Let g : R → R be a function.
A function f : R → R is big-O of g(n), if there exist positive
constants c > 0, and n0 ∈ N such that
f (n) ≤ cg(n), for all n ≥ n0

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 8 / 11

C Programming
Efficiency of Programs
Big Oh Notation

Complexity
Big O
f (n) is bounded above by g(n) from some point onwards.
g(n) is formulated as a simpler function.
g(n) exhibits same trend in growth as f (n).
Since we are interested for large n, it is alright if f (n) ≤ cg(n) for
n > n0 .

Example (4)
f (n) = n2 + 2n + 1 ≤ n2 + 2n2 , if n ≥ 2
= 3n2
Therefore, for c = 3 and n0 = 2, f (n) ≤ cn2 , whenever n ≥ n0 .

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 8 / 11

C Programming
Efficiency of Programs
Big Oh Notation

Complexity
Big O
If f (n) is O(n2 ), is it also O(n3 )?
Since O(n3 ) grows faster than O(n2 ), it is true.
So, our attempt will be to find smallest simple function for which
f (n) is O(g(n)).
Well Known Growth Functions
1, log n, n, n log n, n2 , n3 , 2n , etc.

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 8 / 11

C Programming
Efficiency of Programs
Big Oh Notation

Complexity

Definition (Big Omega)

Ω(g(n)) = {f (n)| there exists positive constants c and n0

such that 0 ≤ cg(n) ≤ f (n) for all n ≥ n0 }

Example (5)
Prove f (n) = n3 + 20n is Ω(n2 )
We must find c > 0, and n0 > 0 such that n3 + 20n ≥ c.n2
√
I.e., c ≤ n + 20
n . RHS is minimum, when n = 20
Therefore, with n0 = 5 and c ≤ 9 f (n) ≥ c.n2 for n ≥ n0 .
Note this is same as saying n2 is O(n3 + 20n).

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 8 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Determining Big-O
Dominance Functions
Find dominant term of the function and find its order.
Any exponential function dominates any polynomial function.
Any polynomial function dominates any logarithmic function.
Any logarithmic function dominates any constant.
Any polynomial of degree k dominates lower degree polynomial
Since dominant term grows more rapid compared to others, it will
quickly outgrow non-dominant terms.

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 9 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Determining Big-O
Other Simple Rules
If T1 (n) = O(f1 (n)), and T2 (n) = O(f2 (n)), then
T1 (n) + T2 (n) = max{ O(f1 (n)), O(f2 (n))}
T1 (n) * T2 (n) = O(f1 (n) * f2 (n))
If T(n) is a polynomial of k then T(n) = Θ(nk )
logk n = O(n) for any constant k
For checking whether g(n) and f (n) are comparable find
lim fg(n)
(n)
→ 1?
n2 2n
Eg: lim n2 +6
= lim 2n = 1
log n (1/n)
lim log n2
= lim 2(1/n) = 1/2.

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 9 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Determining Big-O

Example (6)
Examples of O(n2 ) functions: n2 , n2 + n, n2 + 1000n, 100n2 + 1000n, n,
n/100, n1.99999 , n2 /(log log log n)

Example (7)
log n! = log 1 + log 2 + . . . + log n
≤ log n + log n + . . . + log n = n log n

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 9 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Determining Big-O

Example (8)
2n+1 = 2.2n for all n, so with c = 2, n0 = 1.
So, 2n+1 = O(2n ).
n
But 22 6= O(2n ).
This can be proved it by contradiction.
n
Suppose 0 ≤ 22 = 2n .2n ≤ c.2n , then 2n ≤ c.
But no constant is greater than 2n .

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 9 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Determining Big-O

Example (9)
Prove that n3 + 20n + 1 is not O(n2 ).
By definition we should have n3 + 20n + 1 ≤ c.n2 .
20 1
So n + n + n ≤ c.
Since left side grows with n, c can not be a constant.

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 9 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Determining Big-O

Example (10)
n2 +5 log n
f (n) = 2n+1 is O(n)
5 log n < 5n < 5n2 , for all n > 1
1 1
2n + 1 > 2n, so 2n+1 < 2n for all n > 0
n2 +5 log n n2 +5n2
Thus 2n+1 ≤ 2n = 3n for all n > 1.
So, with c = 3 and n0 = 1 we have f (n) < c.n

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 9 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Determining Big-O

Example (11)
Let f (n) = nk , and m > k, then we can show f (n) = O(nm− ), where
>0
Set  = (m − k)/2, so m −  = (m + k)/2 > k.
Hence, n(m−) dominates nk .

Example (12)
Let f (n) = nk , and m < k, then we can show f (n) = Ω(nm+ ), where
>0
Set  = (k − m)/2, so m +  = (m + k)/2 < k.
Hence, n(m+) is dominated by nk .

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 9 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Determining Big-O

Example (13)
Show f (n) = nk is of O(nlog log n ) for any k > 0
k
nk < nlog log n iff k < log log n, i.e., n > 22 .
k
Setting n0 = 22 , we have nk = O(nlog log n ).

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 9 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Guidelines for Computing Big O

Computing Steps
Loops: for, while and do-while
Number of operations is equal to number of iterations times the
operations in each statement inside loop.
Nested loops:
Number of statements in all loops times the product of the loop sizes.
Consecutive statements:
Use addition rule: O(f (n)) + O(g(n)) = max(g(n), f (n))
If/else and if/else if statement:
Number of operation is equal to running time of conditional evaluation
and maximum of running time of if and else clauses.

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 10 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Guidelines for Computing Big O

Computing Steps
switch statements:
Take the complexity of the most expensive case (with the highest
number of operations).
Methods call:
First, evaluate the complexity of the method being called.
Recursive methods:

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 10 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Guidelines for Computing Big O

Example (14)
1 for ( i = 0; i < n; i ++)
2 a [ i ] = 0;
3 for ( i = 0; i < n ; i ++)
4 for ( j = 0; j < n ; j ++) {
5 sum = i + j;
6 s i z e ++;
7 }

First for loop: n times

Nested for loops: n2 times
Total: O(n + n2 ) = O(n2 )

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 10 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Guidelines for Computing Big O

Example (15)
1 char key ;
2 int X[5];
3 int Y[ 5 ] [ 5 ] ;
4 int i , j ;
5 .........
6 switch ( key ) {
7 case ’ a ’ :
8 f o r ( i = 0 ; i < s i z e o f (X) / s i z e o f (X [ 0 ] ) ; i ++)
9 sum = sum + X [ i ] ;
10 break ;
11 case ’ b ’ :
12 f o r ( i = 0 ; i < s i z e o f (Y) / s i z e o f (Y [ 0 ] ) ; i ++)
13 f o r ( j = 0 ; j < s i z e o f (Y [ 0 ] ) / s i z e o f (Y [ 0 ] [ 0 ] ) ; j ++)
14 sum = sum + Y [ i ] [ j ] ;
15 break ;
16 } // End o f s w i t c h b l o c k

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 10 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Guidelines for Computing Big O

Example (15 contd.)

Case ’a’: O(n)
Case ’b’: O(n2 )
So using switch statement rule: O(n2 )

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 10 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Guidelines for Computing Big O

Example (16)
1 char key ;
2 int A[5][5] , B[5][5] , C[ 5 ] [ 5 ] ;
3 ........
4 i f ( k e y == ’+ ’ ) {
5 f o r ( i = 0 ; i < n ; i ++)
6 f o r ( j = 0 ; j < n ; j ++)
7 C[ i ] [ j ] = A[ i ] [ j ] + B[ i ] [ j ] ;
8 } // End o f i f b l o c k ===> O( n ˆ 2 )
9 e l s e i f ( k e y == ’ x ’ )
10 C = m a t r i x M u l t (A , B ) ; ===> O( n ˆ 3 )
11 else
12 p r i n t f ( ” E r r o r ! E n t e r ’+ ’ o r ’ x ’ ! : ” ) ; ===> O( 1 )

Complexity is thus, O(n3 ).

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 10 / 11

C Programming
Efficiency of Programs
Tricks for Computing Big-O

Exponential Algorithm are Expensive

Let us first prove nk = O(bn ) whenever 0 < k ≤ c,

nk knk−1
lim = lim
bn ln b.bn

The numerator’s exponent decremented after each application of L

Hospital’s rule.
So, bn dominates nk for any finite k.

R. K. Ghosh (IIT-Kanpur) C Programming November 15, 2017 11 / 11