Introduction to Data

Structures
LECTURE 2: TIME COMPLEXITY
Lecture 1
1. Basic terminology of Data Structures
2. Need of Data Structures
3. Advantages of Data Structures
4. Data Structures Classification
4. Algorithm
5. Analysis of Algorithms
7. How to Compare Algorithms
8. Pseudocode
Algorithm
An algorithm is a finite set of instructions which, if followed, accomplish a particular task. In
addition every algorithm must satisfy the following criteria
(i) input: there are zero or more quantities which are externally supplied
(ii) output: at least one quantity is produced;
(iii) definiteness: each instruction must be clear and unambiguous;
(iv) finiteness: if we trace out the instructions of an algorithm, then for all cases the algorithm will
terminate after a finite number of steps
(v) effectiveness: every instruction must be sufficiently basic that it can in principle be carried out
by a person using only pencil and paper. It is not enough that each operation be definite as in (iii),
but it must also be feasible.
 Program Running Time
When is the running time (waiting time for user) noticeable/important?
•Web search
•Database search
•Real-time systems with time constraints
• Withdrawing money from ATM
• Decision taken by an Autonomous Vehicle

Source: Dr. Maunendra, Dr. Zahoor jan, Dr. Jeyakesavan Veerasamy, Jeremy Johnson
Analysis of Algorithms: How?
•Run the program
•Measure the time

•Is it the right thing to do?

•Same algorithm, same input
–Different running times are possible.
–Why?
Factors that determine running time of a
program
•Problem size: n
•Basic algorithm / actual processing
•Memory access speed
•CPU/processor speed
•# of processors?
•Compiler/linker optimization?
Running time of a program or
transaction processing time
•Amount of input: n →min. linear increase*
•Basic algorithm / actual processing →depends on algorithm!
•Memory access speed →by a factor
•CPU/processor speed →by a factor
•# of processors?→yes, if multi-threading or multiple processes are used.
•Compiler/linker optimization? →~20%
Time Complexity
•Measure of algorithm efficiency
•Ignore hardware and environment
–E.g. processor details, threading etc.
•Focus on input size
•Has a big impact on running time.
Experimental Studies
• Write a program implementing the algorithm
•Run the program with inputs of varying size and composition
•Use a function, like the built-in clock()function, to get an
accurate measure of the actual running time

Time
•Plot the results
Limitations of Experiments
•It is necessary to implement the algorithm, which may be difficult
•Results may not be indicative of the running time on other inputs not included in the experiment.
•In order to compare two algorithms, the same hardware and software environments must be used
Theoretical Analysis
•Uses a high-level description of the algorithm instead of an implementation
•Characterizes running time as a function of the input size, n.
•Takes into account all possible inputs
•Allows us to evaluate the speed of an algorithm independent of the hardware/software environment
Simple Example
// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A
int Sum(int A[], int N){
int s=0;
for (int i=0; i< N; i++)
s = s + A[i];
return s;
}
Simple Example
Simple Example: Growth of 5n+3
Estimated running time for different values of N:
N = 10=> 53 steps
N = 100=> 503 steps
N = 1,000=> 5003 steps
N = 1,000,000=> 5,000,003 steps
As N grows, the number of steps grow in linear proportion to N for this function “Sum”
What Dominates in Previous Example?
What about the +3 and 5 in 5N+3?
–As N gets large, the +3 becomes insignificant
–5 is inaccurate, as different operations require varying amounts of time and also does not have any
significant importance

What is fundamental is that the time is linear in N.

Asymptotic Complexity
•The 5N+3 time bound is said to "grow asymptotically" like N
•This gives us an approximation of the complexity of the algorithm
•Ignores lots of (machine dependent) details, concentrate on the bigger picture
Coding example 1
for ( i=0 ; i<n ; i++ )
m += i;

f(n) = 4n+1
Coding example #2
for ( i=0 ; i<n ; i++ )
for( j=0 ; j<n ; j++ )
sum[i] += entry[i][j];

f(n) = 𝟔𝒏𝟐 + 𝟑𝒏 + 𝟏
 What is Rate of Growth?
•The rate at which the running time increases as a function of input is called rate of growth.
•Let us assume that you go to a shop to buy a car and a bicycle. If your friend sees you there and asks what
you are buying, then in general you say buying a car. This is because the cost of the car is high compared to
the cost of the bicycle (approximating the cost of the bicycle to the cost of the car).
Total cost = cost of car + cost of bicycle
Total cost ≈ cost of car (approximation)
•For the above-mentioned example, we can represent the cost of the car and the cost of the bicycle in
terms of function, and for a given function ignore the low order terms that are relatively insignificant (for
large value of input size, n).
•As an example, in the case below, n4, 2n2, 100n and 500 are the individual costs of some function and
approximate to n4 since n4 is the highest rate of growth.
n4 + n2 +100n + 500 ≈ n4
Commonly Used Rates of Growth
Types of Analysis
To analyze the given algorithm, we need to know with which inputs the algorithm takes less time
(performing well) and with which inputs the algorithm takes a long time.
We have already seen that an algorithm can be represented in the form of an expression.
That means we represent the algorithm with multiple expressions: one for the case where it takes less time
and another for the case where it takes more time.
In general, the first case is called the best case and the second case is called the worst case for the
algorithm.
To analyze an algorithm we need some kind of syntax, and that forms the base for asymptotic
analysis/notation
There are three types of analysis:
• Worst case
○ Defines the input for which the algorithm takes a long time (slowest time to complete).
○ Input is the one for which the algorithm runs the slowest.
• Best case
○ Defines the input for which the algorithm takes the least time (fastest time to complete).
○ Input is the one for which the algorithm runs the fastest.
• Average case
○ Provides a prediction about the running time of the algorithm.
○ Run the algorithm many times, using many different inputs that come from some distribution that generates these
inputs, compute the total running time (by adding the individual times), and divide by the number of trials.
○ Assumes that the input is random.
Asymptotic Notation
•Having the expressions for the best, average and worst cases, for all three cases we need to
identify the upper and lower bounds.
•To represent these upper and lower bounds, we need some kind of syntax, and that is the
subject of the following discussion.
•Let us assume that the given algorithm is represented in the form of function f(n).
Big-O Notation [Upper Bounding
Function]
•This notation gives the tight upper bound of the given function.
•Generally, it is represented as f(n) = O(g(n)). That means, at larger values of n, the upper bound
of f(n) is g(n).
•For example, if f(n) = 𝑛4 + 100𝑛2 + 10𝑛 + 50 is the given algorithm, then 𝑛4 is g(n). That
means g(n) gives the maximum rate of growth for f(n) at larger values of n.
Big-O Notation [Upper Bounding
Function]
Let us see the O–notation with a little more detail.
Definition: O–notation defined as O(g(n)) = {f(n): there exist positive constants c and 𝑛0 such that 0 ≤
f(n) ≤ cg(n) for all n > 𝑛0 }. g(n) is an asymptotic tight upper bound for f(n).
•Our objective is to give the smallest rate of growth g(n) which is greater than or equal to the given
algorithms’ rate of growth /(n).
•Generally we discard lower values of n. That means the rate of growth at lower values of n is not
important.
•In the figure, 𝑛0 is the point from which we need to consider the rate of growth for a given algorithm.
Below 𝑛0 , the rate of growth could be different. 𝑛0 is called threshold for the given function
Big-O Visualization
O(g(n)) is the set of functions with smaller or the same order of growth as g(n). For example;
O(𝑛2 ) includes O(1), O(n), O(nlogn), etc.
Note: Analyze the algorithms at larger values of n only. What this means is, below 𝑛0 we do not
care about the rate of growth.
Example for linear function
•When we say that the function 4n+1 is Big O(n), we are saying that the asymptotic behavior of a
linear function is equal to or greater than its own.
•This means that the 4n+1 function will never grow to the point where it surpasses linear
behavior.
•Thus, Big O(n) presents itself as an upper limit, and we know that the function will never exhibit
growth behavior that exceeds this upper limit.

https://algol.dev/en/big-o-notation/
Example: for function 4n+1, there is another function of linear behavior that limits it superiorly. Note in the
graph below that, for values of n>1, the function 4n+1 is surpassed by another function of class n.
Similarly, it is also correct to say that 4n+1 is Big O(n2), because the 4n+1 function will never grow to the point where
it exceeds quadratic behavior. Thus, O(n2) also presents itself as an upper asymptotic limit:
Now it would be wrong to say that 4n+1 is Big O(1), because the 4n+1 function will always grow to the point
where it surpasses constant behavior. Thus, Big O(1) is not presented as an upper asymptotic limit:
Summing up:
4n+1 is Big O(n2), as it can be overcome (superiorly limited) by a quadratic function.
4n+1 is Big O(n), as it can be overcome (superiorly limited) by a linear function.
4n+1 is not Big O(1), as it cannot be surpassed (superiorly limited) by a constant function.
Example for quadratic function
•When we say that the function 5n2-n+1 is Big O(n2), we are saying that the asymptotic behavior
of a quadratic function is equal to or greater than its own.
•This means that the function 5n2-n+1 will never grow to the point where it exceeds quadratic
behavior.
•Thus, O(n2) is presented as an upper limit and we know that the function will never exhibit
growth behavior that exceeds this upper limit.
Example: For the 5n2-n+1 function, there is another quadratic behavior function that limits it superiorly. See
below:
Similarly, it is also correct to say that 5n2-n+1 is Big O(n3​), because the function 5n2-n+1 will never grow to the
point where it exceeds cubic behavior. Thus, O(n3) also presents itself as an upper asymptotic limit:
Now it would be wrong to say that 5n2-n+1 is Big O(n), because the function 5n2-n+1 will always grow to the point
where it surpasses linear behavior. Thus, O(n) does not present itself as an upper asymptotic limit:
First example of mathematical use of Big
O notation
•Let us try to demonstrate by definition that 4n+1=O(n) (reads “It’s Big O of n”).
•By definition, all we have to do is demonstrate that there is a positive constant c and a positive
initial value n0 so that for all values of n greater than or equal to n0, 4n+1 is always less than or
equal to c multiplied by n.
•In summary: we need to prove that 4n+1 ≤ c.n, for a positive c constant and a positive
initial n0 value.
•To do so, the first decision we need to make is to choose a value for the constant c. In this case,
let’s consider c equal to the value 5. Note below a table showing the inequality results
for c=5 and for some values of n:
Note that for n equal to 0 (zero) the function of algorithm 4n+1 is greater than c.n. But when n equals 1, the
values of c.n begin to outperform the 4n+1 algorithm. When we plot a graph for both functions, we see more
clearly this exact moment of overrun of the c.n function:
•This means that O(n) represents an upper asymptotic limit for the 4n+1 function. For from a
given value n, c.n surpasses the function.
•However, what really matters for algorithm analysis is that we can demonstrate our goal, namely
that there is a constant c=5 and an n0=1, such that 4n+1 ≤ c.n for all n values greater than or
equal to n0.
•Thus, it is proved that, in fact, 4n+1=O(n) (is Big O of n).
Second example of the mathematical use
of Big O notation
•Let us try to demonstrate by definition that 5n2-n+1=O(n2) (reads “It’s Big O from n squared”).
•By definition, all we have to do is demonstrate that there is a positive constant c and a positive
initial value n0 so that for all n values greater than or equal to n0​, 5n2-n+1 is always less than or
equal to c multiplied by n2.
•In summary, we need to prove that 5n2-n+1 ≤ c.n2, for a constant c and an initial n0 value.
•Again, we need to choose a value for the constant c. In this case, we will also consider c equal to
the value 5.
•Note below a table showing the inequality results for c=5 and for some values of n:
Note that for n equal to 0 (zero) the algorithm function 5n2-n+1 is greater than c.n2. But when n equals 1 the values
of c.n2 begin to outperform the algorithm 5n2-n+1. When we plot a graph for both functions, we see more clearly this
exact moment of overrun of the c.n2 function:
•This means that O(n2) represents an upper asymptotic limit for the 5n2-n+1 function.
•For from a given value n, c.n2 surpasses the function. However, what really matters for algorithm
analysis is that we can demonstrate our goal, that there is a constant c=5 and n0=1, such
that 5n2-n+1 ≤ c.n2 for all n values greater than or equal to n0.
•Thus, it is proved that, in fact, 5n2-n+1=O(n2) (is Big O from n squared).
Third example: a failed attempt
•Let’s try to demonstrate by definition that 2n2-n=O(n).
•By definition, all we have to do is demonstrate that there is a positive constant c and an initial
value n0 so that for all values greater than n0, 2n2-n is always less than or equal to c multiplied
by n.
•In summary, we need to prove that 2n2-n≤c.n, for a constant c and an initial n0 value
•At first glance, it is clear that 2n2-n has quadratic behavior and c.n has linear behavior.
•So we are tempted to claim that this demonstration is false, since a linear function (c.n) cannot
represent an upper asymptotic limit for a quadratic function (2n2-n).
•But let’s take a closer look to see why this is impossible.
Let’s perform some algebraic transformations in order to isolate the constant c in inequality and
understand its possible values:
•The resulting inequality already shows us that it is impossible for any constant c to make the
expression valid for large values of n (n→∞), because no matter how large the value of c is, the
value of n will always exceed it.
•Below are some graphs illustrating some scenarios of c.n being overcome by the 2n2-n quadratic
function:
Below are some graphs illustrating some scenarios of c.n being overcome by the 2n2-n quadratic function:
•That is, no matter how large the constant c. The 2n2-n function will always outperform the c.n.
•Therefore, we cannot demonstrate our goal. There is no constant c and n0, such that 2n2-n ≤
c.n for all n values greater than or equal to n0. Thus, it is proved that, in fact, 2n2-n≠O(n) (reads
“it’s not Big O of n”).
Big-O Examples
Example-1 Find upper bound for f(n) = 3n + 8
Solution: 3n + 8 ≤ 4n, for all n ≥ 8
∴ 3n + 8 = O(n) with c = 4 and 𝑛0 = 8
Example-2 Find upper bound for f(n) = 𝑛2 + 1
Solution: 𝑛2 + 1 ≤ 2𝑛2 , for all n ≥ 1
∴ 𝑛2 + 1 = O(𝑛2 ) with c = 2 and 𝑛0 = 1
Example-3 Find upper bound for f(n) = 𝑛4 + 100 𝑛2 + 50
Solution: 𝑛4 + 100𝑛2 + 50 ≤ 2𝑛4 , for all n ≥ 11
∴ n4 + 100𝑛2 + 50 = O(𝑛4 ) with c = 2 and 𝑛0 = 11
Example-4 Find upper bound for f(n) = 2𝑛3 – 2𝑛2
Solution: 2𝑛3 – 2𝑛2 ≤ 2𝑛3 , for all n > 1
∴ 2𝑛3 – 2𝑛2 = O(𝑛3 ) with c = 2 and 𝑛0 = 1
Example-5 Find upper bound for f(n) = n
Solution: n ≤ n, for all n ≥ 1
∴ n = O(n) with c = 1 and 𝑛0 = 1
Example-6 Find upper bound for f(n) = 410
Solution: 410 ≤ 410, for all n > 1
∴ 410 = O(1) with c = 1 and 𝑛0 = 1
No Uniqueness?
There is no unique set of values for n0 and c in proving the asymptotic bounds. Let us consider,
100n + 5 = O(n). For this function there are multiple n0 and c values possible.
Solution1: 100n + 5 ≤ 100n + n = 101n ≤ 101n, for all n ≥ 5, n0 = 5 and c = 101 is a solution.
Solution2: 100n + 5 ≤ 100n + 5n = 105n ≤ 105n, for all n > 1, n0 = 1 and c = 105 is also a
solution.
Exercises

1. Try to demonstrate mathematically that n2+800=O(n2).

2. Try to demonstrate mathematically that 2n+10=O(n).
3. Try to demonstrate mathematically that n2=O(n).
4. Try to demonstrate mathematically that 7n-2=O(n).
5. Try to demonstrate mathematically that n2+20n+5=O(n3).
Reference
1. Chapter 3 (Introduction to Algorithms, second edition, T.H.Cormen, C.E.Leiserson,
R.L.Rivest,and C.Stein,PHI Pvt. Ltd./ Pearson Education)
2. Chapter 1 (Data Structures and Algorithms Made Easy: Data Structures and Algorithmic
Puzzles, Narasimha Karumanchi)
3. https://algol.dev/en/big-o-notation/
Thank You