CSC1108

Data Structures & Algorithms

Lecture 3: Analysis of Algorithms

The lecture materials are adapted from ICT008 Data Structures & Algorithms © Singapore Institute of Technology (SIT) 1
Agenda
• Mathematics for Algorithms (review)
• Why analyse algorithms?
• Empirical analysis
• Mathematical models
• Asymptotic rules
• Theory of algorithms
• General plan for Algorithm analysis

2
Recommended Reading
• Algorithms by Robert Sedgewick and Kevin Wayne. Addison-
Wesley Professional. 4th Edition, 2011 – Chapter 1.4

• Runestone Interactive book: “Problem Solving with Algorithms

and Data Structures Using Python” – Section: “Analysis”

3
Mathematics for Algorithms
• Polynomials
• Combinations
• Logarithms
• Summation of Series

4
Polynomials

5
Combinations

6
Logarithm - Definition

7
Law of Logarithm

8
Law of Logarithm

9
Summation of Series
• Arithmetic Series:

10
Summation of Series
• Geometric Series

11
Analysis of Algorithms
• To analyze an algorithm is to determine the amount of
resources (such as time and storage) necessary to execute it.

12
Analysis of Algorithms
• There are two ways to analyze an algorithm

• Empirical / Experimental studies

• Theoretical analyses

13
Empirical Study Method

14
Problems with Empirical Method
• System dependent effects:

• Hardware: CPU, memory, cache

• Software: compiler, interpreter, garbage collector, …
• System: operating system, network, other applications, …

15
Theoretical Analysis
• To analyse the running time of algorithms, use a simple model
of the underlying computer
• Aim: make simplifications to estimate resources to execute an
algorithm
• Exact time, machine instructions for different machines are not
relevant here
• We use a model of a Random Access Machine (RAM)
• Primary challenge: determine the frequency of execution
statements

16
RAM Model
• Consider sequential 1-Processor architecture, no parallelism.
• All data are directly accessible in memory.
• All memory accesses take the same length.
• All elementary operations require constant time.
• Elementary operations are:
• Value assignment;
• Arithmetic operations such as addition, subtraction, multiplication;
• Logical operation such as “AND”, “OR”;
• Comparison operations such as “<”, “>”;
• Commands to control the flow of instructions, such as “IF THEN ELSE”.
• For simplicity, assume each elementary operation takes one time
unit.

17
Example: 1-SUM

18
Example: 2-SUM

!
2N2 + 2 to $ 5$ − 1 + 2
"
19
Example 2-SUM

!
2N2 + 2 to $ 5$ − 1 + 2 20
"
Asymptotic Notation: Comparing
Algorithms
• Consider two algorithms, A and B, for solving a given problem.
• Let the running times of the algorithms be Ta (n) and Tb(n) for
problem size n.
• Suppose the problem size is n0 and

21
Comparing Algorithms

22
Comparing Algorithms
For algorithm analysis,
We emphasize on the operation count’s
Order of growth for large input sizes

23
Asymptotic Notations
In comparing algorithms, consider the
asymptotic behaviour of the two algorithms
for large problem sizes, under worst-case.

24
Big-O Notation
(the “O” stands for “order-of”)

25
Big-O Notation

26
Big-O Notation

27
Big-O Notation

28
Big-O Example: 1-SUM

29
Big-O Example: 2-SUM

1
* 5* − 1 + 2 ≤ kn2 for all n ≥ n0
2
i.e. 5n2 – n + 4 ≤ 2kn2.
Can we find k (> 0) and n0?
We have:
For all n ≥ 1,
Þ 5n2 – n + 4 ≤ 5n2 – n2 + 4n2
Þ 5n2 – n + 4 ≤ 8n2
1 Compare 5n2 – n + 4 ≤ 2kn2
* 5* − 1 + 2 Þ 2k = 8
2
Þ pick k = 4 & n0 is 1, gives:
!
1 * 5* − 1 + 2 ≤ 4n2 for all n ≥ 1
"
* 5* − 1 + 2
2
31
Big-O Rules
If f(n) is a polynomial of degree d, then f(n) = O(nd).
• Drop lower-order terms
• Drop constant factors

Example:

32
Big-O and Growth Rate
The big-O notation gives an upper bound on
the growth rate of a function

33
Common order-of-growth

34
Types of order of growth

35
Common order-of-growth

36
Common Order-of-Growth

37
Complexity Classes
• O(1) denotes constant running time
• O(log n) denotes logarithmic running time
• O(n) denotes linear running time
• O(n log n) denotes log linear running time
• O(nc) denotes polynomial running time (c is a constant)
• O(cn) denotes exponential running time (c is a constant being
raised to a power based on size of input)

38
Growth Rate: Practical Implication

39
Complexity Growth

40
Complexity Classes order by
Low to High

41
Types of Analyses
• Worst case. Upper bound on cost,
• Determined by “most difficult” input
• Provides a guarantee for all inputs.

• Best case. Lower bound on cost.

• Determined by “easiest” input.
• Provides a goal for all inputs.

• Average case. Expected cost for random input.

• Needs a model for “random” input.
• Provides a way to predict performance.
42
Types of Analyses
• Suppose you are given a list L of some length len(L)
• Best case. Minimum running time over all possible input of a given
size len(L)
• Constant for search_for_element
• First element of the list
• Average case. Average running time over all possible inputs of a
give size len(L)
• Practical measure
• Worst case. Maximum running time over all possible input of a given
size len(L)
• Linear in length of list for search_for_element
• Must search entire list and not find it.

43
Theory of Algorithms
• Upper bound.
• Performance guarantee of algorithm for any input.
• My code takes at most this long to run

• Lower bound.
• Proof that no algorithm can do better.
• My code takes at least this long run

• Optimal algorithm.
• Lower bound = upper bound (to within a constant factor)
• My code takes “exactly”* this long to run
• *Except for constant factors and lower order terms

44
Big-O Notation upper bound

45
Big-O Notation upper bound

46
Big-Omega Notation lower bound

47
Big-Theta Notation optimal bound

48
Properties of Asymptotic

49
Properties of Asymptotic

50
General Plan for Algo Runtime
Analysis
• Decide on parameter n indicating input size.
• Identify algorithm’s basic operation – cost model.
• Set up a sum expressing the number of times the basic
operation is executed.
• Simplify the sum using standard formulas and rules to
determine big-O of the running time.

51
Example 1: O(n)
• Provides a Big-O notation (means an upper bound or a worst
case analysis) for the run-time of the following algorithm

52
Example 1: O(n)

53
Example 2: O( lg(n) )

54
Example 3: O( n2 )

55
Example 4: O( n3 )

56
Example 5: O( n2 )

57
Example 6: O( 2n )

58
Complexity of Common Python
Functions

59