You are on page 1of 58

MH1403 Algorithms and Computing

Lecture 1 Algorithm Analysis

1
Outline
• Algorithm analysis
• Growth rate
• Asymptotic notations
• Big-Oh notation
• Big-Omega notation
• Big-Theta notation
• Properties of the Big-Oh notation
• Algorithm analysis general rules

2
Algorithm

3
Algorithm
• What is algorithm?
• An algorithm is a sequence of computational steps that
transform the input into the output
• Algorithm may be specified
• In English (or other languages)
• or as a computer program
• or as pseudo-code (simply speaking, pseudo code is similar to
computer program, except that pseudo code may not follow the
strict syntax of computer program)

4
Algorithm
• We have been learning algorithms since primary
school
• Simple and important arithmetic algorithms:
• Addition/subtraction of two numbers
• Multiplication/division of two numbers
• Gaussian elimination is an efficient algorithm for solving
a system of linear equations
• Gaussian elimination was invented about 2000 years ago (much
early before Gauss was born)
• ….

5
Algorithm
• Besides the algorithms in math, we use many algorithms
frequently in today’s digital world
• For example, Google search engine is a complicated
search algorithm
• Google makes more than 100 billion USD revenue from
advertisement each year
• The advertisement related to Google search is significant to
Google’s revenue

6
Advanced Algorithms
• Since the invention of electronic computers in the
1940s, many algorithms were developed to solve
problems (in economy, engineering, …) efficiently on
computer
• Nowadays, artificial intelligence is changing our
society dramatically. AI are advanced algorithms.
• Medical diagnosis
• Robotics
• Self-driving
• ….

7
Algorithm Analysis

8
Algorithm Analysis
• Why do we need to perform algorithm analysis ?
• Developing a working algorithm is not good enough
• We need to develop efficient algorithm
• If the program is running on a large data set, the running time
becomes an issue
• To tell whether an algorithm is efficient or not, we need
to perform algorithm analysis

9
Algorithm Analysis
• We only analyze correct algorithms
• An algorithm is correct if for every input instance, it
gives the correct output
• Incorrect algorithms do not give the desired answer

10
Algorithm Analysis
• Analyzing an algorithm is to predict the resources that
the algorithm requires
• Resources include
• Computational time (running time)
• Memory (the amount of data being stored at any particular time
during computation)
• Communication bandwidth (the speed of data being
communicated between processors when multiple processors are
used in an program)
• In this course, we mainly analyze the computational
time of an algorithm
• Note that for some algorithms, the memory cost may be
more critical than the computational time
11
Algorithm Analysis
• We typically analyze the worst-case running time of
an algorithm (sometime the average-case)
• Worst-case running time of an algorithm
• The longest running time for any input of size n
• An upper bound on the running time for any input
 guarantee that the algorithm will never take longer
• Example: Sort a set of numbers in increasing order, and the data is
in decreasing order
• The worst case can occur fairly often
• E.g. in searching a database for a particular piece of information
• Best-case running time
• Example: sort a set of numbers in increasing order, and the data is
already in increasing order
• Average-case running time
12
Algorithm Analysis
• It is difficult to predict the precise
computational time of an algorithm
• Many factors affect the computational time
• computer hardware
• programing language, compiler
• input to the algorithm
• The content of the input affects the running time
• Typically, the input size (number of items in the input) is the main
consideration
• E.g. sorting problem  the number of items to be sorted
• E.g. multiplying two matrices together  the total number of
elements in the two matrices

13
Measure the algorithm running time

• To simplify the analysis of the algorithm running time,


we use machine independent approach by assuming
that every basic operation takes constant time:
Basic operation examples:
Addition, Subtraction, Multiplication, Memory Access
Non-basic operation examples:
Sorting, Searching
• The efficiency of an algorithm is the number of basic
operations it performs (we do not distinguish between
the basic operations)
14
Measure the algorithm running time
• In fact, we will not worry about the exact values,
but will look at the growth rates: how the
running time changes as the input size increases
• For example, let there be n inputs. If an algorithm
needs 2n basic operations and another needs 3n basic
operations, we consider them to be in the same
efficiency category.
• Reason: suppose that n is increased from 100 to 1000, the
2×1000−2×100
growth rate of the 2n-operation algorithm is =
2×100
9, the growth rate of the 3n-operation algorithm is
3×1000−3×100
=9
3×100

• However, we distinguish between exp(n), n, n3, log(n) 15


Asymptotic Notations

16
Asymptotic Notations
• In algorithm analysis, asymptotic notations are
used to identify an algorithm’s growth rate
• In asymptotic notations, we try to establish a relative
order among complexity functions for large input size
• Asymptotic analysis is not perfect for algorithm
analysis, but it is the best way available for
analyzing algorithms

17
Asymptotic Notations
• We will learn the following three asymptotic
notations in this course
• Big-Oh notation (commonly written as O )
• Asymptotically upper bound on the growth rate
• Big-Omega notation (commonly written as )
• Asymptotically lower bound on the growth rate
• Big-Theta notation (commonly written as )
• Asymptotically tight bound on the growth rate
• The following two asymptotic notations will not be
covered in this course (not commonly used)
• Small-oh notation
• Small-omega notation
18
Asymptotic Notations
• Note that in the following slides, we need to be
aware of the meaning of the running time of an
algorithm
• The input size n is always a non-negative integer
• The running time is always non-negative

19
Big-Oh
• Big-Oh notation provides us with an asymptotic
upper bound for the growth rate of the running
time of an algorithm
• Let f(n) be the running time of an algorithm (n is
the input size). g(n) is some time complexity
function we are trying to relate to our algorithm.
• If the growth rate of f(n) is less than or equal to the
growth rate of g(n), i.e., if there are positive
constants c and n0 such that
f(n)  c*g(n) when n  n0 ,
we say that f(n) is O(g(n))
20
Big-Oh

• There are positive c and n0 such that f(n)  c*g(n) when n  n0


• f(n) grows no faster than g(n) for “large” n, i.e., c*g(n) is an upper
bound on f(n) for larger n

21
Big-Oh
• “f(N) is O(g(N))” can be written as “f(N) = O(g(N))”
• The equal sign here does not mean “equal”
• The equal sign here does not mean assignment operation
• f(N) = O(g(N)) is read as “f(N) is big-Oh g(N)”

22
Big-Oh: example
• Let f(n) = 2n2. Then
• f(n) = O(n4)
• Reason: f(n)  n4 when n  2
• f(n) = O(n3)
• Reason: f(n)  n3 when n  2
• f(n) = O(n2) (best answer, asymptotically tight)
• Reason: f(n)  2n2 when n  1

23
Big Oh: more examples
• 1 + 4n = O(n)

• 7n2 + 10n + 3 = O(n2)

• sin(N) = O(1), 10 = O(1), 1010 = O(1)


 i =1
N
i 2
 N  N 2
= O ( N 3
)

24
Big-Omega
• Big-Omega notation provides us with
an asymptotic lower bound for the growth rate of
running time of an algorithm
• f(n) is (g(n)) if there are positive constants c and
n0 such that
f(n)  c g(n) when n  n0

• f(n) is (g(n)) indicates that the growth rate of f(n)


is greater than or equal to the growth rate of g(n)
• “f(n) is (g(n))” can be written as “f(n) = (g(n))”
25
Big-Omega

• There are positive constants c , n0 such that f(n)  c g(n)


when n  n0
• f(n) grows no slower than g(n) for large n
26
Big-Omega: example
• Let f(n) = 2n2. Then
• f(n) = (n)
• f(n) = (n2) (best answer)

27
Big-Theta
• Big-Theta notation provides us with an asymptotic
tight bound for the growth rate of running time of
an algorithm
• f(n) is (g(n)) iff f(n) = O(g(n)) and f(n) = (g(n))
• f(n) is (g(n)) means that the growth rate of f(n)
equals the growth rate of g(n)
• “f(n) is (g(n))” can be written as “f(n) = (g(n))”

28
Big-Theta

• the growth rate of f(n) is the same as the growth rate of g(n)

29
Big-Theta: example
• Example: f(n)=n2 , g(n)=2n2
• f(n) = O(g(n)) and f(n) = (g(n)), thus f(n) = (g(n))

30
Some rule of Big-Theta
• If T(n) is a polynomial of degree k, then
T(n) = (nk).

31
Summary of the Notations
Asymptotically less than or equal to O (Big-Oh)
Asymptotically greater than or equal to  (Big-Omega)
Asymptotically equal to  (Big-Theta)

32
Properties of Big-Oh Notation

33
Tight Big-Oh Notation
• The tight big-Oh notation gives the most accurate
asymptotic bound of the growth rate
• We also expect that the tight big-Oh notation gives the
simplest form of big-Oh notation
• Example:
• n is O(n2), but O(n2) is not the tight big-Oh notation of n
• Tight big-Oh notation is not identical to big-Theta
notation
• For example, the computational complexity of quicksort
is O(n2) and (n log n), but it is not (n2)

34
Why is the Big Oh a Big Deal?
• Big Oh is the most commonly used asymptotic
notation
• It gives us a rough idea of how efficient an algorithm is
• Suppose that there are two algorithms, for input
size n, the running time of Algorithm A is O(n2), the
running time of Algorithm B is O(2n). If both big Oh
notations are tight, it is expected that for large
value n, Algorithm A is much faster than Algorithm
B

35
Properties of the Big-Oh Notation (1)
• Constant factors may be ignored for big-Oh notation:
For all the positive constant k, k*f is O(f).
• For example, a*n2 and b*n2 are both O(n2)
• However, the coefficient a in the exponent of 2a*n
should not ignored
• The big-Oh notation of 23n is O(23n), not O(2n)
• The best big-Oh notation of 20.1n is O(20.1n)

36
Properties of the Big-Oh Notation (2)
• Higher powers of n grow faster than lower powers
• Example: a constant complexity has the lowest grows rate.
a constant is O(1)
• Example: The linear complexity a*n+b grows faster
than a constant complexity, and is O(n)
• Example: The quadratic complexity a*n2+bn+c grows faster
than the linear complexity, and is O(n2)
• Example: n5 grows faster than n4

37
Properties of the Big-Oh Notation (3)

• Exponential functions grow faster than powers:


nk is O(bn ), for all b > 1, k  0,
• Example, n4 grows slower than O(2n ) for large n
n4 grows slower than O(20.1n ) for large n

38
Logarithms
• A logarithm is an inverse exponential function
- Exponential functions grow extremely fast
- Logarithm functions grow extremely slow
• Binary search is an example of an O(log(n))
algorithm
• If you have an algorithm which runs in O(logn)
time, take it because it will be very fast

39
Properties of the Big-Oh Notation (4)

• Logarithms grow more slowly than powers:


logbn is O(nk ) for all b > 1, k > 0

• Example: log2n is O(n0.5 )


The best big-Oh notation of log2n is O(log2n)

40
Properties of the Big-Oh Notation (5)

• Asymptotically, the base of the log does not


matter
log c B
There is a property of logarithm: log A B =
log c A

Example: log2n = (1/log102) x log10n,


1/log102 = 3.322 is a constant,
so log2n is O(log10n)

41
Properties of the Big-Oh Notation (6)

• Asymptotically, any polynomial function of n does


not matter after taking logarithm
Example: log(n475 + n2 + n + 96) = O(log n)
Reason: n475 + n2 + n + 96 = O(n475)
and
log(n475) = 475*log(n)

42
Properties of the Big-Oh Notation (7)
• The growth rate of a sum of terms is the growth rate
of its fastest growing term
• The growth rate of a polynomial is given by the growth rate
of its leading term
• If f is a polynomial of degree d, then f is O(nd)
• Example: a*n3 + b*n2 is O(n3 )
• The growth rate of the sum of a polynomial and exponential
is the growth rate of exponential
• Example: a*n3 + b*2n is O(2n )
• Example: a*n3 + b*2n + c*(3n) is O(3n )
• The growth rate of the sum of a polynomial and logarithm is
the growth rate of polynomial
• Example: a*n + b*log(n) is O(n)
43
Properties of the Big-Oh Notation (8)
• The property in the previous slide needs to be
tweaked if the number of terms grows as n
increases.
• The sum of the first n rth powers grows as the
(r + 1) th power:
1 + 2 + 3 + ……. + n = n(n+1)/2 (arithmetic series)
= O(n2)
1 + 22 + 32 +……+ n2 = n(n + 1)(2n + 1)/6
= O(n3)
………
1 + 2r + 3r +……+ nr = O(nr+1)
44
Properties of the Big-Oh Notation (9)

• If f(n) is O(g(n)), g(n) is O(h(n)), then f(n) is O(h(n)).


• Reason:
f(n)  c*g(n) , for some c > 0, n  m
g(n)  d*h(n) , for some d > 0, n  p
so f(n)  (c*d)h(n) , for some c*d > 0, n  max(p,m)

Example: kn is O(n), n is O(n2), so kn is O(n2).

45
Properties of the Big-Oh Notation (10)

• If f(n), g(n) are O(h(n)), then f(n) + g(n) is O(h(n)).


Reason:

f(n)  c*h(n) , for some c > 0, n  m


g(n)  d*h(n) , for some d > 0, n  p
so f(n) + g(n)  c*h(n) + d*h(n) , n  max(m,p)
 (c+d)h(n) , c + d > 0, n  max(m,p)

46
Properties of the Big-Oh Notation (11)
• If T1(n) is O(f(n)), T2(n) is O(g(n)), then
T1(n) + T2(n) is O(max(f(n), g(n))).
Reason:

Let h(n) = max(f(n),g(n)).


T1(n)  c*f(n)  c*h(n) for some c > 0, n  m.
T2(n)  d*g(n)  d*h(n) for some d > 0, n  p.
So T1(n) + T2(n)  (c+d)h(n) for some c+d > 0, n  max(p,m)

47
Properties of the Big-Oh Notation (12)
• T1(n) is O(f(n)), T2(n) is O(g(n)), then
T1(n) T2(n) is O(f(n)g(n)).
Reason:
T1(n)  c*f(n) , for some c > 0, n  m
T2(n)  d*g(n) , for some d > 0, n  p
so T1(n) T2(n)  (c*d)f(n)g(n) , for some c*d > 0, n  max(p,m)

48
Properties of the Big-Oh Notation (13)

• If lim n→ f(n)/g(n) exists and is finite, then f(n) is


O(g(n))

Example: log n is O(nr) for r  0, since limn→log(n)/ nr = 0.

49
Orders of Big-Oh Notations
• The following commonly used big-Oh notations are
listed in increasing order (increasing growth rate):
• Constant O(1)
• Logarithmic O(log n)
• Linear O(n)
• Loglinear, or “n log n” O(n log n)
• Quadratic O(n2)
• Cubic O(n3)
• Exponential O(cn) (c > 1)
• Factorial O(n!)

50
Limitation of Big-Oh Notation
• We should always keep in mind that Big-Oh notation
is the growth rate for large input size
• In practice, there is always an upper limit on the input size.
• Sometimes, we may be working on small input sizes, but
repeat such small problem many times. Then we should
analyse/compare the efficiency of the algorithm with
small input sizes, instead of simply using the Big-Oh
notation.
• We should also keep in mind that constant
coefficient is ignored in Big-Oh notation. When the
constant factor is large, it has significant effect on the
efficiency in practice.
51
Algorithm Analysis General Rules

52
Analysis of statement
• An example of statement:

a = 3*b + 4

• There are 3 basic operations in this statement: one


multiplication, one addition and one assignment
operation “=“
• We consider that there are 3 operations in this
statement

53
Analysis of loop
• Loop (for loop, while loop, …)
• The running time of a loop is the running time of the
statements inside the loop multiplied by the number of
iterations
• Example:
for i from 1 to n
a[i] = i*3
• There are n iterations.
• In each iteration, there are 2 basic operations (one
multiplication, one assignment)
• Total: 2*n = O(n)
(Here we ignored the cost of increasing the value of i, and
ignored the cost of testing the value of i against n. These
computations do not affect the big Oh notation of algorithm)

54
Analysis of nested loops
• Nested loops
for i from 1 to n
a[i] = i*3
for j from 1 to n/2
b[j] = a[i] + j*j
• The inner loop (j) consists of n/2 iterations, 3
computational units in each iteration (b[j] = a[i] + j*j)
• The outer loop (i) consists of n iterations
• Total running time: n * (2 + 3*n/2) = O(n2)

55
Analysis of consecutive statements
• Consecutive statements
for i from 1 to n
a[i] = i*3
for j from 1 to n/2
b[j] = a[i] + j*j
• The first loop (i) consists of n iterations
• The second loop (j) consists of n/2 iterations
• Total running time: 2*n + 3*n/2 = O(n)

56
Analysis of control statements (if/else)

• if/else

if (condition is true)
(….a1 computational units)
else
(….a2 computational units)

• Running time is upper bounded by: max(a1, a2)

57
Summary
• Asymptotic notations are useful for measuring the
grow rate of algorithm
• Big-Oh notation
• Upper bound
• Most commonly used
• Big-Omega notation
• Lower bound
• Big-Theta notation
• Both upper and lower bounds
• Limitations
• measuring the growth rate for large n (the input size may
be small for some problems in applications)
• Constant factors are normally ignored in asymptotic
notations 58

You might also like