Asymptotic Analysis

Reading: Chapter 1

 http://csce.uark.edu  +1 (479) 575-6043  yrpeng@uark.edu

 2.3
Background
❑ Suppose we have two algorithms, how can we tell which is
better?

❑ We could implement both algorithms, run them both

● Expensive and error prone

❑ Preferably, we should analyze them mathematically

● Algorithm analysis

2019/9/9 CSCE 4113/5113: Algorithms 2

 What do we Mean by Analysis?
❑ Analysis is performed with respect to a computational model
❑ We will usually use a generic uniprocessor random-access
machine (RAM)
● All memory is equally expensive to access
● No concurrent operations
● All reasonable instructions take unit time
▪ Except, of course, function calls
● Constant word size
▪ Unless we are explicitly manipulating bits

2019/9/9 CSCE 4113/5113: Algorithms 3

 Example: Searching
❑ Assume we have a sorted array of integers, X[1..N] and we are
searching for “key”

Cost Times
found = 0; C0 1
i = 0; C1 1
while (!found && i < N) { C2 0 <= L < N
if (key == X[i]) C3 1 <= L <= N
found = 1; C5 1
i++; C4 1 <= L <= N
}

T(n) = C0 + C1 + L*(C2 + C3 + C4), where 1 <= L <= N is

the number of times that the loop is iterated.

2019/9/9 CSCE 4113/5113: Algorithms 4

 Example: Searching
❑ What’s the best case?
● Loop iterates just once =>
● T(n) = C0 + C1 + C2 + C3 + C4 + C5
❑ What’s the average (expected) case?
● Loop iterates N/2 times =>
● T(n) = C0 + C1 + N/2 * (C2 + C3 + C4) + C5
● Notice that this can be written as T(n) = a + b*n where a, b are constants
❑ What’s the worst case?
● Loop iterates N times =>
● T(n) = C0 + C1 + N * (C2 + C3 + C4) + C5
● Notice that this can be written as T(n) = a + b*n where a, b are constants

2019/9/9 CSCE 4113/5113: Algorithms 5

 Worst Case Analysis
❑ We will look at WORST CASE running time of an algorithm. Why?
● Worst case is an upper bound on the running time. It gives us a
guarantee that the algorithm will never take any longer

● For some algorithms, the worst case happens fairly often. As in this
search example, the searched item is typically not in the array, so the
loop will iterate N times

● The “average case” is often roughly as bad as the “worst case”. In our
search algorithm, both the average case and the worst case are linear
functions of the input size “n”
❑ However, if the average case is the majority of all cases, then we
will focus on it.
● Like sorting, routing, etc
2019/9/9 CSCE 4113/5113: Algorithms 6
 2.3.1
Maximum Value
❑ For example, the time taken to find the largest object in an array
of n random integers will take n operations
❑ Linear Search
int find_max( int *array, int n ) {
int max = array[0];

for ( int i = 1; i < n; ++i ) {

if ( array[i] > max ) {
max = array[i];
}
}

return max;
}

2019/9/9 CSCE 4113/5113: Algorithms 7

 2.3.2
Linear and Binary Search
❑ There are other algorithms which are significantly faster as the
problem size increases

❑ This plot shows maximum

and average number of
comparisons to find an entry
in a sorted array of size n
● Linear search
● Binary search

n
2019/9/9 CSCE 4113/5113: Algorithms 8
 2.3.3
Quadratic Growth
❑ Consider the two functions
● f(n) = n2 and g(n) = n2 – 3n + 2
❑ Around n = 0, they look very different

2019/9/9 CSCE 4113/5113: Algorithms 9

 2.3.3
Quadratic Growth
❑ Yet on the range n = [0, 1000], they are (relatively)
indistinguishable:

2019/9/9 CSCE 4113/5113: Algorithms 10

 2.3.3
Polynomial Growth
❑ To demonstrate with another example,
f(n) = n6 and g(n) = n6 – 23n5+193n4 –729n3+1206n2 – 648n
● Around n = 0, they are very different

2019/9/9 CSCE 4113/5113: Algorithms 11

 2.3.3
Polynomial Growth
● Still, around n = 1000, the relative difference is less than 3%

2019/9/9 CSCE 4113/5113: Algorithms 12

 Asymptotic Notation
❑ We will study the asymptotic efficiency of algorithms
● To do so, we look at input sizes large enough to make only the order of
growth of the running time relevant
● That is, we are concerned with how the running time of an algorithm
increases with the size of the input in the limit as the size of the input
increases without bound.
● Usually an algorithm that is asymptotically more efficient will be the
best choice for all but very small inputs.
▪ Real-time systems, games, interactive applications need to limit the input
size to sustain their performance.

2019/9/9 CSCE 4113/5113: Algorithms 13

 2.3.3
Asymptotic Analysis
❑ Given an algorithm:
● We need to be able to describe these values mathematically
● We need a systematic means of using the description of the algorithm
together with the properties of an associated data structure
● We need to do this in a machine-independent way

❑ For this, we need Landau symbols and the associated

asymptotic analysis
● Big O, Q, W Notations

2019/9/9 CSCE 4113/5113: Algorithms 14

 2.3.5
Weak Ordering
❑ Consider the following definitions:
● We will consider two functions to be equivalent, f ~ g, if

f ( n)
● where lim =c 0c
n → g ( n)
f ( n)
● We will state that f < g if lim =0
n → g ( n)

❑ For functions we are interested in, these define a weak ordering

2019/9/9 CSCE 4113/5113: Algorithms 15

 2.3.5
Weak Ordering
❑ Let f(n) and g(n) describe either the run-time of two algorithms
● If f(n) ~ g(n), then it is always possible to improve the performance of
one function over the other by purchasing a faster computer
● If f(n) < g(n), then you can never purchase a computer fast enough so
that the second function always runs in less time than the first

❑ Note that for small values of n, it may be reasonable to use an

algorithm that is asymptotically more expensive, but we will
consider these on a one-by-one basis

2019/9/9 CSCE 4113/5113: Algorithms 16

 2.3.5
Big O
❑ The function f(n) has a rate of growth no greater than that of g(n)
● A function f(n) = O(g(n)) if there exists N and c such that
whenever n > N
f(n) < c g(n)

f( n)
❑ If lim =c where 0  c   , it follows that f(n) = O(g(n))
n →
g( n)

2019/9/9 CSCE 4113/5113: Algorithms 17

 2.3.6
Small o
❑ There are other possibilities we would like to describe:
● The function f(n) has a rate of growth less than that of g(n)

f( n)
● If lim
n →
= 0 , we will say f(n) = o(g(n))
g( n)

❑ We would also like to describe the opposite cases:

● f(n) has a rate of growth greater than that of g(n)
f( n ) = ω(g( n ))
● f(n) has a rate of growth greater than or equal to that of g(n)
f( n) = Ω(g( n))

2019/9/9 CSCE 4113/5113: Algorithms 18

 2.3.5
Big Theta Q
❑ A function f(n) = Q(g(n)) if there exist positive N, c1, and c2
such that
c1 g(n) < f(n) < c2 g(n)
whenever n > N
❑ The function f(n) has a rate of growth equal to that of g(n)

f( n)
❑ If lim =c where 0  c   , it follows that f(n) = Q(g(n))
n →
g( n)

2019/9/9 CSCE 4113/5113: Algorithms 19

 2.3.7
Landau Symbols
We will at times use five possible descriptions
f( n )
f( n ) = o(g( n )) lim
n → g( n )
=0

f( n )
f( n) = O(g( n)) lim 
n→ g( n )

f( n )
f( n) = Θ(g( n)) 0  lim 
n→ g( n )

f( n )
f( n) = Ω(g( n)) lim 0
n → g( n )

f( n )
f( n ) = ω(g( n )) lim =
n→ g( n )

2019/9/9 CSCE 4113/5113: Algorithms 20

 2.3.7
Landau Symbols
❑ It can be said that
f(n) = O(g(n)) is equivalent to f(n) = Q(g(n)) or f(n) = o(g(n))
f(n) = W(g(n)) is equivalent to f(n) = Q(g(n)) or f(n) = w(g(n))

❑ Some other observations we can make are:

f(n) = Q(g(n)) ⇔ g(n) = Q(f(n))
f(n) = O(g(n)) ⇔ g(n) = W(f(n))
f(n) = o(g(n)) ⇔ g(n) = w(f(n))

2019/9/9 CSCE 4113/5113: Algorithms 21

 2.3.7
Landau Symbols
❑ Graphically, we can summarize these as follows:

❑ We say

❑ if

2019/9/9 CSCE 4113/5113: Algorithms 22

 2.3.8
Equivalence Relation
❑ If we look at the first relationship, we notice that f(n) = Q(g(n))
seems to describe an equivalence relation:
● f(n) = Q(g(n)) if and only if g(n) = Q(f(n))
● f(n) = Q(f(n))
● If f(n) = Q(g(n)) and g(n) = Q(h(n)), it follows that f(n) = Q(h(n))
❑ For example, all of following are big-Q of each other

n2 100000 n2 – 4 n + 19 n2 + 1000000
323 n2 – 4 n ln(n) + 43 n + 10 42n2 + 32
n2 + 61 n ln2(n) + 7n + 14 ln3(n) + ln(n)

2019/9/9 CSCE 4113/5113: Algorithms 23

 2.3.8
Equivalence Relation
❑ The most common classes are given names:

Q(1) constant
Q(ln(n)) logarithmic
Q(n) linear
Q(n ln(n)) “n log n”
Q(n2) quadratic
Q(n3) cubic
2n, en, 4n, ... exponential

2019/9/9 CSCE 4113/5113: Algorithms 24

 Asymptotic Complexity

250

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

2019/9/9 CSCE 4113/5113: Algorithms 25

 Asymptotic Complexity

5000

4000
f(n) = n
f(n) = log(n)
3000
f(n) = n log(n)
f(n) = n^2
2000 f(n) = n^3
f(n) = 2^n

1000

0
1 3 5 7 9 11 13 15 17 19

2019/9/9 CSCE 4113/5113: Algorithms 26

 2.3.8
Logarithms and Exponentials
❑ Recall that all logarithms are scalar multiples of each other
● Therefore for any base b:
logb(n)= Q(ln(n))
❑ Alternatively, there is no single equivalence class for
exponential functions:
n
● If 1 < a < b, an a
lim n = lim   = 0
n → b n →  b 

● Therefore an = o(bn)

2019/9/9 CSCE 4113/5113: Algorithms 27

 Common Functions
Name Big-Oh Comment
Constant O(1) Can’t beat it!

Log O(logN) Typical time for good searching algorithms

Linear O(N) This is about the fastest that an algorithm can

run given that we need O(n) just to read the
input
N logN O(NlogN) Most sorting algorithms

Quadratic O(N2) Acceptable when the data size is small

(N<10000)
Cubic O(N3) Acceptable when the data size is small
(N<1000)
Exp O(2N) Only good for really small input sizes (n<=20)

Factorial O(N!) Super-fast growth

2019/9/9 CSCE 4113/5113: Algorithms 28