Fundamentals of The Analysis of Algorithm Efficiency: Keun-Liang Sue Information Management NCU

Fundamentals of the Analysis of
Algorithm Efficiency
Keun-Liang Sue
Information Management
NCU
2023/3/9 1
Analysis Framework
Time efficiency
How fast an algorithm in question runs.
Primarily concentrated.
Space efficiency
The extra space the algorithm required.
2023/3/9 Algorithms by KLSue/NCU 2

Measuring an Input’s Size
Investigate an algorithm’s efficiency

A function of the algorithm’s input size.
Measuring size of inputs
Not always equal to the number of input data.
e.g., checking whether an integer n is prime.
the number b of bits in n’s binary representation
b  log 2 n   1
Units for Measuring Running Time
Count the number of times

Each of the algorithm’s operation is executed.
Difficult and unnecessary.
Identify the most important operation
Basic operation
Operation contributing most of total running time.
Compute number of times basic operation is
executed.

 Define :
cop：Execution time of an algorithm’s basic
operation.
C(n)：the number of times this operation needs.
 Estimate running time T(n) by formula
T (n)  cop C (n)

What if double input size

Assume C ( n) 
1
2
n(n  1) ,how much longer if T(2n)
For all but very small values of n:
1 1 1 1
C ( n)  n(n  1)  n 2  n  n 2
2 2 2 2
Therefore:
1 2
T (2n) cop C (2n) 2 (2n)
  1 2 4
T ( n) cop C (n) 2
n
Note
Answer it without knowing the value of cop
Multiplicative constant, ½, for C(n)

It was also cancelled out.
So, concentrates on order of growth.
Orders of Growth

Orders of Growth
logarithmic function(log2n).
Function growing the slowest among these.
Algorithm with logarithmic basic-operation count

run instantaneously on inputs of all realistic sizes.
Orders of Growth
Why log2n? Can be log n?
Sure!!
Can omit a logarithm’s base and write simply logn.
See the formula
log a n  log a b log b n

logab is multiplicative constant, can be ignored.
• Q: Why constant can be ignored?

Orders of Growth
Bad functions
Exponential function 2n and factorial function n!
Grow so fast
The value become large even for small value
Both referred to as “exponential-growth functions”
Remember :
• “Algorithms that require an exponential number of
operations are practical for solving only problems of
very small sizes”
Worst-Case, Best-Case, and Average-

Case Efficiencies

Case Efficiencies
Worst-Case Efficiencies
Definition
Efficiency of the worst-case input of size n
An input the algorithm runs the longest
• among all possible inputs
The way to determine it
Analyze the algorithm
• see what kind of inputs yield the largest value of C(n)
Compute this worst-case value Cworst(n)

Case Efficiencies
Benefits from the analysis
Provide the upper bound of running time
Guarantee that running time will not exceed Cworst(n)

• For any instance of size n
Sequential search, Cworst(n)=n

Case Efficiencies
Best-Case Efficiencies
Definition
Efficiency for the best-case input of size n
Runs the fastest among all possible input of that size
Analyze the best-case efficiency
Determine the kind of inputs
• for which C(n) will be the smallest
Ascertain the value of C(n)
• on the most convenient inputs
• In our example, Cbest(n)=1

Case Efficiencies
Average-Case Efficiencies
Assumptions for Analyze our example
prob. of a successful search is p
(0  p  1)
prob. of first match occurring in ith position of the list
is the same for every i.

Case Efficiencies
In case of successful search

prob. of first match occurring in ith position is p/n
The number of comparison is i
In case of unsuccessful search

the number of comparisons is n
prob. of unsuccessful search is (1-p).

Case Efficiencies
Therefore,
p p p p
Cavg (n)  [1  2     i     n  ]  n  (1  p )
n n n n
p
 [12i n]n(1 p)
n
p n(n1) p(n1)
 n(1 p)  n(1 p)
n 2 2
If p=1, Cavg(n)=(n+1)/2

If p=0, Cavg(n)=n

Case Efficiencies
Note
Average-case is more difficult than two others
Q: Need average-case information?
• Yes
• Many algorithm, average is much better than worst
• Without average-case analysis,
 Computer scientists miss important algorithms
Average-case is not equal to
• Average of best-case and worst case efficiencies

Case Efficiencies
Amortized efficiency (分期攤還)
Definition
It applies to a sequence of operations
• Not to a single run of an algorithm
• performed on the same data structure.
Phenomenon
A single operation can be expensive
• In some situations
Total time for entire n such operations
• always better than worst-case efficiency * n

Asymptotic Notations and Basic Efficiency
Classes
Introduction
To compare and rank such orders of growth
Three Notations:
Ο(big oh),Ω(big omega),Θ(big theta).
Terms:
t(n) and g(n):
• Nonnegative functions defined on natural numbers
t(n): an algorithm’s running time.
g(n): simple function to compare the count with.

Classes
Informal Introduction
Ο(g(n)):
Set of all functions with a smaller or same order of
growth as g(n)
Within a constant multiple, as n goes to infinity
For examples:
1
n  O(n 2 ), 100n  5  O(n 2 ), n(n  1)  O(n 2 )
2
n 3  O(n 2 ), 0.00001n 3  O(n 2 ), n 4  n  1  O(n 2 )
Classes
Ω(g(n)):
set of all functions with a larger or same order of
growth as g(n)
Within a constant multiple, as n goes to infinity.
For examples:
1
n 3  (n 2 ), n(n  1)  (n 2 ), but 100n  5  (n 2 )
2

Classes
Θ(g(n)):
set of all functions have the same order of growth as
g(n)
Within a constant multiple, as n goes to infinity.
Every an 2  bn  c with a  0 is in Θ(n2).

Classes
 Ο-notation
Definition
A function t(n) is said to be in Ο(g(n)), denoted
t (n)  O( g (n))
• if t(n) is bounded above by some constant multiple of g(n)
for all large n.
i.e. exist positive constant c and nonnegative integer
n0 such that
t (n)  cg (n) for all n  n0


Classes
 Ο-notation
For example:
100n  5  O(n 2 )
Indeed,
100n  5  100n  n (for all n  5)  101n  101n 2
Take c =101 and n0=5

Classes

Classes
Ω-notation
Definition
A function t(n) is said to be in Ω(g(n)), denoted
t (n)  ( g (n))
• if t(n) is bounded below by some positive constant
multiple of g(n) for all large n.
i.e. exist positive constant c and nonnegative integer
n0 such that
t (n)  cg (n) for all n  n0
Classes
Ω-notation
example:
Proof that n 3  (n 2 ) :
n 3  n 2 for all n  0
i,e., select c =1 and n0=0

Classes

Classes
Θ-notation
Definition
A function t(n) is said to be in Θ(g(n)), denoted
t (n)  ( g (n))
• if t(n) is bounded both above and below by some positive
constant multiples of g(n) for all large n.
i,e., if there exist positive constant c1 ,c2 and nonnegative
integer n0 such that
c2 g (n)  t (n)  c1 g (n) for all n  n0

Classes
Θ-notation
example:
1
n(n  1)  (n 2 )
2
First, prove the upper bound:
1 1 1 1
n(n  1)  n 2  n  n 2 for all n  0
2 2 2 2

Classes
Θ-notation
Second, prove the lower bound:
1 1 1 1 1 1 1
n(n  1)  n 2  n  n 2  n n (for all n  2)  n 2
2 2 2 2 2 2 4
Select 1 1
c2  , c1  and n0  2
4 2

Classes
Θ-notation

Useful Property Involving the Asymptotic
Notations
Theorem:
If t1 (n)  O( g1 (n)) and t 2 (n)  O( g 2 (n)), then
t1 (n)  t 2 (n)  Omaxg1 (n), g 2 (n)
Proof:
Since t1 (n)  O ( g1 (n)),

t1 (n)  c1 g1 (n) for all n  n1
Since t 2 (n)  O( g 2 (n)),
t 2 ( n )  c2 g 2 ( n ) for all n  n2

Notations
Denote c3=max{c1,c2} and consider n >= max{n1,n2}

So,
t1 (n)  t 2 (n)  c1 g1 (n)  c2 g 2 (n)

 c3 g1 (n)  c3 g 2 (n)  c3 g1 (n)  g 2 (n)
 c3 2 maxg1 (n), g 2 (n)

Notations
Hence,
t1 (n)  t 2 (n)  Omaxg1 (n), g 2 (n)
With constant c=2c3=2max{c1,c2}
and n0=max{n1,n2}
The theorem implies
algorithm’s overall efficiency will be determined
by the part with a larger order of growth.
i.e., by its least efficiency part.
Using Limits for Computing Orders of

Growth
Compare orders of growth of two functions
Compute the limit of ratio of two functions
Three principal cases:
0 implies that t (n) has a smaller order of growth than g (n)
t ( n) 
lim  c implies that t (n) has the same order of growth as g (n)
n  g ( n)

 implies that t (n) has a larger order of growth than g (n).
Question:
First two case means ?
Last two case means ?
The second case means ?
Classes
Useful technique
L’Hôpital’s rule:
t ( n) t ' ( n)
lim  lim
n  g ( n) n  g ' ( n)
Stirling’s formula:
n
n
n! 2n   for large values of n.
e

Classes
Example: compare n(n-1)/2 and n2
1
n ( n 1) 1 n2  n 1 1 1
lim 2
 lim 2  lim(1  ) 
n  n2 2 n  n 2 n  n 2
The limit is equal to a positive constant

The functions have the same order of growth.
Symbolically,
1
n(n  1)  (n 2 )
2

Classes
Example: compare log 2 n and n
1
(log 2 e)
log 2 n (log 2 n) ' n  2 log e lim n  0
lim  lim  lim 2
n  n n  ( n)' n  1 n  n
2 n

Classes
Example: compare n! and 2n
n
2 n ( ) n
n! e nn n n
lim n  lim n
 lim 2 n n n
 lim 2 n ( ) 
n  2 n  2 n  2 e n  2e

Mathematical Analysis of Nonrecursive

Algorithms
Q: Finding value of largest element in a list

Algorithms
Who is basic operation
Two operations
Comparison A[i]>maxval
Assignment maxval←A[i]
Which is the basic operation?
Answer: the comparison A[i]>maxval
Define C(n):
the number of times this comparison is executed.

Algorithms
The sum for C(n):

n 1
C ( n)   1
i 1
Thus,
n 1
C ( n )   1  n  1  ( n )
i 1

General Plan for Analyzing Efficiency of
Nonrecursive Algorithms
Decide a parameter indicating an input’s size.
Identify the algorithm’s basic operation
As a rule, it is in the innermost loop
Check whether NBO depends on size of input.
If also depends on other property,
• The worst-case, average-case and best-case must
be investigated
Note:
• NBO means the number of times the basic
operation is executed
General Plan for Analyzing Efficiency of

Nonrecursive Algorithms
Set up a sum expressing C(n)
Using standard formulas and rules of sum

manipulation
either Find the closed-form formula
or, at least, Establish order of growth

Element uniqueness problem
Whether all elements in array are distinct?

Basic operation:
the comparison of two elements.
Define Cworst(n):
the largest number of comparison among all arrays of
size n
Two kind of worst-case
Array with no equal elements
Last two elements are the only pair of equal elements

Accordingly,
n  2 n 1 n2 n2
Cworst(n)   1  [(n  1)  (i  1)  1]   (n  1  i)
i 0 j i 1 i 0 i 0
n2 n2 n2
(n  2)(n  1)
  (n  1)   i  (n  1)1 
i 0 i 0 i 0 2
(n  2)(n  1) (n  1)n 1 2
 (n  1) 2    n  (n 2 )
2 2 2
Multiple square matrices of order n

Algorithm Efficiency
 Find the time efficiency of the algorithm

Basic operation:
the algorithm’s innermost loop.
M(n):
sum for the total number of multiplication executed.

Only one multiplication on each basic operation.

Governed by k ranging from 0 to n-1.
The number of multiplications is :

for each pair of specific i and j
n 1
1
k 0
Thus, the total number of multiplications M(n):
n 1 n 1 n 1 n 1 n 1 n 1
M (n)  1   n   n 2  n 3
i 0 j 0 k 0 i 0 j 0 i 0

Finds number of binary digits in binary
representation of a decimal integer

The most frequently executed operation:
The comparison n > 1
The comparison will be executed

Repetitions of the loop’s body + 1 times.
Thus, the comparison n > 1 will be executed
log 2 n  1 times
Mathematical Analysis of Recursive
Algorithms
Compute factorial function

Algorithms
Computing F(n)=n!
for an arbitrary non-negative integer n.
n! 1  (n  1)  n  (n  1)!n for n  1

Basic operation:
multiplication
the number of executions is denoted M(n)

Algorithms
F(n)=F(n-1)*n for n>0
M(n) must satisfy the equality:
M (n)  M (n  1)  1 for n  0
to compute F(n -1) to multiply F(n -1) by n
M ( 0)  0

Algorithms
Method of backward substitutions:
M (n)  M (n  1)  1 substitute M (n  1)  M (n  2)  1
 [ M (n  2)  1]  1  M (n  2)  2 substitute M (n  2)  M (n  3)  1
 [ M (n  3)  1]  2  M (n  3)  3
Thus,
M (n)  M (n  1)  1    M (n  i )  i    M (n  n)  n  n.

A General Plan for Analyzing Efficiency of
Recursive Algorithms
Decide a parameter indicating input’s size
Identify the algorithm’s basic operation
Check whether NBO vary on different inputs
of the same size
If also depends on other property
• The worst-case, average-case and best-case
must be investigated
Note:
• NBO means the number of times the basic
operation is executed
A General Plan for Analyzing Efficiency of

Recursive Algorithms
Set up a recurrence relation
for number of times basic operation is executed
with appropriate initial condition
Solve the recurrence

or at least ascertain order of growth of its solution

Tower of Hanoi
Definition
What is Tower of Hanoi ?
Analysis
Input size: the number of disks n
Basic operation: moving one disk
The number of moves: M(n)
Tower of Hanoi

Tower of Hanoi
Generalize the problem
General Case
Base Case
General Case
Tower of Hanoi
Thus,
M (n)  M (n  1)  1  M (n  1) for n  1
Recurrence relation:
M (n)  2 M (n  1)  1 for n  1
M (1)  1

Tower of Hanoi
Solve this recurrence by backward substitutions:
M (n)  2M (n  1)  1 sub. M (n  1)  2M (n  2)  1
 2[2M (n  2)  1]  1  2 2 M (n  2)  2  1 sub. M (n  2)  2 M (n  3)  1
 2 2 [2M (n  3)  1]  2  1  23 M (n  3)  2 2  2  1
After i substitutions:
M (n)  2i M (n  i )  2i 1  2i  2    2  1  2i M (n  i )  2i  1
Tower of Hanoi
Because initial condition is n=1
We let i = n-1
So,
M (n)  2 n 1 M (n  (n  1))  2 n 1  1
 2 n 1 M (1)  2 n 1  1  2 n 1  2 n 1  1  2 n  1

Tower of Hanoi
Tree of tower of Hanoi puzzle

Useful for analysis purpose.
Nodes correspond to recursive calls.
Label them with the value of the parameter
Counting the number of nodes in the tree:
n 1
C (n)   2l (where l is the level in the tree above)  2 n  1
i 0

Recursive version of previous discussion

Algorithms
Analysis
Recurrence:
A(n)  An / 2  1 for n  1
A(1)  0
By smoothness rule, n=2k, so:
A(2 k )  A(2 k 1 )  1 for k  0

A(2 0 )  0

Algorithms
Using backward substitution:
A(2 k )  A(2 k 1 )  1 substitute A(2 k 1 )  A(2 k  2 )  1

 [ A(2 k  2 )  1]  1  A(2 k  2 )  2 substitute A(2 k  2 )  A(2 k 3 )  1
 [ A(2 k 3 )  1]  2  A(2 k 3 )  3

 A(2 k i )  i

 A(2 k  k )  k

Algorithms
Thus,
A(2 k )  A(1)  k  k , and n  2 k  k  log 2 n

A(n)  log 2 n  (log n)
In fact, the exact solution is
A(n)  log 2 n 

Fibonacci Numbers
Fibonacci numbers:
0,1,1,2,3,5,8,13,21,34,…
Can be defined by simple recurrence:
F (n)  F (n  1)  F (n  2) for n  1
F (0)  0, F (1)  1
Explicit Formula for the nth Fibonacci

Number
Theorem
Homogeneous second-order linear recurrence
with constant coefficients:
aX (n)  bX (n  1)  cX (n  2)  0
a, b and c: coefficients of the recurrence
X(n): unknown sequence to be found

Number
Characteristic equation for recurrence:
ar 2  br  c  0
Apply this theorem to the case of Fibonacci
numbers:
F (n)  F (n  1)  F (n  2)  0

Number
Its characteristic equation:
r 2  r 1  0
With the roots:
1  1  4(1) 1  5
r1, 2  
2 2

Number
Use the formula of Theorem1 in Appendix B:
n n
1 5  1 5 
F (n)      
 


 2   2 
Thus,
0 0
1 5  1 5 
F (0)      
 
 0
   0
 2   2 
1 1 1 5  1 5 
1 5  1 5        1
F (1)      
  2 
 1 
2   2 
 2     

Number
We get
  1 / 5 and   -1/ 5
 thus,
n n
1 1 5  1 1 5  1  n n 
F ( n)            
5  2  5  2  5  
1 n
F ( n)   rounded to the nearest integer.
5
“The impact of second term can be obtained by

rounding off the value of first term to nearest integer”

Algorithms for Computing Fibonacci
Numbers

Numbers

Numbers
Analysis
Basic operation: addition
A(n): the number of additions

performed by the algorithm in computing F(n)
A(n)  A(n  1)  A(n  2)  1 for n  1,

A(0)  0, A(1)  0.

Numbers
Inhomogeneous recurrences
Similar to homogeneous recurrence
• But its right-hand side ≠0.
Reduce our inhomogeneous recurrence

• to a homogeneous one by rewriting it :
A(n)  1  A(n  1)  1  A(n  2)  1  0

Numbers
Substituting B(n)=A(n)+1:
B (n)  B (n  1)  B (n  2)  0
B (0)  1, B (1)  1.
The recurrence can be solved:
B(n)  F (n  1)
1  n 1
A(n)  B(n)  1  F (n  1)  1  (   )  1
n 1
5
Hence,
A(n)  ( n )

Numbers
Non-recursive version
Only n-1 additions
Extra array for storing elements can be avoided

Empirical Analysis of Algorithms
Some algorithm is difficult to analyze

The alternative is empirical analysis
Steps for the analysis
Understand the experiment’s purpose.
Decide on efficiency metric M and measure unit
Decide on characteristics of the input sample.

Prepare a program implementing the algorithm.
Generate a sample of inputs.
Run the algorithm on the sample’s inputs
Record the data observed.
Analyze the data obtained.

Random Number Generator
Random Number Generator
recommended parameters
seed: set to the current date and time
m: 2w
w: the computer’s word size
a: integer between 0.01m and 0.99m
b: can be chosen as 1

Scatterplot
Empirical data from experiment must be recorded
Then, presented for an analysis
Data Can be presented in a graph
Called scatterplot
Observation of scatterplot
Help to ascertain the efficiency class of a algorithm
Typical scatterplots
Concave shape: for a logarithmic algorithm.
Straight line: for a linear algorithm.
Convex shape: for
(n log n) or (n 2 ) or (n 3 )

Concave shape
Straight line

Convex shape
Algorithm Visualization
Another way to study algorithm

Use graph elements
Point, line segment, 3-dimension bar
Represent interesting events
principal variations of algorithm visualization
Static algorithm visualization.
Shows an algorithm’s progress.
Through a series of still images.

Algorithm Visualization
Dynamic algorithm visualization
Also called algorithm animation.
A movie-like presentation of an algorithm’s operation.




Fundamentals of The Analysis of Algorithm Efficiency: Keun-Liang Sue Information Management NCU

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Fundamentals of The Analysis of Algorithm Efficiency: Keun-Liang Sue Information Management NCU

Uploaded by

Copyright:

Available Formats

Fundamentals of the Analysis of

2023/3/9 Algorithms by KLSue/NCU 2

Investigate an algorithm’s efficiency

Units for Measuring Running Time

Count the number of times

2023/3/9 Algorithms by KLSue/NCU 4

C(n)：the number of times this operation needs.

 Estimate running time T(n) by formula

T (n)  cop C (n)

Units for Measuring Running Time

What if double input size

Multiplicative constant, ½, for C(n)

So, concentrates on order of growth.

2023/3/9 Algorithms by KLSue/NCU 7

2023/3/9 Algorithms by KLSue/NCU 8

Algorithm with logarithmic basic-operation count

2023/3/9 Algorithms by KLSue/NCU 9

log a n  log a b log b n

2023/3/9 Algorithms by KLSue/NCU 10

2023/3/9 Algorithms by KLSue/NCU 11

Worst-Case, Best-Case, and Average-

2023/3/9 Algorithms by KLSue/NCU 12

2023/3/9 Algorithms by KLSue/NCU 13

Worst-Case, Best-Case, and Average-

Provide the upper bound of running time

Guarantee that running time will not exceed Cworst(n)

Sequential search, Cworst(n)=n

2023/3/9 Algorithms by KLSue/NCU 14

2023/3/9 Algorithms by KLSue/NCU 15

Worst-Case, Best-Case, and Average-

2023/3/9 Algorithms by KLSue/NCU 16

In case of successful search

In case of unsuccessful search

2023/3/9 Algorithms by KLSue/NCU 17

Worst-Case, Best-Case, and Average-

If p=1, Cavg(n)=(n+1)/2

2023/3/9 Algorithms by KLSue/NCU 18

2023/3/9 Algorithms by KLSue/NCU 19

Worst-Case, Best-Case, and Average-

2023/3/9 Algorithms by KLSue/NCU 20

2023/3/9 Algorithms by KLSue/NCU 21

Asymptotic Notations and Basic Efficiency

Within a constant multiple, as n goes to infinity

Within a constant multiple, as n goes to infinity.

Asymptotic Notations and Basic Efficiency

Within a constant multiple, as n goes to infinity.

Every an 2  bn  c with a  0 is in Θ(n2).

2023/3/9 Algorithms by KLSue/NCU 24

t (n)  cg (n) for all n  n0

Asymptotic Notations and Basic Efficiency

100n  5  100n  n (for all n  5)  101n  101n 2

Take c =101 and n0=5

2023/3/9 Algorithms by KLSue/NCU 26

2023/3/9 Algorithms by KLSue/NCU 27

Asymptotic Notations and Basic Efficiency

2023/3/9 Algorithms by KLSue/NCU 29

Asymptotic Notations and Basic Efficiency

2023/3/9 Algorithms by KLSue/NCU 30

c2 g (n)  t (n)  c1 g (n) for all n  n0

2023/3/9 Algorithms by KLSue/NCU 31

Asymptotic Notations and Basic Efficiency

2023/3/9 Algorithms by KLSue/NCU 32

2023/3/9 Algorithms by KLSue/NCU 33