Data Structures and Algorithm Analysis: COMP2015 HKBU/CS/JF/2023 1

Data Structures and Algorithm
Analysis
n Algorithm Analysis:
q How to predict an algorithm’s performance
q How well an algorithm scales up
q How to compare different algorithms for a problem
n Data Structures:
q How to efficiently store, access, manage data
q Data structures affect algorithm’s performance
COMP2015 HKBU/CS/JF/2023 1
Machine Independent Analysis
n A program segment to calculate
int Sum(int N)
{
int i, PartialSum;
PartialSum = 0;
for (i = 1; i <= N; i++)
PartialSum += i*i*i;
return PartialSum;
}
The total basic operations for this function is

1+1+2N+2+4N = 6N + 4
Machine Independent Analysis
n Algorithm 1 uses ? basic operations.

n Algorithm 2 uses ? basic operations.
n Algorithm ? is more efficient.
n It is difficult to derive the exact number of operations,

especially for a large program.
n In fact, we will not worry about the exact values, but will
look at “broad classes” of values, or the growth rates.
Asymptotic Analysis
n Let there be N inputs

q If an algorithm needs N basic operations and another needs
2N basic operations, consider them to be in the same
efficiency category.
q However, distinguish between exp(N), N, log(N)
n Asymptotic analysis
q A way to describe behavior of functions in the limit, for
example, upper bound or lower bound of functions.
q Look at growth of T(N) as N infinity
q Establish a relative order among functions for large N
Asymptotic Notations
n Common notations:
q O » ≤
q W » ³
q Q » =
q o » <
q w » >
n A way to compare “sizes” of functions

n The Q, O, and W notations are used to express whether a
function grows
· at the same rate as,
· at a rate no faster than, or
· at a rate no slower than
another function.
Start from a Simple Problem ...
n Compare T(N) = 1000N and f(N) = N2
q 1000N is larger than N2 for small values of N
q N2 grows faster and will be larger
n The turning point: N = 1000 = n0
n There is some point n0 past which cf(N) (c = 1 in this

case) is always at least as big as T(N).
n We could also use n0 = 10 and c = 100.
n We can say that 1000N = O(N2)
Asymptotic Notation: Big-Oh
n T(N) = O(f(N))
n There exist positive constants c and n0
such that
0 ≤ T(N) ≤ cf(N) for all N ≥ n0
n The growth rate of T(N) is less than or

equal to the growth rate of f(N).
n f(N) is an asymptotic upper bound for
T(N)
n O»≤
n Example: Let f(N) = 2N2

q f(N) = O(N4)
Asymptotic Notation: Big-Omega
n T(N) = W(g(N))
n There exist positive constants c and n0

such that
0 ≤ cg(N) ≤ T(N) for all N ≥ n0
n The growth rate of T(N) is greater than

or equal to the growth rate of g(N).
n g(N) is an asymptotic lower bound for

T(N)
n W»³
n Example: Let T(N) = 2N2. Then

q T(N) = W(N)
Asymptotic Notation: Big-Theta
n T(N) = Q(h(N))
n There exist positive constants c1, c2

and n0 such that
0 ≤ c1h(N) ≤ T(N) ≤ c2h(N) for
all N ≥ n0
§ T(N) = O(h(N)) and T(N) = W(h(N))
§ The growth rate of T(N) equals the

growth rate of h(N).
n h(N) is an asymptotically tight

bound for T(N)
n Q»=
Asymptotic Notation: Little-oh
Ø T(N) = o(p(N))
for all constants c > 0, there exists a constant n0 > 0 such that
0 ≤ T(N) < c p(N) for all N ≥ n0
n T(N) = O(p(N)) and T(N) ¹ Q(p(N))

n The growth rate of T(N) is less than the growth rate of p(N).
n Another view, probably easier to use
n o»<
Asymptotic Notation
n w Notation: Asymptotically bounds a function strictly
from below only
n T(N) = w(q(N))
for all constants c > 0, there exists a constant n0 > 0 such
that
0 ≤ c q(N) < T(N) for all N ≥ n0
n Another view, probably easier to use
n w»>
n Intuitive meaning:
q T(N) = O(f(N)): the growth rate of T(N) £ that of f(N);
q T(N) = W(g(N)): the growth rate of T(N) ³ that of g(N);
q T(N) = Q(h(N)): the growth rate of T(N) = that of h(N);
q T(N) = o(p(N)): the growth rate of T(N) < that of p(N);
q T(N) = w(q(N)): the growth rate of T(N) > that of q(N)
n When T(N) = O(f(N)), we are guaranteeing that the function

T(N) grows at a rate no faster than f(N). Thus, f(N) is an upper
bound on T(N).
n Similarly, when T(N) = W(g(N)), we say g(N) is a lower bound
on T(N).
n Examples:
we have: f(N) = O(g(N)), and f(N) = W(g(N)).
if g(N) = 2N2, are g(N) = O(N4), g(N) = O(N3), and g(N) = O(N2)
all correct?
Growth of Functions
§ We worry about the speed of our algorithms for large input,

so we want to know the growth of running time when the
size of input is getting larger.
n Commonly encountered running times are proportional to

the following functions:
q c :Represents a constant
q Log N :Logarithmic
q N :Linear time
q N log N
q N2 :Quadratic
q N3 :Cubic
q 2N :Exponential
Complexity and Tractability
Assume the computer does 1 billion operations per second
Growth Rate of Different Functions
n Growth rates of some common functions:
(log N) < (N) < (N logN) < (N2) < (N3) < (2N) < (N!)
n An algorithm with smaller complexity is normally

preferable to one with larger complexity.
n Tuning an algorithm without affecting its complexity is

usually not as good as finding an algorithm with better
complexity.
Simplifying the Bound
How to derive f(N) from T(N)?
n T(N) = ck Nk + ck-1 Nk-1 + ck-2Nk-2 + … + c1 N + co
q too complicated
q too many terms
q Difficult to compare two expressions, each with 10 or 20
terms
n Do we really need that many terms?
Simplifications
n Keep just one term!
q The fastest growing term (dominates the running time)
n No constant coefficients are kept

q Constant coefficients affected by machines, languages, etc.
q Machine-dependent constants
n Asymptotic behavior (as N gets large) is determined entirely by

the leading term.
q Example: T(N) = 10N3 + N2 + 40N + 800
n If N = 1,000, then T(N) = 10,001,040,800
n Error is only 0.01% if we drop all but the N3 term.
q In an assembly line, the slowest worker determines the

throughput rate.
Asymptotic Analysis of Functions
n Asymptotic analysis is equivalent to

q ignoring multiplicative constants
q ignoring lower-order terms
q “for large enough inputs”
q Example:
n N2 + 3N + 4 = O(N2)
n N2 + N + N log N + log N + 1 = O(N2)
n 3N + 5 = O(N)
n Big-O and growth rate

q Big-O gives an upper bound on the growth rate of a function
Relative Growth Rates of Two
Functions
n To compare the growth rates of two functions f(N) and g(N)
q use L’Hopital’s rule to evaluate:
Growth rate of f(N) < that of g(N)
Growth rate of f(N) > that of g(N)
Growth rate of f(N) = that of g(N)
Example
n Given , compare their growth rates.
Growth rate of (N3) > Growth rate of (N2)
Question
n If f(N) = NlogN, g(N) = N1.5, which of f(N) and g(N) grows

faster?
Asymptotic Algorithm Analysis
n Time complexity analysis: determine the running time in

big-O notation
n To perform the asymptotic analysis

q Find the worst-case number of primitive operations
executed as a function of the input size
q Express this function with big-O notation
Asymptotic Analysis: Review
n What does it mean to say that an algorithm has running

time O(n log n)?
n n: Problem size
n Big-O: upper bound over all inputs of size n
n Highlight those dominant terms
n “Ignore constant factor”
n “as n grows large”
n Big-O Rules
n If f(n) is a polynomial of degree d, then f(n) is O(nd), i.e.,

1. Drop lower-order terms
2. Drop constant factors
n Use the smallest possible class of functions, if possible

n Say “2n is O(n)” instead of “2n is O(n )”
2
n Use the simplest expression of the class

n Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”
n Big-O Rules
• Examples:
if T1(N) = O(N2) and T2(N)= O(N) then
(a) T1(N) + T2(N) = O(N2)
(b) T1(N)*T2(N) = O(N3)
Questions
n Suppose an algorithm has running time Q(N3)

q Suppose solving a problem of size 1000 takes 10
seconds. How long to solve a problem of size 10000?
n Suppose an algorithm has running time Q(N log N)

q Suppose solving a problem of size 1000 takes 10
seconds. How long to solve a problem of size 10000?
Algorithm Complexity Analysis
Complexity of a loop:
diff = sum = 0;
for (k=0; k < N; k++) Each iteration of the first loop takes 2
basic steps.
sum ® sum + 1;
First loop runs N times.
diff ® diff - 1;
Each iteration of the second loop takes 1
for (k=0; k < 3N; k++) basic step.
sum ® sum - 1; Second loop runs for 3N times.
Overall, 2N + 3N steps
This is O(N)
Running Time Calculations: General
Rules
n Rule 1: for loops
for (i = 0; i < N; i++) O(N)

k++
n The running time of a for loop is at most the running time

of the statements inside the for loop (including tests) times
the number of iterations.
O(no. of iterations in a loop * maximum complexity of
each iteration)
Rules
n Nested for loops:
n Analyze the innermost loop first, complexity of next outer

loop = number of iterations in this loop * complexity of inner
loop, etc…..
sum = 0;
Inner loop: O(N)
for (i=0; i < N; i++)
for (j=0; j < N; j++) Outer loop: N iterations
sum ® sum + 1; Overall: O(N2)
Rules
n Rule 2: Nested for loops
for (i = 0; i < N; i++)

for (j = 0; j < N; j++) O(N2)
k++
n The total running time of a statement inside a group of

nested loops is the running time of the statement multiplied
by the product of the size of all the for loops.
Rules
n Rule 3: Consecutive Statements

Just add together
O(N) + O(N2)
for (i=0; i<N; i++)
A[i] = 0;
for (i=0; i<N; i++)
for (j=0; j<N; j++) O(N2)
A[i] += A[j] +i+j;
Rules
n Rule 4: Condition Statement
if (condition)
S1
else
S2
n The running time of an if/else statement is never more than

the running time of the test plus the larger of the running
times of S1 and S2.
Example
Sum=0
for (j=0; j<N; j++)
for (k=0; k<N*N; k++) O(N3)
Sum++;
Sum=0
for (j=0; j<N; j++)
for (k=0; k<j; k++) O(N2)
Sum++;
Example: Binary Search
n Analysis of Binary Search Problem:
Given an integer X and integers A0,
while (low <= high) A1, …, AN-1, which are presorted
{ mid = (low + high) / 2; and already in memory, find i such
if (x == a [mid]) that Ai = X, or return i = -1 if X is
not in the input.
return (mid);
else if (x < a [mid])
high = mid - 1;
else O(logN)
low = mid + 1;
}
Question
for( i = 0; i < n; i++ )

for( j = i; j < n; j++ )
{
for(k = i; k <= j; k++ )
thisSum += a[ k ];
}

Data Structures and Algorithm Analysis: COMP2015 HKBU/CS/JF/2023 1

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Data Structures and Algorithm Analysis: COMP2015 HKBU/CS/JF/2023 1

Uploaded by

Copyright:

Available Formats

Data Structures and Algorithm

q How well an algorithm scales up

q How to compare different algorithms for a problem

q Data structures affect algorithm’s performance

The total basic operations for this function is

n Algorithm 1 uses ? basic operations.

n It is difficult to derive the exact number of operations,

n Let there be N inputs

n A way to compare “sizes” of functions

q N2 grows faster and will be larger

n The turning point: N = 1000 = n0

n There is some point n0 past which cf(N) (c = 1 in this

n We could also use n0 = 10 and c = 100.

n We can say that 1000N = O(N2)

n The growth rate of T(N) is less than or

n Example: Let f(N) = 2N2

n There exist positive constants c and n0

n The growth rate of T(N) is greater than

n g(N) is an asymptotic lower bound for

n Example: Let T(N) = 2N2. Then

n There exist positive constants c1, c2

§ T(N) = O(h(N)) and T(N) = W(h(N))

§ The growth rate of T(N) equals the

n h(N) is an asymptotically tight

n T(N) = O(p(N)) and T(N) ¹ Q(p(N))

n Another view, probably easier to use

n Another view, probably easier to use

q T(N) = W(g(N)): the growth rate of T(N) ³ that of g(N);

q T(N) = Q(h(N)): the growth rate of T(N) = that of h(N);

q T(N) = o(p(N)): the growth rate of T(N) < that of p(N);

q T(N) = w(q(N)): the growth rate of T(N) > that of q(N)

n When T(N) = O(f(N)), we are guaranteeing that the function

we have: f(N) = O(g(N)), and f(N) = W(g(N)).

§ We worry about the speed of our algorithms for large input,

n Commonly encountered running times are proportional to

Assume the computer does 1 billion operations per second

n Growth rates of some common functions:

n An algorithm with smaller complexity is normally

n Tuning an algorithm without affecting its complexity is

n T(N) = ck Nk + ck-1 Nk-1 + ck-2Nk-2 + … + c1 N + co

n Do we really need that many terms?

n No constant coefficients are kept

n Asymptotic behavior (as N gets large) is determined entirely by

n If N = 1,000, then T(N) = 10,001,040,800

n Error is only 0.01% if we drop all but the N3 term.

q In an assembly line, the slowest worker determines the

n Asymptotic analysis is equivalent to

q ignoring lower-order terms

q “for large enough inputs”

n N2 + N + N log N + log N + 1 = O(N2)

n Big-O and growth rate

Growth rate of f(N) < that of g(N)

Growth rate of f(N) > that of g(N)

Growth rate of f(N) = that of g(N)

n Given , compare their growth rates.

Growth rate of (N3) > Growth rate of (N2)

n If f(N) = NlogN, g(N) = N1.5, which of f(N) and g(N) grows

n Time complexity analysis: determine the running time in

n To perform the asymptotic analysis

n What does it mean to say that an algorithm has running

n If f(n) is a polynomial of degree d, then f(n) is O(nd), i.e.,

2. Drop constant factors

n Use the smallest possible class of functions, if possible