Algorithmic Foundation

Algorithmic Foundations
COMP108
COMP108
Algorithmic Foundation
Prudence Wong
http://www.csc.liv.ac.uk/~pwong/teaching/comp108
1
COMP108
Module information …
2
COMP108
Teaching Team
Dr Prudence Wong
G45.A Chadwick Bldg.
pwong@csc.liv.ac.uk
Office hours: Monday 3-5pm
Miss Cindy Li, Miss Carmen Pardavila,

Miss Chang Su, Mr Cloud Zhang
3
COMP108
Module Structure
Lectures
Tutorials Help channels
Assignments Examination
& Quizzes 4
COMP108
~36 lectures
(Mon 2-3pm, Wed 11am-12pm, Thu 10-11am)
10 tutorials (1 hr per week)
4 Assignments & 1 or 2 Quizzes

(20% of final mark)
A written examination
(80% of final mark)
Office hours, email, appointment (by

email) 5
COMP108
Main Reference
Introduction to the Design
and Analysis of Algorithms
Anany V. Levitin
Villanova University
Addison Wesley
Additional Reading
Introduction to Algorithms
Thomas H. Cormen,
Charles E. Leiserson,
Ronald L. Rivest
The MIT Press
6
COMP108
More about assignments

¾ plagiarism
is the practise of presenting the
thoughts or writings of another or others as
original
¾ consequence?
¾ What happen if you are ill when there is a

deadline for assignment?
¾ Get medical proof and ask for extension
¾ This is not an excuse to commit plagiarism
7
COMP108
Why COMP108?
Pre-requisite for: COMP202, COMP209
(COMP202 is pre-requisite for COMP308, COMP309)
Algorithm design is a foundation for efficient

and effective programs
Year 1 modules do not count towards honour

classification…
(Message from Career Services:
Employers DO consider year 1 module results)
8
COMP108
Aims
What do we mean by good?
¾ To give an overview of the study of algorithms in terms of
their efficiency.
How to achieve?
¾ To introduce the standard algorithmic design paradigms
employed in the development of efficient algorithmic
solutions.
Can we prove?
¾ To describe the analysis of algorithms in terms of the use of
formal models of Time and Space.
Can all problems be solved efficiently?
¾ To give a brief introduction to the subject of computational
complexity theory and its use in classifying computational
problems. 9
COMP108
Ready to start …
10
COMP108
Learning outcomes
¾ Able
to tell what an algorithm is and have
some understanding why we study algorithms
¾ Able to use pseudo code to describe algorithm
¾ Able to carry out simple asymptotic analysis
of algorithms
¾ Seesome examples of polynomial time and
exponential time algorithms
¾ Ableto apply searching/sorting algorithms and
derive their time complexities
11
COMP108
Learning outcomes
¾ Able
of algorithms
12
COMP108
What is an algorithm?
A sequence of precise and concise instructions
that guide you (or a computer) to solve a
specific problem
Input Algorithm Output
Daily life examples: cooking recipe, furniture

assembly manual
(What are input / output in each case?)
13
COMP108
Why do we study algorithms?

The obvious solution to a problem
may not be efficient
Example: We are given a map with n cities and
the traveling cost between the cities. What is
the cheapest way to go from city A to city B?
8
Simple solution
10 ¾ Compute the cost of
6
12 3 4 each route from A to B
3
¾ Choose the cheapest one
5 2 5
3
5
9 7
7 5
1 6 11
5 9 14
COMP108
Shortest path to go from A to B

How many routes between A & B?

involving 1 intermediate city?
Simple solution
2 B
¾ Compute the cost of
each route from A to B

15
COMP108


involving 1 intermediate city? 3?
Simple solution
2 B

Number of routes
A =2
16
COMP108


Simple solution
B

3
Number of routes
A =2+3
17
COMP108


Simple solution
B

Number of routes
A =2+3
18
COMP108


Simple solution
B

Number of routes
A =2+3+6
6 = 11
19
COMP108

TOO
MANY!!
involving 1 intermediate city? 3? 8?
Simple solution
B

For large n, it is
A impossible to check all
routes!
We need more
sophisticated solutions
20
COMP108

There is an algorithm, called Dijkstra's
algorithm, that can compute this shortest path
efficiently.
21
COMP108
Idea of Dijkstra's algorithm
Go one step from A, label the

visited city with the cost.
B
10 8
10 6
12 3 4
0 3
5 2 5
3
5
9
A 7 5
7
1 6 11
5 9 22
COMP108

Choose city with the smallest
cost and go one step from it
similarly.
B
10 8
10 6
12 3 4
0 3
5 2 5
5 3
5
9
A 7 5
7
1 6 11
5 9 23
COMP108
Again ...
B
10 8
10 6
12 3 4
0 3
5 2 5
5 3
5
9
A 7 5
7
1 6 11
5
7 9 24
COMP108

Choose city with the smallest cost.
All neighbours of A have labels already.
Choose the next smallest one.
B
10 8
10 6
12 3 4
0 3
5 2 5
5 3
5
9
A 7 5
7
1 6 11
5
7 9 25
COMP108

This path must NOT An alternative cheaper route is
be used because we found!
find a cheaper Record this route and erase the
alternative path more expensive one.
B
10
8 8
10 6
12 3 4
0 3
5 2 5
5 3
5
9
A 7 5
7
1 6 11
5
7 9 26
COMP108
Repeat and repeat ...
17
B
8 8
10 6
12 3 4
0 3
5 2 5
5 3
5
9
A 7 5
7
1 6 11
5
7 9 27
COMP108

We will further
Finally, the cheapest route
discuss Dijkstra’s
from A to every other city will
algorithm later. be discovered.
?
B
8 8
10 6
12 3 4
0 3 ?
5 2 5 ?
5 3
5
A 7
9
5
? 7
1 6 11
5 ?
7 ?
9 28
COMP108
Lesson to learn
The most straightforward

(or brute force) algorithm
for a problem may run slowly.
We need more sophisticated
solutions.
29
COMP108
Learning outcomes
9 Able
of algorithms
30
COMP108
How to represent
algorithms …
31
COMP108
Algorithm vs Program
Still remember? An algorithm is a sequence of precise and concise
instructions that guide a computer to solve a specific problem
Algorithms are free from grammatical rules

¾ Contentis more important than form
¾ Acceptable as long as it tells people how to
perform a task
Programs must follow some syntax rules

¾ Form is important
¾ Even if the idea is correct, it is still not
acceptable if there is syntax error
32
COMP108
Computing the n-th power

Input: a number a & a non-negative integer n
Output: the n-th power of x
Algorithm:
1. Set a temporary variable p to 1.
2. Repeat the multiplication p = p * a for n times.
3. Output the result p.
33
COMP108
Pseudo Code
p = 1
Another way to
while i <= n do
describe algorithm
begin
is by pseudo code
p = p * a
i = i+1
p = 1 end
for i = 1 to n do output p
p = p * a
output p
similar to programming
language
more like English Combination of both
34
COMP108
Pseudo Code (2)

Assignment statement
variable = expression
e.g., assign the expression 3 * 4 to
the variable result
result = 3 * 4
compound statements
begin
statement1
statement2
end 35
COMP108
Pseudo Code (3)

if a < 0 then
Conditional statement
if condition then
a = -a
statement(s) abs = a
output abs
if condition then
statement(s)
else
statement(s) if a > 0 then
abs = a
else
What do these two abs = -a
algorithms compute? output abs
36
COMP108
Pseudo Code (4)

Iterative statement
for var = start_value to end_value do
statement(s)
while condition do
statement(s)
repeat
statement(s)
until condition
37
COMP108
Pseudo Code (5)

for var = start_value to end_value do
statement(s)
1. var is first assigned the Computing sum of the

value start_value. first n numbers:
2. If var ≤ end_value, input n
execute the statement. sum = 0
3. var is incremented by 1 for i = 1 to n do
and then go back to step 2. begin
sum = sum + i
end
the loop is executed for n output sum
times 38
COMP108
Pseudo Code (6)

while condition do 1. if condition is true,
statement(s) execute the statement
2. else stop
3. go back to step 1
Computing sum of all input numbers:
sum = 0
while (user wants to continue) do
begin
ask for a number the loop is executed
sum = sum + number
end for undetermined
output sum number of times
39
COMP108
Pseudo Code (7)

repeat 1. execute the statement
statement(s) 2. if condition is true, stop
until condition
3. go back to step 1
Computing sum of all input numbers:

sum = 0 ¾ theloop is also
repeat executed for
ask for a number
undetermined
sum = sum + number
until (user wants to stop) number of times
output sum ¾ How it differs
from while-loop?
40
COMP108
Learning outcomes
9 Able
9 Able to use pseudo code to describe algorithm
of algorithms
41
COMP108
Analysis of algorithms …
42
COMP108
Analysis of Algorithms
After designing an algorithm, we analyze it.
¾ Proof of correctness: show that the algorithm
gives the desired result
¾ Time complexity analysis: find out how fast the
algorithm runs
¾ Space complexity analysis: decide how much
memory space the algorithm requires
¾ Look for improvement: determine whether the
algorithm is best possible; can we improve it to run
faster or use less memory? 43
COMP108
Proof of Correctness
There are different proof techniques:
¾ Direct Proof
¾ Proof by Cases
¾ Proof by Contradiction
¾ Proof by Mathematical Induction
¾…
We will learn more of them in later examples.
44
COMP108
Mathematical Induction
¾ Claim: 1+2+…+k = k(k+1)/2 for all positive integers k
¾ Denote p(k) as the statement "1+2+…+k = k(k+1)/2"
¾ Basic step: When k=1, 1+2+…+k = 1 = k(k+1)/2.
Therefore, p(1) is true.
¾ Induction step: Assume that p(k) is true for an integer
k ≥ 1. We need to prove that p(k+1) is true.
¾ 1+2+…+k+(k+1) = k(k+1)/2 + (k+1) = (k+1)(k/2+1) = (k+1)(k+2)/2
¾ Therefore, p(k+1) is true.
¾ Conclusion: p(k) is true for all positive integers
45
COMP108
A nonsense induction
Claim: All COMP108 students are of the same gender.
Let us prove the claim by induction on the number of
students.
Basic step: Consider any group of COMP108 students
which comprises only one student. Same gender of
course.
Induction step: Assume that any group of n COMP108
students are of the same gender.
Next, we consider any group of (n+1) COMP108
students.
46
COMP108
Remove a student, say X

n+1 X
X
n
Induction hypothesis: Induction hypothesis:
Same gender Same gender
Replace a student, say Y, with X
X n
Y
The argument
Therefore, X and Y and works only when
other students are of n>1, i.e., it fails
same gender when n=1
47
COMP108
Proof of Correctness
Proof by Mathematical Induction (MI)
1. In the 0-th iteration, sum equals 0. Correct!
2. Assume sum correctly stores 1+2+...+k after the k-th

iteration, for some k.
3.Then after the (k+1)-th iteration, by the induction
hypothesis, sum stores (1+2+…+k)+k+1.
Therefore, by MI, the algorithm output the correct sum.
input n
sum = 0
Computing sum of the for i = 1 to n do
first n numbers begin
sum = sum + i
end 48
output sum
COMP108
Time Complexity Analysis

How fast is the algorithm?
Code the algorithm and run the program, then
measure the running time
1. Depend on the speed of the computer

2. Waste time coding and testing if the
algorithm is slow
Identify some important operations/steps and

count how many times these operations/steps
needed to executed
49
COMP108

How to measure efficiency?
Number of operations usually expressed in
terms of input size
¾ If we doubled the input size, how much longer

would the algorithm take?
If we trebled the input size, how much longer
would it take?
50
COMP108
Why efficiency matters?

¾ speed of computation by hardware has been
improved
¾ efficiency still matters
¾ ambitionfor computer applications grow with
computer power
¾ demand a great increase in speed of
computation
51
COMP108
Amount of data handled
matches speed increase?
When computation speed vastly increased,
can we handle much more data?
Suppose
• an algorithm takes n2 comparisons to sort n numbers
• we need 1 sec to sort 5 numbers (25 comparisons)
• computing speed increases by factor of 100
Using 1 sec, we can now perform 100x25 comparisons,
i.e., to sort 50 numbers
With 100 times speedup, only sort 10 times more numbers!
52
COMP108

input n
Important operation of sum = 0
summation: addition for i = 1 to n do
begin
How many additions this sum = sum + i
end
algorithm requires? output sum
We need n additions
(depend on the input size n)
53
COMP108
Space Complexity Analysis

input n
We need to store the sum = 0
number n, the variable sum, for i = 1 to n do
begin
and the index i sum = sum + i
end
=> needs 3 memory space output sum
In this example, the memory space

does not depend on the input size n
In other case, it may depend on

the input size n 54
COMP108
Look for improvement

input n
Mathematical formula gives sum = n*(n+1)/2
us an alternative to find output sum
the sum of first n numbers:
1 + 2 + ... + n = n(n+1)/2
We only need 3 operations:

1 addition, 1 multiplication, and 1 division
(no matter what the input size n is)
55
COMP108
Analyze some example

algorithms - finding the
minimum…
56
COMP108
Finding min among 3 numbers

Input: 3 numbers a, b, c
Output: the min value of these 3 numbers
Algorithm: Input a, b, c
Y a≤b? N
Y a≤c? Y b≤c? N
Output a N Output b Output c

Output c 57
COMP108

Example
Input a, b, c
a=3, b=5, c=2
Y
a≤b? N
Y a≤c? Y b≤c? N
Output a N Output b Output c

Output c
2 is output
58
COMP108

input a, b, c Proof by cases
if (a ≤ b) then 1. if a is min:
if (a ≤ c) then both (a ≤ b) & (a ≤ c) are
T, then a is output
output a
else 2. if a is not min, but b:
(a ≤ b) is F & (b ≤ c) is T,
output c then b is output
else 3. if a & b are not min,
if (b ≤ c) then but c:
output b both (a ≤ c) & (b ≤ c) are
else T, then no matter (a ≤ b)
is T or F, c is output
output c 59
COMP108

input a, b, c
Important operation: if (a ≤ b) then
comparison if (a ≤ c) then
output a
How many comparison this else
output c
algorithm requires? else
if (b ≤ c) then
output b
else
output c
It takes 2 comparisons
60
COMP108

We need to store 3 numbers.
So we need 3 memory space.
input a, b, c
if (a ≤ b) then
if (a ≤ c) then
output a
else
output c
else
if (b ≤ c) then
output b
else
output c
61
COMP108
Finding min among n numbers

Input a0, a1, ...
Any
Y a0 ≤ a1 ? N systematic
approach?
Y a0 ≤ a2 ? Y a1 ≤ a2 ?
N N
a0 ≤ a3 ? a1 ≤ a3 ?
a2 ≤ a3 ? a2 ≤ a3 ?
62
COMP108

Input a[0], a[1], ..., a[n-1]
min = 0
i=1
i = i+1
i < n? N
Y
Output a[min]
Y
min = i a[i] < a[min]?
N
63
COMP108

input a[0], a[1], ..., a[n-1]
min = 0
i = 1
while (i < n) do Example
begin
if (a[i] < a[min]) then a[]={5,3,4,2,1}
min = i
i = i + 1 iteration min a[min]
end before 0 5
output a[min] 1 1 3
2 1 3
3 3 2
4 4 1 64
COMP108

¾ Proof: by MI
¾ Time complexity: ?? comparisons
¾ Space complexity: ?? memory space
¾ Can it be reduced? YES. How?
65
COMP108
Proof by induction
Let mini be the index of min value in a[0]...a[i].
Claim: After the i-th iteration, min stores mini.
1. Before any iteration (i=0), min stores 0 = min0
2. Assume after the k-th iteration, min stores mink
3. In the (k+1)-th iteration, a[mink] is compared with
a[k+1] and the index of the smaller value (i.e.,
mink+1) is stored in min
while (i < n) do
input a[0], a[1], ..., a[n-1] begin
min = 0 if (a[i] < a[min]) then
i = 1 min = i
i = i + 1
end
66
output min
COMP108
Finding the 2nd minimum

…
67
COMP108
Find the 2nd min number

Straightforward solution:
1. Run the find min number algorithm once
2. Remove this number
3. Run the find min number algorithm again
[This is the 2nd min number.]
Time complexity: n+(n-1) comparisons
Can we do better?
68
COMP108

a1 a2 a3 a4 a5 a6 a7 a8
a1 ≤ a2 a4 ≤ a3 a6 ≤ a5 a7 ≤ a8
a1 a4 a6 a7
a4 ≤ a1 a6 ≤ a7 a6
a4
a4 ≤ a6 a4
The 2nd min number is either a1

or a3 or a6, two more comparisons
give the answer
69
COMP108

Proof of correctness
a1 a2 a3 a4 a5 a6 a7 a8
a1 ≤ a2 a4 ≤ a3 a6 ≤ a5 a7 ≤ a8
a1 a4 a6 a7
a4 ≤ a1 a6 ≤ a7 a6
a4
a4 ≤ a6 a4
candidates for the 2nd min number
do not required further consideration 70

COMP108

Time complexity analysis
1. comparisons involved in the tree is at most
(n-1)
2. the length of the red path is at most log2n,
finding the min of so many numbers takes
log2n-1 comparisons
¾ Total: n+log2n-2 comparisons
a1 a2 a3 a4 a5 a6 a7 a8
a1 ≤ a2 a4 ≤ a3 a6 ≤ a5 a7 ≤ a8
a1 a4 a6 a7
a4 a6 ≤ a7 a6
a4 ≤ a1
71
a4 ≤ a6 a4
COMP108
Time complexity
- Big O notation …
72
COMP108
Which algorithm is the fastest?

Consider a problem that can be solved by 5
algorithms A1, A2, A3, A4, A5 using different
number of operations (time complexity).
f1(n) = 50n + 20 f2(n) = 10 n log2 n + 100
f3(n) = n2 - 3n + 6 f4(n) = 2n2
f5(n) = 2n/8 - n/4 + 2
n 1 2 4 8 16 32 64 128 256 512 1024 2048
f1(n) = 50n + 20 70 120 220 420 820 1620 3220 6420 12820 #### #### ####
f2(n) = 10 n log2n + 100 100 120 180 340 740 1700 3940 9060 20580 #### #### ####
f3(n) = n 2 - 3n + 6 4 4 10 46 214 934 3910 #### 64774 #### #### ####
f4(n) = 2n 2 2 8 32 128 512 2048 8192 #### 131072 #### #### ####
f5(n) = 2n/8 - n/4 + 2 2 2 3 32 8190 5E+08 2E+18
f4(n) f3(n) f1(n)

Depends on the size of the problem! 73
COMP108
1000000
100000
10000
Time
1000
f1(n)
f1(n) == 50n
50n++20
20
100 f2(n) ==10
f2(n) 10nnlog2n
log2n+ +100
100
f3(n) ==nn2
f3(n) 2 - 3n + 6
- 3n + 6
10 f4(n) ==2n
f4(n) 2
2n2
f5(n) ==22n/8
f5(n)
n/8 - n/4 + 2
- n/4 + 2
1
1 2 4 8 16 32 64 128 256 512 1024 2048 4096
n
74
COMP108
What do we observe?
¾ There is huge difference between
¾ functions involving powers of n (e.g., n, n log n, n2,
called polynomial functions) and
¾ functionsinvolving powering by n (e.g., 2n, called
exponential functions)
¾ Among polynomial functions, those with same
order of power are more comparable
¾ e.g., f3(n) = n2 - 3n + 6 and f4(n) = 2n2
75
COMP108
Big-O notation
f(n) = O(n log n) [read as f(n) is of order n log n]
¾ What does it mean? (note that log n is base 2).
¾ Roughly speaking, f(n) is at most n log n
multiplied by a constant
¾ e.g., f(n) = 3 n log n + 10n + 100
There exists a constant c such that for all

sufficiently large n, f(n) ≤ c n log n
¾ ∃c ∃no ∀n: if n>no then f(n) ≤ c n log n
¾ e.g., if no=2 & c=63, f(2) = 126 ≤ 63 * (2 log 2),
f(3) = ... 76
COMP108
Big-O notation (2)

f(n) is O(g(n)) [read as f(n) is of order g(n)]
¾ There exists a constant c such that for all
sufficiently large n, f(n) ≤ c g(n)
¾ ∃c ∃no ∀n: if n>no then f(n) ≤ c g(n)
¾ Problem: Show that 2n2 + 4n is O(n2)
9 Solution: Since n ≤ n2 for all n>0, we have 2n2 + 4n
≤ 2n2 + 4n2 ≤ 6n2.
Therefore, by definition, 2n2 + 4n is O(n2).
9 On the other hand, since n2 ≤ 2n2 + 4n for all n>0,
it is also true that n2 is O(2n2 + 4n).
¾ Thesetwo functions are said to have the same
order of magnitude 77
COMP108
Which one is the fastest?

Usually we are only interested in the
asymptotic time complexity, i.e., when n is
large
Answer: O(n) < O(n log n) < O(n2) < O(2n)
78
COMP108
Hierarchy of functions
¾ We can define a hierarchy of functions each
having a greater order of magnitude than its
predecessor:
1 log n n n2 n3 ... nk ... 2n
constant logarithmic polynomial exponential
¾ We can further refine the hierarchy by

inserting n log n between n and n2,
n2 log n between n2 and n3, and so on.
79
COMP108
Hierarchy of functions (2)

1 log n n n2 n3 ... nk ... 2n
Important: as we move from left to right,

successive functions have greater order of
magnitude than the previous ones.
As n increases, the values of the later
functions increase more rapidly than the
values of the earlier ones.
⇒ Relative growth rates increase 80
COMP108
Relative growth rate

2n
n2
log n
c
n
81
COMP108
Relative growth rate (2)

1 log n n n2 n3 ... nk ... 2n
¾ Now, when we have a function, we can assign

the function to some function in the
hierarchy:
¾ For example, f(n) = 2n3 + 5n2 + 4n + 7
2n3 is O(n3), 5n2 is O(n2), 4n is O(n), 7 is O(1).
Then, f(n) is O(n3). N.B.: 4n is also O(n2),
but 5n2 is NOT O(n)
82
COMP108
Exercise
Determine the least order of magnitude of the
following functions.
1. n3 + 3n2 + 3 O(n3)
2. 4n2 log n + n3 + 5n2 + n O(n3)
3. 2n2 + n2 log n O(n2 log n)
4. 6n2 + 2n O(2n)
83
COMP108
Some algorithms we learnt

input n
Computing sum of the first sum = 0
n numbers for i = 1 to n do
begin
input n sum = sum + i
sum = n*(n+1)/2 end
output sum
O(?) output sum O(?)
Finding the min value min = 0

for i = 1 to n do
among n numbers if (a[i] < a[min]) then
min = i
output a[min] O(?)
Finding the 2nd min value among n numbers
takes n+log2n-2 comparisons O(?)
84
COMP108
More exercise
What is the time complexity of the following
pseudo code?
for i = 1 to 2n do
for j = 1 to n do
for k = 1 to j do
x = x + 1
O(?)
The outer-most loop iterates for 2n times.
The j-loop iterates for n times.
The k-loop iterates for j times, i.e., 1, 2, …, n.
Total: 2n * (1+2+...+n) = 2n* n(n+1)/2 = n2(n+1).85
COMP108
Example
Question: Give an O(n2)-time algorithm to check
whether all numbers in an array are all
distinct.
Rough idea: For each number, check it is equal
to any of the other numbers. If yes, NOT
distinct. After checking all numbers, then
distinct.
Algorithm: Why O(n2)?
for i = 0 to n-2 do
for j = i+1 to n-1 do
if a[i] == a[j] then
output "Not distinct" and stop
output "distinct" 86
COMP108
Summary
f(n) is O(g(n)) [read as f(n) is of order g(n)]
¾ There exists a constant c such that for all
sufficiently large n, f(n) ≤ c g(n)
¾ ∃c ∃no ∀n: if n>no then f(n) ≤ c g(n)
1 log n n n2 n3 ... nk ... 2n
¾ Usuallywe are interested in the asymptotic

time complexity (when n is large)
¾ Focus on worst-case time complexity 87
COMP108
Exercise
Are the followings correct?
1. n2log n + n3 + 3n2 + 3 O(n2)?
2. n + 1000 O(n)?
3. 6n20 + 2n O(n20)?
4. n3 + 5n2 log3n + n O(n2log3n) ?
88
COMP108
Matrix Multiplication
Consider the multiplication of two nxn matrices
A and B.
What is the time complexity of the definition-
based algorithm?
A col j B C
x =
row i
C[i,j]
C[i,j] = A[i,0]B[0,j] + A[i,1]B[1,j] + ... + A[i,n-1]B[n-1,j]

= x + x + x + x + x 89
COMP108
Matrix Multiplication (2)

C[i,j] = A[i,0]B[0,j] + A[i,1]B[1,j] + ... + A[i,n-1]B[n-1,j]
for i = 0 to n-1 do
for j = 0 to n-1 do
begin
C[i,j] = 0;
for k = 0 to n-1 do
C[i,j] = C[i,j] + A[i,k]*B[k,j]
end
Time complexity:
The i-loop iterates for n times, in which
the j-loop iterates for n times, in which
the k-loop iterates for n times as well.
Total n3 iterations, each takes O(1) steps.
Therefore, time complexity is O(n3). 90
COMP108
String Matching
Given a string of n characters called the text
and a string of m characters (m≤n) called the
pattern.
We want to find a substring of the text that
matches the pattern.
Example
text: N O B O D Y _ N O T I C E _ H I M
pattern: N O T
substring: N O B O D Y _ N O T I C E _ H I M
91
COMP108
String Matching
N O B O D Y _ N O T I C E _ H I M
N O T
N O T
N O T
N O T
for i = 0 to n-m do N O T
N O T
begin N O T
j=0 N O T
while j < m & P[j]==T[i+j] do
j=j+1
if j == m then
report "found!" & stop
end
report "Not found!" 92
COMP108
Time Complexity
How many comparisons this algorithm requires?
Best case: for i = 0 to n-m do
pattern appears in the begin
beginning of the text, j=0
O(m)-time j=j+1
Worst case: pattern if j == m then
appears at the end of end
the text OR pattern report "Not found!"
does not exists,
O(nm)-time
93
COMP108
Space Complexity
We need to store the n characters of the text,
the m characters of the pattern, the variables i,
j, m and n
=> need n+m+4 memory space, O(n+m)-space
for i = 0 to n-m do
begin
j=0
j=j+1
if j == m then
end
report "Not found!" 94
COMP108
Learning outcomes
9 Able
9 Able to use pseudo code to describe algorithm
9 Able to carry out simple asymptotic analysis
of algorithms
95
COMP108
More polynomial time

algorithms - searching …
96
COMP108
Searching
Input: a sequence of n numbers a0, a1, …, an-1;
and a number X
Output: determine whether X is in the sequence
or not
Algorithm (Linear search):
1. Starting from i=0, compare X with ai one by one
as long as i < n.
2. Stop and report "Found!" when X = ai .
3. Repeat and report "Not Found!" when i >= n.
97
COMP108
Linear Search
¾ 12 34 2 9 7 5 six numbers
7 number X
¾ 12 34 2 9 7 5
7
¾ 12 34 2 9 7 5
7
¾ 12 34 2 9 7 5
7
¾ 12 34 2 9 7 5
7 found!
98
COMP108
Linear Search (2)

¾ 12 34 2 9 7 5
10
¾ 12 34 2 9 7 5
10
¾ 12 34 2 9 7 5
10
¾ 12 34 2 9 7 5
10
¾ 12 34 2 9 7 5
10
¾ 12 34 2 9 7 5
10 not found! 99
COMP108
Linear Search (3)

i=0
while i < n do
begin
if X == a[i] then
report "Found!" and stop
else
i = i+1
end
report "Not Found!"
100
COMP108
Time Complexity
i=0
while i < n do
Important operation of begin
searching: comparison if X == a[i] then
report "Found!" & stop
How many comparisons this else
algorithm requires? i = i+1
end
report "Not Found!"
Best case: X is the 1st no., 1 comparison, O(1)

Worst case: X is the last no. OR X is not found, n
comparisons, O(n)
101
COMP108

We need to store the n numbers, the
value of X, and the variables i and n
=> need n+3 memory space, O(n)-space
i=0
while i < n do
begin
if X == a[i] then
report "Found!" and stop
else
i = i+1
end
report "Not Found!" 102
COMP108
Improve Time Complexity?

If the numbers are pre-sorted, then we
can improve the time complexity of
searching by binary search.
103
COMP108
Binary Search
more efficient way of searching when the sequence
of numbers is pre-sorted
Input: a sequence of n sorted numbers a0, a1, …, an-1;
and a number X
Idea of algorithm:
¾ compare X with number in the middle
¾ then focus on only the first half or the second half
(depend on whether X is smaller or greater than the
middle number)
¾ reduce the amount of numbers to be searched by half
104
COMP108
Binary Search (2)

3 7 11 12 15 19 24 33 41 55 10 nos
24 X
19 24 33 41 55
24
19 24
24
24
24 found!
105
COMP108
Binary Search (3)

3 7 11 12 15 19 24 33 41 55 10 nos
30 X
19 24 33 41 55
30
19 24
30
24
30 not found!
106
COMP108
Binary Search (4)

first=0, last=n-1
while (first <= last) do
begin
mid = (first+last)/2
if (X == a[mid])
report "Found!" & stop
else
if (X < a[mid])
last = mid-1
else
first = mid+1
end
report "Not Found!" 107
COMP108
Time Complexity
Best case:
first=0, last=n-1
X is the middle number while (first <= last) do
=> 1 comparison, O(1)-time begin
mid = (first+last)/2
Worst case: if (X == a[mid])
at most log2n+1 comparisons, report "Found!" & stop
O(log n)-time else
if (X < a[mid])
Why? Every comparison last = mid-1
reduces the amount of else
first = mid+1
numbers by at least half end
E.g., 16 => 8 => 4 => 2 => 1 report "Not Found!"
108
COMP108

We need to store the n first=0, last=n-1
numbers, the value of X, while (first <= last) do
begin
and the variables first, last, mid = (first+last)/2
mid and n if (X == a[mid])
report "Found!"
=> need n+5 memory space, else
O(n)-space if (X < a[mid])
last = mid-1
else
first = mid+1
end
report "Not Found!"
109
COMP108
Binary search vs Linear search

Time complexity of linear search is O(n)
Time complexity of binary search is O(log n)
Therefore, binary search is more efficient
than linear search
110
COMP108
More polynomial time

algorithms - sorting …
111
COMP108
Sorting
Input: a sequence of n numbers a0, a1, …, an-1
Output: arrange the n numbers into ascending
order, i.e., from smallest to largest
Example: If the input contains 5 numbers 132,
56, 43, 200, 10, then the output should be 10,
43, 56, 132, 200
There are many sorting algorithms:

insertion sort, selection sort, bubble sort,
merge sort, quick sort 112
COMP108
Selection Sort
¾ find minimum key from the input sequence
¾ delete it from input sequence
¾ append it to resulting sequence
¾ repeat until nothing left in input sequence
113
COMP108
Selection Sort - Example

¾ sort (34, 10, 64, 51, 32, 21) in ascending order
Sorted part Unsorted part Swapped
34 10 64 51 32 21 10, 34
10 34 64 51 32 21 21, 34
10 21 64 51 32 34 32, 64
10 21 32 51 64 34 51, 34
10 21 32 34 64 51 51, 64
10 21 32 34 51 64 --
10 21 32 34 51 64
114
COMP108
Selection Sort Algorithm

for i = 0 to n-2 do
begin
min = i
if a[j] < a[min] then
min = j
swap a[i] and a[min]
end
115
COMP108
Algorithm Analysis
for i = 0 to n-2 do
The algorithm consists of a begin
nested for-loop. min = i
For each iteration of the if a[j] < a[min] then
min = j
outer i-loop, swap a[i] and a[min]
there is an inner j-loop. end
Total number of comparisons i # of comparisons in

inner loop
= (n-1) + (n-2) + … + 1 0 n-1
= n(n-1)/2 1 n-2
O(n2)-time … ...
n-2 1
116
COMP108
Bubble Sort
starting from the last element, swap adjacent
items if they are not in ascending order
when first item is reached, the first item is the
smallest
repeat the above steps for the remaining items
to find the second smallest item, and so on
117
COMP108
Bubble Sort - Example

(34 10 64 51 32 21)
round
34 10 64 51 32 21
1 34 10 64 51 21 32
34 10 64 21 51 32
34 10 21 64 51 32
34 10 21 64 51 32
10 34 21 64 51 32
2 10 34 21 64 32 51
10 34 21 32 64 51
10 34 21 32 64 51
10 21 34 32 64 51
118
COMP108
Bubble Sort - Example (2)

round
10 21 34 32 64 51
3 10 21 34 32 51 64
10 21 34 32 51 64
10 21 32 34 51 64
4 10 21 32 34 51 64
10 21 32 34 51 64
5 10 21 32 34 51 64
119
COMP108
Bubble Sort Algorithm

the smallest will be moved to a[i]
for i = 0 to n-2 do
for j = n-1 downto i+1 do
if (a[j] < a[j-1])
start from a[n-1],
swap a[j] & a[j-1] check up to a[i+1]
34 10 64 51 32 21
i =0 j=5
j=4
j=3
j=2
j=1
i =1 j=5
j=4
j=3
j=2
120
COMP108
Algorithm Analysis
The algorithm consists of a nested for-loop.
for i = 0 to n-1 do
for j = n-1 downto i+1 do
if (a[j] < a[j-1])
swap a[j] & a[j-1]
Total number of comparisons

i # of comparisons in
= (n-1) + (n-2) + … + 1 inner loop
= n(n-1)/2 0 n-1
O(n2)-time 1 n-2
… ...
n-2 1
121
COMP108
Sorting
Input: a sequence of n numbers a0, a1, …, an-1
Output: arrange the n numbers into ascending
order, i.e., from smallest to largest
We have learnt these sorting algorithms:

selection sort, bubble sort
Next: insertion sort
122
COMP108
Insertion Sort
look at elements one by one
build up sorted list by inserting the element at
the correct location
123
COMP108
Example
¾ sort (34, 8, 64, 51, 32, 21) in ascending order
Sorted part Unsorted part int moved
34 8 64 51 32 21
34 8 64 51 32 21 -
8 34 64 51 32 21 34
8 34 64 51 32 21 -
8 34 51 64 32 21 64
8 32 34 51 64 21 34, 51, 64
8 21 32 34 51 64 32, 34, 51, 64
124
COMP108
Insertion Sort Algorithm

for i = 1 to n-1 do
begin using linear search to
find the correct
key = a[i]
position for key
pos = 0
while (a[pos] < key) && (pos < i) do
pos = pos + 1
shift a[pos], …, a[i-1] to the right
a[pos] = key
end
i.e., a[i-1] to a[i],
finally, place key a[i-2] to a[i-1], …,
(the original a[i]) in a[pos] to a[pos+1]
a[pos]
125
COMP108
Algorithm Analysis
for i = 1 to n-1 do
Worst case input begin
key = a[i]
¾ input
is sorted in pos = 0
while (a[pos] < key) && (pos < i) do
descending order pos = pos + 1
shift a[pos], …, a[i-1] to the right
Then, for a[i] a[pos] = key
end
¾ finding
the position takes i
comparisons i # of comparisons in
the while loop
total number of comparisons 1 1
= 1 + 2 + … + n-1 2 2
= (n-1)n/2 O(n2)-time
… ...
n-1 n-1
126
COMP108
Selection, Bubble, Insertion Sort

All three algorithms have time complexity O(n2)
in the worst case.
Are there any more efficient sorting
algorithms? YES, we will learn them later.
What is the time complexity of the fastest
comparison-based sorting algorithm?
O(n log n)
127
COMP108
Some exponential time

algorithms – Traveling
Salesman Problem,
Knapsack Problem …
128
COMP108
Traveling Salesman Problem

Input: There are n cities.
Output: Find the shortest tour from a
particular city that visit each city exactly
once before returning to the city where it
started.
This is known as
Hamiltonian circuit
129
COMP108
Example a
2
b
To find a Hamiltonian
7 8 3 circuit from a to a
5
c d
1
Tour Length
a -> b -> c -> d -> a 2 + 8 + 1 + 7 = 18
a -> b -> d -> c -> a 2 + 3 + 1 + 5 = 11
a -> c -> b -> d -> a 5 + 8 + 3 + 7 = 23
a -> c -> d -> b -> a 5 + 1 + 3 + 2 = 11
a -> d -> b -> c -> a 7 + 3 + 8 + 5 = 23
a -> d -> c -> b -> a 7 + 1 + 8 + 2 = 18 130
COMP108
Idea and Analysis

A Hamiltonian circuit can be represented by a
sequence of n+1 cities v1, v2, …, vn, v1, where
the first and the last are the same, and all
the others are distinct.
Exhaustive search approach: Find all tours in
this form, compute the tour length and find
the shortest among them.
How many possible tours to consider?
(n-1)! = (n-1)(n-2)…1
N.B.: (n-1)! is exponential in terms of
131 n
COMP108
Knapsack Problem
Input: Given n items with weights w1, w2, …, wn
and values v1, v2, …, vn, and a knapsack with
capacity W.
Output: Find the most valuable subset of items
that can fit into the knapsack.
Application: A transport plane is to deliver the
most valuable set of items to a remote
location without exceeding its capacity.
132
COMP108
Example
capacity = 10
w=7 w=3 w=4 w=5

v = 42 v = 12 v = 40 v = 25
item 1 item 2 item 3 item 4 knapsack
total total total total
subset weight value subset weight value
φ 0 0 {2,3} 7 52
{1} 7 42 {2,4} 8 37
{2} 3 12 {3,4} 9 65
{3} 4 40 {1,2,3} 14 N/A
{4} 5 25 {1,2,4} 15 N/A
{1,2} 10 54 {1,3,4} 16 N/A
{1,3} 11 N/A {2,3,4} 12 N/A 133
{1,4} 12 N/A {1,2,3,4} 19 N/A
COMP108
Idea and Analysis

Exhaustive search approach: Try every subset
of the set of n given items, compute total
weight of each subset and compute total value
of those subsets that do NOT exceed
knapsack's capacity.
How many subsets to consider?
2n, why?
134

Algorithmic Foundation

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Algorithmic Foundation

Uploaded by

Copyright:

Available Formats

Algorithmic Foundations

G45.A Chadwick Bldg.

Office hours: Monday 3-5pm

Miss Cindy Li, Miss Carmen Pardavila,

Tutorials Help channels

10 tutorials (1 hr per week)

4 Assignments & 1 or 2 Quizzes

Office hours, email, appointment (by

More about assignments

¾ What happen if you are ill when there is a

Algorithm design is a foundation for efficient

Year 1 modules do not count towards honour

Input Algorithm Output

Daily life examples: cooking recipe, furniture

Why do we study algorithms?

Shortest path to go from A to B

How many routes between A & B?

each route from A to B

Shortest path to go from A to B

How many routes between A & B?

each route from A to B

Shortest path to go from A to B

How many routes between A & B?

each route from A to B

Shortest path to go from A to B

How many routes between A & B?

each route from A to B

Shortest path to go from A to B

How many routes between A & B?

each route from A to B

Shortest path to go from A to B

each route from A to B

Shortest path to go from A to B

Idea of Dijkstra's algorithm

Go one step from A, label the

Idea of Dijkstra's algorithm

Idea of Dijkstra's algorithm

Idea of Dijkstra's algorithm

Idea of Dijkstra's algorithm

Idea of Dijkstra's algorithm

Repeat and repeat ...

Idea of Dijkstra's algorithm

The most straightforward

Algorithms are free from grammatical rules

Programs must follow some syntax rules

Computing the n-th power

Pseudo Code (2)

Pseudo Code (3)

Pseudo Code (4)

Pseudo Code (5)

1. var is first assigned the Computing sum of the

Pseudo Code (6)

Pseudo Code (7)

Computing sum of all input numbers:

We will learn more of them in later examples.

Remove a student, say X

2. Assume sum correctly stores 1+2+...+k after the k-th

Time Complexity Analysis

1. Depend on the speed of the computer

Identify some important operations/steps and

Time Complexity Analysis

¾ If we doubled the input size, how much longer

Why efficiency matters?

Time Complexity Analysis