2023-09-13

TABLE OF CONTENTS
• Course mechanics
• Models of computation

CS341: ALGORITHMS (F23) • Worked example: Bentley’s problem

• Multiple solutions,
Lecture 1
demonstrating different algorithm design techniques
Trevor Brown • Analyzed in different models of computation
https://student.cs.uwaterloo.ca/~cs341
trevor.brown@uwaterloo.ca
1 2

COURSE MECHANICS
• In person
• Lectures
• “Lab” section is for tutorials
• Course website: https://student.cs.uwaterloo.ca/~cs341/
• Syllabus, calendar, policies, slides, assignments…
• Read this and mark important dates.
COURSE MECHANICS
• Keep up with the lectures: Material builds over time…
• Piazza: For questions and announcements.

3 4

ASSESSMENTS TEXTBOOK
• Available for free via library website!
• All sections have same assignments, midterm and final
• You are expected to know
• Sections are roughly synchronized to ensure necessary
content is taught • entire textbook sections,
as listed on course website
• Tentative plan is 5 assignments, midterm, final
• all the material presented in lectures
• See website for grading scheme, etc.
(unless we explicitly say you aren’t
responsible for it)
• Some other textbooks cover some material better… see www

5 6

1
 2023-09-13

ACADEMIC OFFENSES WHY IS CS341 IMPORTANT FOR YOU?

• Beware plagiarism • Algorithms is the heart of CS
• High level discussion • It appears often in later courses
about solutions with individual
students is OK • It dominates technical interviews
• Master this material…
• Don’t take written notes away make your interviews easy!
from such discussions
• Designing algorithms is creative work
• Class-wide discussion of solutions
is not OK (until deadline+2 days) • Useful for some of the more
interesting jobs out there
• And, you want to graduate…
7 8

WHAT IS A COMPUTATIONAL PROBLEM?

• Informally: A description of input,
and the desired output

WHAT IS AN ALGORITHM?
MODELS OF COMPUTATION • Informally: A well-defined
procedure (sequence of steps)
to solve a computational problem Correctness?

9 10

ANALYSIS OF ALGORITHMS But how do we

know how much
time 𝑀 will take
on input 𝐼?
• Every program uses resources
• CPU instructions / cycles → time
Depends on the
• Memory (RAM) → space model of
computation
• Others: I/O, network bandwidth/messages, locks…
(not covered in this course)
• Analysis is the study of how many resources an algorithm uses
• Usually using big-O notation (to ignore constant factors)

11 12

2
 2023-09-13

UNIT COST MODEL

MODELS OF COMPUTATION
• Each variable (or array entry) is a word
• Make analysis possible • Words can contain unlimited bits
• Ones covered in this course • Basic operations on words take O(1) time
• Unit cost model • Read/write a word in O(1)
• Word RAM model • Add two words in O(1)
• Bit complexity model • Multiply two words in O(1)
• Space complexity is the number of words used
(excluding the input)

13 14

BUT SOMETIMES WE CARE ABOUT WORD SIZE WORD RAM MODEL

n in decimal n in binary
• Suppose we want to limit 𝐥𝐨𝐠 𝟐 𝒏 + 𝟏
1 1 0 +1=1
the size of words 2 10 1 +1=2
• Key difference: we care about the size of words
• Must consider how many 3 11 1.58 + 1 = 2 • Words can contain O(lg n) bits,
bits are needed to represent 4 100 2 +1=3
where n is the number of words in the input
5 101
a number 𝒏 2.32 + 1 = 3
6 110 2.58 + 1 = 3 • Word size depends on input size!
7 111 2.81 + 1 = 3
• Intuition: if the input is an array of n words,
Need 𝐥𝐨𝐠 𝟐 𝒏 + 𝟏 bits to store 𝒏 8 1000 3 +1=4
9 1001 3.17 + 1 = 4
a word is large enough to store an array index
i.e., 𝚯(𝐥𝐨𝐠 𝒏) bits 10 1010 3.32 + 1 = 4 • Basic operations on words still take O(1) time
11 1011 3.46 + 1 = 4
12 1100 3.58 + 1 = 4 • (but the values they can contain are limited)

15 16

BIT COMPLEXITY MODEL

• Each variable (or array entry) is a bit string
• Size of a variable x is the number of bits it needs
• It takes O(log v) bits to represent a value v
• So if v is stored in x, the size of x must be Ω(lg 𝑣) bits
• Basic operations are performed on individual bits
• Read/write a bit in O(1)
• Add/multiply two bits in O(1) BENTLEY’S PROBLEM
• Space complexity is the total number of bits used A worked example to demonstrate algorithm design & analysis
(excluding the input)

17 18

3
 2023-09-13

Design: brute force

Try all combinations of 𝒊, 𝒋

And for each combination,
sum over 𝒌 = 𝒊 . . 𝒋
𝒌
Example 1 1 7 4 0 2 1 3 1 Solution: 19
(take all of A[1..8])
Array index 1 2 3 4 5 6 7 8 𝒊 𝒋
1 -7 4 0 2 -1 3 -1
Example 2 -1 -7 -4 -1 -2 -1 -3 -1 Solution: 0
(take no elements of A)
Index 1 2 3 4 5 6 7 8

Example 3 1 -7 4 0 2 -1 3 -1 Solution: 8
(take A[3..7])
Index 1 2 3 4 5 6 7 8

in unit cost model?

19 20

Design: slightly better

brute force

Avoid repeatedly summing over 𝒌 = 𝒊 . . 𝒋

1 -7 4 0 2 -1 3 -1
𝒊=𝒋 𝒋 𝒋 𝒋 Case 1: optimal sol’n
A 9 -3 4 -5 -2 -5 3 -1

is entirely in L L 9 -3 4 -5 R -2 -5 3 -1

Case 2: optimal sol’n

is entirely in R

Case 3: optimal sol’n

A 1 -7 4 0 2 -1 3 0

crosses the partition L 1 -7 4 0 R 2 -1 3 0

in unit cost model?

21 22

WHY 𝐴[𝑖 … 𝑗] IS MAXIMAL

• Suppose not for contradiction
Find: maximum
subarray going over A 1 -7 4 0 2 -1 3 0 • Then some 𝐴[𝑖 ′ … 𝑗 ′ ] that crosses the partition
the middle partition Index 1 2 … n/2 n/2+1 … n has a larger sum
𝒊 𝒋 A
Find 𝒊 that maximizes We can prove 𝐀[𝒊 … 𝒋]
the sum over 𝒊 . . . 𝒏/𝟐 Find 𝒋 that maximizes the is the maximum But both are
𝐿 𝑅
𝒏
sum over 𝟐 + 𝟏 . . . 𝒋
subarray going over impossible!
the middle partition! 𝑖 𝑗
𝐿′ 𝑅′ This sum is bigger
𝑖′ 𝑗′
So either ∑𝐿′ > ∑𝐿
or ∑𝑅′ > ∑𝑅

23 24

4
 2023-09-13

A 9 -3 4 -5 -2 -5 3 -1 A 1 -7 4 0 2 -1 3 0

L 9 -3 4 -5 R -2 -5 3 -1 L 1 -7 4 0 R 2 -1 3 0

maxL = 10 maxR = 3 maxL = 4 maxR = 4

maxM = maxI + maxJ = 5 maxM = maxI + maxJ = 8

A 9 -3 4 -5 -2 -5 3 -1
A 1 -7 4 0 2 -1 3 0
𝑛
Index 1 2 … n/2 +1 … n
Index 1 2 … n/2 𝑛 … n
2 +1
2
maxI = 5 maxJ = 0 maxI = 4
maxJ = 4

Return max( 10, 3, 5 ) = 10 Return max( 4, 4, 8 ) = 8

25 26

ANALYSIS IN THE BIT COMPLEXITY MODEL

(in unit cost model)
• Revisiting Solution 1 Can only add a pair of bits
in O(1) time. How many
How do we analyze this running time? bits are added here?
Need new mathematical techniques!
𝐬𝐢𝐳𝐞 𝐀 𝒌 ∈ 𝐎 𝐥𝐨𝐠 𝐀 𝒌 bits.
Recurrence relations, recursion tree
𝐬𝐢𝐳𝐞 𝐬𝐮𝐦 ∈ ???
methods, master theorem…
𝐬𝐮𝐦 = 𝐀 𝒊 + ⋯ + 𝐀[𝒌 − 𝟏]

This result is really quite good…

but can we do asymptotically better?

so 𝐬𝐢𝐳𝐞 𝐬𝐮𝐦 ∈ 𝐎 𝐥𝐨𝐠 𝐀 𝒊 + ⋯ + 𝐀 𝒌 − 𝟏 bits

How to simplify?

27 28

COMPLEXITY OF ADDITION ADDING SUM AND A[K]

Adding two numbers x+y takes
O(max{size(x), size(y)}) bit operations
Fun fact: the size of x+y can be 1 • Recall size 𝑠𝑢𝑚 ∈ 𝑂 log 𝑘𝑀 , size A 𝑘 ∈ 𝑂 log 𝐴 𝑘 bits
bit larger than either x or y
This can be rewritten O(size(x)+size(y))
(multiplication can double #bits) • Adding them takes
= O(lg x + lg y)
𝑂 log 𝑘𝑀 + log 𝐴 𝑘 bit operations
• And since log 𝐴[𝑘] ≤ log 𝑀 we get:
Let 𝐌 = 𝒎𝒂𝒙{𝑨 𝟏 , … , 𝑨 𝒏 }
𝑂 log 𝑘𝑀 + log 𝑀
𝐬𝐢𝐳𝐞 𝐬𝐮𝐦 ∈ 𝐎 𝐥𝐨𝐠 𝐀 𝒊 + ⋯ + 𝐀 𝒌 − 𝟏
∈ 𝐎 𝐥𝐨𝐠 𝐌 + ⋯ + 𝐌 bits • And the first term asymptotically dominates:
∈ 𝑶 𝐥𝐨𝐠 ((𝐤 − 𝒊)𝐌 bits
𝑶(𝐥𝐨𝐠 𝒌𝑴)
Optional: simplify to 𝑶 𝐥𝐨𝐠 𝐤𝐌

29 30

5
 2023-09-13

ZOOMING OUT TO THE K LOOP ACCOUNTING FOR THE OUTER LOOPS

• The addition happens for all values of 𝑘
Careful to check this
• 𝑘 loop is repeated
𝑗
• Total time for the loop is at most ∑𝑘=𝑖 𝑂(log 𝑘𝑀) does not affect the Θ at most 𝑛2 times
complexity (much).
• Complicated to sum for 𝑘 = 𝑖 … 𝑗 (Check by finding • Each time taking at most
similar Ω result.)
so get an upper bound with 𝑘 = 1 … 𝑛 𝑂(𝑛 log 𝑛𝑀) time
• ∑𝑛𝑘=1 𝑂(log 𝑘𝑀) = 𝑂(log 𝑀 + log 2𝑀 + log 3𝑀 + ⋯ + log 𝑛𝑀) And similarly • So total runtime is
• ⊆ 𝑂 log 𝑛𝑀 + log 𝑛𝑀 + log 𝑛𝑀 + ⋯ + log 𝑛𝑀
for this… 𝑂(𝑛3 log 𝑛𝑀) time

𝒏 Difference is due to
Compare to unit cost model:
(1) growth in variable sizes and
• = 𝑂 𝑛 log 𝑛𝑀 𝑂 𝑛3 time
(2) cost of bitwise addition

31 log-factor difference is common… 32

HOW ABOUT WORD RAM? BENTLEY’S SOLUTIONS: RUNTIME IN PRACTICE

• If each variable fits in a single word, • Consider solutions

the analysis (and result) is as in the unit cost model implemented in C n Sol.4 O(n) Sol.3 O(n lg n) Sol.2 O(n2) Sol.1 O(n3)
100 0 0 0 0
• Some values
• Since there are 𝑛 input words, 1,000 0 0 0 0.12
measured on a 10,000 0 0 0.036 2 minutes
each 𝐴[𝑘] will fit in one word only if size 𝐴 𝑘 ∈ O log 𝑛 Threadripper 3970x 100,000 0 0.002 3.582 33 hours
1M 0.001 0.017 6 minutes 4 years
• i.e., if O log 𝐴 𝑘 = O(log 𝑛) • Red values 10M 0.012 0.195 12 hours 3700 years
100M
• If a variable is too big to fit in a word, extrapolated from 1 billion
0.112 2.168 50 days 3.7M years
1.124 24.57 1.5 years > age of life
it is stored in multiple words, measurements 10 billion 19.15 5 minutes 150 years > age of universe

and analysis looks more like bit complexity model • 0 represents time
under 0.01s

33 34

HOMEWORK: BIG-O REVIEW & EXERCISES

𝑓 𝑛 ∈ 𝑂(𝑔 𝑛 )

35 36

6
 2023-09-13

𝑓 𝑛 𝑓 𝑛 ∈ Θ(𝑔 𝑛 ) 𝑓 𝑛 ∈ O(𝑔 𝑛 )

𝑓 𝑛 ∈ Ω(𝑔 𝑛 ) 𝑔 𝑛
𝑔 𝑛 ∈ Θ(𝑓 𝑛 ) 𝑓 𝑛 ∈ Ω(𝑔 𝑛 )

𝑂+Ω=Θ

37 38

EXERCISE
• Which of the following are true?
𝑓 𝑛 ∈o 𝑔 𝑛 • 𝑛2 ∈ 𝑂(𝑛3 )
But NOT
implies
vice versa
𝑓 𝑛 ∈ O(𝑔 𝑛 ) • 𝑛2 ∈ 𝑜(𝑛3 )
• 𝑛3 ∈ 𝜔(𝑛3 )
• log 𝑛 ∈ 𝑜(𝑛)
𝑓 𝑛 ∈𝜔 𝑔 𝑛 • 𝑛 log 𝑛 ∈ Ω(𝑛)
But NOT
implies
vice versa • 𝑛 log 𝑛2 ∈ 𝜔(𝑛 log 𝑛)
𝑓 𝑛 ∈ Ω(𝑔 𝑛 )
• 𝑛 ∈ Θ(𝑛 log 𝑛)

39 40

EXERCISE
• Which of the following are true?
• 𝑛2 ∈ 𝑂(𝑛3 ) YES
• 𝑛2 ∈ 𝑜(𝑛3 ) YES
• 𝑛3 ∈ 𝜔(𝑛3 ) NO
• log 𝑛 ∈ 𝑜(𝑛) YES
• 𝑛 log 𝑛 ∈ Ω(𝑛) YES COMPARING GROWTH RATES
• 𝑛 log 𝑛2 ∈ 𝜔(𝑛 log 𝑛) NO
• 𝑛 ∈ Θ(𝑛 log 𝑛) NO

41 42

7
 2023-09-13

LIMIT TECHNIQUE
FOR COMPARING GROWTH RATES

43 44

LIMIT RULES 1/3 LIMIT RULES 2/3

All of the identities shown

hold only if the limits exist

45 46

L’HOSPITAL’S RULE
LIMIT RULES 3/3 • Often we take the limit of 𝑔
𝑓 𝑛
where
𝑛
both 𝑓(𝑛) and 𝑔(𝑛) tend to ∞, or
both 𝑓(𝑛) and 𝑔(𝑛) tend to 0
Limit of an Exponential Function • Such limits require L’Hospital’s rule
lim 𝑓 𝑥
lim 𝑏 𝑓 𝑥 = 𝑏 𝑥→𝑎 • This rule says the limit of 𝒇(𝒏)/𝒈(𝒏) in this case
𝑥→𝑎
is the same as the limit of the derivative
Limit of a Logarithm of a Function 𝑓 𝑛
𝑑
𝑓 𝑛
lim log 𝑏 𝑓(𝑥) = log 𝑏 lim 𝑓 𝑥 • In other words, lim = lim 𝑑𝑛
𝑑
𝑥→𝑎 𝑥→𝑎 𝑛→∞ 𝑔 𝑛 𝑛→∞ 𝑔 𝑛
𝑑𝑛

(Where base 𝑏 > 0)

47 48

8
 2023-09-13

USING THE LIMIT METHOD: EXERCISE 1 USING THE LIMIT METHOD: EXERCISE 2
• Compare growth rate of 𝑛2 and 𝑛2 − 7𝑛 − 30 • Compare growth rate of ln 𝑛 2
and 𝑛1/2
𝑛2 −7𝑛−30 𝑑
• lim ln 𝑛 2 ln 𝑛 2
𝑛→∞ 𝑛2 • lim = lim𝑑𝑛
𝑑 1/2
𝑛→∞ 𝑛1/2 𝑛→∞ 𝑛
7 30 𝑑𝑛
• = lim (1 − 𝑛 − 𝑛2 )
𝑛→∞
• =1
• So 𝑛2 − 7𝑛 − 30 ∈ Θ(𝑛2 )

49 50

Try these at home…

USING THE LIMIT METHOD: EXERCISE 2
• Compare growth rate of ln 𝑛 2
and 𝑛1/2
𝑑
𝑑 4 ln 𝑛
ln 𝑛 2 • = lim 𝑑𝑛
• lim𝑑𝑛
𝑑 1/2 𝑛→∞
𝑑 1/2
𝑛
𝑛→∞ 𝑛 𝑑𝑛
𝑑𝑛
4/𝑛
2 ln 𝑛 (1/𝑛) • = lim
• = lim 1 −1/2
1
𝑛→∞ 2𝑛−1/2
𝑛→∞ 𝑛
2
8
4 ln 𝑛 • = lim
• = lim 𝑛→∞ 𝑛1/2
𝑛→∞ 𝑛1/2
• =0
• So, ln 𝑛 2
∈ 𝑜(𝑛1/2 )

51 52

This is included for

your notes

SUMMATIONS
AND SEQUENCES

53 54

9
 2023-09-13

SEQUENCES

55 56

SEQUENCES CONTINUED This is included for This is included for

your notes your notes

57 58

LOGARITHM RULES

59 60

10
 2023-09-13

BASE OF LOGARITHM DOES NOT MATTER!

• Big-O notation does not distinguish between log bases

• Proof:
• Fix two constant logarithm bases b and c
• From log rules, we can change from log 𝑐 to log𝑏
by using formula: logb 𝑥 = log 𝑐 𝑥 / log 𝑐 𝑏
We typically omit the base, LOOP ANALYSIS
• But log𝑐 𝑏 is a constant! and just write 𝜣 𝐥𝐨𝐠 𝒙
• So log 𝑐 𝑥 ∈ Θ(log 𝑏 𝑥) for this reason

61 62

META-ALGORITHM FOR ANALYZING LOOPS TWO BIG-O ANALYSIS STRATEGIES

• Identify operations that require only constant time
• Strategy 1
• The complexity of a loop is the sum of
the complexities of all iterations • Prove a O-bound and a matching Ω-bound
separately to get a Θ -bound. Often easier
• Analyze independent loops separately and add the results
• Strategy 2 (but not always)
• If loops are nested, it often helps to start at the innermost,
and proceed outward… but, • Use Θ-bounds throughout the analysis and thereby obtain
a Θ-bound for the complexity of the algorithm
• sometimes you must express several nested loops together in a
single equation (using nested summations),
• and actually evaluate the nested summations… (can be hard)

63 64

Strategy 1: big-O and big-Ω bounds

EXAMPLE 1

65 66

11
 2023-09-13

Strategy 2: use Θ-bounds throughout the analysis EXAMPLE 2

𝑂(1) Consider this loop alone…
number of loop iterations?

𝑂(1) 𝑗 starts at 𝑖 and is repeatedly divided by

2… this can happen only 𝚯(𝐥𝐨𝐠 𝒊) times

𝑂(1) So inner loop has runtime 𝚯(𝐥𝐨𝐠 𝒊)

𝑂(1)
And the entire inner loop is
executed for 𝑖 = 1,2, … , 𝑛
𝑂(1)

So, we have 𝑻 𝒏 ∈ 𝚯 ∑𝒏𝒊=𝟏 𝐥𝐨𝐠 𝒊

𝑛 𝑛 𝑛 𝑛
𝑛
𝑇 𝑛 ∈ O ෍ log 𝑖 ⊆ O ෍ log 𝑛 ⊆ 𝑶(𝒏 𝐥𝐨𝐠 𝒏) 𝑇 𝑛 ∈ Ω ෍ log 𝑖 ⊆ Ω ෍ log ⊆ 𝜴(𝒏 𝐥𝐨𝐠 𝒏)
𝒏
2
𝑖=1 𝑖=1 𝑖=1 𝑖=𝟐

67 68

EXAMPLE 3 (BENTLEY’S PROBLEM, SOLUTION 1)

Try to analyze this yourself!

One possible solution is
given in these slides…

… ANOTHER EXERCISE
IN LOOP ANALYSIS?

69 70

𝑛
Strategy 1: big-O and big-Ω bounds 𝑂(1)
෍…
Proving a big-Ω bound…
𝒊=1 𝑛 𝑛
𝑛 𝑛 𝒋
Recall: 𝑇 𝑛 ∈ 𝚯 ෍ ෍(𝑗 − 𝑖)
𝑇 𝑛 ∈ Θ 1 + ෍ ෍ Θ 1 + ෍ Θ 1 + Θ(1) 𝑂(1)
𝑖=1 𝑗=𝒊
𝑖=1 𝑗=𝒊 𝑘=𝒊
𝑂(1) 𝑛 𝑛
𝑛 𝑛 𝑛 𝑛
𝑇 𝑛 ∈ 𝛀 ෍ ෍(𝑗 − 𝑖)
𝑇 𝑛 ∈ ෍ ෍ 𝚯 𝒋 − 𝒊 ∈ 𝚯 ෍ ෍(𝑗 − 𝑖) Intuition: 𝑗 − 𝑖 is 𝛀(𝒏) in some iterations. How
𝑖=1 𝑗=𝒊
𝑖=1 𝑗=𝒊 𝑖=1 𝑗=𝒊 𝑂(1) many iterations? Lots?
𝒏/𝟐 𝑛
𝑛 𝑛 𝑛 𝑛
≥ Ω ෍ ෍(𝑗 − 𝑖) To get a good Ω-bound, we ask questions like:
𝑇 𝑛 ∈ 𝑶 ෍ ෍(𝑗 − 𝑖) ≤ 𝑂 ෍ ෍ 𝒏 When do our loops have many iterations?
𝑖=1 𝑗=𝒊
𝑖=1 𝑗=𝒊 𝑖=1 𝑗=𝒊 When is our dominant term large?
𝑛/2 𝑛
𝑛 𝑛
This is the maximum number of ≥ Ω ෍ ෍ (𝑗 − 𝑖) Many iterations: when our 𝒋 loop does 𝛀(𝒏)
≤ 𝑂 ෍෍𝑛 iterations that could be iterations! For example, when 𝒊 ≤ 𝒏/𝟐…
𝑖=1 𝑗=𝟑𝒏/𝟒
𝑖=1 𝒋=𝟏 performed in this loop
Large dominant term: when 𝒋 is much
𝑻 𝒏 ∈ 𝑶(𝒏𝟑 ) larger than 𝒊 (i.e., by a factor of n)
71 72

12
 2023-09-13

Proving a big-Ω bound… continued

𝑛/2 𝑛

Recall: 𝑇 𝑛 ∈ Ω ෍ ෍ (𝑗 − 𝑖)
𝑖=1 𝑗=𝟑𝒏/𝟒
𝑛/2 𝑛
𝟑𝒏 𝒏
≥Ω ෍ ෍ −
𝟒 𝟐
𝑖=1 𝑗=3𝑛/4 Smallest possible value of 𝑗 − 𝑖
𝑛/2 𝑛 for these bounds on 𝒊, 𝒋

= Ω ෍ ෍ 𝒏/𝟒
We will perform at least this much
𝑖=1 𝑗=3𝑛/4
work in every iteration!
𝒏 𝒏 𝑛
≥Ω ⋅ ⋅ = 𝛀(𝒏𝟑 ) This term does not depend on the
𝟐 𝟒 4
loop indexes, so just multiply by the
total number of loop iterations…

Since we have 𝑂 𝑛 3 and Ω 𝑛 3 , we have proved 𝚯 𝒏𝟑

73 74

BONUS
• Study-song of the day
• Tool - Descending
• youtu.be/PcSoLwFisaw

75

13