CS 430: Introduction to Algorithms

1
 The Joy of Algorithms

"Algorithms: a common language for nature, human, and computer."

— Avi Wigderson (2006), A Worldview through the Computational Lens

"For me, great algorithms are the poetry of computation.

Just like verse, they can be terse, allusive, dense, and even
mysterious. But once unlocked, they cast a brilliant new
light on some aspect of computing."

— Francis Sullivan (2000), The Joy of Algorithms

2
 Etymology [Knuth, TAOCP]

 Algorism = process of doing arithmetic using Arabic numerals.

 A misperception: algiros [painful] + arithmos [number].

 True origin: Abu 'Abd Allah Muhammad ibn Musa al-Khwarizm was a
famous 9th century Persian textbook author who wrote Kitab al-
jabr wa'l-muqabala, which evolved into today's high school algebra
text.

3
 What you will learn in this course

 Abstract messy, real-world problems by clean algorithmic problems

 Solve those algorithmic problems
 examine problem structure
 pick and choose from different algorithm design strategies
 Analyze algorithms: quantify different design choices

Prerequisites:
CS 330: Discrete Structures https://moss.cs.iit.edu/cs330
CS 331: Data Structures and Algorithms https://moss.cs.iit.edu/cs331

4
 … and how

• Attending lectures

• Reading the textbook

• Solving lots of practice problems

• About 6 assignments: 40%
• One project: 15%
• Two in-class exams: midterm 20%, final 25%

5
 Administrative matters

Instructor/TAs Email Office hours Room

Peng-Jun Wan wan@iit.edu 4-6pm Wed. SB 236F
Gong Chen gchen31@hawk.iit.edu
Adil Tanveer atanveer2@hawk.iit.edu
Xue Zhang xzhang143@hawk.iit.edu

Course website: blackboard.iit.edu

• Lecture slides and assignments will be posted here.
• Your completed assignments and exams in pdf format will be
submitted here.
• Questions, discussions, announcements
• Also restricted material such as solutions to exercises.
• No late submission without prior permission

6
 You’ve already learnt many algorithms in the past …

“The Euclidean algorithm is the granddaddy of all algorithms, because it

is the oldest nontrivial algorithm that has survived to the present day.”
— Donald Knuth

7
 Algorithms in daily lives …

Coin Changing. Given currency denominations: 1, 5, 10, 25, 100, devise a

method to pay amount to customer using fewest number of coins.

Ex: 34¢.

Cashier's algorithm. At each iteration, add coin of the largest value

that does not take us past the amount to be paid.

Ex: $2.89.

8
 Algorithms in daily lives …

Top 2 players. Given n players with transitively ranked skills, devise a

game to find the top 2 using fewest matches.

7 8

7 5 6 8

1 7 3 5 2 6 4 8

Tournament method: matches using prune and conquer.

Q: selection of best & worst in fewest matches?

9
 Recreational puzzles

Counterfeit coin problem: Given 9 coins, one of them counterfeit and

lighter, find the counterfeit coin in two weighings on a balance scale

@wiki/Balance_puzzle

10
 If you are bored, try…

Q1: Given n coins, one of them fake and lighter, find the minimum
number of weighings that will guarantee finding the fake coin.

Q2: Given n 3 coins, one of them fake and lighter or heavier, find the
minimum number of weighings that will guarantee finding the fake coin
and figuring out whether it is heavier or lighter .

11
Appetizer: elevator movements

An elevator is in a -story building.

It can go up multiple floors at a time,
but can only go down one floor at a time.

In movements of the elevator,

how many floors can it travel upwards?

12
 Counting using potential/state function

Potential story where the elevation stops

UPs DOWNs

Counting:

Useful when total sum of bounds for each step.

Amortized analysis: If , it travels at most floors, or fewer

than floors per move on average.

13
 Gambler's losing bets

John is a gambler. On each bet, he can lose up to and win up to .

When he loses, he always loses at least .

Suppose that
 John starts with at most
 the number of times that he wins bets is at most .
What is an upper bound on the number of times that John loses a bet?

Final amount Initial Amount Winnings Losses

Losses Initial Amount Winnings Final amount

Losses .

14
Five Representative Problems [KT 1.2]
 Interval scheduling

Input. Set of jobs with start times and finish times.

Goal. Find maximum cardinality subset of non-overlapping jobs.

h Time
0 1 2 3 4 5 6 7 8 9 10 11

16
 Weighted interval scheduling

Input. Set of jobs with start times, finish times, and weights.
Goal. Find maximum weight subset of non-overlapping jobs.

23

12

20

26

13

20

11

16 Time
0 1 2 3 4 5 6 7 8 9 10 11

17
 Assignment of jobs to machines

Input. (job, machine) pairs.

Goal. Assign maximum number of jobs.

A 1

B 2

C 3

D 4

E 5

18
 Selection of conflict-free jobs

Input. Conflicting pair of jobs.

Goal. Find maximum number of conflict-free jobs.

1 2

4 5
3

6 7

19
 Competitive job selection

Input. Conflicting pair of jobs, and a weight on each job.

Game. Two competing players alternate in selecting jobs. Not allowed
to select a job if any conflicting job has been selected.

Goal. Select a maximum weight subset of jobs.

10 1 5 15 5 1 5 1 15 10

Second player can guarantee 20, but not 25.

20
 Five Representative Problems

Variations on a theme: independent set.

Interval scheduling: n log n greedy algorithm.

Weighted interval scheduling: n log n dynamic programming algorithm.
Assignment of jobs to machines: nk max-flow based algorithm.
Selection of conflict-free jobs: NP-complete.
Competitive job selection: PSPACE-complete.

21
Asymptotic Order of Growth [KT 2.2]
 Suppress constant factors and lower-order terms

Upper bounds. T(n) is O(f(n)) if there exist constants c > 0 and n0  0

such that for all n  n0 we have T(n)  c · f(n).

Lower bounds. T(n) is (f(n)) if there exist constants c > 0 and n0  0

such that for all n  n0 we have T(n)  c · f(n).

Tight bounds. T(n) is (f(n)) if T(n) is both O(f(n)) and (f(n)).

Ex: T(n) = 32n2 + 17n + 32.

 T(n) is O(n2), O(n3), (n2), (n), and (n2) .
 T(n) is not O(n), (n3), (n), or (n3).

23
 Semantics of asymptotic order

Slight abuse of notation. T(n) = O(f(n)).

 Asymmetric:
– f(n) = 5n3; g(n) = 3n2
– f(n) = O(n3) = g(n)
– but f(n)  g(n).
 Better notation: T(n)  O(f(n)).

Meaningless statement. Any comparison-based sorting algorithm

requires at least O(n log n) comparisons.
 Statement doesn't "type-check."
 Use  for lower bounds.

24
 Properties

Transitivity.
 If f = O(g) and g = O(h) then f = O(h).
 If f = (g) and g = (h) then f = (h).
 If f = (g) and g = (h) then f = (h).

Additivity.
If f = O(h) and g = O(h) then f + g = O(h).
If f = (h) and g = (h) then f + g = (h).
If f = (h) and g = O(h) then f + g = (h).

25
 Asymptotic bounds for some common functions

Polynomials. a0 + a1n + … + adnd is (nd) if ad > 0.

Polynomial time. Running time is O(nd) for some constant d independent

of the input size n.

Logarithms. O(log a n) = O(log b n) for any constants a, b > 0.

can avoid specifying the

base
Logarithms. For every x > 0, log n = O(nx).

log grows slower than every polynomial

Exponentials. For every r > 1 and every d > 0, nd = O(rn).

every exponential grows faster than every polynomial

26
Quickly growing function and its slowly growing inverse

27
 The tower function and iterated logarithm

n T(n) n log*n
1 2 [0,2] 1
2 4 [3,4] 2
3 16 [ 5 , 16 ] 3
4 65,536 [ 17 , 65,536 ] 4
5 265,536 [ 65,537 , 265,536 ] 5

For all practical applications, log*n

28
Nesting / Repeated application

29
Ackermann’s function (one of many variations)

30
Ackermann’s function vs. Tower function

31
Inverse Ackermann function

is the inverse of the function

32
In-class exercise: Sort in ascending order

log*n

comparison strategy:
• divide and conquer
• take the logarithm

33
 Stirling formula

Approximating sums by integrals

Recurrence on counting [CLRS 5.2]
 Recurrence Relation: Invariant total subproblem sizes

Solution.

Assorted proofs. We describe several ways to prove this recurrence.

Initially we assume n is a power of 2 and replace  with =.

36
 Proof by recursion tree

0, if 𝑛 = 1
𝑇 𝑛 =
2𝑇 𝑛/2 + 𝑛, otherwise

n n

n/2 n/2 2(n/2)

n/4 n/4 n/4 n/4 4(n/4)

log2n
...
n / 2k 2k (n / 2k)

...

2 2 2 2 2 2 2 2 n/2 (2)

n log2n

37
 Proof by telescoping

0, if 𝑛 = 1
𝑇 𝑛 =
2𝑇 𝑛/2 + 𝑛, otherwise

For n > 1:

38
 Proof by induction

0, if 𝑛 = 1
𝑇 𝑛 =
2𝑇 𝑛/2 + 𝑛, otherwise

 Base case: n = 1.
 Inductive hypothesis: T(n) = n log2 n.
 Goal: show that T(2n) = 2n log2 (2n).

39
 Induction with floor/ceiling

 Base case: n = 1.
 Define n1 = n / 2 , n2 = n / 2.
 Induction step: assume true for 1, 2, ... , n–1.

40
 Recurrence Relation: Increasing total subproblem sizes

0, if 𝑛 = 1
𝑇 𝑛 ≤
3𝑇 𝑛/2 + 𝑛, otherwise

n n

n/2 n/2 n/2 3(n/2)

log2n n/4 n/4 n/4 n/4 n/4 n/4 n/4 n/4 n/4 9(n/4)

... ...

n / 2k
... ...

2 2 2 2 2 2 2 2 3 lg n (2)

41
 The master theorem for balanced subsproblems

Suppose , where
• is the number of subproblems
• is the factor by which input size shrinks
• is the work to create and combine all the subproblems.

42
 Unbalanced subproblems

n/5 7n/10

n/25 7n/50 7n/50 49n/100

...

Key fact: the total subproblem size drops at the factor 9/10, so does
the total work at each level.

43
CS 330 Topics
 • Introductory Logic propositional logic, predicate logic, proof
methodologies, examples of algorithm correctness

 • Sets, Functions and relations - sets, set operations, functions,

summations, growth of functions, equivalence relations, countable and
uncountable sets, examples of algorithm analysis

 • Counting Methods permutations, combinations, discrete probability,

pigeonhole principle

 • Advanced counting inclusion-exclusion, recurrence relations,

methods of solving recurrences, examples from computer sciences

 • Introduction to Graphs - trees, connectivity, Eulerian traversals,

minimum spanning tree, planarity, Euler formula, matchings.

 • Languages, automata, and Turing machines, computational

complexity
 CS 331: Data Structures and Algorithms

https://moss.cs.iit.edu/cs331/syllabus.html

• lists (unordered and ordered),

• stacks,
• queues,
• expression trees,
• binary search trees,
• heaps, and
• hash tables

45