Algorithms Design and Analysis

Introduction
Evaluation
• Your evaluation in this course is based on three components:
• 20% Minor 1
• 20% Minor 2
• 30% Major
• 30% Homeworks, Quizzes and Projects
Evaluation
• A student who misses exams must provide:
• A Verification of Illness form indicating a severe illness, or
• Other formal documentation, as appropriate
• No re-minors: Weighted average of the Major and the other
Minor will be awarded
• Re-major can take its time. Incomplete may be awarded.
• Late Homeworks:
• 20% penalty per late day.
• 0 will be awarded after the fifth day
Evaluation
• The graders may penalize for unreadable homeworks
• The graders may award bonus points for “nice” homeworks
• The graders may not grade all the assigned problems in the
homeworks
• The graders may not grade all the homeworks
Academic Offences
• Academic Offences include, but are not limited to:
• Infringing unreasonably on the work of other members
• E.g., disrupting classes
• Cheating
• Plagiarism
• Misrepresentations
Introduction
ADA

1-5
What is an algorithm?
• An algorithm is a sequence of unambiguous instructions for
solving a problem, i.e., for obtaining a required output for any
legitimate input in a finite amount of time.

problem

algorithm

input “computer” output

1-6
Algorithm
• An algorithm is a sequence of unambiguous instructions for
solving a problem, i.e., for obtaining a required output for any
legitimate input in a finite amount of time.

1-7
Historical Perspective
• Euclid’s algorithm for finding the greatest common divisor

• Muhammad ibn Musa al-Khwarizmi – 9th century

mathematician
www.lib.virginia.edu/science/parshall/khwariz.html

1-8
Notion of algorithm

problem

algorithm

input “computer” output

Algorithmic solution

1-9
Example of computational problem: sorting
• Statement of problem:
• Input: A sequence of n numbers <a1, a2, …, an>
• Output: A reordering of the input sequence <a´1, a´2, …, a´n>
so that a´i ≤ a´j whenever i < j
• Instance: The sequence <5, 3, 2, 8, 3>
• Algorithms:
• Selection sort
• Insertion sort
• Merge sort
• (many others)

1-10
Selection Sort
• Input: array a[1],…,a[n]
• Output: array a sorted in non-decreasing order
• Algorithm:

for i=1 to n
swap a[i] with smallest of a[i],…a[n]

* see also pseudocode, section 3.1

1-11
Some Well-known Computational Problems
• Sorting
• Searching
• Shortest paths in a graph
• Minimum spanning tree
• Primality testing
• Traveling salesman problem
• Knapsack problem
• Chess
• Towers of Hanoi
• Program termination

1-12
Basic Issues Related to Algorithms
• How to design algorithms
• How to express algorithms
• Proving correctness
• Efficiency
• Theoretical analysis
• Empirical analysis
• Optimality

1-13
Algorithm design strategies
• Brute force
• Divide and conquer
• Decrease and conquer
• Transform and conquer
• Greedy approach
• Dynamic programming
• Backtracking and Branch and bound
• Space and time tradeoffs

1-14
Analysis of Algorithms
• How good is the algorithm?
• Correctness
• Time efficiency
• Space efficiency

• Does there exist a better algorithm?

• Lower bounds
• Optimality

1-15
What is an algorithm?
• Recipe, process, method, technique, procedure, routine,…
with following requirements:
• Finiteness
• terminates after a finite number of steps
• Definiteness
• rigorously and unambiguously specified
• Input
• valid inputs are clearly specified
• Output
• can be proved to produce the correct output given a valid input
• Effectiveness
• steps are sufficiently simple and basic

1-16
Why study algorithms?
• Theoretical importance
• the core of computer science

• Practical importance
• A practitioner’s toolkit of known algorithms
• Framework for designing and analyzing algorithms for new
problems

1-17
Euclid’s Algorithm
• Problem: Find gcd(m,n), the greatest common divisor of two
nonnegative, not both zero integers m and n
• Examples:
• gcd(60,24) = 12, gcd(60,0) = 60, gcd(0,0) = ?
• Euclid’s algorithm is based on repeated application of equality
• gcd(m,n) = gcd(n, m mod n)
• until the second number becomes 0, which makes the
problem trivial.
• Example: gcd(60,24) = gcd(24,12) = gcd(12,0) = 12

1-18
Two descriptions of Euclid’s algorithm
• Step 1
• If n = 0, return m and stop; otherwise go to Step 2
• Step 2
• Divide m by n and assign the value fo the remainder to r
• Step 3
• Assign the value of n to m and the value of r to n.
• Go to Step 1.

• while n ≠ 0 do
• r ← m mod n
• m← n
•n←r
• return m
1-19
Other methods for computing gcd(m,n)

• Consecutive integer • Step 3

checking algorithm • Divide n by t.
• Step 1 • If the remainder is 0, return t
• Assign the value of min{m,n} and stop;
to t • otherwise, go to Step 4
• Step 2 • Step 4
• Divide m by t. • Decrease t by 1 and go to
• If the remainder is 0, go to Step 2
Step 3;
• otherwise, go to Step 4

1-20
Other methods for gcd(m,n) [cont.]
• Middle-school procedure
• Step 1
• Find the prime factorization of m
• Step 2
• Find the prime factorization of n
• Step 3
• Find all the common prime factors
• Step 4
• Compute the product of all the common prime factors and
return it as gcd(m,n)
• Is this an algorithm?

1-21
Sieve of Eratosthenes
• Input: Integer n ≥ 2
• Output: List of primes less than or equal to n
• for p ← 2 to n do A[p] ← p
• for p ← 2 to n do
• if A[p]  0 //p hasn’t been previously eliminated from the list
j←p*p
• while j ≤ n do
• A[j] ← 0 //mark element as eliminated
•j←j+p
• Example: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1-22
Why study algorithms?
• Theoretical importance
• the core of computer science

• Practical importance
• A practitioner’s toolkit of known algorithms
• Framework for designing and analyzing algorithms for new
problems

1-23
Two main issues related to algorithms
• How to design algorithms
• How to analyze algorithm efficiency

1-24
Algorithm design techniques/strategies
• Brute force
• Divide and conquer
• Decrease and conquer
• Transform and conquer
• Space and time tradeoffs
• Greedy approach
• Dynamic programming
• Iterative improvement
• Backtracking
• Branch and bound

1-25
Analysis of algorithms
• How good is the algorithm?
• time efficiency
• space efficiency

• Does there exist a better algorithm?

• lower bounds
• optimality

1-26
Important problem types
• sorting
• searching
• string processing
• graph problems
• combinatorial problems
• geometric problems
• numerical problems

1-27
Algorithm Efficiency

ADA
Analysis of algorithms
• Issues:
• correctness
• time efficiency
• space efficiency
• optimality

• Approaches:
• theoretical analysis
• empirical analysis
Theoretical analysis of time efficiency
• Time efficiency is analyzed by determining the number of
repetitions of the basic operation as a function of input size
• Basic operation: the operation that contributes most towards
the running time of the algorithm

T(n) ≈ cop C(n)

Theoretical analysis of time efficiency
• Time efficiency is analyzed by determining the number of
repetitions of the basic operation as a function of input size
• Basic operation: the operation that contributes most towards
the running time of the algorithm
Theoretical analysis of time efficiency
• Time efficiency is analyzed by determining the number of
repetitions of the basic operation as a function of input size
• Basic operation: the operation that contributes most towards
the running time of the algorithm
input size

T(n) ≈ cop C(n)

running time execution time Number of times

for basic operation basic operation is
executed
Input size and basic operation examples
Problem Input size measure Basic operation

Searching for key Number of list’s

Key comparison
in a list of n items items, i.e. n

Matrix dimensions
Multiplication of Multiplication of
or total number of
two matrices two numbers
elements

Checking n’size = number of

primality of a digits (in binary Division
given integer n representation)

Typical graph #vertices and/or Visiting a vertex or

problem edges traversing an edge
Empirical analysis of time efficiency
• Select a specific (typical) sample of inputs
• Use physical unit of time (e.g., milliseconds)
• or
• Count actual number of basic operation’s executions
• Analyze the empirical data
Best-case, average-case, worst-case
• For some algorithms efficiency depends on form of input:
• Worst case: Cworst(n) – maximum over inputs of size n
• Best case: Cbest(n) – minimum over inputs of size n
• Average case: Cavg(n) – “average” over inputs of size n
• Number of times the basic operation will be executed on typical
input
• NOT the average of worst and best case
• Expected number of basic operations considered as a random
variable under some assumption about the probability
distribution of all possible inputs
Example: Sequential search

• Worst case
• Best case
• Average case
Types of formulas for basic operation’s count
• Exact formula
• e.g., C(n) = n(n-1)/2
• Formula indicating order of growth with specific multiplicative
constant
• e.g., C(n) ≈ 0.5 n2
• Formula indicating order of growth with unknown multiplicative
constant
• e.g., C(n) ≈ cn2
Order of growth
• Most important: Order of growth within a constant multiple as
n→∞
• Example:
• How much faster will algorithm run on computer that is twice as
fast?
• How much longer does it take to solve problem of double input
size?
Values of some important functions as n → 
Asymptotic order of growth
• A way of comparing functions that ignores constant factors
and small input sizes
• O(g(n)): class of functions f(n) that grow no faster than g(n)
• Θ(g(n)): class of functions f(n) that grow at same rate as g(n)
• Ω(g(n)): class of functions f(n) that grow at least as fast as
g(n)
Big-oh
Big-omega
Big-theta
Establishing the order of growth
• Definition: f(n) is in O(g(n)) if order of growth of f(n) ≤ order of
growth of g(n) (within constant multiple),
• i.e., there exist positive constant c and non-negative integer
n0 such that
• f(n) ≤ c g(n) for every n ≥ n0
• Examples:
• 10n is O(n2)
• 5n+20 is O(n)
Properties of asymptotic order of growth
• f(n)  O(f(n))
• f(n)  O(g(n)) iff g(n) (f(n))
• If f (n)  O(g (n)) and g(n)  O(h(n)),
• then f(n)  O(h(n))
• Note similarity with a ≤ b
• If f1(n)  O(g1(n)) and f2(n)  O(g2(n)),
• then f1(n) + f2(n)  O(max{g1(n), g2(n)})
Establishing order of growth using limits
• Given functions, f(n) and g(n), how to decide for whether
• f(n) = O(g(n)) or
• f(n) = Θ(g(n)) or
• f(n) = Ω(g(n))
Establishing order of growth using limits
• Given functions, f(n) and g(n), how to decide for whether
• f(n) = O(g(n)) or
• f(n) = Θ(g(n)) or
• f(n) = Ω(g(n))

• Examples:
• 10n vs. n2
• n(n+1)/2 vs. n2
Establishing order of growth using limits
• Given functions, f(n) and g(n), how to decide for whether
• f(n) = O(g(n)) or
• f(n) = Θ(g(n)) or
• f(n) = Ω(g(n))

• Examples:
• 10n vs. n2
• n(n+1)/2 vs. n2

0 order of growth of T(n) < order of growth of g(n)

lim T(n)/g(n) = c>0 order of growth of T(n) = order of growth of g(n)
n→∞ ∞ order of growth of T(n) > order of growth of g(n)
L’Hôpital’s rule and Stirling’s formula
• L’Hôpital’s rule:
• If limn→ f(n) = limn→ g(n) =  and
• the derivatives f´, g´ exist, then
• limn→ ( f(n) / g(n) ) = limn→ ( f’(n) / g’(n) )

• Stirling’s formula: n!  (2n)1/2 (n/e)n

• Example: log n vs. n
• Example: 2n vs. n!
Orders of growth of some functions
• All logarithmic functions loga n belong to the same class
(log n) no matter what the logarithm’s base a > 1 is
• All polynomials of the same degree k belong to the same
class: aknk + ak-1nk-1 + … + a0  (nk)
• Exponential functions an have different orders of growth for
different a’s
• order log n < order n (>0) < order an < order n! < order nn
Basic asymptotic efficiency classes
1 constant
log n logarithmic
n linear
n log n n-log-n
n2 quadratic
n3 cubic
2n exponential
n! factorial
Time efficiency of nonrecursive algorithms
• General Plan for Analysis
• Decide on parameter n indicating input size
• Identify algorithm’s basic operation
• Determine worst, average, and best cases for input of size n
• Set up a sum for the number of times the basic operation is
executed
• Simplify the sum using standard formulas and rules (see
Appendix A)
Useful summation formulas and rules
• liu 1 = 1+1+…+1 = u - l + 1
• In particular, 1in 1 = n - 1 + 1 = n  (n)
• 1in i = 1+2+…+n = n(n+1)/2  n2/2  (n2)
• 1in i2 = 12+22+…+n2 = n(n+1)(2n+1)/6  n3/3  (n3)
• 0in ai = 1 + a +…+ an = (an+1 - 1)/(a - 1) for any a  1
• In particular, 0in 2i = 20 + 21 +…+ 2n = 2n+1 - 1  (2n )
• (ai ± bi ) = ai ± bi
• cai = cai
• liu ai = lim ai + m+1iu ai
Example 1: Maximum element
Example 2: Element uniqueness problem
Example 3: Matrix multiplication
Example 4: Gaussian elimination
• Algorithm GaussianElimination(A[0..n-1,0..n])
• //Implements Gaussian elimination of an n-by-(n+1) matrix A
for i  0 to n - 2 do
for j  i + 1 to n - 1 do
for k  i to n do
A[j,k]  A[j,k] - A[i,k]  A[j,i] / A[i,i]
• Find the efficiency class and a constant factor improvement.
Example 5: Counting binary digits

• It cannot be investigated the way the previous examples are.

Plan for Analysis of Recursive Algorithms
• Decide on a parameter indicating an input’s size.
• Identify the algorithm’s basic operation.
• Check whether the number of times the basic op. is executed
may vary on different inputs of the same size. (If it may, the
worst, average, and best cases must be investigated
separately.)
• Set up a recurrence relation with an appropriate initial
condition expressing the number of times the basic op. is
executed.
• Solve the recurrence (or, at the very least, establish its
solution’s order of growth) by backward substitutions or
another method.
Example 1: Recursive evaluation of n!
• Definition:
• n ! = 1  2  … (n-1)  n for n ≥ 1 and 0! = 1
• Recursive definition of n!:
• F(n) = F(n-1)  n for n ≥ 1 and F(0) = 1

• Size:
• Basic operation:
• Recurrence relation:
Solving the recurrence for M(n)
• M(n) = M(n-1) + 1, M(0) = 0
Example 2: The Tower of Hanoi Puzzle

• Recurrence for number of moves:

Solving recurrence for number of moves
• M(n) = 2M(n-1) + 1, M(1) = 1
Tree of calls for the Tower of Hanoi Puzzle
n

n-1 n-1

n-2 n-2 n-2 n-2

... ... ...
2 2 2 2

1 1 1 1 1 1 1 1
Example 3: Counting #bits
Fibonacci numbers
• The Fibonacci numbers:
• 0, 1, 1, 2, 3, 5, 8, 13, 21, …
• The Fibonacci recurrence:
• F(n) = F(n-1) + F(n-2)
• F(0) = 0
• F(1) = 1
• General 2nd order linear homogeneous recurrence with
constant coefficients:
• aX(n) + bX(n-1) + cX(n-2) = 0
Solving aX(n) + bX(n-1) + cX(n-2) = 0
• Set up the characteristic equation (quadratic)
• ar2 + br + c = 0
• Solve to obtain roots r1 and r2
• General solution to the recurrence
• if r1 and r2 are two distinct real roots: X(n) = αr1n + βr2n
• if r1 = r2 = r are two equal real roots: X(n) = αrn + βnrn
• Particular solution can be found by using initial conditions
Application to the Fibonacci numbers
• F(n) = F(n-1) + F(n-2) or F(n) - F(n-1) - F(n-2) = 0

• Characteristic equation:

• Roots of the characteristic equation:

• General solution to the recurrence:

• Particular solution for F(0) =0, F(1)=1:

Computing Fibonacci numbers
• Definition-based recursive algorithm
• Nonrecursive definition-based algorithm
• Explicit formula algorithm
• Logarithmic algorithm based on formula:

F(n-1) F(n) n
0 1
=
F(n) F(n+1) 1 1

• for n≥1, assuming an efficient way of computing matrix

powers.
Brute Force
Brute Force
• A straightforward approach, usually based directly on the
problem’s statement and definitions of the concepts involved

• Examples:
• Computing an (a > 0, n a nonnegative integer)

• Computing n!

• Multiplying two matrices

• Searching for a key of a given value in a list

Brute-Force Sorting Algorithm
• Selection Sort Scan the array to find its smallest element and
swap it with the first element. Then, starting with the second
element, scan the elements to the right of it to find the
smallest among them and swap it with the second elements.
Generally, on pass i (0  i  n-2), find the smallest element in
A[i..n-1] and swap it with A[i]:

A[0]  . . .  A[i-1] | A[i], . . . , A[min], . . ., A[n-1]

• in their final positions

• Example: 7 3 2 5
Analysis of Selection Sort
• Time efficiency:
• Space efficiency:
• Stability:
Brute-Force String Matching
• pattern: a string of m characters to search for
• text: a (longer) string of n characters to search in
• problem: find a substring in the text that matches the pattern

• Brute-force algorithm
• Step 1 Align pattern at beginning of text
• Step 2 Moving from left to right, compare each character of
pattern to the corresponding character in text until
• all characters are found to match (successful search); or
• a mismatch is detected
• Step 3 While pattern is not found and the text is not yet
exhausted, realign pattern one position to the right and
repeat Step 2
Examples of Brute-Force String Matching
• Pattern: 001011
Text: 10010101101001100101111010

• Pattern: happy
Text: It is never too late to have a happy childhood.
Pseudocode and Efficiency

• Efficiency:
Brute-Force Polynomial Evaluation
• Problem: Find the value of polynomial
• p(x) = anxn + an-1xn-1 +… + a1x1 + a0
• at a point x = x0
Brute-Force Polynomial Evaluation
• Brute-force algorithm
• p  0.0
• for i  n downto 0 do
• power  1
• for j  1 to i do //compute xi
• power  power  x
• p  p + ai  power
• return p

• Efficiency?
Polynomial Evaluation: Improvement
• We can do better by evaluating from right to left:
Polynomial Evaluation: Improvement
• We can do better by evaluating from right to left:

• Better brute-force algorithm

• p  a[0]
• power  1
• for i  1 to n do
• power  power  x
• p  p + a[i]  power
• return p

• Efficiency?
Closest-Pair Problem
• Find the two closest points in a set of n points (in the two-
dimensional Cartesian plane).
Closest-Pair Problem
• Find the two closest points in a set of n points (in the two-
dimensional Cartesian plane).

• Brute-force algorithm
• Compute the distance between every pair of distinct points
• and return the indexes of the points for which the distance is
the smallest.
Closest-Pair Brute-Force Algorithm (cont.)

• Efficiency?

• How to make it faster?

Brute-Force Strengths and Weaknesses
• Strengths
• wide applicability
• simplicity
• yields reasonable algorithms for some important problems
(e.g., matrix multiplication, sorting, searching, string matching)

• Weaknesses
• rarely yields efficient algorithms
• some brute-force algorithms are unacceptably slow
• not as constructive as some other design techniques
Exhaustive Search
Exhaustive Search
• A brute force solution to a problem involving search for an
element with a special property, usually among combinatorial
objects such as permutations, combinations, or subsets of a
set.

• Method:
• generate a list of all potential solutions to the problem in a
systematic manner

• evaluate potential solutions one by one, disqualifying infeasible

ones and, for an optimization problem, keeping track of the best
one found so far

• when search ends, announce the solution(s) found

Example 1: Traveling Salesman Problem
• Given n cities with known distances between each pair, find
the shortest tour that passes through all the cities exactly
once before returning to the starting city
• Alternatively: Find shortest Hamiltonian circuit in a weighted
connected graph
• Example:

2
a b
5 3
8 4

c d
7
TSP by Exhaustive Search
• Tour Cost
• a→b→c→d→a 2+3+7+5 = 17
• a→b→d→c→a 2+4+7+8 = 21
• a→c→b→d→a 8+3+4+5 = 20
• a→c→d→b→a 8+7+4+2 = 21
• a→d→b→c→a 5+4+3+8 = 20
• a→d→c→b→a 5+7+3+2 = 17

• More tours?

• Less tours?
Example 2: Knapsack Problem
• Given n items:
• weights: w1, w2, … , wn
• values: v1, v2, … , vn
• a knapsack of capacity W
• Find most valuable subset of the items that fit into the
knapsack
Example 2: Knapsack Problem
• Given n items:
• weights: w1 w2 … wn
• values: v1 v2 … vn
• a knapsack of capacity W
• Find most valuable subset of the items that fit into the
knapsack

• Example: Knapsack capacity W=16

item weight value
1 2 $20
2 5 $30
3 10 $50
4 5 $10
Knapsack Problem by Exhaustive Search
• Efficiency? Subset Total weight Total value
{1} 2 $20
{2} 5 $30
{3} 10 $50
{4} 5 $10
{1,2} 7 $50
{1,3} 12 $70
{1,4} 7 $30
{2,3} 15 $80
{2,4} 10 $40
{3,4} 15 $60
{1,2,3} 17 not feasible
{1,2,4} 12 $60
{1,3,4} 17 not feasible
{2,3,4} 20 not feasible
{1,2,3,4} 22 not feasible
Example 3: The Assignment Problem
• There are n people who need to be assigned to n jobs, one
person per job. The cost of assigning person i to job j is C[i,j].
Find an assignment that minimizes the total cost.
Job 0 Job 1 Job 2 Job 3
Person 0 9 2 7 8
Person 1 6 4 3 7
Person 2 5 8 1 8
Person 3 7 6 9 4

• Algorithmic Plan: Generate all legitimate assignments,

compute their costs, and select the cheapest one.
• How many assignments are there?
• Pose the problem as the one about a cost matrix:
Assignment Problem by Exhaustive Search
9 2 7 8
6 4 3 7
C =
5 8 1 8
7 6 9 4
Assignment (col.#s) Total Cost
1, 2, 3, 4 9+4+1+4=18
1, 2, 4, 3 9+4+8+9=30
1, 3, 2, 4 9+3+8+4=24
1, 3, 4, 2 9+3+8+6=26
1, 4, 2, 3 9+7+8+9=33
1. 4, 3, 2 9+7+1+6=23
etc.

• (For this particular instance, the optimal assignment can be

found by exploiting the specific features of the number given.
It is: _________ )
Final Comments on Exhaustive Search
• Exhaustive-search algorithms run in a realistic amount of time
only on very small instances

• In some cases, there are much better alternatives!

• Euler circuits
• shortest paths
• minimum spanning tree
• assignment problem

• In many cases, exhaustive search or its variation is the only

known way to get exact solution
Divide-and-Conquer
Divide-and-Conquer
• The most-well known algorithm design strategy:
• Divide instance of problem into two or more smaller instances

• Solve smaller instances recursively

• Obtain solution to original (larger) instance by combining

these solutions
Divide-and-Conquer Technique (cont.)

a problem of size n

subproblem 1 subproblem 2
of size n/2 of size n/2

a solution to a solution to
subproblem 1 subproblem 2

a solution to
the original problem
Divide-and-Conquer Examples
• Sorting: mergesort and quicksort

• Binary tree traversals

• Binary search (?)

• Multiplication of large integers

• Matrix multiplication: Strassen’s algorithm

• Closest-pair and convex-hull algorithms

General Divide-and-Conquer Recurrence
T(n) = aT(n/b) + f (n) where f(n)  (nd), d  0

1-99
General Divide-and-Conquer Recurrence
T(n) = aT(n/b) + f (n) where f(n)  (nd), d  0

Master Theorem: If a < bd, T(n)  (nd)

If a = bd, T(n)  (nd log n)
If a > bd, T(n)  (nlog b a )

Note: The same results hold with O instead of .

1-100
General Divide-and-Conquer Recurrence
T(n) = aT(n/b) + f (n) where f(n)  (nd), d  0

Master Theorem: If a < bd, T(n)  (nd)

If a = bd, T(n)  (nd log n)
If a > bd, T(n)  (nlog b a )

Note: The same results hold with O instead of .

Examples: T(n) = 4T(n/2) + n  T(n)  ?

T(n) = 4T(n/2) + n2  T(n)  ?
T(n) = 4T(n/2) + n3  T(n)  ?

1-101
Mergesort
• Split array A[0..n-1] in two about equal halves and make
copies of each half in arrays B and C
• Sort arrays B and C recursively
• Merge sorted arrays B and C into array A as follows:
• Repeat the following until no elements remain in one of the
arrays:
• compare the first elements in the remaining unprocessed
portions of the arrays
• copy the smaller of the two into A, while incrementing the index
indicating the unprocessed portion of that array
• Once all elements in one of the arrays are processed, copy the
remaining unprocessed elements from the other array into A.
Pseudocode of Mergesort
Pseudocode of Merge
Mergesort Example
8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 71 5 4

8 3 2 9 7 1 5 4

3 8 2 9 1 7 4 5

2 3 8 9 1 4 5 7

1 2 3 4 5 7 8 9
Analysis of Mergesort
• All cases have same efficiency: Θ(n log n)

• Number of comparisons in the worst case is close to

theoretical minimum for comparison-based sorting:

log2 n! ≈ n log2 n - 1.44n

• Space requirement: Θ(n) (not in-place)

• Can be implemented without recursion (bottom-up)

Quicksort
• Select a pivot (partitioning element) – here, the first element
• Rearrange the list so that all the elements in the first s
positions are smaller than or equal to the pivot and all the
elements in the remaining n-s positions are larger than or
equal to the pivot (see next slide for an algorithm)

A[i]p A[i]p

• Exchange the pivot with the last element in the first (i.e., )
subarray — the pivot is now in its final position
• Sort the two subarrays recursively
Partitioning Algorithm
Quicksort Example
•5 3 1 9 8 2 4 7
Analysis of Quicksort
• Best case: split in the middle — Θ(n log n)
• Worst case: sorted array! — Θ(n2)
• Average case: random arrays — Θ(n log n)

• Improvements:
• better pivot selection: median of three partitioning
• switch to insertion sort on small subfiles
• elimination of recursion
• These combine to 20-25% improvement

• Considered the method of choice for internal sorting of large

files (n ≥ 10000)
Binary Search
• Very efficient algorithm for searching in sorted array:
K vs A[0] . . . A[m] . . . A[n-1]
• If K = A[m], stop (successful search);
• otherwise, continue searching by the same method in
• A[0..m-1] if K < A[m] and in A[m+1..n-1] if K > A[m]

l  0; r  n-1
while l  r do
m  (l+r)/2
if K = A[m] return m
else if K < A[m] r  m-1
else l  m+1
return -1
Analysis of Binary Search
• Time efficiency
• worst-case recurrence: Cw (n) = 1 + Cw( n/2 ), Cw (1) = 1
solution: Cw(n) = log2(n+1)

This is VERY fast: e.g., Cw(106) = 20

• Optimal for searching a sorted array

• Limitations: must be a sorted array (not linked list)

• Bad (degenerate) example of divide-and-conquer

• Has a continuous counterpart called bisection method for

solving equations in one unknown f(x) = 0
Binary Tree Algorithms
• Binary tree is a divide-and-conquer ready structure!

• Ex. 1: Classic traversals (preorder, inorder, postorder)

• Algorithm Inorder(T)
• if T  
• Inorder(Tleft)
• print(root of T)
• Inorder(Tright)

• Efficiency: Θ(n)

1-114
Binary Tree Algorithms (cont.)
• Ex. 2: Computing the height
of a binary tree

• h(T) = max{h(TL), h(TR)} + 1

if T   and h() = -1

• Efficiency: Θ(n) TL TR
Multiplication of Large Integers
• Consider the problem of
multiplying two (large) n-digit
integers represented by
arrays of their digits such as: a1 a2 … an
A = 12345678901357986429 b1 b2 … bn
B = 87654321284820912836 (d10) d11 d12 … d1n
(d20) d21 d22 … d2n
… … … … ...
• The grade-school algorithm:
(dn0) dn1 dn2 … dnn

• Efficiency: n2 one-digit
multiplications
First Divide-and-Conquer Algorithm

• A small example: A  B where A = 2135 and B = 4014

First Divide-and-Conquer Algorithm

• A small example: A  B where A = 2135 and B = 4014

• Write A = (21·102 + 35) and B = (40 ·102 + 14). Then
•AB
• = (21 ·102 + 35)  (40 ·102 + 14)
• = 21  40 ·104 + (21  14 + 35  40) ·102 + 35  14
First Divide-and-Conquer Algorithm

• In general, if A = A1A2 and B = B1B2

• where A and B are n-digit numbers,
• and A1, A2, B1, B2 are n/2-digit numbers,
• Then A  B = A1  B1·10n + (A1  B2 + A2  B1) ·10n/2 + A2  B2

• Recurrence for the number of one-digit multiplications M(n):

• M(n) = 4M(n/2), M(1) = 1

• Solution: M(n) = n2
Second Divide-and-Conquer Algorithm
• A  B = A1B110n + (A1B2 + A2B1)10n/2 + A2  B2

• Can we decrease the number of multiplications from 4 to 3?

Second Divide-and-Conquer Algorithm
• (A1 + A2 )  (B1 + B2 ) = A1  B1 + (A1  B2 + A2  B1) + A2 
B2
• i.e., (A1  B2 + A2  B1) = (A1 + A2 )  (B1 + B2 ) - A1  B1 -
A2  B2,
• A  B = A1B110n + ((A1 + A2 )  (B1 + B2 ) - A1  B1 - A2 
B2)10n/2 + A2  B2

• which requires only 3 multiplications at the expense of (4-1)

extra add/sub.
• Recurrence for the number of multiplications M(n):
M(n) = 3M(n/2), M(1) = 1
Solution: M(n) = 3log 2n = nlog 23 ≈ n1.585
Example of Large-Integer Multiplication
• 2135  4014
Strassen’s Matrix Multiplication
• Strassen observed [1969] that the product of two matrices
can be computed as follows:
Strassen’s Matrix Multiplication
• Strassen observed [1969] that the product of two matrices
can be computed as follows:

C00 C01 A00 A01 B00 B01

= *
C10 C11 A10 A11 B10 B11

M1 + M4 - M5 + M7 M3 + M5
=
M2 + M4 M1 + M3 - M2 + M6
Formulas for Strassen’s Algorithm
• M1 = (A00 + A11)  (B00 + B11)
• M2 = (A10 + A11)  B00
• M3 = A00  (B01 - B11)
• M4 = A11  (B10 - B00)
• M5 = (A00 + A01)  B11
• M6 = (A10 - A00)  (B00 + B01)
• M7 = (A01 - A11)  (B10 + B11)
Analysis of Strassen’s Algorithm
• If n is not a power of 2, matrices can be padded with zeros.

• Number of multiplications:
• M(n) = 7M(n/2), M(1) = 1

• Solution:
• M(n) = 7log 2n = nlog 27 ≈ n2.807
• vs. n3 of brute-force alg.

• Algorithms with better asymptotic efficiency are known but

they are even more complex.
Closest-Pair Brute-Force Algorithm (cont.)
• Find the two closest points in a set of n points (in the two-
dimensional Cartesian plane).
• Brute-force algorithm
• Compute the distance between every pair of distinct points
• and return the indexes of the points for which the distance is the
smallest.

• Efficiency?
• How to make it faster?
Closest-Pair Problem by Divide-and-Conquer
• Step 1 Divide the points given into two subsets S1 and S2 by
a vertical line x = c so that half the points lie to the left or on
the line and half the points lie to the right or on the line.

• How to choose this c?

Closest-Pair Problem by Divide-and-Conquer
• Step 1 Divide the points given into two subsets S1 and S2 by
a vertical line x = c so that half the points lie to the left or on
the line and half the points lie to the right or on the line.
Closest Pair by Divide-and-Conquer (cont.)
• Step 2 Find recursively the closest pairs for the left and right
subsets.

• Step 3 Set d = min{d1, d2}

• We can limit our attention to the points in the symmetric vertical
strip of width 2d as possible closest pair.
• Let C1 and C2 be the subsets of points in the left subset S1
and of the right subset S2, respectively, that lie in this vertical
strip.
• The points in C1 and C2 are stored in increasing order of their y
coordinates, which is maintained by merging during the
execution of the next step.

• Step 4 For every point P(x,y) in C1, we inspect points in

C2 that may be closer to P than d. There can be no more
Closest Pair by Divide-and-Conquer:
Worst Case
Efficiency of the Closest-Pair Algorithm
• Running time of the algorithm is described by

T(n) = 2T(n/2) + M(n), where M(n)  O(n)

• By the Master Theorem (with a = 2, b = 2, d = 1)

T(n)  O(n log n)

Quickhull Algorithm
• Convex hull: smallest convex set that includes given points
• Assume points are sorted by x-coordinate values
• Identify extreme points P1 and P2 (leftmost and rightmost)
• Compute upper hull recursively:
• find point Pmax that is farthest away from line P1P2
• compute the upper hull of the points to the left of line P1Pmax
• compute the upper hull of the points to the left of line PmaxP2
• Compute lower hull in a similar manner
Pmax
P2

P1
Efficiency of Quickhull Algorithm
• Finding point farthest away from line P1P2 can be done in
linear time

• Time efficiency:
• worst case: Θ(n2) (as quicksort)
• average case: Θ(n)
• under reasonable assumptions about distribution of points given

• If points are not initially sorted by x-coordinate value, this can

be accomplished in O(n log n) time

• Several O(n log n) algorithms for convex hull are known

 Chapter 5

Decrease-and-Conquer
Decrease-and-Conquer
• Reduce problem instance to smaller instance of the same
problem
• Solve smaller instance
• Extend solution of smaller instance to obtain solution to
original instance
• Can be implemented either top-down or bottom-up
• Also referred to as inductive or incremental approach
3 Types of Decrease and Conquer

• Decrease by a constant (usually by 1):

• insertion sort
• graph traversal algorithms (DFS and BFS)
• topological sorting
• algorithms for generating permutations, subsets

• Decrease by a constant factor (usually by half)

• binary search and bisection method
• exponentiation by squaring
• multiplication à la russe

• Variable-size decrease
• Euclid’s algorithm
• selection by partition
What’s the difference?
• Consider the problem of exponentiation: Compute an

• Brute Force:

• Divide and conquer:

• Decrease by one:

• Decrease by constant factor:

Insertion Sort
• To sort array A[0..n-1], sort A[0..n-2] recursively and then
insert A[n-1] in its proper place among the sorted A[0..n-2]

• Usually implemented bottom up (nonrecursively)

• Example: Sort 6, 4, 1, 8, 5

6 4 1 8 5
4 6 1 8 5
1 4 6 8 5
1 4 6 8 5
1 4 5 6 8
Pseudocode of Insertion Sort
Analysis of Insertion Sort
• Time efficiency
• Cworst(n) = n(n-1)/2  Θ(n2)
• Cavg(n) ≈ n2/4  Θ(n2)
• Cbest(n) = n - 1  Θ(n) (also fast on almost sorted arrays)

• Space efficiency: in-place

• Stability: yes

• Best elementary sorting algorithm overall

• Binary insertion sort

Graph Traversal
• Many problems require processing all graph vertices (and
edges) in systematic fashion

• Graph traversal algorithms:

• Depth-first search (DFS)

• Breadth-first search (BFS)

Depth-First Search (DFS)
• Visits graph’s vertices by always moving away from last
visited vertex to unvisited one, backtracks if no adjacent
unvisited vertex is available.

• Uses a stack
• a vertex is pushed onto the stack when it’s reached for the first
time
• a vertex is popped off the stack when it becomes a dead end,
i.e., when there is no adjacent unvisited vertex

• “Redraws” graph in tree-like fashion (with tree edges and back

edges for undirected graph)
Pseudocode of DFS
Example: DFS traversal of undirected graph
DFS traversal stack: DFS tree:

a b c d

e f g h
Notes on DFS
• DFS can be implemented with graphs represented as:
• adjacency matrices: Θ(V2)
• adjacency lists: Θ(|V|+|E|)

• Yields two distinct ordering of vertices:

• order in which vertices are first encountered (pushed onto
stack)
• order in which vertices become dead-ends (popped off stack)

• Applications:
• checking connectivity, finding connected components
• checking acyclicity
• finding articulation points and biconnected components
• searching state-space of problems for solution (AI)
Breadth-first search (BFS)
• Visits graph vertices by moving across to all the neighbors of
last visited vertex

• Instead of a stack, BFS uses a queue

• Similar to level-by-level tree traversal

• “Redraws” graph in tree-like fashion (with tree edges and

cross edges for undirected graph)
Pseudocode of BFS
BFS traversal of undirected graph
BFS traversal queue: BFS tree:

a b c d

e f g h
Notes on BFS
• BFS has same efficiency as DFS and can be implemented
with graphs represented as:
• adjacency matrices: Θ(V2)
• adjacency lists: Θ(|V|+|E|)

• Yields single ordering of vertices (order added/deleted from

queue is the same)

• Applications: same as DFS, but can also find paths from a

vertex to all other vertices with the smallest number of edges
Dags and Topological Sorting
• A dag: a directed acyclic graph, i.e. a directed graph with no
(directed) cycles

a b a b
a dag not a dag

c d c d

• Arise in modeling many problems that involve prerequisite

constraints (construction projects, document version control)
• Vertices of a dag can be linearly ordered so that for every
edge its starting vertex is listed before its ending vertex
(topological sorting).
• Being a dag is also a necessary condition for topological
sorting be possible.
Topological Sorting Example
• Order the following items in a food chain

tiger

human
fish
sheep
shrimp

plankton wheat
DFS-based Algorithm
• DFS-based algorithm for topological sorting
• Perform DFS traversal, noting the order vertices are popped off
the traversal stack
• Reverse order solves topological sorting problem
• Back edges encountered?→ NOT a dag!

• Example:
a b c d

e f g h

• Efficiency:
Source Removal Algorithm
• Source removal algorithm
• Repeatedly identify and remove a source (a vertex with no
incoming edges) and all the edges incident to it until either no
vertex is left (problem is solved) or there is no source among
remaining vertices (not a dag)
• Example:

a b c d

e f g h

• Efficiency: same as efficiency of the DFS-based algorithm

Decrease-by-Constant-Factor Algorithms
• In this variation of decrease-and-conquer, instance size is
reduced by the same factor (typically, 2)

• Examples:
• Binary search and the method of bisection

• Exponentiation by squaring

• Multiplication à la russe (Russian peasant method)

• Fake-coin puzzle

• Josephus problem
Exponentiation by Squaring
• The problem: Compute an where n is a nonnegative integer

• The problem can be solved by applying recursively the

formulas:
• For even values of n
• an = (an/2)2 if n > 0 and a0 = 1
• For odd values of n
• an = (a(n-1)/2)2 a

• Recurrence: M(n) = M( n/2 ) + f(n), where f(n) = 1 or 2,

M(0) = 0

• Master Theorem: M(n)  Θ(log n) = Θ(b)

where b = log (n+1)
Russian Peasant Multiplication
• The problem: Compute the product of two positive integers

• Can be solved by a decrease-by-half algorithm based on the

following formulas.

• For even values of n:

• n * m = (n/2) * 2m
• For odd values of n:
• n * m = (n-1)/2 * 2m + m if n > 1 and m if n = 1
Example of Russian Peasant Multiplication
• Compute 20 * 26
n m
20 26
10 52
5 104 104
2 208 +
1 416 416
520

• Note: Method reduces to adding m’s values corresponding to

odd n’s.
Fake-Coin Puzzle (simpler version)
• There are n identically looking coins one of which is fake.
There is a balance scale but there are no weights; the scale
can tell whether two sets of coins weigh the same and, if not,
which of the two sets is heavier (but not by how much).
Design an efficient algorithm for detecting the fake coin.
Assume that the fake coin is known to be lighter than the
genuine ones.

• Decrease by factor 2 algorithm

• Decrease by factor 3 algorithm

Variable-Size-Decrease Algorithms
• In the variable-size-decrease variation of decrease-and-
conquer,
• instance size reduction varies from one iteration to another

• Examples:
• Euclid’s algorithm for greatest common divisor
• Partition-based algorithm for selection problem
• Interpolation search
• Some algorithms on binary search trees
• Nim and Nim-like games
Euclid’s Algorithm
• Euclid’s algorithm is based on repeated application of equality
• gcd(m, n) = gcd(n, m mod n)

• Ex.: gcd(80,44) = gcd(44,36) = gcd(36, 12) = gcd(12,0) = 12

• One can prove that the size, measured by the second

number, decreases at least by half after two consecutive
iterations.
• Hence, T(n)  O(log n)
Selection Problem
• Find the k-th smallest element in a list of n numbers
• k = 1 or k = n
• median: k = n/2
• Example: 4, 1, 10, 9, 7, 12, 8, 2, 15 median = ?

• The median is used in statistics as a measure of an average

value of a sample.
• In fact, it is a better (more robust) indicator than the mean,
which is used for the same purpose.
Digression: Post Office Location Problem
• Given n village locations along a straight highway, where
should a new post office be located to minimize the average
distance from the villages to the post office?
Algorithms for the Selection Problem
• The sorting-based algorithm: Sort and return the k-th element
Efficiency (if sorted by mergesort): Θ(nlog n)

• A faster algorithm is based on using the quicksort-like partition

of the list. Let s be a split position obtained by a partition:

all are ≤ A[s] all are ≥ A[s]

s

• Assuming that the list is indexed from 1 to n:

• If s = k, the problem is solved;
• if s > k, look for the k-th smallest elem. in the left part;
if s < k, look for the (k-s)-th smallest elem. in the right part.
• Note: The algorithm can simply continue until s = k.
Tracing the Median / Selection Algorithm
Here: n = 9,
Example: 4 1 10 9 7 12 8 2 15
k = 9/2 = 5
array index: 1 2 3 4 5 6 7 8 9
4 1 10 9 7 12 8 2 15
4 1 2 9 7 12 8 10 15
2 1 4 9 7 12 8 10 15 --- s=3 < k=5
9 7 12 8 10 15
9 7 8 12 10 15
8 7 9 12 10 15 --- s=6 > k=5
8 7
7 8 --- s=k=5
Solution: median is 8
Efficiency of the Partition-based Algorithm
• Average case (average split in the middle):
• C(n) = C(n/2)+(n+1)
• C(n)  Θ(n)

• Worst case (degenerate split):

• C(n)  Θ(n2)

• A more sophisticated choice of the pivot leads to a

complicated algorithm with Θ(n) worst-case efficiency.
Interpolation Search
• Searches a sorted array similar to binary search but estimates
location of the search key in A[l..r] by using its value v.
• Specifically, the values of the array’s elements are assumed
to grow linearly from A[l] to A[r]
• and the location of v is estimated as the x-coordinate of the
point on the straight line through (l, A[l]) and (r, A[r]) whose y-
coordinate is v.
• x = l + (v - A[l])(r - l)/(A[r] – A[l] )
value

A[r] .
v

A[l] .
index
l x r
Analysis of Interpolation Search
• Efficiency
• average case: C(n) < log2 log2 n + 1
• worst case: C(n) = n

• Preferable to binary search only for VERY large arrays and/or

expensive comparisons

• Has a counterpart, the method of false position (regula

falsi), for solving equations in one unknown
Binary Search Tree Algorithms
• Several algorithms on BST requires recursive processing of
just one of its subtrees, e.g.,
• Searching
• Insertion of a new key
• Finding the smallest (or the largest) key k

<k >k
Searching in Binary Search Tree
Algorithm BTS(x, v)
//Searches for node with key equal to v in BST rooted at node x
• if x = NIL return -1
• else if v = K(x) return x
• else if v < K(x) return BTS(left(x), v)
• else return BTS(right(x), v)

• Efficiency
• worst case: C(n) = n
• average case: C(n) ≈ 2ln n ≈ 1.39log2 n
One-Pile Nim
• There is a pile of n chips.
• Two players take turn by removing from the pile at least 1 and
at most m chips.
• (The number of chips taken can vary from move to move.)
• The winner is the player that takes the last chip.
• Who wins the game – the player moving first or second, if
both player make the best moves possible?

• It’s a good idea to analyze this and similar games

“backwards”, i.e., starting with n = 0, 1, 2, …
Partial Graph of One-Pile Nim with m = 4

• Vertex numbers indicate n,

the number of chips in the
pile.
• The losing position for the
player to move are circled.
1 6 • Only winning moves from a
winning position are shown
2 7
5 10
(in bold).
0
3 8 • Generalization: The player
moving first wins iff n is not a
4 9
multiple of 5 (more generally,
m+1); the winning move is to
take n mod 5 (n mod (m+1))
chips on every move.
Transform-and-Conquer
Transform and Conquer
• This group of techniques solves a problem by a
transformation

• to a simpler/more convenient instance of the same problem

(instance simplification)

• to a different representation of the same instance

(representation change)

• to a different problem for which an algorithm is already

available (problem reduction)
Instance simplification - Presorting
• Solve a problem’s instance by transforming it into another
simpler/easier instance of the same problem

• Presorting
• Many problems involving lists are easier when list is sorted.
• searching
• computing the median (selection problem)
• checking if all elements are distinct (element uniqueness)

• Also:
• Topological sorting helps solving some problems for dags.
• Presorting is used in many geometric algorithms.
How fast can we sort ?
• Efficiency of algorithms involving sorting depends on
efficiency of sorting.

• Theorem: log2 n!  n log2 n comparisons are necessary in

the worst case to sort a list of size n by any comparison-based
algorithm.

• Note: About n log2 n comparisons are also sufficient to sort

array of size n (by mergesort).
Searching with presorting
• Problem: Search for a given K in A[0..n-1]

• Presorting-based algorithm:
• Stage 1 Sort the array by an efficient sorting algorithm
• Stage 2 Apply binary search

• Efficiency: Θ(n log n) + O(log n) = Θ(n log n)

• Good or bad?

• Why do we have our dictionaries, telephone directories, etc.

sorted?
Element Uniqueness with presorting
• Presorting-based algorithm
• Stage 1: sort by efficient sorting algorithm (e.g. mergesort)
• Stage 2: scan array to check pairs of adjacent elements

• Efficiency: Θ(n log n) + O(n) = Θ(n log n)

• Brute force algorithm

Compare all pairs of elements

• Efficiency: O(n2)

• Another algorithm? Hashing

Instance simplification
Gaussian Elimination
• Given: A system of n linear equations in n unknowns with an
arbitrary coefficient matrix.
• Transform to: An equivalent system of n linear equations in n
unknowns with an upper triangular coefficient matrix.
• Solve the latter by substitutions starting with the last equation
and moving up to the first one.
a11x1 + a12x2 + … + a1nxn = b1 a11x1 + a12x2 + … + a1nxn = b1
a21x1 + a22x2 + … + a2nxn = b2 a22x2 + … + a2nxn = b2
⁞ ⁞ ⁞ ⁞ ⁞ ⁞ ⁞
an1x1 + an2x2 + … + annxn = bn annxn = bn
Gaussian Elimination (cont.)
• The transformation is accomplished by a sequence of
elementary operations on the system’s coefficient matrix
(which don’t change the system’s solution):

for i ←1 to n-1 do
replace each of the subsequent rows (i.e., rows i+1, …, n) by
a difference between that row and an appropriate multiple of
the i-th row to make the new coefficient in the i-th column of
that row 0
Example of Gaussian Elimination
2x1 - 4x2 + x3 = 6
• Solve:
3x1 - x2 + x3 = 11
x1 + x2 - x3 = -3

• Gaussian elimination
2 -4 1 6 2 -4 1 6
3 -1 1 11 row2 – (3/2)*row1 0 5 -1/2 2
1 1 -1 -3 row3 – (1/2)*row1 0 3 -3/2 -6
2 -4 1 6
0 5 -1/2 2
row3–(3/5)*row2 0 0 -6/5 -36/5

• Backward substitution
x1 = (-36/5) / (-6/5) = 6
x2 = (2+(1/2)*6) / 5 = 1
x3 = (6 - 6 + 4*1)/2 = 2
Gaussian Elimination
Pseudocode and Efficiency
• Stage 1: Reduction to the upper-triangular matrix
for i ← 1 to n-1 do
for j ← i+1 to n do
for k ← i to n+1 do
A[j, k] ← A[j, k] - A[i, k] * A[j, i] / A[i, i] //improve!
• Stage 2: Backward substitution
for j ← n downto 1 do
t←0
for k ← j +1 to n do
t ← t + A[j, k] * x[k]
x[j] ← (A[j, n+1] - t) / A[j, j]

• Efficiency: Θ(n3) + Θ(n2) = Θ(n3)

Searching Problem
• Problem:
• Given a (multi)set S of keys and a search key K, find an
occurrence of K in S, if any

• Searching must be considered in the context of:

• file size (internal vs. external)
• dynamics of data (static vs. dynamic)

• Dictionary operations (dynamic data):

• find (search)
• insert
• delete
Taxonomy of Searching Algorithms
• List searching
• sequential search
• binary search
• interpolation search

• Tree searching
• binary search tree
• binary balanced trees: AVL trees, red-black trees
• multiway balanced trees: 2-3 trees, 2-3-4 trees, B trees

• Hashing
• open hashing (separate chaining)
• closed hashing (open addressing)
Binary Search Tree
• Arrange keys in a binary tree with the binary search tree
property:

<K >K

Example: 5, 3, 1, 10, 12, 7, 9

Dictionary Operations on BSTs
• Searching – straightforward
• Insertion – search for key, insert at leaf where search
terminated
• Deletion – 3 cases:
• deleting key at a leaf
• deleting key at node with single child
• deleting key at node with two children
• Efficiency depends of the tree’s height: log2 n  h  n-1,
with height average (random files) be about 3 log2 n
• Thus all three operations have
• worst case efficiency: (n)
• average case efficiency: (log n)
• Bonus: inorder traversal produces sorted list
Balanced Search Trees
• Attractiveness of binary search tree is marred by the bad
(linear) worst-case efficiency. Two ideas to overcome it are:

• to rebalance binary search tree when a new insertion

makes the tree “too unbalanced”
• AVL trees
• red-black trees

• to allow more than one key per node of a search tree

• 2-3 trees
• 2-3-4 trees
• B-trees
Balanced trees: AVL trees
• An AVL tree is a binary search tree in which, for every node,
the difference between the heights of its left and right
subtrees, called the balance factor, is at most 1 (with the
height of an empty tree defined as -1)
• Tree (a) is an AVL tree
• Tree (b) is not an AVL tree
1 2

10 10
0 1 0 0
5 20 5 20

1 -1 0 1 -1

4 7 12 4 7

0 0 0 0
2 8 2 8

(a) (b)
Rotations
• If a key insertion violates the
balance requirement at 2 0
some node, the subtree 3 2

rooted at that node is 1

R
0 0
2 > 1 3
transformed via one of the
four rotations. 0
1
(a)

• (The rotation is always Single R-rotation

performed for a subtree
rooted at an “unbalanced” 2 0

node closest to the new 3 2

leaf.) -1
LR
0 0
1 > 1 3

0
2
(c)

Double LR-rotation
General case: Single R-rotation
General case: Double LR-rotation
AVL tree construction - an example
• Construct an AVL tree for the list 5, 6, 8, 3, 2, 4, 7
0 -1 -2 0
L(5)
5 5 5 6
0 -1 > 0 0
6 6 5 8

0
8

1 2 1
6 6 6
1 0 2 0 R (5) 0 0
5 8 5 8 > 3 8

0 1 0 0
3 3 2 5

0
2
AVL tree construction - an example (cont.)
2 0
6 5
-1 0 LR (6) 0 -1
3 8 > 3 6

0 1 0 0 0
2 5 2 4 8

0
4

-1 0
5 5
0 -2 0 0
3 6 3 7
RL (6)
0 0 1 > 0 0 0 0
2 4 8 2 4 6 8

0
7
Analysis of AVL trees
• h  1.4404 log2 (n + 2) - 1.3277
• average height: 1.01 log2 n + 0.1 for large n (found empirically)

• Search and insertion are O(log n)

• Deletion is more complicated but is also O(log n)

• Disadvantages:
• frequent rotations
• Complexity

• A similar idea: red-black trees (height of subtrees is allowed to

differ by up to a factor of 2)
Multiway Search Trees
• A multiway search tree is a search tree that allows
more than one key in the same node of the tree.
• A node of a search tree is called an n-node if it contains n-1
ordered keys (which divide the entire key range into n
intervals pointed to by the node’s n links to its children):
• Note: Every node in a classical binary search tree is a 2-node

k1 < k2 < … < kn-1

< k1 [k1, k2 )  kn-1

2-3 Tree
• A 2-3 tree is a search tree that
• may have 2-nodes and 3-nodes
• height-balanced (all leaves are on the same level)
• A 2-3 tree is constructed by successive insertions of keys
given, with a new key always inserted into a leaf of the tree. If
the leaf is a 3-node, it’s split into two with the middle key
promoted to the parent.
2-node 3-node

K K1, K2

<K >K < K1 (K1 , K 2 ) > K2

2-3 tree construction – an example
• Construct a 2-3 tree the list 9, 5, 8, 3, 2, 4, 7

8 8
>
9 5, 9 5, 8, 9 5 9 3, 5 9

8 3, 8 3, 8
>
2, 3, 5 9 2 5 9 2 4, 5 9

3, 8 > 3, 5, 8 >
3 8

2 4, 5, 7 9 2 4 7 9 2 4 7 9
Analysis of 2-3 trees
• log3 (n + 1) - 1  h  log2 (n + 1) - 1

• Search, insertion, and deletion are in (log n)

• The idea of 2-3 tree can be generalized by allowing more keys

per node
• 2-3-4 trees
• B-trees
Heaps and Heapsort
• A heap is a binary tree with keys at its nodes (one key per
node) such that:
• It is essentially complete, i.e., all its levels are full except
possibly the last level, where only some rightmost keys may
be missing

• The key at each node is ≥ keys at its children

Illustration of the heap’s definition
• Note: Heap’s elements are ordered top down (along any path
down from its root), but they are not ordered left to right

10 10 10

5 7 5 7 5 7

4 2 1 2 1 6 2 1

a heap not a heap not a heap

Some Important Properties of a Heap
• Given n, there exists a unique binary tree with n nodes that is
essentially complete, with h = log2 n

• The root contains the largest key

• The subtree rooted at any node of a heap is also a heap

• A heap can be represented as an array

Heap’s Array Representation
• Store heap’s elements in an array (whose elements indexed,
for convenience, 1 to n) in top-down left-to-right order
• Example:

9
123456
5 3 953142

1 4 2
• Left child of node j is at 2j
• Right child of node j is at 2j+1
• Parent of node j is at j/2
• Parental nodes are represented in the first n/2 locations
Heap Construction (bottom-up)
• Step 0: Initialize the structure with keys in the order given

• Step 1: Starting with the last (rightmost) parental node, fix the
heap rooted at it, if it doesn’t satisfy the heap condition: keep
exchanging it with its largest child until the heap condition
holds

• Step 2: Repeat Step 1 for the preceding parental node

Example of Heap Construction
Construct a heap for the list 2, 9, 7, 6, 5, 8

2 2 2

9 7 > 9 8 9 8

6 5 8 6 5 7 6 5 7

2 9 9

9 8 > 2 8 > 6 8

6 5 7 6 5 7 2 5 7
Pseudopodia of bottom-up heap construction
Heapsort
• Stage 1: Construct a heap for a given list of n keys

• Stage 2: Repeat operation of root removal n-1 times:

• Exchange keys in the root and in the last (rightmost) leaf
• Decrease heap size by 1
• If necessary, swap new root with larger child until the heap
condition holds
Sort the list 2, 9, 7, 6, 5, 8 by heapsort
Stage 1 (heap construction) Stage 2 (root/max removal)

•197658 •968257
•298657 •76825|9
•298657 •86725|9
•928657 •5672|89
•968257 •7652|89
•265|789
•625|789
•52|6789
•52|6789
•2|56789
Analysis of Heapsort
• Stage 1: Build heap for a given list of n keys
• worst-case
• C(n) = h-1
 2(h-i) 2i = 2 ( n – log2(n + 1))  (n)
i=0
# nodes at level i

• Stage 2: Repeat operation of root removal n-1 times (fix heap)

• worst-case
n-1
• C(n) =

2log i  (nlogn)
i=1
2

• Both worst-case and average-case efficiency: (nlogn)

• In-place: yes
• Stability: no (e.g., 1 1)
Priority Queue
• A priority queue is the ADT of a set of elements with
• numerical priorities with the following operations:
• find element with highest priority
• delete element with highest priority
• insert element with assigned priority (see below)

• Heap is a very efficient way for implementing priority queues

• Two ways to handle priority queue in which

highest priority = smallest number
Insertion of a New Element into a Heap
• Insert the new element at last position in heap.
• Compare it with its parent and, if it violates heap condition,
exchange them
• Continue comparing the new element with nodes up the tree
until the heap condition is satisfied
• Example: Insert key 10
• Efficiency: O(log n)
9 9 10

6 8 > 6 10 > 6 9

2 5 7 10 2 5 7 8 2 5 7 8
Horner’s Rule For Polynomial Evaluation
• Given a polynomial of degree n and a specific value of x, find
the value of p at that point.
• p(x) = anxn + an-1xn-1 + … + a1x + a0
• Two brute-force algorithms:

p  0;
for i  n downto 0 do
power  1
for j  1 to i do
power  power * x
p  p + ai * power
return p
p  a0; power  1
for i  1 to n do
power  power * x
p  p + ai * power
return p
Horner’s Rule Example
• p(x) = 2x4 - x3 + 3x2 + x - 5 =
= x(2x3 - x2 + 3x + 1) - 5 =
= x(x(2x2 - x + 3) + 1) - 5 =
= x(x(x(2x - 1) + 3) + 1) - 5

• Substitution into the last formula leads to a faster algorithm

• Same sequence of computations are obtained by simply

arranging the coefficient in a table and proceeding as follows:

• coefficients 2 -1 3 1 -5
• x=3
Horner’s Rule pseudocode
• Efficiency of Horner’s Rule: # multiplications = # additions = n
Horner’s Rule
• Synthetic division of of p(x) by (x-x0)
• Example: Let p(x) = 2x4 - x3 + 3x2 + x - 5. Find p(x):(x-3)
Computing an (revisited)
• Left-to-right binary exponentiation:
• Initialize product accumulator by 1.
• Scan n’s binary expansion from left to right and do the
following:
• If the current binary digit is 0, square the accumulator (S);
if the binary digit is 1, square the accumulator and multiply it by
a (SM).
• Efficiency: (b-1) ≤ M(n) ≤ 2(b-1) where b = log2 n + 1
• Example: Compute a13. Here, n = 13 = 11012.

binary rep. of 13: 1 1 0 1

SM SM S SM
accumulator: 1 12*a=a a2*a = a3 (a3)2 = a6 (a6)2*a= a13
(computed left-to-right)
Computing an (cont.)
• Right-to-left binary exponentiation
• Scan n’s binary expansion from right to left and compute an
as the product of terms a2 i corresponding to 1’s in this
expansion.
• Example Compute a13 by the right-to-left binary
exponentiation. Here, n = 13 = 11012.
1 1 0 1
a8 a4 a2 a : a2^i terms
a8 * a4 *a : product (computed right-to-left)

• Efficiency: same as that of left-to-right binary exponentiation

Problem Reduction
• This variation of transform-and-conquer solves a problem by a
transforming it into different problem for which an algorithm is
already available.

• To be of practical value, the combined time of the

transformation and solving the other problem should be
smaller than solving the problem as given by another method.
Examples of Solving Problems by Reduction
• computing lcm(m, n) via computing gcd(m, n)

• counting number of paths of length n in a graph by raising the

graph’s adjacency matrix to the n-th power

• transforming a maximization problem to a minimization

problem and vice versa (also, min-heap construction)

• linear programming

• reduction to graph problems (e.g., solving puzzles via state-

space graphs)
Space and Time Tradeoffs
Space-for-time tradeoffs
• Two varieties of space-for-time algorithms:
• input enhancement — preprocess the input (or its part) to
store some info to be used later in solving the problem
• counting sorts
• string searching algorithms

• prestructuring — preprocess the input to make accessing its

elements easier
• hashing
• indexing schemes (e.g., B-trees)
Review: String searching by brute force
• pattern: a string of m characters to search for
• text: a (long) string of n characters to search in

• Brute force algorithm

• Step 1 Align pattern at beginning of text
• Step 2 Moving from left to right, compare each character of
pattern to the corresponding character in text until either all
characters are found to match (successful search) or a
mismatch is detected
• Step 3 While a mismatch is detected and the text is not yet
exhausted, realign pattern one position to the right and repeat
Step 2
String searching by preprocessing
• Several string searching algorithms are based on the input
• enhancement idea of preprocessing the pattern

• Knuth-Morris-Pratt (KMP) algorithm preprocesses pattern left

to right to get useful information for later searching

• Boyer -Moore algorithm preprocesses pattern right to left and

store information into two tables

• Horspool’s algorithm simplifies the Boyer-Moore algorithm by

using just one table
Horspool’s Algorithm
• A simplified version of Boyer-Moore algorithm:

• preprocesses pattern to generate a shift table that determines

how much to shift the pattern when a mismatch occurs

• always makes a shift based on the text’s character c aligned

with the last character in the pattern according to the shift
table’s entry for c
How far to shift?
• Look at first (rightmost) character in text that was compared:
• The character is not in the pattern
• .....c...................... (c not in pattern)
• BAOBAB

• The character is in the pattern (but not the rightmost)

• .....O...................... (O occurs once in pattern)
BAOBAB
• .....A...................... (A occurs twice in pattern)
• BAOBAB

• The rightmost characters do match

• .....B......................
Shift table
• Shift sizes can be precomputed by the formula
• distance from c’s rightmost occurrence in pattern
among its first m-1 characters to its right end
• t(c) =
• pattern’s length m, otherwise
• by scanning pattern before search begins and stored in a
table called shift table

• Shift table is indexed by text and pattern alphabet

Eg, for BAOBAB:
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

1 2 6 6 6 6 6 6 6 6 6 6 6 6 3 6 6 6 6 6 6 6 6 6 6 6
Example of Horspool’s alg. application

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _

1 2 6 6 6 6 6 6 6 6 6 6 6 6 3 6 6 6 6 6 6 6 6 6 6 6 6
• BARD LOVED BANANAS
• BAOBAB
• BAOBAB
• BAOBAB
• BAOBAB (unsuccessful search)
Boyer-Moore algorithm
• Based on same two ideas:
• comparing pattern characters to text from right to left

• precomputing shift sizes in two tables

• bad-symbol table indicates how much to shift based on text’s

character causing a mismatch

• good-suffix table indicates how much to shift based on matched

part (suffix) of the pattern
Bad-symbol shift in Boyer-Moore algorithm
• If the rightmost character of the pattern doesn’t match, BM
algorithm acts as Horspool’s
• If the rightmost character of the pattern does match, BM
compares preceding characters right to left until either all
pattern’s characters match or a mismatch on text’s character c
is encountered after k > 0 matches
• text
• pattern c
k matches
• bad-symbol shift d1 = max{t1(c ) - k, 1}
Good-suffix shift in Boyer-Moore algorithm
• Good-suffix shift d2 is applied after 0 < k < m last characters
were matched
• d2(k) = the distance between matched suffix of size k and its
rightmost occurrence in the pattern that is not preceded by the
same character as the suffix

Example: CABABA d2(1) = 4

• If there is no such occurrence, match the longest part of the k-

character suffix with corresponding prefix;
if there are no such suffix-prefix matches, d2 (k) = m

Example: WOWWOW d2(2) = 5, d2(3) = 3, d2(4) = 3, d2(5) =

3
Boyer-Moore Algorithm
• After matching successfully 0 < k < m characters, the
algorithm shifts the pattern right by
• d = max {d1, d2}
• where d1 = max{t1(c) - k, 1} is bad-symbol shift
• d2(k) is good-suffix shift

• Example: Find pattern AT_THAT in

• WHICH_FINALLY_HALTS. _ _ AT_THAT
•

•
Boyer-Moore Algorithm (cont.)
• Step 1 Fill in the bad-symbol shift table
• Step 2 Fill in the good-suffix shift table
• Step 3 Align the pattern against the beginning of the text
• Step 4 Repeat until a matching substring is found or text ends:
• Compare the corresponding characters right to left.
• If no characters match, retrieve entry t1(c) from the bad-
symbol table for the text’s character c causing the mismatch
and shift the pattern to the right by t1(c).
If 0 < k < m characters are matched, retrieve entry t1(c) from
the bad-symbol table for the text’s character c causing the
mismatch and entry d2(k) from the good-suffix table and shift
the pattern to the right by
• d = max {d1, d2}
where d1 = max{t1(c) - k, 1}.
Example of Boyer-Moore alg. application

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _

1 2 6 6 6 6 6 6 6 6 6 6 6 6 3 6 6 6 6 6 6 6 6 6 6 6 6

• BE
k Spattern
S _ K Nd E W _ A B O
U T _ B A O B A2 B S
1 BAOBAB 2
• BA OBAB
2 BAOBAB 5
• d13= t1(K)
BAOBAB= 6 5B A O B A B
• d14= t1(_)-2
BAOBAB = 54
5 BAOBAB
• d2(2) =5 5

• BAOBAB
Hashing
• A very efficient method for implementing a dictionary, i.e., a
set with the operations:
• find
• insert
• delete

• Based on representation-change and space-for-time tradeoff

ideas

• Important applications:
• symbol tables
• databases (extendible hashing)
Hash tables and hash functions
• The idea of hashing is to map keys of a given file of size n into
• a table of size m, called the hash table, by using a predefined
• function, called the hash function,
• h: K → location (cell) in the hash table

• Example: student records, key = SSN. Hash function:

• h(K) = K mod m where m is some integer (typically, prime)
• If m = 1000, where is record with SSN= 314159265 stored?

• Generally, a hash function should:

• be easy to compute
• distribute keys about evenly throughout the hash table
Collisions
• If h(K1) = h(K2), there is a collision

• Good hash functions result in fewer collisions but some

collisions should be expected (birthday paradox)

• Two principal hashing schemes handle collisions differently:

• Open hashing
– each cell is a header of linked list of all keys hashed to it
• Closed hashing
• one key per cell
• in case of collision, finds another cell by
• linear probing: use next free bucket
• double hashing: use second hash function to compute increment
Open hashing (Separate chaining)
• Keys are stored in linked lists outside a hash table whose
• elements serve as the lists’ headers.
• Example: A, FOOL, AND, HIS, MONEY, ARE, SOON,
PARTED
• h(K) = sum of K ‘s letters’ positions in the alphabet MOD 13
Key A FOOL AND HIS MONEY ARE SOON PARTED
h(K) 1 9 6 10 7 11 11 12

0 1 2 3 4 5 6 7 8 9 10 11 12

A AND MONEY FOOL HIS ARE PARTED

SOON
Search for KID
Open hashing (cont.)
• If hash function distributes keys uniformly, average length of
linked list will be α = n/m. This ratio is called load factor.

• Average number of probes in successful, S, and unsuccessful

searches, U:
• S  1+α/2, U = α

• Load α is typically kept small (ideally, about 1)

•
• Open hashing still works if n > m
Closed hashing (Open addressing)
• Keys are stored inside a hash table.

Key A FOOL AND HIS MONEY ARE SOON PARTED

h(K) 1 9 6 10 7 11 11 12

0 1 2 3 4 5 6 7 8 9 10 11 12
A
A FOOL
A AND FOOL
A AND FOOL HIS
A AND MONEY FOOL HIS
A AND MONEY FOOL HIS ARE
A AND MONEY FOOL HIS ARE SOON
PARTED A AND MONEY FOOL HIS ARE SOON
Closed hashing (cont.)
• Does not work if n > m
• Avoids pointers
• Deletions are not straightforward
• Number of probes to find/insert/delete a key depends on load
factor α = n/m (hash table density) and collision resolution
strategy. For linear probing:
• S = (½) (1+ 1/(1- α)) and U = (½) (1+ 1/(1- α)²)
• As the table gets filled (α approaches 1), number of probes in
linear probing increases dramatically:
Chapter 8

Dynamic Programming

Copyright © 2007 Pearson Addison-Wesley. All rights reserved.

Dynamic Programming

Dynamic Programming is a general algorithm design technique

for solving problems defined by recurrences with overlapping
subproblems

• Invented by American mathematician Richard Bellman in the

1950s to solve optimization problems and later assimilated by CS

• “Programming” here means “planning”

• Main idea:
- set up a recurrence relating a solution to a larger instance
to solutions of some smaller instances
- solve smaller instances once
- record solutions in a table
- extract solution to the initial instance from that table
Example: Fibonacci numbers

• Recall definition of Fibonacci numbers:

F(n) = F(n-1) + F(n-2)

F(0) = 0
F(1) = 1

• Computing the nth Fibonacci number recursively (top-down):

F(n)

F(n-1) + F(n-2)

F(n-2) + F(n-3) F(n-3) + F(n-4)

...
Example: Fibonacci numbers (cont.)

Computing the nth Fibonacci number using bottom-up iteration and

recording results:

F(0) = 0
F(1) = 1
F(2) = 1+0 = 1
…
F(n-2) =
F(n-1) =
F(n) = F(n-1) + F(n-2)

0 1 1 . . . F(n-2) F(n-1) F(n)

Efficiency:
- time
- space
Examples of DP algorithms

• Computing a binomial coefficient

• Warshall’s algorithm for transitive closure

• Floyd’s algorithm for all-pairs shortest paths

• Constructing an optimal binary search tree

• Some instances of difficult discrete optimization problems:

- traveling salesman
- knapsack
Computing a binomial coefficient by DP

Binomial coefficients are coefficients of the binomial formula:

(a + b)n = C(n,0)anb0 + . . . + C(n,k)an-kbk + . . . + C(n,n)a0bn

Recurrence: C(n,k) = C(n-1,k) + C(n-1,k-1) for n > k > 0

C(n,0) = 1, C(n,n) = 1 for n  0

Value of C(n,k) can be computed by filling a table:

0 1 2 . . . k-1 k
01
111
.
.
.
n-1 C(n-1,k-1) C(n-1,k)
n C(n,k)
Computing C(n,k): pseudocode and analysis

Time efficiency: Θ(nk)

Space efficiency: Θ(nk)
Knapsack Problem by DP
• Given n items of
• integer weights: w1 w2 … wn
• values: v1 v2 … vn
• a knapsack of integer capacity W
• find most valuable subset of the items that fit into the
knapsack

• Consider instance defined by first i items and capacity j (j 

W).
• Let V[i,j] be optimal value of such instance. Then
• max {V[i-1,j], vi + V[i-1,j- wi]} if j- wi  0
• V[i,j] =
• V[i-1,j] if j- wi < 0
Knapsack Problem by DP (example)
• Example: Knapsack of capacity W = 5
• item weight value
• 1 2 $12
• 2 1 $10
• 3 3 $20
• 4 2 $15 capacity j
• 012345
• 0
• w1 = 2, v1= 12 1
• w2 = 1, v2= 10 2
• w3 = 3, v3= 20 3
• w4 = 2, v4= 15 4 ?
Warshall’s Algorithm: Transitive Closure

• Computes the transitive closure of a relation

• Alternatively: existence of all nontrivial paths in a digraph

• Example of transitive closure:

3 3
1 1

4 4 0010
2 0010 2
1001 1111
0000 0000
0100 1111
Warshall’s Algorithm

Constructs transitive closure T as the last matrix in the sequence

of n-by-n matrices R(0), … , R(k), … , R(n) where
R(k)[i,j] = 1 iff there is nontrivial path from i to j with only first k
vertices allowed as intermediate
Note that R(0) = A (adjacency matrix), R(n) = T (transitive closure)
3 3 3 3 3
1 1 1 1 1

4 4 4 2 4 4
2 2 2 2

R(0) R(1) R(2) R(3) R(4)

0010 0010 0010 0010 0010
1001 1011 1011 1011 1111
0000 0000 0000 0000 0000
0100 0100 1111 1111 1111
Warshall’s Algorithm (recurrence)

On the k-th iteration, the algorithm determines for every pair of

vertices i, j if a path exists from i and j with just vertices 1,…,k
allowed as intermediate

{
R(k)[i,j] = or
R(k-1)[i,j] (path using just 1 ,…,k-1)

R(k-1)[i,k] and R(k-1)[k,j] (path from i to k

and from k to i
using justk 1 ,…,k-1)
i

j
Warshall’s Algorithm (matrix generation)

Recurrence relating elements R(k) to elements of R(k-1) is:

R(k)[i,j] = R(k-1)[i,j] or (R(k-1)[i,k] and R(k-1)[k,j])

It implies the following rules for generating R(k) from R(k-1):

Rule 1 If an element in row i and column j is 1 in R(k-1),

it remains 1 in R(k)

Rule 2 If an element in row i and column j is 0 in R(k-1),

it has to be changed to 1 in R(k) if and only if
the element in its row i and column k and the element
in its column j and row k are both 1’s in R(k-1)
Warshall’s Algorithm (example)

3
1 0010 0010
1001 1011
R(0) = 0000 R(1) = 0000
2 4
0100 0100

0010 0010 0010

1011 1011 1111
R(2) = 0000 R(3) = 0000 R(4) = 0000
1111 1111 1111
Warshall’s Algorithm (pseudocode and
analysis)

Time efficiency: Θ(n3)

Space efficiency: Matrices can be written over their predecessors
Floyd’s Algorithm: All pairs shortest paths

Problem: In a weighted (di)graph, find shortest paths between

every pair of vertices

Same idea: construct solution through series of matrices D(0), …,

D (n) using increasing subsets of the vertices allowed
as intermediate

Example: 4 3
1
1
6
1 5

4
2 3
Floyd’s Algorithm (matrix generation)

On the k-th iteration, the algorithm determines shortest paths

between every pair of vertices i, j that use only vertices among
1,…,k as intermediate

D(k)[i,j] = min {D(k-1)[i,j], D(k-1)[i,k] + D(k-1)[k,j]}

D(k-1)[i,k]
k
i
D(k-1)[k,j]
D(k-1)[i,j]
j
Floyd’s Algorithm (example)

2
1 2 0∞3∞2 0∞3∞
3 6 7 0∞∞ 205∞
D(0) = ∞701 D(1) = ∞701
3
1
4 6∞∞0 6∞90

0∞3∞ 0 10 3 4 0 10 3 4
205∞ 2056 2056
D(2) = 9701 D(3) = 9701 D(4) = 7701
6∞90 6 16 9 0 6 16 9 0
Floyd’s Algorithm (pseudocode and analysis)

Time efficiency: Θ(n3)

Space efficiency: Matrices can be written over their predecessors
Note: Shortest paths themselves can be found, too
Optimal Binary Search Trees

Problem: Given n keys a1 < …< an and probabilities p1 ≤ … ≤ pn

searching for them, find a BST with a minimum
average number of comparisons in successful search.
Since total number of BSTs with n nodes is given by
C(2n,n)/(n+1), which grows exponentially, brute force is hopeless.

Example: What is an optimal BST for keys A, B, C, and D with

search probabilities 0.1, 0.2, 0.4, and 0.3, respectively?
DP for Optimal BST Problem

Let C[i,j] be minimum average number of comparisons made in

T[i,j], optimal BST for keys ai < …< aj , where 1 ≤ i ≤ j ≤ n.
Consider optimal BST among all BSTs with some ak (i ≤ k ≤ j ) as
their root; T[i,j] is the best among them.
ak
C[i,j] =
min {pk · 1 +
i≤k≤j
Optimal Optimal
BST for k-1
BST for
a i , ..., a k-1
∑ ps (level as in T[i,k-1] +1) +
a k+1 , ..., a j

s=i

j
∑ ps (level as in T[k+1,j] +1)}
s =k+1
DP for Optimal BST Problem (cont.)

After simplifications, we obtain the recurrence for C[i,j]:

j
C[i,j]
0 =
1 min {C[i,k-1] + C[k+1,j]}
j n + ∑ ps for 1 ≤ i ≤ j ≤ n
i≤k≤j s=i
C[i,i] = pi for 1 ≤ i ≤ j ≤ n
1 0 p1 goal

0 p2

i C[i,j]

pn

n+1 0
 Example: key A B C D
probability 0.1 0.2 0.4 0.3

The tables below are filled diagonal by diagonal: the left one is filled
using the recurrence j
C[i,j] = min {C[i,k-1] + C[k+1,j]} + ∑ ps , C[i,i] = pi ;
i≤k≤j s=i
the right one, for trees’ roots, records k’s values giving the minima
j j
i 0 1 2 3 4 i 0 1 2 3 4
C
1 0 .1 .4 1.1 1.7 1 1 2 3 3

2 2 3 3 B D
2 0 .2 .8 1.4

3 0 .4 1.0 3 3 3
A
4 0 .3 4 4
optimal BST
5 0 5
Optimal Binary Search Trees
Analysis DP for Optimal BST Problem

Time efficiency: Θ(n3) but can be reduced to Θ(n2) by taking

advantage of monotonicity of entries in the
root table, i.e., R[i,j] is always in the range
between R[i,j-1] and R[i+1,j]
Space efficiency: Θ(n2)

Method can be expended to include unsuccessful searches

Chapter 9
Greedy Technique

Copyright © 2007 Pearson Addison-Wesley. All rights reserved.

Greedy Technique
• Constructs a solution to an optimization problem piece by
• piece through a sequence of choices that are:

• feasible

• locally optimal

• irrevocable

• For some problems, yields an optimal solution for every

instance.
• For most, does not but can be useful for fast approximations.
Applications of the Greedy Strategy
• Optimal solutions:
• change making for “normal” coin denominations
• minimum spanning tree (MST)
• single-source shortest paths
• simple scheduling problems
• Huffman codes

• Approximations:
• traveling salesman problem (TSP)
• knapsack problem
• other combinatorial optimization problems
Change-Making Problem
• Given unlimited amounts of coins of denominations d1 > … >
dm ,
• give change for amount n with the least number of coins

• Example: d1 = 25c, d2 =10c, d3 = 5c, d4 = 1c and n = 48c

• Greedy solution:

• Greedy solution is
• optimal for any amount and “normal’’ set of denominations
• may not be optimal for arbitrary coin denominations
Minimum Spanning Tree (MST)
• Spanning tree of a connected graph G: a connected acyclic
subgraph of G that includes all of G’s vertices

• Minimum spanning tree of a weighted, connected graph G: a

spanning tree of G of minimum total weight

• Example:

6 c
a
4 1
2

d
b 3
Prim’s MST algorithm
• Start with tree T1 consisting of one (any) vertex and “grow”
tree one vertex at a time to produce MST through a series of
expanding subtrees T1, T2, …, Tn

• On each iteration, construct Ti+1 from Ti by adding vertex not

in Ti that is closest to those already in Ti (this is a “greedy”
step!)

• Stop when all vertices are included

Example
•
4 c
a
1
6
2

d
b 3
Notes about Prim’s algorithm
• Proof by induction that this construction actually yields MST

• Needs priority queue for locating closest fringe vertex

• Efficiency
• O(n2) for weight matrix representation of graph and array
implementation of priority queue
• O(m log n) for adjacency list representation of graph with n
vertices and m edges and min-heap implementation of priority
queue
Another greedy algorithm for MST: Kruskal’s
• Sort the edges in nondecreasing order of lengths

• “Grow” tree one edge at a time to produce MST through a

series of expanding forests F1, F2, …, Fn-1

• On each iteration, add the next edge on the sorted list unless
this would create a cycle. (If it would, skip the edge.)
Example
•
4 c
a
1
6
2

d
b 3
Notes about Kruskal’s algorithm
• Algorithm looks easier than Prim’s but is harder to implement
(checking for cycles!)

• Cycle checking: a cycle is created iff added edge connects

vertices in the same connected component

• Union-find algorithms – see section 9.2

Minimum spanning tree vs. Steiner tree
•
1 a c
a c

1 1 vs

b d b d
1
Shortest paths – Dijkstra’s algorithm
• Single Source Shortest Paths Problem: Given a weighted
• connected graph G, find shortest paths from source vertex s
• to each of the other vertices

• Dijkstra’s algorithm: Similar to Prim’s MST algorithm, with

• a different way of computing numerical labels: Among vertices
• not already in the tree, it finds vertex u with the smallest sum
• dv + w(v,u)
• where
• v is a vertex for which shortest path has been already found
on preceding iterations (such vertices form a tree)
• dv is the length of the shortest path form source to v
w(v,u) is the length (weight) of edge from v to u
 4
Example 3
b
2 5
c
6
a 7 4
d e
•
Tree vertices Remaining vertices 4
b c
3 6
a(-,0) b(a,3) c(-,∞) d(a,7) e(-,∞) 2 5
a d e
7 4

4
b c
b(a,3) c(b,3+4) d(b,3+2) e(-,∞) 3 6
2 5
a d e
7 4

4
d(b,5) c(b,7) e(d,5+4) 3
b c
6
2 5
a d e
7 4
4
b c
c(b,7) e(d,9) 3 6
2 5
a d e
7 4
e(d,9)
Notes on Dijkstra’s algorithm
• Doesn’t work for graphs with negative weights

• Applicable to both undirected and directed graphs

• Efficiency
• O(|V|2) for graphs represented by weight matrix and array
implementation of priority queue
• O(|E|log|V|) for graphs represented by adj. lists and min-heap
implementation of priority queue

• Don’t mix up Dijkstra’s algorithm with Prim’s algorithm!

Coding Problem
• Coding: assignment of bit strings to alphabet characters

• Codewords: bit strings assigned for characters of alphabet

• Two types of codes:

• fixed-length encoding (e.g., ASCII)
• variable-length encoding (e,g., Morse code)

• Prefix-free codes: no codeword is a prefix of another

codeword

• Problem: If frequencies of the character occurrences are

• known, what is the best binary prefix-free code?
Huffman codes
• Any binary tree with edges labeled with 0’s and 1’s yields a
prefix-free code of characters assigned to its leaves

• Optimal binary tree minimizing the expected (weighted

average) length of a codeword can be constructed as follows

• Huffman’s algorithm
• Initialize n one-node trees with alphabet characters and the
tree weights with their frequencies.
• Repeat the following step n-1 times: join two binary trees with
smallest weights into one (as left and right subtrees) and
make its weight equal the sum of the weights of the two trees.
• Mark edges leading to left and right subtrees with 0’s and 1’s,
respectively.
Example
• character A B C D _
0.1 0.15 0.2 0.2 0.35

• frequency 0.35 0.1 0.2 0.2 B _ C D A

0.15 0.2 0.2

0.25
0.35
C D A

0.1 0.15
B _

• codeword 11 100 00 01 101 0.25 0.35 0.4

A

0.1 0.15 0.2 0.2

B _ C D

• average bits per character: 0.4 0.6

2.25 0.2 0.2

0.25 0.35
C D
A

• for fixed-length encoding: 3 0.1

B
0.15
_

• compression ratio: (3- 0

1.0
1

2.25)/3*100% = 25% 0.4 0.6

0 1 0 1

0.2 0.2 0.35

0.25
C D A
0 1

0.1 0.15
B _
Chapter 10
Iterative
Improvement

Copyright © 2007 Pearson Addison-Wesley. All rights reserved.

Iterative Improvement
• Algorithm design technique for solving optimization problems

• Start with a feasible solution

• Repeat the following step until no improvement can be found:
• change the current feasible solution to a feasible solution with a
better value of the objective function
• Return the last feasible solution as optimal

• Note: Typically, a change in a current solution is “small” (local

search)

• Major difficulty: Local optimum vs. global optimum

Important Examples
• simplex method

• Ford-Fulkerson algorithm for maximum flow problem

• maximum matching of graph vertices

• Gale-Shapley algorithm for the stable marriage problem

• local search heuristics

Linear Programming
• Linear programming (LP) problem is to optimize a linear
function of several variables subject to linear constraints:

• maximize (or minimize) c1 x1 + ...+ cn xn

• subject to ai 1x1+ ...+ ain xn ≤ (or ≥ or =) bi , i = 1,...,m x1 ≥ 0,
... , xn ≥ 0

• The function z = c1 x1 + ...+ cn xn is called the objective

function;
• constraints x1 ≥ 0, ... , xn ≥ 0 are called nonnegativity
constraints
Example
• maximize 3x + 5y
• subject to x + y ≤ 4
• x + 3y ≤ 6
• x ≥ 0, y ≥ 0 y

x + 3y = 6
• ( 0, 2 )
( 3, 1 )

x
Feasible region is the ( 0, 0 ) ( 4, 0 )

set of points defined x+y=4

by the constraints
Geometric solution
• maximize 3x + 5y
• subject to x + y ≤ 4
• x + 3y ≤ 6 y

• x ≥ 0, y ≥ 0

( 0, 2 )
( 3, 1 )

x
( 0, 0 ) ( 4, 0 )
3x + 5y = 20

• Optimal solution: x = 3, y = 1
3x + 5y = 14

3x + 5y = 10
Extreme Point Theorem Any LP problem with a nonempty bounded feasible
region has an optimal solution; moreover, an optimal solution can always be
found at an extreme point of the problem's feasible region.
3 possible outcomes in solving an LP problem
• has a finite optimal solution, which may no be unique

• unbounded: the objective function of maximization

(minimization) LP problem is unbounded from above (below)
on its feasible region

• infeasible: there are no points satisfying all the constraints, i.e.

the constraints are contradictory
The Simplex Method
• The classic method for solving LP problems;
one of the most important algorithms ever invented

• Invented by George Dantzig in 1947

• Based on the iterative improvement idea:

• Generates a sequence of adjacent points of the problem’s
feasible region with improving values of the objective function
until no further improvement is possible
Standard form of LP problem
• must be a maximization problem
• all constraints (except the nonnegativity constraints) must be
in the form of linear equations
• all the variables must be required to be nonnegative

• Thus, the general linear programming problem in standard

• form with m constraints and n unknowns (n ≥ m) is
•
maximize c1 x1 + ...+ cn xn
• subject to ai 1x1+ ...+ ain xn = bi , i = 1,...,m, x1 ≥ 0, ... , xn ≥
0

• Every LP problem can be represented in such form

Example

• maximize 3x + 5y maximize 3x + 5y + 0u + 0v
• subject to x + y ≤ 4 subject to x + y + u = 4
• x + 3y ≤ 6 x + 3y + v = 6
• x≥0, y≥0 x≥0, y≥0, u≥0, v≥0
•
• Variables u and v, transforming inequality constraints into
• equality constrains, are called slack variables
Basic feasible solutions
• A basic solution to a system of m linear equations in n
unknowns (n ≥ m) is obtained by setting n – m variables to 0
and solving the resulting system to get the values of the other
m variables. The variables set to 0 are called nonbasic; the
variables obtained by solving the system are called basic.
• A basic solution is called feasible if all its (basic) variables are
nonnegative.

• Example x + y + u = 4
• x + 3y + v = 6

• (0, 0, 4, 6) is basic feasible solution

• (x, y are nonbasic; u, v are basic)
Simplex Tableau
• maximize z = 3x + 5y + 0u +
0v
• subject to x + y + u = 4 x y u v
u 1 1 1 0 4
• x + 3y + v = 6
v 1 3 0 1 6
• x≥0, y≥0, u≥0, v≥0
3 5 0 0 0

•
basic basic feasible solution
variables (0, 0, 4, 6)

objective row value of z at (0, 0, 4, 6)

Outline of the Simplex Method
• Step 0 [Initialization] Present a given LP problem in standard
form and set up initial tableau.
• Step 1 [Optimality test] If all entries in the objective row are
nonnegative — stop: the tableau represents an optimal
solution.
• Step 2 [Find entering variable] Select (the most) negative
entry in the objective row. Mark its column to indicate the
entering variable and the pivot column.

• Step 3 [Find departing variable] For each positive entry in the

pivot column, calculate the θ-ratio by dividing that row's entry
in the rightmost column by its entry in the pivot column. (If
there are no positive entries in the pivot column — stop: the
problem is unbounded.) Find the row with the smallest θ-ratio,
mark this row to indicate the departing variable and the pivot
row.
Example of Simplex Method Application
•
maximize z = 3x + 5y + 0u + 0v x y u v
subject to x + y + u = 4 u 1 1 1 0 4

x + 3y + v = 6 v 1 3 0 1 6
x≥0, y≥0, u≥0, v≥0
3 5 0 0 0

x y u v
x 1 0 3/2 1/3 3

y 0 1 1/2 1/2 1

x y 0 u 0 v 2 1 14
2 1
u 0 1 2
3 3
1 1
y 1 0 2
3 3
basic feasible sol. basic feasible sol. basic feasible sol.
4 5
0 0
(0, 0, 4, 6) (0, 2, 2, 0) 3 (3, 31, 0,100)
z=0 z = 10 z = 14
Notes on the Simplex Method
• Finding an initial basic feasible solution may pose a problem

• Theoretical possibility of cycling

• Typical number of iterations is between m and 3m, where m

is
the number of equality constraints in the standard form

• Worse-case efficiency is exponential

• More recent interior-point algorithms such as Karmarkar’s

algorithm (1984) have polynomial worst-case efficiency and
have performed competitively with the simplex method in
empirical tests
Maximum Flow Problem
• Problem of maximizing the flow of a material through a
transportation network (e.g., pipeline system, communications
or transportation networks)

• Formally represented by a connected weighted digraph with n

vertices numbered from 1 to n with the following properties:
• contains exactly one vertex with no entering edges, called the
source (numbered 1)
• contains exactly one vertex with no leaving edges, called the
sink (numbered n)
• has positive integer weight uij on each directed edge (i.j), called
the edge capacity, indicating the upper bound on the amount of
the material that can be sent from i to j through this edge
Example of Flow Network

4
3
Source

2 5 2
1 2 3 6

3 Sink
1

4
Definition of a Flow
• A flow is an assignment of real numbers xij to edges (i,j) of a
given network that satisfy the following:
• flow-conservation requirements
The total amount of material entering an intermediate
vertex must be equal to the total amount of the material
leaving the vertex

∑ xji = ∑ xij for i = 2,3,…, n-1

j: (j,i) є E j: (i,j) є E
• capacity constraints
• 0 ≤ xij ≤ uij for every edge (i,j)  E
Flow value and Maximum Flow Problem
• Since no material can be lost or added to by going through
intermediate vertices of the network, the total amount of the
material leaving the source must end up at the sink:
• ∑ x1j = ∑ xjn

• The value of the flowj: (1,j) є E j: (j,n)as

is defined є E the total outflow from the
source (= the total inflow into the sink).

• The maximum flow problem is to find a flow of the largest

value (maximum flow) for a given network.
Maximum-Flow Problem as LP problem
• Maximize v = ∑ x1j
• j: (1,j)  E
• subject to

∑ xji - ∑ xij = 0 for i = 2, 3,…,n-1

j: (j,i)  E j: (i,j)  E

0 ≤ xij ≤ uij for every edge (i,j)  E

Augmenting Path (Ford-Fulkerson) Method
• Start with the zero flow (xij = 0 for every edge)

• On each iteration, try to find a flow-augmenting path from

source to sink, which a path along which some additional flow
can be sent

• If a flow-augmenting path is found, adjust the flow along the

edges of this path to get a flow of increased value and try
again

• If no flow-augmenting path is found, the current flow is

maximum
Example 1

0/3 0/4

0/2 0/5 0/2

1 2 3 6

0/3
0/1
xij/uij
4 Augmenting path:
1→2 →3 →6
Example 1 (cont.)

0/3 0/4

2/2 2/5 2/2

1 2 3 6

0/3
0/1

4
Augmenting path:
1 →4 →3←2 →5 →6
Example 1 (maximum flow)

1/3 1/4

2/2 1/5 2/2

1 2 3 6

1/3
1/1
max flow value = 3
4
Finding a flow-augmenting path
• To find a flow-augmenting path for a flow x, consider paths
from source to sink in the underlying undirected graph in
which any two consecutive vertices i,j are either:
• connected by a directed edge (i to j) with some positive unused
capacity rij = uij – xij
• known as forward edge ( → )
• OR
• connected by a directed edge (j to i) with positive flow xji
• known as backward edge ( ← )
• If a flow-augmenting path is found, the current flow can be
increased by r units by increasing xij by r on each forward
edge and decreasing xji by r on each backward edge, where

r = min {rij on all forward edges, xji on all backward edges}

Finding a flow-augmenting path (cont.)
• Assuming the edge capacities are integers, r is a positive
integer

• On each iteration, the flow value increases by at least 1

• Maximum value is bounded by the sum of the capacities of

the edges leaving the source; hence the augmenting-path
method has to stop after a finite number of iterations

• The final flow is always maximum, its value doesn’t depend on

a sequence of augmenting paths used
Performance degeneration of the method
• The augmenting-path method doesn’t prescribe a specific way
for generating flow-augmenting paths
• Selecting a bad sequence of augmenting paths could impact
the method’s efficiency

• Example 2
2
0/U 0/U

1 0/1 3 U = large positive integer

0/U 0/U
4
Example 2 (cont.)

2 2
0/U 0/U 1/U 0/U

0/1 1 1/1 3 V=1

1 3

0/U 0/U 0/U 1/U

1→2→4→3 4
4
1→4←2→3

2 2
U/U U/U 1/U 1/U
V=2U
1 0/1 3 ●●● 1 0/1 3 V=2

U/U U/U 1/U 1/U

4 4
Requires 2U iterations to reach
maximum flow of value 2U
Shortest-Augmenting-Path Algorithm
• Generate augmenting path with the least number of edges by
BFS as follows.
• Starting at the source, perform BFS traversal by marking new
(unlabeled) vertices with two labels:
• first label – indicates the amount of additional flow that can be
brought from the source to the vertex being labeled
• second label – indicates the vertex from which the vertex being
labeled was reached, with “+” or “–” added to the second label
to indicate whether the vertex was reached via a forward or
backward edge
Vertex labeling
• The source is always labeled with ∞,-

• All other vertices are labeled as follows:

• If unlabeled vertex j is connected to the front vertex i of the
traversal queue by a directed edge from i to j with positive
unused capacity rij = uij –xij (forward edge), vertex j is labeled
with lj,i+, where lj = min{li, rij}

• If unlabeled vertex j is connected to the front vertex i of the

traversal queue by a directed edge from j to i with positive flow
xji (backward edge), vertex j is labeled lj,i-, where lj = min{li, xji}
Vertex labeling (cont.)
• If the sink ends up being labeled, the current flow can be
augmented by the amount indicated by the sink’s first label

• The augmentation of the current flow is performed along the

augmenting path traced by following the vertex second labels
from sink to source; the current flow quantities are increased
on the forward edges and decreased on the backward edges
of this path

• If the sink remains unlabeled after the traversal queue

becomes empty, the algorithm returns the current flow as
maximum and stops
Example: Shortest-Augmenting-Path
Algorithm
5
0/3 0/4

0/2 0/5 0/2

1 2 3 6
2,2+
0/3 0/1 5
0/3 0/4
4
∞,- 2,1+ 2,3+
Queue: 1 2 4 3 5 6 0/2 0/5 0/2
↑↑↑↑ 1 2 3 6
2,2+

0/3 0/1
4 3,1+
Augment the flow by 2 (the sink’s first
label) along the path 1→2→3→6
 Example (cont.)
5
0/3 0/4

2/2 2/5 2/2

1 2 3 6
1,2+
0/3 0/1 5
0/3 0/4
4
Queue: 1 4 3 2 5 6 ∞,- 1,3- 1,5+
↑↑↑↑↑ 2/2 2/5 2/2
1 2 3 6
1,4+

0/3 0/1
4 3,1+
Augment the flow by 1 (the sink’s first
label) along the path 1→4→3←2→5→6
 Example (cont.)
5
1/3 1/4

2/2 1/5 2/2

1 2 3 6

1/3 1/1 5
1/3 1/4
4
Queue: 1 4
↑↑ ∞,- 2/2 1/5 2/2
1 2 3 6

1/3 1/1
4 2,1+
No augmenting path (the sink is unlabeled)
the current flow is maximum
Definition of a Cut
• Let X be a set of vertices in a network that includes its source
but does not include its sink, and let X, the complement of X,
be the rest of the vertices including the sink. The cut induced
by this partition of the vertices is the set of all the edges with a
tail in X and a head in X.
• Capacity of a cut is defined as the sum of capacities of the
edges that compose the cut.

• We’ll denote a cut and its capacity by C(X,X) and c(X,X)

• Note that if all the edges of a cut were deleted from the
network, there would be no directed path from source to sink
• Minimum cut is a cut of the smallest capacity in a given
network
Examples of network cuts
• If X = {1} and X = {2,3,4,5,6}, C(X,X) = {(1,2), (1,4)}, c = 5
5
• If X ={1,2,3,4,5} and X = {6}, C(X,X) = {(3,6), (5,6)}, c = 6
3 4
• If X = {1,2,4} and X ={3,5,6}, C(X,X) = {(2,3), (2,5), (4,3)}, c = 9
1 2 2 5 3 2
6

3
1

4
Max-Flow Min-Cut Theorem
• The value of maximum flow in a network is equal to the
capacity of its minimum cut

• The shortest augmenting path algorithm yields both a

maximum flow and a minimum cut:
• maximum flow is the final flow produced by the algorithm
• minimum cut is formed by all the edges from the labeled
vertices to unlabeled vertices on the last iteration of the
algorithm
• all the edges from the labeled to unlabeled vertices are full, i.e.,
their flow amounts are equal to the edge capacities, while all
the edges from the unlabeled to labeled vertices, if any, have
zero flow amounts on them
Time Efficiency
• The number of augmenting paths needed by the shortest-
augmenting-path algorithm never exceeds nm/2, where n and
m are the number of vertices and edges, respectively

• Since the time required to find shortest augmenting path by

breadth-first search is in O(n+m)=O(m) for networks
represented by their adjacency lists, the time efficiency of the
shortest-augmenting-path algorithm is in O(nm2) for this
representation

• More efficient algorithms have been found that can run in

close to O(nm) time, but these algorithms don’t fall into the
iterative-improvement paradigm
Bipartite Graphs
• Bipartite graph: a graph whose vertices can be partitioned into
two disjoint sets V and U, not necessarily of the same size, so
that every edge connects a vertex in V to a vertex in U
• A graph is bipartite if and only if it does not have a cycle of an
odd length

V 1 2 3 4 5

U 6 7 8 9 10
Bipartite Graphs (cont.)
• A bipartite graph is 2-colorable: the vertices can be colored in
two colors so that every edge has its vertices colored
differently

V 1 2 3 4 5

U 6 7 8 9 10
Matching in a Graph
• A matching in a graph is a subset of its edges with the
property that no two edges share a vertex

V 1 2 3 4 5
a matching
in this graph
M = {(4,8), (5,9)}

U 6 7 8 9 10

A maximum (or maximum cardinality) matching is a matching

with the largest number of edges
• always exists
• not always unique
Free Vertices and Maximum Matching

V 1 2 3 4 5 A matched
A matching vertex
in this graph (M)

A free
vertex
U 6 7 8 9 10

For a given matching M, a vertex is called free (or unmatched) if

it is not an endpoint of any edge in M; otherwise, a vertex is said
to be matched
• If every vertex is matched, then M is a maximum matching
• If there are unmatched or free vertices, then M may be able to be improved
• We can immediately increase a matching by adding an edge connecting two
free vertices (e.g., (1,6) above)
Augmenting Paths and Augmentation
• An augmenting path for a matching M is a path from a free
vertex in V to a free vertex in U whose edges alternate
between edges not in M and edges in M
• The length of an augmenting path is always odd
• Adding to M the odd numbered path edges and deleting from
it the even numbered path edges increases the matching size
by 1 (augmentation)
•VOne-edge
1 2 path
3 between
4 5 two free vertices
V 1 is 2special
3 case
4 of5
augmenting path

U 6 7 8 9 10 U 6 7 8 9 10

Augmentation along path 2,6,1,7

Augmenting Paths (another example)

V 1 2 3 4 5 V 1 2 3 4 5

U 6 7 8 9 10 U 6 7 8 9 10

Augmentation along
3, 8, 4, 9, 5, 10

• Matching on the right is maximum (perfect matching)

• Theorem A matching M is maximum if and only if there exists
no augmenting path with respect to M
Augmenting Path Method (template)
• Start with some initial matching
• e.g., the empty set

• Find an augmenting path and augment the current matching

along that path
• e.g., using breadth-first search like method

• When no augmenting path can be found, terminate and return

the last matching, which is maximum
BFS-based Augmenting Path Algorithm
• Initialize queue Q with all free vertices in one of the sets (say
V)

• While Q is not empty, delete front vertex w and label every

unlabeled vertex u adjacent to w as follows:
• Case 1 (w is in V)
If u is free, augment the matching along the path ending at u by
moving backwards until a free vertex in V is reached. After that,
erase all labels and reinitialize Q with all the vertices in V that
are still free
If u is matched (not with w), label u with w and enqueue u
• Case 2 (w is in U) Label its matching mate v with w and
enqueue v

• After Q becomes empty, return the last matching, which is

maximum
Example (revisited)

Initial Graph Resulting Graph

V 1 2 3 4 5 V 1 2 3 4 5

U 6 7 8 9 10 U 6 7 8 9 10
1

Queue: 1 2 3 Queue: 1 2 3 Augment

from 6

Each vertex is labeled with the vertex it was reached from. Queue deletions are
indicated by arrows. The free vertex found in U is shaded and labeled for clarity;
the new matching obtained by the augmentation is shown on the next slide.
Example (cont.)

Initial Graph Resulting Graph

6 8

V 1 2 3 4 5 V 1 2 3 4 5

U 6 7 8 9 10 U 6 7 8 9 10
2 1 3

Queue: 2 3 Queue: 2 3 6 8 1 4 Augment

from 7
Example (cont.)

Initial Graph Resulting Graph

6 8

V 1 2 3 4 5 V 1 2 3 4 5

U 6 7 8 9 10 U 6 7 8 9 10
3 3 4 4

Queue: 3 Queue: 3 6 8 2 4 9 Augment

from 10
Example: maximum matching found
• This matching is maximum since there are no remaining free
vertices in V (the queue is empty)
V 1 2 3 4 5
• Note that this matching differs from the maximum matching
found earlier

U 6 7 8 9 10

maximum
matching
Notes on Maximum Matching Algorithm
• Each iteration (except the last) matches two free vertices (one
each from V and U). Therefore, the number of iterations
cannot exceed n/2 + 1, where n is the number of vertices in
the graph. The time spent on each iteration is in O(n+m),
where m is the number of edges in the graph. Hence, the time
efficiency is in O(n(n+m))

• This can be improved to O(sqrt(n)(n+m)) by combining

multiple iterations to maximize the number of edges added to
matching M in each search

• Finding a maximum matching in an arbitrary graph is much

more difficult, but the problem was solved in 1965 by Jack
Edmonds
Conversion to Max-Flow Problem
• Add a source and a sink, direct edges (with unit capacity) from
the source to the vertices of Vsand1 from the vertices of U to
1 1 1 1
the sink
V 1 2 3 4 5
• Direct all edges from V to U with unit capacity
1
1
1 1
1 1 1 1
1 1

U 6 7 8 9 10

1 1 1 1
1
t
Stable Marriage Problem
• There is a set Y = {m1,…,mn} of n men and a set X =
{w1,…,wn} of n women. Each man has a ranking list of the
women, and each woman has a ranking list of the men (with
no ties in these lists).
• A marriage matching M is a set of n pairs (mi, wj).
• A pair (m, w) is said to be a blocking pair for matching M if
man m and woman w are not matched in M but prefer each
other to their mates in M.
• A marriage matching M is called stable if there is no blocking
pair for it; otherwise, it’s called unstable.
• The stable marriage problem is to find a stable marriage
matching for men’s and women’s given preferences.
Instance of the Stable Marriage Problem
• An instance of the stable marriage problem can be specified
either by two sets of preference lists or by a ranking matrix, as
in the example below.
• men’s preferences women’s preferences
• 1st 2nd 3rd 1st 2nd 3rd
• Bob: Lea Ann Sue Ann: Jim Tom Bob
• Jim: Lea Sue Ann Lea: Tom Bob Jim
• Tom: Sue Lea Ann Sue: Jim Tom Bob

• ranking matrix {(Bob, Ann) (Jim, Lea) (Tom, Sue)} is unstable

{(Bob, Ann) (Jim, Sue) (Tom, Lea)} is stable
• Ann Lea Sue
• Bob 2,3 1,2 3,3
• Jim 3,1 1,3 2,1
Stable Marriage Algorithm (Gale-Shapley)
• Step 0 Start with all the men and women being free

• Step 1 While there are free men, arbitrarily select one of them
and do the following:
Proposal The selected free man m proposes to w, the next
woman on his preference list
• Response If w is free, she accepts the proposal to be
matched with m. If she is not free, she compares m with her
current mate. If she prefers m to him, she accepts m’s
proposal, making her former mate free; otherwise, she simply
rejects m’s proposal, leaving m free

• Step 2 Return the set of n matched pairs

Example

Ann Lea Sue

Bob 2,3 1,2 3,3

Bob proposed to Lea
Jim Free
3,1men:1,3 2,1
Bob, Jim, Tom Lea accepted
Tom 3,2 2,1 1,2

Ann Lea Sue

Bob 2,3 1,2 3,3

Free men: Jim proposed to Lea
Jim, Tom Jim 3,1 1,3 2,1 Lea rejected

Tom 3,2 2,1 1,2

Example (cont.)

Ann Lea Sue

Bob 2,3 1,2 3,3

Jim proposed to Sue
Jim Free
3,1men:1,3 2,1
Jim, Tom Sue accepted
Tom 3,2 2,1 1,2

Ann Lea Sue

Bob 2,3 1,2 3,3

Free men: Tom proposed to Sue
Tom Jim 3,1 1,3 2,1 Sue rejected

Tom 3,2 2,1 1,2

Example (cont.)

Ann Lea Sue

Bob 2,3 1,2 3,3

Tom proposed to Lea
Jim Free
3,1men:1,3 2,1
Tom Lea replaced Bob
Tom 3,2 2,1 1,2 with Tom

Ann Lea Sue

Bob 2,3 1,2 3,3

Free men: Bob proposed to Ann
Bob Jim 3,1 1,3 2,1 Ann accepted

Tom 3,2 2,1 1,2

Analysis of the Gale-Shapley Algorithm
• The algorithm terminates after no more than n2 iterations with
a stable marriage output

• The stable matching produced by the algorithm is always

man-optimal: each man gets the highest rank woman on his
list under any stable marriage. One can obtain the woman-
optimal matching by making women propose to men

• A man (woman) optimal matching is unique for a given set of

participant preferences

• The stable marriage problem has practical applications such

as matching medical-school graduates with hospitals for
residency training
Chapter 11
Limitations of
Algorithm Power

Copyright © 2007 Pearson Addison-Wesley. All rights reserved.

Lower Bounds
• Lower bound: an estimate on a minimum amount of work
needed to solve a given problem

• Examples:
• number of comparisons needed to find the largest element in
a set of n numbers
• number of comparisons needed to sort an array of size n
• number of comparisons necessary for searching in a sorted
array
• number of multiplications needed to multiply two n-by-n
matrices
Lower Bounds (cont.)
• Lower bound can be
• an exact count
• an efficiency class ()

• Tight lower bound: there exists an algorithm with the same

efficiency as the lower bound

• Problem Lower bound Tightness

• sorting (nlog n) yes
• searching in a sorted array (log n) yes
• element uniqueness (nlog n) yes
• n-digit integer multiplication (n) unknown
• multiplication of n-by-n matrices (n2) unknown
Methods for Establishing Lower Bounds
• trivial lower bounds

• information-theoretic arguments (decision trees)

• adversary arguments

• problem reduction
Trivial Lower Bounds
• Trivial lower bounds: based on counting the number of items
that must be processed in input and generated as output

• Examples
• finding max element

• polynomial evaluation

• sorting

• element uniqueness

• Hamiltonian circuit existence

Decision Trees
• Decision tree — a convenient model of algorithms involving
• comparisons in which:
• internal nodes represent comparisons
• leaves represent outcomes

• Decision tree for 3-elementabcinsertion sort

yes no
a<b

abc bac
yes no yes no
b< c a<c

a<b<c acb bca

b<a<c
yes no yes no
a<c b<c

a<c<b c<a<b b<c<a c<b<a

Decision Trees and Sorting Algorithms
• Any comparison-based sorting algorithm can be represented
by a decision tree

• Number of leaves (outcomes)  n!

• Height of binary tree with n! leaves  log2n!

• Minimum number of comparisons in the worst case  log2n!

for any comparison-based sorting algorithm

• log2n!  n log2n

• This lower bound is tight (mergesort)

Adversary Arguments
• Adversary argument: a method of proving a lower bound by
• playing role of adversary that makes algorithm work the
hardest
• by adjusting input

• Example 1: “Guessing” a number between 1 and n with

yes/no
• questions
• Adversary: Puts the number in a larger of the two subsets
generated by last question

• Example 2: Merging two sorted lists of size n

• a1 < a2 < … < an and b1 < b2 < … < bn
Lower Bounds by Problem Reduction
• Idea: If problem P is at least as hard as problem Q, then a
lower
• bound for Q is also a lower bound for P.
• Hence, find problem Q with a known lower bound that can
be reduced to problem P in question.

• Example: P is finding MST for n points in Cartesian plane

Q is element uniqueness problem (known to be in (nlogn))
Classifying Problem Complexity
• Is the problem tractable, i.e., is there a polynomial-time
(O(p(n)) algorithm that solves it?

• Possible answers:
• yes (give examples)

• no
• because it’s been proved that no algorithm exists at all
(e.g., Turing’s halting problem)
• because it’s been be proved that any algorithm takes
exponential time

• unknown
Problem Types: Optimization and Decision
• Optimization problem: find a solution that maximizes or
minimizes some objective function

• Decision problem: answer yes/no to a question

• Many problems have decision and optimization versions.

• E.g.: traveling salesman problem

• optimization: find Hamiltonian cycle of minimum length
• decision: find Hamiltonian cycle of length  m

• Decision problems are more convenient for formal

investigation of their complexity.
Class P
• P: the class of decision problems that are solvable in O(p(n))
time, where p(n) is a polynomial of problem’s input size n

• Examples:
• searching

• element uniqueness

• graph connectivity

• graph acyclicity

• primality testing (finally proved in 2002)

Class NP
• NP (nondeterministic polynomial): class of decision problems
whose proposed solutions can be verified in polynomial time =
solvable by a nondeterministic polynomial algorithm

• A nondeterministic polynomial algorithm is an abstract two-

stage procedure that:
• generates a random string purported to solve the problem
• checks whether this solution is correct in polynomial time
• By definition, it solves the problem if it’s capable of generating
and verifying a solution on one of its tries

• Why this definition?

• led to development of the rich theory called “computational
complexity”
Example: CNF satisfiability
• Problem: Is a boolean expression in its conjunctive normal
form (CNF) satisfiable, i.e., are there values of its variables
that makes it true?

• This problem is in NP. Nondeterministic algorithm:

• Guess truth assignment
• Substitute the values into the CNF formula to see if it
evaluates to true

• Example: (A | ¬B | ¬C) & (A | B) & (¬B | ¬D | E) & (¬D | ¬E)

• Truth assignments:
•ABCDE
•00000
What problems are in NP?
• Hamiltonian circuit existence
• Partition problem: Is it possible to partition a set of n integers
into two disjoint subsets with the same sum?
• Decision versions of TSP, knapsack problem, graph coloring,
and many other combinatorial optimization problems. (Few
exceptions include: MST, shortest paths)

• All the problems in P can also be solved in this manner (but

no guessing is necessary), so we have:
• P  NP

• Big question: P = NP ?
NP-Complete Problems
• A decision problem D is NP-complete if it’s as hard as any
• problem in NP, i.e.,
• D is in NP
• every problem in NP is polynomial-time reducible to D

NP problems

NP-complete
problem

• Cook’s theorem (1971): CNF-sat is NP-complete

NP-Complete Problems (cont.)
• Other NP-complete problems obtained through polynomial-
• time reductions from a known NP-complete problem

NP problems

known
NP-complete
problem
candidate
for NP -
completeness

• Examples: TSP, knapsack, partition, graph-coloring and

hundreds of other problems of combinatorial nature
P = NP ? Dilemma Revisited
• P = NP would imply that every problem in NP, including all
NP-complete problems, could be solved in polynomial time
• If a polynomial-time algorithm for just one NP-complete
problem is discovered, then every problem in NP can be
solved in polynomial time, i.e., P = NP
NP problems

NP-complete
problem

• Most but not all researchers believe that P  NP , i.e. P is a

proper subset of NP
Chapter 12
Coping with the
Limitations of
Algorithm Power

Copyright © 2007 Pearson Addison-Wesley. All rights reserved.

Tackling Difficult Combinatorial Problems
• There are two principal approaches to tackling difficult
combinatorial problems (NP-hard problems):

• Use a strategy that guarantees solving the problem exactly

but doesn’t guarantee to find a solution in polynomial time

• Use an approximation algorithm that can find an approximate

(sub-optimal) solution in polynomial time
Exact Solution Strategies
• exhaustive search (brute force)
• useful only for small instances

• dynamic programming
• applicable to some problems (e.g., the knapsack problem)

• backtracking
• eliminates some unnecessary cases from consideration
• yields solutions in reasonable time for many instances but
worst case is still exponential

• branch-and-bound
• further refines the backtracking idea for optimization problems
Backtracking
• Construct the state-space tree
• nodes: partial solutions
• edges: choices in extending partial solutions

• Explore the state space tree using depth-first search

• “Prune” nonpromising nodes

• dfs stops exploring subtrees rooted at nodes that cannot lead to
a solution and backtracks to such a node’s parent to continue
the search
Example: n-Queens Problem

• Place n queens on an n-by-n

chess board so that no two
of them are in the same row,
column, or diagonal

1 2 3 4
1 queen 1
2 queen 2
3 queen 3
4 queen 4
State-Space Tree of the 4-Queens Problem
Example: Hamiltonian Circuit Problem

0
with 3 w/o 3

3 0
a b with 5 w/o 5 with 5 w/o 5

8 3 5 0
c f w/o 6 with 6 w/o 6 with 6 w/o 6 X
with 6
0+13<15

14 8
d e with 7 w/o 7
9 3 11 5
X X X X X
14+7>15 9+7>15 3+7<15 11+7>14 5+7<15
15 8
solution X
8<15
Branch-and-Bound
• An enhancement of backtracking

• Applicable to optimization problems

• For each node (partial solution) of a state-space tree,

computes a bound on the value of the objective function for all
descendants of the node (extensions of the partial solution)

• Uses the bound for:

• ruling out certain nodes as “nonpromising” to prune the tree – if
a node’s bound is not better than the best solution seen so far
• guiding the search through state-space
Example: Assignment Problem

Select one element in each row of the cost matrix C so that:

• no two selected elements are in the same column
• the sum is minimized

Example
Job 1 Job 2 Job 3 Job 4
Person a 9 2 7 8
Person b 6 4 3 7
Person c 5 8 1 8
Person d 7 6 9 4

Lower bound: Any solution to this problem will have total cost
at least: 2 + 3 + 1 + 4 (or 5 + 2 + 1 + 4)
Example: First two levels of the state-space
tree
Example (cont.)
Example: Complete state-space tree
Example: Traveling Salesman Problem
Approximation Approach
• Apply a fast (i.e., a polynomial-time) approximation algorithm
to get a solution that is not necessarily optimal but hopefully
close to it

• Accuracy measures:
• accuracy ratio of an approximate solution sa
• r(sa) = f(sa) / f(s*) for minimization problems
• r(sa) = f(s*) / f(sa) for maximization problems
• where f(sa) and f(s*) are values of the objective function f for
the approximate solution sa and actual optimal solution s*

• performance ratio of the algorithm A

• the lowest upper bound of r(sa) on all instances
Nearest-Neighbor Algorithm for TSP
• Starting at some city, always go to the nearest unvisited city,
and, after visiting all the cities, return to the starting one
• AB 1
•
• sa : A – B – C – D – A of length 10
6 2
3 3
• s* : A – B – D – C – A of length 8
• DC
1

• Note: Nearest-neighbor tour may depend on the starting city

• Accuracy: RA = ∞ (unbounded above) – make the length of

AD
arbitrarily large in the above example
Multifragment-Heuristic Algorithm
• Stage 1: Sort the edges in nondecreasing order of weights.
Initialize the set of tour edges to be constructed to
empty set

• Stage 2: Add next edge on the sorted list to the tour, skipping
those whose addition would’ve created a vertex of
degree 3 or a cycle of length less than n. Repeat
this step until a tour of length n is obtained

• Note: RA = ∞, but this algorithm tends to produce better tours

than the nearest-neighbor algorithm
Twice-Around-the-Tree Algorithm
• Stage 1: Construct a minimum spanning tree of the graph
(e.g., by Prim’s or Kruskal’s algorithm)

• Stage 2: Starting at an arbitrary vertex, create a path that

goes
twice around the tree and returns to the same vertex

• Stage 3: Create a tour from the circuit constructed in Stage 2

by
making shortcuts to avoid visiting intermediate vertices
more than once

• Note: RA = ∞ for general instances, but this algorithm tends to

produce better tours than the nearest-neighbor algorithm
Example

12
a e

9 9
8 4 7 11

b d
8

6 10

a e

b d
Walk: a – b – c – b – d – e – d – b – a Tour: a – b – c – d – e – a
c
Christofides Algorithm
• Stage 1: Construct a minimum spanning tree of the graph

• Stage 2: Add edges of a minimum-weight matching of all the

odd
vertices in the minimum spanning tree

• Stage 3: Find an Eulerian circuit of the multigraph obtained in

Stage 2

• Stage 3: Create a tour from the path constructed in Stage 2 by

making shortcuts to avoid visiting intermediate vertices
more than once

• RA = ∞ for general instances, but it tends to produce better

tours than the twice-around-the-minimum-tree alg.
Example: Christofides Algorithm

a e

4 4 7 11

12 b d
a e 8

9 9 c
8 4 7 11

b d a e
8

6 10

c 4 9 7 11

b d

c
Euclidean Instances
• Theorem If P ≠ NP, there exists no approximation algorithm
for TSP with a finite performance ratio.
• Definition An instance of TSP is called Euclidean, if its
distances satisfy two conditions:
• 1. symmetry d[i, j] = d[j, i] for any pair of cities i and j
2. triangle inequality d[i, j] ≤ d[i, k] + d[k, j] for any cities i, j, k

• For Euclidean instances:

• approx. tour length / optimal tour length ≤ 0.5(⌈log2 n⌉ + 1)
• for nearest neighbor and multifragment heuristic;
• approx. tour length / optimal tour length ≤ 2
• for twice-around-the-tree;
• approx. tour length / optimal tour length ≤ 1.5
Local Search Heuristics for TSP
• Start with some initial tour
(e.g., nearest neighbor). On
each iteration, explore the C4 C3
current tour’s neighborhood
by exchanging a few edges
in it. If the new tour is
shorter, make it the current C1 C2

tour; otherwise consider

another edge change. If no
change yields a shorter tour,
the current tour is returned
as the output. C4 C3

• Example of a 2-change C1 C2
Example of a 3-change

C5 C4

C6 C3

C1 C2

C5 C4

C6 C3

C1 C2

C5 C4

C6 C3

C1 C2
Empirical Data for Euclidean Instances
Greedy Algorithm for Knapsack Problem
• Step 1: Order the items in decreasing order of relative values:
v1/w1…  vn/wn
• Step 2: Select the items in this order skipping those that don’t
• fit into the knapsack
•
Example: The knapsack’s capacity is 16
• item weight value v/w
• 1 2 $40 20
• 2 5 $30 6
• 3 10 $50 5
• 4 5 $10 2
•
Accuracy
Approximation Scheme for Knapsack
Problem
• Step 1: Order the items in decreasing order of relative values:
v1/w1…  vn/wn
• Step 2: For a given integer parameter k, 0 ≤ k ≤ n, generate all
subsets of k items or less and for each of those that fit the
knapsack, add the remaining items in decreasing
order of their value to weight ratios
• Step 3: Find the most valuable subset among the subsets
generated in Step 2 and return it as the algorithm’s output

• Accuracy: f(s*) / f(sa) ≤ 1 + 1/k for any instance of size n

• Time efficiency: O(knk+1)
• There are fully polynomial schemes: algorithms with
polynomial running time as functions of both n and k
Bin Packing Problem: First-Fit Algorithm
• First-Fit (FF) Algorithm: Consider the items in the order given
and place each item in the first available bin with enough
room for it; if there are no such bins, start a new one

• Example: n = 4, s1 = 0.4, s2 = 0.2, s3 = 0.6, s4 = 0.7

• Accuracy
• Number of extra bins never exceeds optimal by more than
70% (i.e., RA ≤ 1.7)
• Empirical average-case behavior is much better. (In one
experiment with 128,000 bins, the relative error was found
to be no more than 2%.)
Bin Packing: First-Fit Decreasing Algorithm
• First-Fit Decreasing (FFD) Algorithm: Sort the items in
decreasing order (i.e., from the largest to the smallest). Then
proceed as above by placing an item in the first bin in which it
fits and starting a new bin if there are no such bins

• Example: n = 4, s1 = 0.4, s2 = 0.2, s3 = 0.6, s4 = 0.7

• Accuracy
• Number of extra bins never exceeds optimal by more than
50% (i.e., RA ≤ 1.5)
• Empirical average-case behavior is much better, too
Numerical Algorithms
• Numerical algorithms concern with solving mathematical
problems such as
• evaluating functions (e.g., x, ex, ln x, sin x)
• solving nonlinear equations
• finding extrema of functions
• computing definite integrals

• Most such problems are of “continuous” nature and can be

solved only approximately
Principal Accuracy Metrics
• Absolute error of approximation (of * by  )
• | - *|

• Relative error of approximation (of * by  )

• | - *| / |*|
• undefined for * = 0
• often quoted in %
Two Types of Errors
• truncation errors
• Taylor’s polynomial approximation

ex  1 + x + x2/2! + … + xn/n!

absolute error  M |x|n+1/(n+1)! where M = max et for 0  t  x

• composite trapezoidal rule

 f(x)dx  (h/2) [f(a) + 21=i =n-1 f(xi) + f(b)], h = (b - a)/n

b
absolute error  (b-a)h2 M2 / 12 where M2 = max |f(x)| for a x
a
b

• round-off errors
Solving Quadratic Equation
• Quadratic equation ax2 + bx + c = 0 (a 0)
• x1,2 = (-b  D)/2a where D = b2 - 4ac
• Problems:
• computing square root
• use Newton’s method: xn+1 = 0.5(xn + D/xn)

• subtractive cancellation
• use alternative formulas (see p. 411)
• use double precision for D = b2 - 4ac

• other problems (overflow, etc.)

Notes on Solving Nonlinear Equations
• There exist no formulas with arithmetic ops. and root
extractions for roots of polynomials
• anxn + an-1xn-1 ... + a0 = 0 of degree n 5

• Although there exist special methods for approximating roots

of polynomials, one can also use general methods for
• f(x) = 0

• Nonlinear equation f(x) = 0 can have one, many, infinitely

many, and no roots at all

• Useful:
• sketch graph of f(x)
• separate roots
Three Classic Methods
• Three classic methods for solving nonlinear equation
• f(x) = 0
• in one unknown:

• bisection method

• method of false position (regula falsi)

• Newton’s method
Bisection Method
• Based on
• Theorem: If f(x) is continuous on ax b and f(a) and f(b) have
opposite signs, then f(x) = 0 has a root on a < x < b
• binary search idea
f(x)

x
a x1 b

• Approximations xn are middle points of shrinking segments

• |xn - x*|  (b - a)/2n
• xn always converges to root x* but slower compared to others
Example of Bisection Method Application y

• Find the root of f(x) = x3 - x -1

• x³ - x - 1=0
• with the absolute error not . .
0 2
x

larger than 0.01

n an bn xn f(xn)
1 0.0- 2.0+ 1.0 -1.0
2 1.0- 2.0+ 1.5 0.875
3 1.0- 1.5+ 1.25 -0.296875
4 x ≈ 1.3203125
5
6
7
8 1.3125- 1.328125+ 1.3203125 -0.018711
Method of False Position
• Similar to bisection method but uses x-intercept of line
through
• (a, f(a)) and (b, f(b)) instead of middle point of [a,b]
.
f(x)

an
. .xn
. bn
x

• Approximations xn are computed by the formula

• xn = [anf(bn) - bnf(an)] / [f(bn) - f(an)]
Newton’s Method
• Very fast method in which xn’s are x-intercepts of tangent
lines
• to the graph of f(x)

. f(xn )

. .
xn+1 x n
x

• Approximations xn are computed by the formula

• xn+1 = xn - f(xn) / f(xn)