CS 473ug: Algorithms

Chandra Chekuri
chekuri@cs.uiuc.edu
3228 Siebel Center
University of Illinois, Urbana-Champaign

Fall 2007

Chekuri

CS473ug

Longest Increasing Subsequence

Part I
Longest Increasing Subsequence

Chekuri

CS473ug

Longest Increasing Subsequence

Longest Increasing Subsequence

Sequence: an ordered list a1 , a2 , . . . , an . Length of sequence is n
ai1 , ai2 , . . . , aik is a subsequence of a1 , a2 , . . . , an if
1 i1 < i2 < . . . < ik n.
A sequence is increasing if a1 < a2 < . . . < an . It is non-decreasing
if a1 a2 . . . an . Similarly decreasing and non-increasing.

Chekuri

CS473ug

Longest Increasing Subsequence

Longest Increasing Subsequence

Sequence: an ordered list a1 , a2 , . . . , an . Length of sequence is n
ai1 , ai2 , . . . , aik is a subsequence of a1 , a2 , . . . , an if
1 i1 < i2 < . . . < ik n.
A sequence is increasing if a1 < a2 < . . . < an . It is non-decreasing
if a1 a2 . . . an . Similarly decreasing and non-increasing.
Longest Increasing Subsequence
Input A sequence of numbers a1 , a2 , . . . , an
Goal Find an increasing subsequence ai1 , ai2 , . . . , aik of
maximum length

Chekuri

CS473ug

Longest Increasing Subsequence

Example

Chekuri

CS473ug

Longest Increasing Subsequence

A First Recursive Approach

L(i): length of longest common subsequence in first i elements
a1 , a2 , . . . , ai .

Chekuri

CS473ug

Longest Increasing Subsequence

A First Recursive Approach

L(i): length of longest common subsequence in first i elements
a1 , a2 , . . . , ai .

Can we write L(i) in terms of L(1), L(2), . . . , L(i 1)?

Case 1 : L(i) does not contain ai , then L(i) = L(i 1)
Case 2 : L(i) contains ai , then L(i) =?

Chekuri

CS473ug

Longest Increasing Subsequence

A First Recursive Approach

L(i): length of longest common subsequence in first i elements
a1 , a2 , . . . , ai .

Can we write L(i) in terms of L(1), L(2), . . . , L(i 1)?

Case 1 : L(i) does not contain ai , then L(i) = L(i 1)
Case 2 : L(i) contains ai , then L(i) =? What is the element in the
subsequence before ai ? If it is aj then it better be the case
that aj < ai since we are looking for an increasing sequence.

Chekuri

CS473ug

Longest Increasing Subsequence

A First Recursive Approach

L(i): length of longest common subsequence in first i elements
a1 , a2 , . . . , ai .

Can we write L(i) in terms of L(1), L(2), . . . , L(i 1)?

Case 1 : L(i) does not contain ai , then L(i) = L(i 1)
Case 2 : L(i) contains ai , then L(i) =? What is the element in the
subsequence before ai ? If it is aj then it better be the case
that aj < ai since we are looking for an increasing sequence.
Do we know which j? No!

Chekuri

CS473ug

Longest Increasing Subsequence

A First Recursive Approach

L(i): length of longest common subsequence in first i elements
a1 , a2 , . . . , ai .

Can we write L(i) in terms of L(1), L(2), . . . , L(i 1)?

Case 1 : L(i) does not contain ai , then L(i) = L(i 1)
Case 2 : L(i) contains ai , then L(i) =? What is the element in the
subsequence before ai ? If it is aj then it better be the case
that aj < ai since we are looking for an increasing sequence.
Do we know which j? No! So we try all possibilities
L(i) = 1 +

Chekuri

max

j<i and aj <ai

CS473ug

L(j)

Longest Increasing Subsequence

A First Recursive Approach

L(i): length of longest common subsequence in first i elements
a1 , a2 , . . . , ai .

Can we write L(i) in terms of L(1), L(2), . . . , L(i 1)?

Case 1 : L(i) does not contain ai , then L(i) = L(i 1)
Case 2 : L(i) contains ai , then L(i) =? What is the element in the
subsequence before ai ? If it is aj then it better be the case
that aj < ai since we are looking for an increasing sequence.
Do we know which j? No! So we try all possibilities
L(i) = 1 +

max

j<i and aj <ai

Is the above correct?

Chekuri

CS473ug

L(j)

Longest Increasing Subsequence

A First Recursive Approach

L(i): length of longest common subsequence in first i elements
a1 , a2 , . . . , ai .

Can we write L(i) in terms of L(1), L(2), . . . , L(i 1)?

Case 1 : L(i) does not contain ai , then L(i) = L(i 1)
Case 2 : L(i) contains ai , then L(i) =? What is the element in the
subsequence before ai ? If it is aj then it better be the case
that aj < ai since we are looking for an increasing sequence.
Do we know which j? No! So we try all possibilities
L(i) = 1 +

max

j<i and aj <ai

L(j)

Is the above correct? No, because we do not know that L(j) is

a subsequence that actually ends at aj !
Chekuri

CS473ug

Longest Increasing Subsequence

A Correct Recursion

L(i): longest common subsequence in first i elements a1 , a2 , . . . , ai

that ends in ai

Chekuri

CS473ug

Longest Increasing Subsequence

A Correct Recursion

L(i): longest common subsequence in first i elements a1 , a2 , . . . , ai

that ends in ai
L(i) = 1 +

max

j<i and aj <ai

What is the final solution?

Chekuri

CS473ug

L(j)

Longest Increasing Subsequence

A Correct Recursion

L(i): longest common subsequence in first i elements a1 , a2 , . . . , ai

that ends in ai
L(i) = 1 +

max

j<i and aj <ai

What is the final solution? maxni=1 L(i)

How many subproblems?

Chekuri

CS473ug

L(j)

Longest Increasing Subsequence

A Correct Recursion

L(i): longest common subsequence in first i elements a1 , a2 , . . . , ai

that ends in ai
L(i) = 1 +

max

j<i and aj <ai

What is the final solution? maxni=1 L(i)

How many subproblems? O(n)

Chekuri

CS473ug

L(j)

Longest Increasing Subsequence

Table based Iterative Algorithm

Recurrence:
L(i) = 1 +

max

j<i and aj <ai

Iterative algorithm:
for i = 1 to n do
L[i] = 1
for j = 1 to i 1 do
if aj < ai and 1 + L[j] > L[i]
L[i] = 1 + L[j]
Output maxni=1 L[i]

Running Time:

Chekuri

CS473ug

L(j)

Longest Increasing Subsequence

Table based Iterative Algorithm

Recurrence:
L(i) = 1 +

max

j<i and aj <ai

Iterative algorithm:
for i = 1 to n do
L[i] = 1
for j = 1 to i 1 do
if aj < ai and 1 + L[j] > L[i]
L[i] = 1 + L[j]
Output maxni=1 L[i]

Running Time: O(n2 )

Space:

Chekuri

CS473ug

L(j)

Longest Increasing Subsequence

Table based Iterative Algorithm

Recurrence:
L(i) = 1 +

max

j<i and aj <ai

Iterative algorithm:
for i = 1 to n do
L[i] = 1
for j = 1 to i 1 do
if aj < ai and 1 + L[j] > L[i]
L[i] = 1 + L[j]
Output maxni=1 L[i]

Running Time: O(n2 )

Space: O(n)

Chekuri

CS473ug

L(j)

Longest Increasing Subsequence

A DAG Based Approach

Given sequence a1 , a2 , . . . , an create DAG as follows:

for each i there is a node vi
if i < j and ai < aj add an edge (vi , vj )
find longest path

Chekuri

CS473ug

Longest Increasing Subsequence

Dynamic Programming Methods

There is always a DAG of computation underlying every dynamic

program but it is not always clear.

Chekuri

CS473ug

Longest Increasing Subsequence

Dynamic Programming Methods

There is always a DAG of computation underlying every dynamic

program but it is not always clear.
Three ways to come up with dynamic programming solutions
recursion followed by memoization with a polynomial number
of subproblems
bottom up via tables
identify a DAG followed by a shortest/longest path in DAG
But there is always a recursion+memoization that is implicit in the
other methods!

Chekuri

CS473ug

Longest Increasing Subsequence

Dilworths Theorem
Given sequence a1 , a2 , . . . , an the longest increasing subsequence
can be of size 1.

Chekuri

CS473ug

Longest Increasing Subsequence

Dilworths Theorem
Given sequence a1 , a2 , . . . , an the longest increasing subsequence
can be of size 1.
Example: a1 > a2 > a3 > . . . > an

Chekuri

CS473ug

Longest Increasing Subsequence

Dilworths Theorem
Given sequence a1 , a2 , . . . , an the longest increasing subsequence
can be of size 1.
Example: a1 > a2 > a3 > . . . > an
In the above example the longest decreasing sequence is of size n.

Chekuri

CS473ug

Longest Increasing Subsequence

Dilworths Theorem
Given sequence a1 , a2 , . . . , an the longest increasing subsequence
can be of size 1.
Example: a1 > a2 > a3 > . . . > an
In the above example the longest decreasing sequence is of size n.
Question: Given a sequence of length n, can the longest increasing
sequence and the longest decreasing sequence be both small?

Chekuri

CS473ug

Longest Increasing Subsequence

Dilworths Theorem
Given sequence a1 , a2 , . . . , an the longest increasing subsequence
can be of size 1.
Example: a1 > a2 > a3 > . . . > an
In the above example the longest decreasing sequence is of size n.
Question: Given a sequence of length n, can the longest increasing
sequence and the longest decreasing sequence be both small?

Dilworths Theorem Consequence: In any sequence of length n,

either the longest increasing sequence or the longest decresing

sequence is of length b nc + 1.
Exercise: give a sequence of length n in which both the longest
increasing subsequence and longest decreasing susbequence are no

more than b nc + 1.
Chekuri

CS473ug

Edit Distance

Part II
Edit Distance

Chekuri

CS473ug

Edit Distance

Spell Checking Problem

Given a string exponen that is not in the dictionary, how should

a spell checker suggest a nearby string?

Chekuri

CS473ug

Edit Distance

Spell Checking Problem

Given a string exponen that is not in the dictionary, how should

a spell checker suggest a nearby string?
What does nearness mean?
Question: Given two strings x1 x2 . . . xn and y1 y2 . . . ym what is a
distance between them?

Chekuri

CS473ug

Edit Distance

Spell Checking Problem

Given a string exponen that is not in the dictionary, how should

a spell checker suggest a nearby string?
What does nearness mean?
Question: Given two strings x1 x2 . . . xn and y1 y2 . . . ym what is a
distance between them?
Edit Distance: minimum number of edits to transform x into y .

Chekuri

CS473ug

Edit Distance

Edit Distance

Edit Distance: minimum number of edits to transform x into y .

Edit operations:
delete a letter
add a letter
substitute a letter with another letter

Chekuri

CS473ug

Edit Distance

Edit Distance

Edit Distance: minimum number of edits to transform x into y .

Edit operations:
delete a letter
add a letter
substitute a letter with another letter
Why is substitute not delete plus add?
In general different edits can have different costs and using
substitution as a edit allows a single operation as far as distance is
concerned

Chekuri

CS473ug

Edit Distance

Example

Chekuri

CS473ug

Edit Distance

Edit Distance as Alignment

Chekuri

CS473ug

Edit Distance

Edit Distance Problem

Input Two strings x = x1 x2 . . . xn and y = y1 y2 . . . ym over

some fixed alphabet
Goal Find edit distance between x and y : minimum
number of edits to transform x into y

Chekuri

CS473ug

Edit Distance

Edit Distance Problem

Input Two strings x = x1 x2 . . . xn and y = y1 y2 . . . ym over

some fixed alphabet
Goal Find edit distance between x and y : minimum
number of edits to transform x into y
Note: EditDist(x,y) = EditDist(y,x)

Chekuri

CS473ug

Edit Distance

Towards a Recursive Solution

Think of aligning the strings. Can we express the full problem as a
function of subproblems?

Chekuri

CS473ug

Edit Distance

Towards a Recursive Solution

Think of aligning the strings. Can we express the full problem as a
function of subproblems?
Case 1 xn is aligned with ym . Then x1 x2 . . . xn1 is aligned with
y1 y2 . . . ym1

Chekuri

CS473ug

Edit Distance

Towards a Recursive Solution

Think of aligning the strings. Can we express the full problem as a
function of subproblems?
Case 1 xn is aligned with ym . Then x1 x2 . . . xn1 is aligned with
y1 y2 . . . ym1
Case 2a xn is aligned with and ym is to left of . Then x1 x2 . . . xn1
is aligned with y1 y2 . . . yn .

Chekuri

CS473ug

Edit Distance

Towards a Recursive Solution

Think of aligning the strings. Can we express the full problem as a
function of subproblems?
Case 1 xn is aligned with ym . Then x1 x2 . . . xn1 is aligned with
y1 y2 . . . ym1
Case 2a xn is aligned with and ym is to left of . Then x1 x2 . . . xn1
is aligned with y1 y2 . . . yn .
Case 2b ym is aligned with and xn is to left of . Then x1 x2 . . . xn is
aligned with y1 y2 . . . ym1 .

Chekuri

CS473ug

Edit Distance

Towards a Recursive Solution

Think of aligning the strings. Can we express the full problem as a
function of subproblems?
Case 1 xn is aligned with ym . Then x1 x2 . . . xn1 is aligned with
y1 y2 . . . ym1
Case 2a xn is aligned with and ym is to left of . Then x1 x2 . . . xn1
is aligned with y1 y2 . . . yn .
Case 2b ym is aligned with and xn is to left of . Then x1 x2 . . . xn is
aligned with y1 y2 . . . ym1 .
Subproblems involve aligning prefixes of the two strings.
Find edit distance between prefix x[1..i] of x and prefix y [1..j] of y
EditDist(x,y) is the distance between x[1..n] and y [1..m]

Chekuri

CS473ug

Edit Distance

A Recursive Solution
E [i, j]: edit distance between x[1..i] and y [1..j]

Chekuri

CS473ug

Edit Distance

A Recursive Solution
E [i, j]: edit distance between x[1..i] and y [1..j]
Case 1 xi aligned with xj .
E [i, j] = diff (xi , yj ) + E [i 1, j 1]
where diff (xi , yj ) = 0 if xi = yj , otherwise diff (xi , xj ) = 1

Chekuri

CS473ug

Edit Distance

A Recursive Solution
E [i, j]: edit distance between x[1..i] and y [1..j]
Case 1 xi aligned with xj .
E [i, j] = diff (xi , yj ) + E [i 1, j 1]
where diff (xi , yj ) = 0 if xi = yj , otherwise diff (xi , xj ) = 1
Case 2a xi is mapped to

and xi is to right of xj
E [i, j] = 1 + E [i 1, j]

Chekuri

CS473ug

Edit Distance

A Recursive Solution
E [i, j]: edit distance between x[1..i] and y [1..j]
Case 1 xi aligned with xj .
E [i, j] = diff (xi , yj ) + E [i 1, j 1]
where diff (xi , yj ) = 0 if xi = yj , otherwise diff (xi , xj ) = 1
Case 2a xi is mapped to

and xi is to right of xj
E [i, j] = 1 + E [i 1, j]

Case 2b yj is mapped to

and xi is to left of yj
E [i, j] = 1 + E [i, j 1]

Chekuri

CS473ug

Edit Distance

A Recursive Solution

E [i, j] = min{diff (xi , yj )+E [i 1, j 1], 1+E [i 1, j], 1+E [i, j 1]

Base cases:

Chekuri

CS473ug

Edit Distance

A Recursive Solution

E [i, j] = min{diff (xi , yj )+E [i 1, j 1], 1+E [i 1, j], 1+E [i, j 1]

Base cases:
E [i, 0] = i for i 0
E [0, j] = j for i 0
How many subproblems?

Chekuri

CS473ug

Edit Distance

A Recursive Solution

E [i, j] = min{diff (xi , yj )+E [i 1, j 1], 1+E [i 1, j], 1+E [i, j 1]

Base cases:
E [i, 0] = i for i 0
E [0, j] = j for i 0
How many subproblems? O(mn)

Chekuri

CS473ug

Edit Distance

Iterative Solution

What is table?

Chekuri

CS473ug

Edit Distance

Iterative Solution

What is table? E is a two-dimensional array of size (n + 1)(m + 1)

How do we order the computation?

Chekuri

CS473ug

Edit Distance

Iterative Solution

What is table? E is a two-dimensional array of size (n + 1)(m + 1)

How do we order the computation?
To compute E [i, j] need to have computed
E [i 1, j 1], E [i 1, j], E [i, j 1].

Chekuri

CS473ug

Edit Distance

Iterative Solution

Chekuri

CS473ug

Edit Distance

Iterative Solution
for i = 0 to n
E [i, 0] = i
for j = 0 to m
E [0, j] = j
for i = 1 to n do
for j = 1 to m do
E [i, j] = min{diff (xi , yj ) + E [i, j], 1 + E [i 1, j], 1 + E [i, j 1]}

Running Time:

Chekuri

CS473ug

Edit Distance

Iterative Solution
for i = 0 to n
E [i, 0] = i
for j = 0 to m
E [0, j] = j
for i = 1 to n do
for j = 1 to m do
E [i, j] = min{diff (xi , yj ) + E [i, j], 1 + E [i 1, j], 1 + E [i, j 1]}

Running Time: O(nm)

Space:

Chekuri

CS473ug

Edit Distance

Iterative Solution
for i = 0 to n
E [i, 0] = i
for j = 0 to m
E [0, j] = j
for i = 1 to n do
for j = 1 to m do
E [i, j] = min{diff (xi , yj ) + E [i, j], 1 + E [i 1, j], 1 + E [i, j 1]}

Running Time: O(nm)

Space: O(nm)

Chekuri

CS473ug

Edit Distance

Iterative Solution
for i = 0 to n
E [i, 0] = i
for j = 0 to m
E [0, j] = j
for i = 1 to n do
for j = 1 to m do
E [i, j] = min{diff (xi , yj ) + E [i, j], 1 + E [i 1, j], 1 + E [i, j 1]}

Running Time: O(nm)

Space: O(nm)
Can reduce space to O(n + m) if only distance is needed but not
obvious how to actually compute the edits. Can do it with a clever
scheme (see Jeffs notes).
Chekuri

CS473ug

Edit Distance

Example

Chekuri

CS473ug

Edit Distance

Where is the DAG?

one node for each (i, j), 1 = 0 i n, 0 j m.

Edges for node (i, j): from (i 1, j 1) of cost diff (xi , yj ),
from (i 1, j) of cost 1, from (i, j 1) of cost 1
find shortest path from (0, 0) to (n, m)

Chekuri

CS473ug